Start with the customer promise
“Add AI to our product” leaves too much undecided. Write the job in terms a customer could check: “Prepare a support reply using our approved policy, with a source a teammate can verify before sending.” Then name the failures that would make the feature unusable. A convincing demo is only one input to this decision.
Keep the present workflow in the comparison. You may discover that a better search page, a clearer form, or an ordinary rule solves the immediate problem. If the requirement is still unclear, the next step is to clarify it before selecting a supplier or architecture.
Separate the model from the workflow
Building an AI feature does not necessarily mean training a model. You can use a hosted model while owning the interface, permitted context, evaluation cases, and review step. Write down which parts each option supplies and which parts your team must maintain.
- Buy a finished application
- Consider it when the available workflow fits. Verify what you can configure, how data enters and leaves, and whether you can inspect failures. Include integration and human review work in the estimate.
- Build a workflow on a model service
- Consider it when a specific behavior matters enough to own. Name who will maintain the integration, test changes, handle unavailable responses, and support users.
- Keep or improve the current process
- Consider it when the volume, evidence, or operating capacity does not justify a new system yet. Define the condition that would make a new evaluation worthwhile.
Use a comparison worksheet with hard gates
Fill this in for each option before testing. Mark an unanswered requirement “unknown” rather than giving it a neutral score. A hard failure should remain visible even if the option performs well elsewhere.
Decision and owner: Customer job and explicit exclusions: Option: buy / build on a service / current workflow Required behavior: Evidence that it works on our permitted test cases: Hard failures and unresolved requirements: Data needed; access and processing permissions: Setup, integration, and migration effort: Recurring service, review, and maintenance effort: Who owns failures and future changes: How we export our data and switch away: Current response: Act / Monitor / Ignore / Clarify Smallest authorized next step: Review date or condition that reopens the choice:
Use the same cases and reviewer rules for the candidates. The AI tool evaluation scorecard explains how to record correctness, unsupported claims, corrections, latency, and review effort. Keep the raw observations beside the worksheet.
Count the work after the first demo
Choose a planning period and show one-time and recurring work separately. Include setup, usage, retries, verification, support, integration fixes, and a possible switch away. Keep uncertain estimates as ranges with their assumptions. Do not compare a supplier’s subscription with only your first afternoon of coding.
Fictional arithmetic: suppose an option costs $120 per month and needs three hours of review. At an assumed internal planning rate of $40 per hour, those two items total $240 per month. That excludes setup, tax, support, and other work. It is a worksheet example, not a vendor quote, a market rate, or evidence of savings.
Also name the work your team would postpone to build or maintain the feature. Do not convert that opportunity into invented revenue. A concrete statement such as “the same engineer would delay the export feature” is useful even when its value is uncertain.
A worked choice: clarify before building
Illustrative scenario: a small team wants policy-grounded support drafts. A finished tool cannot demonstrate the source behind a reply. A custom prototype can display a source but has no assigned maintainer. Neither option has passed the team’s acceptance criteria.
The useful response is “Clarify”: agree on the source-verification requirement and operating owner, then run a bounded test with approved synthetic inputs. Keep the live inbox disconnected. The decision is about the next test, not permission to launch either option. Record that distinction in a decision journal.