How to Evaluate AI Vendors: A Buyer's Scorecard for Enterprise AI

To evaluate an AI vendor, score twelve things: problem fit, grounding, memory ownership, exit path, evaluation evidence, failure design, human review, security and permissions, integration reality, roadmap dependency, total cost including verification, and support model. Most procurement processes evaluate the demo and the price, then sign a three-year agreement on the strength of a roadmap slide. The demo is the least informative artifact in the entire process, because it was built to be informative about nothing.

This is a buyer's playbook. It assumes you have already selected a problem worth solving and are deciding whether to buy the capability, build it, or do neither.

Before You Evaluate Anyone

Three decisions belong to you, not to a vendor. Make them before the first call or the vendors will make them for you — in a shape that favors their product.

1. Is this infrastructure or advantage?

Run the test: could a competitor buy this capability tomorrow?

If yes, it is infrastructure. Buy it, buy it efficiently, and stop describing it as strategy. Table stakes arrive on the same release schedule for everyone, so the goal is a fair price and low switching cost — not differentiation, which is not available in this aisle at any price.

If no — because it depends on your operational memory, your proprietary process, or data your own operations generate — be very careful about outsourcing the part that makes it yours.

Most AI procurement failures come from buying infrastructure at strategy prices, or from handing the strategic layer to a vendor because it came bundled with the infrastructure.

2. What is the baseline?

If you cannot state the current cost, cycle time, error rate, and review burden of the process, you cannot evaluate any proposal about improving it. Every vendor claim becomes unfalsifiable, and the decision defaults to whoever presents most confidently.

Two weeks of measurement. An agreed definition of an error. Numbers your process owner will stand behind afterward.

3. Where will the memory land?

As the product runs, knowledge accumulates: corrections, feedback, tuned behavior, exception handling, an index over your documents. Decide now whether that lands in a store you control or inside the vendor's product.

This single decision determines your negotiating position at every renewal for the life of the relationship. Decide it before you are dependent, because afterward it is not a negotiation.

The Twelve-Criterion Scorecard

Score each criterion 1 to 5. Weight according to your situation, but do not let a strong score on demo quality compensate for a 1 on exit path.

#	Criterion	1	3	5
1	Problem fit	Generic platform, your case is an example	Configurable to your case	Purpose-built for this problem
2	Grounding	Model answers from general knowledge	Retrieval over your documents	Retrieval with provenance on every output
3	Memory ownership	Accumulates inside their product only	Partial export, proprietary format	Full export, open format, demonstrated
4	Exit path	No migration story	Data export, no configuration export	Full export plus documented migration
5	Evaluation evidence	Benchmarks on public datasets	Case studies with named metrics	Will run your evaluation set and show results
6	Failure design	"The model is very accurate"	Documented failure modes	Failure modes plus configurable stop conditions
7	Human review	Bolted on or absent	Review UI exists	Review UI that captures structured corrections
8	Security and permissions	Answer-level filtering	Retrieval-time permission enforcement	Retrieval-time enforcement plus full audit trail
9	Integration reality	"We have an API"	Documented API, some connectors	Working connectors for your actual systems
10	Roadmap dependency	Core need is on the roadmap	Core need partially shipped	Core need shipped and in production elsewhere
11	Total cost	License only	License plus implementation	License, implementation, and verification cost modeled
12	Support model	Ticket queue	Named contact	Named contact with domain competence

How to read the result

A total is less useful than the pattern. Three specific patterns should stop a purchase regardless of the total:

• Criterion 3 or 4 scores 1. You are agreeing to build institutional knowledge inside something you cannot leave. The price of that is not on the quote.
• Criterion 10 scores 1. You are buying a roadmap. Roadmaps slip, and the feature you need is the one most likely to slip because it is the one that is not built.
• Criterion 5 scores 1 and the vendor will not run your evaluation set. A vendor confident in the product runs your data. A vendor who deflects to public benchmarks is telling you something.

The Exit Test

Ask this in the first meeting, before you are invested enough to soften it:

"If we terminate in year three, what exactly do we take with us, in what format, and can you show me an export from an existing customer?"

The answer tells you more than the rest of the evaluation combined. Four responses, in descending order of quality:

1. A documented export covering source data, configuration, accumulated corrections, and evaluation sets, in open formats, which they will demonstrate. Rare, and worth a great deal.
2. Data export in an open format, configuration not portable. Workable. Price the rebuild cost into the total.
3. "We can provide a data dump." Ask what is in it. Usually the documents you supplied — which you already had — and nothing that accumulated.
4. Discomfort, deflection, or a reference to the account team. That is the answer.

What should be in the export

Asset	Why it matters	Commonly excluded
Source documents	You already own these	No
Retrieval configuration	Represents months of tuning	Frequently
Reviewer corrections	Your highest-value accumulated asset	Almost always
Evaluation sets and results	Required to compare a replacement	Almost always
Prompt and workflow configuration	Represents your process knowledge	Frequently
Usage and quality history	Required to justify the switch internally	Usually

The rows marked "almost always" are the ones that matter most. Negotiate them into the contract at signature, while you still have room to negotiate, in the contract schedule rather than the sales conversation.

Demo Questions That Actually Discriminate

A demo is a controlled environment built by people who know exactly which inputs work. Treat it as a starting point, not evidence.

Eight questions. They take ten minutes and they end weak sales processes early.

"Show me it being wrong." Every system has failure modes. A vendor who has characterized theirs can show you one and explain the handling. A vendor who claims not to have any has not looked.

"Run this on my data, live, right now." Bring three real cases, including one ugly one. Watch what happens to the demo.

"Where did that answer come from?" If the product cannot show provenance for an output, it cannot be audited, and any regulated use is off the table.

"What happens when the source document changes?" Re-indexing cadence, staleness detection, and version handling. Silent staleness is one of the most damaging production failure modes because nothing looks broken.

"How does permission enforcement work — at retrieval or at the answer?" Answer-level filtering means restricted content entered the context window. That is a finding in a security review, and it is common.

"When a reviewer corrects an output, what happens to the correction?" The best question in the list. If the answer is "it improves the model over time," ask how, whether it is your model or a shared one, and whether you can export the corrections. Frequently the honest answer is that the correction is discarded.

"Which of the capabilities you have shown me are generally available today?" Ask it plainly and write down the answer.

"Who else runs this in production on our scale and workflow, and can I speak to their operator rather than their executive sponsor?" The sponsor bought it. The operator runs it. They give different interviews.

The Proof of Concept

If the scorecard clears, run a proof of concept. Most are worthless because they are designed to succeed.

Design rules

Rule	Why
Your data, including the ugly cases	Clean data proves nothing about production
Your evaluation set, built before the PoC	Otherwise success gets defined retroactively
Written pass and fail thresholds, agreed up front	Otherwise the result is negotiated afterward
Your reviewers, not the vendor's team	Review burden is what you are actually measuring
Measured against your baseline	The comparison is the current process, not zero
Fixed end date	Open-ended PoCs become unpaid production
Correction capture running	If it cannot capture corrections now, it will not later

What to measure

Metric	Pass condition
Output quality on your evaluation set	Meets the agreed standard
Review time per case	Below your baseline
Exception rate	Within tolerance
Traceability	Every output traceable to a source
Reviewer sentiment	They would rather use it than not
Corrections captured	Structured and exportable

That last row is a pass criterion. A PoC that produces good outputs and captures nothing has taught your organization nothing, and you will start the next one from the same place.

The honest failure

If review time is not below baseline, the product may still be useful — but it is a drafting assistant, not automation. Those have very different economics. Name it accurately in the business case rather than presenting one as the other, because the difference will surface at the first budget review and it is better to have said it first.

Contract Terms Worth the Argument

Commercial terms are outside this playbook. These five are operational, they are frequently ignored, and each one is materially harder to obtain after signature.

1. Export rights, itemized. Not "data export" — an enumerated list including corrections, configuration, and evaluation sets, in named formats, with a stated turnaround.
2. Model change notification. You are told before the underlying model changes, with a window to re-run your evaluation set. Silent model swaps have broken production workflows with no code change on either side.
3. Evaluation access. The right to run your own evaluation set against the production system on an ongoing basis, not only during the PoC.
4. Data use boundaries. Explicit terms on whether your data, your corrections, or your usage patterns train anything shared with other customers.
5. Named stop authority. Your side can disable the system unilaterally, immediately, without a support ticket.

Numbers 2 and 3 together are the difference between a system you operate and a system that changes underneath you.

When Not to Buy

• The process is not defined. A vendor cannot automate a process you have not articulated. You will pay them to discover it and own the result.
• The memory does not exist yet. If the knowledge the decision depends on is undocumented, capturing it is the project. Buying a product first means the product accumulates your knowledge on its terms.
• A simpler system solves it. A rule engine, a redesigned form, or a well-built report is frequently the correct answer and is dramatically cheaper to own.
• Nobody will own it after launch. An unowned system decays. Do not buy one without a named operator.
• The only argument is that competitors have it. That is an argument for infrastructure spending at infrastructure prices, not for a strategic commitment. Buy the commodity cheaply and put the attention where it compounds.

What Good Procurement Optimizes For

Not the best demo. Not the lowest price. Not the longest feature list.

It optimizes for a position where the capability is available, the knowledge accumulates on your side of the boundary, and switching remains possible at a known cost. That combination keeps the vendor honest for the entire life of the relationship, and it is worth paying somewhat more to obtain.

Rent the capability. Own what accumulates. Then negotiate like someone who can leave.

Frequently Asked Questions

What should you evaluate first when choosing an AI vendor? Whether the capability is infrastructure or advantage. Infrastructure should be bought efficiently with low switching cost. Advantage depends on your operational memory and proprietary process, and outsourcing the layer that makes it yours is the most common expensive mistake in AI procurement.

How do you avoid vendor lock-in with an AI product? Decide before signature where accumulated knowledge lands. Negotiate itemized export rights covering corrections, configuration, and evaluation sets in open formats, and require a demonstrated export rather than a contractual promise.

What is the single best question to ask an AI vendor? "If we terminate in year three, what exactly do we take with us, in what format, and can you show me an export from an existing customer?" It surfaces the dependency structure of the entire relationship in one answer.

How long should an AI proof of concept run? Long enough to see real input variety including edge cases, and to accumulate enough reviewer corrections to be informative — typically six to eight weeks for a transactional workflow. Set a fixed end date at the start; open-ended PoCs turn into unpaid production with no decision.

Should a vendor run a proof of concept on our data? Yes, and you should build the evaluation set before the PoC begins. A vendor who will not run your data and will only show public benchmarks is telling you something about how the product behaves outside a controlled environment.

What does retrieval-time permission enforcement mean and why does it matter? It means access control is applied when documents are retrieved, so restricted content never enters the model's context. The weaker alternative filters the answer after generation, which means restricted content was processed. That distinction matters in any security review and in most regulated environments.

Is it better to build or buy enterprise AI capability? Buy the commodity layers — models, retrieval infrastructure, orchestration tooling — because they improve on someone else's roadmap and no advantage is available there. Own the memory layer and the workflow design, because those are specific to your operations and cannot be purchased by you or by a competitor.