How to test an AI assistant before it reaches customers

Most AI assistants get tested the way a new hire gets tested on day one. Someone asks it five questions, likes four of the answers, and calls it ready. That works for a demo. It does not work for something that will answer your customers at two in the morning, when nobody is watching.

What you need is a short, slightly boring routine that runs every time anything changes: the prompt, the documents behind it, or the model underneath. This is the one we use.

Start with questions people actually asked

Pull real questions from your support inbox, chat logs and sales calls. Skip the tidy ones you wish customers asked. Keep the messy ones: two questions in one message, a contract clause quoted by the wrong number, a refund request that is really a complaint about onboarding.

For each question, write down what a good answer has to contain. That is usually the fact itself, the document it comes from, and what the assistant should do next. Sometimes the right next step is to answer. Sometimes it is to ask a follow-up, or to hand the conversation to a person.

A first set of a hundred or so questions is enough to learn a lot. Group them by topic, so you can see when a change helps billing questions and quietly hurts technical ones. On our AI development page, the example release gate uses three groups: billing and refunds, technical setup, and contract and policy.

Decide what "good" means before you score anything

Anthropic's guide to defining success criteria says criteria should be specific and measurable. "Accurate" is hard to count. "Cites the correct policy section on billing questions" can be counted on every run.

For a customer-facing assistant we usually track four numbers:

Correct answers, by topic.
Answers that cite a source, and whether that source says what the answer claims.
Actions the assistant took that it should not have, such as approving a credit above its limit. This one should be zero, every time.
Response time and cost per answer.

Write the thresholds down first. If you set them after you see the scores, you will set them to whatever you got.

Score it more than one way

Some checks are mechanical, and code should do them. Did the answer include a link? Did the assistant try to issue a refund above the limit? Those have yes-or-no answers.

Other checks need judgment. A second model can grade each answer against the expected one you wrote down, and it handles most of the volume well. Anthropic's write-up on evals for AI agents recommends combining grader types for exactly this reason: each one misses things the others catch.

Then a person reads every answer that changed between the old version and the new one. Teams skip this step more than any other, and it catches the strangest failures. It takes less time than you would think, because you only read the differences.

Agree the shipping rule in advance

The rule we write into every project goes roughly like this. A new version ships if no topic gets worse beyond an agreed margin, the count of unsafe actions stays at zero, and every regression has been read and accepted by a named person on your team.

Then roll it out in steps. Send a small share of live traffic to the new version, watch the same numbers on real questions, and keep the old version one click away. If something looks off on a Tuesday afternoon, you switch back and look at it on Wednesday.

Keep the question set alive

Every week, add the questions that went badly in production, with the answer they should have had. The set grows with the product.

After a few months it becomes the most useful thing in the project, more useful than the prompt. It describes what your customers need in a form that any future model can be tested against. New models arrive every few months, and with a good set, trying one is an afternoon of testing instead of a leap of faith.

Where to start this week

Export a hundred recent support conversations.
For each one, write the answer you would want and the document it should come from.
Run your current assistant against them.
Count the misses by topic.

If the count surprises you, better to learn it now than from a customer. If you want help setting this up, here is how we build and test AI assistants.