How to build and measure an AI assistant for your website

A website assistant is easy to launch and hard to trust. The launch takes an afternoon. Trust comes from a set of decisions made before the first visitor arrives, and from numbers you watch afterwards.

This is the plan we would follow, in the order we would follow it. It builds on what the research says about AI chat.

1. Choose jobs, not a personality

Start by naming two or three jobs a visitor can finish in one sitting. For a services firm, good ones are: work through a problem and leave with a short brief, get a second opinion on a plan, and prepare the case for an internal decision.

Each job should produce something the visitor can use, such as an exhibit, a list of questions or a one-page summary. Hours, locations and simple questions do not need an assistant. Link to the page that answers them.

2. Ground it in your own content

If your whole site fits in a prompt, put it all there. A site of about 20,000 words is roughly 27,000 tokens, which fits easily. Cache that prompt so repeat reads cost a fraction of the normal input price, and skip the vector database until the site is many times larger. A small corpus in context is simpler, cheaper and more accurate than retrieval.

Every claim the assistant makes about your firm should link to the page it came from. If it cannot find one, it should say so.

3. Let the model choose from a menu

Do not let the model write the page. Define a short list of blocks, such as an issue tree, a chart with sliders, a brief and a list of findings, and have the model return structured data that your code validates and draws. The Vercel AI SDK documents this pattern as tool calls that render components.

It keeps every screen on brand, it removes a class of security problems, and it makes the output testable, because each block either matches its schema or it does not.

4. Write the guardrails down

Put them in the prompt and in the tests:

Never invent statistics, clients, people or testimonials.
Say "I don't know" and offer a person when unsure.
Quote only the pricing your site already states.
Give no legal, tax or investment advice, and say nothing about named competitors.
Say it is an AI at the start of every session, and let the visitor ask for a person at any point.

5. Test before every release

Build a release gate of around 120 questions and run it again whenever the prompt, the content or the model changes. A split we would start with: 30 on facts about the firm, 30 on using the methods correctly, 20 on refusals, 20 traps for invented numbers and 20 on knowing when to hand off. Our note on testing an AI assistant before it reaches customers covers how to build and score the set, and Anthropic's guide to evals for AI agents explains why to combine grader types.

6. Control the cost from day one

Four levers do most of the work: cache the site prompt, use a small model for routing and simple turns, limit the number of turns in a session, and set a hard monthly cap that falls back to a person instead of an error.

At October 2026 list prices, our estimate for a typical seven-step session with a cached 30,000-token prompt is about a cent with a small model, about 15 to 20 cents when a mid-size model does the reasoning and a small one handles routing, and about 30 cents with a large model. Those numbers assume a visitor who arrives every few minutes or less, so each session pays to write the cache. Prices change, so measure your own cost per session in a pilot before you set the cap.

7. Measure from a baseline

Agree the numbers before launch, run four weeks, then set targets. We would track:

Sessions started, and the share that reach a first useful output
Briefs produced, and how many are copied or forwarded
Handoffs requested, and how many become real conversations compared with the contact form
The share of visitors who leave after the first answer. One vendor benchmark treats more than 20% as a sign the chat is hurting, so use it as a rough guide, not a target.
Release-gate pass rate, and cost per session and per handoff

8. Roll it out in stages

Run a private prototype first. Have a few colleagues hold about 20 real conversations and read every one. Then launch one job, watch the numbers, and add the next. Promise a reply time on handoffs only if someone will keep it.

Where to start this week

Name the two or three jobs and what each one produces.
Write 40 questions you would want the assistant to pass.
Decide a monthly cost cap.
Choose who receives handoffs and how quickly they reply.

If you want help planning or building one, here is how we approach AI development.