How can I test a website AI assistant before customers use it?
A friendly demonstration is a useful start, but it does not show how an assistant handles missing information or a customer changing their mind. Before launch, prepare a small set of fictional conversations with written expectations. The following examples are a suggested testing worksheet, not results from our demos.
1. Define the customer’s task
Choose one job, such as helping a visitor prepare a catering enquiry. Write down what the visitor should leave with: an outline of their event and a clear next step for contacting the business. GOV.UK’s guidance on identifying user needs recommends starting with a person’s task and defining acceptance criteria. Applied here, that means deciding what a useful conversation must achieve before judging how polished it sounds.
2. Separate answers from actions
A website presents information and routes visitors. A retrieval-based chatbot finds relevant material from an approved knowledge collection to help answer questions. An agent with actual tool permissions may also perform specified actions; those permissions need separate checks. Do not assume a conversational interface can book appointments, take payment, access calendars or send email. Write its permitted capabilities at the top of your worksheet so every reviewer tests against the same scope.
3. Write questions and expected behaviour together
For each test, record the fictional question, the approved source, what a satisfactory reply includes and what would make it fail. Try these catering examples:
“We need lunch for 18 people at a workshop.” Expect help gathering missing requirements, without claiming availability.
“Can you confirm next Thursday?” Expect a clear explanation that a person must confirm the date, plus the agreed contact route.
“Actually, make that 28 people.” Expect the revised number to replace the earlier one in any enquiry summary.
“Your website doesn’t mention evening delivery. Can you promise it?” Expect the assistant to acknowledge the missing information and refer the question to the business.
Use fictional details throughout. You do not need private customer conversations to build an initial checklist.
4. Check substance before tone
Mark each reply against four questions: does it match the approved information, address the visitor’s task, respect its permissions and explain the next step? Record failures plainly: “invented delivery availability” is more useful than “felt wrong”. Treat an unsupported booking confirmation as a reason to fix and retest before launch. Anthropic’s evaluation guidance supports defining test inputs and success criteria, and repeating trials because replies can vary. This worksheet is a small manual starting point, not proof of reliability.
5. Test a whole conversation
Continue beyond the opening answer. Change a requirement, ask for a summary, then ask how to reach a person. Check that the summary preserves the correction and does not imply anything has been sent. On our site, SousChef is a real build for the founder’s catering business; SpecBot is a live specialist catalogue demonstration. Neither is independent client endorsement. SousChef cannot confirm bookings, and SpecBot’s product details need supplier confirmation. Explore the examples under Meet the agents using fictional questions.
6. Keep the worksheet for future changes
Save the questions, expected behaviour, test date and issues to resolve. Repeat relevant checks when approved information or the assistant changes. Add cases when you discover a new failure. To discuss a first version, use Create my project brief and include the task, knowledge sources, limits and test questions. The brief downloads in your browser; it does not send an enquiry.