AI Chatbot Testing and QA: The Pre-Launch Checklist

The golden question set: your AI chatbot testing backbone

A golden set is 50-200 real customer queries, each paired with the expected answer (or the key facts it must contain), scored on every change to the system. Pull the queries from actual tickets and chat logs, cover your top 20 intents proportionally to their volume, keep the typos and sloppy phrasing intact, and include 10-20 questions the bot should not answer, where the correct behavior is a graceful decline and escalation.

Score three-way, pass, partial, fail, because "partially right" is the most dangerous category in support: technically-true-but-incomplete answers generate re-contacts and erode trust. Automate the run (an LLM-as-judge comparing responses to expected facts works well at this scale) but human-review every failure and a 10% sample of passes weekly, since automated judges drift too. The launch gate: 90%+ pass on in-scope questions, 100% correct declines on out-of-scope ones. If your bot is trained on your website and KB, most golden-set failures trace back to missing or contradictory source content, fix the docs, not just the bot.

The adversarial suite: attack it before customers do

Real users will try all of this in week one, some out of malice, most out of boredom. Test each category with 15-30 variations:

Grade by severity: a slightly stiff refusal is cosmetic; an invented refund policy, echoed PII, or a bot trash-talking your own product is a P1 that blocks launch. Give the suite to your support agents to extend, they have heard every manipulation script that exists, and they will find holes your engineers cannot imagine.

Edge cases most teams skip

• • Typos and phonetic spellings. "cancelation," "subscribtion," product names mangled three ways. Retrieval that only matches clean spelling fails 10-15% of real traffic.
• • Mixed and non-English languages. A Spanish question against an English KB should get a Spanish answer grounded in the same sources, or a clean escalation, never English boilerplate.
• • Multi-question messages. "Can I upgrade mid-cycle, what happens to my data, and do you offer refunds?" The bot must address all three or explicitly sequence them, answering only the first reads as evasion.
• • Long pastes. A 2,000-character error log or forwarded email thread. The bot should extract the actual question, not summarize the noise back.
• • Cross-turn references. "And how much does that cost?" three turns later. Context loss here is a top-3 source of nonsense answers.
• • Empty and emoji-only messages. A "👍" or a blank send should not trigger a fallback lecture.

Regression testing: every KB edit is a code change

The most common post-launch failure is silent: someone updates a pricing page or rewrites a returns article, retrieval re-ranks, and 3-5 golden-set answers quietly change, nobody notices until a customer screenshots the contradiction. Treat content like code: wire the golden set to run automatically on every KB publish, prompt tweak, guardrail threshold change, and model version update, and surface an answer diff (old response vs new) for anything that changed. Version your KB so you can roll content back as easily as code. Weekly scheduled runs catch the drift that event-triggered runs miss, model providers update behavior under stable version names more often than release notes admit.

Staged rollout: internal → 10% → full

Stage 1, internal (1 week). The whole company uses the bot for real questions; support agents actively try to break it. Exit gate: golden set at 90%+, zero P1 adversarial failures, handoff flow verified end-to-end including off-hours.

Stage 2 to 10% of live traffic (1-2 weeks). Random assignment, with a visible path to a human for everyone. Watch fallback rate (healthy: 10-20%), bot-CSAT versus your human baseline (within 0.2 points), and read escalated transcripts daily. This is where fallback strategy tuning happens, clarify first, escalate second, never dead-end.

Stage 3 to 50%, then 100%. Scale only when each gate holds for a full week. Keep the human handoff prominent at every stage: hiding it inflates your numbers and torches trust, and you will not learn which queries genuinely need people.

Production monitoring: the test that never ends

Four continuous signals, with alerts configured before launch: fallback rate (a climb past 25% means KB gaps or retrieval regression; a sudden drop may mean the bot started guessing, check it against hallucination symptoms); bot-CSAT (alert on a 0.3-point week-over-week dip); novel intent clusters (new topics you never tested, each one goes into the golden set); and deflection quality, meaning deflection paired with resolution confirmation, not abandonment, see the deflection rate guide for the honest way to count. On top of the dashboards, sample 25-50 full conversations weekly for human review; aggregates hide individual disasters.

The pre-launch checklist

Frequently Asked Questions

Last updated July 2, 2026. Thresholds and gate values reflect common practice across EzyConn deployments and published LLM-evaluation guidance, 2024-2026. View more guides.