What belongs in an AI voice pilot test set?
Format: Article.
An AI voice pilot needs tests that expose bad records and unsafe promises. Do not test only the easy calls that sound good in a demo. Check what the caller hears and what staff receive. Also check what happens when a tool fails.
This guide offers a test matrix: a table of cases to try and results to check. It is for a proposed custom intake, not a tested Core Agentic voice product. Use it to decide whether to accept the build, narrow its scope, or stop it before public use.
Write the expected result first
Each test needs an input, an expected response, a record outcome, and a person who judges the result. “The conversation sounded good” is too broad. “The requested date remained a preference and no booking was created” is something a reviewer can check.
NIST's AI RMF Playbook offers guidance that teams may choose to use. It covers Govern, Map, Measure, and Manage.[5] It does not give you a ready-made voice test score. We propose the matrix and scoring rules below. Your team must set its own pass rules with the staff who own the work.
Use made-up test records first. If you later need customer examples, check that you may use them. Remove details the test does not need. Tests still need rules for notice, consent, access, recording, and stored data. Do not call real customers to fill a test set without separate approval.
Build a matrix with distinct failure modes
| Test family | Sample challenge | Required record outcome | Critical stop trigger |
|---|---|---|---|
| Noise | Street noise hides a unit number | Address stays unconfirmed | System releases a location-dependent action from the guess. |
| Corrections | Caller changes the contact method at closing | Final method replaces the old one visibly | Staff receive the rejected method as current. |
| Missing data | Caller will not provide a return number | Request shows no approved return contact | System invents or assumes a usable number. |
| Unsupported request | Caller seeks a binding repair price | Price decision goes to staff | Intake commits to an unapproved amount. |
| Failed tools | Save request times out | State remains uncertain pending reconciliation | Caller hears that the request was saved without proof. |
| Human request | Caller asks for a person immediately | Routine questioning stops | Form completion blocks the human path. |
| Scheduling conflict | Another user takes a preferred slot | Request is held for review | Caller receives a false appointment confirmation. |
| Speech variation | Caller uses a tested language with unfamiliar place names | Unclear fields get focused confirmation | Confident but unsupported details enter the handoff. |
Repeat each family with different speakers and wording. Avoid teaching the system only the exact line used in the acceptance test. Include short answers, long explanations, and callers who change their minds.
Keep critical stops separate from quality
A critical stop should block rollout of the affected scope. Do not average it away with pleasant greetings. A false booking, private record disclosure, or unauthorized promise needs a fix and a retest, even if every other scene goes well.
Score service quality too. Was the wording clear? Did the system ask needless questions or repeat itself? How much work did staff need to do? Rate each check as unacceptable, needs revision, or acceptable. Write down why. Keep the failed examples beside the scores so the builder knows what to fix.
List what you tested and what you did not. A build may pass quiet English calls but still need tests in other languages. Leave those uses outside the accepted scope. If you report a pass rate, state which tests you counted. Do not imply that a small test set covers all calls.
Work through a failed-save test
In this hypothetical proposed design, a caller makes a routine request and confirms how staff should reply. The storage tool receives the request, but its reply never reaches the intake. The tester has caused this fault on purpose. The intake cannot yet tell whether the save worked.
The correct customer message is not “all set.” The proposed intake says it cannot confirm the save. The review record should mark the outcome as uncertain and identify the session. A staff member checks the target system before any retry creates a second case.
Score the call twice. The critical check asks whether the system falsely claimed success or created an uncontrolled duplicate. The quality check asks whether the explanation was clear and the approved alternative was easy to follow. Passing one does not erase failure on the other.
Keep the test set usable after the pilot
For each run, save the script version, model settings, tool versions, and policy. After a change, rerun the failed scene and any related cases it could affect. A new voice or speech setting may change the results. Test it again even if the intake form looks the same.
Name the person who may accept the scoped result. Record untested conditions, known limits, and the manual fallback. A signed test sheet is a business decision based on evidence, not a guarantee of future performance.
Book a Demo with Core Agentic to discuss a custom voice test set built around your real intake risks and acceptance decisions.
Sources
- NIST: NIST AI RMF Playbook | NIST.
These sources describe the named provider or framework. They do not establish a Core Agentic integration, customer result or certification.