How to QA AI Voice Agent Calls: A Scorecard
A QA scorecard for AI voice agent calls: critical fails, weighted criteria, a sampling plan, why listening beats reading, and how findings turn into fixes.
Respuesta rápida
To QA AI voice agent calls, review each call's recording, transcript and outcome against a scorecard with two layers: critical fails that void the call, such as an invented price, a wrong booking or a missed handoff, and weighted criteria such as accuracy, interruptions, silence and name confirmation. Sample every new version, every flagged call and a random set weekly. Listen to the audio, then turn findings into prompt or knowledge fixes.
A human agent's call is reviewed because the agent might forget a script line or lose patience. An AI voice agent fails differently. It rarely forgets, but it can say something fluent and wrong, with total confidence, on many calls before anyone notices. QA for AI calls is less about tone coaching and more about catching the confident error early and finding which setting caused it.
This article gives you a scorecard you can copy, a sampling plan, and the loop from finding to fix. It is written for operators who run an AI agent on live calls, whatever platform is underneath.
Why AI calls need their own QA
You can start from the call center QA scorecard template for human agents, but several of its lines do not transfer.
- Empathy and rapport matter, but the failure you most need to catch is factual: a price, an opening hour, a policy the agent was never given.
- Consistency is a different risk. A human varies from call to call. An AI agent repeats the same mistake on every call that touches the same topic, so one finding can affect hundreds of callers.
- Timing is part of quality. Silence before a reply, and talking over the caller, are scorable defects with no human equivalent.
- The cause is a setting, not a person. The fix is a change to instructions, knowledge or flow, not a coaching conversation.
The same logic applies to the broader outbound call quality assurance routine: sampling, scoring and feedback. The difference is what you feed back to.
What you review: the recording, the transcript and the outcome
Every call by the DialerBee agente de voz de IA leaves a result with three parts, and a QA review uses all three.
| Artículo | What it answers |
|---|---|
| Grabación | How the call sounded: pace, silence, interruptions, pronunciation. See grabación de llamadas. |
| Transcripción | What was said, word for word, and where to search for key phrases. See transcripción. |
| Resultado | What the agent recorded as the result: a booking, an answer, a message, a transfer. |
Compare the three. The most useful single check is whether the outcome matches the conversation: a booking recorded for a call where the caller never confirmed a time is a critical fail that a transcript read would catch only if you compare the two on purpose.
A note on accuracy: a transcript is itself produced by speech recognition, so it can contain errors that the agent did not make. When a transcript looks wrong, check the audio before blaming the agent. For measuring that layer, see how to measure call transcription accuracy (WER).
Critical fails
A critical fail voids the call whatever else was good. Define a short list and keep it short, so reviewers apply it the same way.
- Invented fact or price. The agent stated a figure, hour, policy or promise that is not in your documents or catalogue. The DialerBee agent's price-truth guard is meant to keep spoken prices to your catalogue, but you verify it by listening.
- Wrong booking. The wrong person, service, day or time was booked, or a booking was claimed that was never saved.
- Missed handoff. The caller asked for a person, or showed distress, and was not passed on. See Transferencia a una persona desde el agente de voz con IA.
- No disclosure where required. Where your rules or local law require the caller to be told they are speaking with an automated system, or that the call is recorded, the statement was missing. Check the rules that apply to you, with advice from a lawyer where needed.
- Disclosure of private data. The agent read out another person's details, or details the caller had not been verified for.
Add your own only when a failure could harm a customer or the business. A long list of "critical" items makes every call fail and tells you nothing.
Weighted criteria
The remaining criteria are scored and weighted. The weights below are a starting point to adjust, not a standard.
- Accuracy against the knowledge base. Did every answer match the uploaded document? Highest weight.
- Task completion. Did the call reach the outcome the caller wanted?
- Confirming names and numbers. Did the agent read back a name, date, phone number or amount before acting on it? Names and numbers are easy for speech recognition to mishear, and costly when it does.
- Interruptions. When the caller spoke, did the agent stop? Barge-in is a behaviour of the DialerBee agent, so a call where it kept talking over the caller is worth flagging.
- Silence. Were there long gaps before a reply? Note the time in the recording. The measurement side is covered in AI voice agent latency.
- Language and register. Did the agent answer in the caller's language and the right style? For Arabic, see the Arabic voice agent testing checklist.
- Closing. Did the agent say what happens next, and stop cleanly?
A sampling plan
You will not review every call. Review the calls most likely to teach you something, plus a random set so you do not only see what you expected.
| Sample | What goes in it | Por qué |
|---|---|---|
| New version | Every call, or a large share, in the first day after you promote a version | A change is when new errors appear |
| Flagged calls | Very short calls, very long calls, transfers, repeat contacts, transcripts that contain a request for a person | These are likelier to be failures |
| Random sample | A fixed number of other calls each week, picked at random | Keeps the picture honest and finds the unknown |
| Contained calls | A sample of calls that ended with no transfer | Catches false success; see measuring containment |
Keep the random sample the same size each week, even when volume changes, so you can compare scores over time. Record the version number against each reviewed call, so a drop in score can be traced to the version that caused it.
Listen, don't only read
The transcript is the quick way to find a call and the wrong way to judge it. Some failures do not exist in text:
- A pause long enough that the caller says "hello?"
- The agent starting to answer just as the caller begins a sentence.
- A name that is spelled correctly in the transcript and pronounced wrongly.
- A stiff or rushed delivery that makes a caller hang up.
The caller's ear is the test. A line the system generated is not the same as a line the caller heard, so when a caller or a colleague says a call sounded wrong, trust that over the log. Agent Studio makes the same point for building: place a test call and listen to what the agent actually says, not the script it was given.
From finding to fix
A QA programme earns its keep only when a finding changes the agent. Use a short loop.
- Name the layer. Is the cause in the persona and instructions, in a knowledge base document, or in the flow? A wrong opening hour is a document problem. A rude phrase is an instruction problem.
- Change one thing in Agent Studio. Edit the document or the persona. Change one layer at a time so you know what worked.
- Place a test call. Reproduce the failing question and listen to the answer. Do this on a real phone call rather than only in a browser, since the phone path can add delay and change how audio sounds.
- Publish and promote. Every publish is a version, and promoting it is a deliberate act.
- Review the next sample. Put the new version in your "new version" sample, and confirm the failure is gone.
- Roll back if it got worse. Because versions are kept, rollback is a decision, not a rebuild.
If you want to compare two wordings on live traffic, Agent Studio supports A/B experiments between versions. Judge them on reviewed calls, not only on a headline number. When you are still choosing a platform rather than checking live calls, the matriz de evaluación de agentes de voz con IA is the right tool.
Scorecard template (copy-paste table)
Score each criterion from 0 to the maximum shown. Any critical fail sets the call score to zero. The weights are illustrative; change them to fit your business.
| Sección | Criterio | Max | Notas |
|---|---|---|---|
| Critical (pass/fail) | No invented fact or price | Pass/fail | |
| Critical (pass/fail) | Booking matches what the caller confirmed | Pass/fail | |
| Critical (pass/fail) | Handoff made when asked or needed | Pass/fail | |
| Critical (pass/fail) | Required disclosure given | Pass/fail | |
| Exactitud | Answers match the knowledge base | 30 | |
| Resultado | Reached the result the caller wanted | 20 | |
| Confirmation | Names, numbers and dates read back | 15 | |
| Turn-taking | Stopped when interrupted; no talking over | 10 | |
| Silence | No long gaps before replies | 10 | Note timestamp |
| Idioma | Right language, register and pronunciation | 10 | |
| Closing | Stated next step; ended cleanly | 5 |
Also record: call ID, agent version, call reason, reviewer, the layer at fault (instructions, knowledge, flow, speech recognition, other) and the fix ticket. The "layer at fault" column is what turns a score into a to-do list.
This article is operational guidance. DialerBee does not provide legal advice or guarantee regulatory compliance. Check disclosure and recording rules for each country you call with a qualified adviser.
Preguntas frecuentes
How many AI calls should I review each week?
Enough that one reviewer can finish the set, and the same size every week so results compare. Add every call from a newly promoted version, every flagged call, and a random sample on top. Start small and widen the sample after any change to the prompt, knowledge base or voice, because that is when errors appear.
Is reading the transcript enough?
No. A transcript can read smoothly over a call that sounded wrong: long silences, talking over the caller, a mispronounced name or a stiff tone. Read the transcript to find the exact words, then listen to the recording to judge how the call felt. The caller's ear is the real test.
What should count as a critical fail?
Any error the business cannot absorb: an invented fact or price, a booking made for the wrong person, day or service, a caller who asked for a human and was not handed over, or a missing disclosure where your rules require one. A critical fail scores the call as failed whatever the other marks were.
Does DialerBee score AI calls with its QA scorecards?
The site does not say that its QA scorecards score AI calls. What every AI call leaves behind is a result with its recording, its transcript and its outcome. Use those with a scorecard of your own, such as the table in this article, and record the scores in a sheet or your own QA tool.
How do I turn a QA finding into a fix?
Find which layer caused it: the persona and instructions, the knowledge base document, or the flow. Change that one thing in Agent Studio, place a test call and listen, then promote the new version. Every publish is a version, so if the change makes things worse you can roll back.
Should the same person who built the agent review its calls?
Prefer a second reviewer where you can. The builder knows what the agent was meant to say and tends to hear that, not what it said. If one person must do both, score a blind sample without looking at the instructions first.
Lecturas relacionadas
- Call center QA scorecard template
- Outbound call quality assurance guide
- Arabic voice agent testing checklist
- How to measure call transcription accuracy
- Grabación de llamadas
- Transcripción de llamadas
- Agent Studio
- matriz de evaluación de agentes de voz con IA
Lecturas relacionadas: What Is Prompt Injection?
Artículos relacionados
Diseño de menús IVR: opciones, orden y guiones
Lectura de 11 minutos
OperacionesDevolución de llamada en cola (espera virtual): cómo configurarla
Lectura de 10 minutos
CumplimientoReglas de llamadas de los agentes de recuperación del RBI en India a partir de 2027
9 minutos de lectura
CumplimientoReglas de llamadas de cobranza de la CONDUSEF en México y el REDECO
Lectura de 10 minutos
¿Listo para ver DialerBee en acción?
Solicite una demostración en vivo de 15 minutos o comience una prueba gratuita de 14 días: sin presentaciones, sin compromiso. Las llamadas empiezan en cuanto su propia troncal SIP o PBX está conectada y aprobada.
Prueba gratuita de 14 días · crédito de IA gratuito · sin tarjeta · 11 idiomas · BYOC · controles que apoyan el cumplimiento