Buyer's guide · Scoring matrix
AI Voice Agent Evaluation Matrix
Twenty criteria, each with what to test, how to test it on a pilot, and the red flags. The matrix is blank on purpose: you score vendors from your own calls, not from anyone's marketing, ours included.
Quick answer
How do you evaluate an AI voice agent? Run a short pilot on your own calls and score each vendor on the same criteria: telephony integration, BYOC, SIP and SBC interoperability, inbound and outbound handling, barge-in, response latency measured as median and p95, tool calling, knowledge grounding, human handoff, CRM write-back, observability, auditability, languages and Arabic dialects, security, deployment, data residency, pricing transparency and support. For each criterion, test it yourself, force failures on purpose, and score only what you observed. Do not rely on published scores for any vendor.
How do you use the matrix?
An AI voice agent demo is a performance rehearsed on a call the vendor chose. The matrix replaces it with evidence from calls you chose. Before you contact any vendor:
- Decide the call types. Score agents on the calls you actually plan to automate. If you have not chosen them, work through AI or Human? Which Calls to Automate first.
- Set the weights first. Give each criterion a weight from 1 to 3 before any pilot starts. A clinic booking line might weight tool calling and Arabic dialects at 3 and outbound at 1; a collections team might do the opposite.
- Score 0 to 3 from evidence. 3 means observed working on your calls; 2 means working with minor issues; 1 means partly working or only shown in a demo; 0 means not working or not available. Multiply by the weight and add up.
- Mark the deal-breakers. Some red flags end the evaluation whatever the total, such as an agent that confirms actions that failed, or AI outbound that bypasses your DNC list.
For the written questions to send before the pilot, use the AI voice agents section of the Contact Center RFP Checklist.
The evaluation matrix
Blank and reusable. Print one copy per vendor. Scroll sideways on a phone.
| Criterion | What to test | How to test it on a pilot | Red flags | Weight (1–3) | Score (0–3) |
|---|---|---|---|---|---|
| Telephony integration | Whether the agent runs on the same telephony as your human agents, so it can take transfers, use your queues and present your caller IDs. | Route one real DID to the agent and place a transfer from the agent to a human queue and back. Check the caller ID your test phone sees on outbound. | The agent only works on a separate number or app; transfers drop context or require a new call. | ||
| BYOC | Whether the agent can carry calls over your existing carrier or SIP trunk. | Run the pilot on your own trunk, not the vendor's demo trunk. Compare audio quality and call setup on both. | Pilot only possible on the vendor's carrier; per-minute fees on calls over your own trunk that were not in the quote. | ||
| SIP and SBC | Interoperability with your PBX or SBC: signalling, codecs, DTMF, and how failures are handled. | Have your telephony engineer review a call trace. Test DTMF entry, hold, and a far-end hang-up mid-sentence. | Nobody on the vendor side can read a SIP trace; one-way audio or DTMF issues dismissed as edge cases. | ||
| Inbound | How the agent answers, identifies the caller, and routes or resolves the request. | Call in twenty times with your top five real reasons for calling, from different phones, and log what happens on each. | Works on the demo script only; asks the caller for information the system already has. | ||
| Outbound | Whether the agent can run outbound campaigns with the same list, pacing, calling-hours and DNC controls as human campaigns. | Run a small outbound list to internal test numbers only. Include one number on your DNC list and one outside calling hours; both must be blocked. | AI outbound bypasses the controls that apply to human campaigns; no record of why a call was or was not placed. | ||
| Barge-in | Whether a caller can interrupt and the agent stops, listens and responds to the interruption. | Interrupt in the middle of the greeting, mid-sentence and mid-number. Try a short interjection and a cough; neither should derail the call. | Agent keeps talking over the caller, or stops at every background noise. | ||
| Latency | Response time from the end of the caller's speech to the first audible word of the reply. Measure the median and the p95, not just an average. | Measure from your own recordings of pilot calls, on your trunk, in your languages. Report median and p95 per language and per call type, including turns that call a tool. | Vendor quotes a single figure but cannot say how it was measured; long silences on turns that look up data; latency shown only from a lab, not your calls. | ||
| Tool calling | Whether the agent can take real actions (book, reschedule, look up an order, log a payment promise) and confirms the action succeeded before telling the caller. | Book, move and cancel test appointments, then check the target system. Force a failure, such as a full slot, and listen to what the caller is told. | Agent says “done” when the action failed; actions not visible in your system afterwards. | ||
| Knowledge grounding | Whether answers come from your approved content, and what the agent does when the answer is not there. | Ask ten questions your documents answer and five they do not. Ask for a price that is not in your price list. | Invents an answer or a price rather than saying it does not know and offering a person. | ||
| Human handoff | When and how the agent hands over to a person, and what the person receives. | Ask for a human directly, sound upset, and raise a topic the agent is not allowed to handle. Check the human sees the transcript and reason. | Caller has to repeat everything; no handoff outside business hours and no callback offered; agent argues with a request for a person. | ||
| CRM write-back | Whether outcomes, summaries, recordings and transcripts land in your CRM against the right record. | After each pilot call, open the CRM record. Check the outcome, summary and links are present and attached to the correct contact. | Results stay in the vendor's console; write-back needs a custom project quoted separately. | ||
| Observability | What you can see while it runs: live calls, errors, latency, handoff rate, failed tool calls. | Ask your supervisor to find, without vendor help, the three worst calls of the pilot and the reason each went wrong. | Only aggregate dashboards; no per-call recording and transcript; errors visible only to the vendor. | ||
| Auditability | Whether you can reconstruct what the agent said and did on any call, and which configuration version was live at the time. | Pick a call from a week ago. Retrieve its recording, transcript, actions taken, and the agent version that handled it. | Configuration changes are not versioned; no way to tell which prompt or knowledge base answered a past call. | ||
| Languages | Which languages work in production on real calls, including language switching mid-call. | Use native speakers from your own team for every language you need. Switch language halfway through one call. | Language list taken from the speech provider's catalogue, not from tested calls; accent breaks recognition. | ||
| Arabic dialects | Whether the agent understands and answers in the dialect your callers use (for example Gulf, Levantine, Egyptian), not only Modern Standard Arabic. | Have callers from each dialect you serve make the same requests. Ask them whether the replies sounded natural or like a translation. | Replies in formal Arabic to a dialect caller; numbers, dates or names misheard in dialect. | ||
| Security | Encryption, tenant isolation, access control and audit logs, plus how AI providers process your call data. | Run your standard security questionnaire. Ask which AI providers receive audio or text and whether your data trains shared models. | Certifications described as held that are only planned; no answer on where audio is processed; single sign-on described vaguely with no roadmap date. | ||
| Deployment | How the agent is configured, tested, published and rolled back, and who on your team can do it. | Have your own admin change the greeting, publish it, test it, and roll it back without vendor help. | Every change needs a vendor ticket; no test environment; no rollback. | ||
| Data residency | Where audio, transcripts, recordings and AI processing are located, and whether that can be chosen. | Get the answer in writing for each data type, including the AI model provider, and check it against your regulator's requirements. | Residency stated for the database only, while audio goes to an AI provider in another country. | ||
| Pricing transparency | How usage is billed (per minute, per call, per token), what is included, and whether you can cap spend. | Ask for an invoice estimate for the pilot volume and compare it with the actual pilot bill. | Pilot price differs from production price; charges for failed or transferred calls not disclosed; no spend cap. | ||
| Support | Who helps when something breaks, in which hours and languages, and how fast. | Raise one real support ticket during the pilot and time the first response and the resolution. | Pilot supported by the sales engineer, production by a queue you never met; no escalation path. | ||
| Weighted total (sum of weight × score) | |||||
Do not score competitors from their websites, and do not ask a vendor to fill in this matrix for you. Every cell should be filled in by your team from calls you placed, heard or reviewed.
How should you measure latency?
Latency is the criterion vendors most often answer with a single number, and the one buyers most often accept without checking. Measure it yourself:
- Define the gap. From the moment the caller stops speaking to the first audible word of the reply, as heard on the caller's phone. Not the time the model took, and not the time measured inside the vendor's server.
- Use your recordings. Take the recordings of your pilot calls and measure the gap on every agent turn, or on a fixed sample of turns.
- Report median and p95. The median is the typical reply. The p95 is the slowest one in twenty, and it is usually where callers start saying “hello?”.
- Split it. Report separately per language and for turns where the agent calls a tool, such as checking a calendar. Those turns are slower, and they matter most.
- Use your network. Measure on your own trunk and your callers' networks. Latency on a vendor's demo line tells you about the demo line.
Then set your own threshold, before you see any vendor's result, for the median and the p95 you will accept on your calls.
Two weeks
How do you run a 2-week AI voice agent pilot?
Run the same plan with every vendor on your shortlist, ideally in parallel, so the conditions are the same.
- 1
Before day 1
Agree the scope in writing
Pick one or two call types from the AI good fit list in the AI or Human framework. Write down what success means: the share of calls fully resolved, the share correctly handed to a person, the latency you will accept at median and p95, and the errors that end the pilot immediately. Agree that the pilot runs on your own numbers, your own trunk where possible, and your own content.
- 2
Days 1 to 3
Configure and test internally
Load your knowledge content and price list, connect the calendar or CRM the agent must write to, and set the handoff destination. Your own staff, not the vendor, make test calls covering the criteria above. Fix configuration issues before any customer hears the agent.
- 3
Days 4 to 5
Run the scripted test set
Run a fixed set of test calls against every vendor in the same way: the same requests, the same interruptions, the same failures forced on purpose, the same dialects. Record every call. This is the like-for-like part of the comparison.
- 4
Days 6 to 12
Live traffic, small and supervised
Route a small, defined slice of real calls to the agent, for example one DID or one hour of the day, with a supervisor listening and able to take over. Keep the kill switch with your team. Review a sample of recordings every day and log every failure by type.
- 5
Days 13 to 14
Measure and score
Calculate median and p95 response latency from recordings, resolution and handoff rates, and failed actions. Fill in the matrix from what you observed. Then ask the people who handled the handoffs and the callers you can reach what they thought: the caller's experience outranks any dashboard.
Ground rules for the live part of the pilot
- Outbound pilot calls go only to people who have agreed to receive them, inside your calling hours, with your DNC list applied.
- Tell callers they are speaking with an AI agent where your rules or your own policy require it, and always offer a person.
- Keep the switch that stops the agent in your team's hands, not only the vendor's.
- Stop the pilot at once if the agent confirms an action that did not happen, or gives advice it was not configured to give.
Where does DialerBee fit?
We publish no scores for ourselves on this matrix, for the same reason we publish none for anyone else. What we can tell you is what to look for when you pilot our AI voice agent, Layla, so you can check it:
- It runs on the same platform as human agents: inbound calls routed from a DID, outbound AI campaigns, and handoff to a human agent.
- Callers can interrupt (barge-in), answers come from your knowledge base, and only prices in your service catalogue can be spoken.
- Every candidate reply passes guardrails before text-to-speech, and actions such as bookings must return a confirming status before the agent says they are done.
- Arabic dialect handling covers Gulf, Levantine and Egyptian Arabic, plus ten more languages.
- Each AI call produces a result with recording and transcript, with a push into the CRM. Agent versions can be published and rolled back, and kill switches stop an agent, a campaign or all AI calling.
- It is billed by usage, on managed model keys or your own keys. SOC 2 Type II is planned, not held, and single sign-on is available; the Trust Center lists the status of each item.
More detail is on the AI voice agent and Agent Studio pages. Then put it through the matrix like anyone else.
Questions
Evaluating AI voice agents: FAQ
What should I measure when comparing AI voice agents?
Why median and p95 latency instead of an average?
Why does this matrix not include scores for any vendor?
How long should an AI voice agent pilot run?
What is the biggest red flag in an AI voice agent pilot?
Should we test Arabic dialects separately?
Pilot it on your own calls
Book a demo to plan a pilot around your call types and languages, or start a 14-day free trial of the platform.