AI Voice Agent Latency: What Callers Notice
Where the silence on an AI phone call comes from, how fast people expect a reply, how to measure response time per call, and the fixes that shorten dead air.
Réponse rapide
AI voice agent latency is the gap between the end of the caller's speech and the first audio of the reply. It comes from the phone path, speech recognition and end-of-turn detection, the language model, text-to-speech and the way back. People minimise silence between turns in every language studied, so dead air stands out. Log two timestamps on every call, read the slow end, and test on real phone calls.
The first thing a caller notices about an AI voice agent is rarely its voice. It is the pause. A noticeable silence after a simple question makes the caller wonder if the line dropped, and a caller who wonders starts talking again, which collides with the reply that was about to arrive.
This article explains where that pause comes from, what research says about how fast people take turns, how to measure it on your own calls, and which fixes are worth making. It does not state a DialerBee latency figure, because the site makes no such claim. The one on-call behaviour it relies on is barge-in: the caller can interrupt, and the agent stops and listens.
What latency means on an AI call
Define it from the caller's side. Latency is the time from the end of the caller's speech to the first audio back. It is not the time the model takes, and not the time to finish the sentence. A reply that starts quickly and then streams for ten seconds has low latency and a long answer. A reply that is short and arrives after a long pause has high latency.
Two related quantities are worth keeping apart:
- Response latency: end of caller speech to the first audio of the reply. This is the dead air.
- Answer completion: the time until the whole reply has played. This depends on how long the reply is, which you control by writing shorter sentences.
Where the time goes
Between the caller's last syllable and the agent's first sound, the call passes through several stages. Each adds a little, and the total is what the caller hears.
| Stage | What happens | What makes it slower |
|---|---|---|
| Phone path, inbound | Audio travels from the caller through the carrier to the platform | Distance, extra hops, transcoding |
| End-of-turn detection | The system decides the caller has finished | A long safety wait after silence |
| Speech recognition | Speech becomes text | Waiting for the full utterance, noisy audio |
| The model | Reads the text, retrieves facts, chooses a reply | Long instructions, slow knowledge lookups, tool calls such as a calendar check |
| Text-to-speech | The reply becomes audio | Waiting to synthesise a whole paragraph before playing any of it |
| Phone path, outbound | Audio returns to the caller | Same as inbound, plus jitter buffering |
Two stages matter most for the fixes below. End-of-turn detection is a deliberate wait, so it is a setting rather than a speed limit. The model stage is where tool calls happen, and a turn that waits on a tool call, such as a calendar check, is usually slower than one that does not.
How fast people take turns
Callers carry a lifetime of expectations from talking to people. A study published in the Proceedings of the National Academy of Sciences, Stivers and colleagues (2009), measured turn-taking in conversation across 10 different languages, from traditional indigenous communities to major world languages. They found that all of the languages showed a general avoidance of overlapping talk and a minimisation of silence between turns. The languages did differ in their average gap between turns, but within a range of 250 ms from the cross-language mean (Stivers et al., 2009; see Sources).
Read this carefully. It does not set a target for a machine, and nothing here says a particular delay is acceptable on a call. What it supports is narrower and useful: in human talk, the gap between turns is kept short across cultures, and the differences between languages are small. A long silence from an agent is therefore noticeable to a caller in any language, including a caller who has never used a voice bot.
Network delay
The phone path adds delay before and after every other stage. The ITU-T publishes Recommendation G.114, titled "One-way transmission time". The ITU page lists the 05/03 edition as in force, together with an amendment that adds guidance on one-way delay for Voice over IP. The page checked for this article gave no millisecond limits, so no figure is quoted here. If you need the limits, read the Recommendation itself (see Sources).
What you can say without a number: delay accumulates, and the phone path is not under the agent's control the way the model is. Delay from a distant media server, from transcoding between codecs, or from a jitter buffer all add to the silence the caller hears. See jitter, and for the quality score that summarises a call's audio, MOS. Delay before the call even rings is a different measure, covered in post-dial delay. When audio goes missing in one direction altogether, use the one-way audio troubleshooting guide.
End-of-turn detection versus cutting callers off
The system must guess when the caller has finished. It has two ways to be wrong.
- Too eager. It decides the turn is over at a pause. The agent starts talking while the caller is still thinking, reading out a number, or searching for a word. The caller is cut off, and has to start again.
- Too cautious. It waits a long time after silence to be sure. Every reply then carries that wait as dead air.
There is no setting that removes both errors. Callers who read out an account number slowly need a patient system, and callers who answer "yes" need a fast one. Treat the wait as a trade-off, tune it against recordings, and watch both kinds of failure.
Why barge-in matters
Barge-in is the safety net. The DialerBee agent vocal IA stops talking when the caller speaks, and listens. This means a wrong guess at the end of a turn is recoverable: if the agent starts early, the caller can speak over it and the agent yields. Without barge-in, a premature reply becomes an announcement the caller must sit through. Barge-in does not shorten the silence, but it limits the damage of the fast settings.
Measure it per call
An average hides the problem. Callers do not experience the average; they experience the one reply that took too long. Log two timestamps for every turn and compute the gap.
- End of speech: when the caller stopped talking.
- First audio back: when the agent's first audio was sent to the caller.
Store the difference per turn, with the call, the version and the question type. Then look at the distribution:
- The median shows the typical turn.
- The slow end, such as the slowest tenth of turns, shows what callers complain about.
- Split by question type. A turn that triggers a calendar check will be slower than a greeting, and that is where a spoken fallback helps.
The recording is the cross-check. A logged gap tells you the system's view, and the recording tells you the caller's. If they disagree, believe the recording. This is the same principle as in how to QA AI voice agent calls, where silence is a scored criterion.
Fixes that shorten dead air
Start with a shorter first sentence
The caller hears the first sentence first. If the instructions tell the agent to open with a long preamble, the speech has to be generated before it plays. Tell the agent to answer the question in the first short sentence and add detail after.
Stream speech
If the platform allows it, begin playing audio as the first words of the reply are ready, instead of waiting for the whole reply. This changes the first-audio time directly. Check with your provider how this is handled, since it depends on the stack.
Put timeouts on tool calls and speak a fallback
Calendar checks and lookups are the usual cause of the slowest turns. Give every tool call a timeout. If it passes, the agent should say something honest, such as that it is still checking, or offer to take a message, rather than stay silent. A fixed fallback sentence is better than ten seconds of nothing.
Use fillers only where they help
A brief spoken acknowledgement before a known slow step tells the caller the line is alive. Used on every turn, fillers add their own delay and quickly sound mechanical. Apply them to the specific steps you measured as slow, and check by ear.
Keep the instructions and lookups lean
A very long instruction block and a broad knowledge search both add to the model stage. Trim what the agent does not need on every call. See AI voice API versus AI contact center platform for who owns each stage when you assemble the stack yourself, and who does not.
Reduce the phone path
Avoid needless transcoding and long routes where you can, and check how the call reaches the media server. If callers use browser phones rather than desk phones, the access path differs; see Téléphones WebRTC versus SIP.
Test on real phone calls, not a browser demo
A web demo skips the carrier, the codecs and the handset. It can feel quicker than the real thing, and it will not show the delay a caller on a mobile network hears. Before go-live, and after every change, place real calls through the same route callers use, in the languages you serve, and listen. DialerBee's own Agent Studio offers test calls for this reason: the ear is the acceptance test.
For a checklist that includes silence and interruption in Arabic, see the Arabic voice agent testing checklist.
Foire aux questions
What is latency on an AI voice call?
It is the time from the moment the caller stops speaking to the moment the first audio of the agent's reply reaches the caller's ear. It is the sum of several steps: the phone path, speech recognition and end-of-turn detection, the language model, text-to-speech and the return path. Measure it per call, not as a single average.
How fast should an AI voice agent reply?
Human conversation sets the expectation. A study of 10 different languages found that speakers minimise the silence between turns, and that language averages sit within 250 ms of the cross-language mean. This site does not give a target for DialerBee, so set your own limit from listened-to calls and watch the slow end, not just the average.
Why does the agent sometimes cut me off?
The system decided your turn had ended when you had only paused. Ending the turn sooner makes replies faster but risks interrupting people who pause to think or to read out a number. Ending it later is safer but adds dead air. Tune it against recorded calls, and keep barge-in on so the caller can always interrupt.
Does the phone network add delay?
Yes. Every network hop, codec and device adds some. ITU-T Recommendation G.114, titled One-way transmission time, covers this delay, and an amendment to it adds guidance on one-way delay for Voice over IP. The ITU page checked for this article did not state millisecond figures, so none are quoted here; read the Recommendation itself for its limits.
Are fillers like um a good fix?
Only where they help. A short spoken acknowledgement before a slow tool call, such as a calendar check, tells the caller the agent is working. Used before every reply, fillers add their own delay and sound repetitive. Test with real callers.
Why test on a phone call and not in the browser?
A browser demo skips the phone path: the carrier, the codecs and the caller's handset. It can feel faster than a real call. Always place the test through the same route callers use, then listen.
Sources
- Stivers T, Enfield NJ, Brown P, et al. Universals and cultural variation in turn-taking in conversation. Proceedings of the National Academy of Sciences, 2009; 106(26): 10587-10592. doi.org/10.1073/pnas.0903616106; full text at europepmc.org/article/PMC/PMC2705608 (abstract retrieved via the Europe PMC service, checked November 7, 2026)
- ITU-T Recommendation G.114, One-way transmission time (edition 05/03 listed as in force, with Amendment 1, "Guidance on one-way delay for Voice over IP"). itu.int/rec/T-REC-G.114 (consulté le 7 novembre 2026)
Lectures complémentaires
Articles connexes
Prêt à voir DialerBee en action ?
Demandez une démo en direct de 15 minutes, ou démarrez un essai gratuit de 14 jours — sans diaporama, sans engagement. Les appels démarrent dès que votre propre trunk SIP ou PBX est connecté et approuvé.
Essai gratuit de 14 jours · crédit IA gratuit · sans carte · 11 langues · BYOC · contrôles qui contribuent à la conformité