Back to Blog
Technical

Voice AI latency benchmarks: how fast must an AI phone agent be?

Humans leave about 200 ms between turns. Here is what research, ITU standards and vendor data say about voice AI latency and how fast a phone agent must be.

K

Klevere AI Team

Engineering

17 November 202612 min read

Voice AI latency is the single number that decides whether an AI phone agent feels like a conversation or like a bad satellite link. People are remarkably fast at taking turns: a study of turn-taking across 10 languages found that every language tested showed a general avoidance of overlapping talk and a minimisation of silence between turns. An AI agent that waits too long breaks that rhythm, and callers notice before they can explain why.

This post pulls together the published numbers: what conversation research says about human gaps, what telecoms standards say about delay on a phone line, and what vendors publish about their own speech and model latency. We reconcile the figures, give a realistic target range for a production phone agent, and explain how to measure it on your own calls.

Quick answer

Humans typically leave gaps of around 200 milliseconds between turns, and pauses of 600 to 700 milliseconds or more start to carry meaning, often hesitation or a coming refusal. A good AI phone agent should aim to start speaking within roughly one second of the caller finishing, measured end to end, with faster always better.

Why voice AI latency matters more on the phone than in chat

In a text chat, a two second wait for a reply is barely noticed. On a call, the same two seconds is a long, awkward silence. The reason is that spoken conversation runs on tightly timed turn-taking, and listeners read meaning into the timing itself. Levinson and Torreira's review in Frontiers in Psychology reports that gaps between turns are short, of the order of 200 milliseconds, and that gaps of 700 milliseconds or more are associated with dispreferred actions such as declining an invitation or disagreeing.

That finding has a practical consequence for anyone deploying an AI voice agent. A caller who asks whether a slot is free and hears a long silence may already be bracing for a no. Slow responses also cause callers to start talking again just as the agent begins its answer, which produces overlap, interruptions and repeated questions. Latency is not only a comfort issue; it shapes how the caller interprets what the agent says.

There is a second reason the phone is unforgiving. Unlike a website chatbot, a phone agent has no typing indicator or spinner to show it is working. Silence is the only signal, and silence on a phone line reads as either a dropped call or a struggling system.

What the research says about human turn-taking

The conversation analysis literature gives a clear baseline. The study of 10 languages by Stivers and colleagues, published in PNAS in 2009 and summarised by the Max Planck Institute for Psycholinguistics, found that languages differ in their average gap between turns, but only within a range of 250 milliseconds from the cross-language mean. In other words, the expectation of quick turn-taking holds whether the caller is speaking English, Japanese or Danish, with modest cultural variation around it.

Levinson and Torreira add the striking detail that humans manage these short gaps even though producing speech is slow. Their review notes that language production latencies are over 600 milliseconds, around 600 milliseconds and up just to encode a new word. People hit a 200 millisecond gap because they predict the end of the other person's turn and start planning their reply while still listening.

The same review offers the most useful thresholds for AI design:

  • The modal gap at speaker transitions sits at around 200 milliseconds.
  • Gaps longer than the norm, above about 300 milliseconds, decrease the likelihood of an unqualified acceptance.
  • Gaps of 600 milliseconds or longer generate inferences that an unwelcome response is coming.
  • Gaps of 700 milliseconds or more are associated with dispreferred actions.
  • An AI agent cannot yet predict turn ends as well as a person, so matching a 200 millisecond gap on every turn is unrealistic today. But these thresholds tell you where the discomfort starts and why shaving off even a few hundred milliseconds changes how a call feels.

    What telecoms standards say about delay on the line

    Before any AI processing happens, the phone network itself adds delay. The International Telecommunication Union's recommendation on one-way transmission time, ITU-T G.114, is the long-standing reference here and remains in force. It recommends not exceeding a one-way delay of 400 milliseconds for general network planning, and states that if delays are kept below 150 milliseconds, most applications will experience essentially transparent interactivity. It also notes that highly interactive tasks, including some speech, may be affected by delays below 100 milliseconds.

    These figures matter because they are one way. In a voice AI call, audio travels from the caller to your system and the reply travels back, so network delay is paid twice on every turn. If your telephony provider, speech recognition, language model and speech synthesis sit in different regions, each hop adds more. G.114 was written for human to human calls, but it is a useful reminder that a phone agent's latency budget is consumed partly by the network before your software even starts work.

    Published voice AI latency benchmarks compared

    The table below collects published figures from independent research, a standards body and vendors who state how their number was measured. Each line gives the source, the figure, the year and the scope.

  • Stivers et al., PNAS (2009): average gap between turns varies by language within 250 ms of the cross-language mean. Scope: conversation recordings across 10 languages.
  • Levinson and Torreira, Frontiers in Psychology (2015): modal gap around 200 ms; production latency over 600 ms; gaps of 700 ms or more linked to dispreferred responses. Scope: review of turn-taking and psycholinguistic research.
  • ITU-T G.114 (2003, in force): below 150 ms one-way delay is essentially transparent; do not exceed 400 ms for general planning. Scope: telecoms network planning.
  • OpenAI GPT-4o announcement (2024): audio responses in as little as 232 ms, average 320 ms; earlier Voice Mode pipeline averaged 2.8 s (GPT-3.5) and 5.4 s (GPT-4). Scope: vendor-reported, model response to audio input.
  • ElevenLabs Flash v2.5 model documentation (current): median latency around 100 ms, excluding application and network latency. Scope: vendor-reported text-to-speech model latency.
  • Deepgram text-to-speech docs (current): illustrative request with time to first byte of 616 ms and total latency of 745 ms; network latency grows with distance to the endpoint. Scope: vendor example, explicitly not a fixed result.
  • Google Cloud Speech-to-Text docs (current): 100 ms audio frame size recommended as a trade-off between latency and efficiency. Scope: vendor guidance for streaming recognition.
  • Two things stand out. First, the human baseline and the network standard agree closely: around 150 to 200 milliseconds is where interaction feels instant. Second, vendor component figures look small in isolation, but they measure different things. ElevenLabs' figure explicitly excludes application and network latency. Deepgram's own example shows how a request that synthesises quickly can still take hundreds of milliseconds end to end once the network is included.

    How we compiled this

    We opened every source listed and used only figures stated on that page. Vendor numbers are reported as the vendor describes them, including their caveats about what is and is not measured. We did not combine vendor figures into a single headline number, because they come from different test conditions. None of the figures in this post are Klevere's own measurements.

    Where the delay comes from in an AI phone agent

    Most production phone agents today are either a cascaded pipeline or a speech-to-speech model. In a cascade, the caller's audio is transcribed by speech recognition, the text goes to a language model, and the reply is turned back into audio by text-to-speech. OpenAI described this pattern directly when launching its Realtime API, noting that stitching together speech recognition, a text model and text-to-speech often resulted in loss of emotion, emphasis and accents, plus noticeable latency.

    The GPT-4o announcement put numbers on the gap. The earlier ChatGPT Voice Mode was a pipeline of three separate models with average latencies of 2.8 seconds with GPT-3.5 and 5.4 seconds with GPT-4. GPT-4o, handling audio in a single model, could respond to audio in as little as 232 milliseconds with an average of 320 milliseconds, which OpenAI described as similar to human response time. That is a vendor-reported figure for the model itself, not for a full phone call over a telephone network.

    In practice, the delay a caller hears breaks down into these stages:

  • End of speech detection: the system has to decide the caller has finished, rather than paused for breath. Waiting longer reduces interruptions but adds directly to the gap.
  • Speech recognition: streaming recognisers process audio in small frames. Google's Speech-to-Text guidance recommends a 100 millisecond frame size as a good trade-off, noting that larger frames are more efficient but add latency.
  • Language model response: time to the first token of the reply, which depends on the model, the provider's load, the prompt length and any tool calls the agent makes.
  • Text-to-speech: time to the first byte of audio. Deepgram's latency documentation breaks total latency into network time, time to first byte and audio synthesis.
  • Network and telephony: every hop between the caller's carrier, your telephony provider and each AI service.
  • The biggest wins usually come from streaming every stage so they overlap, keeping all services in the same region as the telephony infrastructure, and avoiding slow tool calls in the middle of a turn. A model that is technically brilliant but slow on the first token can make the whole agent feel worse than a smaller, faster one; our guide to LLM API pricing covers how model choice also affects running cost.

    What latency do callers actually notice

    Putting the research together gives a practical set of bands for end-to-end response time, measured from the moment the caller stops speaking to the moment they hear the agent's first sound.

  • Under about 300 milliseconds: within the range of natural human gaps. Rare for today's phone agents once network delay is included.
  • Around 300 to 700 milliseconds: noticeable to some callers, but the conversation still flows. This sits between the point where Levinson and Torreira report reduced unqualified acceptance and the point where gaps signal a dispreferred response.
  • Around 700 milliseconds to 1 second: clearly slower than a person, but tolerable for most service calls, especially if the agent sounds natural.
  • Beyond 1 second: callers start to talk over the agent, repeat themselves, or ask whether anyone is there. The earlier multi-second pipelines reported by OpenAI sit well into this zone.
  • These bands are our synthesis of the published research rather than a measured standard, so treat them as design targets. The realistic goal for most business phone agents is to keep typical turns under one second, end to end, with the slowest turns not far behind. Averages hide the problem; a caller who hits one four second silence will remember it more than ten fast replies.

    Context also matters. A caller booking a table or checking an order status expects brisk answers. A caller asking a complex question may tolerate a short pause, especially if the agent uses a natural filler such as a brief acknowledgement while it looks something up. Fillers should be honest and short, not a way to hide a slow system.

    How to measure voice AI latency on your own calls

    Vendor benchmarks are a starting point, not a guarantee. Network distance alone changes results: Deepgram's documentation states that network latency increases with the distance between your application and the endpoint, and suggests regional endpoints for users in or near the EU, Australia or India. A stack that performs well from a test machine in one region can be noticeably slower for real callers elsewhere.

    A sensible measurement approach looks like this:

  • Measure end to end on real phone calls, from the end of the caller's speech to the first audible sound of the reply, not just inside your application logs.
  • Log each stage separately: end of speech detection, transcription, model first token, first audio byte and playback start.
  • Report percentiles, not just the average. The median shows typical experience; the 90th and 95th percentiles show the turns that frustrate callers.
  • Test from the regions and carriers your callers actually use, including mobile networks.
  • Re-test after any change to model, prompt length, tools or provider region, since each can shift latency.
  • Track overlap events, where the caller and agent speak at the same time, as a symptom of latency or poor end of speech detection.
  • Latency issues also tend to appear alongside other production problems such as timeouts and failed tool calls. Our post on AI agent failure modes covers how to monitor those together.

    How Klevere approaches voice AI latency

    When we build phone agents, latency is treated as a requirement from day one rather than something to tune after launch. Across 500+ AI agents deployed in 12 industries, the pattern we see is consistent: callers forgive an occasional imperfect answer more readily than a long silence.

    In practice that means choosing the architecture to suit the job. Some use cases suit a speech-to-speech model for the most natural timing; others need a cascaded pipeline because the agent must call booking systems, CRMs or knowledge bases mid-conversation, and the trade-offs need to be weighed openly. We keep services co-located with the telephony layer, stream every stage, set end of speech thresholds per use case, and instrument each stage so latency is reported by percentile from the first test call. For businesses using an AI receptionist, we also design what the agent says while it waits on a slow system, so the caller is never left in silence.

    We are also candid when latency is the wrong thing to optimise. If an agent needs to run several lookups before it can answer, the better design is often to restructure the conversation, gathering details first and confirming later, rather than chasing milliseconds that the backend cannot deliver.

    Frequently asked questions

    What is a good latency for an AI voice agent?

    For most business phone calls, aim for the agent to start speaking within about one second of the caller finishing, measured end to end on real calls. Faster is better: human conversation research shows typical gaps of around 200 milliseconds, and pauses of 600 to 700 milliseconds or more begin to signal hesitation or an unwelcome reply to the listener.

    How fast do humans respond in conversation?

    Research across many languages finds that people typically leave gaps of around 200 milliseconds between turns. A study of 10 languages found that every one avoided overlapping talk and minimised silence, with average gaps varying only within 250 milliseconds of the cross-language mean. People achieve this by predicting when the other speaker will finish.

    Why do AI phone agents feel slow?

    Delay builds up across several stages: detecting that the caller has finished, transcribing speech, generating a reply with a language model, synthesising audio and sending it across the network. Each stage may be fast on its own, but in a pipeline the delays add up, and the phone network adds delay in both directions on every turn.

    Is speech-to-speech AI faster than a speech-to-text pipeline?

    Often, yes. OpenAI reported that GPT-4o could respond to audio in an average of 320 milliseconds, compared with multi-second averages for its earlier three-model Voice Mode pipeline. That is a vendor figure for the model, not a full phone call. Pipelines remain useful when you need specific speech models, voices or tight control over each step.

    How do you measure voice AI latency?

    Measure from the moment the caller stops speaking to the moment they hear the first sound of the reply, on real phone calls from the regions your callers use. Log each stage separately and report percentiles as well as averages, because occasional long silences shape how callers feel about the whole call far more than the typical turn.

    Does network location affect voice AI latency?

    Yes. Network delay grows with distance between your application and each service it calls, and vendors such as Deepgram recommend regional endpoints for this reason. The ITU guidance on one-way transmission time treats delay under 150 milliseconds as essentially transparent, so keeping telephony and AI services in the same region protects a large share of your budget.

    If you are planning a phone agent and want to know whether your systems can support fast, natural calls, start with a free AI audit. We will look at your call volumes, the systems the agent would need to reach, and where latency is likely to be won or lost before anything is built.

    Ready to implement AI in your business?

    Let's discuss how AI agents can transform your operations and reduce costs.