Why Milliseconds Define the AI Voice Experience
In human conversation, the average gap between one person finishing a sentence and another responding is 200-300 milliseconds. This timing is so deeply ingrained that even slight deviations feel unnatural. Add 500ms and the conversation feels sluggish. Add a full second and it feels broken — like talking to someone on a bad satellite connection.
This is the fundamental challenge of voice AI: the system needs to hear what was said, understand it, decide how to respond, generate the response text, convert it to speech, and begin playing audio — all within a window that feels instantaneous to the caller.
Most voice AI systems take 1.5-3 seconds to respond. Callers notice. They start talking over the AI, repeat themselves, or simply decide the system is broken. The experience feels robotic not because of voice quality, but because of timing.
The Latency Budget
To achieve natural conversation timing, we work with a strict latency budget. Every millisecond matters, and every component in the pipeline must be optimized:
Speech-to-Text (STT): The time from the caller finishing their sentence to having a text transcription ready. This includes endpoint detection (knowing the caller has stopped speaking) and transcription processing.
Natural Language Understanding: Processing the transcribed text to determine intent, extract entities, and decide what action to take.
Response Generation: Producing the appropriate response — whether from a knowledge base lookup, a business logic decision, or a language model.
Text-to-Speech (TTS): Converting the response text into natural-sounding audio using our TTS provider, Cartesia.
Network Transit: Moving audio packets between the caller's phone, our infrastructure, and back.
Each component has a budget measured in tens of milliseconds. Exceed the budget in any single component and the entire experience degrades.
Architectural Decisions That Shave Milliseconds
Streaming Everything
The naive approach processes each step sequentially: wait for complete transcription → process intent → generate full response → synthesize all audio → play. This waterfall adds latency at every handoff.
Our approach streams at every stage. Transcription begins producing partial results while the caller is still speaking. Intent recognition starts on partial transcripts. Response generation streams tokens as they're produced. TTS begins synthesizing the first sentence while later sentences are still being generated.
This pipelining means the caller hears the first word of the response while the system is still generating the last word. The perceived latency drops dramatically because the caller doesn't wait for the complete response to be ready.
Edge-Optimized Endpoint Detection
Knowing when a caller has finished speaking is surprisingly difficult. Pause too briefly and you cut them off mid-thought. Wait too long and you add unnecessary latency to every turn.
We use a multi-signal approach: acoustic energy levels, linguistic completeness (is the sentence grammatically finished?), prosodic cues (falling intonation typically signals completion), and contextual expectation (if we asked a yes/no question, a single word is likely complete).
This detection runs at the edge, close to the caller, rather than waiting for audio to traverse the network to a central processing location. Shaving network round-trips from endpoint detection alone saves meaningful time.
Predictive Response Preparation
For many conversational patterns, the likely response is predictable before the caller finishes speaking. If a caller says "I'd like to schedule an appointment for—" we can begin preparing the scheduling flow before hearing the specific date.
This speculative execution means that by the time the caller finishes their sentence, much of the response preparation is already complete. If the prediction was wrong, we discard the prepared response and generate the correct one — but the prediction is right often enough to meaningfully reduce average latency.
TTS Optimization with Cartesia
Text-to-speech is often the largest single contributor to latency in voice AI systems. Traditional TTS engines process the entire text before producing any audio. We use Cartesia's streaming synthesis, which begins producing audio from the first few words while later portions are still being processed.
The result is that audio playback begins within milliseconds of the response text being available, rather than waiting for the complete utterance to be synthesized.
Connection Persistence
Every new network connection adds latency for handshaking and setup. Our architecture maintains persistent connections between all pipeline components. The STT service, NLU engine, response generator, and TTS service are always connected and ready — no cold-start delays when a new utterance arrives.
The Perception Gap
Technical latency (measured in milliseconds from end-of-speech to start-of-response) and perceived latency (how fast the conversation feels) are different things. Several techniques exploit this gap:
Filler acknowledgments. A quick "mm-hmm" or "let me check that" buys processing time while signaling to the caller that they've been heard. This mirrors natural human behaviour — people often produce brief acknowledgments while formulating their full response.
Progressive disclosure. Starting the response with a brief framing phrase ("Great question—") while the substantive answer is still being generated. The caller perceives immediate engagement even though the full answer takes slightly longer.
Prosodic continuity. When the AI's response audio begins, its initial prosody (rhythm and intonation) signals that more is coming. This prevents the caller from jumping in during the brief pauses between streamed audio chunks.
What Happens When Latency Is Too High
We've extensively tested how different latency levels affect caller behaviour and satisfaction:
Under 500ms: Conversation feels natural. Callers interact as they would with a human. Completion rates are high, satisfaction is high.
500ms-1 second: Noticeable but tolerable. Some callers start speaking over the AI. Slight increase in "hello? are you there?" interruptions.
1-2 seconds: Clearly unnatural. Callers frequently interrupt, repeat themselves, or ask if the system is working. Abandonment rates increase measurably.
Over 2 seconds: Conversation breaks down. Callers treat the interaction as broken. Most either hang up or immediately request a human agent.
The threshold for "natural" isn't a hard line, but our data consistently shows that staying under 500ms keeps the conversation in the zone where callers don't consciously notice they're talking to AI.
Infrastructure Choices
Achieving consistent low latency requires infrastructure decisions that prioritize speed:
Geographic distribution. Processing happens close to the caller. We maintain infrastructure in multiple regions so that a caller in New York isn't waiting for packets to travel to a distant data center and back.
Dedicated compute. AI inference shares no resources with other workloads. There are no noisy-neighbor effects or queue delays from unrelated processing competing for the same hardware.
Redundant paths. If any single component experiences a latency spike, traffic automatically routes to a backup. The caller never experiences the degradation — the system self-heals faster than a single conversational turn.
Warm models. Language models and TTS models are always loaded and ready. There's no cold-start penalty for the first call of the day or after a quiet period.
Measuring and Monitoring
You can't optimize what you don't measure. Every call produces detailed latency telemetry:
- End-of-speech detection time
- Transcription completion time
- Intent classification time
- Response generation time (first token and complete)
- TTS first-audio time
- Total turn latency (end-of-caller-speech to start-of-AI-audio)
We track these at the 50th, 95th, and 99th percentiles. Optimizing average latency isn't enough — a system that's fast 95% of the time but has 3-second spikes 5% of the time still frustrates callers regularly.
The Ongoing Challenge
Low latency isn't a problem you solve once. It's a continuous engineering discipline:
Model improvements often add latency. Larger, more capable language models produce better responses but take longer to generate them. Every model upgrade requires re-validating that latency budgets are maintained.
Feature additions can introduce delays. Adding a new capability (like real-time database lookups or CRM integration) means adding a new component to the pipeline with its own latency contribution.
Scale changes affect performance. What works at 100 concurrent calls may not work at 1,000. Infrastructure must scale without sacrificing per-call latency.
Network conditions vary. Callers on poor cellular connections, VoIP systems, or international routes introduce variable latency that the system must accommodate.
Why This Matters for Business Outcomes
Ultra-low latency isn't just a technical achievement — it directly impacts business metrics:
Higher completion rates. Callers who experience natural conversation timing stay on the line longer and complete their intended action (booking, enquiry, purchase).
Better caller satisfaction. Post-call surveys consistently show that perceived "naturalness" correlates more strongly with satisfaction than any other factor — including whether the AI resolved their issue.
Reduced "press 0" behaviour. When the AI responds quickly and naturally, callers don't frantically try to reach a human. They engage with the AI as a capable conversational partner.
The goal isn't just fast AI — it's AI that feels like a natural conversation. Every millisecond we shave from response latency moves the experience closer to that standard.