Definitions

Definitions

The budget

Results

1. Streaming each stage saved about 600 ms at the median in the model, more than most other single changes.
2. Every knob has a cost on the other side: faster endpointing cuts people off; a noise filter delays interruptions.
3. A truthful history matters: an agent that thinks it said what the caller never heard will mislead them later.

Design rules

• Set a turn-gap budget (for example p50 under a second, p95 under two) and measure each stage.
• Stream every stage and start speaking at the first sentence.
• Tune endpointing on real calls; consider a semantic end-of-turn signal and treat the cut-in rate as a metric.
• Support barge-in: stop audio, flush buffers, cancel generation, and write the truncated reply to history.
• Require a short minimum speech duration and echo cancellation to avoid self-interruption.
• Write for the ear: short turns, grouped numbers, read-back of critical details, offers to continue.
• Confirm irreversible actions out loud before executing them.
• Play fillers for slow tools, and never block silently for seconds.
• Plan a human handoff and a text follow-up (email or SMS) for dense information.
• Disclose AI use and recording as the law and your policy require; log with care for privacy (Level 10 module 6).

Interview questions, with the follow-ups

Common mistakes

• Batch pipeline: waiting for the full reply before speaking.
• Chasing the shortest endpoint until the agent cuts people off.
• Ignoring barge-in so callers talk over a monologue.
• Leaving the full reply in history after an interruption.
• Replies written for reading: long, with lists and URLs.
• Reading numbers as one long string.
• No confirmation before irreversible actions.
• Silence during slow tool calls.
• Testing on clean studio audio instead of phone-quality recordings.
• No AI disclosure or recording consent where required.

What the example does not do

• No audio at all: every timing is a number drawn from an invented distribution, not a measurement.
• No echo, background noise, codec or packet-loss modelling; the false barge-in model is a single duration threshold.
• Endpointing is a silence threshold; real systems use learned end-of-turn detection.
• The model reply is a fixed number of tokens; real replies and tool calls vary.
• No concurrency or cost model: real voice systems pay per minute and handle many simultaneous calls.
• No dialog management (clarification, repair, corrections such as "make that tomorrow").
• Speaking rate is fixed at 165 words per minute, ignoring accent, voice and context.
• Optional real experiment (I did not run this): record ten real calls with your stack, timestamp user-stop, first audio and each stage, and compute your own p50 and p95 gap and cut-in rate.

For the curious What the user heard is what the history should say

When the user interrupts, stop playback and flush the audio buffer. If the history kept the full reply, it recorded on average 17 words as said that the user never heard; truncating the history at the interruption cut that to 0. Every stage latency here is an assumed round number, not a measurement of any product.

What you must be able to say (one page)

• A voice agent is a real-time pipeline: audio in, voice activity detection and endpointing (has the user finished?), speech-to-text, the model and tools, text-to-speech, audio out. Some systems use one end-to-end speech model instead.
• Latency is the product. Latency has been the major barrier to voice interfaces, and low-latency recognition plus efficient models have reduced it. Think in a turn-gap budget: the silence between the user stopping and the agent's first audio. In my simulation (every stage latency assumed; no real product measured) a streaming pipeline had a mean gap of about 1.8 seconds: 600 ms endpointing, 150 ms transcript finalisation, 744 ms model start plus first sentence, 230 ms first audio chunk, 80 ms network. A batch pipeline (wait for the full reply, then synthesise it all) had p50 2,399 ms and p95 2,885 ms; streaming 1,764 and 2,268; streaming with a faster model (250 ms to first token instead of 450) 1,564 and 1,861. People expect gaps of a few hundred milliseconds (general knowledge, I did not measure), so every stage counts; the largest single item in my budget was endpointing.
• Endpointing is a trade-off, not a setting to minimise. A short wait feels fast but cuts people off during pauses. With my assumed pause distribution (hesitations of about 300 ms median, 1.5 pauses per utterance), the agent started talking over an unfinished sentence in 57% of utterances at 300 ms, 33% at 450, 18% at 600, 8% at 800, 4% at 1,000, while the median gap rose from 1,464 to 2,164 ms. Smarter endpointing (semantic end-of-turn models, punctuation, intonation) tries to improve the curve.
• Interruptions (barge-in) are part of the design. Handle interruptions, mid-sentence corrections ("Book me a table for... oh wait, make that tomorrow") and topic shifts gracefully, and truncate the assistant's output as soon as the user starts speaking. Mechanics: detect speech, stop playback and flush the audio buffer, cancel or restart generation, and update the conversation history with what the user actually heard. In my study, with a 35-word reply and the user speaking at a random moment, the agent talked over the user for about 200 ms at 150 ms detection (149 ms at 100, 345 ms at 300, 632 ms at 600). If the history kept the full reply, it recorded on average 17 words as said that the user never heard; truncating the history at the interruption cut that to 0. Without truncation the agent later says "as I said..." about things the caller never heard.
• Filter noise, but pay for it. Requiring a minimum speech duration before stopping playback reduced false barge-ins from a cough or a door slam: 30 in 10 minutes with no minimum, 6 at 200 ms, 2 at 250 ms, at the price of about 100 ms more talk-over (198 to 296 ms) in my model. Echo (the agent hearing itself on speaker) needs echo cancellation; I did not simulate it.
• Write for the ear. People speak at about 150 to 180 words per minute, read at about 250 to 300 and skim above 500, so voice is slow for dense information. My example reply of 35 words takes about 12.7 seconds to say versus 7.6 to read. Rules: short sentences, two sentences then "want the rest?", numbers and ids in small groups, confirm critical details back (names, dates, amounts), no tables, links or lists.
• Hide tool latency honestly. When a tool is expected to take longer than about 700 ms, play a short filler ("one moment, I am checking that") at once. In my simulation (tool times median 900 ms) the p95 silence fell from 2,376 ms to 1,176 ms. Fillers hide waiting; they do not excuse slow tools.
• Other production concerns: a phone-grade audio stack (telephony codecs, packet loss, jitter), WebRTC or WebSocket transport, streaming everywhere, accents and noise, barge-in during tool calls (do not execute half-heard commands), confirmation before irreversible actions (a misheard "cancel" is expensive), consent to record and disclosure that the caller is talking to an AI (check local law; Level 10 module 10 is not legal advice), voice cloning and fraud risk, and cost per minute (Level 11 module 5).
• Evaluate with recorded calls and scripts: task success, word error rate on your domain terms, turn-gap p50 and p95, interruption handling, false barge-in rate, escalation to a human, caller satisfaction, and cost per call.
• Honest limits: I simulated timings with invented distributions. There is no audio, no STT or TTS, no telephony, and no real model, so the numbers show the structure of the trade-offs, not what a real system achieves.