Turn-Taking & Interruption
The Turn Boundary Problem
Deciding when the user has finished speaking sets the floor on perceived latency and determines whether the agent talks over people or leaves dead air. A half-duplex pipeline (STT, then LLM, then TTS) has to commit to that decision explicitly: too early clips the user mid-sentence, too late leaves a hesitation they read as the agent being slow. Interruption is the same problem inverted, deciding whether overlapping speech is a claim on the floor, a noise artefact, or someone saying "mhm" to show they are listening.
Endpointing Models
Four approaches, roughly in order of how much they know about what is being said. Each rung buys responsiveness and costs simplicity.
Wait a constant period of silence, typically 500 to 800ms, then declare the turn over. Deterministic, trivial to implement, and completely blind to content. The threshold is a direct latency tax on every turn, paid whether or not the user had actually finished.
A neural voice-activity detector scores each frame for speech, with thresholds that adapt to the noise floor and the speaker's rate. Much better rejection of background noise and non-speech audio, but still purely acoustic: it knows whether someone is talking, never whether they have finished a thought.
A small, fast classifier runs over the streaming transcript and predicts whether the utterance is complete. A syntactically finished sentence can end the turn almost immediately; a trailing conjunction or a dangling preposition extends the wait. This decouples responsiveness from a single global threshold and is where most production platforms have converged.
The model listens and speaks continuously, with turn-taking learned as part of the training objective rather than decided by an external component. There is no explicit endpoint to tune. Removes the boundary problem outright, at the cost of a control surface: there is no longer a discrete moment at which to gate, inspect, or intervene.
Overlap Taxonomy
Overlapping speech is a category, not an event. The correct response differs completely across the category, and a system that collapses them into a single "barge-in" signal will get most of them wrong.
| Type | What the User Is Doing | Correct Agent Response | If Misread |
|---|---|---|---|
| Backchannel | Signalling attention ("mhm", "yeah", "right") without claiming the floor | Keep speaking. Do not cancel, do not treat as input. | Agent cannot finish a sentence |
| False Trigger | Not speaking to the agent at all: background TV, a second person, or the agent's own echo | Suppress via echo cancellation and, where available, speaker verification. | Random self-interruption |
| Clarification | Interjecting a question about what is being said right now | Stop, answer the question, offer to resume the original turn. | Question ignored or answered late |
| Correction / Redirect | Changing the request mid-answer, as in "no, the other account" | Stop immediately, discard the pending plan, re-plan from the correction. | Agent completes work the user already rejected |
| Hard Stop | Explicit cancellation: "stop", "wait", "cancel that" | Cancel speech and any in-flight tool call, then yield the floor silently. | Agent keeps acting after being told to stop |
| Simultaneous Start | Both parties beginning at the same moment after a pause | Yield by default. The agent should lose the race with the human. | Both retreat, then both restart, in a loop |
Backchannels are the expensive case to get wrong because they are frequent, short, and mean the opposite of an interruption. The listener is signalling keep going. An agent that halts on every "mhm" cannot hold the floor long enough to answer anything.
Truncation and Reconciliation
When a turn is cut short, the agent's own record of the conversation becomes wrong. Text was generated, audio was synthesised, and some prefix of it reached the user's ear. Those are three different quantities. Writing the full generated turn into history leaves the agent convinced it said things nobody heard, and it will refer back to them.
The full LLM output for the turn. Usually complete well before the audio finishes playing, because generation outruns speech.
On cancellation: Discard everything past the spoken prefix. This is the text most systems wrongly commit to history in full.
TTS output, typically buffered ahead of playback to absorb jitter. The buffer is the gap between what exists as audio and what the user has actually heard.
On cancellation: Flush the buffer. Anything unplayed never happened as far as the user is concerned.
The prefix that genuinely reached the user. The only one of the three that describes the shared conversational state.
On cancellation: This is the ground truth. Reconstruct the spoken text from playback marks and write that to history.
Tool calls the turn already executed. Cancelling speech does not unwind a booking, a payment, or a write.
On cancellation: Nothing rolls back automatically. Either gate side effects behind turn completion or make them explicitly reversible.
The practical requirement is word-level or phoneme-level playback marks from the TTS layer, so the spoken prefix can be reconstructed and written to history in place of the generated turn. Without them, truncation is a guess. See Conversation Management for what that history then has to carry forward.
Latency Budget
Response latency is usually attacked at the model layer, but in a well-built pipeline the endpoint decision is frequently the largest single term, and the cheapest to reduce, because it is a policy choice rather than an engineering constraint.
| Stage | Typical Contribution | Reducible By |
|---|---|---|
| Endpoint decision | Often the largest single term, being the full silence threshold on a fixed model | Semantic endpointing; shorter thresholds only where clipping risk is low |
| STT finalisation | Small if streaming partials are used; significant if waiting for a final transcript | Start LLM prefill on stable partials rather than the final result |
| LLM time-to-first-token | Moderate, and the most commonly optimised | Smaller or distilled models, prompt caching, speculative decoding |
| TTS time-to-first-byte | Moderate, and the first audio chunk gates everything the user perceives | Streaming synthesis; begin on the first clause rather than the full response |
| Network and jitter buffer | Small but highly variable, and worst on the calls that matter | Edge deployment; adaptive buffering rather than a fixed floor |
Dropping a fixed threshold from 700ms to 400ms removes 300ms from every single turn, more than most model swaps deliver, at the cost of clipping people who pause mid-thought. Semantic endpointing exists to take that win without paying that price. Related: Latency and Design Constraints.
Failure Modes
The agent commits to a response while the user is mid-thought. Dictated numbers, addresses and spellings trigger it most reliably, because natural pauses there are long and frequent.
Fix: Extend thresholds inside known slot-filling states. A turn model can be state-aware: collecting a card number is not the same context as open conversation.
A dead-air gap after every utterance. Users read the silence as the agent being broken and start repeating themselves, which restarts the turn and compounds the delay.
Fix: Semantic endpointing for confident complete sentences. If a long wait is unavoidable, fill it with an acknowledgement token rather than silence.
Every "mhm" halts the agent. The agent never completes a sentence, and the user escalates their volume trying to be understood, which triggers more cancellations.
Fix: Classify short overlapping utterances before acting. Duration, lexical content and prosody separate continuers from floor-claims cheaply.
The agent hears its own output through the microphone and interrupts itself. Characteristic of speakerphone and poorly configured telephony paths.
Fix: Acoustic echo cancellation as a hard requirement, not a tuning option. Gate barge-in detection on the residual after cancellation.
The user interrupts while a tool is in flight. Speech stops, but the call completes and its effect lands. The user believes they cancelled and the system disagrees.
Fix: Propagate cancellation to the tool layer, not just the audio layer. Where a call cannot be cancelled, say so before starting it.
Truncated speech is written to history in full, so the agent later references information the user never heard. It is the most confusing failure for users because it is invisible from their side.
Fix: Commit only the spoken prefix, reconstructed from playback marks. Treat the audio layer as the source of truth for what was said.
Choosing a Turn Model
| Situation | Why It Matters | Choice |
|---|---|---|
| Open-ended conversation | Users pause naturally and unpredictably; a fixed threshold either clips or lags | Semantic |
| Slot filling: numbers and addresses | Long mid-utterance pauses are normal and clipping is costly to recover from | Extended threshold |
| Noisy or shared environments | False triggers dominate; content-based signals help less than rejection does | Adaptive VAD first |
| Strict regulatory logging | Every gate, disclosure and interruption must be inspectable after the fact | Avoid full-duplex |
| Latency is the product | Perceived responsiveness outweighs fine-grained control of the turn boundary | Full-duplex |
| Constrained IVR-style flows | The valid response set is small and turn boundaries are structurally obvious | Fixed threshold |
Turn behaviour is testable. Simulated callers who interrupt, trail off, or talk over tool calls are a standard scenario class. See Agent Testing for building those suites, V2V Risks for what turn detection means at the security layer, Stream Safeguards for policy-driven interruption of the agent's own output, and Agentic Errors for turn-taking as a social failure mode.