Speech-to-speech models handle turn-taking inside the model seamlessly. But plenty of us still run cascades because we need a specific ASR for a language, our own customized LLM, or a particular synthetic voice. And in a cascade architecture, by default nobody decides who talks when. You have to design and build it yourself.
Cascade architecture in voice agents involves a sequential processing of audio input through three main components: Speech-to-Text (STT), Large Language Model (LLM), and Text-to-Speech (TTS). Each stage operates independently, allowing for modular optimization and flexibility.
Cascade remains the production default for most voice agent workloads due to its ability to provide component-level debugging and compliance-ready audit trails across all three pipeline boundaries.
You can run the absolute best models on the market, but callers will still hate the bot if it jumps in while they pause to think, or if it keeps talking after they try to correct it. Users notice broken turn-taking long before they notice anything about your underlying LLM’s intelligence.
Here is the key mental model: split the problem into “when does the caller’s turn start?” and “when does it end?” These are two fundamentally different problems, requiring distinct logic and safeguards.
| When the caller’s turn starts | When the caller’s turn ends |
|---|---|
| Use an interruption gate to decide whether the caller is genuinely cutting in. | Use VAD, transcript-based turn detection, silence fallback, and a watchdog. |
| Goal: avoid letting “mhm,” coughs, or background sound stop the agent. | Goal: avoid jumping in while the caller is pausing, thinking, or continuing. |
1. Turn Start: Don’t Let “Uh-Huh” Kill the Agent
Pure VAD start means any sound can interrupt the bot: backchannels, coughs, or a TV in the background. A better fix is not a hard floor of two or three words, because the most important interruptions are often one word: “stop,” “no,” and “wait.” We handle this with time instead: a minimum interruption duration, with 0.2s as a practical default. A higher value rejects short backchannels like “mhm” and “yeah,” while still allowing a real short interruption through. LiveKit has a word-count gate, but the useful version is duration plus a floor of one word, not two or three.
| User sound | Should interrupt? | Why |
|---|---|---|
| “mhm” | No | Usually a backchannel, not a correction. |
| Cough or background noise | No | Sound alone should not stop the agent. |
| “stop” | Yes | A short, direct interruption should be allowed. |
| “wait” | Yes | The caller is trying to take control of the turn. |
2. Turn End: Silence Alone Is a Bad Signal
“I need to change my…” followed by a pause is not the end of a turn. What works better is VAD to catch the pause plus a small turn detection model, such as LiveKit’s turn model, that checks whether the transcript looks finished. If it does not, wait a few seconds of silence before ending the turn.
And if the caller starts talking again while the model is still deciding, throw the verdict away. Otherwise, a stale “end of turn” result can merge two utterances or cut the caller off.
3. Always Have a Watchdog
If nothing closes the turn — not the model, silence, or ASR — force-close it after about five seconds. A conversation that silently stalls is worse than one that occasionally ends a turn early.
4. Don’t Wait for the Final Transcript
Once the turn detector says “done,” start inference on what you have and give ASR a short grace window — a few hundred milliseconds — to update the text. Waiting for the final transcript adds latency you feel on every single turn.
5. Handle Silence on the Agent’s Side Too
Start an idle timer when the agent finishes speaking. If the caller says nothing, re-ask or check that they are still there, then close with a polite goodbye if needed.
6. Make Interruptions Switchable Mid-Call
Interruption control should be explicit, but it is not always available at every level of the call. In our graph today, we can control the welcome message with welcome_allow_interruptions, and we have one broader switch for the rest of the call. That means we should not claim that a legal disclaimer or payment terms can be made non-interruptible per message unless the graph exposes that control. LiveKit can support per-utterance behavior, but this implementation does not expose it yet.
7. Clean the Audio Before VAD, Not After
Phone lines are noisy, so clean audio as early as the stack allows. Noise cancellation is handled by LiveKit’s telephony model on the room input, and in this setup it applies only for SIP. Treat it as input conditioning for the call, not as part of the same worker-side turn-taking module.
8. Watch Out for Multi-Language Nuances
The language warning is still real, but the failure mode is different in a transcript-based pipeline. This setup is not mainly judging intonation the way an audio model such as Pipecat Smart Turn would. It is judging text. If speech-to-text is weak, the turn model is judging a bad transcript; and for languages outside the model, the system falls back to VAD. In those cases, longer pauses still help. A higher confidence knob does not exist on our side, so we should not recommend one.
The Ideal Pipeline Setup for Smart Turn Detection
The practical setup is not one media-server module that bundles everything together. Silero VAD and the LiveKit turn model run in the agent worker. Noise cancellation runs separately as LiveKit’s telephony model on the room input, and only for SIP. So the accurate way to describe the system is a cascade with explicit start and stop strategy: VAD for speech activity, transcript-based turn detection for whether the caller sounds finished, cancellation on the room input when SIP is involved, and a watchdog so the call never stalls.
| Pipeline step | What happens |
|---|---|
| 1. Caller audio | Phone or SIP input enters the call. |
| 2. Room input cleanup | LiveKit telephony noise cancellation runs on SIP room input when available. |
| 3. Agent worker | Silero VAD detects speech activity. |
| 4. Turn decision | The LiveKit turn model checks whether the transcript sounds complete. |
| 5. Safety fallback | A watchdog force-closes the turn if nothing else does. |
| 6. Agent response | The assistant responds once the caller’s turn is complete. |
None of this is rocket science, but it is the exact difference between a neat demo and a voice system that survives real human callers.
I’m curious how you and your team deal with the backchannel issue. Is it just a matter of using a simple classifier, or have you found a cleaner architectural trick? Leave a comment below or join the discussion—I’d love to hear what’s working for you in production.