Barge-in and interruption handling for on-device voice agents

Watch a person interrupt a voice agent and you learn everything about how it was built. A good one stops mid-word the instant you start talking, the way a human does. A bad one keeps monologuing over you for a second and a half, then either ignores what you said or hears its own voice as your input and answers a question nobody asked. The feature is called barge-in, and it's the part of a voice stack that never makes the demo script and always makes the difference.

Barge-in is also where the "just glue three open-source projects together" approach quietly falls apart. Speech-to-text, a language model, and text-to-speech can each run fine in isolation and still produce an agent that can't be interrupted, because interruption isn't a feature of any one of them — it's a property of the loop that connects them. This post is about that loop: what barge-in actually requires, the three hard problems inside it, and why owning the whole thing in one process on the device is the difference between "stops when you talk" and "talks over you."

What barge-in actually requires: full duplex

Most voice pipelines are half-duplex by default. They listen, then they talk, then they listen — a walkie-talkie taking turns. Barge-in demands full duplex: the agent has to keep listening while it is speaking, because the whole point is to catch you starting to talk in the middle of its own sentence.

That one requirement cascades into a surprising amount of engineering. The microphone is now open while the speaker is playing the agent's own voice into the same room. Your voice-activity detector, which is supposed to fire when a human speaks, is staring at an audio stream that is mostly the agent's own TTS bouncing off the walls. Naively, every word the agent says looks like a user interruption. Solve that wrong and the agent interrupts itself into silence; solve it not at all and it can never be interrupted at all.

So the real barge-in problem breaks into three parts that have to work together, frame by frame, on live audio:

Hear the user over yourself — cancel the agent's own voice out of the microphone signal.
Decide the interruption is real — a genuine take-the-floor interruption, not a cough or an "mm-hmm."
Stop, fast — tear down the in-flight TTS (and the generation behind it) within a couple of audio frames.

Problem 1: hearing the user over your own voice

The technical name is acoustic echo cancellation (AEC), and the reason it's tractable at all is that the agent has an unfair advantage: it knows exactly what it is playing through the speaker, because it generated that audio. That known signal is the reference. An echo canceller like WebRTC's AEC3 takes the reference, models how the room and the speaker-to-microphone path delay and distort it, and subtracts that predicted echo from the microphone signal before anything downstream sees it (Switchboard's AEC3 breakdown is a good technical walkthrough).

The hard case is double-talk — when the agent and the user are both speaking at once, which is exactly the barge-in moment you care about. During double-talk the user's speech is present in the same signal the canceller is trying to drive to zero, so a naive adaptive filter will try to model the human as if they were echo and corrupt itself. AEC3 handles this by watching the coherence and energy between the reference and the microphone capture, and freezing filter adaptation when it detects double-talk so the filter doesn't diverge. Get that logic wrong and the first half-second of every interruption — the half-second that matters — is where your echo canceller is at its worst.

There's a subtler requirement hiding here: the reference signal and the microphone capture have to be time-aligned to the sample. The canceller needs to know that this block of mic audio corresponds to that block of what you played, offset by the real acoustic delay. On one device with one audio clock that's an alignment problem. Across separate processes with separate buffers and separate clocks, it's a moving target.

Problem 2: deciding the interruption is real

Once you can hear the user cleanly, you have to decide whether they've actually taken the floor. Not every sound is an interruption. Humans constantly emit backchannels — "uh-huh," "okay," "right," "mm-hmm" — little acknowledgments that mean keep going, not stop. An agent that halts every time you say "yeah" is as broken as one that can't be stopped at all.

This is a layered decision, and each layer buys you something:

Voice-activity detection (VAD) is the fast acoustic gate: is there speech energy here at all? A production-grade detector like Silero VAD processes a 30ms-plus audio chunk in under a millisecond on a single CPU thread, from a model that's only one to two megabytes — cheap enough to run continuously on every frame while the agent talks.
Endpointing works one level up, at the transcript. It watches the partial STT output for signs the utterance is a real, complete thought rather than a filler sound, using trailing silence, punctuation, or an explicit end-of-utterance signal from the recognizer.
Model-based turn detection goes further still: a small classifier reads the partial transcript and predicts, from semantic completeness, whether the user actually intends to take the turn — which lets you distinguish "stop —" from "uh, okay, sure" before the silence even lands. LiveKit's write-up on VAD, endpointing, and model-based turn detection is a clear tour of the tradeoffs.

The point is that a good barge-in decision is a pipeline of increasingly smart, increasingly slow checks, and it has to run against the same audio your echo canceller just cleaned — while the agent is still talking. Every one of these stages needs the others' state.

Problem 3: stopping fast

Deciding to stop is nothing; stopping is everything. When the user takes the floor, the agent has to kill the audio coming out of the speaker now. Interruptible pipelines model this as a cancel event: user speech is confirmed, an interrupt fires, the in-flight TTS is torn down, and — critically — the language-model generation feeding that TTS is cancelled too, so the agent doesn't keep producing tokens for a sentence nobody will hear. Frameworks like Pipecat and LiveKit build this in as a first-class signal that propagates through the whole graph; the Hamming AI interruption runbook is a good field guide to the failure modes.

Here's the budget that makes this brutal. Audio is processed in frames of roughly 10–30ms. To feel like it stops "instantly," the whole chain — cancel the user's echo, confirm speech on VAD, fire the interrupt, drain the TTS buffer, mute the speaker — has to complete inside a small handful of those frames. Humans take turns with a gap of about 200 milliseconds on average across languages, and anything past roughly half a second of the agent talking over you reads as the machine being rude or broken. We laid out the full response-latency budget in why voice agents live or die on latency; barge-in is that same budget, run backwards, under harder conditions, while two people are talking at once.

Why the DIY stack gets barge-in wrong

None of the three problems above is unsolvable. The trouble is that in the standard duct-taped stack — Whisper for STT, llama.cpp for the model, Piper for TTS, wired together with Python glue — each of these lives in a different process with a different buffer and a different clock, and barge-in is precisely the thing that needs them to share all three.

The AEC reference lives in the wrong place. The signal you're playing (Piper's output) and the signal you're capturing (mic into Whisper) are owned by different components. Aligning them to the sample, across process boundaries and Python's scheduling jitter, is exactly the alignment problem that makes echo cancellation fail during double-talk.
The interrupt has to cross seams. A "stop now" decision made near the VAD has to travel to the TTS process to mute the speaker and to the model process to cancel generation. Every seam it crosses adds queueing and adds milliseconds — and it's spending them during the one moment your entire latency budget is already gone.
Nobody owns the loop. STT, the SLM, and TTS each do their job correctly and the interruption still fails, because the failure lives in the integration nobody wrote. That's the same lesson we learned the hard way when we ripped cloud voice out of our robots: the glue is the product, and the glue was the part we'd bolted on last.

Barge-in is the clearest example of why "assemble three great open-source projects" produces a worse agent than the sum of its parts. The projects are great. The seams between them are where interruption goes to die.

How on-device C++ owns the loop

Put the whole thing in one process, on the device, sharing one audio clock and one block of memory, and the seams disappear.

One clock, one reference. The TTS you're playing and the microphone you're capturing are in the same process, so the AEC reference is sample-aligned by construction — no cross-process delay estimation, no clock drift to chase. Double-talk cancellation gets the clean, aligned reference it needs.
The interrupt is a function call, not a network of queues. When VAD-plus-turn-detection confirms the user has the floor, cancelling the TTS and the SLM generation is an in-memory signal that lands in microseconds, not a message hopping between Python processes. The agent stops inside a frame or two because there's nothing between the decision and the action.
It's deterministic, and it's offline. No network in the interruption path means no radio variance deciding whether this particular barge-in feels crisp or broken. The loop costs the same milliseconds every time, which for an embodied agent is half of what makes it feel alive — and it keeps working with the network unplugged, which for a robot in a warehouse or a field is the whole point.

This full-duplex loop — AEC, always-on VAD, turn detection, and instant TTS/generation cancellation, all in C++ with no glue-code seams — is exactly the layer EdgeAI's voice AI platform and its streaming orchestration exist to own, running a right-sized on-device SLM on Jetson-class hardware. It's the same integration work we walk through, with latency numbers, in the offline voice assistant on Jetson Orin guide — barge-in is the reason that integration has to be one thing and not three.

FAQ

What is barge-in in a voice agent?

Barge-in is the ability to interrupt a voice agent mid-response by starting to speak, and have it stop talking and listen — the way a human would. It requires full-duplex audio (listening while speaking), acoustic echo cancellation so the agent doesn't hear its own voice as user input, turn detection to tell real interruptions from backchannels like "uh-huh," and a fast cancellation path that tears down the in-flight speech and generation within a few audio frames.

Why does barge-in cause echo or feedback problems?

Because the microphone is open while the speaker is playing the agent's own TTS, the voice detector sees the agent's voice as if it were the user. Acoustic echo cancellation fixes this by subtracting a known reference (the audio the agent is playing) from the microphone signal. The hard case is double-talk — agent and user speaking at once — where the canceller must freeze its adaptation so it doesn't mistake the human for echo. That's the exact moment barge-in depends on.

How fast does a voice agent need to stop when interrupted?

Fast enough to feel instant. Audio is handled in 10–30ms frames, and the full stop — cancel echo, confirm user speech, fire the interrupt, mute the speaker — needs to finish within a small handful of frames. Human conversational turns have a gap of about 200ms on average, and an agent that keeps talking for much past half a second after you start reads as broken.

Why is barge-in harder with a DIY Whisper + llama.cpp + Piper stack?

Because interruption is a property of the loop connecting STT, the model, and TTS — not of any one component. In a duct-taped stack those live in separate processes with separate buffers and clocks, so the echo-cancellation reference is hard to align, and the "stop now" signal has to cross process seams to reach the TTS and the model, adding latency during the one moment you have none to spare. An on-device stack in a single process shares one clock and one memory space, so alignment is free and the interrupt is a function call.

The takeaway

Barge-in isn't a feature you add to a voice agent — it's a full-duplex loop that has to hear the user over the agent's own voice, tell a real interruption from a nod, and stop within a couple of audio frames. Each of those is solvable; the DIY stack loses because it scatters them across processes that don't share a clock or a memory space. Own the loop in one place, on the device, and interruption stops being the thing that breaks your demo.

If you'd rather start from a stack where the full-duplex loop is already built: try EdgeAI free, skim the getting-started docs, or talk to us and bring the interruption cases that are breaking your agent today.