C++ vs Python for real-time voice agents on the edge

Every "should I use C++ or Python" thread ends up in the same cul-de-sac: someone posts a microbenchmark showing C++ is 50x faster at a loop, someone else points out that the heavy lifting is in a C library anyway, and nobody's robot gets any better.

That argument is the wrong one, because throughput is not the constraint in a voice agent. The constraint is a deadline. Your audio callback fires every 10 to 20 milliseconds forever, and it does not care about your average case. It cares whether you returned in time, every single time, including the time the garbage collector woke up.

This post is about where that deadline actually bites on a Jetson, what Python costs you at each point, and — the part most of these threads miss — the fact that your Python voice stack is already C++ underneath, which changes what the decision is really about.

The deadline nobody writes down

Humans enforce turn-taking gaps of about 200 milliseconds, which is the number that sets your whole voice agent latency budget. But that's the outer deadline. Inside it there's a much tighter one.

Audio arrives in buffers. At 16kHz with a 20ms block, that's 320 frames every 20ms, and your callback has to consume that block and return before the next one lands. Miss it and you don't get "slower" — you get a dropped buffer, a click, a chopped-off first syllable, a VAD that fires late.

The rules for code on that thread have been settled for two decades. Ross Bencina's Real-time audio programming 101 states the core one plainly: don't allocate memory in the audio callback, because the allocator may take a lock contended by every other thread in the process, or go ask the OS for pages and wait.

You don't have to take it from a blog post, either. Here is what python-sounddevice's own documentation says about the callback you are about to write in Python:

The PortAudio stream callback runs at very high or real-time priority. It is required to consistently meet its time deadlines. Do not allocate memory, access the file system, call library functions or call other functions from the stream callback that may block or take an unpredictable amount of time to complete.

Now hold that next to what CPython does on every function call.

Where Python actually costs you

Not "Python is slow." Four specific, nameable failure modes.

1. The GIL — improved, not gone

Every Python audio callback has to acquire the global interpreter lock before it can run a single bytecode. If another thread is holding it — your inference thread, your ROS executor, your logging thread — your real-time thread waits on a lock whose hold time you do not control.

Free-threading is real progress here. PEP 703 landed the free-threaded build as experimental in 3.13, and PEP 779 promoted it to officially supported in 3.14. The single-threaded penalty dropped from roughly 40% in 3.13 to about 5–10% in 3.14, depending on platform and compiler.

Two caveats that matter for robotics specifically:

• The ecosystem gates you. C extensions must be rebuilt against the free-threaded ABI. If any dependency in your import graph isn't ported, you're back on the GIL — and on a Jetson your import graph includes CUDA bindings, camera drivers, and whatever vendor SDK shipped with your sensor.
• Removing a lock doesn't add determinism. Free-threading is a parallelism win. It does nothing about the next three items.

2. The cyclic collector's stop-the-world scan

CPython's reference counting is deterministic. The supplemental cycle collector is not: a full collection linearly scans every tracked object in the process, and it does it while everything else is stopped.

The clearest published illustration is Ben Hoyt's writeup of a 4.5-second GC pause on a server holding ~10 million Python objects — a page that normally rendered in 15ms, every 445th request. That's a 2013 web app, not a robot, and the number is not a voice-agent benchmark. What transfers is the scaling law: pause time grows with the number of live tracked objects, and it fires on an allocation-count trigger you did not schedule.

A long-running voice agent holds conversation history, rolling audio buffers, transcript objects, and tool-call state. That is a growing population of tracked objects sitting in a process with a 20ms deadline. You can tune gc.set_threshold, you can gc.freeze() after startup, you can disable it and leak cycles on purpose. Every one of those is you hand-managing memory — the thing you chose Python to avoid.

3. The callback can't obey the rules it's given

Read that sounddevice warning again, then look at an idiomatic Python callback:

def callback(indata, frames, time, status):
    if status:
        print(status)          # I/O on the real-time thread
    q.put(indata.copy())       # allocation + lock, per block

indata.copy() allocates. q.put takes a lock. print touches stderr. Every one of those is explicitly on the do-not-do list, and the idiomatic version of this code — the one in the docs, the one every tutorial copies — does all three. Not because Python programmers are careless, but because in Python the allocation-free version isn't really expressible; boxing, refcounting, and temporary objects are the substrate.

The C++ equivalent is boring, which is the point:

// Real-time thread: no allocation, no locks, no syscalls.
void on_audio(const int16_t* in, size_t frames) noexcept {
    ring_.write(in, frames);          // preallocated SPSC ring buffer
    vad_.process(in, frames);         // fixed-size scratch, no heap
}

Everything expensive happens on a worker that drains the ring. The hot path has a bounded worst case you can actually reason about. This is the same discipline that makes barge-in and interruption handling work: cutting TTS within a couple of audio frames of detecting real speech requires that the detection path never stalls.

4. The process-boundary tax

The last cost is architectural, and it's the one that quietly dominates.

Because Python can't hold the whole pipeline in one hot loop, DIY stacks split into processes and talk over sockets. Home Assistant's Wyoming protocol makes this explicit — STT and TTS run as separate network services. That's a sane design for a home server. On a robot it means every audio chunk and every partial transcript crosses a serialization boundary on the critical path.

ROS 2 gives us a clean measurement of what a Python binding costs at that boundary. From the rclpy maintainers' own issue on large-message publishing: publishing a 10MB PointCloud takes about 2.8ms in rclcpp and about 92ms in rclpy — roughly 33x. Audio messages are far smaller than point clouds, so don't transplant that number onto your mic topic. But on a robot your voice pipeline shares an executor and a memory bus with perception traffic that is that big, and the tax is paid in jitter on your thread.

What Python is genuinely better at

An honest comparison has to include this, because the answer isn't "C++ everywhere."

• Model and pipeline experimentation. Swapping an SLM, testing a new VAD threshold, sweeping quantization settings — iteration speed dominates, and there's no deadline.
• Evaluation and data tooling. WER scoring, transcript diffing, dataset curation, plotting. C++ here is self-harm.
• Behavior and mission logic. The layer deciding what the robot should do runs at human timescales. Python is fine, and often better.
• Ecosystem reach. Every new model ships a Python reference implementation first, sometimes only.

The mistake isn't using Python. It's letting Python own the loop with the deadline.

The scoreboard

The part nobody says out loud

Here's the thing that should end the argument: if you've built a fast local voice pipeline in Python, you already chose C++. You just chose it three times, from three different vendors, with Python in between.

• whisper.cpp — C/C++ on ggml
• llama.cpp — C/C++
• Piper — VITS models executed through ONNX Runtime's C++ API
• TensorRT — C++

The evidence that this is where the performance lives is in NVIDIA's own numbers. On a Jetson Orin Nano, WhisperTRT transcribes a 20-second clip with base.en in 0.86s versus 2.55s for the PyTorch reference, using 439MB versus 666MB of memory. For tiny.en it's 0.64s (TensorRT) against 0.85s (faster-whisper) and 1.74s (PyTorch). The win comes from leaving the Python runtime, not from a better algorithm.

So the real question was never "C++ or Python for inference." Inference is already C++. The question is:

What language runs the orchestration between those C++ components?

And orchestration — capture, VAD, endpointing, the streaming handoff from partial transcript to first token to first audio frame, barge-in cancellation — is precisely the part with the hard real-time deadline. Python is currently the layer holding the stopwatch while every stage it coordinates is written in something else.

That's the seam. It's also the reason a Jetson Orin Nano Super with 67 TOPS and 8GB of LPDDR5 at $249 can feel sluggish running a voice agent: the silicon isn't the bottleneck, the seams between four processes are.

What to do on Monday

You don't need a rewrite. You need to move the boundary.

1. Find your real deadline. Log callback entry-to-exit for every block for an hour. Look at max, not mean. If p99.9 is anywhere near your buffer period, you're already dropping audio and blaming the model.
2. Instrument the GC. Register a gc.callbacks hook that records collection duration and generation. If gen-2 collections show up during conversations, that's your mystery stutter.
3. Get everything off the callback thread. One lock-free ring buffer, one consumer. Nothing else on that thread, ever.
4. Count your process boundaries. Every socket hop between mic and speaker is serialization plus a scheduler round-trip you can delete by putting the stages in one address space.
5. Draw the line and hold it. C++ from microphone to speaker. Python above it, talking to the loop through a queue. Measure on the target — desktop numbers don't transfer to a 25W module sharing memory bandwidth with a perception stack.

Steps 3 through 5 are most of a year of work if you build them yourself, on top of three upstream projects with independent release cadences and no shared threading model. That integration layer — a single C++ process owning capture, local STT and SLM inference, on-device models, TTS, and the orchestration between them — is exactly what EdgeAI's voice stack exists to be, so that the seam isn't yours to maintain. We took the same route the long way around, and wrote up why we ripped cloud voice out of our robots on the way.

FAQ

Is Python fast enough for a real-time voice agent?

For the parts that aren't real-time, yes — model loading, configuration, evaluation, behavior logic. For the audio callback, no. That callback must return within one buffer period (typically 10–20ms) with no allocation, no blocking, and no unpredictable pauses, and CPython offers no bounded guarantee on any of the three.

Does Python 3.14 removing the GIL fix real-time audio?

It helps with parallelism, not determinism. PEP 779 made free-threaded builds officially supported in 3.14, and the single-threaded penalty fell from ~40% in 3.13 to roughly 5–10%. But the cyclic collector, dynamic allocation, and unbounded C-extension behavior remain, and any extension not rebuilt for the free-threaded ABI can re-enable the GIL process-wide.

Why is rclpy slower than rclcpp in ROS 2?

Messages cross the Python/C boundary and get copied and converted. The rclpy maintainers' own issue reports a 10MB PointCloud publish at ~2.8ms in rclcpp versus ~92ms in rclpy. Audio messages are much smaller, but on a robot your voice pipeline shares an executor and memory bus with traffic that isn't.

Should I rewrite my whole robot stack in C++?

No — draw the line at the real-time boundary. Capture, VAD, endpointing, barge-in, and the streaming handoff between STT, model, and TTS belong in C++ with a bounded, allocation-free hot path. Mission logic and tooling stay in Python.

Is my Python voice pipeline actually running Python inference?

Almost certainly not. whisper.cpp and llama.cpp are C/C++, Piper runs through ONNX Runtime's C++ API, TensorRT is C++. Python is the glue between C++ components — which puts the language boundary exactly where the latency-critical scheduling happens.

The takeaway

C++ vs Python for a real-time voice agent isn't a performance argument, it's a determinism argument. Python's problem on the audio thread isn't that it's slow; it's that its worst case is unbounded and its idioms violate the callback rules its own audio bindings publish. And since every fast local STT, LLM, and TTS runtime is already C++, the only question left is whether the orchestration between them — the part actually holding the deadline — is written in the same language as the parts it schedules.

Start from a stack where it is: try EdgeAI free, read the getting-started docs, or talk to us and bring your callback histograms.