We put a voice agent on Genie's delivery robot. Then it rode away.

Genie Mobility builds fully autonomous delivery e-bikes for dense city streets: road-legal vehicles that run up to 28 mph in bike lanes and traffic, sitting in the gap between sidewalk rovers and delivery vans. Andrew Hughes founded it in 2025.

In late July we took Friday, our on-device voice agent, live with Genie on the streets of San Francisco.

Five things about that integration I'll state plainly, because they're the five I can stand behind:

• Ten days, from starting the integration to speech-to-reasoning-to-response running on the vehicle. That number is now one day, and why it collapsed is further down.
• No new hardware. The whole voice stack runs on the vehicle's existing NVIDIA Jetson AGX Orin, alongside the autonomy software already there.
• Eyes, without a new sensor. Friday runs a vision-language model for scene understanding on the cameras the bike already carried for autonomy.
• No audio on the wire, ever. Speech becomes text at whichever device heard it. Nothing crossing a network is audio, and nothing goes to a cloud service.
• The autonomy stack was never touched. Navigation, obstacle avoidance, path planning, and every safety decision stayed exactly where they were.

The engineering went about how we expected. Everything that actually taught us something arrived as a problem we hadn't designed for.

The obvious version, which was wrong

We started where everyone starts: put the voice agent on the robot.

It's the intuitive design. The machine is the thing you want to talk to, so the ears and the mouth go on the machine. Microphone on the vehicle, speaker on the vehicle, agent on the Orin. It worked. You could stand next to a Genie bike, ask it what it was doing, and it would tell you — locally, with no network.

Then we watched it in the field and hit the thing that should have been obvious from the first line of the product description.

The bike's entire job is to leave.

An autonomous delivery vehicle is, by definition, not where you are. That's the point of it. It's three blocks away in a bike lane, or at a handoff, or parked somewhere it decided was safe. The operator is at a desk. The customer is in an apartment. The person who most needs to ask the machine a question is almost never within earshot of it.

Voice-on-the-machine quietly assumes co-location. Autonomy is the thing that breaks that assumption. We had built a natural interface to a robot, then deployed it on the one class of robot engineered to remove itself from the room.

That's not a bug you find in a lab. Everything works in a lab, because in a lab the robot is right there.

Splitting the stack: ears travel, the agent stays

The answer was an iOS app. Talk to your machine from anywhere, whether it's across the street or across the city. Every robotics company has an app, so that part is unremarkable.

What's worth describing is where we drew the line, because we did not put a second agent on the phone.

The phone runs the audio front-end, entirely on the device: voice activity detection, acoustic echo cancellation, speech recognition, and text-to-speech. The iPhone turns your voice into text locally and speaks the reply locally. AEC is the unglamorous one that matters most. It's what lets you interrupt mid-sentence instead of waiting politely for the assistant to finish.

Friday stays on the machine. Reasoning, memory, the behavior profile, and the calls into the robot's capabilities all live on the AGX Orin, where the machine's state actually is.

Two things fall out of that split, and neither was the plan going in.

Audio never crosses the network. Not to us, not to a cloud provider, not even to the robot. What travels is a sentence. That's a stronger property than "we don't log your audio," and we got it by accident of architecture rather than by policy.

There is exactly one agent, and it lives where the state is. Put a second agent on the phone and you have two systems holding two versions of the truth about one robot, and you spend the rest of the year reconciling them. The machine knows what the machine is doing. The phone is a very good remote pair of ears and a mouth.

The reframe: from "why voice?" to "what makes voice worth having?"

We started this project asking a feature question: why would this robot benefit from a voice agent? You can answer that one with a demo. Ask the bike a question, it answers, everyone nods.

The split forced a better question: what does a machine need in order for a voice to be worth having at all?

Because once you're not standing next to the robot, an interface that only does speech-to-text-to-answer is worse than the dashboard it replaced. A dashboard at least shows you state. Voice without state is a remote control with extra steps and a worse error rate.

"I wanted to create a product interface that went far beyond a standard dashboard. A way to communicate with your robot that could answer any question, fill in the blanks and feel like a useful colleague."

— Andrew Hughes, founder, Genie Mobility

What makes it worth having is that the thing on the other end is an agent. Four properties, all of them living on the machine:

Memory. It remembers this machine, this route, this operator, and what you asked twenty minutes ago. You don't re-establish context every time you open your mouth. Without memory, every interaction starts from zero — exactly the friction a voice interface is supposed to remove.

A behavior profile. It knows what kind of machine it is and how it's supposed to act: what it may do, what it must escalate, how it speaks to a customer at a handoff versus a field operator at a depot. Same runtime, different persona and different permissions, defined rather than emergent.

Actions, not just answers. It invokes a bounded, explicit set of capabilities. Everything else is off-limits by construction.

Persistent state. It carries what's happening right now, continuously, not only when you address it.

That last one changes the product, because state is what makes an agent proactive. An agent holding live state doesn't wait to be interrogated. It can tell you the bike has been stationary for four minutes before you thought to check. It can flag a battery margin that won't cover the return leg. The interaction stops being you, remembering to ask and becomes the machine, telling you.

And it only works because the agent sits on the vehicle. A phone-resident agent would have to poll for the thing it's supposed to volunteer.

Supervision, not queries: the contract that has to exist first

Put those four properties together and the thing you've built stops being a question-answering interface. An operator with memory, live state, and a bounded action set isn't querying a robot. They're supervising one, in conversation, from anywhere. Ask why it stopped, hear the reason, and redirect the mission: one exchange instead of a dashboard, a phone call, and a remote-assistance session. Point the same agent at a customer with a different behavior profile and different permissions, and the delivery recipient gets an interface too, backed by the same live state.

That's the product. It's also where the engineering stops being about voice.

You cannot put a language model in a supervisory position over a safety-critical control stack on the strength of a good prompt. Models are nondeterministic; autonomy stacks are not allowed to be. Bridging those two things is the actual research problem, and it's the one this pilot pushed us into.

The shape we landed on separates the agent from the machine with two layers that each do one job:

The bridge is the data plane. One ROSbridge connection to topics, services, and TF, so the capability layer reads real machine state instead of a cached summary. One bridge, not one integration per feature.

The capability servers are the contract. Command and response, health and diagnostics, route and navigation, system status and controls. Each is an MCP server exposing a typed, enumerable set of verbs. MCP earns its place here for an unglamorous reason: it makes the agent's action space a declaration rather than a prompt. The agent cannot invoke what no server exposes. Add a capability and the agent discovers it. Revoke one and it disappears. You can read the whole action space without reading the model.

Underneath, nothing changed. Sensors, perception, planning, navigation and controls stay exactly where they were, owned by the team that certified them. The agent asks; the autonomy stack decides whether the ask is admissible right now. We're the mouth and the ears, never the hands on the wheel.

What's still open. Enumerating verbs is the easy half. The half we're still working on is making the contract formal, the part where safety and determinism stop being conventions and become properties you can check:

• Preconditions and admissibility. A mission-level request is valid in some machine states and not others. Where does that predicate live, and who owns it — the capability server, or the autonomy stack that knows what it's currently doing?
• Deadlines. Every capability call needs a bounded response time and a defined behavior when it blows it. A supervisory interface that hangs is worse than one that refuses.
• Idempotency and replay. Speech recognition will occasionally hand you the same sentence twice. "Return to the pickup point" must not queue twice.
• Arbitration. When the agent's request conflicts with what the planner is already doing, the resolution rule needs to be written down rather than emergent.
• Auditability. Every invocation, its arguments, the state it saw, and the decision the autonomy stack returned — recorded, replayable, and explicable after the fact. This is what an industrial buyer actually purchases.

None of that is a voice problem. It's the interface between a probabilistic reasoner and a deterministic machine, and I think it's the thing that decides whether conversational supervision ships beyond pilots.

What the ten days were actually spent on

Ten days is a good number. I was pleased with it at the time. It was also ten days of me doing by hand something the tooling should have been doing by itself.

Almost none of it was voice engineering. The ASR, the reasoning, the tool calling, the TTS — already built, already tested, already fast. The ten days went to fitting: working out which model would fit in the memory left over after autonomy, sizing quantization against the thermal envelope, wiring the audio path, connecting to the machine's interfaces, then tuning until the numbers were acceptable.

Every one of those depends on facts the software can discover for itself. How much RAM is actually free. What accelerator is present. What the sustained thermal budget looks like. I was acting as a very slow, very expensive hardware profiler.

So we moved that work into the tooling. Installing the stack is now three commands:

pip install edgeai-cli
edgeai login
edgeai install

That resolves what your platform needs (it knows a Jetson from a Mac), pulls the product binaries and models together, and writes the launchers. edgeai doctor tells you what's broken when something is. The same speech-to-reasoning-to-response path we spent ten days hand-fitting now comes up in a day.

That's the same lesson as the first one, wearing different clothes. The software should conform to the situation, to where the human is standing and to the machine it woke up on, instead of asking the situation to conform to it.

The first version worked. It wasn't fast.

Fitting it to the hardware was one problem. Making it feel fast was a different one, and on day ten we had not solved it. Nothing in that first build was optimized, and the number describes an integration, not a tuned system. Four things we changed once it was running, roughly in order of how much they moved.

Count the tokens going in, not just the ones coming out. On a constrained device, time-to-first-token is dominated by prefill, and prefill scales with everything you put in the context window. The system prompt, the tool schemas, the conversation history, and the machine state you inject all bill against your response time before the model emits a single token. Our first pass pushed a generous blob of robot state into every turn because that was the easy thing to do. Working out what the model actually needed out of that blob, and cutting the rest, took a turn from roughly 8,000 input tokens to a little over 2,000, and roughly halved our time to first token. That was a bigger win than anything we did to the model itself. Most of the latency work turned out to be editing, not engineering. Background on why this metric is the one that matters: time-to-first-token.

Expose fewer tools per turn. Every tool schema in the prompt costs you twice: tokens on the way in, and a wider space for a small model to pick the wrong thing. Four capability servers with a full verb set each is a lot of surface to hand the model every time somebody asks about battery. So the capability contract became the ceiling rather than the working set. The servers still declare everything, and the agent is handed a subset selected for what's actually being discussed. The declared action space stays auditable and complete; the live one stays small.

Put the latency on the screen. We spent longer than I'd like tuning against a stopwatch and a feeling. Surfacing per-stage timing in the app (endpoint fired, transcript final, prefill done, first audio out) turned "that felt slow" into a named stage regressing. It changed field reports too. When an operator can see where a turn went, they stop telling you the agent is broken and start telling you the thing that's actually broken.

Give it eyes. A text-only agent knows what the machine believes. It does not know what the machine is looking at. "Why did you stop?" answered out of planner state is a variable name. Answered out of the scene, it's something a person can act on. So we moved Friday to a vision-language model. The bike already carried cameras for autonomy, which meant scene understanding cost no new sensor and no new hardware, the same constraint that shaped everything else in this integration. The VLM runs on the same AGX Orin, so frames stay on the vehicle along with everything else. It also runs straight into the first lesson: vision tokens are the most expensive thing you can put in a context window on edge hardware, and paying for them without giving back the latency we had just won is the live tension.

This generalizes, and not just to delivery

Every mobile autonomous machine has this shape. AMRs cross a warehouse. Tractors work a field the operator drove away from. Inspection robots go into places specifically because people shouldn't. Drones are the extreme case.

In all of them the machine's value is that it operates without you, and the direct consequence is that your interface can't assume you're there. If you're planning voice for a machine that moves, plan for both endpoints from the start. We didn't, and retrofitting the second one is where the real design work turned out to be.

A short version of what I'd ask before starting:

1. Where will the human actually be when they most need to talk to this machine? If the honest answer is "not next to it," you need ears and a mouth in two places and an agent in one.
2. Can a program call your machine's functions? ROS 2, an internal API, a documented service layer. A day becomes a month if the only control path is undocumented.
3. What's genuinely free on your compute after perception, autonomy, and safety-critical loops are resident? Not total RAM — free.
4. What's the bounded set of capabilities you'd let an agent invoke, what are the preconditions on each, and what is permanently off the list?
5. What state would be worth the machine volunteering, unprompted? That answer is your proactive roadmap, and it's usually more valuable than anything on the command list.

What I'm not claiming

No absolute latency figure for this deployment. We have committed, reproducible end-to-end results on other hardware, measured acoustically, speech-end to first audible sample, p50 and p95, over a fixed corpus. We don't yet have that run on an AGX Orin, which is what this deployment uses, and quoting a number from different silicon would be exactly the thing I complain about when other people do it. The halving above is a relative before-and-after on the same build and the same vehicle, which is a different kind of claim from a figure you'd hold up against somebody else's. The harness is being open-sourced and the AGX Orin numbers publish with it. Background on why this measurement is harder than it looks: time-to-first-token.

The installer matches your platform, not yet your capacity. edgeai install resolves the right binaries and models for the OS and architecture it finds. What it does not yet do is probe how much memory is genuinely free and pick a model sized to that, the part I did by hand over those ten days. Automatic model selection is specced and being built. It isn't shipping, and I'd rather say so than let a phrase like "fits your hardware" do more work than it has earned.

The formal contract is a research direction, not a finished spec. The capability layer and the bridge are running. Preconditions, deadlines, arbitration, and replayable audit are where the work is, and I'd rather describe the open questions honestly than imply we've closed them.

This is a pilot, and it's ongoing. I'm not dressing it up as more than that. Commercial terms are between me and Genie.

No customer quote here. If Andrew wants to say something about it, it'll be in his words, not mine.

One machine class. An autonomous e-bike is not a warehouse AMR and not a tractor. Treat these as lessons from one deployment, because that's what they are.

What we built, and what I'd ask

EdgeAI is the on-device voice and agentic layer for physical AI. Friday is the agent that runs on it. One C++ runtime:

Speech → Reasoning → Guardrails → Machine Capabilities → Response

Local ASR and voice activity detection, echo cancellation, interruption handling, small-language-model inference with tool calling, conversation memory, deterministic tool controls, and TTS. Built to sit alongside an existing perception stack rather than compete with it. The audio front-end runs wherever the human is (the machine, a phone), while the agent stays with the machine's state, reaching the autonomy stack only through declared capabilities over a single data bridge. Hardware-agnostic across NVIDIA, Qualcomm, Apple, and AMD silicon. EdgeAI is a member of the NVIDIA Inception program.

I wrote C++ for machines that have to work in real time. Autonomous systems at NASA and Rivian, then voice at Humane. That's both halves of this problem: the machine that has to keep itself safe, and the person who won't wait more than a beat for an answer.

I'd rather tell you your compute budget can't take this than sell you a pilot that disappoints you in week three.

If you're shipping or piloting a machine on Jetson, Linux, or ROS and voice is on the roadmap under "figure out later," run those five questions. Question one is the one I'd get wrong again if I weren't paying attention, and it's cheap to answer now and expensive to answer after you've built.

EdgeAI has a limited number of slots in our Design Partner Program. We'll put Friday on your existing hardware, wire it to one real workflow, and tell you honestly if it doesn't fit.

No new box. No cloud. No touching your autonomy stack.

Get in touch, or email me at andrew@runedge.ai. Tell me what your machine needs to say — and where you'll be standing when it says it.

To Andrew Hughes at Genie: thanks for putting this on a vehicle that has to survive real San Francisco traffic. Building a road-legal autonomous delivery bike solo is a harder thing than making it talk, and I'd rather people knew that part.

Sources

Primary

• Genie Mobility — Founders, Inc. portfolio. "Genie Mobility builds fully autonomous delivery e-bikes for dense city streets." Up to 28 mph on local roads and bike lanes; founded 2025; founder Andrew Hughes.
• NVIDIA Jetson Orin modules — module specifications. AGX Orin: 32GB / 64GB 256-bit LPDDR5.

Related reading

• Barge-in and interruption handling for on-device voice agents
• Time-to-first-token: why voice agents live or die on latency
• How to build an offline voice assistant on NVIDIA Jetson Orin
• Matic proved people want to talk to their robots