Measured first.
Then made it fast.
The voice provider wasn't the bottleneck. The agent's prompt was.
−39%
End of speech to first audio — 6.4 s down to 3.9 s
32 → 4 tools
Prompt 65% smaller: voice gets only the tools it needs
~4.70 Kč
All-in cost of a 2.5-minute call. Model cost cut by 68%.
Architecture
Latency breakdown · median per turn
From the user going quiet to the first sound back, measured live.
Hover or tap a segment
The telephony decision
For phone calls, the client could switch to ElevenLabs Agents. I ran both paths with the same voice, model and questions, twice each.
End of speech → first audio · hover a point
Recommendation
Switching works, at the same speed and cost. It goes live once:
- Tool calls reliably under 15 s
- Give the agent the current time
- A filler line while it works — “One moment, let me check…”
- A stable endpoint instead of a tunnel
The test caught the hidden timeout before customers on the phone did.
Side quest · local model on the same VPS
For background jobs like sorting e-mail and pulling out amounts and company IDs. 15 test messages, 4 fields each.
| Model | All 4 correct | Median / msg | Peak RAM | Tokens/s |
|---|---|---|---|---|
| Gemma 4 E2BQAT | 15/15 | 5.8 s | ~5.3 GB | 19.5 |
| Qwen 3.5 2BQ4_K_M | 10/15 | 4.8 s | ~3.0 GB | 21.1 |
| Qwen 3.5 4BQ4_K_M | 10/15 | 11.5 s | ~5.2 GB | 10.5 |
Long input · Gemma 4 E2B · time to first answer
Input tokens. Answers stayed correct at every length; speed is what breaks.
Gemma 4 E2B is right for background jobs, not for chat.
Also shipped on the engagement
- 01Agent identityThe agent became “Jarvis” in chat and voice, with global instructions documented and verified.
- 02Files round-trip planHow the agent can hand back edited files, and why Word/Excel formatting can’t survive.
- 03Ops hygieneBackups, rollbacks and a port audit on a shared production server.
I add voice to existing AI agents, chats and internal tools, and measure it until it's fast and cheap enough to use every day.