Voice is the least forgiving channel for AI. A chat user tolerates a two-second pause. A caller hears silence and assumes the line dropped. A chat user can scroll back. A caller who mishears an account balance has no transcript to check.
Voice agents in customer service work when three things are engineered deliberately: latency, handoff and trust. Get any one wrong and callers learn to say “agent” until a human picks up.
Latency is a pipeline, not a model setting
A voice turn has several stages: speech recognition, turn detection, reasoning, any tool calls, speech synthesis and network transport both ways. Each adds delay. Swapping to a faster model fixes one stage and ignores the rest.
What helps in practice:
- Stream everything. Start synthesising the first sentence while the rest is still being generated.
- Tune turn detection. End-of-speech detection that waits too long feels slow; too short and the agent interrupts.
- Support barge-in. Callers interrupt. The agent should stop talking and listen, not finish its paragraph.
- Pre-fetch context. Look up the caller’s account from the number or IVR input before the first question, not after.
- Mask tool latency honestly. “Let me check that order for you” is fine. Dead air is not.
- Measure per stage. Track time from caller silence to first audio, broken down by component, at percentiles rather than averages.
Run load tests with realistic audio — accents, background noise, poor mobile lines. Lab recordings flatter every system.
Handoff is the product
The most important feature of a voice agent is how it hands a call to a person. Most frustration comes from failed or blind transfers: the caller repeats everything, or waits in a queue with no context passed along.
Design handoff as a first-class flow:
- Clear triggers. Repeated misunderstanding, detected frustration, a request for a human, high-risk intents such as complaints, bereavement, fraud or vulnerability indicators.
- Warm context transfer. The human sees a summary, verified identity status, the intent and what the agent already tried.
- No dead ends. Every path, including failures, ends with a human option or a clear callback.
- Queue awareness. If wait times are long, offer a callback rather than a transfer into silence.
Measure handoff quality, not only containment. A high containment rate that hides callers giving up is not success.
Trust comes from boundaries
Callers trust a voice agent that is clear about what it is and what it can do.
- Disclose that the caller is speaking with an automated system where required, and ideally everywhere.
- Confirm critical details back — amounts, dates, addresses — before acting.
- Keep authentication strong. Voice alone is not a reliable identity check; use the same verification steps your human agents use.
- Restrict actions by risk class. Reading an order status and issuing a refund are not the same permission.
- Do not let the agent improvise policy. Answers about fees, eligibility or terms should come from retrieved, versioned sources.
Recording and transcription rules vary by region and sector. Confirm consent and retention requirements with counsel before launch.
Evaluation for voice
Text evaluation is necessary but not sufficient. Add:
- Audio test sets covering accents, languages and noisy environments
- Scripted adversarial calls — interruptions, topic switches, attempts to bypass verification
- Transcript sampling reviewed by experienced agents each week
- Regression runs whenever the speech, reasoning or synthesis component changes
Your best human agents are the right reviewers. They know what a good call sounds like.
Operations after go-live
Voice agents need the same run discipline as any production system: dashboards for latency and handoff, alerts on error spikes, a runbook for degrading to the traditional IVR, and a named owner on call. Plan for telephony and provider outages — the fallback path should be tested, not assumed.
Where to start
- Pick one high-volume, low-risk intent such as order status or appointment changes
- Map the existing call flow and handoff rules with frontline supervisors
- Set latency and handoff targets before choosing components
- Pilot on a share of calls with full transcript review
- Expand intents only when handoff quality holds
Incubics builds voice agents with handoff and evaluation designed in from discovery, not added after the first complaint.