Testing AI phone agents before customers do.

I run seven live voice agents on Vapi and Retell. Language models aren't deterministic, so three good test calls prove little, and the calls people test with are the easy ones. Real callers interrupt, refuse details, ask for the owner, or call at 2am with water coming through the ceiling. Some of them try to manipulate the agent.

I needed answers to two questions before any agent touched a real phone number: does it do the right thing on the hard calls, and can it be talked into leaking data or breaking the rules?

The eval suite

• 38 scenarios across three agents (dental, plumbing, self-storage), each with a persona, a goal, facts revealed only when asked, and a success outcome. Each runs 3 times: 114 calls, about $0.25 a run.
• Deterministic checks first (did it call the booking tool, save the right fields, quote only real prices, hand over when it should); an LLM judge with a written rubric only for what code can't decide.
• Metrics that resist gaming: containment where a hang-up never counts, correct vs unnecessary transfers, missed handoffs as a critical failure, hallucination rate, and pass on all three runs. Baselines and a regression gate in GitHub Actions.

The security track (OWASP Top 10 for LLM Applications 2025)

• 21 attack scenarios, 7 per agent: direct and indirect prompt injection (including instructions hidden in CRM notes and tool results, and attacks in Arabic), system-prompt extraction, requests for other customers' details, excessive agency and unbounded replies.
• A random canary token planted in each call's system prompt, Faker fixture customers to detect cross-customer leaks, and a tool guard for saves or transfers the caller isn't entitled to.
• Every transcript passes through PII redaction (Presidio plus a regex engine) before it is judged, stored or published, with a publish guard that refuses to publish if anything slips through.

• The first functional run caught 5 missed emergency handoffs in 41 plumbing calls, including a caller reporting a gas smell, on the cheaper model. The agent sounded calm the whole time; it just did the wrong thing.
• The final run had zero criticals across all 177 calls. PII redaction scored 1.00 precision and recall on a 26-string labelled set; 135 tests run in CI, Presidio included.

• Test audio, not just text: latency, barge-in and speech-recognition errors on real phone audio.
• Push the least-privilege context builder into every live CRM integration, and add a tool-reply sanitizer for the open finding.
• Run the security track nightly against the deployed prompts, not only on changes.