How is data accuracy measured?

A call counts as accurate only if every detail the agent saved matched exactly; calls that save nothing are excluded from this metric. The Cascade baseline scores highest here at 94.1%.

Is there a cost comparison?

Yes — cost per minute is taken from each vendor's published rates. For example, GPT Realtime 2.1 Mini is $0.020/min, GPT-Live 1 + GPT-6 Sol is $0.055/min, and Phonic v1 is $0.140/min. Some models (Gemini 3.8 Live, Gemini 3.8 Live Extended Thinking, Amazon Nova 2 Sonic) have no published rate.

Which model should I pick?

No model wins on everything. GPT Realtime 2.1 is the most reliable overall and best on Medicare calls; Azure Realtime is the fastest and best at handling caller corrections; Phonic v1 never stalls; GPT Realtime 2.1 Mini is the cheapest. The site recommends picking based on which measure matters most to you.

Cekura S2S Benchmarks — Speech-to-speech models tested on real phone calls

Cekura Bench is a public speech-to-speech (S2S) model benchmark from Cekura that ranks realtime voice AI models on how they perform on actual phone calls. Each model answered live test calls for two healthcare scenarios — booking a doctor's appointment and a Medicare intake line — and was scored on reliability, task success, data accuracy, response time, stalled calls, and cost per minute. The leaderboard covers 11 speech-to-speech models from OpenAI, Microsoft, Google, xAI, Amazon, Deepslate, and Phonic, plus a non-ranked cascade baseline (speech-to-text, then an LLM, then text-to-speech) for reference.

Key Features

Real-call testing

Every model answered the same real phone calls: around 59 scenarios for a healthcare receptionist (book, move, or cancel an appointment) and 23 scenarios for a Medicare intake line that collects consent and details, then routes the caller to the right team.

Five scored metrics plus cost

Each call is checked on response time, stalled calls (10-second silences after the caller speaks), data accuracy (whether saved details match exactly), success rate (right words and right details), and reliability (pass³ — passing every check on all 3 runs).

Ranked leaderboard

Models are ranked on reliability (pass³), with sortable columns for success rate, data accuracy, stalled calls, response time, and published cost per minute. As of the Oct 7, 2026 update, OpenAI's GPT Realtime 2.1 leads at 79.3% reliability, followed by GPT-Live 1 + GPT-6 Sol (75.6%) and Azure Realt

Failure breakdowns by call type

Per-model analysis shows where each model breaks: plain bookings, multiple requests per call, caller corrections, background noise, interruptions and barge-in, silence/cough/packet loss, accents and slow speech, tool or system errors, safety and scope, and Medicare compliance and routing.

Qualitative call reviews

Written reviews from listening through every call per model, noting what each is good at and where it fails — e.g., GPT Realtime 2.1 'almost never fails' but struggles on noisy phone lines, while Azure Realtime is the best at handling caller corrections but often drops IDs from Medicare handoff note

Cascade baseline for context

A traditional pipeline (Flux speech-to-text, then GPT-4.1, then ElevenLabs Flash text-to-speech) is included for reference, not ranked — it scored 82.9% reliability, 93.1% success rate, and 94.1% data accuracy, beating every native S2S model on accuracy.

How It Works

Step 1

Each model is given the same set of scripted call scenarios: 59 short-form healthcare receptionist calls and 23 long-form Medicare intake calls.

Step 2

Every scenario is run three times, and a scenario only counts as 'reliable' (pass³) if the model passed every check on all 3 runs.

Step 3

Calls are graded automatically on five checks — quick reply, no long silences, details saved right, right outcome, and consistent results — plus per-minute cost from each vendor's published rates where available.

Step 4

Results are published as a sortable leaderboard plus per-scenario-type breakdowns and human-written reviews of what went wrong on the actual calls.

Pros & Cons

Pros

Tests on real phone calls across two concrete suites (59 clinic booking scenarios, 23 Medicare intake scenarios) instead of synthetic single-turn tasks
Reliability is measured with pass³ — three runs per scenario — which captures consistency, not just one-off success
Covers realistic failure modes most benchmarks ignore: barge-in, background noise, packet loss, caller corrections, and tool errors
Includes practical buyer dimensions: median response time, stalled-call rate, data accuracy, and published cost per minute
Ships with written call-by-call reviews explaining why each model fails, not just numbers

Who It's For

Best for

Teams building realtime voice agents who need to pick a speech-to-speech model for phone-based workloads
Buyers deciding between OpenAI, Microsoft, Google, xAI, Amazon, Deepslate, and Phonic voice models based on reliability and cost
Teams in healthcare or other regulated phone workflows who care about data accuracy and compliance-style handoffs
Anyone who wants to know how native S2S models compare to a traditional cascade pipeline on real calls

Not ideal for

Teams needing benchmark coverage outside healthcare call scenarios
Teams evaluating text-to-speech or speech-to-text components in isolation rather than end-to-end S2S models

Use Cases

Choosing a speech-to-speech model for a realtime voice agent before committing to one

Comparing models on reliability rather than a single success metric

Understanding how models fail on realistic call conditions like background noise, interruptions, accents, and tool errors

Benchmarking S2S models against a traditional cascade pipeline for healthcare phone workloads

Tracking cost per minute and cost trade-offs across vendors like OpenAI, Microsoft, Google, xAI, Amazon, Deepslate, and Phonic

FAQ

What is Cekura Bench?

Cekura Bench is a public benchmark that ranks speech-to-speech AI voice models by how they perform on real phone calls. Models are scored on reliability, task success, data accuracy, response time, stalled calls, and cost per minute.

Which models are benchmarked?

Eleven speech-to-speech models: GPT Realtime 2.1, GPT-Live 1 + GPT-6 Sol (low thinking), Azure Realtime, Phonic v1, Grok Voice Think Fast 2.0, Gemini 3.1 Flash Live Preview, Deepslate Opal, Gemini 3.8 Live Extended Thinking, Amazon Nova 2 Sonic, GPT Realtime 2.1 Mini, and Gemini 3.8 Live. A cascade pipeline (Flux → GPT-4.1 → ElevenLabs Flash) is included as a reference baseline, not ranked.

What do the models get tested on?

Two call suites: a healthcare receptionist (59 short-form scenarios — patients booking, moving, or cancelling appointments) and a Medicare intake line (23 long-form scenarios — collecting consent and details, then routing the caller to the right team).

What does 'reliability (pass³)' mean?

Cekura S2S Benchmarks