How to A/B test Deepgram voices.
“The order is 418. Sorry, 480.” A transcript can contain every word and still leave the next system with the wrong order number.
Stephen Stilwell · September 14, 2026 · Product analysis · Content updated October 5, 2026What the product offers
Deepgram offers speech-to-text, text-to-speech, and voice agent APIs. Official source ↗
Build a Deepgram voice comparison plan.
The short answer: keep your scripts fixed, render them with two supported voice models, and compare blinded clips in a counterbalanced order. Measure pronunciation, clarity and task understanding separately from latency.
Deepgram selects Aura voices using a model name. Its voice model reference lists the choices. Its latency guide distinguishes time to first byte from total request time. References checked October 25, 2026. The study design below is my proposed method, not a native experimentation feature.
Your entries stay in this page until you leave. Use fictional data and download your plan. This planner does not make API requests or run a statistical test.
Five scripts worth including
- Correction: “Your appointment is at two. Sorry, three thirty. Please confirm three thirty.” Check what the listener understands as final.
- Numbers: “Your reference is four eight zero.” Ask the listener to repeat it.
- Acronyms: “You can export the CSV from your CRM.” Define the required pronunciation in your reference.
- Names: Use a fictional name with an agreed reference pronunciation.
- Handoff: “I couldn’t complete the change. A person will help you next.” Check clarity and the expected next step.
For a first listening pilot, assign anonymous clip labels and alternate A/B and B/A order across listeners and scripts. Record clarity and pronunciation ratings on a defined 1–5 scale, listener preference, response errors and the number of completed comparisons. A small pilot is useful for finding problems; it does not establish a universal winner. For a live conversion experiment, define assignment, a primary outcome and the required sample size before launch.
Keep recognition separate: testing a TTS voice does not test speech recognition or your booking integration. Use the voice agent checklist to inspect the final record as well as the conversation.
The decision that matters
Separate transcription from task interpretation. First ask whether the words were captured accurately. Then ask whether the consuming application understands the correction. Names, numbers, background noise, and interruptions deserve explicit examples because a general accuracy score may not describe your business task.
Try one bounded task.
Prepare ten short consented or self-recorded clips using fictional names and order numbers. Include corrections and modest background noise. Create a reference transcript and expected final value for each clip. Compare both transcript fidelity and the downstream value.
What to capture
Count transcription errors and final-value errors separately. Record audio conditions, sample size, processing settings, and any manual fixes.
Deepgram filler words: what should the transcript keep?
Deepgram documents a filler_words=true speech-to-text option. With the option off or omitted, “uh” and “um” are removed for readability. Its help page lists Nova, Nova-2 and Nova-3 general models for this feature, so check support for your selected model rather than assuming every endpoint behaves alike. Deepgram filler-word documentation, checked October 5, 2026.
For a proposed test, send the same consented clip with the setting off and on. Keep the model and other settings fixed. Compare the transcript to a reference, then check the final order number. Keeping a hesitation does not establish that the application understood a correction. Record both outcomes separately.
How to A/B test voice variations in Deepgram
Testing an incoming transcript and choosing the voice that speaks back are separate jobs. Deepgram’s Aura documentation selects a text-to-speech voice through its model name, for example aura-2-thalia-en. Use the current voice and language list to choose two candidates appropriate to your audience. Source checked October 5, 2026.
This is my proposed evaluation method, not a built-in Deepgram experiment feature or a completed test.
- Keep the words fixed. Use the same short scripts for both voices: a greeting, a corrected booking, a number, an unfamiliar name and an escalation. Keep output format, playback device and volume consistent.
- Hide the voice names. Label the recordings A and B, and vary which plays first. Ask listeners to repeat the appointment details and identify words they missed before asking which voice they prefer.
- Separate quality from delivery. Record comprehension errors and requests to repeat. If latency matters, measure time to first audio separately under the same network conditions and record failures.
- Choose the rule before listening. A pleasant voice that makes the date harder to understand should not win a booking task. Retain ratings, sample size, script, model IDs and generation date.
- Retest in a conversation. The selected voice still needs checks for interruptions, corrections, handoffs and the final system record. A small listening comparison cannot establish production reliability.
For the full conversation, follow the Retell AI appointment-correction test plan and use the voice AI agent testing checklist. For longer narration, the ElevenLabs audio evaluation asks what needs changing when a reader becomes a listener.
Keep an experiment record with each voice clip.
Separate the model ID, fixed input text, expected pronunciation, audio format, environment and observed result. A preferred voice can still mispronounce a name, and correct speech can still lead to an incorrect booking.
| Record | Example or decision |
|---|---|
| Experiment and script | voice-pilot-01 / correction-03; labels you define |
| Candidate models | Two supported model IDs from the current voice list |
| Reference | The exact text and agreed pronunciation, with the final time or number |
| Listening order | Anonymous labels; alternate A/B and B/A across listeners and scripts |
| Outcomes | Pronunciation errors, clarity rating, task comprehension and preference |
| Timing | Time to first byte and total request time recorded separately |
Deepgram documents a tag request parameter for grouping and filtering TTS usage. Tags are reporting labels; they do not randomize users or run an experiment. Keep personal information out of labels. See Deepgram’s tagging reference.
Names, acronyms and numbers need their own check.
Set the required pronunciation before listening. Test the original spelling with both voices first. If you change spelling or punctuation to improve the output, record the exact change and rerun both candidates against that version. Deepgram’s TTS prompting guide discusses pronunciation and pacing techniques. Those techniques still need checking against the actual audio for your task.
Sources checked October 25, 2026. This is a proposed evaluation workflow. No Deepgram audio was generated or benchmarked for this update.
Download the listening scorecard ↓Where this analysis stops
No recordings or API requests were made for this article. Ten clips would be a small diagnostic sample, not a general benchmark.
Source reviewed September 14, 2026. Product access and terms can change. These source links have no affiliate tracking added.