🎙️ How to Test Whisper Accuracy on Your Own Audio

I wasted two weeks trusting a 5.6% number I found on a leaderboard. My actual audio (three-person remote calls, one strong Glaswegian accent, a lot of product jargon) scored nowhere near it, and I only found out after shipping captions that mangled a customer's name in twelve places. The fix was not a better model. The fix was a test set I built in an afternoon and a scoring script I should have written first.

Every page ranking for "whisper accuracy" hands you someone else's word error rate. LibriSpeech. An Open ASR Leaderboard composite. A vendor model card. None of them recorded your microphone, your room, or your speakers. So this post gives you no accuracy number at all. It gives you the protocol to produce your own in about 90 minutes, plus the cost model to decide where to run it.

By the numbers#

• ~1% of transcriptions in the peer-reviewed Koenecke et al. FAccT '24 study contained entire hallucinated phrases absent from the audio; 38% of those included explicit harms such as invented associations or false authority.
• 809M vs 1550M parameters: large-v3-turbo against large-v3, achieved by cutting the decoder from 32 layers to 4. OpenAI's own wording is "way faster, at the expense of a minor quality degradation."
• 8 datasets are averaged into the Open ASR Leaderboard headline score, from LibriSpeech audiobooks to CallHome telephone audio.
• $0.006 per minute and a 25 MB upload cap on OpenAI's hosted transcription API. At common MP3 bitrates that ceiling can arrive around the half-hour mark.
• ~50 to ~550 hours/month: the spread of break-even estimates across competing published analyses, which tells you nobody shares a cost model.

Build the test set before you install anything#

Thirty to sixty minutes of stratified audio beats ten hours of one type. The goal is coverage of failure conditions, not volume.

Sample deliberately across the axes that actually move the score. Graham & Roll, publishing in JASA Express Letters in early 2024, found Whisper recognises American English better than British and Australian English, native accents better than non-native, and read speech better than conversational speech. That is four axes handed to you for free. Add your own noise floor, your overlapping-talk clips, and your domain vocabulary — product names, drug names, ticker symbols, whatever your business runs on.

Then the part nobody else tells you: include pure silence and non-speech on purpose. Room tone. Hold music. A minute of keyboard clatter with no voice. The Koenecke study found hallucinations occurred disproportionately for speakers with longer shares of non-vocal duration, a common symptom of aphasia. Silence is the trigger condition, not speech. A test set made only of continuous clean speech will never surface the failure mode that does the most damage.

I aim for something like: 20 clips of 90 seconds each, roughly 40% your dominant recording condition, 30% the hard accents and crosstalk, 20% jargon-dense passages, 10% silence and non-speech with empty references. Keep clips short so a bad one is cheap to re-transcribe by hand.

So what: an afternoon of sampling gives you a number that predicts production. A leaderboard gives you a number that predicts LibriSpeech.

Write the transcription protocol down, then make the ground truth#

Your reference transcript decides your WER. Two humans transcribing the same clip will disagree on enough tokens to shift the result by a percentage point or more, so the protocol is part of the measurement and must be published with it.

Decide and document, before anyone types:

• Disfluencies. Are "um", "uh" and false starts in the reference or not? Whisper generally drops them. If your reference keeps them, every one becomes a deletion and your WER inflates for a behaviour you probably wanted.
• Numbers. "Twenty twenty-six" or "2026"? Pick one and enforce it, or handle it in normalization (see below) rather than in the reference.
• Proper nouns. Spell out the canonical form in a glossary file. This doubles as your keyword-error-rate word list later.
• Crosstalk. When two people talk over each other, do you transcribe both in speaker order, or mark the region as unintelligible and exclude it from scoring? Either is defensible; silently mixing the two is not.

Store references as plain UTF-8 text next to the audio, one file per clip, plus a PROTOCOL.md in the same folder. When someone later disputes your number, that folder is the whole argument.

Score it with jiwer, and publish the normalization#

Here is the step the entire "run Whisper locally" cluster skips. Those guides walk you through pip install faster-whisper or a whisper.cpp CMake build and end at "you now have a transcript." You never learn whether the model you installed is good enough for your audio.

jiwer, the standard Python library, deliberately makes normalization opt-in. It exposes ToLowerCase, RemovePunctuation, ExpandCommonEnglishContractions, RemoveMultipleSpaces and ReduceToListOfListOfWords as transforms you pass via truth_transform and hypothesis_transform. The measured WER is a function of that pipeline, not of the model alone. An unstated normalization makes a quoted percentage meaningless.

import jiwer

norm = jiwer.Compose([
    jiwer.ToLowerCase(),
    jiwer.ExpandCommonEnglishContractions(),
    jiwer.RemovePunctuation(),
    jiwer.RemoveMultipleSpaces(),
    jiwer.Strip(),
    jiwer.ReduceToListOfListOfWords(),
])

out = jiwer.process_words(
    refs, hyps,
    reference_transform=norm,
    hypothesis_transform=norm,
)
print(out.wer, out.substitutions, out.deletions, out.insertions, out.hits)

Report the substitution, deletion and insertion breakdown, never the headline percentage alone. The shape tells you what to fix. Heavy substitutions on your glossary terms is a prompting or fine-tuning problem. Heavy deletions usually means chunking or VAD is eating audio. Heavy insertions means fabrication, and you go straight to the next section.

I keep the whole thing as one score.py in the repo with the transform block at the top, so the number and the pipeline that produced it live in the same commit. Same discipline I apply to caption accuracy testing, where a stray punctuation rule can flip a pass into a fail.

So what: commit the transform chain. A WER without it is a vibe.

Metrics that decide usability, beyond WER#

WER treats every token equally, which is wrong for almost every real job. Track these alongside it:

Keyword / entity error rate. Score only the words in your glossary — customer names, SKUs, drug names. A 6% overall WER with 40% of proper nouns wrong is unusable for a searchable archive and fine for gist summaries. Compute it by filtering both strings to glossary tokens before scoring.

Insertion rate on empty references. Your hallucination proxy. Any non-empty output against a silent clip is fabricated.

Timestamp drift. Vanilla Whisper emits utterance-level timestamps that can be off by several seconds. If you are cutting video to those timestamps, measure the offset on a handful of clips by hand before you trust them.

Diarization error rate. Only if you need to know who spoke. Whisper has no native speaker diarization at all.

The hallucination test, as a runnable procedure#

Run your silence subset. Count insertions. That is the test.

OpenAI's own model card admits the models are "prone to generating repetitive texts, which can be mitigated to some degree by beam search and temperature scheduling but not perfectly," and that because the model combines next-word prediction with transcription, it may output text that was never spoken. This is documented vendor behaviour, not a bug you can report.

Three mitigations worth measuring:

1. Temperature fallback tuple rather than a hard temperature=0.0. Greedy decoding at zero is exactly the regime that loops. The standard fallback ladder (0.0, 0.2, 0.4, 0.6, 0.8, 1.0) lets the decoder retry a bad segment.
2. compression_ratio_threshold. Repetitive output compresses unusually well; the threshold catches loops and triggers the fallback.
3. VAD preprocessing. Strip long non-vocal stretches before decoding. Since the FAccT finding points at non-vocal duration as the trigger, removing it removes the condition. This is the single highest-impact change I have made to a transcription pipeline.

Honest admission: I have not been able to reproduce the FAccT study's 1% rate on my own material, and I do not know whether that is because my mitigations work, because my audio differs, or because my sample of silence clips is too small to detect a 1-in-100 event reliably. Twenty silent clips cannot measure a 1% rate. Treat my insertion counts as a smoke alarm, not a rate estimate.

Sweep the models on your own set#

Once score.py exists, running a sweep costs you an evening of GPU time. Build a table with a WER column and a real-time-factor column on stated hardware: tiny, base, small, medium, large-v3, large-v3-turbo, distil-whisper, and both backends — faster-whisper and whisper.cpp. State your GPU, your batch size and your compute type, because a speed number without hardware is noise.

Expect the turbo tradeoff to be visible: 809M parameters against 1550M, with the decoder cut from 32 layers to 4. Whether "minor quality degradation" is minor for your jargon is precisely what the sweep answers.

Also test at least one non-Whisper model. On the Open ASR Leaderboard as reported in November 2025, NVIDIA's Parakeet recorded 6.34% WER against Whisper 6.43% while running at 3,332.74 RTFx versus 68.56: roughly comparable accuracy at about 48x the throughput. Whisper is no longer the automatic answer for English batch work.

Speed work on Whisper itself moves the same axis. This walkthrough covers the batching, quantization and kernel changes behind the usual order-of-magnitude claims, which is the context you want before you read your own sweep's speed column:

If your sweep says a different model wins on your audio, believe your sweep. Hardware sizing for the local runs is covered in the GPU guide, and quantized variants change the speed column considerably — see quantization explained.

So what: the sweep converts a religious argument about models into a two-column table.

The cost model, honestly#

The arithmetic everyone publishes is $0.006/min versus "free." That comparison is missing most of the cost.

A usable model has five terms: GPU or Mac capex amortized over its service life; electricity at your actual tariff and actual draw under load; engineer hours for setup and for every breakage after; the API bill you avoid; and the re-run rate — the fraction of local jobs that silently truncate, loop, or come back wrong and get processed twice.

That last term is the one that decides it, and it is the one nobody models. A published calculation puts a $600 Mac mini or a $700 to $900 used RTX 3090 at payback against gpt-4o-transcribe pricing after roughly 1,670 to 2,500 hours of audio. Other analyses in the same search results quote break-even anywhere from about 50 to 550 hours per month. That spread is not a disagreement about GPU prices; it is the absence of a shared model. Measure your own re-run rate during the sweep and plug it in. Run the numbers against your volume in the cost calculator, and the broader local-versus-hosted tradeoffs are laid out in local vs cloud AI.

For most readers the deciding variable turns out not to be price. It is whether the audio can legally leave the building, whether you need sub-second latency, or whether the 25 MB API cap forces you to write a splitter and re-joiner anyway. Whisper was trained on 680,000 hours of audio; none of that helps you upload a two-hour board meeting in one call.

What Whisper structurally cannot do#

Budget for these rather than discovering them in production:

• No speaker diarization. Add pyannote.audio.
• Utterance timestamps off by seconds. WhisperX adds word-level timestamps to roughly ±50 ms via wav2vec2 forced alignment, and wires in pyannote.audio 3.1 for diarization — which requires accepting a licence and supplying a Hugging Face token.
• 30-second window chunking. Long audio is stitched from windows, and the seams are where repetitions and drops appear.
• 25 MB upload cap on the hosted API, plus no streaming from the batch endpoint.

Takeaway#

Delete every borrowed WER from your notes. Spend the afternoon: 20 clips, a written protocol, a committed jiwer transform chain, a silence subset with empty references. You will finish with a number that is yours, reproducible by a colleague, and defensible when someone asks why the captions are wrong.