Whisper Accuracy by Language: Measured, Not Guessed

Almost every guide to Whisper says the same thing about model size: base is the sweet spot, small is overkill for most people. That advice was written by people testing in English.

Here is the same sentence-for-sentence content, recorded in eleven languages, run through both models:

Word error rate for space-delimited languages, character error rate for Chinese, Japanese and Korean. Lower is better.

Spanish and French are perfect on the smaller model. Polish is 30% wrong on it. Both are Latin-script European languages.

Language family predicts nothing

The two rows worth staring at:

• Polish, 30% on base. Latin script, Indo-European, geographically surrounded by languages that score near zero. Intuition files it next to Spanish. It performs like Russian.
• Swedish, 45% on base and 5.0% on small. A nine-fold difference — and Swedish sits in the same Germanic branch as Dutch, which scores an identical 3.7% on both models and genuinely does not need the bigger one.

Dutch and Swedish are close relatives with opposite answers. Whatever determines Whisper’s per-language accuracy, it is not the language family, and it is not the writing system either — Chinese and Japanese both land mid-table while Turkish, written in Latin script, is the worst result in the set.

The most likely explanation is simply how many hours of each language were in the 680,000-hour training set, and that distribution is not published per-language in a form you can plan around.

Practical rule: either measure the language you care about, or default to the larger model for anything that is not English.

Some languages are not usable at any browser-sized model

Four languages stayed bad even on small:

A 25% error rate does not read like a broken transcript. It reads like a slightly odd one. Every fourth word is wrong, the grammar is intact, and unless you have the audio in front of you there is no signal telling you which words to distrust.

That is why this site lists sixteen languages instead of ninety. A tool that offers you Hindi and returns a 52%-wrong transcript has not given you a starting point — it has given you something worse than nothing, because you will edit it as if it were mostly right.

If you need these languages, use a server-side medium or large model. The browser-sized ones are not close.

How this was measured

Nothing here is from a benchmark paper, so it is worth being explicit about the method and its limits.

1. One passage of meeting-style content — three topics, ordinary business vocabulary, no jargon — written in each language.
2. Rendered to speech with macOS say, one voice per language, 16 kHz mono WAV.
3. Transcribed with onnx-community/whisper-base and onnx-community/whisper-small, q8 quantized, 30-second chunks with 5-second overlap, language explicitly specified.
4. Scored by Levenshtein distance against the source text, punctuation stripped. Chinese scored after converting Traditional output to Simplified — skipping that step inflates the error rate to 28.4% purely from character variants.

Why these numbers are optimistic

Synthesized speech is clean: no background noise, no accent, no crosstalk, no microphone six feet away, consistent pace. Real recordings will score worse than this table.

Use these numbers to compare models against each other — that comparison holds, because both models heard exactly the same audio. Do not read them as the accuracy you will personally get.

Which model you actually get

On this site the choice is made from the language you pick, not left to you:

• base (73 MB) — English, Spanish, French, German, Dutch
• small (237 MB) — everything else

That is a 3.3× larger download for most languages, which is a real cost on a slow connection. It buys Russian users a drop from 24% to 3.4%, and Polish users a drop from 30% to zero. That trade is not close.

English keeps the smaller model because English is where base genuinely is enough — the same reason the conventional advice exists in the first place.

Common questions

Does a bigger model always win? No. Dutch scores 3.7% on both. And on English technical vocabulary, base and small trade wins — in one test small transcribed hyperthyroidism as hypothyroidism, a word meaning the opposite condition, which base got right. Bigger models fail less often but not strictly less.

Is quantization hurting these numbers? Some. Chinese on base scores 11.2% quantized to q8 versus 5.2% at full precision. But full precision base is larger than quantized small while still scoring worse than it, so it is not a useful middle option for browser use.

What about accents within a language? Not measured here, and it is a real gap — synthesized voices have no regional accent. Expect accented speech to widen the gap between model sizes rather than narrow it.

Can I check my own language before trusting it? Yes, and it takes two minutes: transcribe a recording where you already know every word, then count the errors. Any language, any tool. It is the only number that describes your audio.

Ready to try it — transcribe a file in your browser, pick your language from the list, and nothing gets uploaded.