So messen Sie die Genauigkeit der Anruftranskription (WER)
Measure speech-to-text accuracy on your own calls: build a reference set, normalise text, calculate word error rate per language and read the result honestly.
Kurze Antwort
To measure call transcription accuracy, calculate word error rate (WER): substitutions plus deletions plus insertions, divided by the number of words in a human-checked reference transcript. Build the reference from a random sample of your own calls, split by language, dialect and speaker channel. Apply the same text normalisation to reference and system output, score with a tool such as NIST sclite, then report results per group, and separately check names, amounts, dates and opt-out phrases.
Every speech-to-text vendor has an accuracy number. Almost none of those numbers come from your calls: your customers, your accents, your phone lines, your background noise and your vocabulary. The only figure that predicts how a transcript will behave in your contact centre is one you measure yourself.
This guide is a neutral method. It shows how to build a reference set, agree normalisation rules, run a comparison and read the results honestly, including the errors that a single percentage hides. It quotes no accuracy figure for any product, including ours.
What WER counts
Word error rate compares two versions of the same speech: the reference, a transcript checked by a person, and the hypothesis, the output of the speech-to-text system. After aligning them word by word, every difference is one of three kinds:
- Substitution (S): a word was replaced by a different word.
- Deletion (D): a word in the reference is missing from the output.
- Insertion (I): the output contains a word that was not said.
WER = (S + D + I) divided by N, where N is the number of words in the reference.
Worked example (illustrative). Reference: "please send the invoice to my office today" (8 words). Output: "please send invoice to my new office today". The word "the" was deleted (D = 1) and "new" was inserted (I = 1), and nothing was substituted. WER = (0 + 1 + 1) / 8 = 25 per cent. Now suppose the output instead read "please send the invoice to my officer today": one substitution, WER = 1 / 8 = 12.5 per cent, yet that single word changed the meaning more than the 25 per cent example did. This is the first lesson: WER treats every word the same, and your business does not.
Because insertions are counted, WER can exceed 100 per cent on very poor output. A low WER also does not mean a usable transcript if the few errors are the words that matter.
Build a reference set from your own calls
The reference set is the part that takes effort, and the part that decides whether the number means anything.
1. Sample at random, then stratify
Pull calls at random from a period long enough to include busy and quiet days. Then make sure you cover every group you care about. Split the sample by:
- Sprache. Measure each language separately. A blended figure hides a weak language behind a strong one.
- Dialect and accent. For Arabic, that means the dialects your callers actually speak, not only Modern Standard Arabic. An Arabic call-domain benchmark paper on arXiv describes the challenge directly: dialectal diversity, phonetic complexity and telephone-specific problems such as audio quality, background noise and conversational speech (Sources, item 3).
- Channel. If agent and customer are recorded on separate channels, score them separately. Customers usually sound worse: mobile lines, speakerphone, traffic.
- Call type. Collections, support and sales calls use different vocabulary and pacing.
- Audioqualität. Include poor calls on purpose. Production traffic contains them.
2. Choose a sample size you can defend
Count words, not calls. A reference set of a few hundred words per group will move a lot with one difficult call. Aim for several thousand reference words per language and channel you intend to rely on. If a group is small, label its result as indicative.
3. Have people write the reference
Use native speakers of each language and dialect. Give them a written style guide so two transcribers handle numbers, hesitations and foreign words in the same way. Where possible, have a second person check a share of each file. Disagreement between your own transcribers is useful information: it sets a floor under what any system can score.
4. Mind privacy
Recordings contain personal data. Check your consent, retention and access rules before copying calls into a test folder, restrict who can open them, and delete the working copies when the test ends. Qualified counsel can tell you what your jurisdiction requires.
Normalisation rules that change the score
Before comparing, both texts must be put in the same form. Otherwise the scorer counts differences in writing as recognition errors. Decide these rules on paper, apply them to reference and output alike, and then do not change them between runs.
| Issue | Example | Decision to make |
|---|---|---|
| Numbers | "15" vs "fifteen" | Pick digits or words, and convert both sides |
| Punctuation and case | "Hello, John." vs "hello john" | Usually remove punctuation and lower-case |
| Hesitations | "um", "uh", false starts | Keep or drop them on both sides |
| Arabic orthography | Alef and hamza variants, ta marbuta vs ha, optional diacritics | Unify variant letters; usually strip diacritics |
| Dialect spellings | Colloquial words with no fixed spelling | Write a spelling list for common words |
| Code-Switching | An English brand name inside an Arabic sentence | Decide Latin or native script, and stick to it |
| Compound words and clitics | Attached prefixes in Arabic, compounds in German | Agree what counts as one word |
Normalisation can move a score by several points, so quote the rules whenever you quote the number. A result that cannot be reproduced under stated rules is just a claim.
Running the comparison
You can write a short script for word-level edit distance, but a long-established scorer saves time and gives a result others can check. The NIST Scoring Toolkit (SCTK) is such a package. NIST's tools page lists it as a speech recognition scoring toolkit that includes SCLITE, ASCLITE, tranfilt, hubscr, SLATreport and utf_filt (Sources, item 2). The project README lists sclite ("Score Lite") alongside sc_stats, described as sclite's statistical significance comparison tool, rover (Recognition Output Voting Error Reduction) and asclite. It can be compiled from source, or built and run with the Dockerfile included in the repository, using a command of the form sclite -i wsj -r ref.txt -h hyp.txt, where -r is the reference file and -h the hypothesis file (Sources, item 1). When we checked it, the GitHub repository said it had been archived by its owner on October 7, 2026 and was read-only, so pin the version you use and record it.
A practical run looks like this:
- Export the system transcript for each sampled call, one channel at a time.
- Normalise both reference and system files with the same script.
- Convert to the line format the scorer expects, with a stable identifier per utterance.
- Score each language and channel group separately.
- Keep the aligned output, not only the totals, so you can read the actual errors.
Reading the results honestly
Per speaker channel
Report agent and customer sides separately. If the customer channel scores far worse, the usual cause is the phone line, and the fix is not the same as for a poor agent channel (headset, room, microphone gain).
Per language and dialect
Do not average across languages. A system that is strong in one language and weak in another gives a middling blended score that describes neither. Rank your groups by call volume, and look first at the weakest large group.
Confidence, in plain words
Your sample is a small part of all your calls, so the result is an estimate. Two things make it more reliable: more words, and a random sample. A simple check is to split the sample in two, score each half, and compare. If the halves disagree by a lot, you need more data. When comparing two systems on the same files, a paired significance test is the right tool, which is what sc_stats is for (Sources, item 1). If the difference is small and the sample is small, treat the two as tied.
Read the errors, not only the rate
Sort the substitutions by frequency. Often a handful of words (a product name, a place, a common colloquial word) account for a large share of the errors. That is good news: those can be handled with a vocabulary list or a process fix, or at least known and worked around.
Errors that matter more than WER
For contact-centre use, score a second set of items on their own:
- Names. Customer and company names, and street names.
- Amounts. Currency values, including the unit. "Fifteen" versus "fifty" is a one-syllable difference on a bad line.
- Dates and times. Appointments, due dates, payment promises.
- Account and reference numbers. Digit-by-digit exactness.
- Opt-out and consent phrases. "Do not call me again", "stop", and their equivalents in each language and dialect you serve.
- Negations. A missed "not" reverses the meaning of a sentence and costs one word of WER.
For each category, count how many occurrences there were in the reference and how many came out exactly right. A category-level exact-match rate is easier to explain to a compliance or operations manager than a WER.
Using the result
- QA sampling. Know which language and channel groups have noisier transcripts, so reviewers listen to the audio where the text is least reliable.
- Zusammenfassungen. Check that key facts in a summary (amount, date, outcome) match the call. The AI call summaries guide explains the review habit that keeps summaries trustworthy.
- Compliance keyword search. A search for opt-out phrases is only as good as the transcript of those phrases. Test it with the exact phrases your callers use, in each dialect.
- Vendor comparison. Run every candidate on the same files with the same rules. Our comparison of multilingual AI and English-only tools explains why language coverage should be tested, not assumed.
- Re-testing. Repeat the test on fresh calls every few months, and after any change of provider or settings.
For background on how transcripts feed review work, see Was ist Sprachanalyse? und die QA-Bewertungsbögen page.
Wie DialerBee passt
DialerBee's Anrufprotokoll supports 11 languages, with dialect awareness. According to the feature page, it separates the agent and customer audio channels, transcribes each side independently with timestamps and detects the language automatically. That channel separation fits the per-channel scoring described above. DialerBee does not publish a word error rate figure, and this article does not claim one: use the method above on a sample of your own calls, in your own languages, before you rely on transcripts for decisions.
Häufig gestellte Fragen
What is a good word error rate for call transcription?
There is no single good number, because the score depends on the audio, the language, the dialect and the way you normalise text. A figure quoted by a vendor on its own test set tells you little about your calls. Measure on a sample of your own recordings, per language and per channel, and judge the result against what you need to do with the transcript: summaries, search or compliance checks.
How do I calculate word error rate by hand?
Align the reference transcript with the system transcript, count the substitutions, deletions and insertions, add them up and divide by the number of words in the reference. For example, 1 substitution, 1 deletion and 1 insertion against a 10-word reference is 3 errors out of 10 words, a word error rate of 30 per cent. Because insertions count, the rate can exceed 100 per cent.
How many calls do I need to measure accuracy?
Enough that every language, dialect and channel you care about has its own sample of several thousand reference words, drawn at random from real calls. A handful of clean calls says very little. If a group has only a small sample, report the result as indicative and widen the sample before you act on it.
Why does normalisation change my WER score?
Because the scorer treats any difference in the written form as an error. If the reference writes 15 and the system writes fifteen, or one side keeps punctuation and the other drops it, you get errors that are not recognition mistakes. Agree the rules first, apply them to both reference and system output, and keep them fixed between tests.
Is WER enough to judge a transcription system?
No. WER weights every word equally, but a wrong name, amount, date or opt-out phrase can matter far more than ten missed filler words. Score the high-stakes items separately, for example the share of amounts and dates transcribed exactly right, and read a sample of full transcripts by eye.
Which tool should I use to score WER?
The NIST Scoring Toolkit (SCTK) includes sclite, a long-established scorer that compares a reference file against a hypothesis file. You can also write a short script around any word-level edit-distance routine. What matters is that the same tool and the same normalisation are used every time, so scores are comparable.
Quellen
- NIST Scoring Toolkit (SCTK), including sclite, sc_stats, rover and asclite. github.com/usnistgov/SCTK (abgerufen am 7. November 2026)
- NIST Information Technology Laboratory, Tools page (Evaluation Tools), listing the Speech Recognition Scoring Toolkit. nist.gov/itl/iad/mig/tools (abgerufen am 7. November 2026)
- Abo Obaidah et al., "A New Benchmark for Evaluating Automatic Speech Recognition in the Arabic Call Domain" (abstract page). arxiv.org/abs/2403.04280 (abgerufen am 7. November 2026)
Weiterführende Lektüre
- Anrufprotokoll
- AI Call Summaries and After-Call Work
- Was ist Sprachanalyse?
- QA-Bewertungskarten
- Mehrsprachige KI vs. rein englischsprachige KI
- Arabic speech recognition across dialects in call centres
- Arabic-English code-switching in voice AI and transcription
- Word Error Rate (WER)
- Arabic voice agent testing checklist before you go live
Verwandte Artikel
How to Measure VoIP Call Quality (MOS, Jitter)
11 Minuten Lesezeit
TechnologieOutbound Caller ID Not Showing: SIP Fixes
10 Minuten Lesezeit
BetriebHow to QA AI Voice Agent Calls: A Scorecard
10 Minuten Lesezeit
EinhaltungIndien: RBI-Regeln für Anrufe von Inkassoagenten ab 2027
9 Minuten Lesezeit
Bereit, DialerBee in Aktion zu sehen?
Fordern Sie eine 15-minütige Live-Demo an oder starten Sie eine 14-tägige kostenlose Testphase – ohne Folien, ohne Verpflichtung. Das Telefonieren beginnt, sobald Ihr eigener SIP-Trunk oder Ihre TK-Anlage verbunden und freigegeben ist.
14-tägige kostenlose Testphase · kostenloses KI-Guthaben · keine Kreditkarte · 11 Sprachen · BYOC · Kontrollen zur Unterstützung der Compliance