English-Hebrew Code-Switching ASR Leaderboard
How well does speech recognition handle English with Hebrew words in it: "pick up some bamba from the makolet", "I have to go to Bituach Leumi"? Models are scored on a frozen test set (Hebrish/English-Hebrew-Codeswitching-ASR-Benchmark, commit 3cb0b2b3b3e4) with the open runner at GitHub.
Term recall is the share of Hebrew words written with the canonical romanized spelling (case-insensitive, whole word). It is the headline: a model can have a low WER and still mangle every Hebrew word. Unseen-term recall counts only Hebrew words absent from the public training split. Single speaker, read speech: see About before drawing conclusions.
Status: early-stage, work in progress (test set v1). Known issue: 22 of the 139 labelled terms are English tech words or product names, not Hebrew (JSON, grep, Proxmox, ...). Ranks are unaffected; absolute term recall is 2 to 6 points high. v1.1 will relabel them and rescore every model from its stored transcripts. Contributions welcome: more speakers, better labels, more models.
Benchmark repo · Inference code (Modal GPU runner) · Scorer · Test set · Result files · Write-up · Fine-tune website: hebrish.app
10 | Code-switching fine-tune | 73.3 | 70.1 | 76.2 | 63.9 | 10.5 | 1.1 | 2.2 | 1,550 | ⚠ Trained on English-Hebrew-Mixed-Sentences (2025-11-17), which overlaps 33 non-headline clips; 10 unseen-flagged clean term occurrences appear in its training text. |
Ranked by term recall, then WER. Lenient also accepts lexicon spelling variants (tsaharon for tzaharon), which is fairer to models never taught the canonical spelling. English control is 24 clips of the same speaker with no Hebrew: a fine-tune that wrecks English shows up there.
Adds 37 clips excluded from the headline: 33 overlap Whisper-Hebrish's training data (26 identical recordings, 7 re-recordings of training sentences) and 4 are older desktop-mic recordings. Comparable with the numbers on the Parakeet fine-tune's model card, which used all 121. Not a fair comparison for Whisper-Hebrish.
10 | Code-switching fine-tune | 68.2 | 63.7 | 70.7 | 60.2 | 11.1 | 1.3 | 2.2 | 1,550 | ⚠ Trained on English-Hebrew-Mixed-Sentences (2025-11-17), which overlaps 33 non-headline clips; 10 unseen-flagged clean term occurrences appear in its training text. |
Clean subset (84 clips, 172 Hebrew term occurrences) unless stated. Hover for values. With 172 occurrences, a gap of 2 to 3 points is about 4 to 5 words: treat it as noise.
Hits / occurrences for every Hebrew term in the clean subset. Hardest terms first.
Google Workspace | yes | 2 | 0/2 | 0/2 | 0/2 | 0/2 | 0/2 | 0/2 | 0/2 | 0/2 | 0/2 | 0/2 |
Every published benchmark round, newest first. A round is frozen when it is published; the leaderboard tabs always show the latest result per model.
2026-10-19: Round 3: Parakeet v1.0 and the Nemotron streaming fine-tune
Parakeet code-switching fine-tune v1.0 (run r6-en30, 339 min) and the experimental Nemotron streaming fine-tune (r6-nemotron-en30, same data, scored offline). Both on Modal L40S, 2026-10-19. Contamination check: no benchmark clip in either training manifest.
Test set v1 (dataset commit 3cb0b2b3b3e4). Full result files with every transcript: 8d3e79e.
| # | Model | Type | Term recall % | Unseen % | Lenient % | WER % | English control WER % |
|---|---|---|---|---|---|---|---|
| 1 | Parakeet v2 EN-HE code-switching (v1.0) | CS fine-tune | 73.3 | 70.1 | 76.2 | 3.3 | 2.2 |
| 2 | Parakeet v2 EN-HE code-switching (v3) | CS fine-tune | 67.4 | 63.2 | 70.3 | 4.0 | 2.5 |
| 3 | Parakeet v2 EN-HE code-switching (v2) | CS fine-tune | 65.7 | 60.7 | 67.4 | 4.2 | 2.4 |
| 4 | Parakeet v2 EN-HE code-switching (v1) | CS fine-tune | 62.2 | 61.5 | 63.9 | 4.6 | 2.5 |
| 5 | Nemotron Streaming EN-HE code-switching (checkpoint 2) | CS fine-tune | 62.2 | 54.7 | 64.5 | 4.7 | 2.5 |
| 6 | Whisper-Hebrish ⚠ | CS fine-tune | 54.1 | 56.4 | 56.4 | 6.5 | 4.7 |
| 7 | Whisper Large V3 | Stock | 45.9 | 46.2 | 47.7 | 7.5 | 2.4 |
| 8 | Whisper Large V3 Turbo | Stock | 40.1 | 39.3 | 41.9 | 8.9 | 2.4 |
| 9 | Parakeet TDT 0.6B v2 | Stock | 32.6 | 32.5 | 34.3 | 9.5 | 2.7 |
| 10 | Parakeet TDT 0.6B v3 | Stock | 31.4 | 31.6 | 35.5 | 10.5 | 2.7 |
2026-10-18: Round 2: code-switching fine-tune v3
Adds Parakeet EN-HE code-switching v3 (196 min of code-switching audio). Other models unchanged from round 1.
Test set v1 (dataset commit 3cb0b2b3b3e4). Full result files with every transcript: 3ff5e61.
| # | Model | Type | Term recall % | Unseen % | Lenient % | WER % | English control WER % |
|---|---|---|---|---|---|---|---|
| 1 | Parakeet v2 EN-HE code-switching (v3) | CS fine-tune | 67.4 | 63.2 | 70.3 | 4.0 | 2.5 |
| 2 | Parakeet v2 EN-HE code-switching (v2) | CS fine-tune | 65.7 | 60.7 | 67.4 | 4.2 | 2.4 |
| 3 | Parakeet v2 EN-HE code-switching (v1) | CS fine-tune | 62.2 | 61.5 | 63.9 | 4.6 | 2.5 |
| 4 | Whisper-Hebrish ⚠ | CS fine-tune | 54.1 | 56.4 | 56.4 | 6.5 | 4.7 |
| 5 | Whisper Large V3 | Stock | 45.9 | 46.2 | 47.7 | 7.5 | 2.4 |
| 6 | Whisper Large V3 Turbo | Stock | 40.1 | 39.3 | 41.9 | 8.9 | 2.4 |
| 7 | Parakeet TDT 0.6B v2 | Stock | 32.6 | 32.5 | 34.3 | 9.5 | 2.7 |
| 8 | Parakeet TDT 0.6B v3 | Stock | 31.4 | 31.6 | 35.5 | 10.5 | 2.7 |
2026-10-15: Round 1: baselines
First published round. Seven models: four stock (Whisper Large V3, Whisper Large V3 Turbo, Parakeet TDT 0.6B v2 and v3) and three code-switching fine-tunes (Parakeet v2 EN-HE v1 and v2, Whisper-Hebrish). All run on Modal L40S, greedy decoding.
Test set v1 (dataset commit 3cb0b2b3b3e4). Full result files with every transcript: aa09470.
| # | Model | Type | Term recall % | Unseen % | Lenient % | WER % | English control WER % |
|---|---|---|---|---|---|---|---|
| 1 | Parakeet v2 EN-HE code-switching (v2) | CS fine-tune | 65.7 | 60.7 | 67.4 | 4.2 | 2.4 |
| 2 | Parakeet v2 EN-HE code-switching (v1) | CS fine-tune | 62.2 | 61.5 | 63.9 | 4.6 | 2.5 |
| 3 | Whisper-Hebrish ⚠ | CS fine-tune | 54.1 | 56.4 | 56.4 | 6.5 | 4.7 |
| 4 | Whisper Large V3 | Stock | 45.9 | 46.2 | 47.7 | 7.5 | 2.4 |
| 5 | Whisper Large V3 Turbo | Stock | 40.1 | 39.3 | 41.9 | 8.9 | 2.4 |
| 6 | Parakeet TDT 0.6B v2 | Stock | 32.6 | 32.5 | 34.3 | 9.5 | 2.7 |
| 7 | Parakeet TDT 0.6B v3 | Stock | 31.4 | 31.6 | 35.5 | 10.5 | 2.7 |
Data dictionary
Every term, field and metric used by the benchmark, in one place. The methodology and the reasoning behind these choices are in About. The code that computes every metric is ehcs_bench/score.py.
Concepts
| Term | Meaning |
|---|---|
| Code-switching | Using words from a second language inside a sentence in the first. Here: English sentences with Hebrew words in them ("pick up some bamba from the makolet"). |
| Hebrew term | A Hebrew word or multi-word phrase inside an English sentence (makolet, Bituach Leumi, teudat zehut). Proper nouns (people, places) count. |
| Term occurrence | One appearance of a term in one clip. Recall is computed over occurrences, so a term that appears 3 times counts 3 times. |
| Canonical spelling | The one romanized (Latin-letter) spelling the reference uses for a term. A convention, not a truth: see lenient. |
| Variant | An alternative romanization accepted by lenient recall (tsaharon for tzaharon). From the Hebrew-Latin-Token-Classifier lexicon. |
| Unseen term | A term absent from the train split of the source dataset Read-Aloud-Voice-Sentences. Measures generalisation for models trained on that split. |
| Seen term | A term that does appear in that train split. |
| Hit | The term is found in the transcript as a whole word, case-insensitive, hyphen and space interchangeable. Never a substring match (chool is not in school). Hebrew script is never a hit. |
| Contamination | Overlap between a model's training data and the test set: identical recordings (contaminated_audio) or new recordings of training sentences (contaminated_text). Those clips are excluded from the headline. |
| Oracle check | Scoring the references themselves as if they were transcripts. Must give 0% WER and 100% on every term metric; anything else is a bug in the test set or scorer. |
| Benchmark round | A dated snapshot of the leaderboard (Archive tab). Rounds are immutable once published. |
Subsets
| Subset | Clips | In headline | Description |
|---|---|---|---|
clean |
84 | Yes | Phone recordings (2026-10) of sentences that appear in no published training set |
contaminated_text |
7 | No | New recordings of sentences Whisper-Hebrish trained on |
contaminated_audio |
26 | No | Recordings that are in Whisper-Hebrish's training set |
desktop_mic |
4 | No | 2025 desktop-microphone recordings |
english_control |
24 | Regression check | Same speaker, English only, no Hebrew terms |
Scoring groups (keys under metrics.subsets in a result file):
| Group | Pools | Use |
|---|---|---|
clean |
clean |
Headline leaderboard |
all_code_switch |
clean + contaminated_text + contaminated_audio + desktop_mic (121 clips) |
Comparable with the Parakeet fine-tune's model card. Unfair to Whisper-Hebrish |
english_control |
english_control |
Does a fine-tune damage plain English? |
Metrics
| Metric | Field in result file | Definition | Better |
|---|---|---|---|
| Term recall | terms.all.recall |
Hits ÷ term occurrences, canonical spelling | Higher |
| Unseen-term recall | terms.unseen.recall |
Term recall over unseen terms only | Higher |
| Seen-term recall | terms.seen.recall |
Term recall over seen terms only | Higher |
| Lenient recall | terms.*.recall_lenient |
A hit on the canonical spelling or any lexicon variant | Higher |
| Exact-case recall | terms.*.recall_exact_case |
A canonical hit whose casing also matches the reference (Bituach Leumi, not bituach leumi) | Higher |
| WER | wer |
Word error rate after normalisation (lowercase, hyphens to spaces, punctuation removed except in-word apostrophes). jiwer 3.1.0 |
Lower |
| CER | cer |
Character error rate, same normalisation | Lower |
| Raw WER | wer_raw |
WER on the cased, punctuated text, no normalisation | Lower |
| English control WER | english_control.wer |
WER on the English-only clips | Lower |
| Occurrences | terms.*.occurrences |
Number of term occurrences in the group (clean: 172) | — |
| Terms | terms.*.terms |
Number of distinct terms (clean: 139) | — |
Test-set fields (metadata.jsonl in the dataset)
| Field | Type | Meaning |
|---|---|---|
id |
string | Clip id: UTC recording time plus a random suffix |
file_name |
string | Path of the 16 kHz mono WAV inside the dataset |
text |
string | Reference transcript, with canonical term spellings |
subset |
string | One of the subsets above |
duration_s |
float | Clip length in seconds |
hebrew_terms |
list | One entry per term occurrence: term (canonical), variants (list), unseen (bool) |
categories |
list | Content tags: code_switch, tech, email, message, names, general, long, homophone |
recording |
string | phone_2026 or desktop_mic_2025 |
source_dataset |
string | Upstream dataset and pinned commit the clip came from |
source_text |
string or null | The upstream transcript when the reference corrected its spelling, else null |
Result-file fields (results/<model>.json)
| Field | Meaning |
|---|---|
schema |
Result format version (currently 1) |
benchmark.version, benchmark.revision |
Test-set version and the exact dataset commit it was scored against. The leaderboard only shows results for the current revision |
model.id, model.revision |
Model repo and the resolved commit that was run |
model.name |
Display name |
model.adapter |
nemo, whisper, or external (transcribed outside the runner) |
model.base_model |
What it was fine-tuned from, if anything |
model.fine_tuned_for_code_switching |
True for English-Hebrew code-switching fine-tunes |
model.params_m |
Parameters, millions |
model.settings |
Decoding settings (language, greedy/beam, chunk length, dtype) |
model.contamination |
Declared overlap between the model's training data and the test set |
scored_at |
UTC time the result was scored |
run.host, run.wall_seconds, run.audio_seconds |
Where it ran and how long it took for how much audio |
metrics.subsets.<group> |
n, minutes, wer, cer, wer_raw, and terms as above |
metrics.per_term.<term> |
Clean subset only: n, strict, lenient, exact hit counts and unseen |
metrics.missing |
Clip ids with no transcript (scored as empty) |
utterances[] |
Every clip: id, subset, ref, hyp. ehcs-bench rescore recomputes all metrics from these |
Archive fields (archive/<date>.json)
| Field | Meaning |
|---|---|
date, title, notes |
When the round was published and what it contained |
benchmark.version, benchmark.revision |
Test set the round was scored on |
repo_commit |
GitHub commit holding the full result files (with transcripts) for the round |
models[] |
Per model: identity, revision, contamination, scored_at, host, and the subsets metrics block |
What is measured
English speakers in Israel drop Hebrew words into English: "grab some bamba from the makolet", "I have to go to Bituach Leumi". English ASR has rarely heard these words and writes the nearest English-sounding thing (Macaulay for makolet). This benchmark measures how often a model gets them right, and whether it still transcribes plain English well.
The references write each Hebrew word in one canonical romanized spelling in Latin letters (makolet, teudat zehut, Yerushalayim). Hebrew script in the output is never a hit.
Test set (v1)
145 clips, frozen at a pinned commit of Hebrish/English-Hebrew-Codeswitching-ASR-Benchmark. They are the held-out test split of Read-Aloud-Voice-Sentences (commit 3bdaa2a50d52). Each clip is tagged with a subset:
| Subset | Clips | Headline? | Why |
|---|---|---|---|
clean |
84 (11.8 min) | Yes | Phone recordings (2026-10) of sentences that appear in no published training set |
contaminated_text |
7 | No | New recordings of sentences that Whisper-Hebrish trained on |
contaminated_audio |
26 | No | Recordings that are in Whisper-Hebrish's training set (the identical files) |
desktop_mic |
4 | No | Clean 2025 desktop-mic recordings, too few to be a subset of their own |
english_control |
24 (3.3 min) | Regression check | The same speaker, English only: email, tech, names, homophones |
How contamination was established (2026-10-15). Whisper-Hebrish was trained on 2025-11-17 on English-Hebrew-Mixed-Sentences at commit e7dc91ef7301, using its whisper_train and whisper_validation files. 26 of the 30 imported desktop clips in the test split have ids in those files. 7 of the phone clips re-read a sentence from those files (normalised text match). Later commits of that dataset (2026-10-14) appended phone takes, so comparing against its main branch gives the wrong answer. The Parakeet fine-tune's test split is term-disjoint by design: no test clip is in its training data.
Term labels. Every clip lists its Hebrew terms with the accepted spelling variants from the Hebrew-Latin-Token-Classifier lexicon and an unseen flag. The flag is set when the term is absent from the source dataset's train split. The clean subset has 172 term occurrences (139 distinct). 117 of the occurrences are unseen. unseen is defined against that one public train split, so it fits models trained on it, such as the Parakeet fine-tune. Whisper-Hebrish was trained on different data: 19 clean occurrences are of terms that appear in its training sentences, and 10 of those are flagged unseen.
Four source transcripts had a variant spelling where their own term label was canonical (heder madregot, machane Yehuda). With them, even a perfect transcript could not score a hit. The benchmark references use the canonical term and keep the original in source_text. An oracle check (references scored as transcripts) gives 0% WER and 100% on every term metric. That check is the guard against this class of bug.
Metrics
All on the clean subset unless stated.
| Metric | Definition |
|---|---|
| Term recall | Share of Hebrew term occurrences whose canonical spelling appears in the transcript as a whole word, case-insensitive, with hyphen and space interchangeable. Never substring: chool is not found in school. |
| Unseen-term recall | The same, restricted to terms absent from the public training split. This is the generalisation number. |
| Lenient recall | Also accepts the lexicon's variant spellings (tsaharon for tzaharon). It is fairer to stock models, which were never taught the canonical spelling. |
| Exact-case recall | A strict hit where the casing also matches the reference (Bituach Leumi). |
| WER / CER | Lowercased, hyphens to spaces, punctuation removed except in-word apostrophes (jiwer 3.1.0). wer_raw (cased, punctuated) is in the result files. |
| English control WER | WER on english_control. A fine-tune that damages English shows up here. |
The normalisation and term matcher are the same ones behind the Parakeet fine-tune's model card. That card's numbers are not directly comparable, though: it scored all 121 code-switching clips (the All code-switching tab) and counted only its own 65 held-out terms as unseen.
Limits
- One speaker. The speaker is Irish-born, English-speaking and lives in Jerusalem. Results rank models on this speaker's code-switching. They do not show performance on other accents.
- Read speech, mostly on one phone microphone.
- Small. 84 headline clips and 172 term occurrences. A difference of 2 to 3 percentage points in term recall is about 4 to 5 hits, which is within noise.
- Not every labelled term is Hebrew (found 2026-10-15, open). 22 of the 139 clean-subset terms are English or product names: tech jargon (JSON, YAML, grep, venv, Proxmox, Ubuntu, sherpa-onnx, VS Code, ...) and brands (Tuya, Astro, Pinecone). They are 26 of the 172 occurrences. They came from the source dataset's term labels, which tag hard vocabulary in general. Excluding them changes no ranks but lowers every model's term recall by 2 to 6 points (Parakeet v2 EN-HE (v2): 65.7% to 63.7%; Whisper Large V3: 45.9% to 39.7%). A relabelled test set will be a new version.
- The canonical spelling is a convention. Lenient recall is there because a model writing tsaharon is not wrong in any absolute sense.
Submitting a result
- Run the model. NeMo and Hugging Face Whisper checkpoints run directly:
ehcs-bench run --model nemo:<repo>or--model whisper:<repo>. It runs on a GPU withmodal run modal_app.py --model .... - Anything else (an API, a phone app, a GGUF runtime): transcribe the 145 WAVs yourself. Write
{"id", "hyp"}JSONL and runehcs-bench score --preds preds.jsonl --name ... --model-id .... - Open a pull request on GitHub that adds the result file to
results/. CI-free check:ehcs-bench rescore results/<file>.jsonmust printOK.
Declare any overlap between your training data and this test set with --contamination. Training on the train split of the source datasets is fine and expected. That is what the unseen metric exists for.