English-Hebrew Code-Switching ASR Leaderboard
How well does speech recognition handle English with Hebrew words in it: "pick up some bamba from the makolet", "I have to go to Bituach Leumi"? Models are scored on a frozen test set (Hebrish/English-Hebrew-Codeswitching-ASR-Benchmark, test set v2, commit 48ec85a301dc) with the open runner at GitHub.
Term recall is the share of Hebrew words written with the canonical romanized spelling (case-insensitive, whole word). It is the headline: a model can have a low WER and still mangle every Hebrew word. Unseen-term recall counts only Hebrew words absent from the public training split. Single speaker, read speech: see About before drawing conclusions. Contributions welcome: more speakers, better labels, more models.
Benchmark repo · Inference code (Modal GPU runner) · Scorer · Test set · Result files (v2) · Write-up · Fine-tune website: hebrish.app
Test set v2 (current, 2026-10-25). 129 code-switching clips balanced across 13 Hebrew vocabulary domains (about 10 each), none of them trained on by any Hebrish model, plus a 25-clip English control and 25 tech clips. Only Hebrew lexicon entries are labelled. Public and ungated. v1 results are in the Test set v1 tab; the two sets are not comparable. Dataset commit
48ec85a301dc.
1 | Code-switching fine-tune | 65.6 | 56.2 | 67.4 | 57.9 | 13.7 | 1.7 | 2.4 | 4.8 |
Ranked by term recall, then WER. Lenient also accepts lexicon spelling variants (tsaharon for tzaharon), which is fairer to models never taught the canonical spelling. English control is 25 clips of the same speaker with no Hebrew: a fine-tune that wrecks English shows up there. Tech is 25 English tech and AI dictation clips with no Hebrew (WER only).
Clean subset split by the Hebrew vocabulary domain each clip was sampled for (the hebrish.app lexicon domains). About 10 clips and 15 to 20 term occurrences per domain, so a single word moves a domain by 5 to 7 points: read the pattern, not the decimals.
bureaucracy | 10 | 15 | 53.3 / 4.9 | 60.0 / 4.3 | 50.0 / 10.9 | 60.0 / 4.9 | 13.3 / 15.8 |
Clean subset (129 clips, 221 Hebrew term occurrences) unless stated. Hover for values. With 221 occurrences, a gap of 2 to 3 points is about 4 to 7 words: treat it as noise.
Hits / occurrences for every Hebrew term in the clean subset. Hardest terms first.
leshana tova tikatevu | yes | 5 | 0/5 | 0/5 | 0/5 | 0/5 | 0/5 |
Test set v1 (2026-10-15, superseded by v2 on 2026-10-25). Known issue: 22 of the 139 labelled terms are English tech words or product names, not Hebrew (JSON, grep, Proxmox, ...). Ranks are unaffected; absolute term recall is 2 to 6 points high. v2 fixes this. Scores here are not comparable with v2. Dataset commit
3cb0b2b3b3e4.
10 | Code-switching fine-tune | 73.8 | 68.4 | 76.7 | 64.5 | 10.5 | 1.1 | 2.2 | 1,550 | ⚠ Trained on English-Hebrew-Mixed-Sentences (2025-11-17), which overlaps 33 non-headline clips; 10 unseen-flagged clean term occurrences appear in its training text. |
Ranked by term recall, then WER. Lenient also accepts lexicon spelling variants (tsaharon for tzaharon), which is fairer to models never taught the canonical spelling. English control is 24 clips of the same speaker with no Hebrew: a fine-tune that wrecks English shows up there.
Adds 37 clips excluded from the headline: 33 overlap Whisper-Hebrish's training data (26 identical recordings, 7 re-recordings of training sentences) and 4 are older desktop-mic recordings. Comparable with the numbers on the Parakeet fine-tune's model card, which used all 121. Not a fair comparison for Whisper-Hebrish.
10 | Code-switching fine-tune | 68.7 | 62.3 | 71.1 | 60.7 | 11.1 | 1.3 | 2.2 | 1,550 | ⚠ Trained on English-Hebrew-Mixed-Sentences (2025-11-17), which overlaps 33 non-headline clips; 10 unseen-flagged clean term occurrences appear in its training text. |
Clean subset (84 clips, 172 Hebrew term occurrences) unless stated. Hover for values. With 172 occurrences, a gap of 2 to 3 points is about 3 to 5 words: treat it as noise.
Hits / occurrences for every Hebrew term in the clean subset. Hardest terms first.
Google Workspace | yes | 2 | 0/2 | 0/2 | 0/2 | 0/2 | 0/2 | 0/2 | 0/2 | 0/2 | 0/2 | 0/2 | 0/2 |
Every published benchmark round, newest first. A round is frozen when it is published; the leaderboard tabs always show the latest result per model.
2026-10-20: Round 4: Sabich v1.0
Test set v1 (dataset commit 3cb0b2b3b3e4). Full result files with every transcript: 7420d0c. Scored on test set v1, not the current v2: compare it with the v1 tab, not the v2 leaderboard.
| # | Model | Type | Term recall % | Unseen % | Lenient % | WER % | English control WER % |
|---|---|---|---|---|---|---|---|
| 1 | Parakeet v2 EN-HE Sabich v1.0 | CS fine-tune | 73.8 | 68.4 | 76.7 | 3.3 | 2.0 |
| 2 | Parakeet v2 EN-HE code-switching (v1.0) | CS fine-tune | 73.3 | 70.1 | 76.2 | 3.3 | 2.2 |
| 3 | Parakeet v2 EN-HE code-switching (v3) | CS fine-tune | 67.4 | 63.2 | 70.3 | 4.0 | 2.5 |
| 4 | Parakeet v2 EN-HE code-switching (v2) | CS fine-tune | 65.7 | 60.7 | 67.4 | 4.2 | 2.4 |
| 5 | Parakeet v2 EN-HE code-switching (v1) | CS fine-tune | 62.2 | 61.5 | 63.9 | 4.6 | 2.5 |
| 6 | Nemotron Streaming EN-HE code-switching (checkpoint 2) | CS fine-tune | 62.2 | 54.7 | 64.5 | 4.7 | 2.5 |
| 7 | Whisper-Hebrish ⚠ | CS fine-tune | 54.1 | 56.4 | 56.4 | 6.5 | 4.7 |
| 8 | Whisper Large V3 | Stock | 45.9 | 46.2 | 47.7 | 7.5 | 2.4 |
| 9 | Whisper Large V3 Turbo | Stock | 40.1 | 39.3 | 41.9 | 8.9 | 2.4 |
| 10 | Parakeet TDT 0.6B v2 | Stock | 32.6 | 32.5 | 34.3 | 9.5 | 2.7 |
| 11 | Parakeet TDT 0.6B v3 | Stock | 31.4 | 31.6 | 35.5 | 10.5 | 2.7 |
2026-10-19: Round 3: Parakeet v1.0 and the Nemotron streaming fine-tune
Parakeet code-switching fine-tune v1.0 (run r6-en30, 339 min) and the experimental Nemotron streaming fine-tune (r6-nemotron-en30, same data, scored offline). Both on Modal L40S, 2026-10-19. Contamination check: no benchmark clip in either training manifest.
Test set v1 (dataset commit 3cb0b2b3b3e4). Full result files with every transcript: 8d3e79e. Scored on test set v1, not the current v2: compare it with the v1 tab, not the v2 leaderboard.
| # | Model | Type | Term recall % | Unseen % | Lenient % | WER % | English control WER % |
|---|---|---|---|---|---|---|---|
| 1 | Parakeet v2 EN-HE code-switching (v1.0) | CS fine-tune | 73.3 | 70.1 | 76.2 | 3.3 | 2.2 |
| 2 | Parakeet v2 EN-HE code-switching (v3) | CS fine-tune | 67.4 | 63.2 | 70.3 | 4.0 | 2.5 |
| 3 | Parakeet v2 EN-HE code-switching (v2) | CS fine-tune | 65.7 | 60.7 | 67.4 | 4.2 | 2.4 |
| 4 | Parakeet v2 EN-HE code-switching (v1) | CS fine-tune | 62.2 | 61.5 | 63.9 | 4.6 | 2.5 |
| 5 | Nemotron Streaming EN-HE code-switching (checkpoint 2) | CS fine-tune | 62.2 | 54.7 | 64.5 | 4.7 | 2.5 |
| 6 | Whisper-Hebrish ⚠ | CS fine-tune | 54.1 | 56.4 | 56.4 | 6.5 | 4.7 |
| 7 | Whisper Large V3 | Stock | 45.9 | 46.2 | 47.7 | 7.5 | 2.4 |
| 8 | Whisper Large V3 Turbo | Stock | 40.1 | 39.3 | 41.9 | 8.9 | 2.4 |
| 9 | Parakeet TDT 0.6B v2 | Stock | 32.6 | 32.5 | 34.3 | 9.5 | 2.7 |
| 10 | Parakeet TDT 0.6B v3 | Stock | 31.4 | 31.6 | 35.5 | 10.5 | 2.7 |
2026-10-18: Round 2: code-switching fine-tune v3
Adds Parakeet EN-HE code-switching v3 (196 min of code-switching audio). Other models unchanged from round 1.
Test set v1 (dataset commit 3cb0b2b3b3e4). Full result files with every transcript: 3ff5e61. Scored on test set v1, not the current v2: compare it with the v1 tab, not the v2 leaderboard.
| # | Model | Type | Term recall % | Unseen % | Lenient % | WER % | English control WER % |
|---|---|---|---|---|---|---|---|
| 1 | Parakeet v2 EN-HE code-switching (v3) | CS fine-tune | 67.4 | 63.2 | 70.3 | 4.0 | 2.5 |
| 2 | Parakeet v2 EN-HE code-switching (v2) | CS fine-tune | 65.7 | 60.7 | 67.4 | 4.2 | 2.4 |
| 3 | Parakeet v2 EN-HE code-switching (v1) | CS fine-tune | 62.2 | 61.5 | 63.9 | 4.6 | 2.5 |
| 4 | Whisper-Hebrish ⚠ | CS fine-tune | 54.1 | 56.4 | 56.4 | 6.5 | 4.7 |
| 5 | Whisper Large V3 | Stock | 45.9 | 46.2 | 47.7 | 7.5 | 2.4 |
| 6 | Whisper Large V3 Turbo | Stock | 40.1 | 39.3 | 41.9 | 8.9 | 2.4 |
| 7 | Parakeet TDT 0.6B v2 | Stock | 32.6 | 32.5 | 34.3 | 9.5 | 2.7 |
| 8 | Parakeet TDT 0.6B v3 | Stock | 31.4 | 31.6 | 35.5 | 10.5 | 2.7 |
2026-10-15: Round 1: baselines
First published round. Seven models: four stock (Whisper Large V3, Whisper Large V3 Turbo, Parakeet TDT 0.6B v2 and v3) and three code-switching fine-tunes (Parakeet v2 EN-HE v1 and v2, Whisper-Hebrish). All run on Modal L40S, greedy decoding.
Test set v1 (dataset commit 3cb0b2b3b3e4). Full result files with every transcript: aa09470. Scored on test set v1, not the current v2: compare it with the v1 tab, not the v2 leaderboard.
| # | Model | Type | Term recall % | Unseen % | Lenient % | WER % | English control WER % |
|---|---|---|---|---|---|---|---|
| 1 | Parakeet v2 EN-HE code-switching (v2) | CS fine-tune | 65.7 | 60.7 | 67.4 | 4.2 | 2.4 |
| 2 | Parakeet v2 EN-HE code-switching (v1) | CS fine-tune | 62.2 | 61.5 | 63.9 | 4.6 | 2.5 |
| 3 | Whisper-Hebrish ⚠ | CS fine-tune | 54.1 | 56.4 | 56.4 | 6.5 | 4.7 |
| 4 | Whisper Large V3 | Stock | 45.9 | 46.2 | 47.7 | 7.5 | 2.4 |
| 5 | Whisper Large V3 Turbo | Stock | 40.1 | 39.3 | 41.9 | 8.9 | 2.4 |
| 6 | Parakeet TDT 0.6B v2 | Stock | 32.6 | 32.5 | 34.3 | 9.5 | 2.7 |
| 7 | Parakeet TDT 0.6B v3 | Stock | 31.4 | 31.6 | 35.5 | 10.5 | 2.7 |
Data dictionary
Every term, field and metric used by the benchmark, in one place. The methodology and the reasoning behind these choices are in About. The code that computes every metric is ehcs_bench/score.py.
Concepts
| Term | Meaning |
|---|---|
| Code-switching | Using words from a second language inside a sentence in the first. Here: English sentences with Hebrew words in them ("pick up some bamba from the makolet"). |
| Test-set version | v1 (2026-10-15, data/test/, dataset commit 3cb0b2b3b3e4) or v2 (2026-10-25, data/v2/, commit 48ec85a301dc, current). Versions never change in place; a result is only comparable with results on the same version and commit. |
| Hebrew term | A Hebrew word or multi-word phrase inside an English sentence (makolet, Bituach Leumi, teudat zehut). Proper nouns (people, places) count. In v2, only entries of the hebrish.app lexicon are labelled; v1 labels also included some English tech and product words. |
| Domain | v2 only. One of the 13 vocabulary domains of the hebrish.app lexicon: places, people, politics, religious, food, health, bureaucracy, money, home, army, transport, brands, everyday. Each term has one; each clean clip gets one by the rule in About (majority of its terms, names only as a last resort). |
| Term occurrence | One appearance of a term in one clip. Recall is computed over occurrences, so a term that appears 3 times counts 3 times. |
| Canonical spelling | The one romanized (Latin-letter) spelling the reference uses for a term. A convention, not a truth: see lenient. |
| Variant | An alternative romanization accepted by lenient recall (tsaharon for tzaharon). From the Hebrew-Latin-Token-Classifier lexicon. |
| Unseen term | A term absent from the train split of the source dataset Read-Aloud-Voice-Sentences. Measures generalisation for models trained on that split. |
| Seen term | A term that does appear in that train split. |
| Hit | The term is found in the transcript as a whole word, case-insensitive, hyphen and space interchangeable. Never a substring match (chool is not in school). Hebrew script is never a hit. |
| Contamination | Overlap between a model's training data and the test set: identical recordings or new recordings of training sentences. v1 tagged the clips that overlap Whisper-Hebrish (contaminated_audio, contaminated_text) and kept them out of the headline. v2 leaves out every clip whose id or normalised text is in any Hebrish training or validation manifest, and the Whisper-Hebrish overlaps as well. |
| Oracle check | Scoring the references themselves as if they were transcripts. Must give 0% WER and 100% on every term metric; anything else is a bug in the test set or scorer. |
| Benchmark round | A dated snapshot of the leaderboard (Archive tab). Rounds are immutable once published. |
Subsets, v2 (current)
| Subset | Clips | In headline | Description |
|---|---|---|---|
clean |
129 | Yes | Code-switching phone recordings, about 10 per domain, never trained on by any Hebrish model |
english_control |
25 | Regression check | Same speaker, everyday English, no Hebrew |
tech |
25 | No | Same speaker, English tech and AI dictation, no Hebrew. WER only |
Scoring groups, v2: clean, english_control, tech, each pooling the subset of the same name. There is no all_code_switch. The domains block breaks clean down by domain (see result-file fields).
Subsets, v1 (history)
| Subset | Clips | In headline | Description |
|---|---|---|---|
clean |
84 | Yes | Phone recordings (2026-10) of sentences that appear in no published training set |
contaminated_text |
7 | No | New recordings of sentences Whisper-Hebrish trained on |
contaminated_audio |
26 | No | Recordings that are in Whisper-Hebrish's training set |
desktop_mic |
4 | No | 2025 desktop-microphone recordings |
english_control |
24 | Regression check | Same speaker, English only, no Hebrew terms |
Scoring groups, v1 (keys under metrics.subsets in a result file):
| Group | Pools | Use |
|---|---|---|
clean |
clean |
Headline leaderboard |
all_code_switch |
clean + contaminated_text + contaminated_audio + desktop_mic (121 clips) |
Comparable with the Parakeet fine-tune's model card. Unfair to Whisper-Hebrish |
english_control |
english_control |
Does a fine-tune damage plain English? |
Metrics
| Metric | Field in result file | Definition | Better |
|---|---|---|---|
| Term recall | terms.all.recall |
Hits ÷ term occurrences, canonical spelling | Higher |
| Unseen-term recall | terms.unseen.recall |
Term recall over unseen terms only | Higher |
| Seen-term recall | terms.seen.recall |
Term recall over seen terms only | Higher |
| Lenient recall | terms.*.recall_lenient |
A hit on the canonical spelling or any lexicon variant | Higher |
| Exact-case recall | terms.*.recall_exact_case |
A canonical hit whose casing also matches the reference (Bituach Leumi, not bituach leumi) | Higher |
| WER | wer |
Word error rate after normalisation (lowercase, hyphens to spaces, punctuation removed except in-word apostrophes). jiwer 3.1.0 |
Lower |
| CER | cer |
Character error rate, same normalisation | Lower |
| Raw WER | wer_raw |
WER on the cased, punctuated text, no normalisation | Lower |
| English control WER | english_control.wer |
WER on the English-only clips | Lower |
| Tech WER | tech.wer |
v2: WER on the English tech clips | Lower |
| Per-domain recall, WER | domains.<domain>.terms.recall, domains.<domain>.wer |
v2: term recall (and its lenient and exact-case forms) and WER over the clean clips of one domain. About 10 clips each | Higher / lower |
| Occurrences | terms.*.occurrences |
Number of term occurrences in the group (clean: 221 in v2, 172 in v1) | — |
| Terms | terms.*.terms |
Number of distinct terms (clean: 187 in v2, 139 in v1) | — |
Test-set fields (metadata.jsonl: data/v2/ for v2, data/test/ for v1)
| Field | Type | Meaning |
|---|---|---|
id |
string | Clip id: UTC recording time plus a random suffix |
file_name |
string | Path of the 16 kHz mono 16-bit WAV, relative to the version folder (audio/<id>.wav) |
text |
string | Reference transcript, with canonical term spellings |
subset |
string | One of the subsets above |
domain |
string or null | v2 only: the clip's domain for clean clips, null for the English subsets |
duration_s |
float | Clip length in seconds (v2: of the 16 kHz file) |
hebrew_terms |
list | One entry per labelled term: term (canonical), variants (list), unseen (bool), and in v2 domain. Empty for English-only clips |
categories |
list | Content tags from the source dataset: code_switch, names, religious, yiddish, tech, ai_interactions, linux, smart_home, email, message, general, homophone, target_word, medical, long, ... |
recording |
string | phone_2026 or desktop_mic_2025 |
source_dataset |
string | Upstream dataset and pinned commit the clip came from |
source_text |
string or null | The upstream transcript when the reference corrected its spelling, else null |
source_audio |
string | v2 only: the source file's format before conversion (wav 16000 Hz PCM_16, flac 48000 Hz PCM_24) |
Result-file fields (results/<model>.json)
| Field | Meaning |
|---|---|
schema |
Result format version (currently 1) |
benchmark.version, benchmark.revision |
Test-set version and the exact dataset commit it was scored against. The leaderboard has one tab per test-set version (the current one is the default) and each tab shows only results scored on that version's revision. v1 results live in results/<name>.json, v2 results in results/v2/<name>.json |
model.id, model.revision |
Model repo and the resolved commit that was run |
model.name |
Display name |
model.adapter |
nemo, whisper, or external (transcribed outside the runner) |
model.base_model |
What it was fine-tuned from, if anything |
model.fine_tuned_for_code_switching |
True for English-Hebrew code-switching fine-tunes |
model.params_m |
Parameters, millions |
model.settings |
Decoding settings (language, greedy/beam, chunk length, dtype) |
model.contamination |
Declared overlap between the model's training data and the test set |
scored_at |
UTC time the result was scored |
run.host, run.wall_seconds, run.audio_seconds |
Where it ran and how long it took for how much audio |
metrics.subsets.<group> |
n, minutes, wer, cer, wer_raw, and terms as above |
metrics.per_term.<term> |
Clean subset only: n, strict, lenient, exact hit counts and unseen |
metrics.domains.<domain> |
v2 only: n clips, wer, and terms (occurrences, terms, recall, recall_lenient, recall_exact_case) over the clean clips of that domain |
metrics.missing |
Clip ids with no transcript (scored as empty) |
utterances[] |
Every clip: id, subset, ref, hyp. ehcs-bench rescore recomputes all metrics from these |
Archive fields (archive/<date>.json)
| Field | Meaning |
|---|---|
date, title, notes |
When the round was published and what it contained |
benchmark.version, benchmark.revision |
Test set the round was scored on |
repo_commit |
GitHub commit holding the full result files (with transcripts) for the round |
models[] |
Per model: identity, revision, contamination, scored_at, host, and the subsets metrics block |
What is measured
English speakers in Israel drop Hebrew words into English: "grab some bamba from the makolet", "I have to go to Bituach Leumi". English ASR has rarely heard these words and writes the nearest English-sounding thing (Macaulay for makolet). This benchmark measures how often a model gets them right, and whether it still transcribes plain English well.
The references write each Hebrew word in one canonical romanized spelling in Latin letters (makolet, teudat zehut, Yerushalayim). Hebrew script in the output is never a hit.
Test set v2 (current, 2026-10-25)
v2 is the set Hebrish uses for its own internal benchmarks, and the one new results are scored on. It is in data/v2/ of Hebrish/English-Hebrew-Codeswitching-ASR-Benchmark (config v2), pinned at dataset commit 48ec85a301dc, and built by scripts/build_testset_v2.py. Its source is the held-out test split of Read-Aloud-Voice-Sentences at commit 2cec2ebc85e9 (the revision Sabich v2.1 trained on). All 179 clips are phone recordings from 2026-10-14 to 2026-10-25.
| Subset | Clips | Headline? | What it is |
|---|---|---|---|
clean |
129 (14.8 min) | Yes | Code-switching sentences, about 10 per Hebrew vocabulary domain, never trained on by any Hebrish model |
english_control |
25 (2.0 min) | Regression check | The same speaker, everyday English with no Hebrew: email, general sentences, homophones |
tech |
25 (2.3 min) | No | English tech and AI dictation with no Hebrew. WER only |
Why a v2. v1 drew its headline from whatever the test split held on 2026-10-15, so a few kinds of vocabulary dominated it. Its term labels also included English tech and product words (see Limits). v2 samples the same number of clips from each domain of the hebrish.app lexicon, so the headline weighs places, food, bureaucracy and the army equally. It labels only Hebrew lexicon entries. It also excludes every clip that any Hebrish model trained on.
Clean clips per domain:
| Domain | Clips | Term occurrences in these clips | of which unseen | Eligible clips |
|---|---|---|---|---|
places |
10 | 19 | 12 | 44 |
people |
10 | 12 | 5 | 11 |
politics |
9 | 15 | 12 | 9 |
religious |
10 | 14 | 11 | 111 |
food |
10 | 18 | 10 | 39 |
health |
10 | 19 | 12 | 39 |
bureaucracy |
10 | 21 | 12 | 23 |
money |
10 | 16 | 9 | 23 |
home |
10 | 18 | 12 | 37 |
army |
10 | 15 | 11 | 20 |
transport |
10 | 22 | 10 | 15 |
brands |
10 | 15 | 11 | 19 |
everyday |
10 | 17 | 10 | 43 |
| Total | 129 | 221 (187 distinct) | 137 |
Politics had 9 eligible clips, one short of the target of 10. The other domains had enough.
How it was built. These are the steps in scripts/build_testset_v2.py. Its counts are in data/v2/build-report.json in the dataset.
- Eligible takes. Of the 838 test-split rows, a take is eligible if it is the owner's own (
speakerDaniel Rosehill or missing), has a transcript, and is not a 2025 desktop-mic import (30 dropped). It must not be a re-reading of a sentence Whisper-Hebrish trained on (16 dropped). It must not be a special-delivery task take: spelling alphabet, singing, whispering, odd voices, repetitions, disfluency, numbers, long-form (49 dropped). Only one take per sentence is kept (13 duplicates dropped). - Never trained on (checked 2026-10-25). A take is dropped if its id or its normalised text (lowercase, hyphens to spaces, punctuation removed) is in
runs/<run>/train.jsonlorval.jsonlon the Modal volumeparakeet-cs-finetune, for any of the 22 runs there. Those runs includer10-all(Sabich v2.1),r9-all(Sabich v2.0),r7-all(Sabich v1.0) andr6-en30(Jachnun v1.0). 13 test-split takes were dropped. All 13 matched by text (the same sentence recorded again, in train), none by id. Each of r10-all, r9-all and r7-all contained all 13; r6-en30 contained 10. The manifests are private, so the per-run matches are committed inscripts/inputs/v2-trained-exclusions.json, and the builder rebuilds the same set from that file. - Headline candidates. These are code-switching takes (
code_switchcategory) with no tech category, because code-switching and tech are kept in separate sentences (20 dropped). Each needs at least one labelled Hebrew term (59 dropped). - Term labels.
hebrew_termsholds the terms from the take's source label (target_terms) that are entries in hebrish.app's lexicon. The lexicon is frozen inscripts/inputs/hebrish-lexicon-2026-10-20.json(3,642 terms in 13 domains, built 2026-10-20). Each term is written in the lexicon's canonical spelling, matched by word boundary, and carries its domain. English tech and product words are not in the lexicon, so v1's JSON, grep, Proxmox and Tuya problem cannot recur. Across the headline candidates, 49 distinct source labels (61 occurrences) are not lexicon entries and are left unlabelled. Most are Hebrew words the lexicon lacks (Maccabi Zahav in 5 takes, Makor Baruch, moshav, yeshiva). Some are spellings the lexicon has in another form (abba vs aba, Modiin, Rishon LeTsion). A few are not Hebrew (Daniel Rosehill, 8200). The full list istarget_terms_not_labelledin the build report. These words still count in WER. Lexicon terms the source label missed are not added, because several lexicon brands are also English words (Yes, Max, Partner) and a text scan would mislabel them. Spelling variants come from Hebrew-Latin-Token-Classifier at commit0459b004a775.unseenmeans the term is absent from the source train split'starget_termsat2cec2ebc85e9. - One domain per clip. A clip's domain is the most common domain among its terms.
peopleterms count only when the clip has nothing else, because a name is usually the addressee ("Hi Gal, ..."), not the subject. Ties go to the domain with fewer eligible clips (to help the scarce ones), then alphabetically. 63 of the 129 clips have terms from more than one domain. Domains are assigned per term by the lexicon, not by reading the sentence, so a few look odd: Shas (the Talmud, in context) and machshev are under brands, and cheder madregot is under army. These are the lexicon's labels, kept as they are. - Sampling. In each domain, candidates are ordered by
sha256("ehcs-v2:" + id). A first pass takes clips that add a term not yet chosen in that domain, and the rest of the order fills up to 10.english_controlis the first 25 takes in the same order whose categories are onlygeneral,email,message,medical,target_wordorhomophone, and which carry no Hebrew category or lexicon term.techis the first 25 takes with a tech category and no Hebrew category. Rerunning the builder gave identical files. - References and audio. Two source transcripts spelled heder madregot where the label is the canonical cheder madregot. The reference uses the canonical term and keeps the original in
source_text. Audio is 16 kHz mono 16-bit WAV. The 66 clips recorded as 48 kHz / 24-bit FLAC (from 2026-10-22) were downmixed and resampled withscipy.signal.resample_poly, as ASR-Training'scs_lib.to_16k_wavdoes for training. - Oracle check. Scoring the references as transcripts gives 0% WER, CER and raw WER on all three subsets. It gives 100% strict, lenient and exact-case term recall on all, unseen and seen terms, and in every domain. It was run on the copy downloaded from the Hub at
48ec85a301dc.
v2 overlaps v1 in 24 clean clips and 5 English clips. The rest are new.
Scoring. A v2 result carries metrics.subsets.clean, english_control and tech. There is no all_code_switch in v2, because no contaminated clips are kept. A new metrics.domains.<domain> block gives n, wer and term recall per domain over clean. Each per-domain figure rests on about 10 clips and 12 to 22 term occurrences. Read it as a pointer to where a model is weak, not as a measurement.
Test set v1 (2026-10-15, history)
v1 is kept unchanged in data/test/ and pinned at dataset commit 3cb0b2b3b3e4; every result scored on it stays valid and can be rescored. It was the leaderboard set until 2026-10-25. What follows is its original description.
145 clips, frozen at a pinned commit of Hebrish/English-Hebrew-Codeswitching-ASR-Benchmark. They are the held-out test split of Read-Aloud-Voice-Sentences (commit 3bdaa2a50d52). Each clip is tagged with a subset:
| Subset | Clips | Headline? | Why |
|---|---|---|---|
clean |
84 (11.8 min) | Yes | Phone recordings (2026-10) of sentences that appear in no published training set |
contaminated_text |
7 | No | New recordings of sentences that Whisper-Hebrish trained on |
contaminated_audio |
26 | No | Recordings that are in Whisper-Hebrish's training set (the identical files) |
desktop_mic |
4 | No | Clean 2025 desktop-mic recordings, too few to be a subset of their own |
english_control |
24 (3.3 min) | Regression check | The same speaker, English only: email, tech, names, homophones |
How contamination was established (2026-10-15). Whisper-Hebrish was trained on 2025-11-17 on English-Hebrew-Mixed-Sentences at commit e7dc91ef7301, using its whisper_train and whisper_validation files. 26 of the 30 imported desktop clips in the test split have ids in those files. 7 of the phone clips re-read a sentence from those files (normalised text match). Later commits of that dataset (2026-10-14) appended phone takes, so comparing against its main branch gives the wrong answer. The Parakeet fine-tune's test split is term-disjoint by design: no test clip is in its training data.
Term labels. Every clip lists its Hebrew terms with the accepted spelling variants from the Hebrew-Latin-Token-Classifier lexicon and an unseen flag. The flag is set when the term is absent from the source dataset's train split. The clean subset has 172 term occurrences (139 distinct). 117 of the occurrences are unseen. unseen is defined against that one public train split, so it fits models trained on it, such as the Parakeet fine-tune. Whisper-Hebrish was trained on different data: 19 clean occurrences are of terms that appear in its training sentences, and 10 of those are flagged unseen.
Four source transcripts had a variant spelling where their own term label was canonical (heder madregot, machane Yehuda). With them, even a perfect transcript could not score a hit. The benchmark references use the canonical term and keep the original in source_text. An oracle check (references scored as transcripts) gives 0% WER and 100% on every term metric. That check is the guard against this class of bug.
Metrics
All on the clean subset unless stated. They are the same in v1 and v2.
| Metric | Definition |
|---|---|
| Term recall | Share of Hebrew term occurrences whose canonical spelling appears in the transcript as a whole word, case-insensitive, with hyphen and space interchangeable. Never substring: chool is not found in school. |
| Unseen-term recall | The same, restricted to terms absent from the public training split. This is the generalisation number. |
| Lenient recall | Also accepts the lexicon's variant spellings (tsaharon for tzaharon). It is fairer to stock models, which were never taught the canonical spelling. |
| Exact-case recall | A strict hit where the casing also matches the reference (Bituach Leumi). |
| WER / CER | Lowercased, hyphens to spaces, punctuation removed except in-word apostrophes (jiwer 3.1.0). wer_raw (cased, punctuated) is in the result files. |
| English control WER | WER on english_control. A fine-tune that damages English shows up here. |
The normalisation and term matcher are the same ones behind the Parakeet fine-tune's model card. That card's numbers are not directly comparable, though: it scored all 121 code-switching clips (the All code-switching tab) and counted only its own 65 held-out terms as unseen.
Limits
- One speaker. The speaker is Irish-born, English-speaking and lives in Jerusalem. Results rank models on this speaker's code-switching. They do not show performance on other accents.
- Read speech, mostly on one phone microphone.
- Small. v2 has 129 headline clips and 221 term occurrences; v1 had 84 and 172. A difference of 2 to 3 percentage points in term recall is about 5 hits, which is within noise. Per-domain numbers (v2) rest on about 10 clips each.
- v1 only: not every labelled term is Hebrew (found 2026-10-15, fixed in v2). 22 of the 139 clean-subset terms are English or product names: tech jargon (JSON, YAML, grep, venv, Proxmox, Ubuntu, sherpa-onnx, VS Code, ...) and brands (Tuya, Astro, Pinecone). They are 26 of the 172 occurrences. They came from the source dataset's term labels, which tag hard vocabulary in general. Excluding them changes no ranks but lowers every model's term recall by 2 to 6 points (Parakeet v2 EN-HE (v2): 65.7% to 63.7%; Whisper Large V3: 45.9% to 39.7%). v2 labels only hebrish.app lexicon entries, so it does not have this problem.
- The canonical spelling is a convention. Lenient recall is there because a model writing tsaharon is not wrong in any absolute sense.
Submitting a result
- Run the model. NeMo and Hugging Face Whisper checkpoints run directly:
ehcs-bench run --model nemo:<repo>or--model whisper:<repo>. It runs on a GPU withmodal run modal_app.py --model .... - Anything else (an API, a phone app, a GGUF runtime): transcribe every WAV of the pinned test set yourself (v2: the 179 files in
data/v2/audio/). Write{"id", "hyp"}JSONL and runehcs-bench score --preds preds.jsonl --name ... --model-id ....--version v1scores against the old set instead; the default is the current set, v2. A v2 result file is written toresults/v2/<name>.json(v1 results stay inresults/). - Open a pull request on GitHub that adds the result file to
results/. CI-free check:ehcs-bench rescore results/<file>.jsonmust printOK.
Declare any overlap between your training data and this test set with --contamination. Training on the train split of the source datasets is fine and expected. That is what the unseen metric exists for.