# ERA V5 · Session 2 — one 10,000-token BPE tokenizer for en / hi / te / mai

Write-up and live widget: https://vdanepalli.com/courses/erav5/session-2/assignments

## Reproduce (Python 3.11+)

    pip install -r requirements.txt
    # corpus: the course's reference snapshot (../corpus/*.faithful.txt), the files its evaluator reads
    python train_tokenizer.py --out tokenizer.base.json --weights en=10,hi=12,te=19,mai=19
    python equalize.py tokenizer.base.json tokenizer.eq.json     # equalise the four fertilities
    python build_heldout.py                                       # held-out articles (for fill_vocab + analysis)
    python fill_vocab.py tokenizer.eq.json tokenizer.json         # back to exactly 10,000 tokens
    python evaluate_tokenizer.py tokenizer.json --json metrics.json

`tokenizer.json` rebuilds byte for byte (sha256 1af7df9245c1c5b4…). Counting is
`len(tokenizer.encode(whole_file).ids)`, exactly as the course's evaluate_tokenizer.py.

## Files
- `eval_common.py`        faithful-unit definition (same regex as the course evaluator)
- `train_tokenizer.py`    HF BPE · Metaspace (▁) · no normaliser · byte fallback · weighted documents
- `search_weights.py`     coordinate search over per-language training weights
- `equalize.py`           vocabulary surgery in both directions (remove / add single-language merges)
- `fill_vocab.py`         add held-out merges that provably never fire on the four files → 10,000
- `evaluate_tokenizer.py` counts, fertilities, spread, score, Hindi penalty, faithfulness
- `build_heldout.py`, `analysis.py`   held-out articles and ../findings.json
- `tokenizer.base.json`, `tokenizer.eq.json`   the tokenizer after training and after equalising