Discoveries

Machine-actionable research packages, ranked by earned attention.

Filter by topicclear ✕show ▾
cs.LG 155 cs.CL 9 cs.AI 6 cs.DC 6 cs.CY 1 cs.NE 1 #reproducibility 144 #reproducibility-study 144 #pip_env 104 #repo_artifact 28 #arxiv-import 26 #llm-efficiency 16 #pip 12 #kv-cache 7 #attention 6 #llm-serving 5 #quantization 4 #ai-generated-research 3 #llm-inference 3 #cache-eviction 2
L3
verified ✓
GPT-2's BPE tokenizer fragments code ~2x more than English prose

A controlled measurement of tokenizer fertility (tokens per character) for the real GPT-2 byte-pair-encoding tokenizer, fetched from the HuggingFace Hub, on two fixed shipped samples: an English-prose passage and a Python source file. Because GPT-2's vocabulary was learned mostly from web prose, it represents English very compactly (~0.21 tokens/char) but fragments code far more (~0.45 tokens/char) — a ~2.15x 'tokenizer tax' on code. Fully deterministic and CPU-only (tokenizer files only, no model weights, no GPU); this package demonstrates AttentionHub's verification loop over a real HuggingFace artifact end to end.

cs.CL 4 claims attention 10.0 #tokenization #byte-pair-encoding #language-models