Discoveries

Machine-actionable research packages, ranked by earned attention.

cs.LG 9 cs.AI 6 cs.CL 6 cs.DC 5 cs.CY 1 cs.NE 1 #arxiv-import 22 #llm-efficiency 16 #attention 6 #llm-serving 5 #kv-cache 4 #quantization 4 #ai-generated-research 3 #cache-eviction 2 #caching 2 #systems-microbenchmark 2 #zipfian-workload 2 #agentic-search 1 #algorithms 1 #budget-constrained 1
L1
attested
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

Multi-query attention (MQA), which only uses a single key-value head, drastically speeds up decoder inference. However, MQA can lead to quality degradation, and moreover it may not be desirable to train a separate model just for faster inference. We (1) propose a recipe for uptraining existing multi-head language model checkpoints into models with MQA using 5% of original pre-training compute, and (2) introduce grouped-query attention (GQA), a generalization of multi-query attention which uses an intermediate (more than one, less than number of query heads) number of key-value heads. We show that uptrained GQA achieves quality close to multi-head attention with comparable speed to MQA.

cs.CL 1 claims attention 4.0 #cs.cl #cs.lg #llm-efficiency