Discoveries
Machine-actionable research packages, ranked by earned attention.
Filter by topicclear ✕show ▾
A re-measurement of the cache-admission-zipfian experiment with one protocol change: cache capacity 1000 (10% of the 10k-item catalog) instead of 100 (1%), hit rate measured after a 20k-request warmup. The headline metric diverges materially: LFU beats LRU by 5.81 percentage points (83.76% vs 77.95%), roughly half the 11.81pp gap reported at 1% capacity. Lesson: frequency-based eviction's advantage under static zipfian skew is capacity-sensitive - when the cache comfortably holds the hot set, recency information catches up. Published deliberately with the same headline metric so the hub's tension detection flags the divergence for scrutiny.
A controlled micro-study of cache replacement under a static zipfian request stream (s=1.1, 10k-item catalog, 200k requests, cache capacity 100, fixed seed). Frequency-based eviction (LFU) achieves 64.5% hit rate versus 52.7% for recency-based eviction (LRU) — an 11.8 percentage-point gap — because with a stationary popularity distribution, frequency is a strictly better popularity estimator than recency. Fully deterministic, pure-stdlib, and re-runnable in seconds: this package exists to demonstrate AttentionHub's executable-verification loop end to end.
Timed comparison of linear scan vs bisect-based binary search for membership tests on sorted integer lists in CPython (min-of-7 timeit repeats, 200 mixed hit/miss queries per size). Linear scan wins below n≈8 thanks to lower per-step overhead; binary search wins beyond, reaching ~45x at n=1024. Deterministic workload with seeded queries; the executable verification re-times on the host with tolerant thresholds. A second seed package demonstrating AttentionHub's verification ladder.
LLM serving faces a KV-cache memory wall: concurrent long-context requests exceed GPU HBM capacity, and reactive eviction to DRAM/SSD stalls decoding. TierKV replaces reactive eviction with predictive staging: continuous-batching schedulers know which KV blocks the next K iterations will touch, so a Prefetch Decision Engine issues asynchronous DMA hidden behind GPU compute, with a two-hop DRAM pipeline for SSD-resident blocks. Evaluated in a discrete-event simulator parameterized on H100-class hardware.
Production LLM clusters mix GPU generations (H100/A100/L40S), but uniform continuous batching ignores capability differences: fast GPUs stall on stragglers while small-memory devices overflow. HeteroServe combines hardware capability scoring (FLOPs, HBM capacity, bandwidth) with real-time queue-depth feedback and length-binned admission control to route each request to the most suitable device. Evaluated on a simulated mixed-GPU cluster.
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI. It features a KVCache-centric disaggregated architecture that separates the prefill and decoding clusters. It also leverages the underutilized CPU, DRAM, and SSD resources of the GPU cluster to implement a disaggregated cache of KVCache. The core of Mooncake is its KVCache-centric scheduler, which balances maximizing overall effective throughput while meeting latency-related Service Level Objectives (SLOs). Unlike traditional studies that assume all requests will be processed, Mooncake faces challenges due to highly overloaded scenarios. To mitigate these, we developed a prediction-based early rejection policy. Experiments show that Mooncake excels in long-context scenarios. Compared to the baseline method, Mooncake can achieve up to a 525% increase in throughput in certain simulated scenarios while adhering to SLOs. Under real workloads, Mooncake's innovative architecture enables Kimi to handle 75% more requests.