L1
Integrity-checked
claims carry evidence but haven't been re-run (attested)
TierKV: Prefetch-Aware Memory Tiering for KV Cache in LLM Serving
LLM serving faces a KV-cache memory wall: concurrent long-context requests exceed GPU HBM capacity, and reactive eviction to DRAM/SSD stalls decoding. TierKV replaces reactive eviction with predictive staging: continuous-batching schedulers know which KV blocks the next K iterations will touch, so a Prefetch Decision Engine issues asynchronous DMA hidden behind GPU compute, with a two-hop DRAM pipeline for SSD-resident blocks. Evaluated in a discrete-event simulator parameterized on H100-class hardware.
Claims
≈ attested
c1
performance
In discrete-event simulation at 3x HBM oversubscription, scheduler-lookahead prefetching (K=4) achieves a 100% prefetch hit rate.
system TierKV
workload 3x-hbm-oversubscription-sim
metric prefetch-hit-rate
value 100.0
unit %
higher_is_better True
baseline reactive-LRU
hardware H100-class (simulated)
≈ attested
c2
comparison
Simulated mean time-per-output-token improves 3.6x over reactive LRU eviction at 3x oversubscription.
system TierKV
workload 3x-hbm-oversubscription-sim
metric tpot-speedup
value 3.6
unit x
higher_is_better True
baseline reactive-LRU
hardware H100-class (simulated)
≈ attested
c3
comparison
Simulated system throughput improves 2.9x over reactive LRU at 3x oversubscription; larger models benefit proportionally more due to wider per-iteration overlap budget.
system TierKV
workload 3x-hbm-oversubscription-sim
metric throughput-speedup
value 2.9
unit x
higher_is_better True
baseline reactive-LRU
hardware H100-class (simulated)
Artifacts 1 files · code, data, logs — integrity-checked
| role | location | size | integrity |
|---|---|---|---|
| paper | paper.pdf | 904630 | ✓ 07e7f6bae845 |