News · AI Reliability & Operations · AI for SaaS & Software Companies · Enterprise AI Governance
KVGuard checks KV cache memory safety in LLM serving
An arXiv paper describes silent KV cache memory bugs in vLLM and SGLang and a checker that the authors say catches them with low latency overhead.
Large language model serving systems can carry a class of memory safety bug in their key-value (KV) cache that silently corrupts outputs or breaks isolation between requests without crashing anything, according to a paper by Jonguk Cheong, Myungbin Hwang, Eulgyu Kim and Byoungyoung Lee first posted to arXiv on November 4, 2026. The authors present KVGuard, which they describe as the first systematic memory-safety abstraction for KV cache management, a checker for this class of bug, and report that a prototype on vLLM and SGLang covered all 29 historical KV cache safety bugs from those projects and found 8 previously unknown ones. The results are the authors' own and have not been independently replicated.
What the paper reports
Modern serving systems manage the KV cache through PagedAttention, which the abstract describes as a custom GPU allocator. It divides one large GPU memory region into fixed-size physical blocks and maps each request to its blocks through a block table built on the CPU.
The authors argue that this block-based design exposes a previously unstudied class of bugs: releasing the same block twice, using a block after it has been released, out-of-bounds access, reading uninitialized entries, and permission violations. Their point is that these bugs do not crash the system. They corrupt outputs or break request isolation quietly.
Existing memory sanitizers cannot detect them, the authors say, because they reason about raw GPU addresses rather than blocks, requests and block tables, and their guard-region and quarantine techniques do not fit the scarce GPU memory of LLM serving.
KVGuard keeps a compact shadow block for each physical KV block, recording which requests may use it and which entries have been initialized. It uses that metadata for five checks across the PagedAttention workflow: CPU allocation, block table construction and translation, GPU-side block table lookup, and KV entry access. Each check fires at the boundary where a violation is introduced, so a bug surfaces during development rather than as corrupted output many requests later.
The authors report these results for their prototype:
- It covers all 29 historical real-world KV cache safety bugs they collected from the vLLM and SGLang patch histories.
- It found 8 previously unknown safety violations, with no false positives reported.
- It adds 3.0 percent (vLLM) and 7.4 percent (SGLang) geometric-mean overhead in time per output token (TPOT), measured on Qwen 2.5 14B.
- Its memory overhead is at most 0.11 percent of the KV cache, which the authors describe as two orders of magnitude below NVIDIA Compute Sanitizer on the same workloads.
The abstract reports latency overhead for one model on two serving systems. It does not say whether KVGuard is meant to run in production or only during development and testing, though it frames early detection during development as the goal.
Why it matters
Most AI safety discussion looks at prompts and outputs. This paper looks one layer down. Where many users or agents share a serving system, keeping each request's cache state with that request is a reliability property as much as a security one, and a fault there looks like an answer that is wrong for reasons nobody upstream can see.
The 29 historical bugs and 8 new findings turn cache isolation from a theoretical concern into a testing gap that projects and operators can act on. This brief describes the class of issue and the proposed checker only, not how any bug might be triggered.
What it means for businesses
Add serving-layer isolation to the questions asked of any AI host. Ask which serving software and version runs the workload, how request isolation in the cache is tested, and how a silent isolation fault would be detected and reported. For sensitive workloads, ask whether dedicated serving is available and what it means for latency. The Enterprise AI Governance hub covers the adjacent application-side questions.
What it means for AI users
On a shared hosted service, users depend on the host to keep each session's state separate, without being able to see it. If an answer seems to draw on context you never provided, or is garbled in a way that does not fit your request, report it to the provider. The paper's point is that such faults need not cause any visible crash.
What it means for developers
Teams that run their own inference on vLLM or SGLang should follow upstream fixes in the KV cache path and consider block-level checks in development and CI builds, where an overhead in the range the authors report is easier to accept. Add cache isolation to the service's reliability objectives and monitor for outputs inconsistent with a request's own inputs. The AI Reliability & Operations hub covers making such checks routine, and the KVGuard paper is the reference for the checker's design.
Sources
- KVGuard: A KV Cache Safety Checker for LLM Serving Systems (arXiv) (accessed 2026-11-06)