On this page
Long-context inference has made the KV cache one of the main bottlenecks in serving large language models.
This isn’t because KV cache is a bad idea or implementation; KV cache is one of the core reasons why autoregressive decoding is reasonable nowadays. During generation, a transformer stores the keys and values from previous tokens so it can reuse them at later decoding steps instead of recomputing the entire context from scratch.
However, as with most engineering trade-offs, the real problem emerges at scale i.e the KV cache grows with context length. As prompts move from a few thousand tokens to hundreds of thousands, the cache begins to consume a large fraction of GPU memory. Another issue that commonly arises is that it also has to be read repeatedly during decoding. Each new token depends on the previous context, so every step places pressure on memory capacity and memory bandwidth.
For short prompts, it is natural to think mostly about model compute: matrix multiplications, attention, feed-forward layers, and GPU utilization. For long-context serving, memory is no longer a problem that can be ignored. As a result, the system has to store large caches, move them through the memory hierarchy, batch requests around them, and sometimes reuse them across requests that share long prefixes.
KV cache helps with this issue. If the cache is too large, make it smaller. Token-dropping methods keep only part of the cache. Quantization methods store the same cache at lower precision. Both reduce memory footprint and bandwidth pressure, and both can improve throughput.
However, compression changes the computation. The KV cache is not just stored context. It is an input to every future attention operation. Once the cache is compressed, the model is no longer decoding from the same state it would have used with the full cache. The next-token distribution changes. Even if the change is small at each step, generation is autoregressive, so those differences compound over time.
Lossy KV compression can produce outputs that look close to full-cache outputs under soft metrics such as F1, ROUGE, perplexity, or semantic similarity. But many LLM applications are not evaluated by soft similarity. Code generation, tool calling, structured outputs, and agent workflows often require exact behavior. A syntactically invalid diff, an incorrect function argument, or a malformed JSON object is not partially correct in the way a summary might be.
So the tradeoff becomes: full KV inference preserves the model’s original behavior, but costs more memory and bandwidth. Compressed KV inference is faster, but may change the output.
This is the core problem VeriCache is built around. In a recent paper, the authors introduce VeriCache as a system for making KV cache compression safer for correctness-sensitive generation. The key idea is, lossy compression can be useful for speed, but it should not be trusted blindly. VeriCache treats the compressed cache as a fast way to draft possible tokens, while the full KV cache remains the authority that checks whether those tokens match what the original model would have produced.
This turns KV compression into a speculative execution mechanism. The compressed cache is no longer treated as a replacement for the full cache. It becomes a fast approximation used to propose work. The full cache remains the reference computation used to guarantee the same output as full-KV decoding.
That distinction is the most interesting part of VeriCache. The paper is not claiming that lossy compression is harmless. It assumes the opposite. Once the KV cache is compressed, the model may no longer behave exactly as it would under full-cache decoding, and for many real applications, that isn’t good enough.
In this article, I’ll walk through the problem VeriCache is solving, why lossy KV compression fails for correctness-sensitive workloads, how compressed-cache drafting and full-cache verification work, and why the system design matters as much as the decoding algorithm.
The Tradeoff VeriCache Is Trying to Escape
The paper’s tradeoff is not abstract. It comes from two very concrete places where KV cache starts to dominate inference.
The first is within a single request. LLM inference has two phases: prefill and decode. During prefill, the model processes the prompt and builds the KV cache. During decode, the model generates one token at a time using that cache. The important difference is that prefill is relatively parallel, while decode is sequential. Once generation starts, every new token depends on the tokens before it.
That means the KV cache is read again and again. For a short context, this is manageable. For a long context, it becomes expensive in two separate ways. The cache takes up GPU memory, which limits batch size. It also has to be read during decoding, which puts pressure on memory bandwidth.
For Qwen-32B on a single 80GB H100, a 2K-token context needs roughly 0.3GB of KV cache per request. At that size, the system can fit a batch of around 50 requests. At 100K tokens, the KV cache grows to roughly 15GB per request, and the batch size falls to one.
That is the general nature of serving. The same thing happens on the bandwidth side. For Llama-3.1–8B-1M, the paper reports that a decode step at 5K context reads around 0.6GB of KV and takes about 5ms. At 500K context, the KV cache is around 60GB and the decode step takes about 25ms. The model did not suddenly become larger. The generated token did not become more complicated. The system is paying to move more cached state.
This is the first issue: long-context decoding turns the KV cache into a per-token memory cost.
The second issue appears across requests. Many production workloads reuse long prefixes i.e agroup of users may share the same system prompt. A coding assistant may repeatedly work over the same repository. An agent may carry forward a long interaction history. Rather than recomputing the prefix every time, serving systems can precompute the KV cache once and reuse it.
That avoids redundant prefill, but it creates another bottleneck: the cache now has to be loaded onto the serving GPU.
The paper gives a good example of this. Loading Qwen-32B’s precomputed KV cache from S3 at around 3GB/s takes about 0.5 seconds for a 10K-token context. At 100K tokens, the same load takes about 5 seconds. At that point, prefix reuse has not removed the cost. It has moved the cost from computation to transfer.
If the cache is the object consuming memory, reduce its size. If the cache is the object being read every decode step, reduce the number of bytes read. If the cache is the object being transferred for prefix reuse, send a smaller version.
Existing KV compression methods usually fall into two families. Token-dropping methods reduce the number of cached positions. They make a decision about which tokens, layers, or attention heads are less important to keep around during decoding. Some methods preserve attention sinks and recent tokens; others identify heads that need long-range context and compress the rest; others score cached entries during prefill and evict the ones that seem least useful. Quantization methods preserve the shape of the cache, but store each entry with fewer bits. The sequence positions remain, but the representation becomes cheaper to store, read, and transfer.
From a systems perspective, both approaches are very reasonable. If long-context serving is constrained by memory capacity, bandwidth, or transfer time, then reducing the size of the KV cache directly attacks the bottleneck. A smaller cache leaves more room for batching. It reduces the amount of data read during decoding. It also makes prefix reuse cheaper, since less data has to be moved from storage or across the network when multiple requests share the same long context.
The paper does not argue that existing compression methods are ineffective. They often do exactly what they are designed to do: reduce memory footprint and improve throughput. The question is whether the usual way of measuring their quality captures the failures that matter in modern LLM workloads.
Many KV compression methods are evaluated with token-level or similarity-based metrics such as F1, ROUGE, perplexity, or cosine similarity. These metrics are useful when the output has room to vary. In summarization or open-ended question answering, there may be many acceptable continuations. A model can choose a different phrase, reorganize an explanation, or produce a slightly different response while still satisfying the user’s request.
Code generation and tool calling are very harsh. A patch has to be syntactically valid, use the right imports, preserve the surrounding interfaces, and fit into the repository it is modifying. A tool call has to use the correct function name and provide arguments in the format expected by the downstream system. In these settings, similarity to the full-cache output is only a weak proxy for success. The output has to remain executable, parseable, or callable.
This is the motivation behind the VeriCache paper. Under KVzip compression, the authors show that token-level F1 can remain above 75%, while code-format accuracy falls close to zero at 4× compaction. In the tool-calling setting, function-call accuracy drops below 10% under the same compression level. The compressed-cache output can still share enough surface structure with the full-cache output to receive partial credit from a soft metric, while failing the stricter requirement imposed by the application.
This is the tradeoff the paper dives into: compressed KV cache improves throughput because it reduces the amount of memory the system has to keep close to the GPU. At the same time, compression changes the cache, and the cache participates in every future attention computation. For workloads where the model output becomes part of a larger execution path, that change can turn into a failed program edit, an invalid tool invocation, or a malformed structured response.
The target of the paper is beyond just preserving semantic similarity. VeriCache aims to preserve the output of full-KV inference while retaining as much of the speed benefit of compression as possible. That shifts the problem from building a compressor that is accurate on average to building an inference system that can use approximate cached state without allowing that approximation to determine the final answer.
The Core Idea
VeriCache’s core idea is to change the role of KV compression. Instead of using the compressed cache as the final state from which the model generates, VeriCache uses it as a speculative state. The compressed cache is allowed to be approximate because its outputs are not immediately trusted.
At the algorithm level, the system keeps two versions of the cache. The compressed KV cache is used to draft tokens quickly. The full KV cache is used to check whether those drafted tokens match what exact full-cache decoding would have produced. This is the main idea in the paper: compression is no longer replacing exact computation; it is proposing work for exact computation to verify.
The algorithm loops looks something like this:
- First, the model generates a block of candidate tokens using the compressed KV cache.
- If the draft length is (x), the compressed-cache model autoregressively produces (x) tokens. These are the tokens the system hopes to accept without paying the cost of full-cache decoding one step at a time.
- Then VeriCache runs a verification pass using the full KV cache. This pass checks the drafted positions in parallel.
- For each drafted token, the verifier asks what the model would have predicted at that position if it had been using the full cache and the previously accepted draft prefix.
- In one forward pass, the verifier gets predictions for all (x) drafted positions, plus one additional prediction after the last drafted token.
The accept step is where the guarantee comes from. VeriCache compares the draft tokens against the full-cache predictions from left to right. If the first few draft tokens match, they are accepted. If the system finds a mismatch, it accepts the verified prefix up to that point, inserts the full-cache correction at the mismatch, and discards the rest of the draft. If all drafted tokens match, the system accepts the entire draft and also keeps the verifier’s extra bonus token.
Figure 1: A single VeriCache verification round. The compressed KV cache drafts candidate tokens, the full KV cache verifies those positions, and the system accepts tokens until the first mismatch. At the mismatch, the full-cache token is inserted as the correction, and the remaining draft is discarded because it was generated from a prefix exact decoding would not have followed.
The verifier is not just checking whether the compressed cache produced a plausible continuation. It is checking whether the compressed cache produced the same continuation as the full cache. That is what lets VeriCache preserve the output of full-KV decoding under the paper’s greedy-decoding setting.
Let’s walk through an example:
Draft: A B C D E F G H
Verify: A B C D E X ...
Emit: A B C D E X
Discard: G HThe verifier accepts the shared prefix, replaces the first wrong token with the full-KV prediction, and drops the rest because those later draft tokens were generated from the wrong prefix. Now to put this into code terms, VeriCache implementation looks something like this:
def vericache_step(prefix, kv_comp, kv_full, draft_len):
# Draft with compressed KV
draft = []
for _ in range(draft_len):
token = model.next_token(prefix + draft, kv_cache=kv_comp)
draft.append(token)
# Verify all drafted positions with full KV
verified = model.verify(prefix, draft, kv_cache=kv_full)
# verified has draft_len + 1 predictions:
# one for each drafted position, plus one bonus token
# Accept until first mismatch
for i, token in enumerate(draft):
if token != verified[i]:
return draft[:i] + [verified[i]]
# If all draft tokens match, accept draft + bonus token
return draft + [verified[-1]]Written this way, it makes VeriCache look similar to speculative decoding; traditional speculative decoding also uses a cheap path to draft tokens and an expensive path to verify them. The difference is what changes between the drafter and the verifier. In standard speculative decoding, the drafter is often a smaller model. It has different weights, different capacity, and a different next-token distribution. The verifier is the larger target model.
This makes VeriCache look similar to speculative decoding; traditional speculative decoding also uses a cheap path to draft tokens and an expensive path to verify them. The difference is what changes between the drafter and the verifier. In standard speculative decoding, the drafter is often a smaller model. It has different weights, different capacity, and a different next-token distribution. The verifier is the larger target model.
VeriCache keeps the model fixed i.e the drafter and verifier use the same model weights. The core change occurs in the cache, the draft runs the target model with compressed KV, while the verifier runs the target model with full KV. That difference is important because the compressed-cache drafter is usually much closer to the verifier than a separate small model would be.
This is why the VeriCache can draft over longer horizons; a small drafter model may diverge from the target model after only a few tokens, so verification has to happen frequently. VeriCache’s drafter still uses the target model and preserves much of the same attention structure, even though the cache has been compressed. The paper reports that compressed-KV drafting can sustain accepted runs of 25–40 tokens per verification round, compared with only 2–3 tokens for typical small-model drafters.
That longer acceptance length is one of the reasons why this approach is reasonable + practical. The verification is not free, if the system had to verify every couple of tokens, the overhead would quickly erase the benefit of compression. VeriCache depends on the compressed cache being close enough to the full cache for many consecutive tokens, while still being checked often enough to prevent divergence from becoming part of the final output.
Verifying Without Giving Back the Speedup
At this point, VeriCache has solved the correctness problem through the algorithm we discussed in the above section. The compressed cache is allowed to be lossy because it only drafts candidate tokens. The full cache remains the reference computation, so every token that enters the final output is either matched against full-KV decoding or corrected by it.
However, this masks the systems problem the paper is trying to solve. If VeriCache still needs the full KV cache for verification, then the system has to decide where that full cache lives. Keeping it on the GPU all the time is safe, but it weakens the benefit of compression. The compressed cache is useful because it reduces the amount of GPU memory each request occupies. If the full cache remains resident beside it, the system is now paying for both versions of the same context: the compressed one for fast drafting and the full one for correctness. That will ensure correctness of the output, but it gives back much of the memory advantage that made compression useful.
The alternative is to keep the full KV cache outside GPU memory and bring it in only when verification is needed. This preserves the benefit of drafting with a small cache, but the problem now is that long-context KV caches are large. Loading the full cache from CPU memory, storage, or another machine can take long enough to stall generation. If every request drafts for a while and then reaches verification at the same time, the system creates a synchronized bottleneck. The interconnect is busy moving full KV caches. GPU memory has to temporarily hold them. The GPU then has to run the verifier over the drafted positions. In a naive implementation, verification can become expensive enough to erase the speedup compression was supposed to provide.
As a result of this problem, this is where VeriCache is beyond an algorithmic trick, it becomes a systems paper. The draft-and-verify loop makes lossy compression safe, but the serving system has to make verification cheap. The paper’s main systems move is to avoid treating drafting and verification as two global phases. Instead of making all requests draft together and then verify together, VeriCache staggers verification across requests.
The reason this helps is that drafting and verification put pressure on different hardware resources. Drafting from compressed KV is mostly a GPU memory-bandwidth problem. The GPU repeatedly reads the model weights and the compressed cache from HBM while generating tokens. Since decoding is token-by-token, each step has relatively little compute compared with the amount of memory traffic.
Verification is slightly different, it needs access to the full KV cache, which may live in CPU memory or storage until the verifier needs it. That puts pressure on the CPU-GPU interconnect, PCIe, NVLink, or a network link. The verifier also checks multiple drafted positions at once, so it uses compute differently from ordinary one-token decoding. The key point is that compressed-cache drafting and full-cache verification do not bottleneck on exactly the same part of the system.
VeriCache exploits this difference through cross-resource staggering. While one request is verifying against its full KV cache, other requests can continue drafting from their compressed caches. While the full cache for one request is being transferred, the GPU can still make progress on another request whose compressed cache is already resident. Verification does not disappear, but some of its cost can be overlapped with useful work the system needed to do anyway.
Figure 2: Cross-resource staggering in VeriCache. In a lock-step schedule, requests reach verification at the same time, causing full-KV transfer and verification work to pile up. VeriCache spreads verification across time, so one request can verify against full KV while other requests continue drafting from compressed KV.
This scheduling idea also explains why long accepted runs matter. Verification is still not free, even if it is staggered. If the system has to verify every two or three tokens, verification happens too often. If the compressed-cache draft diverges quickly from the full-cache output, most of the drafted work is discarded. In both cases, the verifier dominates the runtime.
VeriCache works because the compressed-cache drafter is usually close enough to the full-cache verifier for many tokens at a time. This is the payoff from using the same model weights on both sides. The drafter is not a tiny auxiliary model trying to imitate a larger target model, rather it is the target model itself, running with an approximate cache. The cache has changed, so the draft is not guaranteed to be correct, but the dominant computation is still similar enough that many tokens can often be accepted before the verifier has to intervene.
Long acceptance runs make the full-cache cost easier to amortize. A single verification round can validate a block of drafted tokens instead of just one next token. If the whole draft matches, the verifier can even contribute an extra bonus token. The more tokens accepted per verification round, the less often the system has to pay the full cost of loading and checking against the full cache.
The paper applies this idea in two serving settings.
- In long-context decoding, the compressed KV cache stays on the GPU for fast drafting, while the full KV cache can live in CPU memory and be reloaded only when verification is needed.
- In remote prefix caching, a system may store precomputed KV caches for long shared prefixes. Sending the compressed cache to the remote serving GPU makes decoding cheaper to start, while the full cache remains available elsewhere for verification.
The details differ, but the principle is the same: keep the approximate state close to the GPU, keep the exact state accessible, and schedule verification so exactness does not become a global stall.
Final Takeaways from VeriCache
Now taking a look at the evaluation benchmarking; the evaluation is testing two claims at the same time. The first is algorithmic: compressed-cache drafting has to stay close enough to full-cache decoding that many tokens can be accepted per verification round. The second is systems-level: the runtime has to hide enough of the full-cache verification cost that compression still improves throughput. If either side fails, the idea becomes much less useful. A fast drafter that diverges too often wastes work. A correct verifier that stalls the system gives back the speedup.
The paper’s results suggest that these two pieces can work together. VeriCache preserves the output of full-KV inference while recovering much of the throughput benefit of compressed KV. In long-context decoding, the compressed cache stays close to the GPU for fast drafting, while the full cache is kept outside GPU memory and brought back only for verification. In remote prefix caching, the compressed cache can be sent to the serving GPU to reduce transfer cost, while the full cache remains available elsewhere for exact checking. Across these settings, the paper reports up to 4× higher throughput than full-KV inference for long-context decoding and up to 2× higher throughput for remote prefix caching, while producing identical outputs.
That result is different from simply saying that compression makes inference faster. Existing compression methods already do that, the cool part is that VeriCache separates speed from authority. The compressed cache is used where approximation is helpful: proposing likely continuations cheaply. The full cache is used where approximation is dangerous: deciding what actually gets emitted. This is why the system can use lossy compression without allowing lossy computation to silently determine the final answer.
There are still important limits to the idea. VeriCache does not eliminate the full KV cache. It still has to exist somewhere, whether in CPU memory, storage, or another part of the serving system. Verification is also not free; the paper’s system design tries to overlap and amortize that cost, not make it disappear. The benefits depend on the compressed cache staying close enough to the full cache for long accepted runs. If a workload or compressor causes frequent mismatches, the system would spend more time verifying and throwing away drafted tokens. The cleanest version of the guarantee is also easiest to explain under greedy decoding; sampling-based generation requires additional machinery.
This is the broader lesson to take from VeriCache. As LLM serving becomes more memory-bound and long-context workloads become more common, systems will increasingly rely on approximate representations, caches, routing decisions, and speculative work. The hard part is not only making those approximations good on average. It is deciding where they are allowed to influence the computation. VeriCache draws that boundary cleanly: compressed KV can propose tokens, but full KV decides which tokens enter the output.