Sliding-Window Prefix Caches Need Physical Proof

A token-identical prefix is not necessarily a valid cache hit. In a hybrid sliding-window attention model, the radix tree may remember the prefix after the local KV slots it points to have been evicted, reused, or remapped.

That turns prefix caching from a string-matching optimization into a memory-consistency problem. If your serving engine does not understand the model's attention layout, a great hit-rate chart can hide recomputation, crashes, or incorrect output.

Full-attention prefix caching has one convenient invariant

During generation, each transformer layer computes keys and values for every token. A KV cache keeps them so the next token does not recompute the earlier sequence. Prefix caching extends that reuse across requests: a prompt beginning with known tokens can reuse their cached tensors and skip most of prefill.

A radix tree is a compact prefix tree whose edges hold token sequences. SGLang's RadixAttention uses one to map shared prompt prefixes to paged KV tensors. For full attention, the rule is pleasantly simple: equal tokens imply equal KV, provided the pages are still resident. Matching walks from the start of the prompt until the first missing block.

Sliding-window attention limits selected layers to the most recent $W$ tokens instead of the full history. Hybrid models interleave these local layers with global layers that retain KV for the whole sequence, breaking the single-lifetime assumption.

MiMo-V2.5-Pro makes the split concrete: 10 full-attention layers, 60 sliding-window layers, and a window of 128 tokens. Xiaomi estimates that layout at roughly one-seventh of the full-attention compute and KV storage for long contexts. The saving only exists if the runtime actually releases old sliding-window KV instead of allocating full-length storage for every layer.

The prefix tree and the KV slots now age differently

A radix node describes a logical token sequence. Sliding-window KV is physical state with a shorter lifetime. The node can remain useful to global layers after the local layers have discarded everything except its tail.

If the matcher checks token equality alone, it can report a pseudo-hit. The scheduler skips prefill, then an attention kernel reads a slot that belongs to another request or no longer exists. This is the nasty sort of cache bug where the metadata looks confident.

A safe matcher keeps the full-attention blocks for the logical prefix, proves that the sliding-window tail is still live, then intersects the two hit lengths. A mapping cached before eviction or load-back is not evidence about the current pool.

Recent SGLang fixes show the failure modes. In PR #25889, a HiCache commit or load-back could rebuild the full-to-window mapping without invalidating a cached translation. The stale indices produced out-of-bounds writes, and an end-to-end regression showed log-probability divergence when stale slots were reused.

In PR #25805, recycled full-KV pages retained mappings from a previous request. Eviction could then double-free sliding-window pages owned by another request. The reported six-node GB300 test crashed after roughly 2,000 requests before the fix and completed 10,240 requests without errors afterward.

These are specific bugs in fast-moving code paths, not evidence that hybrid attention is inherently unreliable. They are evidence that logical prefix identity and physical KV validity must be tested separately.

Correct matching intersects layer-specific cache hits

vLLM's hybrid KV cache design groups layers by attention type and gives each group its own allocation and matching rules. Full attention checks cached blocks from left to right. Sliding-window attention checks whether the last $W - 1$ tokens needed at the proposed resume point are present, scanning backward. The usable prefix is the intersection of those results.

This also explains why a minimal round-robin window buffer is insufficient. It saves memory, but token positions are overwritten in place, so prefix reuse cannot safely refer to their old addresses. vLLM instead allocates distinct blocks and frees blocks that leave the window.

SGLang's newer ShadowRadix design for DeepSeek-V4 takes a related approach. A virtual full-token coordinate projects into separate physical pools, each with its own lifetime. Matching extends only after it finds 128 consecutive live sliding-window tokens. Dropping local slots does not destroy compressed or global KV that remains reusable.

The cost is more metadata, more allocator states, and more invalidation paths. Raw hit rate may also fall because a token match that lacks live local KV becomes a miss. That is the honest result. Capacity improves only when correctness constrains reuse.

Test cache correctness before chasing hit rate

If you serve a hybrid model, verify the runtime path rather than assuming model support includes cache support:

1. Confirm that the engine has attention-type-specific KV pools or cache groups, plus a documented hybrid prefix-matching rule.
2. Compare log probabilities with prefix caching enabled and disabled across long shared prefixes, eviction pressure, cache load-back, and request termination.
3. Add churn tests that combine prefill/decode disaggregation, speculative decoding, and recycled pages. These features create the lifecycle transitions where stale mappings survive.
4. Trace match length per attention group, physical slot validity, mapping generation, evictions, and recomputed tokens. One aggregate hit-rate counter is not enough.
5. If the engine cannot prove the window tail is live, disable prefix reuse for that model or retain full KV. Slower and correct beats fast and haunted.

Sliding-window attention earns its memory reduction by letting old local state die. Prefix caching earns its speedup by reusing live state. A production runtime has to preserve both truths at once.