RSQR (Raw Survivor Query Rotation) - A new perspective efficient approach to StreamingLLM

How this started

So my day started as any other where i was sitting in class (as usual), and i had recently started being invested in vLLM (high throughput ml inference engine) repository and was looking through issues and open PR’s seeing if there was any cool contribution i could make. I had just finished my latest project - which was a segment tree based sharding distributed engine (it’s pretty cool if you want to go check it out on my github).

Before this my prior experience to vllm had been through my last blog - which was regarding a reasoning engine, but vllm popped up a lot in my research during my 2 weeks with it.

There was one particular PR which caught my eye -

[Core] Experimental session KV eviction with attention sinks- #43374
...
After evicting positions [s, e), surviving tokens shift from position
p to p - (e - s). Their cached K still carries the old RoPE
rotation. Without re-rotation, attention against post-eviction queries is
numerically wrong on every RoPE model (LLaMA / Mistral / Qwen / Gemma /
DeepSeek / Phi-3 / …).
...
This PR ships the engine plumbing + extension hook. A downstream runner...

Wait, you guys are deleting KV cache out here? Isn’t that kinda dangerous?

I mean I was no stranger to introducing deliberate roughness to models since AI models are auto regressive in nature, they generally have a good amount of leniency while also landing on the (usually) correct answer each time.

My laptop is severely kind of limited by constraints, I have a GTX 1050 Ti graphics card which still has the Pascal architecture which limits what i can test from the vllm repository. So seeing this - i was sort of surprised that i could actually probably contribute to this PR, given i understood enough of what was being talked about.

What’s KV Cache? (for newbies)

KV cache is essentially the stored key and value vectors from all the layers of the model

What does that mean?

So essentially in simpler terms what KV cache is, is the memory of an LLM. How an LLM remembers the conversation. Usually what happens is that when LLM starts generating text token by token (synonymous to word by word), it does this specific calculation.

\[A = \text{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)\]

Still with me? Every new token that is generated is multiplied by all the Keys -> which is the K vector in our formula to get the Attention matrix.

So the thing about the KV Cache is that it increases linearly per every token consumed. And for each latest token to be generated it has to be multiplied by all the keys previously generated, so that it can properly consume and understand the context for it’s next word that it’s about to speak.

Now obviously, i’m majorly simplifying but that’s the jist of what YOU need to know as the reader to understand.

Why should i care?

You see, KV Cache, especially at large contexts, becomes so big sometimes that generating the next token becomes an increasingly difficult task. This is a major challenge for the biggest of the companies out there. When they handle big models, like Gemini, Fable, etc. long running contexts (lot of memory behind each context) is what generates massive data centers to fix the issue.

Got it?

Now people smarter than me have obviously tried to tackle this issue. We’ve come relatively far in learning how to compress KV Cache, make less of it as well as use less of it. So i’m not going to claim that i’ve done a revolutionary thing, i’m pretty new to the field myself.

So what does the PR say?

The PR is essentially saying this in a nutshell -

Hey, I want to add an experimental feature that evicts parts of the KV Cache and retains the important parts so that we can save memory and the model can understand the context

“Okay… that’s a good idea. But won’t deleting random parts of it’s memory cause it to become faulty?”

Well you see, in massive industrial use, ML inference engines are concerned with saving on cost as much as possible, because running millions of calculations for every single token takes a lot of compute. So risking a model’s context learning by a little bit, to save memory overhead from millions of people using it worldwide - that’s a pretty damn good deal. Save costs - save water.

How it does that

Before we get into what it tries to do, we’ll discuss a little thing called RoPE.

Usually, in training, LLM models need a way to know which part of the context is related which other context and by how much. Let’s take some context for example -

K1 - “My dog hasn’t been eating for a while. K2 - “I think that it is stressed out.” K3 - “I personally like eating.” K4 - “There’s a good food establishment called Burger King.”

Ignore the creativity behind that sentence - i’m a programmer not exactly a writer.

In this context, it’s blatantly clear to us, the audience which sentence relates to which. LLM’s actually have no concept of spacing - unless it’s provided to them using a clear technique.

Essentially, an LLM cannot differentiate the relation of how these contexts are close to each other, they don’t see the sequence. To them if you multiply all of these raw keys together it looks like K4 came at exactly the same time as K1 - which is awful because mentioning eating at Burger King right as you talk about your stressed out starving dog really isn’t a funny premise.

And also it’s blatantly obvious that K1 and K2 are closer in contextual meaning to each other in terms of raw distance, and K3 K4 are the same.

So to fix this we introduce something in training called Rotatory Position Embedding.

Essentially we rotate the keys by a little amount say K1 - we rotate by 1 degree, K2 we rotate by 2 degrees, K3 by 3 and K4 by 4 degrees.

Now, assumption being that the LLM Model has learnt the value of time when it was in training via RoPE - can now successfully differentiate time passing by in terms of angles. Like Doctor Who.

The difference between K1 and K4 is 3 degrees which means they lie further from each other being mentioned than K1 and K2 which are only 1 degree apart, which are much closely mentioned than K1 and K4.

Still with me?

The Problem

Evicting random KV cache blocks does something these angles. The model had only trained on angles that were neatly packed and close to each other till now. Now we’re introducing gaps into it’s KV Cache. A KV cache that used to look like this:

K1 K2 K3 K4 K5 K6 K7

Is now starting to look like this

K1 .. K3 .. .. K6 K7

And these gaps, are actually not too bad for the model. See the problem isnt the fact that the model isn’t able to cope with gaps in the KV Cache, it’s what comes after that it cant handle.

Suppose you keep running your model forever. You keep evicting KV cache blocks smartly and only keep the ones that are actually contextually important perfectly. Your blocks look like this after roughly 10,000 tokens -

K1 .. K2 .. K120 .. K500 .. K800 .. K5000 .. K10000

When the context used to be shorter it was extremely easy for the model to identify the gaps and somehow work around them because the overall distance between the blocks was shorter aswell. But introducing HUGE gaps did a little thing called RoPE dampening.

This effect can be thought of like this

• 1 degree, 2 degree 3 degree gaps don’t bother a model, because it gives ENOUGH magnitude for a vector to project itself onto another vector and make a noticeable statistic for the model to notice while decoding.

• 300 degrees gap between 2 blocks means that the model can literally not associate these 2 vectors together, not because it’s incapable, the value projected between the 2 is so low that it tricks the model into thinking there’s zero association.

• This “zero association” means that blocks at huge gaps get isolated and guess what, the model has never seen isolated blocks in it’s training, ever.

• This causes a valid crashout from the model who now complains that “hey i only did what i was supposed to do with what i was given”

gtiwa’s Solution

Ok i got it, what’s the solution here?

Solution - rerotate the blocks. How do you rerotate blocks? by applying the rotation matrix. See, RoPE isn’t actually a new novel thing, it’s use case is - it’s just a very well known tool wrapped up in a way that suits LLM training.

Hypothetically if we renumber all the blocks

K1 .. K2 .. K120 .. K500 .. K800 .. K5000 .. K10000

to

K1 K2 K3 K4 K5 K6 K7

The model should gain back some of it’s ability to function. This isn’t a new technique - Many papers have discussed the effects of RoPE dampening at long contexts and tried to improve upon it.

A new contender!?

So this PR was written by gtiwa - who’s currently a Meta engineer from what i can see from the PR. I understood pretty well what gtiwa wanted to achieve in his PR.

But then a new person came by on that same exact thread - and their name was nvbfalk, he’s currently in NVIDIA. Now this person had quite an interesting resolution to this exact same problem.

nvbfalk’s Solution in his RFC

So how do you tackle this problem?

We keep the gaps.

Haha yeah, no seriously… what?

We keep the gaps.

What.

So it turns out that nvbfalk had done his own research

Apparently the laziest route you could take to evicting cache was LITERALLY JUST DELETING IT without a single concern for the model. The LLM model can actually, believe it or not, sustain long amounts of streaming and context, if you don’t do a single thing to repair the KV cache after deleting parts of it.

This is insane. What the fuck?

Now this doesn’t actually remove from the fact that eventually, the context will get long enough for RoPE dampening to happen. That’s bound to happen regardless

So nvbfalk’s approach right after that was to do a full “consolidation”. which meant running a forward pass (which is expensive af) on all the surviving kv cache blocks before it.

Why it worked

I had a major suspicion that nvbfalk’s approach worked primarily because he was working on m-Rope, for long streaming video sessions over 24 hour long durations. This meant that a LARGE amount of what his “setup” actually processed was a still image from a CCTV capture somewhere. So it was actually a pretty neat idea, he says that he’d been doing his approach for a while now and just wanted to draft it up as a change to VLLM.

Issues with both the approaches

So far we have 2 problems.

• nvbfalk’s approach was dicey - it relied on the fact that after a huge session, a consolidation would immediately reset the order of things - but this introduced a major lagspike and a major amount of compute to do a “reset”

• gtiwa’s approach was launching a custom kernel for doing reset on command. You could technically choose to evict cache whenever you wanted by running a function, and it would do that for you and rerotate all the blocks. This was flagged as an experimental feature for vllm because you wouldn’t know WHEN to rerotate vectors.

• Also the main part of the entire thing was rooted in knowing which one approach gave higher accuracy and recall and maintained latency.

Both of these were talking on the same wavelength - just different approaches. That’s why nvbfalk considered going to gtiwa for help in the first place.

My first approach

Okay so I had an architecture in place already that i thought would decrease latency by precomputing those swaps for rerotation - called Amortized Precompute Mechanism. I’ll discuss what it was later. For now we’ll just be testing an idea which was the fundamental basis for it.

The heaviest claim that architecture would need is that rerotation was better than leave-gaps method, other-wise precomputing makes zero sense.

So the test began for doing automated kv cache eviction -

The general approach i had revolved around evicting every delta-th token in the kv cache. This means we’re automatically picking which tokens to evict on the fly, then measuring their accuracy on recall ability on a lot of trials.

I changed up things quite a lot - instance of video streaming I went in a “longform text context” direction and tried to build the exact same features they both talked about.

https://github.com/null-Exception1/auto-kv-cache-eviction

The writeup has a lot of judgement and is heavily detailed but what you really need to focus on is the results:

My basic methodology for fact recall was to keep multiple facts in a context which WOULD be protected throughout the eviction of other kv cache blocks process. It’s not the most accurate thing to a real world scenario but it’s something.

n_cycles	mean evictions	A: full replay	B: renumber + re-rotate	B: leave gap, no re-rotation
4	4.0	84.0%	82.7%	90.0%
8	12.0	90.0%	89.3%	95.3%
12	20.0	87.3%	88.7%	96.0%
16	28.0	90.7%	91.3%	95.3%

so far, seems like leave-gaps is winning

So, Leave-Gaps method seems to have a pretty lead over our Renumber + Re-rotate strategy. And that’s not because it’s an inherently better strategy. It’s because of another reason, i found out later.

My entire job with this was to actually find out whether gtiwa’s renumbering strategy would give the same if not better accuracy than the leave-gaps model. That’s not really a way to live - because it let me down. I sulked for quite a bit.

Failure

After taking a moment to myself (crying), I kind of felt over the last week that what i was doing was inherently flawed for its following reasons -

• Firstly, I’m doing this experiment on long form text context - text is inherently more info dense than video streaming sessions, so applying these kv cache eviction strategies wouldn’t fit properly.
• Secondly, the only thing i’m measuring is the effect of rerotation on output - since the facts themselves are protected and we’re only removing fillers - this doesn’t replicate a real world scenario at all. Streaming sessions in comparison, have wayyyy more fillers, so the probability of accidently removing a fact is lower
• Thirdly, i realised that leave-gaps and rerotation was supposed to have ROUGHLY the same accuracy because gtiwa’s method was built to combat the problem of LONG context sessions, and the current state of our context was around 200 tokens only per cycle.

Infact the entire point of this experiment was to test whether THIS core architecture was possible -

https://github.com/null-Exception1/auto-kv-cache-eviction/blob/main/amortized_precompute_mechanism.md

What was Amortized Precompute Mechanism?

I’m sorry for making you wait but here you go -

I came up with this mechanism which in short does this -

• At a micro scale in each block, every delta-th token is kept as a survivor token and it’s cache has a duplicate copy of precomputed cache of what would happen after the swap.

• After exiting that block, the swap happens and all the precomputed survivors after the swap slot right into their respective positions as we had already precomputed for those positions.

• Rinse and repeat when you exit the next block.

• The size of block and delta are specified ahead of time, its not a dynamic value.

What were the benefits of this?

• First of all you’ve got the obvious O(1) precompute time save
• Second you’re rerotating the rotated vectors - StreamingLLM had to rotate all the unrotated cache at runtime

In short, the results of my auto-kv-cache-eviction basically invalidated the basis for amortized-precompute-mechanism

Okay so look

This is actually a pretty good mechanism on paper. On paper it doesn’t refuse to do what it says. It makes use of concepts we know already and there’s very little scope for error.

But something wasn’t sitting right with me after that auto-kv-eviction test we did in the previous section. How could it be that rerotated cache (which was the fundamentally correct idea) get beaten by it’s predecessor the leave-gaps model.

Epiphanies

Something i realised at night after tiring day of classes - I was scrolling through my old Colab notebook

There was a cell to check whether RoPE was working correctly or not so i ran this extremely small test cell -

Cell

test_content = torch.randn(1, N_KV_HEADS, 1, HEAD_DIM, device=device)
P, evict_n = 16, 8


rotated_then_corrected = apply_rope(test_content, torch.tensor([float(P)], device=device), FREQS)

rotated_then_corrected = apply_rope(rotated_then_corrected, torch.tensor([float(-evict_n)], device=device), FREQS)

fresh_at_target_position = apply_rope(test_content, torch.tensor([float(P - evict_n*1)], device=device), FREQS)

print("max abs diff:", (rotated_then_corrected - fresh_at_target_position).abs().max().item())

Output

max abs diff: 2.384185791015625e-07

And I said hmm that’s not that unusual, 10 ^ -7 is generally golden standard for margin of error, as i thought to myself - stupidly.

Then later at night it bugged me - “there IS a small margin of error but it shouldn’t mess with my core idea - it shouldn’t be prone to failure within 10^-7 margin of error”.

But the mechanism behind B_corrected and B_uncorrected was exactly the same with the difference of RoPE added to B_corrected for correction.

So i ran a little diagnosis test.

import random

print("rerotate 1 time")
for i in range(10):
  test_content = torch.randn(1, N_KV_HEADS, 1, HEAD_DIM, device=device)
  P, evict_n = random.randint(800,1000),random.randint(500,600)

  rotated_then_corrected = apply_rope(test_content, torch.tensor([float(P)], device=device), FREQS)

  rotated_then_corrected = apply_rope(rotated_then_corrected, torch.tensor([float(-evict_n)], device=device), FREQS)

  fresh_at_target_position = apply_rope(test_content, torch.tensor([float(P - evict_n*1)], device=device), FREQS)

  print("max abs diff:", (rotated_then_corrected - fresh_at_target_position).abs().max().item())

print("rerotate 2 time")
for i in range(10):
  test_content = torch.randn(1, N_KV_HEADS, 1, HEAD_DIM, device=device)
  P, evict_n = random.randint(800,1000),random.randint(500,600)

  rotated_then_corrected = apply_rope(test_content, torch.tensor([float(P)], device=device), FREQS)

  rotated_then_corrected = apply_rope(rotated_then_corrected, torch.tensor([float(-evict_n)], device=device), FREQS)

  rotated_then_corrected = apply_rope(rotated_then_corrected, torch.tensor([float(-evict_n)], device=device), FREQS)


  fresh_at_target_position = apply_rope(test_content, torch.tensor([float(P - evict_n*2)], device=device), FREQS)

  print("max abs diff:", (rotated_then_corrected - fresh_at_target_position).abs().max().item())

rerotate 1 time
max abs diff: 2.2977590560913086e-05
max abs diff: 1.8835067749023438e-05
max abs diff: 3.361701965332031e-05
max abs diff: 2.205371856689453e-05
max abs diff: 0.00011551380157470703
max abs diff: 1.990795135498047e-05
max abs diff: 1.1563301086425781e-05
max abs diff: 2.473592758178711e-05
max abs diff: 1.0728836059570312e-05
max abs diff: 5.036592483520508e-06
rerotate 2 time
max abs diff: 3.680586814880371e-05
max abs diff: 2.6941299438476562e-05
max abs diff: 6.365776062011719e-05
max abs diff: 5.179643630981445e-05
max abs diff: 5.793571472167969e-05
max abs diff: 3.001093864440918e-05
max abs diff: 1.1995434761047363e-05
max abs diff: 6.301701068878174e-05
max abs diff: 3.4809112548828125e-05
max abs diff: 2.5987625122070312e-05

Well shit. The margin of error I just got by messing around with numbers and more varied data points shows that at higher values the margin of error of RoPE degrades by 100x.

And so i put this into Google

“Hey is 10^-5 still an okay margin of error for RoPE”

…

“Nah buddy you’ve got KV crashing on your hands go get it checked”

Are you kidding me?

More research by dissecting

Okay so we’re miles ahead and still miles behind -

• After the block wise cache eviction strategy i actually decided that since blockwise eviction would be inconsistent for accuracy and be depended on block size it would be unclean architectural - purely an intuitive choice

• So I pivoted to making a “latest window” type design where the survivor tokens are stored as soon as they exit the window as into the “historical” bucket

• StreamingLLM has roughly the same design choice - but what they do is incredibly unique - they keep the unrotated caches in memory and ONLY rotate them at runtime.

• Then i realised that RoPE actually dodges error if you only rotate an unrotated block it gives MASSIVELY less drift as compared to doing it twice or thrice. This must be why StreamingLLM is precise and still works.

• This idea started to set inside my head - I had done the literature preview PRIOR to making the design - but it had never set in for me that what they were doing accidently dodged the main issue with RoPE error margins i just got myself into.

• Not only that but now i could confirm why exactly the gap between B_correct and B_uncorrected was a consistent gap not a compounding gap over n_cycles increasing.

• In the actual mechanism we were only doing one extra rotate - whose margin of error was just enough to make the LLM believe that some positions were overlapping each other - and then fail just enough to not raise suspicion either.

• The entire thing was starting click now.

The missing piece

I realised that my design was missing one core piece - the idea to scrape out survivor tokens and put them behind the “recent” window wasn’t wrong - it was JUST lacking one piece. And believe it or not i was inclining to this SPECIFIC architecture because my raw intuition told me that if i kept the survivors consecutive with the window’s rotation, i could try to rotate the baseline so that the overall rotation renumbered back from 0 as everything was relative on the scale.

i’m rambling but my insight at the time was kind of giving “i’m either an idiot or a genius”

So I did another literature review - for the last time

MiniPIC is used for something called Flexible Position-Independent Caching

In the MiniPIC architecture, rotating the query vector acts as the “relocation anchor” that aligns new text inputs with reused, unrotated text chunks stored in the cache.Instead of forcing you to recompute or re-rotate the stored text blocks to fit a new prompt structure, MiniPIC shifts all the alignment work onto the active generation phase.

Bingo.

See my main idea was actually along this line -

I deeply apologize for my bad handwriting, I was in a hurry to get to class, your honour

• See essentially, when a survivor token is in the window (which we know ahead of time when because of our fixed offset delta approach), we compute it’s rotated block with respect to global position as well as keep it’s raw block in memory
• All the non survivor tokens in the window are rotated as per usual according to their global position don’t worry about them
• As soon as we’re out of the window - we shunt out the rotated block, and keep the unrotated cache with us of the survivor
• We put this in the survivor bucket, it has it’s own global position though - just because it’s unrotated doesn’t mean it doesn’t hold it’s position
• Now everytime ON runtime we rotate ONLY the survivor tokens consecutively till they hit the rear end of the window

• This means our survivor tokens AND the window are now consecutively numbered, and we only had to rotate the survivor tokens - because all of the blocks in our window are rotated once already and anything that falls off it is either a survivor or evicted

• Here’s the special crabby patty sauce ingredient - we use MiniPIC to ROTATE the query to right behind the survivor tokens - since all of them are distances relative to each other in the grand scheme of rotation - where our survivor token starts from doesn’t matter, it becomes a 0th block

       HISTORICAL SURVIVOR BUCKET             ACTIVE STREAMING WINDOW
  ┌──────────────┐┌──────────────┐┌──────────────┐ ┌───────────────────────────┐
  │    T_16/5    ││    T_17/10   ││    T_18/15   │ │ T_19  . . . . . . T_latest│
  └──────┬───────┘└──────┬───────┘└──────┬───────┘ └─────────────┬─────────────┘
         │               │               │                       │
         └───────────────┼───────────────┘                       │
                         ▼                                       ▼
             Eviction Strategy Window                   Keys remain unchanged!
            (Only rerotate at runtime)                (Zero runtime rotation cost)

By altering the Query rotation at runtime, we fake out the model into thinking it is looking at a perfectly chronological, contiguous timeline:

\[\text{Runtime Update: } \quad (R_{m + 16} Q_m) \cdot (R_n K_n)^T\]

The sparse history gets re-indexed instantly from base zero at runtime without touching the Key vectors in memory:

┌──────┐┌──────┐┌──────┐┌──────┐
│ T_1/5││ T_2/6││ T_3/7││ T_4  │ . . . T_latest
└──────┘└──────┘└──────┘└──────┘

Here is the absolute coolest side-effect of this architecture: **We just decoupled physical sequence length from the attention timeline.**

Because we aren't dragging keys through consecutive rotation passes, we can use a **variable and fragmented timeline**. You can stitch together completely separate pieces of context—like modular system prompts, few-shot examples, and historical RAG vectors—recorded across wildly different times. 

By adjusting the query shift relative to *any* arbitrary set of survivors, the LLM treats them as a perfectly smooth, continuous timeline. It’s basically virtual memory paging, but for LLM context.

Advantages

• No longer we care about RoPE precision errors - RoPE is applied to each unrotated block only ONCE
• StreamingLLM’s design choice was based on rotating ALL of the tokens on runtime - this took way more FLOPS than needed and this design will sincerely destroy it purely on the basis of less FLOPS it needs
• At the same time we’re not bounded by questions like “is rerotation good?” StreamingLLM’s research was hell bent on doing rerotation because according to their OWN testing they found out how leave-gaps method is bad - the architecture we improved upon was purely from the algorithmic point of view - and exploiting the well known properties of rotation matrices and RoPE - none of the core research behind StreamingLLM’s original thesis is invalid.

link to the draft

An improvement - the Precompute-ahead variant

So basically this as an addition to the previous mechanism - we’re talking about relieving the kernel launches from the mechanism as they cost a ton. My genuine thought process was that that if the kernel launch doesn’t have enough work to do to satisfy the cost of the launch itself - then we give it a reason to.

• In this case - the computations we were doing for survivor tokens becomes “ahead-of-time” precomputed, we calculate multiple rotations of the same unrotated blocks for future runtimes - This basically loads more work in the beginning and consumes more memory
• By the saturation point - when the precomputations are no longer beneficial, and the number of survivor tokens in the survivor bucket has increased to the point where doing a kernel launch purely for computing those survivor tokens is good enough we switch back to purely computing and not precomputing anything else
• The memory overhead is NOT extra and is purely early in the beginning
• This also improves latency in the beginning when kernel launches might cause unnecessary overhead - not too much just unnecessary.

The only current massive question trailing me is the Tail-latency issue - is it bumpy because of micro-batching? and by how much. Although we optimized the architecture to consume decreased FLOPS - it’s the tail latency which is important when serving models like StreamingLLM so - yeah i need to work on that.

Mathematical Proof

Open Questions

• Batched (eviction-boundary) rotation latency vs. StreamingLLM’s continuous 31→65ms/token curve. Does a per-eviction-event rotation batch, amortized per token between events, come in above or below that continuous-cost curve? This is the load-bearing unmeasured claim in this proposal.
• Bump severity as a function of window size and survivor density. Does a smaller window (more frequent eviction boundaries) degrade tail latency enough to erode the aggregate-compute win? Needs a sweep over window size and Δ (survivor selection interval), not a single configuration.
• Drift-scaling sanity check. Extend the 10-iteration test in §2 to ~100 iterations to confirm the 6.02e-6 figure is linear and not accelerating. Partially addressed by §2.1: a follow-up sweep varying P and evict_n magnitude (rather than iteration count) shows a separate, magnitude-driven error floor that dominates at larger P/evict_n, plus a compounding term on top of it that becomes visible at that same larger magnitude. Still open: isolating P from evict_n from target-position magnitude, and the ~100-iteration extension at fixed magnitude originally proposed here.
• Raw-shadow-copy memory accounting. Quantify the memory cost of holding a raw copy for flagged survivors while still in-window, as a function of Δ and window size. For the precompute-ahead variant (§3.4), this extends to the cost of holding a chain of precomputed correction states per flagged block, front-loaded earlier in the session than the base mechanism requires — bounded to the same memory the session would need eventually regardless (§3.4), but not yet quantified as a concrete number.
• Precompute-ahead saturation point (§3.4). At what survivor-batch size does additional ahead-of-time precompute stop yielding kernel-launch benefit, and how much earlier does the precompute-ahead variant reach that saturation point versus the passive (grow-naturally) base mechanism? This is the concrete, falsifiable version of “whether precompute is worth reintroducing” — worth measuring directly rather than assuming, and only meaningful once (1) establishes whether there’s a real tail-latency problem to solve in the first place.
• Attention-score sensitivity to the §2.1 magnitude floor. Does a K-tensor diff in the 1e-5–1e-4 range (as measured at P∈[800,1000]) move softmax attention scores or downstream recall accuracy at all, or is it invisible past that point? This is the load-bearing question raised by §2.1 and is a prerequisite for treating the magnitude floor as anything more than a tensor-diff curiosity.
• Isolate P magnitude from evict_n magnitude in the §2.1 sweep. The follow-up sweep varied both together; a cleaner sweep holding one fixed while varying the other would attribute the error floor to absolute position, correction size, or target position specifically — relevant for whether frequent-small-δ eviction is meaningfully safer than infrequent-large-δ eviction.

References

• Xiao, G., Tian, Y., Chen, B., Han, S., & Lewis, M. (2023). Efficient Streaming Language Models with Attention Sinks. arXiv:2309.17453.
• Anonymous et al. (2025). MEPIC: Memory Efficient Position Independent Caching for LLM Serving. arXiv:2512.16822.

Conclusion

Alright thanks for reading this blog guys. The core architecture’s theoretical proof is intact so i’m going to start working on the testing and build a good implementation enough to make a working proof of concept for this.

This has consumed maybe the last 2 weeks of my life so, probably good to know someone read this.

See ya