the structure, code and framing are theirs. this page re-tells it from zero, filling in the basics (what a decoder is, what a cache is) the original assumes.
22,580 GPT-2s fit inside one Kimi K3.
GPT-2 (2019) has 124 million parameters. Kimi K3 (2026) has 2.8 trillion. That is a 22,580× jump in seven years. The lazy story is "we made it bigger." The real story is a chain of specific fixes: each new architecture changed what the model remembers, how it overwrites old memories, and how it reads them back. This page walks the whole chain from scratch, no background assumed, every line of code explained.
scroll · ~40 min read
A language model plays one game: guess the next token.
Before any architecture, fix the job description. A language model receives a piece of text and outputs a probability for every possible next chunk. Feed it "the cat sat on the" and it should assign high probability to "mat". That is the entire game, played billions of times during training.
Text first gets split into
tokens.
Each token is an integer id into a vocabulary. GPT-2's vocabulary has about 50,000 entries. The model never sees letters; it sees a list of integers like [464, 3797, 3332].
Each integer is then swapped for an embedding, a list of numbers the model can do math on. In GPT-2 each token becomes 768 floats. From this point until the very last step, the model only pushes vectors around: it never touches text again.
At the very end, the model converts its final vector back into a score for every vocabulary entry. Those raw scores are called logits, and a softmax turns them into probabilities. Pick one token, append it to the input, run the whole thing again. Generation is just this loop.
GPT-2 is a decoder-only transformer. Here is what that phrase unpacks to.
The original 2017 transformer was built for translation and had two halves. An encoder read the whole source sentence at once; every word could look at every other word, forwards and backwards. A decoder wrote the translation one word at a time, and it was only allowed to look at words it had already written, plus the encoder's output.
GPT's move was to throw the encoder away. If your only job is "predict the next token," you never need to peek forward, because forward is exactly what you are trying to predict. What remains is a single stack that reads left-to-right. That stack is the decoder, hence decoder-only.
The "only look left" rule is enforced by a causal mask. Mechanically the model still processes the whole sequence in parallel during training; the mask just zeroes out any information flow from future positions to past ones. One forward pass over a 1,000-token sentence therefore trains 1,000 next-token predictions at once, one per position.
The entire forward pass is ten lines.
This is the actual shape of GPT-2 (the code below is from Karpathy's nanoGPT, which the post uses too). Read it once, then we will take it apart line by line.
tok_emb = self.transformer.wte(idx) # token embeddings, shape (b, t, n_embd) pos_emb = self.transformer.wpe(pos) # position embeddings, shape (t, n_embd) x = self.transformer.drop(tok_emb + pos_emb) for block in self.transformer.h: x = block(x) x = self.transformer.ln_f(x) logits = self.lm_head(x) return logits
idx is the batch of token ids, shape (b, t): b sentences, t tokens each. wte is a big lookup table with one 768-float row per vocabulary entry. Each integer id gets replaced by its row. No math, just a copy. Output: (b, t, 768).tok_emb + pos_emb is the model's working representation: what the token is plus where it sits.x a little. Bigger models mostly stack more of these (and make them wider). The next section opens one up.(768, 50304) maps each position's 768-float vector to one score per vocabulary entry: the logits. Softmax those and you have next-token probabilities.One block, zoomed in, is just two sub-steps wired together with additions:
class Block(nn.Module): def forward(self, x): x = x + self.attn(self.ln_1(x)) # 1. talk to other tokens x = x + self.mlp(self.ln_2(x)) # 2. think alone return x
x = x + f(x): the block adds its result onto x instead of replacing it.Every token asks a question. Softmax decides who answers.
Attention is the piece everything later in this post will rewrite, so it is worth building the intuition properly. Every token produces three vectors by multiplying its embedding with three learned matrices:
A token's query is dotted against every earlier token's key. Big dot product = strong match. Softmax turns those match scores into weights that sum to 1, and the output is the weighted average of the matched tokens' values. That is all attention is: a soft dictionary lookup. Here is GPT-2's real implementation, then the line-by-line.
B, T, C = x.size() # batch, sequence length, embedding dim (768) # one big matmul produces q, k, v for all heads at once q, k, v = self.c_attn(x).split(self.n_embd, dim=2) k = k.view(B, T, self.n_head, C // self.n_head).transpose(1, 2) # (B, nh, T, hs) q = q.view(B, T, self.n_head, C // self.n_head).transpose(1, 2) # (B, nh, T, hs) v = v.view(B, T, self.n_head, C // self.n_head).transpose(1, 2) # (B, nh, T, hs) att = (q @ k.transpose(-2, -1)) * (1.0 / math.sqrt(k.size(-1))) att = att.masked_fill(self.bias[:,:,:T,:T] == 0, float('-inf')) att = F.softmax(att, dim=-1) y = att @ v # (B, nh, T, T) x (B, nh, T, hs) -> (B, nh, T, hs) y = y.transpose(1, 2).contiguous().view(B, T, C) # stitch heads back together y = self.resid_dropout(self.c_proj(y)) return y
(768, 3×768) computes q, k and v in a single multiply, then split slices the result into three (B, T, 768) tensors. Purely an efficiency trick; conceptually there are three separate matrices Wq, Wk, Wv.hs = head size = 64). Each head runs its own independent attention over its own slice. The transpose just moves the head axis next to the batch axis so PyTorch treats the 12 heads like 12 extra batch items.(T, T) grid per head where cell (i, j) says "how much does token i care about token j". This grid is the famous O(N²): doubling the sequence length quadruples this matrix. Remember this cost. The whole rest of the post is a war against it.self.bias is a lower-triangular matrix of ones. Wherever it is zero (the upper triangle = future positions), the score is set to negative infinity. After softmax, e-∞ = 0: future tokens get exactly zero weight.Generation wastes work, so we cache it. Then the cache becomes the problem.
Watch what happens when the model generates. It runs the full forward pass over "the cat sat on the", computes representations for every position, and then uses only the last position's logits to pick "mat". Append "mat", run again... and without help it would recompute keys and values for "the cat sat on the" from scratch, even though none of them changed.
The fix is bookkeeping: store every token's k and v vectors the first time they are computed. On the next step, only the new token needs fresh projections; its query attends against the stored keys and values. That store is the KV cache. Generation splits into two phases: prefill (process the whole prompt at once, fill the cache) and decode (one new token at a time, reading the cache).
Now the catch. The cache grows by one entry per token, per layer, per head: O(N) in sequence length. At long contexts it becomes gigabytes. And at every decode step, the GPU must read the entire cache out of HBM just to produce one token. Decoding stops being limited by arithmetic and becomes limited by memory traffic. This is the memory-bandwidth bottleneck, and it is the villain that motivates every architecture in the rest of this post.
124M → 2.8T is a factor of 22,580. But is it just scale?
That is the post's framing question. Seven years, four and a half orders of magnitude. If the answer were "we just made GPT-2 bigger," this story would be over: more blocks, wider vectors, done.
The actual answer: the skeleton (decoder-only, blocks, residual stream) survived untouched, but the two expensive organs, attention and the MLP, were both replaced. Attention because its cache grows without bound. The MLP because activating 2.8 trillion parameters for every single token would be absurd.
The rest of the post follows one thread: the search for an attention whose memory does not grow. It runs through linear attention (2020), DeltaNet, Gated DeltaNet, and lands at Kimi Delta Attention inside K3. The MLP's replacement, Mixture-of-Experts, gets its own section near the end.
Move one parenthesis, and the memory stops growing.
Here is the pivotal piece of algebra, and it is genuinely just a parenthesis. Softmax attention must compute softmax(q·kT)·v. The exponential inside softmax couples every query to every key: you cannot simplify eq·k, so you must build the full score matrix, and you must keep every k and v around to build it.
Linear attention (Katharopoulos et al., 2020) asks: what if we make the scores non-negative without the exponential? Apply a
feature map
like elu(x)+1 to q and k separately. Then the score is just a plain dot product φ(q)·φ(k), and plain dot products obey the associative law:
Read the right-hand side slowly, because it is the founding move of everything that follows. kT·v is an
outer product:
one token's key (D numbers) times its value (D numbers) makes a D×D matrix, a little stamp recording "this key goes with this value." Add up the stamps of all past tokens and you get one D×D matrix S, the state. A new query just multiplies q·S to read from all of history at once. The past no longer exists as a list; it exists as a sum.
def forward(self, x, mask=None, past_kv=None): b,t,d = x.shape qkv = self.qkv_proj(x) # project x into q, k, v (one matmul) q = qkv[:, :, :d].view(b,t,h,d_head).transpose(1,2) k = qkv[:, :, d:2*d].view(b,t,h,d_head).transpose(1,2) v = qkv[:, :, 2*d:].view(b,t,h,d_head).transpose(1,2) if past_kv is not None: # DECODE: t=1, glue new k,v onto cache k = torch.cat((past_kv[0], k), dim=2) v = torch.cat((past_kv[1], v), dim=2) scores = (q @ k.transpose(-1,-2)) / math.sqrt(d_head) if past_kv is None: # PREFILL: need the causal mask scores = scores.masked_fill(causal_mask, float('-inf')) attn = scores.softmax(-1) o = attn @ v return self.o_proj(o), (k, v) # hand the grown cache back
dim=2 is the time axis; each decode step concatenates one more column. After 100k tokens, this cat is stitching your new 1-token key onto a 100k-token history, and the following matmul reads all of it.def forward(self, x, mask=None, cache=None): b,t,d = x.shape # ... same qkv projection and head reshaping as before ... k = F.elu(k) + 1 # feature map: make k non-negative k = k.transpose(-1,-2) # set up k as a column for the outer product q = F.elu(q) + 1 # same map on q S, z = cache if cache is not None else (0.0, 0.0) S = S + k @ v # fold this token's kᵀv stamp into the D×D state z = z + k # running sum of keys (the normalizer) o = q @ S # read: one matmul against the state denom = q @ z # what softmax's "divide by the sum" becomes o_scaled = o / denom return self.o_proj(o_scaled), (S, z) # cache is (S, z): FIXED SIZE
z accumulates all keys, so q·z equals the sum of all individual scores q·ki. Dividing by it makes the weights sum to 1, exactly like softmax's denominator.(k, v) lists that grow forever vs (S, z), one D×D matrix and one D-vector, the same size at token 10 and token 10 million. Each decode step is a couple of small matmuls regardless of history length.A historical footnote the post is honest about: the 2020 paper sold itself on fixing O(N²) compute, which reads strangely today because FlashAttention (2022) later showed softmax attention's real problem was memory traffic, not arithmetic. The framing aged badly; the mechanism did not. What matters for our story is the state: a fixed-size memory instead of a growing list.
A fixed whiteboard, and nobody ever erases.
Think of the state S as a whiteboard with room for roughly D independent facts (D×D numbers can only hold so much). Softmax attention never had this problem: every token got its own private slot in the cache, retrievable perfectly, forever.
On the whiteboard, token 4,001's stamp lands on top of the 4,000 stamps before it. When a query later reads the board, it gets its answer plus a smear of everything else whose keys point in similar directions. This is interference, and it gets strictly worse as the sequence outgrows the board, the regime the Fast Weight Programmers paper (Schlag et al., 2021) calls overcapacity.
Their diagnosis, lightly compressed: once sequence length exceeds storage capacity, the model must dynamically interact with the memory, deciding which key-value associations to keep and which to delete. A purely additive update cannot do that. Endlessly adding associations to a finite memory inevitably hits a wall.
Read what is there. Subtract it. Write the difference.
DeltaNet's insight: before writing to the board, check what the board already says at that key. If the slot for "capital of France" currently reads "Paris" and you want it to read "Lyon", do not add "Lyon" on top of "Paris". Add (Lyon − Paris), the delta, so the old entry cancels and the new one takes its place. An update that erases as it writes.
def forward(self, x, mask=None, cache=None): # ... same qkv projection and head reshaping ... q = F.normalize(F.silu(q), dim=-1) # unit-length queries k = F.normalize(F.silu(k), dim=-1) # unit-length keys (this matters!) beta = torch.sigmoid(self.w_beta(x)).view(b, 1, t, 1) # NEW: per-token write strength in (0,1) S = cache if cache is not None else 0.0 v_old = k @ S # 1. READ what the board says at this key u = beta * (v - v_old) # 2. the delta: only what is actually new S = S + k.transpose(-1,-2) @ u # 3. WRITE it, same outer-product as before o = q @ S # read for output (no denominator now) return self.o_proj(o), S
S = kTv. Read it back with the same key: k @ (kTv) = (k·kT) v = (squared length of k) × v. If k has length 1, you get back exactly v. Normalizing keys makes the board a faithful dictionary: what you read is really what was stored, so the subtraction in the next line removes exactly the right thing.u = 0: no write, no interference. If the board holds something stale, u carries precisely the correction. Compare with linear attention, which would have blindly added all of v every time.Making recurrence run like attention.
The post's author says this section took him seven hours. Here is the problem in one sentence: the delta rule is a loop where step i needs the state produced by step i−1, and a loop that must run one token at a time is poison for training, where you want to process a million-token batch in a few giant matrix multiplies. (This is the prefill/training problem; one-at-a-time is fine during decode, where tokens arrive one at a time anyway.)
S = torch.zeros(b, h, dh, dh) outs = [] for i in range(t): # one token at a time. for t = 1M: 1M tiny steps k_i = k[:, :, i:i+1] v_i = v[:, :, i:i+1] b_i = beta[:, :, i:i+1] v_old = k_i @ S # read depends on S... u_i = b_i * (v_i - v_old) S = S + k_i.transpose(-1,-2) @ u_i # ...and S depends on every step before outs.append(q[:, :, i:i+1] @ S)
The escape route is chunking, and it is easiest to see on plain linear attention first (no delta yet). Split the sequence into chunks of C tokens, say C=64. Then for each chunk, split its attention into two parts:
S = torch.zeros(b, h, dh, dh) outs = [] for i in range(t // C): # loop over CHUNKS, not tokens q_c = q[:, :, i*C:(i+1)*C] # this chunk's C queries/keys/values k_c = k[:, :, i*C:(i+1)*C] v_c = v[:, :, i*C:(i+1)*C] o_prev = q_c @ S # PAST: read everything before this chunk # from the state (recurrent order) attn = (q_c @ k_c.transpose(-1,-2)).tril() # PRESENT: real C×C masked attention o_curr = attn @ v_c # within the chunk (score order) o = o_prev + o_curr # each token: all of past + its chunk S = S + k_c.transpose(-1,-2) @ v_c # fold whole chunk into the state at once outs.append(o)
(kTv) then q..tril() (lower triangle) is the causal mask. This is the score order: (qkT) then v.
C is a dial between the two worlds. C = N (one chunk = whole sequence) collapses to standard O(N²) attention. C = 1 collapses to token-by-token linear attention. The FLOPs split cleanly in two: a fixed piece, 2Ld², the state work, which does not care about C, plus a growing piece, 2LCd, the C×C score triangles along the diagonal. Set C = L and that second term becomes 2L²d: quadratic attention recovered exactly. Smaller C means fewer FLOPs, but C=1 is slowest in wall-clock, because GPUs finish more arithmetic per second when the work arrives as chunky matmuls.
Tensor cores
like 64 or 128, so C is usually 64 or 128.
Direct chunking fails for DeltaNet because of one line: v_old = k_i @ S. Token i's correction needs the state as of token i−1, which includes the corrections of tokens inside the same chunk. The corrections depend on each other in a chain. To break the chain, the authors rewrite the update. The recurrent form:
# delta rule, written as one algebraic step (per token t): S_t = S_{t-1} @ (I - β_t k_t k_tᵀ) + β_t v_t k_tᵀ # │ │ │ # old state "erase along k_t" "write v at k_t"
def chunk_delta_rule_forward(Q, K, V, beta, C): L, d = Q.shape Q, K, V = map(lambda x: x.reshape(-1, C, d), [Q, K, V]) # cut into chunks beta = beta.reshape(-1, C) K_beta = K * beta.unsqueeze(-1) # keys pre-scaled by write strength V_beta = V * beta.unsqueeze(-1) # values pre-scaled by write strength # T resolves the WITHIN-CHUNK dependency chain (eq. 10): T = -(K_beta @ K.t()).tril(-1) # how much each token's write disturbs for i in range(1, C): # each LATER token's read (lower tri) T[i, :i] = T[i, :i] + (T[i, :, None] * T[:, :i]).sum(-2) T += torch.eye(C) # forward substitution: a C×C solve, # tiny because C is 64 W = T @ K_beta # "reader" keys, corrected for chunk-mates U = T @ V_beta # values, corrected for chunk-mates S = torch.zeros(d, d) O = torch.empty_like(V) for i in range(L // C): # the only sequential loop: over chunks q_i, k_i, w_i = Q[i], K[i], W[i] u_i = U[i] - w_i @ S # ALL C deltas at once, given incoming S o_inter = q_i @ S # read the past from the state A_i = (q_i @ k_i.t()).tril() # within-chunk attention scores o_intra = A_i @ u_i # attention over the CORRECTED values S += k_i.t() @ u_i # fold the chunk's corrections into S O[i] = o_intra + o_inter # present + past, same as before return O.reshape(L, d)
K_beta @ K.t() measures how much every token's key overlaps every other's: exactly how much token j's write will contaminate token i's read-back. .tril(-1) keeps only j < i (earlier tokens affect later ones, strictly). The minus sign: contamination must be subtracted.v_old = k_i @ S done token by token: same meaning, batched.Two erasers: one surgical, one global.
The delta rule replaces a fact when it has a specific replacement. But suppose the document changes topic entirely. Hundreds of stale associations should fade, and no new key is coming to overwrite each one individually. DeltaNet has no move for this.
Mamba-2 (a parallel line of research on state-space models) had the opposite toolkit. Its update is embarrassingly simple:
S_old = cache S_new = k @ v # cache = S_old + S_new ← linear attention: hoard forever cache = alpha * S_old + S_new # mamba-2: fade the past, write the present
alpha is a learned, data-dependent scalar in (0,1), produced per token from the input. At α=0.99 memories linger; at α=0.1 the board is nearly wiped. Because the model computes α from the current token, it can learn "this is a section break, flush the state." The state can no longer grow stale without bound.So the two lineages have exactly complementary blind spots: Delta rule: replace one fact precisely, cannot decay globally. Mamba-2: decay globally, cannot touch one fact. The Gated Delta rule (Gated DeltaNet, 2024) simply composes them: decay first, then delta-write.
S_t = α_t * ( S_{t-1} @ (I - β_t k_t k_tᵀ) ) + β_t v_t k_tᵀ # │ │ │ # global fade surgical erase at k_t write v_t at k_t # α=1 → pure DeltaNet (no forgetting) # β=0, α→0 → memory wipe
The chunked training story survives with one addition. Decay compounds multiplicatively: a fact written at step x and read at step x+t has been faded by αx·αx+1·...·αx+t. The kernels precompute cumulative-decay products γ along each chunk (the γr/γi ratio in the post's code), the multiplicative cousin of a running total, and fold them into the same T-matrix machinery from the previous section. Same skeleton, now with fading.
Full attention went on its own diet: MLA.
Before Kimi Linear, meet the other attention in its hybrid. While the linear-attention lineage replaced the KV cache with a state, a separate lineage (DeepSeek, 2024) kept softmax attention but shrank the cache. That is Multi-head Latent Attention, and Kimi K3 uses it too, so it needs two minutes here.
The observation: per-head keys and values are hugely redundant. So instead of caching all of them, MLA compresses each token's K/V information into one small latent vector (hundreds of dims instead of head-count × head-size), caches only that, and re-expands with learned up-projection matrices at read time. The cache shrinks by an order of magnitude; softmax retrieval quality survives, because the compression is learned end-to-end.
The same low-rank idea shows up again later under the name LoRA: factoring one big matrix into two skinny ones through a narrow middle. K3 applies it to MLA's query projection ("MLA query LoRA") purely to save parameters.
A decay knob per channel, and a hybrid that beat full attention.
Kimi Linear (2025) made news for one controlled-comparison claim: it outperformed full attention while decoding up to 6× faster, as a drop-in replacement. Its attention layer, Kimi Delta Attention (KDA), is Gated DeltaNet with one upgrade: instead of one scalar α fading the whole board uniformly, KDA learns a separate decay per channel, a whole vector of αs per token (in the code: alpha.reshape(nb, C, d), one decay value per feature dimension).
Why that matters: the state's D channels end up specializing. Some hold fast-moving local context (what was the last verb?); others hold slow facts (what language is this document in?). A single α forces one forgetting speed on all of them. Per-channel α lets local channels flush quickly while durable channels persist, fine-grained memory management, learned end to end.
Beside the per-channel gate, Kimi Linear makes two structural moves that carry into K3. It goes hybrid: mostly-KDA with MLA interleaved (the two failure modes cancel: KDA forgets but never runs out of memory, MLA never forgets but pays O(N) for it). And it swaps the MLP for a Mixture-of-Experts layer, next section.
2.8T parameters, but each token wakes only a sliver.
Now the MLP's turn. In GPT-2, every token passes through every MLP parameter. Scale that honestly to 2.8T and each token costs 2.8T parameter-reads: absurd. A Mixture-of-Experts layer replaces the one big MLP with many small MLPs (experts) and a router: a tiny learned layer that scores each expert per token (a dot-product similarity) and sends the token to the top-k scorers only.
So K3 can be a 2.8T-parameter model whose per-token compute is closer to a dense model a fraction of that size. Parameters become cheap storage for knowledge; compute is spent only on the slice of knowledge each token needs. This is the second organ swap (MLP → MoE), and it is what makes the 22,580× parameter count economically survivable.
The assembly: 92 layers, every trick with a job.
Kimi K3's language backbone is Kimi Linear scaled up and refined. The layout: 23 macrocycles of 4 layers each (3× KDA + 1× MLA = 92 layers). The first layer uses a plain dense feed-forward; every other layer uses the latent MoE. On top of that, a short list of upgrades. The big one, AttnRes, gets the next section; the rest go here.
Each MLA layer's output now passes through an elementwise gate: a sigmoid projection of the layer input, multiplied feature-by-feature into the attention output. Retrieval decides what could enter the residual stream; the gate decides what actually does, feature by feature. A bouncer between attention and the conveyor belt.
d = x.shape[-1] // 2 # expert's up-projection made a double-wide vector; gate = x[..., :d] # half becomes the gate... up = x[..., d:] # ...half becomes the payload situ_a = beta * torch.tanh(gate / beta) * torch.sigmoid(gate) if self.linear_beta is not None: up = linear_beta * torch.tanh(up / linear_beta) return situ_a * up # gate × payload, elementwise
K3's experts do not operate at the full residual width. Inputs are down-projected into a compressed latent space, the experts run there, and the summed result is up-projected back (the shared experts too). Smaller matrices per expert ≈ half the FLOPs per forward pass, which conveniently offsets the SiTU slowdown. The same compression trick as MLA, applied to the feed-forward path.
The last trick: attention over depth.
Remember the conveyor belt from section 03: layer l's input is the embedding plus the plain sum of every earlier layer's output, all weighted equally, forever. Two problems emerge at 92 layers deep. Dilution: layer 80 might need exactly what layer 9 computed, but that signal is buried under 70 other layers' additions. Instability: to be heard over the accumulated pile, later layers learn ever-larger outputs, and the hidden state's magnitude balloons.
AttnRes replaces the equal-weight sum with a weighted one, and computes the weights the way this whole post computes everything: with attention. Each layer gets a learned query; the keys and values are snapshots of the residual stream from earlier depths; softmax over the depth axis picks which past representations to pull forward. The same query-key-softmax machine, rotated 90°: instead of "which earlier token do I need," it asks "which earlier layer's work do I need."
V = torch.stack(blocks + [partial_block]) # [N+1, B, T, D] · snapshots of the # stream at each block boundary K = norm(V) # normalized copies act as the keys logits = torch.einsum('d, n b t d -> n b t', proj.weight.squeeze(), K) h = torch.einsum('n b t, n b t d -> b t d', logits.softmax(0), V) return h
[N+1, B, T, D]: depth, batch, tokens, width.proj.weight is one learned d-dim vector, the query. Dot it with every snapshot's every token vector (the d axes multiply and sum away), leaving one score per depth per token: [N+1, B, T]. Exactly q·k, with depth playing the role of sequence.dim=0 is the depth axis: for each token, the N+1 scores across depths become weights summing to 1. Not across tokens, across layers.att @ v from section 04, one-for-one.Running this at every layer would be too expensive, so K3 applies it blockwise: snapshots are taken every 12 decoder layers, giving 8 AttnRes blocks across the 92-layer stack, for about 2% extra inference latency (the post also credits it with a 1.25× compute advantage in training efficiency). And notice the symmetry the post lands on: KDA forgets across time, MLA recovers across time, AttnRes recovers across depth. Three selective-read mechanisms, one per axis where information gets lost.
Seven years, one column at a time.
Read this table top to bottom and the post's argument tells itself: each row exists because the row above had a named, specific failure.
| architecture | memory | write rule | can forget? | fixed by the next row |
|---|---|---|---|---|
| GPT-2 · softmax attn | KV cache, grows O(N) | append every token | never (perfect recall) | cache becomes a memory-bandwidth wall |
| Linear attention | fixed D×D state | add kᵀv stamps | no, and writes collide | interference once over capacity |
| DeltaNet | fixed D×D state | read, subtract, write β(v−v_old) | only by overwriting a key | no global decay for topic shifts |
| Gated DeltaNet | fixed D×D state | decay α·S, then delta-write | yes, uniformly | one α fades all channels alike |
| KDA / Kimi Linear | fixed state + periodic MLA | per-channel decay + delta-write | yes, per channel | fixed state still cannot do exact long-range recall alone |
| Kimi K3 | state (time) + MLA (time) + AttnRes (depth) | all of the above + MoE routing | yes, everywhere, selectively | to be continued |
And the closing argument, which is worth memorizing as a single sentence: a fixed-capacity memory needs an eviction policy. Purely additive writing fills any finite store and then corrupts it. So learned selection, gating, routing, decay, is not an optimization; it is a requirement. And attention, the query-key-softmax pattern, keeps winning as the best selective read: over tokens (MLA), over experts (the router), over depths (AttnRes).
Go to the originals.
This page is a from-scratch expansion of a worklog by ali (@waterloo_intern). The structure, code excerpts and the GPT-2 → K3 framing are theirs; the beginner scaffolding, diagrams and line-by-line walkthroughs are mine. Numbers describing Kimi K3 (2.8T, 898 experts, macrocycle layout, SiTU, AttnRes) follow the post.