the structure, code and framing are theirs. this page retells it from the beginning and fills in the basics the original assumes you know (what a decoder is, what a cache is).
22,580 GPT-2s fit inside one Kimi K3.
GPT-2 (2019) has 124 million parameters. Kimi K3 (2026) has 2.8 trillion. That is a 22,580× jump in seven years. You could say "we made it bigger," but that misses most of the story. The models also went through a chain of specific fixes. Each new architecture changed what the model remembers, how it overwrites old memories, and how it reads them back. This page walks through that chain from the beginning. It assumes no background and explains every line of code.
scroll · ~40 min read
A language model plays one game: guess the next token.
Before we get to any architecture, let's pin down the job. A language model takes in a piece of text and outputs a probability for every possible next chunk. Feed it "the cat sat on the" and it should give "mat" a high probability. During training it plays this one game billions of times.
Text first gets split into
tokens.
Each token is an integer id into a vocabulary. GPT-2's vocabulary has about 50,000 entries. The model never sees letters; it sees a list of integers like [464, 3797, 3332].
Each integer is then swapped for an embedding, a list of numbers the model can do math on. In GPT-2 each token becomes 768 floats. From this point until the very last step, the model works only with these vectors. It never touches text again.
At the very end, the model converts its final vector back into a score for every vocabulary entry. Those raw scores are called logits, and a softmax turns them into probabilities. Pick one token, append it to the input, and run the model again. Generating text means repeating that loop.
GPT-2 is a decoder-only transformer: one stack that reads left to right.
The original 2017 transformer was built for translation and had two halves. An encoder read the whole source sentence at once; every word could look at every other word, forwards and backwards. A decoder wrote the translation one word at a time, and it was only allowed to look at words it had already written, plus the encoder's output.
GPT threw the encoder away. If your only job is to "predict the next token," you never need to look ahead, because what comes next is the thing you are trying to predict. What is left is a single stack that reads left to right. That stack is the decoder, hence decoder-only.
A causal mask enforces the "only look left" rule. During training the model still processes the whole sequence in parallel. The mask blocks any information from flowing from later positions back to earlier ones. So one forward pass over a 1,000-token sentence trains 1,000 next-token predictions at once, one per position.
GPT-2's whole forward pass fits in under ten lines.
Here is GPT-2's forward pass as real code (from Karpathy's nanoGPT, which the post also uses). Read it once, then we will take it apart line by line.
tok_emb = self.transformer.wte(idx) # token embeddings, shape (b, t, n_embd) pos_emb = self.transformer.wpe(pos) # position embeddings, shape (t, n_embd) x = self.transformer.drop(tok_emb + pos_emb) for block in self.transformer.h: x = block(x) x = self.transformer.ln_f(x) logits = self.lm_head(x) return logits
idx is the batch of token ids, shape (b, t): b sentences, t tokens each. wte is a big lookup table with one 768-float row per vocabulary entry. Each integer id gets replaced by its row. There is no math here, only a copy. Output: (b, t, 768).tok_emb + pos_emb is what the model works with from here on: what the token is plus where it sits.x a little. Bigger models mostly stack more of these (and make them wider). We open one up below.(768, 50304) maps each position's 768-float vector to one score per vocabulary entry: the logits. Apply softmax to those and you get next-token probabilities.Zoom in on one block and there are only two steps, joined by additions:
class Block(nn.Module): def forward(self, x): x = x + self.attn(self.ln_1(x)) # 1. talk to other tokens x = x + self.mlp(self.ln_2(x)) # 2. think alone return x
x = x + f(x): the block adds its result onto x instead of replacing it.Each token asks a question, and softmax decides which earlier tokens answer.
Attention is the part that every later architecture in this post rewrites, so it is worth understanding well. Every token makes three vectors by multiplying its embedding by three learned matrices:
Take one token's query and compute its dot product with every earlier token's key. A big dot product means a strong match. Softmax turns those match scores into weights that sum to 1. The output is the weighted average of those tokens' values. So attention is a soft dictionary lookup: instead of returning one entry, it returns a blend of entries, weighted by how well each key matches. Here is GPT-2's real implementation, then a walk through it line by line.
B, T, C = x.size() # batch, sequence length, embedding dim (768) # one big matmul produces q, k, v for all heads at once q, k, v = self.c_attn(x).split(self.n_embd, dim=2) k = k.view(B, T, self.n_head, C // self.n_head).transpose(1, 2) # (B, nh, T, hs) q = q.view(B, T, self.n_head, C // self.n_head).transpose(1, 2) # (B, nh, T, hs) v = v.view(B, T, self.n_head, C // self.n_head).transpose(1, 2) # (B, nh, T, hs) att = (q @ k.transpose(-2, -1)) * (1.0 / math.sqrt(k.size(-1))) att = att.masked_fill(self.bias[:,:,:T,:T] == 0, float('-inf')) att = F.softmax(att, dim=-1) y = att @ v # (B, nh, T, T) x (B, nh, T, hs) -> (B, nh, T, hs) y = y.transpose(1, 2).contiguous().view(B, T, C) # stitch heads back together y = self.resid_dropout(self.c_proj(y)) return y
(768, 3×768) computes q, k and v in a single multiply, then split slices the result into three (B, T, 768) tensors. This is only for speed. Conceptually there are three separate matrices Wq, Wk, Wv.hs = head size = 64). Each head runs its own attention over its own slice. The transpose moves the head axis next to the batch axis, so PyTorch treats the 12 heads like 12 extra batch items.(T, T) grid per head, where cell (i, j) says "how much does token i care about token j". This grid is the famous O(N²) cost: double the sequence length and this matrix gets four times bigger. Remember this cost. The rest of the post is about ways around it.self.bias is a lower-triangular matrix of ones. Wherever it is zero (the upper triangle, meaning future positions), the score is set to negative infinity. After softmax, e-∞ = 0: future tokens get exactly zero weight.Generation repeats work, so we cache the results. Then the cache itself becomes the problem.
Look at what happens during generation. The model runs the full forward pass over "the cat sat on the", computes representations for every position, and then uses only the last position's logits to pick "mat". Now append "mat" and run again. Without some help, it would recompute the keys and values for "the cat sat on the" from scratch, even though none of them changed.
The fix is bookkeeping: store every token's k and v vectors the first time they are computed. On the next step, only the new token needs its q, k and v computed. Its query attends to the stored keys and values. That store is the KV cache. Generation now has two phases: prefill (process the whole prompt at once and fill the cache) and decode (produce one new token at a time, reading the cache).
The cache grows by one entry per token, per layer, per head, so its size is O(N) in sequence length. At long contexts it reaches gigabytes. And at every decode step, the GPU must read the entire cache out of HBM to produce a single token. Decoding is then limited by how fast memory can move data, while the arithmetic units wait. This is the memory-bandwidth bottleneck, and every architecture in the rest of this post is an attempt to get around it.
124M → 2.8T is a factor of 22,580. But is it just scale?
That is the question the post is built around. It is four and a half orders of magnitude in seven years. If the answer were "we made GPT-2 bigger," the story would be short: add more blocks, widen the vectors, done.
The skeleton (decoder-only, blocks, residual stream) did stay the same. But the two expensive parts, attention and the MLP, were both replaced. Attention was replaced because its cache grows without limit. The MLP was replaced because running all 2.8 trillion parameters for every token would cost far too much.
The rest of the post follows one thread: the search for an attention whose memory does not grow. It goes from linear attention (2020) to DeltaNet, then Gated DeltaNet, and ends at Kimi Delta Attention inside K3. The MLP's replacement, Mixture-of-Experts, gets its own section near the end.
Move one parenthesis, and the memory stops growing.
The key piece of algebra here comes down to where you put one pair of parentheses. Softmax attention must compute softmax(q·kT)·v. The exponential inside softmax ties every query to every key. You cannot split eq·k into a part that depends only on q and a part that depends only on k. So you have to build the full score matrix, and you have to keep every k and v around to build it.
Linear attention (Katharopoulos et al., 2020) asks: what if we make the scores non-negative without the exponential? Apply a
feature map
like elu(x)+1 to q and k separately. Then the score is a plain dot product φ(q)·φ(k), and with plain products you can regroup the multiplication however you like (the associative law):
Read the right-hand side slowly, because every later architecture builds on it. kT·v is an
outer product:
one token's key (D numbers) times its value (D numbers) makes a D×D matrix. Think of it as a small stamp that records "this key goes with this value." Add up the stamps of all past tokens and you get one D×D matrix S, called the state. A new query computes q·S to read from all of history at once. The model no longer keeps the past as a list of tokens. It keeps one running sum.
def forward(self, x, mask=None, past_kv=None): b,t,d = x.shape qkv = self.qkv_proj(x) # project x into q, k, v (one matmul) q = qkv[:, :, :d].view(b,t,h,d_head).transpose(1,2) k = qkv[:, :, d:2*d].view(b,t,h,d_head).transpose(1,2) v = qkv[:, :, 2*d:].view(b,t,h,d_head).transpose(1,2) if past_kv is not None: # DECODE: t=1, glue new k,v onto cache k = torch.cat((past_kv[0], k), dim=2) v = torch.cat((past_kv[1], v), dim=2) scores = (q @ k.transpose(-1,-2)) / math.sqrt(d_head) if past_kv is None: # PREFILL: need the causal mask scores = scores.masked_fill(causal_mask, float('-inf')) attn = scores.softmax(-1) o = attn @ v return self.o_proj(o), (k, v) # hand the grown cache back
dim=2 is the time axis, and each decode step adds one more column. After 100k tokens, this cat joins your new 1-token key onto a 100k-token history, and the next matmul reads all of it.def forward(self, x, mask=None, cache=None): b,t,d = x.shape # ... same qkv projection and head reshaping as before ... k = F.elu(k) + 1 # feature map: make k non-negative k = k.transpose(-1,-2) # set up k as a column for the outer product q = F.elu(q) + 1 # same map on q S, z = cache if cache is not None else (0.0, 0.0) S = S + k @ v # fold this token's kᵀv stamp into the D×D state z = z + k # running sum of keys (the normalizer) o = q @ S # read: one matmul against the state denom = q @ z # what softmax's "divide by the sum" becomes o_scaled = o / denom return self.o_proj(o_scaled), (S, z) # cache is (S, z): FIXED SIZE
z adds up all the keys, so q·z equals the sum of all the individual scores q·ki. Dividing by it makes the weights sum to 1, the same job softmax's denominator does.(k, v), lists that grow forever. Now it is (S, z), one D×D matrix and one D-vector, the same size at token 10 and at token 10 million. Each decode step is a couple of small matmuls, however long the history is.A historical footnote, which the post points out: the 2020 paper pitched itself as a fix for O(N²) compute. That reads strangely today, because FlashAttention (2022) later showed that softmax attention's real problem was memory traffic rather than arithmetic. The pitch aged badly, but the mechanism held up. What matters for our story is the state: a memory of fixed size instead of a growing list.
The state is a whiteboard of fixed size, and nothing ever gets erased.
Think of the state S as a whiteboard with room for roughly D independent facts (D×D numbers can only hold so much). Softmax attention never had this problem: every token got its own private slot in the cache, retrievable perfectly, forever.
On the whiteboard, token 4,001's stamp goes on top of the 4,000 stamps before it. When a query later reads the board, it gets its answer plus a smear of everything else whose keys point in similar directions. This is interference. It keeps getting worse as the sequence grows past what the board can hold. The Fast Weight Programmers paper (Schlag et al., 2021) calls this situation overcapacity.
Their diagnosis, lightly paraphrased: once the sequence is longer than the memory can store, the model must dynamically interact with the memory, deciding which key-value associations to keep and which to delete. An update that only adds cannot do that. If you keep adding associations to a finite memory, it eventually fills up.
DeltaNet reads what is stored, subtracts it, and writes only the difference.
DeltaNet's idea: before writing to the board, check what the board already says at that key. Say the slot for "capital of France" reads "Paris" and you want it to read "Lyon". Adding "Lyon" on top would leave both mixed together. Instead, add (Lyon − Paris), the delta. The old entry cancels and the new one takes its place. So the update erases as it writes.
def forward(self, x, mask=None, cache=None): # ... same qkv projection and head reshaping ... q = F.normalize(F.silu(q), dim=-1) # unit-length queries k = F.normalize(F.silu(k), dim=-1) # unit-length keys (this matters!) beta = torch.sigmoid(self.w_beta(x)).view(b, 1, t, 1) # NEW: per-token write strength in (0,1) S = cache if cache is not None else 0.0 v_old = k @ S # 1. READ what the board says at this key u = beta * (v - v_old) # 2. the delta: only what is actually new S = S + k.transpose(-1,-2) @ u # 3. WRITE it, same outer-product as before o = q @ S # read for output (no denominator now) return self.o_proj(o), S
S = kTv. Read it back with the same key: k @ (kTv) = (k·kT) v = (squared length of k) × v. If k has length 1, you get back exactly v. With normalized keys, what you read back is what was stored, so the subtraction below removes the right amount.u = 0: nothing is written, so nothing interferes. If the board holds something out of date, u is the correction. Linear attention, by contrast, would have added all of v every time.How to train a step-by-step loop in parallel, like attention.
The post's author says this section took him seven hours. The problem: the delta rule is a loop where step i needs the state produced by step i−1. A loop that must run one token at a time is terrible for training, where you want to process a million-token batch in a few giant matrix multiplies. (This is a prefill/training problem. During decode, one at a time is fine, because tokens arrive one at a time anyway.)
S = torch.zeros(b, h, dh, dh) outs = [] for i in range(t): # one token at a time. for t = 1M: 1M tiny steps k_i = k[:, :, i:i+1] v_i = v[:, :, i:i+1] b_i = beta[:, :, i:i+1] v_old = k_i @ S # read depends on S... u_i = b_i * (v_i - v_old) S = S + k_i.transpose(-1,-2) @ u_i # ...and S depends on every step before outs.append(q[:, :, i:i+1] @ S)
The way out is chunking. It is easiest to see on plain linear attention first, before adding the delta. Split the sequence into chunks of C tokens, say C=64. Then split each chunk's attention into two parts:
S = torch.zeros(b, h, dh, dh) outs = [] for i in range(t // C): # loop over CHUNKS, not tokens q_c = q[:, :, i*C:(i+1)*C] # this chunk's C queries/keys/values k_c = k[:, :, i*C:(i+1)*C] v_c = v[:, :, i*C:(i+1)*C] o_prev = q_c @ S # PAST: read everything before this chunk # from the state (recurrent order) attn = (q_c @ k_c.transpose(-1,-2)).tril() # PRESENT: real C×C masked attention o_curr = attn @ v_c # within the chunk (score order) o = o_prev + o_curr # each token: all of past + its chunk S = S + k_c.transpose(-1,-2) @ v_c # fold whole chunk into the state at once outs.append(o)
(kTv) then q..tril() (lower triangle) is the causal mask. This is the score order: (qkT) then v.
C is a dial between the two approaches. With C = N (one chunk is the whole sequence), you get standard O(N²) attention. With C = 1, you get token-by-token linear attention. The FLOPs split into two parts. One part, 2Ld², is the state work, and it does not depend on C. The other part, 2LCd, is the C×C score triangles along the diagonal, and it grows with C. Set C = L and that second term becomes 2L²d, which is exactly quadratic attention again. Smaller C means fewer FLOPs, but C=1 is the slowest in wall-clock time, because GPUs get through more arithmetic per second when the work comes as large matmuls.
Tensor cores
work best on tiles of 64 or 128, so C is usually 64 or 128.
Chunking DeltaNet directly does not work, because of one line: v_old = k_i @ S. Token i's correction needs the state as of token i−1, and that state includes the corrections from earlier tokens in the same chunk. So the corrections form a chain, each one depending on the one before. To break the chain, the authors rewrite the update. Here is the recurrent form:
# delta rule, written as one algebraic step (per token t): S_t = S_{t-1} @ (I - β_t k_t k_tᵀ) + β_t v_t k_tᵀ # │ │ │ # old state "erase along k_t" "write v at k_t"
def chunk_delta_rule_forward(Q, K, V, beta, C): L, d = Q.shape Q, K, V = map(lambda x: x.reshape(-1, C, d), [Q, K, V]) # cut into chunks beta = beta.reshape(-1, C) K_beta = K * beta.unsqueeze(-1) # keys pre-scaled by write strength V_beta = V * beta.unsqueeze(-1) # values pre-scaled by write strength # T resolves the WITHIN-CHUNK dependency chain (eq. 10): T = -(K_beta @ K.t()).tril(-1) # how much each token's write disturbs for i in range(1, C): # each LATER token's read (lower tri) T[i, :i] = T[i, :i] + (T[i, :, None] * T[:, :i]).sum(-2) T += torch.eye(C) # forward substitution: a C×C solve, # tiny because C is 64 W = T @ K_beta # "reader" keys, corrected for chunk-mates U = T @ V_beta # values, corrected for chunk-mates S = torch.zeros(d, d) O = torch.empty_like(V) for i in range(L // C): # the only sequential loop: over chunks q_i, k_i, w_i = Q[i], K[i], W[i] u_i = U[i] - w_i @ S # ALL C deltas at once, given incoming S o_inter = q_i @ S # read the past from the state A_i = (q_i @ k_i.t()).tril() # within-chunk attention scores o_intra = A_i @ u_i # attention over the CORRECTED values S += k_i.t() @ u_i # fold the chunk's corrections into S O[i] = o_intra + o_inter # present + past, same as before return O.reshape(L, d)
K_beta @ K.t() measures how much each token's key overlaps every other token's key. That overlap is how much token j's write will disturb token i's read. .tril(-1) keeps only j < i, since only strictly earlier tokens affect later ones. The minus sign is there because that disturbance has to be subtracted.v_old = k_i @ S, done token by token. It means the same thing, done in one batch.Gated DeltaNet has two erasers: one for a single key, one for everything at once.
The delta rule replaces a fact when a specific replacement comes along. But suppose the document changes topic completely. Hundreds of old associations should fade, and no new key is coming to overwrite each one. DeltaNet has no way to handle this.
Mamba-2 (from a separate line of research on state-space models) had the opposite tool. Its update is very simple:
S_old = cache S_new = k @ v # cache = S_old + S_new ← linear attention: hoard forever cache = alpha * S_old + S_new # mamba-2: fade the past, write the present
alpha is a learned, data-dependent scalar in (0,1), produced per token from the input. At α=0.99 memories last a long time; at α=0.1 the board is almost wiped. Because the model computes α from the current token, it can learn "this is a section break, clear the state." Old content can no longer pile up forever.So the two lines of work have opposite blind spots. The Delta rule can replace one fact precisely but cannot fade everything. Mamba-2 can fade everything but cannot touch one fact. The Gated Delta rule (Gated DeltaNet, 2024) combines them: decay first, then do the delta write.
S_t = α_t * ( S_{t-1} @ (I - β_t k_t k_tᵀ) ) + β_t v_t k_tᵀ # │ │ │ # global fade surgical erase at k_t write v_t at k_t # α=1 → pure DeltaNet (no forgetting) # β=0, α→0 → memory wipe
Chunked training still works, with one addition. Decay multiplies up over time: a fact written at step x and read at step x+t has been faded by αx·αx+1·...·αx+t. The kernels precompute the cumulative decay products γ along each chunk (the γr/γi ratio in the post's code). These work like a running total, with multiplication in place of addition. They get folded into the same T-matrix method from the previous section.
Full softmax attention got a much smaller cache: MLA.
Before Kimi Linear, meet the other kind of attention in its hybrid. The linear-attention line of work replaced the KV cache with a state. A separate line (DeepSeek, 2024) kept softmax attention but shrank the cache. That is Multi-head Latent Attention. Kimi K3 uses it too, so it gets a short section here.
The starting observation: the keys and values across heads repeat a lot of the same information. So instead of caching all of them, MLA compresses each token's K/V information into one small latent vector (hundreds of dims instead of head-count × head-size). It caches only that vector and expands it back with learned up-projection matrices at read time. The cache shrinks by an order of magnitude, and softmax retrieval quality holds up, because the compression is learned end to end.
The same low-rank idea appears again under the name LoRA: replace one big matrix with two thin ones that pass through a narrow middle. K3 applies it to MLA's query projection ("MLA query LoRA") only to save parameters.
A decay knob per channel, and a hybrid that beat full attention.
Kimi Linear (2025) made news for one claim from a controlled comparison: used as a drop-in replacement, it outperformed full attention while decoding up to 6× faster. Its attention layer, Kimi Delta Attention (KDA), is Gated DeltaNet with one upgrade. Instead of one scalar α fading the whole board evenly, KDA learns a separate decay per channel: a vector of αs for each token (in the code, alpha.reshape(nb, C, d), one decay value per feature dimension).
This matters because the state's D channels end up specializing. Some hold fast-changing local context (what was the last verb?). Others hold slow facts (what language is this document in?). A single α forces the same forgetting speed on all of them. A per-channel α lets the local channels clear quickly while the long-lived ones persist. The result is fine-grained memory management, learned end to end.
Besides the per-channel gate, Kimi Linear makes two structural changes that carry over into K3. First, it is a hybrid: mostly KDA layers, with MLA layers mixed in. The two cover each other's weaknesses. KDA forgets but never runs out of memory; MLA never forgets but pays O(N) memory for it. Second, it swaps the MLP for a Mixture-of-Experts layer, covered in the next section.
2.8T parameters, but each token uses only a small slice.
Now the MLP. In GPT-2, every token passes through every MLP parameter. Scale that directly to 2.8T and each token costs 2.8T parameter reads, which is far too much. A Mixture-of-Experts layer replaces the one big MLP with many small MLPs (experts) and a router. The router is a tiny learned layer that scores each expert for each token (a dot-product similarity) and sends the token only to the top-k scorers.
So K3 can be a 2.8T-parameter model whose per-token compute is closer to that of a dense model a fraction of its size. Parameters become cheap storage for knowledge, and compute is spent only on the part of that knowledge each token needs. This is the second swap (MLP → MoE), and it is what makes the 22,580× parameter count affordable to run.
Kimi K3 puts all of these pieces together in 92 layers.
Kimi K3's language backbone is Kimi Linear, scaled up and refined. The layout is 23 macrocycles of 4 layers each (3× KDA + 1× MLA = 92 layers). The first layer uses a plain dense feed-forward, and all the other layers use the latent MoE. On top of that come a few upgrades. The biggest one, AttnRes, gets the next section. The others are below.
Each MLA layer's output now passes through an elementwise gate. The gate is a sigmoid projection of the layer's input, and it is multiplied into the attention output one feature at a time. Attention decides what could enter the residual stream. The gate decides what does enter, feature by feature.
d = x.shape[-1] // 2 # expert's up-projection made a double-wide vector; gate = x[..., :d] # half becomes the gate... up = x[..., d:] # ...half becomes the payload situ_a = beta * torch.tanh(gate / beta) * torch.sigmoid(gate) if self.linear_beta is not None: up = linear_beta * torch.tanh(up / linear_beta) return situ_a * up # gate × payload, elementwise
K3's experts work at a smaller width than the residual stream. Inputs are down-projected into a compressed latent space, the experts run there, and the summed result is up-projected back (this applies to the shared experts too). Smaller matrices per expert give roughly half the FLOPs per forward pass, which happens to offset the SiTU slowdown. It is the same compression idea as MLA, applied to the feed-forward path.
AttnRes runs attention across layers instead of tokens.
Remember the conveyor belt from section 03: layer l's input is the embedding plus the plain sum of every earlier layer's output, all weighted equally. At 92 layers deep, this causes two problems. Dilution: layer 80 might need what layer 9 computed, but that signal is buried under 70 other layers' additions. Instability: to stand out from everything already on the belt, later layers learn larger and larger outputs, and the size of the hidden state keeps growing.
AttnRes replaces the equal-weight sum with a weighted one, and it computes the weights with attention, the same tool this post keeps coming back to. Each layer gets a learned query. The keys and values are snapshots of the residual stream from earlier depths. A softmax over the depth axis picks which earlier representations to bring forward. It is the same query-key-softmax machinery turned 90°: instead of asking "which earlier token do I need," it asks "which earlier layer's work do I need."
V = torch.stack(blocks + [partial_block]) # [N+1, B, T, D] · snapshots of the # stream at each block boundary K = norm(V) # normalized copies act as the keys logits = torch.einsum('d, n b t d -> n b t', proj.weight.squeeze(), K) h = torch.einsum('n b t, n b t d -> b t d', logits.softmax(0), V) return h
[N+1, B, T, D]: depth, batch, tokens, width.proj.weight is one learned d-dim vector, the query. Take its dot product with every token vector in every snapshot (the d axes multiply and sum away). That leaves one score per depth per token: [N+1, B, T]. This is q·k, with depth playing the role of the sequence.dim=0 is the depth axis. For each token, its N+1 scores across depths become weights that sum to 1. Earlier sections ran softmax across tokens; this one runs it across layers.att @ v from section 04 step for step.Running this at every layer would be too expensive, so K3 applies it blockwise. Snapshots are taken every 12 decoder layers, which gives 8 AttnRes blocks across the 92-layer stack and costs about 2% extra inference latency. (The post also credits it with a 1.25× compute advantage in training efficiency.) The post ends on a pattern: KDA forgets across time, MLA recovers across time, AttnRes recovers across depth. That makes three selective-read mechanisms, each working along an axis where information gets lost.
Seven years of architecture, in one table.
Read this table from top to bottom and you can see the post's argument: each row exists because the row above it had a specific, named failure.
| architecture | memory | write rule | can forget? | fixed by the next row |
|---|---|---|---|---|
| GPT-2 · softmax attn | KV cache, grows O(N) | append every token | never (perfect recall) | cache becomes a memory-bandwidth wall |
| Linear attention | fixed D×D state | add kᵀv stamps | no, and writes collide | interference once over capacity |
| DeltaNet | fixed D×D state | read, subtract, write β(v−v_old) | only by overwriting a key | no global decay for topic shifts |
| Gated DeltaNet | fixed D×D state | decay α·S, then delta-write | yes, uniformly | one α fades all channels alike |
| KDA / Kimi Linear | fixed state + periodic MLA | per-channel decay + delta-write | yes, per channel | fixed state still cannot do exact long-range recall alone |
| Kimi K3 | state (time) + MLA (time) + AttnRes (depth) | all of the above + MoE routing | yes, everywhere, selectively | to be continued |
The closing argument fits in one sentence worth remembering: a fixed-capacity memory needs an eviction policy, a rule for what to throw out. Writing that only adds will fill any finite store and then corrupt it. So learned selection (gating, routing, decay) is a requirement. And attention, the query-key-softmax pattern, keeps turning out to be the best way to read selectively: over tokens (MLA), over experts (the router), over depths (AttnRes).
Go to the originals.
This page expands on a worklog by ali (@waterloo_intern), starting from the basics. The structure, code excerpts and the GPT-2 → K3 framing are theirs. The beginner explanations, diagrams and line-by-line walkthroughs are mine. The numbers describing Kimi K3 (2.8T, 898 experts, macrocycle layout, SiTU, AttnRes) come from the post.