Pre-training, step by step.

The pipeline is a sequence of filters, and each one throws away most of what it sees. A snapshot arrives as WARC files: raw HTTP responses, HTML and all. trafilatura pulls the main article out of the HTML. FineWeb's first finding was that doing your own extraction from WARC beats using the pre-extracted WET files: cleaner text, better downstream models.

Then language identification: a fastText classifier scores each page, and FineWeb keeps English above 0.65. Then the quality heuristics inherited from Gopher and C4: drop documents with too many repeated lines, too few words, no terminal punctuation, too many symbols, the word "javascript", a "lorem ipsum". FineWeb added three of its own after ablations: too many short lines, too many lines without punctuation, too much duplicated character content.

Then deduplication. MinHash over 5-grams, 112 hash functions in 14 buckets of 8, which works out to catching documents about 75 percent similar. FineWeb's counterintuitive result: deduplicating across all snapshots at once made models worse than deduplicating within each snapshot, because global dedup preferentially removed the good, widely-mirrored text and kept the unique junk. Finally, PII: emails and IP addresses are replaced with placeholders.

Then the classifier

Heuristics get you to FineWeb. The next big jump was a learned quality classifier. FineWeb-Edu asked Llama-3-70B-Instruct to score about 460K pages on a 0 to 5 "educational value" scale, trained a small regression head on top of an embedding model with those labels, ran it over all 15T tokens, and kept pages scoring 3 or above. That left 1.3T tokens that beat the full set on knowledge benchmarks. DCLM did the same job with a fastText classifier trained to tell OpenHermes instruction data and good r/ELI5 answers apart from random web pages, and kept the top 10 percent. Every serious dataset since uses some version of this: a big model labels a sample, a cheap model scores everything.

datatrove is FineWeb's pipeline as code. A pipeline is a list of steps; each step is a class that consumes and yields documents. The real FineWeb file is longer, but this is the skeleton, with the real filter names and thresholds.

from datatrove.executor import SlurmPipelineExecutor
from datatrove.pipeline.readers import WarcReader
from datatrove.pipeline.extractors import Trafilatura
from datatrove.pipeline.filters import (URLFilter, LanguageFilter, GopherRepetitionFilter,
                                        GopherQualityFilter, C4QualityFilter, FineWebQualityFilter)
from datatrove.pipeline.writers import JsonlWriter

pipeline = [
    WarcReader("s3://commoncrawl/crawl-data/CC-MAIN-2024-10/segments/"),
    URLFilter(),                                       # blocklists + bad words in the url
    Trafilatura(favour_precision=True),                # html -> main text
    LanguageFilter(languages=["en"], language_threshold=0.65),
    GopherRepetitionFilter(),                          # dup lines / paragraphs / n-grams
    GopherQualityFilter(min_doc_words=50, max_doc_words=100_000),
    C4QualityFilter(filter_no_terminal_punct=False),   # fineweb kept most of c4, dropped this one
    FineWebQualityFilter(),                            # short-line and punctuation ratios
    JsonlWriter("s3://my-bucket/fineweb/filtered/"),
]
SlurmPipelineExecutor(pipeline, tasks=8000, time="10:00:00", partition="cpu").run()

Deduplication is a separate pipeline with four stages, because MinHash needs a global view: compute signatures, bucket them, find connected components of near-duplicates, then remove all but one per cluster.

from datatrove.pipeline.dedup import (MinhashDedupSignature, MinhashDedupBuckets,
                                      MinhashDedupCluster, MinhashDedupFilter, MinhashConfig)

cfg = MinhashConfig(hash_config=HashConfig(hash_fc="sha1", precision=64),
                    num_buckets=14, hashes_per_bucket=8, n_grams=5)   # 112 hashes ≈ 75% jaccard
stage1 = [JsonlReader(filtered), MinhashDedupSignature(output_folder=sigs, config=cfg)]
stage2 = [MinhashDedupBuckets(input_folder=sigs, output_folder=buckets, config=cfg)]
stage3 = [MinhashDedupCluster(input_folder=buckets, output_folder=clusters, config=cfg)]
stage4 = [JsonlReader(filtered), MinhashDedupFilter(input_folder=clusters), JsonlWriter(deduped)]

And the classifier stage, in essence. Score everything with a cheap model, keep the top of the distribution. This is FineWeb-Edu's: a regression head on a small embedding model, trained once on the Llama-70B labels.

from transformers import AutoTokenizer, AutoModelForSequenceClassification
tok = AutoTokenizer.from_pretrained("HuggingFaceFW/fineweb-edu-classifier")
clf = AutoModelForSequenceClassification.from_pretrained("HuggingFaceFW/fineweb-edu-classifier")

def keep(doc):
    score = clf(**tok(doc["text"], truncation=True, max_length=512)).logits.item()
    doc["score"] = score                            # e.g. 3.375
    doc["int_score"] = round(max(0, min(score, 5)))  # -> 3
    return doc["int_score"] >= 3                    # fineweb-edu keeps 3, 4, 5 -> 1.3T tokens

{
  "id": "<urn:uuid:673b1bf6-2c30-40ae-992b-c387d00a836a>",
  "dump": "CC-MAIN-2013-20",
  "text": "No. 24; Updated March 2011 ... Parents are usually the first to recognize ...",
  "url": "https://www.aacap.org/AACAP/Families_and_Youth/Facts_for_Families/FFF-Guide/When-to-Seek-Help-for-Your-Child-024.aspx",
  "date": null,
  "file_path": "s3://commoncrawl/crawl-data/CC-MAIN-2013-20/segments/1368696381249/warc/CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz",
  "language": "en", "language_score": 0.927742, "token_count": 755,
  "score": 3.375, "int_score": 3
}

What the other datasets add

// attributes/<tagger>/ file, same row order, holds the derived signals:// attributes/<tagger>/ file, same row order, holds the derived signals:// attributes/<tagger>/ file, same row order, holds the derived signals:als:
{"source": "...", "id": "...", "attributes": {"toxicity": 0.7,
  "olmo_mix_v1_taggers__ft_lang_id_en_paragraph_with_doc_score_v2__en": [[0, 300, 0.9], [300, 540, 0.3]]}}

What breaks at scale

• It is a CPU job, not a GPU job. The FineWeb recipe runs as thousands of Slurm CPU tasks over S3. The crawl is petabytes across snapshots; you stream it and never hold it.
• Dedup is the memory problem. MinHash signatures for 25B documents need a global shuffle, which is why it is four passes and not one function. DCLM's Bloom-filter approach (BFF, in Rust, 13-gram overlap at paragraph and document level) trades exactness for a single streaming pass.
• Decontamination. Benchmark questions leak into the web. Every lab now removes 13-gram overlaps with their eval sets before training, and it still is not perfect.
• The classifier decides the model's taste. FineWeb-Edu's labeling prompt asks for "educational" text, which tilts the corpus toward textbooks and away from fiction and dialogue. That is a design choice with downstream consequences, and it is why labs keep several classifiers and mix. Text extraction is the same kind of choice: trafilatura (FineWeb) and resiliparse (DCLM) produce different corpora from the same crawl.
• Licensing and opt-out. robots.txt and publisher opt-outs are honored at crawl time by some pipelines and not others. Code datasets filter by license. This is where legal, not engineering, sets the limits.

Modern tokenizers are byte-level BPE. Start with the 256 possible bytes as your vocabulary. Count every adjacent pair in the training text. Merge the most frequent pair into a new token. Repeat until you hit the vocabulary size you chose. Because you started from bytes, any string in any language can be encoded; there is no "unknown" token.

The part people miss is the pre-tokenizer regex. Before BPE runs, text is split into chunks that merges may never cross: a word with its leading space, a run of up to three digits, a run of punctuation, whitespace. That regex is why " generates" is one token and why "3,141" is three. GPT-4 introduced the three-digit rule so that numbers tokenize uniformly, and Llama 3 copied it almost verbatim.

Vocabulary size

The vocab size is a tradeoff between sequence length and embedding cost. A bigger vocabulary means fewer tokens per document (cheaper attention, longer effective context) but a bigger embedding table and output head, and rarer tokens that each get less training signal.

The other thing decided here: special tokens. A base model needs one, the document separator. But the chat format that post-training will teach needs more (turn markers, tool-call brackets), and the tokenizer cannot change later. So they are reserved up front. Llama 3 reserves 256 of them; nanochat defines nine, from <|bos|> to <|output_end|>, before pre-training even starts.

The regexes, verbatim. Read them as "things a merge may not cross".

# GPT-2 (tiktoken r50k)
'(?:[sdmt]|ll|ve|re)| ?\p{L}++| ?\p{N}++| ?[^\s\p{L}\p{N}]++|\s++$|\s+(?!\S)|\s

# GPT-4 (cl100k): digits capped at 3, case-insensitive contractions
'(?i:[sdmt]|ll|ve|re)|[^\r\n\p{L}\p{N}]?+\p{L}++|\p{N}{1,3}+| ?[^\s\p{L}\p{N}]++[\r\n]*+|\s++$|\s*[\r\n]|\s+(?!\S)|\s

# Llama 3 (llama-models/models/llama3/tokenizer.py, pat_str)
(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+

# nanochat (nanochat/tokenizer.py, SPLIT_PATTERN): digits capped at 2
'(?i:[sdmt]|ll|ve|re)|[^\r\n\p{L}\p{N}]?+\p{L}+|\p{N}{1,2}| ?[^\s\p{L}\p{N}]++[\r\n]*|\s*[\r\n]|\s+(?!\S)|\s+

Training is a loop that any of us could write. This is the algorithm from Karpathy's minbpe, which is what the Rust version in nanochat does with better data structures.

def train_bpe(text: str, vocab_size: int):
    ids = list(text.encode("utf-8"))               # start from bytes: 256 base tokens
    merges = {}
    for new_id in range(256, vocab_size):
        counts = {}
        for a, b in zip(ids, ids[1:]):             # count adjacent pairs (inside regex chunks, in practice)
            counts[(a, b)] = counts.get((a, b), 0) + 1
        pair = max(counts, key=counts.get)         # the most frequent pair becomes one token
        merges[pair] = new_id
        ids = merge(ids, pair, new_id)             # replace every occurrence
    return merges                                  # the tokenizer *is* this ordered list of merges

nanochat's script is the production shape of the same idea: cap the training text, cap each document, train in Rust, then hand the merges to tiktoken for fast encoding.

# scripts/tok_train.py: defaults --max-chars 2_000_000_000 --doc-cap 10_000 --vocab-size 32768
tokenizer = RustBPETokenizer.train_from_iterator(text_iter, args.vocab_size)   # rustbpe, GPT-4-style split

# nanochat/tokenizer.py: wrap the learned merges in tiktoken for encode_ordinary_batch speed
enc = tiktoken.Encoding(name="rustbpe", pat_str=SPLIT_PATTERN,
                        mergeable_ranks=mergeable_ranks, special_tokens=SPECIAL_TOKENS)
SPECIAL_TOKENS = ["<|bos|>", "<|user_start|>", "<|user_end|>", "<|assistant_start|>", "<|assistant_end|>",
                  "<|python_start|>", "<|python_end|>", "<|output_start|>", "<|output_end|>"]

What production adds

• Pad the vocabulary. The embedding table and output head are padded to a multiple of 64 or 128 rows so the matmuls tile onto tensor cores. nanochat: pad_vocab_size_to=64; OLMo pads to 128. The extra rows are never produced.
• Measure in bits per byte, not loss. Cross-entropy per token depends on the tokenizer. Divide by the bytes each token covers and runs with different vocabularies become comparable. nanochat reports val_bpb for exactly this reason.
• Digit policy. GPT-4 and Llama 3 chunk digits by three, nanochat by two, Qwen by one. Each is a bet on how the model will do arithmetic. There is no consensus, only ablations.
• Byte-level vs byte fallback. Llama 2 used SentencePiece with byte_fallback: normal tokens over characters, bytes only for what is left. Llama 3 switched to tiktoken's byte-level BPE, and so did nearly everyone else.
• Multilingual fertility. A vocabulary trained mostly on English spends three to five tokens per word on Thai or Hindi. Qwen3's 151K and Gemma's 262K vocabularies exist to bring that down. Apertus, trained on 1,800 languages, treats tokenizer coverage as a first-class design goal.
• The tokenizer is data too. nanochat trains it on the first 2B characters of the same corpus it will pre-train on. Train it on the wrong distribution and code or math tokenize badly for the life of the model.

The 2023 default, still in nanoGPT and torchtitan's simplest loader, is "concatenate with EOS, cut every T". It wastes nothing but it cuts documents mid-sentence and lets attention leak across them. Two refinements are now standard.

Best-fit packing treats the row as a bin. nanochat's loader starts every row with BOS, then repeatedly places the largest document that still fits; when nothing fits, it crops one to fill the row exactly. Result: no padding, every token can see its own document's start, and about 35 percent of tokens get cropped at T = 2048. The paper behind it is "Fewer Truncations Improve Language Modeling" (Ding et al., 2024).

Document masking fixes the leak. The attention mask becomes block-diagonal within the row, so a token never attends past its own document's start. It needs a kernel that understands variable-length sequences: FlashAttention's varlen interface with cu_seqlens, or FlexAttention with a document mask. OLMo-core passes doc_lens through the whole stack for this.

Mixing, and the three stages

The corpus is not one pile; it is dozens of sources with weights. The mixture says how many tokens come from each, and a source can be up-sampled past one epoch if it is scarce and valuable (OLMo-core's max_repetition_ratio). SmolLM3's config lists about 45 sources: FineWeb-Edu, DCLM, papers, Wikipedia, StackExchange, 14 languages of FineWeb2, two math sets, 15 languages of The Stack, pull requests, Kaggle notebooks, GitHub issues.

And the weights change over the run. The 2025 to 2026 norm is three stages: a long general stage at short context, a shorter "midtraining" stage where math, code and higher-quality text are up-weighted while the learning rate decays to zero, then a brief long-context stage at 32K to 64K tokens.

nanochat's loader tokenizes parquet shards on the fly and packs with best-fit. This is the core of tokenizing_distributed_data_loader_with_state_bos_bestfit, trimmed.

row_capacity = T + 1                                  # +1 so inputs and targets can both be T long
token_lists = tokenizer.encode(doc_batch, prepend=bos_token, num_threads=tokenizer_threads)
doc_buffer.extend(token_lists)
for row_idx in range(B):
    pos = 0
    while pos < row_capacity:
        remaining = row_capacity - pos
        best_idx, best_len = -1, 0
        for i, doc in enumerate(doc_buffer):          # largest document that fits entirely
            if len(doc) <= remaining and len(doc) > best_len:
                best_idx, best_len = i, len(doc)
        if best_idx >= 0:
            doc = doc_buffer.pop(best_idx)
            row_buffer[row_idx, pos:pos + len(doc)] = torch.tensor(doc); pos += len(doc)
        else:                                         # nothing fits: crop the shortest to fill exactly
            doc = doc_buffer.pop(shortest_idx)
            row_buffer[row_idx, pos:pos + remaining] = torch.tensor(doc[:remaining]); pos += remaining
inputs, targets = row_buffer[:, :-1], row_buffer[:, 1:]   # (B, T) each; targets are inputs shifted left

Document masking, two ways. modded-nanogpt cuts the stream at BOS positions and hands FlashAttention-3 the boundaries; torchtitan builds the same thing as a FlexAttention block mask.

# modded-nanogpt: boundaries as cumulative sequence lengths, batch size 1
# seqlens = [0, 5, 14, 16, ...]  cumulative document starts within the packed stream
y = flash_attn_varlen_func(q[0], k[0], v[0], cu_seqlens_q=seqlens, cu_seqlens_k=seqlens,
                           max_seqlen_q=max_len, max_seqlen_k=max_len,
                           causal=True, window_size=(bm_size, 0))          # sliding window too

# torchtitan: the same mask as a predicate for FlexAttention
def document_causal(b, h, q_idx, kv_idx):
    return (q_idx >= kv_idx) & (doc_id[b, q_idx] == doc_id[b, kv_idx])
block_mask = create_block_mask(document_causal, B, None, T, T)
y = flex_attention(q, k, v, block_mask=block_mask)

Mixing is configuration, not code. OLMo-core's source mixture is a list of sources with target ratios; the official mix manifest is 951 lines of one file path each.

SourceMixtureConfig(source_name="dclm",      target_ratio=0.55, paths=[...], max_repetition_ratio=1.0)
SourceMixtureConfig(source_name="stack_edu", target_ratio=0.15, paths=[...], max_repetition_ratio=2.0)  # up-sample: two epochs
SourceMixtureConfig(source_name="math",      target_ratio=0.05, paths=[...])

# data/mixes/OLMo-mix-0925-official.txt · one line per tokenized shard
common_crawl_adult_content,preprocessed/dolma3-0625/v0.1-official/{TOKENIZER}/common_crawl/adult_content/000000.npy

What the tokens look like on disk

Where it gets hard

• Cropping vs padding. nanochat accepts 35 percent cropped tokens because data is abundant. With scarce data (a specialist domain, a small language) you keep the older loader and live with mid-document cuts.
• Naive masking is slow. A dense (T×T) mask through standard attention costs the same as full attention and then some. The varlen and flex kernels skip the zero blocks entirely; that is the whole point of them.
• Long context is mostly a RoPE change. Raise the rotary base (10K to 500K for Llama 3 and OLMo, 1M for Qwen3, 1.5M then 5M for SmolLM3), train briefly at the longer length, extrapolate further with YaRN at inference. The training-time cost is context parallelism: OLMo 3 splits 65K-token sequences across 8 GPUs.
• The published base model is the post-anneal checkpoint. Midtraining is where the learning rate reaches zero and where math and code get up-weighted, so "the base model" you download is not the constant-LR model. Annealing runs are also how labs A/B test data: branch from the stable checkpoint, anneal on candidate mixes, compare.
• Epochs vs rephrasing. Kimi K2 reuses high-value tokens by rephrasing them with an LLM rather than repeating them, on the evidence that repeats past a couple of epochs stop helping.
• Resume must be deterministic. The loader state is checkpointed with the model so a restart continues from the same document. nanochat's is approximate (it advances one row group to avoid repeats); OLMo's is exact by construction of the index.

Read the middle column top to bottom. RMSNorm replaced LayerNorm: it only divides by the root-mean-square, no mean subtraction, no bias, and nanochat drops the learnable gain too. Cheaper and just as good.

RoPE replaced learned position embeddings. Instead of adding a position vector to the input, rotate the query and key vectors by an angle that grows with position, so the dot product between two tokens depends only on their distance. It generalizes past the training length, which learned positions never did, and it is what makes long-context extension a matter of changing one number, the base frequency.

GQA (grouped-query attention) shrinks the KV cache: 32 query heads share 8 key-value heads in Llama 3 8B, 64 share 8 in Qwen3-32B. Inference memory drops fourfold or more for a tiny quality cost. QK-norm applies RMSNorm to queries and keys before the dot product, which caps attention logits and is the single most common fix for training instability in the 2025 recipes (OLMo 2 and 3, Qwen3, nanochat).

SwiGLU replaced the GELU MLP: three matrices instead of two, a gate multiplied into an up-projection, hidden width cut to 8/3 of the old 4× so the parameter count stays the same. The speedrun world uses ReLU² instead, one matrix fewer and just as good at that scale. Biases are gone everywhere, and so is dropout. Embeddings are usually untied from the output head.

Newer: sliding-window layers. Three layers attend only to the last 768 or 4,096 tokens and every fourth attends to everything (nanochat's pattern is "SSSL", OLMo 3's is [4096, 4096, 4096, -1]). Most of the cost of attention goes away and the full layers keep long-range access. And a logit softcap: 15 · tanh(logits / 15) on the output, from Gemma 2, which stops the final logits from running away.

nanochat's gpt.py is the 2026 dense block in 555 lines. The attention forward, lightly trimmed. Note what is not there: no positional embedding table, no bias, no dropout.

def forward(self, x, ve, cos_sin, window_size, kv_cache):
    B, T, C = x.size()
    q = self.c_q(x).view(B, T, self.n_head, self.head_dim)
    k = self.c_k(x).view(B, T, self.n_kv_head, self.head_dim)      # n_kv_head < n_head: GQA
    v = self.c_v(x).view(B, T, self.n_kv_head, self.head_dim)
    if ve is not None:                                              # value embeddings on alternate layers
        gate = 3 * torch.sigmoid(self.ve_gate(x[..., :self.ve_gate_channels]))
        v = v + gate.unsqueeze(-1) * ve
    cos, sin = cos_sin
    q, k = apply_rotary_emb(q, cos, sin), apply_rotary_emb(k, cos, sin)   # RoPE
    q, k = norm(q), norm(k)                                         # QK-norm
    y = flash_attn.flash_attn_func(q, k, v, causal=True, window_size=window_size)  # (768, 0) or (T, 0)
    return self.c_proj(y.contiguous().view(B, T, -1))

class MLP(nn.Module):
    def forward(self, x):
        return self.c_proj(F.relu(self.c_fc(x)).square())           # relu²: one matrix fewer than SwiGLU

class Block(nn.Module):
    def forward(self, x, ve, cos_sin, window_size, kv_cache):
        x = x + self.attn(norm(x), ve, cos_sin, window_size, kv_cache)   # pre-norm, residual
        x = x + self.mlp(norm(x))
        return x

def norm(x):
    return F.rms_norm(x, (x.size(-1),))                             # no learnable gain

The SwiGLU version, from the Llama reference, so you can see where the 8/3 comes from.

class FeedForward(nn.Module):
    def __init__(self, dim, hidden_dim, multiple_of, ffn_dim_multiplier):
        hidden_dim = int(2 * hidden_dim / 3)                          # 4d -> 8d/3: same params as a 2-matrix mlp
        if ffn_dim_multiplier is not None:
            hidden_dim = int(ffn_dim_multiplier * hidden_dim)         # llama 3.1 8b: 1.3 -> 14,336
        hidden_dim = multiple_of * ((hidden_dim + multiple_of - 1) // multiple_of)
        self.w1 = Linear(dim, hidden_dim, bias=False)                 # gate
        self.w2 = Linear(hidden_dim, dim, bias=False)                 # down
        self.w3 = Linear(dim, hidden_dim, bias=False)                 # up
    def forward(self, x):
        return self.w2(F.silu(self.w1(x)) * self.w3(x))

What the frontier models add

MLA, multi-head latent attention, is DeepSeek's answer to the KV cache: compress keys and values into a 512-dimensional latent per token, store only that, and decompress on the fly. A separate 64-dimensional "rope key" carries position, because the rotation cannot pass through the compression. Queries get their own low-rank path (rank 1,536). In absorb mode the decompression matrix folds into the score computation, so inference never materializes full keys at all. DeepSeek-V3 and Kimi K2 both use it; Kimi K3 gates it and interleaves it with a linear-attention layer (KDA), 69 linear to 24 gated-MLA layers.

MTP, multi-token prediction: one extra small block that predicts the token after next from the main model's hidden state, trained with its own cross-entropy at weight 0.3 for the first 10T tokens and 0.1 after. It densifies the training signal and doubles as a draft model for speculative decoding. torchtitan's models/deepseek_v3/MTP.md documents the block: enorm, hnorm, eh_proj, one transformer block, a shared output head.

DeepSeek V4 (April 2026) goes further on attention memory: sliding windows plus heavily compressed attention and compressed sparse attention, with a 1M context. Kimi K3 is a hybrid with linear attention. Nemotron 3 is a hybrid Mamba-Transformer MoE. The direction is consistent: attention over the full sequence is now the exception, not the rule.

What the speedrun adds, and a warning

modded-nanogpt's file has value embeddings, U-net style skip connections between early and late layers (now "MUDD" skips), a bigram hash embedding, paired-head attention, learnable cross-sequence attention, and an FP8 output head. Its README says plainly that "some methods used in the speedrun are unlikely to scale, particularly those imposing additional network structure, such as logit softcapping."

Stability choices, side by side

The router is a single linear layer. It maps the token's hidden state to one score per expert (256 of them in V3), squashes the scores with a sigmoid, and picks the top-k (8). Those experts run on the token; their outputs are summed, weighted by the renormalized scores. One shared expert runs on every token regardless, so common knowledge does not have to be duplicated across the routed ones. Kimi K2 keeps the same shape at 384 experts; Kimi K3 has 896 with top-16 and two shared; Qwen3-MoE has 128, top-8, and no shared expert.

The failure mode is load imbalance. Nothing stops the router from sending every token to the same three experts, which then become a small dense model while the other 253 sit idle. The classic fix is an auxiliary loss that penalizes uneven expert usage, but that loss fights the language-modeling loss. DeepSeek-V3's fix is auxiliary-loss-free: a per-expert bias added to the scores only for the purpose of selection. After each step, the bias of an overloaded expert goes down by a small constant and an underloaded one goes up. The weights the model uses to mix outputs never see the bias. Toggle it in the diagram above.

The memory-versus-compute math

Read the first column against the second. Per-token FLOPs scale with active parameters, so V3 trains and serves like a 37B dense model. Total parameters scale memory, so it needs the weights of a 671B model resident across the cluster. That asymmetry is the whole bet: parameters are cheap storage for knowledge, compute is the scarce thing. Kimi K2's report fits a "sparsity scaling law" to pick 384/8 = 48 as the ratio.

DeepSeek-V3's inference code is the clearest reference I have found. The gate, verbatim except for comments. Everything is in it: sigmoid scores, the balancing bias, node-limited routing, top-k, renormalization.

def forward(self, x):                                    # class Gate, inference/model.py
    scores = linear(x, self.weight)                                  # (tokens, n_experts)
    scores = scores.softmax(-1, dtype=torch.float32) if self.score_func == "softmax" else scores.sigmoid()
    original_scores = scores
    if self.bias is not None:
        scores = scores + self.bias                                  # balancing bias: affects *which*, not *how much*
    if self.n_groups > 1:                                            # group-limited routing: pick 4 of 8 groups first
        scores = scores.view(x.size(0), self.n_groups, -1)
        group_scores = scores.amax(-1) if self.bias is None else scores.topk(2, -1)[0].sum(-1)
        indices = group_scores.topk(self.topk_groups, -1)[1]
        mask = scores.new_ones(x.size(0), self.n_groups, dtype=bool).scatter_(1, indices, False)
        scores = scores.masked_fill_(mask.unsqueeze(-1), float("-inf")).flatten(1)
    indices = torch.topk(scores, self.topk, dim=-1)[1]               # the 8 experts for each token
    weights = original_scores.gather(1, indices)                     # mixing weights come from the *unbiased* scores
    if self.score_func == "sigmoid":
        weights /= weights.sum(dim=-1, keepdim=True)                 # renormalize to 1
    weights *= self.route_scale                                      # 2.5 in v3
    return weights.type_as(x), indices

The MoE layer itself is a loop over experts. Simple, correct, and the version to understand before the fast one.

def forward(self, x):                                    # class MoE
    shape = x.size()
    x = x.view(-1, self.dim)
    weights, indices = self.gate(x)                      # (N, k), (N, k)
    y = torch.zeros_like(x)
    counts = torch.bincount(indices.flatten(), minlength=self.n_routed_experts).tolist()
    for i in range(self.experts_start_idx, self.experts_end_idx):    # only this rank's experts
        if counts[i] == 0:
            continue
        expert = self.experts[i]
        idx, top = torch.where(indices == i)              # which tokens chose expert i, and in which slot
        y[idx] += expert(x[idx]) * weights[idx, top, None]
    z = self.shared_experts(x)                            # every token, always
    if world_size > 1:
        dist.all_reduce(y)                                # sum the partial results from every rank's experts
    return (y + z).view(shape)

The bias update, which lives in the training loop rather than the model, is one line in essence. γ was 0.001 for the first 14.3T tokens of V3 and 0 for the rest.

load = torch.bincount(indices.flatten(), minlength=n_experts).float()   # tokens per expert this step
gate.bias += gamma * torch.sign(load.mean() - load)                        # overloaded: down, underloaded: up

Production replaces the Python loop with a permutation and a grouped GEMM: sort tokens by expert, run one batched matmul over the variable-sized groups, unsort. torchtitan's GroupedExperts does this with torch._grouped_mm; MegaBlocks did it first with block-sparse kernels.

order = indices.flatten().argsort()                       # tokens grouped by expert id
x_sorted = x.repeat_interleave(k, 0)[order]               # each token appears k times, once per chosen expert
offsets = counts.cumsum(0)                                # where each expert's group ends
h = torch._grouped_mm(x_sorted, w1, offs=offsets)         # one kernel, all experts, no padding
h = F.silu(torch._grouped_mm(x_sorted, w_gate, offs=offsets)) * h
out_sorted = torch._grouped_mm(h, w2, offs=offsets)
y = torch.zeros_like(x).index_add_(0, token_of[order], out_sorted * weights.flatten()[order, None])

Expert parallelism

256 experts do not fit on one GPU, so they are spread across an expert-parallel group and tokens travel to their experts: an all-to-all before the experts (dispatch) and another after (combine). DeepSeek-V3 trained with 64-way EP across 8 nodes, and its node-limited routing exists precisely so a token's 8 experts sit on at most 4 nodes, bounding the cross-node traffic. DeepEP is their open kernel for the all-to-all. Kimi K2 used 16-way EP with 16-way pipeline parallelism.

EP composes with FSDP awkwardly: the experts are sharded one way (by expert) and everything else another (by parameter). torchtitan keeps a separate "edp" mesh for the expert weights and prefetches expert all-gathers around the all-to-all.

Where MoE training breaks

• Router collapse. Without balancing, a few experts win early and the rest never learn. The Switch Transformer loss is α · N · Σᵢ fᵢ · Pᵢ (fraction of tokens routed to i, times mean router probability for i). DeepSeek-V3 keeps a tiny sequence-wise version at α = 0.0001 alongside the bias. Qwen3 computes the balance over the global batch, not per micro-batch, so that specialization within a domain is not penalized.
• Router in fp32. The scores are computed in float32 even in a bf16 or FP8 run; tiny score differences decide routing and low precision makes them noisy. DeepSeek keeps the gating modules out of FP8 entirely.
• Capacity factor vs dropless. Old MoEs capped tokens per expert and dropped the overflow. Dropless MoE (MegaBlocks) processes everything with variable-sized groups, which is what grouped GEMM makes possible.
• Upcycling. Copy a dense model's MLP into every expert and continue training. Qwen and MiniCPM did this; it avoids a cold start but the experts begin identical and take a while to diverge.
• Fine-tuning fragility. Small post-training sets barely touch most experts. Kimi K2's advice is to fine-tune a Muon-pretrained MoE with Muon; LoRA on MoE layers is still an open question in the open literature.
• Serving is a different problem. The 37B-active model still needs 671B parameters resident, so batching many requests is what makes it economical; single-user latency does not benefit.

Initialization

Nearly every lab initializes every matrix from a truncated normal with standard deviation 0.02 (OLMo's InitMethod.normal, torchtitan's default), and scales the output projections down with depth so that the residual stream does not grow layer by layer. The speedrun world goes further and zero-initializes the output projections of attention and the MLP, so every block starts as the identity. nanochat's table is worth reading in full, because every line is a deliberate choice.

wte (embedding):     normal, std=0.8   (then RMS-normed, so the scale is free)
lm_head:             normal, std=0.001
attn.c_q, c_k, c_v:  uniform, std=1/sqrt(n_embd)   (uniform avoids outliers)
attn.c_proj:         zeros
mlp.c_fc:            uniform, std=0.4/sqrt(n_embd)
mlp.c_proj:          zeros
resid_lambdas:       1.15 -> 1.05 across depth     (per-layer residual scale)
x0_lambdas:          0.20 -> 0.05 across depth     (blend the input embedding back in)

AdamW, the boring consensus

β₁ = 0.9, β₂ = 0.95, weight decay 0.1, gradient clipping at 1.0, and no weight decay on the embeddings. That line describes Llama 3, DeepSeek-V3, OLMo 3, SmolLM3 and Qwen3. Peak learning rates: 3e-4 for OLMo 3 7B, 2e-4 for SmolLM3 and Kimi K2, 2.2e-4 for DeepSeek-V3. Warmup of 2,000 steps or so. The differences between labs are in the schedule and the batch, not the optimizer.

Muon

Muon is the exception. For the 2-D weight matrices inside the blocks, it takes the momentum-averaged gradient and orthogonalizes it: replace the update matrix with the nearest matrix whose singular values are all one, computed with five Newton-Schulz iterations in bf16. The effect is that every direction in the update gets the same step size, instead of a few dominant directions taking all of it. Embeddings, the output head and scalars stay on AdamW. It won the speedrun in 2024, Moonshot showed it scales ("Muon is Scalable", 16B on 5.7T tokens, about half the FLOPs of AdamW for equal loss), and Kimi K2 trained a trillion-parameter model with it. MuonClip is K2's addition: after each step, if any attention head's maximum logit exceeds a threshold τ, rescale that head's query and key weights down. K2 reports 15.5T tokens without a single loss spike.

The schedule, the batch, the budget

Two schedule families coexist. Cosine decay to about 10 percent of peak (Llama 2 and 3, OLMo 3, Kimi K3). WSD, warmup then stable then decay (nanochat, OLMo-core's option, SmolLM3, DeepSeek-V3 and Kimi K2 in a multi-phase form). WSD's flat middle is what makes annealing experiments and run extensions possible. A batch-size ramp is standard too: DeepSeek-V3 grows from 3,072 to 15,360 sequences over the first 469B tokens; nanochat settled on 1M tokens per step.

And the budget. Chinchilla's 20 tokens per parameter is now a floor, not a target, because inference cost depends on parameters and not on training tokens.

Muon, verbatim from the modded-nanogpt README. The three constants are a quintic polynomial tuned so the iteration converges fast in bf16.

def zeroth_power_via_newtonschulz5(G, steps=5, eps=1e-7):
    a, b, c = (3.4445, -4.7750, 2.0315)
    X = G.bfloat16() / (G.norm() + eps)                  # normalize so the iteration converges
    if G.size(0) > G.size(1): X = X.T
    for _ in range(steps):
        A = X @ X.T
        B = b * A + c * A @ A
        X = a * X + B @ X                                # X -> polynomial in X X^T, pushes singular values to 1
    if G.size(0) > G.size(1): X = X.T
    return X.to(G.dtype)

# the step, per matrix parameter
buf.mul_(momentum).add_(grad)                           # nesterov momentum
g = grad.add(buf, alpha=momentum)
g = zeroth_power_via_newtonschulz5(g)                    # orthogonalize
p.add_(g, alpha=-lr * max(1, p.size(0) / p.size(1)) ** 0.5)   # scale by aspect ratio

nanochat's optimizer setup shows the split: which parameters get Muon, which get AdamW, and how the AdamW learning rates scale with width so that one config works for every depth.

dmodel_lr_scale = (model_dim / 768) ** -0.5              # AdamW lr ∝ 1/√d_model, tuned at 768
param_groups = [
    dict(kind='adamw', params=lm_head_params,   lr=0.004 * dmodel_lr_scale, betas=(0.8, 0.96),  weight_decay=0.01),
    dict(kind='adamw', params=embedding_params, lr=0.2   * dmodel_lr_scale, betas=(0.8, 0.995), weight_decay=0.001),
    dict(kind='adamw', params=[resid_lambdas],  lr=0.005, betas=(0.8, 0.95), weight_decay=0.05),
    dict(kind='adamw', params=[x0_lambdas],     lr=0.5,   betas=(0.96, 0.95), weight_decay=0.0),
]
for shape in sorted({p.shape for p in matrix_params}):   # every 2-D weight inside the blocks
    param_groups.append(dict(kind='muon', params=[p for p in matrix_params if p.shape == shape],
                             lr=0.02, momentum=0.95, ns_steps=5, beta2=0.9, weight_decay=weight_decay))
optimizer = MuonAdamW(param_groups)

The schedule is twenty lines. This is nanochat's, a WSD with a long linear warmdown; the Muon momentum has its own schedule alongside it.

def get_lr_multiplier(it):                               # scripts/base_train.py
    warmup_iters = args.warmup_steps                     # 40
    warmdown_iters = round(args.warmdown_ratio * num_iterations)   # 0.65 of the run
    if it < warmup_iters:
        return (it + 1) / warmup_iters
    elif it <= num_iterations - warmdown_iters:
        return 1.0
    else:
        progress = (num_iterations - it) / warmdown_iters
        return progress * 1.0 + (1 - progress) * args.final_lr_frac   # down to 0.05

def get_muon_momentum(it):                               # 0.85 -> 0.97 over 400 steps, back to 0.90 in the warmdown
    ...

num_iterations = (target_param_data_ratio * num_scaling_params) // total_batch_size   # the budget, from one dial

MuonClip, in essence, from the Kimi K2 report. It runs after the optimizer step and leaves the forward and backward of the current step untouched.

for h in heads:
    S_max = (1 / sqrt(d)) * max_over_batch(Q_h @ K_h.T)          # this head's largest attention logit
    if S_max > tau:                                              # tau = 100
        gamma = tau / S_max
        W_q[h] *= gamma ** alpha                                 # alpha = 0.5: split the shrink between q and k
        W_k[h] *= gamma ** (1 - alpha)

What the ablation diaries say

• Weight decay scales with width. nanochat's log found the optimum goes as 1/width² (0.22 at depth 12, 0.08 at depth 20), and schedules it linearly to zero over the run.
• Gradient clipping can go. With Muon and QK-norm, nanochat deleted clipping: "grad norm never exceeds 1.0 naturally", and modded-nanogpt does not clip either. Every production run with AdamW still clips at 1.0.
• Muon keeps evolving. nanochat replaced Newton-Schulz with Polar Express coefficients, added row equilibration, renormalizes updates to a target Frobenius norm, and uses NorMuon's factored second moments and "cautious" weight decay (decay only where the update and the weight agree in sign). Kimi K3 orthogonalizes per attention head. Megatron-Core added Muon support in 2026.
• z-loss vs softcap. OLMo 3 adds 1e-5 · log²(Σ exp logits) to keep the output logits from drifting; DCLM used 5e-6. The speedrun family uses a tanh softcap instead. Both solve the same instability.
• Nobody uses muP in production. Every verified recipe (Llama 3, DeepSeek-V3, OLMo 3, Kimi K2 and K3, Qwen3, SmolLM3) fits its learning rate and batch size from scaling-law sweeps on small models rather than from muP. Qwen3's report says it "sets the predicted optimal learning rate and batch size for each dense or MoE model". nanochat derives everything from --depth.
• The full DeepSeek-V3 schedule, for the record: 2K warmup steps to 2.2e-4, constant until 10T tokens, cosine to 2.2e-5 over the next 4.3T, constant at 2.2e-5 for 333B, then 7.3e-6 for the final 167B.

The loop, with accumulation

On one node the loop is nanochat's: a few forward-backward passes per optimizer step (gradient accumulation, so a 1M-token batch fits in memory), then one step. bf16 activations, fp32 master weights and optimizer state, torch.compile for fusion. The multi-node version has the same shape with four kinds of splitting layered on.

Data parallel, then FSDP

Plain data parallelism copies the whole model to every GPU and averages gradients. It stops working when the model does not fit on one GPU. FSDP (fully sharded data parallel, ZeRO-3 in DeepSpeed's naming) keeps only a slice of every parameter on each GPU, all-gathers a layer's weights right before it runs, and reduce-scatters its gradients right after. Memory per GPU drops by the number of GPUs; the cost is communication that has to be overlapped with compute. FSDP2 in PyTorch is one function, fully_shard, called once per block and once on the root. OLMo 3 trained its 7B on 512 H100s with just this, in hybrid form (replicate across nodes, shard within).

Tensor parallel

Split each matrix across the GPUs of one node: column-wise for the first matmul of a block, row-wise for the second, so only one all-reduce is needed per sub-block. Sequence parallel extends the split to the norms and dropout between blocks. It lives inside a node because it needs NVLink bandwidth every layer. Llama 3 405B uses 8-way TP.

Pipeline parallel

Put layers 0 to 15 on one set of GPUs and 16 to 31 on another. The batch is cut into micro-batches that flow through the stages, and the schedule (1F1B, interleaved, zero-bubble, DeepSeek's DualPipe) decides how much of the time a stage sits idle waiting for its neighbor. Kimi K2 used 16-way pipeline with virtual stages; DeepSeek-V3, 16-way with DualPipe.

Context and expert parallel

Context parallel splits the sequence itself across GPUs and passes keys and values around a ring, which is how OLMo 3 trains 65K-token sequences with a degree of 8. Expert parallel puts different experts on different GPUs and moves tokens to them with all-to-all; DeepSeek-V3 ran 64-way EP across 8 nodes. torchtitan's rule for composing them: the dense part of the model is sharded over (dp × cp × tp) and the sparse part over (efsdp × ep), covering the same ranks.

Precision

Default: bf16 for activations and the matmuls, fp32 for the master weights, the optimizer moments and the gradient reduction. FP8 goes one step further on the big linear layers only. DeepSeek-V3's recipe quantizes activations per 1×128 tile and weights per 128×128 block in E4M3, and promotes to fp32 accumulation every 128 elements. Embeddings, the output head, the MoE router, norms and attention stay in bf16 or fp32. Llama 4 trained in FP8; Nemotron 3 went to NVFP4; Kimi K3 uses MXFP4 weights with quantization-aware training from SFT onward, so the served model matches the trained one.

nanochat's loop, verbatim and trimmed. The next batch is fetched while the GPU is busy with backward.

for micro_step in range(grad_accum_steps):
    loss = model(x, y)
    train_loss = loss.detach()
    loss = loss / grad_accum_steps
    loss.backward()
    x, y, dataloader_state_dict = next(train_loader)      # prefetch next batch while GPU is busy
lrm = get_lr_multiplier(step)
muon_momentum = get_muon_momentum(step)
for group in optimizer.param_groups:
    group["lr"] = group["initial_lr"] * lrm
    if group["kind"] == "muon":
        group["momentum"] = muon_momentum
optimizer.step()                                           # gradient sync happens inside, no DDP
model.zero_grad(set_to_none=True)

# grad_accum_steps = total_batch_size // (device_batch_size * max_seq_len * world_size)

FSDP2 is this. Each call makes one unit that all-gathers its bf16 parameters before forward and backward and reduce-scatters bf16 gradients into fp32 sharded gradients afterward.

from torch.distributed.fsdp import fully_shard, MixedPrecisionPolicy

mp_policy = MixedPrecisionPolicy(param_dtype=torch.bfloat16, reduce_dtype=torch.float32)
fsdp_config = {"mesh": dp_mesh, "mp_policy": mp_policy}
for layer_id, transformer_block in model.layers.items():
    fully_shard(transformer_block, **fsdp_config, reshard_after_forward=reshard_after_forward)
fully_shard(model, **fsdp_config)                          # the root: embeddings, norm, head

torchtitan's step adds what a cluster needs: a gradient norm that is correct across pipeline stages and expert meshes, and a hard stop on a non-finite loss before the optimizer can apply it.

grad_norm = dist_utils.clip_grad_norm_(
    [p for m in self.model_parts for p in m.parameters()],
    self.config.training.max_norm, foreach=True,
    pp_mesh=parallel_dims.get_optional_mesh("pp"), ep_enabled=parallel_dims.ep_enabled)
step_is_finite = loss_is_finite.logical_and(torch.isfinite(grad_norm).all())
torch._assert_async(step_is_finite,
    "Loss or gradient norm is not finite on at least one rank at "
    f"step {self.step}. Stopping training before the optimizer update.")
self.checkpointer.maybe_wait_for_staging()               # async checkpoint copy must finish before params change
self.optimizers.step()
self.lr_schedulers.step()

The parallel layout is configuration. torchtitan dropped its TOML files in 2026; configs are Python now. This is the registered DeepSeek-V3 671B entry, trimmed.

# torchtitan/models/deepseek_v3/config_registry.py · deepseek_v3_671b
optimizer=default_adamw(lr=2.2e-4),
lr_scheduler=LRSchedulersContainer.Config(warmup_steps=2000, decay_ratio=0.8, decay_type="cosine", min_lr_factor=0.1),
training=TrainingConfig(num_tokens_per_microbatch_per_dp_rank=4 * model_spec.max_context_length, steps=10000),
parallelism=ParallelismConfig(pipeline_parallel_schedule="Interleaved1F1B", expert_parallel_degree=2),
checkpoint=CheckpointManager.Config(interval=500),
activation_checkpoint=SelectiveAC.Config(),
compile=CompileConfig(enable=True, components=["loss"]),

# ParallelismConfig fields (defaults): data_parallel_replicate_degree=1, data_parallel_shard_degree=-1,
#   tensor_parallel_degree=1, pipeline_parallel_degree=1, pipeline_parallel_schedule="1F1B",
#   context_parallel_degree=1, expert_parallel_degree=1
# on the command line:  --parallelism.data_parallel_shard_degree 2 --parallelism.tensor_parallel_degree 2

And FP8 in essence, from DeepSeek-V3's inference kernel: per-tile scaling with the E4M3 maximum of 448.

@triton.jit
def act_quant_kernel(x_ptr, y_ptr, s_ptr, BLOCK_SIZE: tl.constexpr, scale_fmt: tl.constexpr):
    pid = tl.program_id(axis=0)
    offs = pid * BLOCK_SIZE + tl.arange(0, BLOCK_SIZE)      # one 1x128 tile of activations
    x = tl.load(x_ptr + offs).to(tl.float32)
    amax = tl.max(tl.abs(x))
    amax = tl.maximum(amax, 1e-4)
    s = amax / 448.                                          # 448 = the largest e4m3 value
    y = x / s
    tl.store(y_ptr + offs, y.to(y_ptr.dtype.element_ty))     # float8_e4m3fn
    tl.store(s_ptr + pid, s)                                 # one fp32 scale per tile

The memory arithmetic

What the communication looks like

• FSDP: one all-gather per layer forward, one all-gather and one reduce-scatter per layer backward, overlapped with the next layer's compute. reshard_after_forward frees the gathered weights between forward and backward; torchtitan keeps them for the last block and under pipeline parallel.
• TP: two all-reduces per block (after attention, after the MLP), or reduce-scatter and all-gather pairs with sequence parallel. Needs NVLink; never crosses a node. Async TP overlaps these with the matmuls.
• PP: point-to-point activations between stages. The bubble is the idle fraction: (stages − 1) / micro-batches for 1F1B, which is why micro-batch counts are large and why zero-bubble and DualPipe schedules exist.
• EP: two all-to-alls per MoE layer, dispatch and combine. DeepEP is DeepSeek's kernel for it, with FP8 dispatch; node-limited routing caps how many nodes a token's experts span.
• CP: keys and values pass around a ring. torchtitan balances the causal work with a head-and-tail split per rank.

Where it breaks

• Order of wrapping matters. torchtitan applies declarative sharding (TP), then activation checkpointing, then compile, then FSDP. Get it wrong and compile graphs break or hooks fire twice.
• Clipping needs a global norm. The stock clip_grad_norm_ only sees one rank's slice under PP; torchtitan reduces norm^p across stages and takes the p-th root. nanochat once clipped per rank instead of globally, and the ablation log records the bug.
• The router must not be quantized. torchtitan's float8 converter refuses to touch the router gate, and the 671B config filters lm_head and router.gate out of FP8. Small matmuls lose to quantization overhead too, so there is an auto-filter by shape.
• FP8 needs compile. Without it the scale-and-cast kernels are not fused and FP8 is slower than bf16. Rowwise scaling keeps all communication in high precision; only the matmuls run in e4m3.
• MFU is what you tune for. Picotron reports 38 percent on Llama 2 7B across 64 H100s and about 50 percent on a 1.7B model on 8. Anything above 40 at scale is good; MoE and long-context runs land lower.

Checkpoints

Every 500 to 1,000 steps, write the model, the optimizer state and the dataloader position to storage. PyTorch's distributed checkpoint (DCP) has every rank write its own shard in parallel, and can load the result onto a different number of GPUs, which is what lets you resume a 512-GPU run on 480 after a node dies. Saving is asynchronous: the GPU-to-host copy is staged, the write happens in a background process, and the next optimizer step waits only for the staging to finish. OLMo 3 saved every 1,000 steps and each stage of its run starts from a checkpoint URL of the previous one; every intermediate checkpoint is published.

Loss spikes

The loss suddenly jumps and sometimes never comes back. The cause is almost always attention logits growing until the softmax saturates. Three defenses, in order of preference. Prevent them in the architecture: QK-norm, reordered norm, z-loss, no weight decay on embeddings. Prevent them in the optimizer: MuonClip, which is how Kimi K2 got through 15.5T tokens with zero spikes. Or skip the bad step: OLMo-core's SkipStepOptimizer skips any update whose loss or gradient norm is more than 6 standard deviations above a rolling 128-step window. DeepSeek-V3 reports no irrecoverable spikes and no rollbacks; torchtitan refuses to continue on a non-finite loss and makes you restart from a checkpoint.

What to watch

Loss, smoothed. Gradient norm, because it rises before a spike. Tokens per second and MFU, model FLOPs utilization: the fraction of the hardware's peak you are actually using, computed from a formula rather than measured. Held-out loss every few hundred steps, in bits per byte. And downstream evals every few thousand steps: HellaSwag, ARC, MMLU subsets, a code and a math task, scored by the log-likelihood of the right continuation, since a base model cannot follow instructions.

CORE

nanochat's definition of "GPT-2 grade" is the DCLM CORE score: 22 few-shot tasks, each scored as centered accuracy, (accuracy − random baseline) / (1 − random baseline), then averaged. GPT-2 1.6B scores 0.2565; the current speedrun beats it in 1.65 hours. One number that moves smoothly over a run is worth more than twenty that do not.

DCP, from torchtitan's checkpointer. The to_hf branch writes safetensors that Hugging Face tooling can load.

if to_hf:
    state_dict = self.sd_adapter.to_hf(state_dict)                    # rename keys into the HF layout
    storage_writer = HuggingFaceStorageWriter(path=save_path, save_distributed=True, ...)
if async_mode == AsyncMode.ASYNC:
    ret = dcp.async_save(state_dict, storage_writer=storage_writer,
                         checkpoint_id=checkpoint_save_id, process_group=self.pg)
elif async_mode == AsyncMode.ASYNC_WITH_PINNED_MEM:                     # stage to pinned host memory first
    ret = dcp.async_save(state_dict, storage_writer=storage_writer, checkpoint_id=checkpoint_save_id,
                         process_group=self.pg, async_checkpointer_type=AsyncCheckpointerType.PROCESS,
                         async_stager=self.stager)
else:
    ret = dcp.save(state_dict, storage_writer=storage_writer, checkpoint_id=checkpoint_save_id)

nanochat saves less, but saves the thing people forget: where the dataloader was.

checkpoint = {
    "model": orig_model.state_dict(),
    "optimizer": optimizer.state_dict(),
    "dataloader": dataloader_state_dict,            # {"pq_idx": ..., "rg_idx": ..., "epoch": ...}
    "loop": {"step": step, "min_val_bpb": min_val_bpb, "smooth_train_loss": smooth_train_loss,
             "total_training_time": total_training_time},
}

MFU is arithmetic. Six FLOPs per parameter per token for the matmuls (forward plus backward), plus the attention term, over the peak the vendor publishes for dense bf16.

# nanochat GPT.estimate_flops, generalized for sliding windows
num_flops_per_token = 6 * self.num_matmul_params()                    # embeddings excluded
for layer in layers:
    num_flops_per_token += 12 * n_head * head_dim * min(window, T)      # attention: q·k and softmax·v
mfu = 100 * (num_flops_per_token * tokens_per_sec) / (peak_flops * world_size)
# peak_flops: H100 SXM 989e12 · H100 PCIe 756e12 · A100 312e12   (dense bf16, no sparsity)

And CORE, in essence.

def core_score(results):                                   # nanochat/core_eval.py, the idea
    centered = []
    for task, acc in results.items():
        baseline = RANDOM_BASELINE[task]                    # e.g. 0.25 for a 4-way multiple choice
        centered.append((acc - baseline) / (1 - baseline))  # 0 = random, 1 = perfect
    return sum(centered) / len(centered)                    # mean over 22 tasks

What a month on a cluster is like

• Failures are constant. Llama 3's paper counted 466 job interruptions over 54 days of the 405B run, 419 of them unexpected, most from GPU faults. The training loop is a small part of the job; the automation that detects a dead node, restarts from the last checkpoint and gets back to full speed is the rest.
• Checkpoint cost is a budget. Pick the interval so that the expected lost work (half an interval times the failure rate) balances the save overhead. Async staging makes a save nearly free in wall-clock; storage bandwidth per rank sets the floor.
• A seed checkpoint makes runs reproducible. torchtitan writes the initialized weights once so that a run can be restarted from identical numbers on any GPU count.
• Resume is approximate unless you make it exact. nanochat advances one row group past the saved position to avoid repeating data; OLMo's index makes the position exact. RNG state is rarely saved, so dropout-free architectures help here too.
• Metrics can lie. DCLM re-based CORE from v1 to v2 in September 2025, so scores across that line are not comparable. Held-out loss depends on the tokenizer, hence bits per byte. Benchmark contamination inflates everything.
• Souping is normal. OLMo 3 32B averages two midtraining runs and its last three long-context checkpoints. The published "base" is often an average, not a single step.

What a base model does is completion. Given "The capital of France is", a good one writes " Paris." and then, because it is still predicting the most likely continuation, keeps going: " The capital of Germany is Berlin." Given a question formatted like a forum post, it writes a forum answer, then another post. Nothing in it knows when to stop, except the document separator it learned to predict at the end of documents.

The trick people used before post-training existed was few-shot prompting: show the model three examples of the format you want and it completes the fourth. That is still how base models are evaluated. HellaSwag, MMLU and the CORE tasks are all rendered as a few examples plus a candidate continuation, scored by the probability the model assigns to the right one. A base model can score well on knowledge benchmarks and still be useless in a chat.

nanochat's README describes the speedrun model as "a bit like talking to a kindergartener": it knows the sky is blue and will invent the reason. That is what 1.65 hours of H100 time buys. The frontier base models are the same object, just trained a few million times longer.

Sampling from a base model is the inference loop with nothing around it: one forward pass per token, take the argmax or sample, append, repeat.

ids = [tokenizer.get_bos_token_id()] + tokenizer.encode("The capital of France is")
for _ in range(16):
    logits = model(torch.tensor([ids]))[0, -1]                  # only the last position matters
    next_id = int(logits.argmax())                               # temperature 0: greedy
    ids.append(next_id)
    if next_id == tokenizer.get_bos_token_id():                  # the document separator: "this text is over"
        break
print(tokenizer.decode(ids[1:]))

The evaluation harness renders benchmarks the same way. A multiple-choice question becomes several candidate strings, and the score is which candidate the model finds most probable.

context = few_shot_examples + question                          # 5 solved examples, then the real one
scores = []
for choice in choices:                                          # "A) ...", "B) ...", ...
    ids = tokenizer.encode(context + choice)
    logp = model.log_prob_of_continuation(ids, start=len(tokenizer.encode(context)))
    scores.append(logp)                                         # sum of log-probs of the choice tokens
correct = scores.index(max(scores)) == answer_idx

The handoff

Which checkpoint becomes "the base model" is a choice. Usually it is the post-anneal one, after the learning rate has decayed to zero on the high-quality mix, and sometimes it is an average of several. OLMo 3 publishes the whole trail: Base, then SFT, then DPO, then RLVR, for both an Instruct track and a Think track, with every intermediate checkpoint downloadable. Kimi K2's report recommends fine-tuning a Muon-pretrained checkpoint with Muon. Kimi K3 does quantization-aware training and tunes its multi-token-prediction draft head during post-training so the served model matches the trained numerics.

The base checkpoint goes into supervised fine-tuning first, where it learns the chat format and the special tokens the tokenizer reserved back in step 2. That is the next page.