How a Frontier LLM Is Built.

One step does almost all the work. The rest is polish.

A state-of-the-art LLM in 2026 is not one thing trained one way. It is an assembly line of four stages. The single most useful fact about that line is how lopsided it is: one box costs almost all the money, and the three after it are what actually make the model feel smart. Step through them.

One builds capability. The other builds usability.

Everything in LLM training falls into one of two buckets. Get this split straight and the rest stops looking like alphabet soup. Pre-training reads trillions of tokens and learns one thing: predict the next token. Post-training takes the parameters pre-training already set and nudges them so the model answers instead of rambles.

Learning to predict the next token.

This is the expensive one. Take a transformer with billions of parameters, point it at trillions of tokens, and ask it to do exactly one thing over and over: given everything so far, guess what comes next. Get it wrong and a loss function pushes the weights to be a little less wrong. Do that across the whole internet and grammar, facts, code, arithmetic, and a usable world model all fall out as a side effect.

How do labs pick how big the model and how much data should be? Not by guessing. By scaling laws. Chinchilla fit a clean curve, loss as a function of parameters N and data D, and found the compute-optimal recipe is roughly 20 tokens of data for every parameter.

What the data actually looks like

People picture a clever curated dataset. It is mostly the opposite. Open sets like FineWeb are built from deduplication and filtering of raw Common Crawl, roughly 15T tokens from 114 web snapshots. Each row is just a document.

{
  "text": "The mitochondrion is a double-membrane-bound organelle found in most eukaryotic cells. It generates most of the cell's supply of ATP...",
  "id": "urn:uuid:d66bc6fe-8477-4adf-b430-f6a558ccc8ff",
  "dump": "CC-MAIN-2024-10",
  "url": "https://en.wikipedia.org/wiki/Mitochondrion",
  "language": "en", "language_score": 0.97, "token_count": 717
}

What the loss actually measures

At each position the model does not pick a token. It produces a probability for every token in its vocabulary, tens of thousands of them. The cross-entropy loss reads just one of those numbers: the probability it placed on the token that actually came next.

The loss is the negative log of that one probability. Put a high probability on the right token and the loss is near zero. Put a low one and the model was surprised, so the loss is large. Minimizing that number, averaged over trillions of tokens, is the whole of pre-training.

The objective, in one screenful

The whole training signal is cross-entropy on the next token. Exponentiate it and you get perplexity. In essence, one step is this.

import torch.nn.functional as F

# One pretraining step: predict every next token at once.
logits  = model(tokens[:, :-1])     # (batch, seq-1, vocab)
targets = tokens[:, 1:]             # shift left by one
loss = F.cross_entropy(
    logits.flatten(0, 1),          # (batch*(seq-1), vocab)
    targets.flatten(),
)
loss.backward()                     # nudge billions of weights

Run that for trillions of tokens and the loss curve has a particular shape: it plunges early, when the model learns the cheap structure of language, then flattens into a long grind of diminishing returns. The flat part is where the real money goes.

A base model can't answer you

Architecture is part of the recipe. Llama 3.1 405B is dense, every parameter fires for every token, chosen for training stability. DeepSeek-V3 uses a Mixture of Experts, carrying 671B total parameters but spending only 37B activated per token. Either way, the output is the same kind of thing.

Teaching it to answer.

First stop in post-training. We have a model that completes text; we want one that responds to instructions. The fix is almost embarrassingly simple: show it thousands of examples of the behavior we want. Supervised fine-tuning trains the base model on curated pairs of (prompt, ideal answer). Same machinery, same cross-entropy loss. The only real change is the data.

What the data actually looks like

An SFT example is a short conversation: an id, a list of messages with roles, and where it came from. Here is a real-shaped row from the Tülu 3 mixture.

{
  "id": "oasst1_5921",
  "messages": [
    { "role": "user", "content": "Explain photosynthesis to a 10-year-old." },
    { "role": "assistant", "content": "Photosynthesis is how plants make their own food. They take in sunlight, water, and the air we breathe out, and turn it into sugar for energy and oxygen for us..." }
  ],
  "source": "ai2-adapt-dev/oasst1_converted"
}

The objective, in one screenful

Here is the one twist that makes SFT work. You do not want the model graded on predicting the user's question, only on producing the answer. So it is the exact same cross-entropy as pre-training, masked to the assistant tokens.

# SFT: identical cross-entropy, but only assistant tokens count.
logits  = model(input_ids[:, :-1])
targets = input_ids[:, 1:]
tok_loss = F.cross_entropy(logits.flatten(0, 1), targets.flatten(),
                           reduction="none")
loss = (tok_loss * response_mask[:, 1:].flatten()).sum() / response_mask.sum()
loss.backward()

Aligning with what people prefer.

SFT teaches the model to answer. Preference tuning teaches it which answer is better. This is where tone, helpfulness, and safety get dialed in, and it is the stage most responsible for a model feeling polished. The classic recipe is RLHF, introduced at scale by InstructGPT.

That optimization step is reinforcement learning. Classic RLHF uses PPO, which is powerful but heavy, with a KL penalty as a brake.

The shortcut that took over: DPO

That whole apparatus is a lot. DPO showed you can skip most of it. With a bit of algebra you can train directly on the preference pairs, with no separate reward model and no PPO loop. Preference data is a triple: a prompt, a chosen answer, and a rejected one.

{
  "prompt": "Write a haiku about coffee.",
  "chosen": "Bean falls, water hums, one quiet golden morning.",
  "rejected": "Coffee is a drink. It is brown. People enjoy it daily."
}

The objective, in one screenful

Push the chosen answer's probability up and the rejected one's down, but anchor both to a frozen reference model so the tuned model cannot wander off. The strength of that anchor is a single knob, beta.

# DPO: no reward model. Push chosen above rejected, anchored to a
# frozen reference model by strength beta.
def dpo_loss(pi_chosen, pi_rejected, ref_chosen, ref_rejected, beta=0.1):
    chosen   = pi_chosen   - ref_chosen
    rejected = pi_rejected - ref_rejected
    return -F.logsigmoid(beta * (chosen - rejected)).mean()

When the judge is another model

Human labeling is slow and hard to scale. Constitutional AI swaps the human for the model itself. The model critiques and revises its own answers against a written constitution, and generates its own preference data. This RLAIF idea is core to how Claude is aligned, and part of a broad 2026 shift from human labels toward synthetic preference data.

Learning to think before answering.

The newest phase, and the one that reshaped the competitive landscape in 2026. Preference tuning makes a model pleasant. It does not make it solve a hard math problem. For that you need a different reward: reasoning RL drops human preference entirely and rewards only one thing, getting the answer right, checked by a machine. This is RLVR.

The eye-opener was DeepSeek-R1. Its R1-Zero variant started from DeepSeek-V3-Base and applied pure RL with no SFT first at all. With nothing but verifiable rewards, the model taught itself to produce long chain of thought, to check its own work, even to pause and reconsider. The paper calls it an aha moment. On AIME 2024, accuracy climbed from 15.6% to 71% over the course of RL.

What the data actually looks like

There are no human-written answers to imitate here. There is a problem, a known correct answer, and a rule for scoring.

{
  "problem": "What is the smallest n such that n! has more than 100 digits?",
  "gold_answer": "70",
  "reward": "1.0 if final answer == gold_answer else 0.0"
}

The objective, in one screenful

DeepSeek's method is GRPO. The clever part is what it removes. PPO needs a critic; GRPO throws it out. It samples a whole group of answers to the same prompt, grades each with the verifier, and uses the group's average score as the baseline.

# RLVR with GRPO: sample a GROUP of answers, grade each with a
# verifier (1/0), use the group mean as the baseline. No critic,
# no reward model.
group   = [model.generate(prompt) for _ in range(G)]   # e.g. G = 16
rewards = [verify(ans, gold) for ans in group]         # 1.0 or 0.0
baseline = sum(rewards) / G
advantages = [r - baseline for r in rewards]           # group-relative
loss = -sum(a * logprob(ans) for a, ans in zip(advantages, group)) / G
loss.backward()

Note how different this is from RLHF. There is no learned reward model to game, because the reward is ground truth. The flavor is closer to AlphaGo learning from the rules of the game by self-play than to a model learning human taste.

Same model. Four very different jobs.

One pre-training run on the left of the line, three post-training phases on the right. The line between row one and the rest is the whole pre-training vs post-training divide.

A few things that aren't true.

Stubborn misconceptions that make this picture harder to hold than it needs to be. Click each to see what's actually going on.

The lopsidedness is the whole story.

A frontier model is one enormous bet followed by a handful of cheap corrections. Pre-training spends tens of millions to compress the internet into a next-token predictor that knows a great deal but cannot answer you. Then a stack costing under 1% as much teaches it to respond, to prefer the better answer, and to think before it commits.

Capability is bought once, at great expense. Usability is layered on top, almost for free. Keep that asymmetry in your head and every headline about training cost, alignment, and reasoning models lands in the right place.

Every number traces to one of these. The 2026 trend notes (reasoning RL as differentiator, MoE at the frontier, distillation, synthetic preference data) are qualitative, not hard figures.