How a Frontier LLM Is Built.

One stage costs almost all the compute. Three cheap stages follow it.

A state-of-the-art LLM in 2026 is trained in four stages, one after another, like an assembly line. The most useful thing to know about that line is how lopsided it is. The first stage costs almost all the money. The three after it cost very little, yet they are what make the model feel smart. Step through them below.

Pre-training builds what the model can do. Post-training makes it usable.

Every part of LLM training falls into one of two buckets. Once you see that split, the pile of acronyms gets much easier to sort. Pre-training reads trillions of tokens and learns one thing: predict the next token. Post-training takes the parameters pre-training already set and nudges them, so the model answers your question instead of rambling on.

Learning to predict the next token.

This is the expensive one. Take a transformer with billions of parameters, point it at trillions of tokens, and ask it to do one thing over and over: given everything so far, guess what comes next. When it guesses wrong, a loss function pushes the weights to be a little less wrong. Repeat that across the internet, and the model picks up grammar, facts, code, arithmetic, and a usable model of the world as side effects.

How do labs decide how big the model should be and how much data to feed it? They use scaling laws. The Chinchilla paper fit a smooth curve that predicts the loss from the number of parameters (N) and the amount of data (D). It found that the compute-optimal recipe is roughly 20 tokens of data for every parameter.

What the data looks like

You might picture a carefully hand-built dataset. Mostly it is raw web pages. Open sets like FineWeb take raw Common Crawl, a public archive of scraped web pages, and clean it with deduplication and filtering. The result is roughly 15T tokens from 114 web snapshots. Each row is one document.

{
  "text": "The mitochondrion is a double-membrane-bound organelle found in most eukaryotic cells. It generates most of the cell's supply of ATP...",
  "id": "urn:uuid:d66bc6fe-8477-4adf-b430-f6a558ccc8ff",
  "dump": "CC-MAIN-2024-10",
  "url": "https://en.wikipedia.org/wiki/Mitochondrion",
  "language": "en", "language_score": 0.97, "token_count": 717
}

What the loss measures

At each position, the model gives a probability to every token in its vocabulary, tens of thousands of them, rather than picking a single one. The cross-entropy loss looks at only one of those numbers: the probability the model gave to the token that came next in the text.

The loss is the negative log of that one probability. Put a high probability on the right token and the loss is near zero. Put a low one and the model was surprised, so the loss is large. Pre-training is the job of making that number small, averaged over trillions of tokens.

The objective, in one screenful

The only training signal is cross-entropy on the next token. Raise e to the power of that loss and you get perplexity. Here is one training step, with everything else stripped away.

import torch.nn.functional as F

# One pretraining step: predict every next token at once.
logits  = model(tokens[:, :-1])     # (batch, seq-1, vocab)
targets = tokens[:, 1:]             # shift left by one
loss = F.cross_entropy(
    logits.flatten(0, 1),          # (batch*(seq-1), vocab)
    targets.flatten(),
)
loss.backward()                     # nudge billions of weights

Run that over trillions of tokens and the loss curve takes a typical shape. It drops fast at first, while the model learns the easy patterns of language. Then it flattens out, and each extra token buys a smaller improvement. Most of the money is spent on that long flat part.

A base model can't answer you

The model's architecture is also part of the recipe. Llama 3.1 405B is dense: every parameter is used for every token, a choice made for training stability. DeepSeek-V3 uses a Mixture of Experts. It has 671B parameters in total, but only 37B are activated for each token. Either way, the result is the same kind of model, and it behaves like this.

Teaching it to answer.

This is the first stage of post-training. We have a model that completes text, and we want one that follows instructions. The fix is surprisingly simple. Show it thousands of examples of the behavior we want. Supervised fine-tuning trains the base model on curated pairs of (prompt, ideal answer). It uses the same machinery and the same cross-entropy loss as pre-training. Only the data changes.

What the data looks like

An SFT example is a short conversation: an id, a list of messages with roles, and where it came from. Here is a row in the same format as the Tülu 3 mixture.

{
  "id": "oasst1_5921",
  "messages": [
    { "role": "user", "content": "Explain photosynthesis to a 10-year-old." },
    { "role": "assistant", "content": "Photosynthesis is how plants make their own food. They take in sunlight, water, and the air we breathe out, and turn it into sugar for energy and oxygen for us..." }
  ],
  "source": "ai2-adapt-dev/oasst1_converted"
}

The objective, in one screenful

SFT has one twist. You want the model graded only on writing the answer, not on predicting the user's question. So it uses the same cross-entropy as pre-training, with a mask that counts only the assistant's tokens.

# SFT: identical cross-entropy, but only assistant tokens count.
logits  = model(input_ids[:, :-1])
targets = input_ids[:, 1:]
tok_loss = F.cross_entropy(logits.flatten(0, 1), targets.flatten(),
                           reduction="none")
loss = (tok_loss * response_mask[:, 1:].flatten()).sum() / response_mask.sum()
loss.backward()

Aligning with what people prefer.

SFT teaches the model to answer. Preference tuning teaches it which answer is better. This is where tone, helpfulness, and safety get tuned, and it does more than any other stage to make a model feel polished. The classic recipe is RLHF, introduced at scale by InstructGPT.

The step where the model updates toward the reward is reinforcement learning. Classic RLHF uses PPO, which works well but is heavy to run. It adds a KL penalty as a brake.

The shortcut that took over: DPO

That is a lot of machinery. DPO showed you can skip most of it. With a bit of algebra, you can train directly on the preference pairs, with no separate reward model and no PPO loop. Each piece of preference data is a triple: a prompt, a chosen answer, and a rejected one.

{
  "prompt": "Write a haiku about coffee.",
  "chosen": "Bean falls, water hums, one quiet golden morning.",
  "rejected": "Coffee is a drink. It is brown. People enjoy it daily."
}

The objective, in one screenful

Push the chosen answer's probability up and the rejected one's down. Both are measured against a frozen reference model, which keeps the tuned model from wandering off. A single knob, beta, sets how strong that anchor is.

# DPO: no reward model. Push chosen above rejected, anchored to a
# frozen reference model by strength beta.
def dpo_loss(pi_chosen, pi_rejected, ref_chosen, ref_rejected, beta=0.1):
    chosen   = pi_chosen   - ref_chosen
    rejected = pi_rejected - ref_rejected
    return -F.logsigmoid(beta * (chosen - rejected)).mean()

When the judge is another model

Human labeling is slow and hard to scale. Constitutional AI replaces the human with the model itself. The model critiques and revises its own answers against a written set of principles, the constitution, and produces its own preference data. This RLAIF idea is central to how Claude is aligned. It is also part of a broad 2026 shift from human labels toward synthetic preference data.

Learning to think before answering.

This is the newest phase, and in 2026 it is where labs compete hardest. Preference tuning makes a model pleasant, but that doesn't help it solve a hard math problem. For that you need a different reward. Reasoning RL drops human preference and rewards only one thing: getting the answer right, as checked by a program. This is called RLVR.

The surprise came from DeepSeek-R1. Its R1-Zero version started from DeepSeek-V3-Base and went straight to RL, with no SFT first at all. With only verifiable rewards to go on, the model taught itself to write long chain of thought, to check its own work, and even to stop and reconsider. The paper calls this an aha moment. On AIME 2024, its accuracy rose from 15.6% to 71% over the course of RL training.

What the data looks like

Here there are no human-written answers to imitate. Each example is a problem, its known correct answer, and a rule for scoring.

{
  "problem": "What is the smallest n such that n! has more than 100 digits?",
  "gold_answer": "70",
  "reward": "1.0 if final answer == gold_answer else 0.0"
}

The objective, in one screenful

DeepSeek's method is called GRPO. It is a simpler PPO with one part taken out. PPO needs a critic, a second network that predicts what score to expect. GRPO drops it. Instead, it samples a group of answers to the same prompt, grades each one with the verifier, and uses the group's average score as the baseline. Answers that beat the average are made more likely. Answers below it are made less likely.

# RLVR with GRPO: sample a GROUP of answers, grade each with a
# verifier (1/0), use the group mean as the baseline. No critic,
# no reward model.
group   = [model.generate(prompt) for _ in range(G)]   # e.g. G = 16
rewards = [verify(ans, gold) for ans in group]         # 1.0 or 0.0
baseline = sum(rewards) / G
advantages = [r - baseline for r in rewards]           # group-relative
loss = -sum(a * logprob(ans) for a, ans in zip(advantages, group)) / G
loss.backward()

This is very different from RLHF. In RLHF, the model can learn to fool the reward model. Here there is no reward model to fool, because the reward is the true answer. It is more like AlphaGo improving through self-play, where the rules of the game decide who won, than like a model learning human taste.

The same model, trained four different ways.

The first row is the one pre-training run. The next three rows are the post-training phases. The break between row one and the rest is the pre-training vs post-training divide.

A few things that aren't true.

These common misconceptions make the picture harder to understand than it needs to be. Click each one to see what is true instead.

Pre-training is expensive. Everything after it is cheap.

A frontier model comes from one enormous, expensive run followed by a few cheap adjustments. Pre-training spends tens of millions of dollars to compress the internet into a next-token predictor. That predictor knows a great deal but cannot answer you. Then a few stages costing under 1% as much teach it to respond, to prefer the better answer, and to think before it commits to an answer.

Capability is paid for once, at great expense. Usability is added on top for almost nothing. Keep that imbalance in mind and you will know where each headline about training cost, alignment, or reasoning models fits.

Every number on this page comes from one of these sources. The 2026 trend notes (reasoning RL as what sets labs apart, MoE at the frontier, distillation, synthetic preference data) are general observations, not hard figures.