Post-training, step by step.

There is no single recipe. There is a menu, and every lab orders differently. Almost all of them start with supervised fine-tuning, most add a preference stage, and the ones that care about math and code finish with reinforcement learning against an automatic checker.

Where RLHF fits

You will see RLHF everywhere, including on this page. It stands for reinforcement learning from human feedback, and it names one specific recipe: the one behind the first ChatGPT. It has three parts. First comes SFT. Then people pick the better of two answers, and a reward model learns to score answers the way they would. Then comes reinforcement learning against that reward model. The model writes answers, the reward model scores them, and PPO pushes the model toward higher scores. A KL penalty keeps it close to the SFT model.

Most of this page covers what came after, and each newer method replaced one half of RLHF. DPO kept the human preferences and dropped the reinforcement learning. RLVR kept the reinforcement learning and dropped the human: a program checks the answer instead. None of this happens in pre-training, which has no human feedback and no reward, only the next-token loss. People also use RLHF loosely to mean everything after SFT. On this page it means the three-part recipe.

What it costs

DeepSeek-V3 published the split: 2.664 million H800-hours of pre-training, and roughly 5,000 hours of post-training. That is about two tenths of one percent. MiniMax-M1's full reinforcement-learning run took three weeks on 512 H800s and cost about $535,000. Its pre-training bill was tens of times larger.

Here is all of post-training, zoomed out far enough that you can see the stages. Each line is a script in allenai/open-instruct or huggingface/trl.

base = load("Qwen3-8B-Base")                       # from the pre-training page

sft   = train_sft(base, messages_dataset)          # learn the chat format and the habit
                                                   #   loss on assistant tokens only
pref  = train_dpo(sft, preference_pairs)           # learn a taste, from (chosen, rejected)
                                                   #   or: rm = train_reward_model(pairs)
final = train_grpo(pref, prompts, verifier)        # learn to be right, by being checked
                                                   #   sample 8 answers, keep what worked

release(final)                                     # + safety data, merging, evals

The reading list

Post-training has better open code than pre-training, because the runs are small enough for people to reproduce them.

The template is a Jinja string that ships inside the tokenizer. apply_chat_template runs a list of messages through it and gives you the exact text above. Every later stage is defined in terms of that text: which tokens the loss covers, what the prompt is during reinforcement learning, and where a tool result gets inserted.

The most important token is the one that ends an assistant turn: <|im_end|>, <|eot_id|>, <end_of_turn>. It is the stop token. The inference server watches for it and stops generating there. A model that never learned to produce it keeps talking until it hits the length limit, every time. This failure is common, and it usually has one of two causes: the end-of-turn token was left out of the trained span, or it was left out of the server's stop list.

Thinking blocks

Reasoning models add one more pair of markers. Qwen3 renders <think>…</think> before the answer, and its template has a quirk: assistant turns before the last user message have their thinking removed, and a kept turn with no thinking gets an empty block. Turning thinking "off" means pre-filling that empty block, so the model starts after it and goes straight to the answer. DeepSeek-R1 goes further and ends its generation prompt with <think>, so the model always starts by reasoning.

Rendering is one call. Note the two different shapes: the full transcript is the SFT target, and the prompt with add_generation_prompt=True is what reinforcement learning samples from.

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
msgs = [{"role": "system", "content": "You are a helpful assistant."},
        {"role": "user",   "content": "What is 2+2?"},
        {"role": "assistant", "content": "4."}]

text   = tok.apply_chat_template(msgs, tokenize=False)              # the SFT target
prompt = tok.apply_chat_template(msgs[:2], tokenize=False,          # what RL samples from
                                 add_generation_prompt=True)
# prompt ends with:  <|im_start|>assistant\n
# with enable_thinking=False it ends with: <|im_start|>assistant\n<think>\n\n</think>\n\n

A tool call and its result are more text in the same stream, with their own markers. This is how Qwen3 lays them out.

<|im_start|>assistant
<tool_call>
{"name": "get_weather", "arguments": {"city": "Paris"}}
</tool_call><|im_end|>
<|im_start|>user
<tool_response>
{"temp_c": 21, "sky": "clear"}
</tool_response><|im_end|>
<|im_start|>assistant
It is 21 C and clear in Paris.<|im_end|>

Prefix stability, and the bug it causes

To find the assistant's span, masking code renders messages[:i] with a generation prompt, renders messages[:i+1], and trains on the difference. That only works if the rendered prefix is exactly the start of the rendered whole. Qwen3's thinking-stripping and DeepSeek's <think> injection both break that rule. That is why TRL ships its own modified *_training.jinja templates, whose only job is to keep prefixes stable.

OLMo 3 hit this bug in practice. Their tokenization masked the first <think> token as if it were part of the prompt, and the workaround was to tokenize Think-mode data with the Instruct template. They now publish separate -dev and -release tokenizer repos, because the template used in training and the template shipped to users had drifted apart.

The tokens, for reference

What the loss mask does

Render the conversation, tokenize it, then set the label of every token that is not the assistant's to -100, the value PyTorch's cross-entropy ignores. The model reads the prompt as context but is never graded on predicting it. Skip this step and you train the model to predict user turns too, so it starts writing them itself.

To see what the mask changes, you first need the number it changes. Every later step on this page, DPO and GRPO included, is arithmetic on this same number.

One guess per position. At every position, the model reads everything before it. It then gives a probability to every token in its vocabulary, about 150,000 of them, and those probabilities add up to 1. Training looks at only one of them: the probability of the token that actually came next in the data. Call it p.

From probability to penalty. Training turns p into a penalty by taking −log p. If the model gave the right token probability 1, the penalty is 0. At p = 0.5 the penalty is 0.69. At p = 0.1 it is 2.30. As p gets close to 0, the penalty grows without limit, so the most expensive mistake is to be confident and wrong. This penalty is the cross-entropy loss for one token. Drag the slider to see how it changes.

The loss is an average. One training step makes a guess at every position, gives each guess a penalty, and averages the penalties into one number. Gradient descent then adjusts every weight in the direction that makes that number smaller. The mask decides which positions go into the average. Here is what the mask changes on a twelve-token example. The probabilities were chosen for the example. Every number after them was computed from them.

Why a log. Multiply the five graded probabilities and you get the probability that the model writes the whole answer, “It is 4.” plus its stop token. Each of those guesses already takes the earlier tokens into account: 0.40 × 0.85 × 0.92 × 0.78 × 0.96 = 0.2342. The log turns that product into a sum. −log 0.2342 = 1.4515, which is the five green bars added up. Divide by five and you get the loss, 0.2903. So lowering the loss means making the whole correct answer more likely.

loss      e^(-loss)
0.1000    0.9048     about 9 in 10
0.2903    0.7480     about 3 in 4        the five answer tokens above
0.5710    0.5650     a bit over 1 in 2   all twelve tokens above
0.6931    0.5000     1 in 2
0.7715    0.4623     a bit under 1 in 2  the seven prompt tokens above
2.3026    0.1000     1 in 10

Later steps use log p instead of −log p. It is the same number with the opposite sign. It is never above 0, and a value closer to 0 means more likely. When the DPO step says an answer has a log-prob of −12.40, that is these per-token numbers added up over the whole answer.

Where the data comes from

In 2023 it was written by hand. In 2026 it is almost entirely synthetic. Prompts are generated from personas, answers are generated by a stronger model, and the results are filtered and decontaminated. OpenThoughts3 is 75,000 questions answered 16 times each by QwQ-32B, for 1.2 million rows. DeepSeek-R1's second supervised stage used 600,000 reasoning samples generated by its own RL-trained checkpoint, keeping only the correct ones (rejection sampling).

OLMo 3's preference data shows how strange this can get. Their best pairs used a strong model's answer as the chosen one and a tiny model's answer as the rejected one, instead of a good and a bad answer from the same model. Training on the strong answers alone made things worse. Pairing each with a weak answer worked. What the model learns from is the difference between the two answers, not either answer alone.

nanochat builds the mask while it renders, which makes the idea easy to see: each token is added with a flag that says whether it counts toward the loss.

# nanochat/tokenizer.py  ::  render_conversation
add_tokens(bos, 0)
# user turn: context, never a target
add_tokens(user_start, 0); add_tokens(value_ids, 0); add_tokens(user_end, 0)
# assistant turn: this is what we grade
add_tokens(assistant_start, 0)
add_tokens(value_ids, 1)                                  # the text
add_tokens(python_start, 1); add_tokens(value_ids, 1); add_tokens(python_end, 1)   # a tool CALL is ours
add_tokens(output_start, 0); add_tokens(value_ids, 0); add_tokens(output_end, 0)   # its OUTPUT is not
add_tokens(assistant_end, 1)                              # learn to stop

# scripts/chat_sft.py
targets = batch[:, 1:]
targets[mask[:, 1:] == 0] = -1                            # -1 is nanochat's ignore_index

TRL does the same thing through the template, which is another reason it ships its own training templates. The part that should be graded is wrapped in {% generation %}, and apply_chat_template returns a mask for it.

# trl/chat_templates/qwen3_training.jinja
{{- '<|im_start|>' + message.role + '\n' }}
{%- generation %}
{{- content }}{{- '<|im_end|>\n' }}            # note: the end token is INSIDE the block
{%- endgeneration %}

# SFTConfig(assistant_only_loss=True) then does:
labels = pad(labels, padding_value=-100)

Packing, and the cost of getting it wrong

Conversations range from 50 to 30,000 tokens, so they are packed into fixed-length rows the same way as in pre-training, with the same two choices: crop or best-fit, and mask across document boundaries or not. TRL's "bfd" (best-fit-decreasing) packing plus position_ids is the current default. nanochat pads with BOS instead of cropping, because at SFT scale it is better to waste a token than to cut an answer off mid-sentence.

What labs ran

Where it goes wrong

• Training on the prompt. This is the worked example above: the loss roughly doubles and the model learns to write user turns. Always check the mask by printing one rendered example with the ignored tokens grayed out. nanochat has visualize_tokenization for this.
• The end-of-turn token outside the span. The model never learns to stop.
• Template drift. Training with one template and serving with another makes the model worse across the board, and nothing in the stack will warn you. OLMo 3 documents shipping this bug.
• Contamination. If the model that generated your synthetic SFT data saw the benchmarks, it can leak them into your data. Every serious dataset now removes anything that overlaps the eval suites, by n-gram matching.
• Catastrophic forgetting. Two epochs on a narrow dataset will make the model worse at everything else. The defense is a broad mixture, which is why Tulu 3 publishes a source field on every row.

A verifier is a piece of code. It pulls out the boxed answer and compares it to the correct one, or runs the unit tests, or checks that the output is valid JSON with the required keys. It is free and instant, and no clever wording can talk it into a higher score. It only exists where correctness can be checked mechanically, which is why math and code drove the 2025 wave of reasoning models.

A reward model is a language model whose final layer is replaced by one that outputs a single score. You train it on preference pairs, and then it can score any answer. The training records look much the same everywhere.

{
  "domain": "code",
  "context": [{"role": "user", "content": "Write a JavaScript program that ..."}],
  "response1": "Here's a JavaScript program that meets your requirements. ...",
  "response2": "```javascript\nconst readline = require('readline'); ...",
  "overall_preference": -2,
  "individual_preference": [
    {"score": -1, "reasoning": "@Response 1 is slightly better than @Response 2. ..."},
    {"score": -3, "reasoning": "@Response 1 is much better because ..."},
    {"score": -2, "reasoning": "@Response 1 is better because ..."}
  ]
}

Bradley-Terry, and why only the gap matters

The reward model is trained only to score the better answer higher. The loss is -log σ(r_chosen − r_rejected). There is no target for the scores themselves. Only the margin between them appears, so the scale can drift anywhere. Add 10 to every score and the loss does not change.

pair   r(chosen)  r(rejected)   margin   sigma(margin)   -log sigma
A        +2.10      +0.40      +1.70       0.8455       0.1678
B        +1.25      +0.90      +0.35       0.5866       0.5334
C        +0.80      -0.60      +1.40       0.8022       0.2204
D        +1.40      +0.55      +0.85       0.7006       0.3559
                                                total   1.2775

now score pair D backwards (margin -0.85):
D        +0.55      +1.40      -0.85       0.2994       1.2059
                                                total   2.1275

the increase is exactly 0.8500, the size of the margin it got wrong.

The third source is another language model with a rubric. Kimi K2 has the model critique its own rollouts against a checklist that includes strict rules like "responses must not begin with compliments directed at the user" and "no sentence that explains why the response is good". Those rules exist because the model had learned to write both of those things to please the judge.

The reward model is five lines on top of a sequence classifier.

# trl/trainer/reward_trainer.py
rewards_chosen, rewards_rejected = torch.chunk(outputs.logits.squeeze(-1), chunks=2)
loss = -nn.functional.logsigmoid(rewards_chosen - rewards_rejected).mean()
if self.args.center_rewards_coefficient is not None:              # recommended 1e-2
    loss += self.args.center_rewards_coefficient * torch.mean((rewards_chosen + rewards_rejected) ** 2)

That second term is there because the loss only constrains differences. Without it the scores can drift to any value, and a score from one batch cannot be compared with a score from another.

A verifier is ordinary code. In open-instruct the interface is a function, plus a column in the dataset that says which function to call.

# open_instruct/ground_truth_utils.py
class VerifierFunction:
    def __call__(self, tokenized_prediction, prediction, label, query, rollout_state) -> VerificationResult:
        ...                                                   # returns score, cost, reasoning

# the registry: GSM8KVerifier, MathVerifier, StrictMathVerifier, IFEvalVerifier,
#               CodeVerifier (posts to a sandbox), LMJudgeVerifier, RubricVerifier
reward_fn = reward_fn_mapping[row["dataset"].lower()]         # dispatch on the dataset column

{
  "data_source": "openai/gsm8k",
  "prompt": [{"role": "user", "content": "Natalia sold clips to 48 friends ... Let's think step by step and output the final answer after \"####\"."}],
  "ability": "math",
  "reward_model": {"style": "rule", "ground_truth": "72"},
  "extra_info": {"split": "train", "index": 0}
}

Reward hacking, with real examples

Give a good enough optimizer any reward you can measure, and it will find a way to score well without doing what you wanted. This has happened in practice. Anthropic published three hacks found in the production reinforcement-learning environments used for Claude 3.7 Sonnet. Each one makes a test suite pass without fixing anything.

import sys
sys.exit(0)                    # exit code 0 means "tests passed". The asserts never run.

class AlwaysEqual:             # returned from the function under test
    def __eq__(self, other): return True          # every assertEqual now passes

# conftest.py, which pytest imports automatically
def pytest_runtest_makereport(item, call):
    report.outcome = "passed"  # monkey-patch the reporter itself

The uncomfortable part of that paper is how far the behavior spread: models that learned these tricks became more misaligned on unrelated evaluations. The reward taught them a general strategy, cheating the grader, instead of the task.

Other hacks, and what they cost

• Length. GRPO's per-sequence normalization ends up rewarding long wrong answers. Dr. GRPO removes the normalization. DAPO adds an explicit penalty past a soft length limit. Kimi gives each task a token budget.
• Language mixing. DeepSeek-R1-Zero's chains of thought drifted between languages because nothing scored readability. R1 added a language-consistency reward, and reports that it slightly hurts the benchmarks. They shipped it anyway.
• Judge flattery. Models learn to open with a compliment and to explain why their answer is good, because judges reward it. That is why Kimi's rules ban both.
• Format gaming. A verifier that accepts any \boxed{} will get answers with a hundred boxes. Strict extraction and a penalty for never producing the stop token are now standard.
• Process rewards. DeepSeek tried scoring each reasoning step and gave up. They wrote that "the neural reward model may suffer from reward hacking", and they found no good way to define what a step is. They also tried tree search, and it did not work either.

The margin asks a specific question: has the model moved further toward the chosen answer than toward the rejected one, compared with the frozen reference? It does not ask whether the model prefers the chosen answer outright. The reference stops the model from lowering the loss by collapsing onto short, safe text, because every gain is measured from where it started.

β is the only setting. It controls how sharply the loss separates a pair the model already has right from a pair it has backwards. Three examples show the difference, with every number computed.

                       log-ratio   log-ratio
example                 chosen      rejected     margin
A  already right        +0.80       -0.60        +1.40
B  near tie             -0.10       +0.10        -0.20
C  backwards            -1.20       +1.40        -2.60

beta = 0.1                              beta = 0.5
example      loss    |gradient|         loss    |gradient|
A            0.6256    0.0465           0.4032    0.1659
B            0.7032    0.0505           0.7444    0.2625
C            0.8316    0.0565           1.5410    0.3929

at beta 0.1 the three losses sit within 0.21 of each other, and of
log 2 = 0.6931: the badly-wrong example gets almost the same gradient
as the already-correct one.
at beta 0.5 the backwards example gets 8.4x the gradient of the
already-right one.

The loss is eight lines. This version is worth memorizing, because every variant is a one-line change to it.

# the four sequence log-probs, each a sum over the completion's tokens
policy_chosen, policy_rejected = model_logps(batch)
with torch.no_grad():
    ref_chosen, ref_rejected = ref_model_logps(batch)         # frozen

chosen_logratios   = policy_chosen   - ref_chosen
rejected_logratios = policy_rejected - ref_rejected
margin = chosen_logratios - rejected_logratios
loss = -F.logsigmoid(beta * margin).mean()                    # loss_type="sigmoid"

# what the model implicitly thinks each answer is worth, for logging:
chosen_rewards = (beta * chosen_logratios).detach()

Each variant differs by one line. ipo replaces the logsigmoid with a squared error against a target margin. hinge uses a hinge loss. sigmoid_norm divides the margin by the completion length. That is SimPO's idea, and Tulu 3 found it was the only variant that beat plain SFT. KTO drops pairs and works from single answers labeled good or bad. ORPO combines the SFT loss and the preference loss into one stage, with no reference model at all.

The reference model, in practice

There are three ways to provide it, listed by how much memory they cost. Keep a frozen second copy (simple, but it doubles the weights in memory). Precompute the reference log-probs for the whole dataset once, then drop the model (precompute_ref_log_probs=True, the usual choice). Or, with LoRA, turn the adapter off for one forward pass, because the base weights are the reference.

When to use DPO, and when not to

One more finding worth remembering: Llama 3 ran six rounds of reward modeling, rejection-sampling SFT and DPO, and averaged checkpoints trained with different data and hyperparameters at each stage. At that scale, post-training is less a pipeline than a loop you repeat until the evals stop improving.

The advantage, computed

The baseline needs no network. Take the group's rewards, subtract the mean, and divide by the standard deviation.

group of 8, rewards [1, 0, 0, 1, 0, 1, 0, 0]      3 correct

mean                          0.3750
std  (population)             0.4841

advantage of a correct answer   (1 - 0.3750) / 0.4841  =  +1.2910
advantage of a wrong answer     (0 - 0.3750) / 0.4841  =  -0.7746

Dr. GRPO, which skips the division by std:
                                 +0.6250   and   -0.3750

What the advantage is for

The advantage says how much better an answer did than its group. The loss uses it in a simple way. It takes the SFT loss from step 02, −log p averaged over an answer’s tokens, and multiplies it by the advantage. The only other change is where the answers come from: the model wrote them, not a person.

So this kind of reinforcement learning is supervised fine-tuning on the model’s own answers, with each answer weighted by how much it beat its group. A positive advantage makes the answer more likely, as SFT would. A negative one makes it less likely. An advantage of 0 does nothing at all, and that case turns out to be common.

The group that teaches nothing

Now set every answer to correct. The mean is 1, the standard deviation is 0, and every advantage is 0. The same happens when every answer is wrong. A group where every answer gets the same score produces no gradient at all, and you still paid full price to generate it.

This happens a lot, and it becomes the main waste as training succeeds. The easy prompts get solved every time and the impossible ones never do, so a growing share of every batch teaches nothing. DAPO's fix is dynamic sampling: generate more groups than you need, throw away the ones where every answer got the same score, and keep sampling until the batch is full of groups with mixed results. OLMo 3 calls its version active sampling and reports that it is about four times faster.

Why this works at all

It works because the ability is usually already in the model, but not reliably. A model that is right 30 percent of the time on a problem gets it right at least once in eight tries 94 percent of the time.

p = 0.30 per attempt

k        pass@k = 1 - (1-p)^k
1          0.3000
4          0.7599
8          0.9424
16         0.9967

nanochat's version is the shortest one that hides nothing. It has no KL penalty and no clipping, it is on-policy, and its docstring lists the four ways it simplifies the method in the paper.

# scripts/chat_rl.py
mu = rewards.mean()
advantages = rewards - mu                                   # mean-only baseline

logp = -model(inputs, targets, loss_reduction='none').view_as(inputs)   # (B, T)
pg_obj = (logp * advantages.unsqueeze(-1)).sum()
pg_obj = pg_obj / (num_valid * num_passes * examples_per_rank)
loss = -pg_obj
# num_samples=16, temperature 1.0, max_new_tokens=256, reward = GSM8K exact match

Look at the line that computes logp. model(inputs, targets, loss_reduction='none') is the SFT loss, token by token, run on answers the model sampled. The next line multiplies it by the advantage. In nanochat, that is the whole algorithm.

The production version adds the importance ratio and the clip, because the answers were sampled by a slightly older policy than the one now being updated.

# verl/trainer/ppo/core_algos.py
ratio = torch.exp(torch.clamp(log_prob - old_log_prob, -20, 20))
pg_losses1 = -advantages * ratio
pg_losses2 = -advantages * torch.clamp(ratio, 1 - cliprange_low, 1 + cliprange_high)
clip_pg_losses1 = torch.maximum(pg_losses1, pg_losses2)        # = -min(A*r, A*clip)
pg_losses3 = -advantages * clip_ratio_c                        # dual-clip, default 3.0
clip_pg_losses2 = torch.min(pg_losses3, clip_pg_losses1)
pg_losses = torch.where(advantages < 0, clip_pg_losses2, clip_pg_losses1)
pg_loss = agg_loss(pg_losses, response_mask, loss_agg_mode)    # "token-mean" by default

And the advantage, grouped by prompt:

# compute_grpo_outcome_advantage, in essence
for uid, group in group_by_prompt(scores):
    m, s = group.mean(), group.std()
    scores[group.idx] = (group - m) / (s + 1e-6)        # or just (group - m) for Dr. GRPO
advantages = scores.unsqueeze(-1).expand_as(response_mask)   # every token of an answer
                                                            # carries the answer's advantage

What the clip is for

The answers in a group were sampled by the policy as it was at the start of the step. After one update the policy is different, so those answers describe it a little less well. The importance ratio corrects for that. The clip limits how far the correction is allowed to push.

The defaults, and what they say about the field

What the KL penalty measures

KL measures how far the trained model has moved from where it started. Take an answer the model just wrote. For each token, compare log p under the model now with log p under a frozen copy of the original. Then average the differences.

token     log p now   log p at the start   difference
"Wait"      -0.10          -0.30             +0.20
","         -0.50          -0.60             +0.10
" let"      -1.20          -2.00             +0.80
                                  average    +0.3667    the KL estimate

The most surprising line there is beta 0.0. The KL penalty to the reference model was the centerpiece of the original RLHF recipe. It is now off by default in TRL, and off in DAPO, OLMo 3 and MiniMax. With a verifiable reward there is nothing to hack, so there is less need to hold the model close to where it started, and holding it close costs capability.

Clip-higher: the ceiling costs rare tokens the most

A single clip range, 1 ± ε, sounds symmetric, but its effect on probabilities is not. A token already at 0.95 can barely be raised anyway. A token at 0.10 is the kind that starts an unusual line of reasoning, and the ceiling is what keeps it from growing enough to get tried.

epsilon 0.2, a good answer (A = +1):

token          p_old     ratio   clipped   objective
t1 common      0.8187   1.1052   1.1052   +1.1052
t2 common      0.6065   0.7408   0.8000   +0.7408
t3 RARE        0.1003   1.4918   1.2000   +1.2000   <- clipped
t4 v.common    0.9512   1.0305   1.0305   +1.0305
t5 rare        0.0498   0.6065   0.8000   +0.6065

what the ceiling means in absolute probability:

token          p_old    max p_new at 1.2   at 1.28 (DAPO)
t1 common      0.8187      0.9825           ceiling not binding
t3 RARE        0.1003      0.1203           0.1283
t5 rare        0.0498      0.0597           0.0637

raising the ceiling to 1.28 changes exactly one token here: t3.
t1, t2, t4 and t5 are byte-identical to the run at 0.2.

Dynamic sampling and token-level loss

Dynamic sampling fixes the degenerate-group problem from the last step. Generate more prompts than you need, discard the groups where every answer got the same score, and keep going until the batch is full of groups with mixed results. DAPO generates 1,536 prompts to train on 512.

Token-level loss is subtler. If you average the loss per sequence, every answer counts the same. A 10,000-token wrong answer is then penalized as gently as a 50-token one, which works out to almost nothing per token. If you average over all the tokens in the batch instead, long answers count in proportion to their length. In DAPO's ablation, removing it cost six points on AIME24, more than removing dynamic sampling.

Dr. GRPO: two normalizations that turned out to be biases

Dividing by the group's standard deviation gives every group an equally large update, including the ones where seven of eight answers were right. Dividing by the answer's length makes long answers cheap. Between them they push toward longer answers, and toward treating easy prompts as seriously as hard ones. Dr. GRPO removes both. The same paper showed that Qwen base models already produce reasoning-style text, so some of the famous "aha moment" came from a template mismatch rather than a behavior that emerged during training.

GSPO: one ratio for the whole answer

The reward was earned by the answer as a whole, so Qwen argues the importance weight should belong to the answer as a whole too. Instead of one ratio per token, GSPO takes the geometric mean over the answer: the exponential of the average log-ratio.

token        logp_old   logp_new    delta    token ratio
tok1          -0.10      -0.12     -0.02      0.9802
tok2          -0.20      -0.18     +0.02      1.0202
tok3          -0.05      -0.07     -0.02      0.9802
tok4 NOISY    -2.50      -1.60     +0.90      2.4596   <- clipped at 1.2
tok5          -0.15      -0.16     -0.01      0.9900

sum of deltas                         0.8700
sequence ratio  exp(0.87 / 5)         1.1901   <- inside the clip range
without length normalisation exp(0.87) 2.3869   <- would be clipped just as hard

This matters most for a mixture of experts. The same token can be sent to different experts in the rollout engine and in the trainer, which makes its per-token ratio close to meaningless. Qwen reports that GSPO removed the need for "routing replay", a workaround that existed only to keep token ratios comparable, and that GSPO contributed to the Qwen3 models.

# verl · compute_policy_loss_gspo
negative_approx_kl_seq = torch.sum(negative_approx_kl * response_mask, -1) / seq_lengths
log_seq_importance_ratio = log_prob - log_prob.detach() + negative_approx_kl_seq.detach().unsqueeze(-1)
seq_importance_ratio = torch.exp(torch.clamp(log_seq_importance_ratio, max=10.0))
pg_losses = torch.maximum(-advantages * seq_importance_ratio,
                          -advantages * torch.clamp(seq_importance_ratio, 1 - clip_low, 1 + clip_high))
# TRL: importance_sampling_level="sequence"

CISPO: clip the weight, keep the gradient

MiniMax noticed which tokens the clip was removing from the update: the ones where the reasoning changes direction, like "Wait", "However" and "Let me recheck". Those tokens are rare, so their ratios move a lot, so they get clipped, so the model never learns to use them. CISPO clips the importance weight but leaves every token's gradient in place. It matched DAPO in half the steps.

# instead of dropping the token from the loss, down-weight it and keep it
pg_losses = -clipped_ratio.detach() * advantages * log_prob
#            ^ the weight is clipped and detached; log_prob still carries the gradient

What to turn on

As of 2026, labs largely agree on how to configure a verifiable-reward run, and most of the settings come from these fixes.

Four networks run at once. The actor is the model being trained. The reward model scores the finished answer. The reference model is a frozen copy of the starting point. The per-token KL between actor and reference is subtracted from the reward, which stops the actor from drifting into gibberish that happens to score well. The critic, a fourth transformer, is trained to predict, from any point in an answer, how good the rest of the answer will turn out.

The critic exists because of a hard problem. The reward arrives once, at the end, but the credit has to be shared among hundreds of tokens. GAE is how PPO shares it. At each step, compute the surprise: how much better or worse things look than the critic expected one step earlier. This is called the temporal-difference error. Then each token's advantage is its own error plus a discounted share of all the errors after it.

gamma = 1.0, lambda = 0.95, reward only at the end

t     r     V(s_t)   V(s_t+1)   delta = r + V' - V      A_t
0    0.00    0.50      0.55          +0.0500         +0.436389
1    0.00    0.55      0.60          +0.0500         +0.406725
2    0.00    0.60      0.70          +0.1000         +0.375500
3    0.00    0.70      0.80          +0.1000         +0.290000
4    1.00    0.80      0.00          +0.2000         +0.200000

read the last two rows:
   A_3 = 0.10 + 0.95 * 0.20 = 0.29
   A_4 = 0.20, with nothing after it to add

# openrlhf/models/utils.py :: compute_reward
kl_reward = -kl_coef * kl                                     # per token, every token
eos_indices = action_mask.size(1) - 1 - action_mask.long().fliplr().argmax(dim=1, keepdim=True)
last_reward = torch.zeros_like(kl).scatter_(dim=1, index=eos_indices, src=r.unsqueeze(1))
reward = last_reward + kl_reward                              # one scalar at the end, KL everywhere

# get_advantages_and_returns  (GAE)
delta = rewards[:, t] + gamma * nextvalues - values[:, t]
lastgaelam = delta + gamma * lambd * lastgaelam

# models/loss.py
ratio = (log_probs - old_log_probs).clamp(-20, 20).exp()
loss = -torch.min(ratio * advantages, ratio.clamp(1 - eps_low, 1 + eps_high) * advantages)
values_clipped = old_values + (values - old_values).clamp(-clip_eps, clip_eps)
vloss = 0.5 * torch.max((values_clipped - returns) ** 2, (values - returns) ** 2)

verl's PPO step runs in this order, and reading it is the fastest way to see how many models are involved: generate_sequences → compute_log_prob → compute_ref_log_prob → compute_values → compute reward → compute_advantage → update_critic → update_actor.

Why almost everyone switched

• The critic is another full model. It is usually the same size as the policy, trained at the same time, and the hardest of the four to tune. DeepSeek-R1 describes GRPO as "foregoing the critic model that is typically the same size as the policy".
• With a 0/1 reward there is little for it to learn. A value function is worth its cost when rewards are dense and shaped. When the only signal is "the final answer was right", a group average is a perfectly good baseline and needs no training.
• Memory. Four transformer-sized networks sit in memory at once, and two of them carry optimizer state.
• It did not go away. Tulu 3's RLVR was PPO as late as Nov 2024, and OpenRLHF, verl and NeMo-RL all still ship it. Where the reward is dense or the horizon is long, the critic is still the right tool.

The split is the same everywhere: vLLM or SGLang generates, and FSDP or Megatron trains. verl calls this arrangement HybridFlow, with one controller process driving both. The two can share GPUs (colocated: every GPU is used, but the two phases take turns) or sit on separate GPUs (disaggregated: both stay busy, but you need more hardware).

After every step the new weights have to reach the engine. This sounds easy and turns out to be hard. Moonshot's checkpoint-engine moves a full one-trillion-parameter Kimi K2 update across thousands of GPUs in about twenty seconds, using a three-stage pipeline: copy from host to device, broadcast into CUDA IPC buffers, then have each engine reload its own shard. Qwen3-235B on eight H800s takes 6.2 seconds. When the trainer and engine share a machine, slime passes CUDA IPC handles instead. A handle is a reference to memory already on the GPU, so only the handle crosses the process boundary and the weights themselves do not move at all.

Going asynchronous

Synchronous RL wastes half the cluster. While the engine generates, the trainer sits idle, and then they swap. One-step-off-policy generation overlaps the two: the engine produces batch k+1 using the weights from step k while the trainer updates on batch k. verl measures this as 23 percent faster with FSDP2 and 40 percent faster with Megatron, with no loss in accuracy. Fully asynchronous designs go further. They add a separate rollout process (a rollouter), a message queue and a limit on how stale a rollout may be, and get 2.3 to 2.8 times the throughput.

But once generation and training are off by even one step, the answers you train on were produced by weights that no longer exist. The importance ratio from step 05 corrects for this. Asynchrony is what makes those ratios necessary rather than a formality.

Same weights, different probabilities

This one surprises people. Even with identical weights, the rollout engine and the trainer report different probabilities for the same token, because they use different kernels, different precision and different batch composition. Yao et al. found tokens where vLLM said probability 1 and FSDP said 0. If you do not correct for this, reward growth stalls or collapses.

token           logp rollout   logp trainer    diff     ratio exp(diff)
t1                 -0.500        -0.520      -0.020       0.9802
t2                 -1.200        -1.150      +0.050       1.0513
t3                 -0.050        -0.080      -0.030       0.9704
t4 MISMATCH        -3.500        -2.600      +0.900       2.4596
t5                 -0.900        -1.100      -0.200       0.8187

sequence weight, raw product                              2.0138
sequence weight, each token truncated at 2.0              1.6375
no correction at all                                      1.0000

Here are the fixes, from simplest to most thorough. Truncated importance sampling multiplies the loss by min(π_train/π_rollout, C) with C around 2. verl, TRL and open-instruct all do this. Masking drops tokens whose ratio falls outside a window, instead of clipping them. MiniMax traced their mismatch to the output head and ran it in FP32, which raised the correlation between the two engines from 0.987 to 0.997 and let training make progress again. Thinking Machines went to the root cause: the kernels are not batch-invariant. With batch-invariant implementations the sampler and trainer become bitwise identical, the KL between them is exactly zero, and the run trains fine with no correction.

# prime-rl · the most explicit version: mask the disagreeing tokens and penalise disagreement
abs_probs_diff = torch.abs(torch.exp(trainer_logprobs) - torch.exp(inference_logprobs))
keep_mask = loss_mask & ~(abs_probs_diff > loss_config.eps)          # eps = 0.1
pg_loss = keep_mask * loss_config.adv_tau * advantages * importance_ratio
kl_loss = loss_mask * log_importance_ratio ** 2
per_token_loss = -pg_loss + loss_config.kl_tau * kl_loss

When the rollout uses tools

A multi-turn rollout interleaves text the model wrote with text a tool returned. Only the first kind should be trained on. This is the same loss-mask idea from step 02, now applied during generation. verl's agent loop returns a response_mask that is 1 for model tokens and 0 for tool output.

The documentation stresses one rule above the others: token in, token out. Never turn the conversation back into a string and re-tokenize it between turns. Tokenizing pieces separately does not always give the same tokens as tokenizing the joined text, so a re-tokenized transcript can differ from the one the model produced. That brings back the mismatch above, this time through your own code.

Budgets, and what they cost

Safety is mostly more data

Safety uses the same SFT and preference machinery as everything else, applied to refusal data. There is no separate safety algorithm. Tulu 3's mix is about 12 percent safety: 10,983 examples of when not to comply, 50,000 adversarial jailbreaks, and 50,000 prompt and response harm labels. Safety pairs also go into the preference mix. In RL, a judge penalizes both harmful compliance and over-refusal, because a model that refuses everything would score perfectly on the first and be useless.

Constitutional AI is the same loop, with the model generating its own labels. The model critiques and revises its answers against written principles, and you fine-tune on the revisions. Then a model picks the better of two answers according to a principle, and you train a preference model on those choices. The 2025 descendant of that idea is the rubric reward from step 03.

Merging, which sounds like cheating

Averaging the weights of several fine-tunes of the same base model works, and everyone does it. Llama 3 averaged checkpoints trained with different data and hyperparameters at every stage. SmolLM3 merged 0.9 of its preference-tuned "soup" (itself an average of several models) with 0.1 of a mid-training checkpoint, to recover the long-context ability that post-training had worn down. OLMo 3 32B averaged two midtraining runs and its last three long-context checkpoints.

There is a simple reason this works. The models being averaged are small changes to the same starting point, so their weights sit in the same basin, a low region of the loss. The average of two points in a basin is still in the basin, and often lower than either one.

What gets measured

The number that moves most visibly during reasoning RL is AIME. DeepSeek-R1-Zero went from 15.6 to 71.0 pass@1, and to 86.7 with majority voting over 16 samples. Qwen3's 235B went from 70.1 to 85.1 in 170 RL steps. Those are the numbers that made 2025 look like a step change.

What gets released

Labs release a Base model, then an Instruct model, then a Thinking variant, and more and more often all three come from one family. Qwen3 first shipped a single hybrid model with a /think switch and later split it into separate Instruct and Thinking releases, because the two modes needed different training. OLMo 3 publishes Base, Instruct, Think and RL-Zero, plus every intermediate SFT and DPO checkpoint. It is the only place where you can see every stage on this page, one checkpoint at a time.

One last operational warning

open-instruct's OLMo 3 documentation has a warning worth ending on. The chat template used in training and the one shipped in the release had drifted apart: the system prompt was different, and <think> was handled differently in the generation prompt. The team now maintains separate -dev and -release tokenizer repositories. Every stage on this page works on a rendered string, so the template is the contract that all of them depend on, and it is the part most likely to be wrong without anyone noticing.