It's the same model. It just thinks first.

A reasoning model spends tokens before it answers.

A plain model maps your question almost straight to an answer. If the answer needs five steps of work, the model has to do all five in a single forward pass, with no scratch paper. On anything hard, it guesses.

A reasoning model is trained to first generate a long chain of thought — sometimes thousands of "thinking" tokens — and only then write the answer. The thinking tokens are just more next-token prediction. But because each one is fed back in, the model gets to read its own work, catch mistakes, and try another route. The scratch paper is the whole point.

OpenAI's o-series, DeepSeek-R1, Claude's extended thinking, Gemini's thinking modes, and Qwen's QwQ are all this idea. The model that "reasons" is the model that was taught to use a long scratch pad well.

The architecture is unchanged. The training is not.

This is the part people get wrong. A reasoning model is, in almost every case, the exact same decoder-only transformer as the base model. No new layer type, no extra module bolted on for "reasoning." You could not tell a reasoning model from a chat model by inspecting the network diagram.

What changed is how the weights were tuned. After normal pretraining, the model is pushed through reinforcement learning with verifiable rewards: give it math and code problems where the answer can be checked by a script, reward the chains of thought that reach the right answer, and let the model discover for itself which kinds of thinking pay off.

DeepSeek showed this starkly with R1-Zero: using an RL recipe called GRPO, a base model with no supervised reasoning examples at all gradually taught itself to write longer and longer chains, pause, and re-check its work — the much-quoted "aha moment." Nobody wrote those reasoning traces. The reward did.

# the model is unchanged: same transformer, same forward pass.
# RL just reshapes which chains-of-thought it tends to produce.
for question, checker in verifiable_problems:
    group = [model.generate(question) for _ in range(G)]   # sample G long answers
    rewards = [1.0 if checker(a.final) else 0.0 for a in group]
    baseline = mean(rewards)                            # the "group relative" part
    for a, r in zip(group, rewards):
        reinforce(a.tokens, advantage=r - baseline)     # upvote chains that worked
    # over time: longer reasoning, self-checking, backtracking — all emergent

The detailed training pipeline — pretraining, SFT, preference tuning, then this reasoning-RL phase — gets its own teardown in the LLM training explainer. Here we care about what it costs to run.

Two things are called "the decoder."

The word is overloaded, and the confusion is worth clearing up because both meanings matter for reasoning models.

The decoder stack is the architecture: a tower of identical transformer layers, each running self-attention under a causal mask and then a feed-forward network. "Decoder-only" is the GPT lineage: no encoder, just this stack predicting the next token. The decoding step is the other meaning: taking the model's output probabilities and choosing a token — greedily, or with temperature and top-p sampling.

The expensive shortcut that makes this loop affordable is the KV cache. Without it, every new token would force the model to re-process the entire sequence from the start. With it, the model only computes attention for the one new token and reuses the cached keys and values for everything before it. The cache is what makes long reasoning traces tractable — and, as we will see, what eats the memory.

Inference has two halves, and they stress the chip differently.

Running the model splits cleanly into prefill and decode. Prefill reads your prompt all at once: every prompt token can be processed in parallel, so the chip is doing huge dense matrix multiplies and is compute-bound. Decode is the opposite: one token at a time, each requiring a full sweep over the model's weights, so the chip is memory-bandwidth-bound.

Here is the punchline for reasoning models. Prefill happens once. Decode happens once per output token. A chat reply might decode a few dozen tokens. A reasoning trace decodes thousands. So a reasoning model spends almost its entire life in the memory-bound decode phase, streaming the full weight set out of memory thousands of times for a single answer. The number that matters is no longer FLOPs. It is memory bandwidth and how long the loop runs.

Why reasoning models love fast memory.

In the decode phase, generating one token means reading every weight the active path uses, once. So the ceiling on tokens-per-second is roughly memory bandwidth ÷ model size. Not how many FLOPs the chip can do — how fast it can feed them. This single fact explains the entire inference-hardware landscape.

Figures are peak per-device memory bandwidth from vendor specs; real throughput depends on model size, batch, and quantization. SRAM chips trade tiny capacity for absurd bandwidth.

Two strategies fall out of this. The HBM path (NVIDIA H100/H200/Blackwell and now Rubin, Google TPUs, AWS Trainium) keeps the model in large, very fast off-chip memory and wins on capacity and batching. The SRAM path (Groq's LPU, Cerebras's wafer-scale engine) keeps the weights in on-chip SRAM and posts the lowest per-token latency in the industry — which is exactly what a model emitting thousands of sequential thinking tokens wants. When the answer is gated by a long serial chain, time-per-token is the whole game.

The chip roadmap now says this out loud. NVIDIA's Rubin generation, launched January 2026 and shipping to clouds in the second half of the year, jumps to 288 GB of HBM4 at 22 TB/s per GPU, nearly triple Blackwell, and is marketed on one number: up to 10x lower cost per inference token, pitched squarely at reasoning and agentic workloads. The decode loop is now designing the silicon.

How serving stacks make thinking cheap enough.

If reasoning means decoding thousands of tokens, the economics only work because of a stack of inference tricks, most aimed squarely at the memory-bound decode phase.

Notice the theme: nearly every trick is about moving fewer bytes per token, or amortizing one weight sweep across more useful work. That is what it means to optimize for a memory-bound workload.

One pass can only think so far.

Start from the hardware fact. A transformer does a fixed amount of computation per token. The stack has a set number of layers, and producing one token is exactly one trip down it. There is no loop inside, no "keep working until done." So a single forward pass is shallow, bounded computation. An easy question fits inside that budget. A hard one does not, and no amount of cleverness in the weights changes how deep a single pass can go.

The escape is to write. Every token the model emits is appended to the input and fed back in, so it can park an intermediate result on the page and read it back on the next pass. The context window becomes a scratchpad, and a chain of thought is just many forward passes chained end to end. Effective depth is no longer set by the architecture. It grows with how much the model writes. A token of thinking is, quite literally, one more quantum of serial computation. A fixed-depth network that could only ever do so much in one shot can now unroll a step-by-step procedure for as long as it keeps writing. The page is the tape.

"Fixed depth" sounds like a budget you could just enlarge. It is deeper than that. Look at what a pass physically is: a token enters the bottom layer and rises through each of the model's hundred-odd layers exactly once, in order. That fixed count is the entire serial budget, and it is the same for 2 + 2 as for a competition math problem. Inside any one layer, every position is processed in parallel, all at once. So a forward pass is a wide, shallow computation: astronomically wide, but only a fixed handful of steps deep. Complexity theorists place it in a low, constant-depth class: the kind of circuit that does enormous parallel work but cannot carry out a long chain of dependent steps.

And that is the real ceiling, because many problems are inherently sequential: the value at each step depends on the step before. Carrying digits across a long addition. Following a chain of logical implications. Tracing a variable through a loop. You cannot fold those into constant depth no matter how wide the network gets or how good the weights are, for the same reason nine women cannot make a baby in one month. The work is a line, not a pile. A single pass has to cram the whole line into its fixed layers, and past some size that is not hard, it is impossible.

Writing a token is how the model buys one more step in that line. The token is appended, and the next pass attends over everything written so far, reads the running state, and adds one symbol to it. That is precisely one move of a sequential machine: the context window is the tape, attention is the read head, the emitted token is the write. So a chain of T thinking tokens is not "more effort" in some vague sense. It is T sequential steps of computation the architecture flatly could not perform in one pass. The theory makes this exact: give a fixed-depth transformer a long enough chain of thought and it can simulate computations a single pass provably cannot, stepping through an algorithm one token at a time. The thinking tokens are the steps.

Two axes, same currency. Deeper: one long chain, so the model can carry a derivation, catch its own slips, and backtrack. Wider: many independent chains, then keep the answer most of them agree on (self-consistency) or the one a checker likes best (best-of-N). You can do both at once, which is what tree search is.

And here is why spending compute this way actually pays: verifying is easier than solving. For math, code, and proofs, checking a candidate answer is far cheaper and more reliable than producing one. So it is rational to generate many attempts and spend a little compute picking the good one. That asymmetry is exactly what RL with verifiable rewards trained the model to exploit during training, and what parallel sampling cashes in at run time.

# both spend more forward passes. the model and its weights never change.

# (1) DEEPER: one long chain. the model writes its own working memory.
answer = model.generate(question, thinking_budget=8000)     # long CoT

# (2) WIDER: many independent chains, then aggregate them.
tries = [model.generate(question) for _ in range(N)]          # N samples
answer = majority_vote([t.final for t in tries])           # self-consistency
answer = max(tries, key=lambda t: verifier(t))           # or best-of-N, if you can check

# bigger budget or bigger N -> higher odds a correct path exists and is picked.

Nothing here is free forever, and the reason is also first-principles. The cheap, common reasoning paths fix the easy problems first. What remains are problems the model genuinely cannot do, where more samples just repeat the same blind spot, and very long chains where a small per-step error rate compounds into a wrong final line. Coverage saturates: each doubling of the budget catches fewer new answers than the one before. So accuracy climbs steeply, then flattens. Which is exactly the dial you are about to drag.

Thinking longer is a dial now.

The shift this creates is strategic. For years you made a model smarter by making it bigger and training it longer, a cost paid up front, once. The dial from the last section moves that spending onto a second axis, test-time compute, and the bill is now paid per query, every single time you let the model think.

That is why providers expose a reasoning-effort knob. Drag it and you are trading dollars and latency for correctness. The returns are real but they bend over, steep then flat, the saturation from the last section showing up as a price.

Curve is illustrative, but the shape is real: OpenAI reported o1's accuracy scaling smoothly with test-time compute, and every reasoning model since shows the same diminishing-returns bend. Past a point you pay a lot of tokens for a little correctness.

So what are you actually dragging? Under the hood the thinking is just tokens, emitted into a delimited block before the answer. Open models make this literal: DeepSeek-R1 wraps its reasoning in <think> ... </think> and the model itself decides when to close the tag and start answering. Left alone, a reasoning model emits an end-of-thinking token when it feels done. The effort knob just overrides that decision with a budget.

It runs in two directions, and both are blunt. To make it think less, the server caps the block: when the budget runs out it forces the closing delimiter, so the model has to stop reasoning and answer with whatever it has. Set the budget to zero and you are back to a plain LLM. To make it think more, you do the opposite, suppress the end-of-thinking token and splice in a nudge like Wait so the model keeps going instead of concluding. Researchers named this trick budget forcing, and the surprising part is how well something so crude works.

Aim this back at the datacenter and the picture snaps together. Every extra thinking token is one more memory-bound decode step on a chip somewhere. "Make the model reason more" and "buy more inference silicon" are the same sentence. That is why the industry's compute spend is tilting from training toward inference, and why low-latency chips suddenly matter as much as big ones.

Same engine, longer leash.

A reasoning model is not a smarter species of network. It is the same decoder-only transformer, predicting one token at a time, given a long leash and a training signal that taught it to use that leash to think out loud, check itself, and try again.

Everything downstream follows from that. The thinking is just tokens. Tokens come from the memory-bound decode loop. The decode loop runs on chips whose real currency is bandwidth and latency, not raw FLOPs. And how long you let that loop run is now a dial with a price tag. Understand the loop, and the whole stack — from the reward function to the HBM stack — lines up behind it.

The interactive diagrams here are schematic, built to carry the intuition rather than exact numbers. Bandwidth figures come from vendor specifications; the accuracy curve is illustrative of the diminishing-returns shape reported for test-time scaling, not a measured benchmark.