A reasoning model is an ordinary model trained to think first.

A reasoning model spends tokens before it answers.

A plain model goes from your question almost straight to an answer. If the answer needs five steps of work, the model has to do all five inside one forward pass (one trip through the network), with no scratch paper. On anything hard, it guesses.

A reasoning model is trained to first write a long chain of thought, sometimes thousands of "thinking" tokens, and only then write the answer. The thinking tokens are ordinary next-token predictions. But each one is fed back in, so the model can read its own work, catch mistakes, and try another route. The thinking tokens are its scratch paper.

OpenAI's o-series, DeepSeek-R1, Claude's extended thinking, Gemini's thinking modes, and Qwen's QwQ all work this way. A model that "reasons" is a model that was taught to use a long scratch pad well.

Reasoning models keep the same architecture and change the training.

A reasoning model is, in almost every case, the same decoder-only transformer as the base model. There is no new type of layer and no extra module added for "reasoning." If you looked at the network diagrams, you could not tell a reasoning model from a chat model.

What changed is how the weights were tuned. After normal pretraining, the model goes through reinforcement learning with verifiable rewards. You give it math and code problems whose answers a script can check. You reward the chains of thought that reach the right answer. Over many rounds, the model learns for itself which kinds of thinking pay off.

DeepSeek showed this clearly with R1-Zero. They trained with an RL recipe called GRPO, starting from a base model that had seen no supervised reasoning examples at all. Over training, it taught itself to write longer and longer chains, to pause, and to re-check its work. This is the much-quoted "aha moment." No person wrote those reasoning traces. The model learned them from the reward.

# the model is unchanged: same transformer, same forward pass.
# RL just reshapes which chains-of-thought it tends to produce.
for question, checker in verifiable_problems:
    group = [model.generate(question) for _ in range(G)]   # sample G long answers
    rewards = [1.0 if checker(a.final) else 0.0 for a in group]
    baseline = mean(rewards)                            # the "group relative" part
    for a, r in zip(group, rewards):
        reinforce(a.tokens, advantage=r - baseline)     # upvote chains that worked
    # over time: longer reasoning, self-checking, backtracking, all emergent

The full training pipeline (pretraining, SFT, preference tuning, then this reasoning-RL phase) is explained step by step in the LLM training explainer. Here we care about what it costs to run.

Two things are called "the decoder."

The word has two meanings, and both matter for reasoning models, so it is worth separating them.

The decoder stack is the architecture. It is a tower of identical transformer layers. Each layer runs self-attention under a causal mask and then a feed-forward network. "Decoder-only" is the GPT family of models: there is no encoder, only this stack predicting the next token. The decoding step is the other meaning. It takes the model's output probabilities and chooses a token, either greedily (always the most likely one) or with temperature and top-p sampling.

The shortcut that makes this loop affordable is the KV cache. Without it, every new token would make the model process every earlier token again from the start. With it, the model computes attention only for the one new token and reuses the stored keys and values for everything before it. The cache is what makes long reasoning traces practical. As we will see, it is also what uses up the memory.

Inference has two phases, and each is limited by a different part of the chip.

Running the model splits cleanly into prefill and decode. Prefill reads your prompt all at once. Every prompt token can be processed in parallel, so the chip spends its time on large, dense matrix multiplications and is compute-bound. Decode makes one token at a time, and each token needs a full pass over the model's weights, so the chip is memory-bandwidth-bound.

Prefill happens once. Decode happens once per output token. A chat reply might decode a few dozen tokens. A reasoning trace decodes thousands. So a reasoning model spends almost all of its time in the memory-bound decode phase, reading the full set of weights out of memory thousands of times for a single answer. For reasoning models, the numbers that matter are memory bandwidth and how long the loop runs, not FLOPs.

Reasoning models need fast memory.

In the decode phase, generating one token means reading every weight on the active path from memory, once. So the top speed in tokens per second is roughly memory bandwidth ÷ model size. The limit is how fast the chip can feed data to its math units, not how many FLOPs those units can do. This one fact explains why inference hardware looks the way it does.

Figures are peak memory bandwidth per device, from vendor specs. Real throughput depends on model size, batch size, and quantization. SRAM chips hold very little, but move data enormously fast.

This leads to two strategies. The HBM path (NVIDIA H100/H200/Blackwell and now Rubin, Google TPUs, AWS Trainium) keeps the model in large, very fast off-chip memory. It wins on capacity and on batching many requests together. The SRAM path (Groq's LPU, Cerebras's wafer-scale engine) keeps the weights in SRAM on the chip itself and gets the lowest per-token latency in the industry. That suits a model writing thousands of thinking tokens one after another. When the answer has to wait for a long chain of steps in sequence, the time per token decides how long you wait.

Chip makers now say this openly. NVIDIA's Rubin generation launched in January 2026 and is shipping to clouds in the second half of the year. Each GPU has 288 GB of HBM4 at 22 TB/s, nearly triple Blackwell's bandwidth. NVIDIA markets it on one number, up to 10x lower cost per inference token, and aims it at reasoning and agentic workloads. Chips are now being designed around the decode loop.

How serving stacks make thinking cheap enough.

Reasoning means decoding thousands of tokens. That is only affordable because of a set of inference tricks, and most of them target the memory-bound decode phase.

Nearly every trick either moves fewer bytes per token or spreads one weight sweep across more useful work. That is what optimizing for a memory-bound workload means.

A single forward pass can only take so many steps.

A transformer does a fixed amount of computation per token. The stack has a set number of layers, and producing one token is exactly one trip through them. There is no loop inside, nothing that says "keep working until done." So a single forward pass is a shallow computation with a fixed limit. An easy question fits inside that limit. A hard one does not, and no amount of cleverness in the weights changes how deep a single pass can go.

The way around this is to write. Every token the model emits is added to the input and fed back in. So the model can write down an intermediate result and read it back on the next pass. The context window becomes a scratchpad, and a chain of thought is many forward passes linked end to end. The model's effective depth (how many steps it can take in a row) is no longer set by the architecture. It grows with how much the model writes. Each token of thinking adds one more step of sequential computation. A fixed-depth network that could only do so much in one pass can now carry out a step-by-step procedure for as long as it keeps writing.

"Fixed depth" sounds like a budget you could make bigger. The limit is more basic than that. Look at what happens physically in one pass. A token enters the bottom layer and rises through each of the model's hundred-odd layers exactly once, in order. That fixed count is the total serial budget, and it is the same for 2 + 2 as for a competition math problem. Inside any one layer, every position is processed in parallel, all at once. So a forward pass is a wide, shallow computation. It is enormously wide, but only a fixed handful of steps deep. Complexity theorists place it in a low, constant-depth class: the kind of circuit that does a huge amount of parallel work but cannot carry out a long chain of dependent steps.

That is a real limit, because many problems are inherently sequential: each step needs the result of the step before. Examples are carrying digits across a long addition, following a chain of logical implications, or tracing a variable through a loop. You cannot squeeze those into a constant number of steps, no matter how wide the network gets or how good the weights are. It is the same reason nine women cannot make a baby in one month. The steps have to happen one after another. A single pass would have to fit the whole chain into its fixed layers, and past some length that is impossible, not merely hard.

Writing a token gives the model one more step in that chain. The token is added to the context. The next pass looks over everything written so far, reads the current state, and adds one symbol to it. That is one move of a sequential machine, like a Turing machine: the context window is the tape, attention is the read head, and the emitted token is the write. So a chain of T thinking tokens gives T sequential steps of computation, steps the architecture could not perform in one pass. Researchers have proved this formally. Give a fixed-depth transformer a long enough chain of thought, and it can simulate computations that a single pass provably cannot, stepping through an algorithm one token at a time. Each thinking token is one step of that algorithm.

There are two ways to spend extra forward passes. Deeper: one long chain, so the model can carry a derivation, catch its own slips, and backtrack. Wider: many independent chains, then keep the answer most of them agree on (self-consistency) or the one a checker scores highest (best-of-N). You can do both at once, and that is what tree search does.

Spending compute this way pays off because verifying is easier than solving. For math, code, and proofs, checking a candidate answer is far cheaper and more reliable than producing one. So it makes sense to generate many attempts and spend a little compute picking the good one. RL with verifiable rewards relies on this same gap to train the model, and parallel sampling uses it again at run time.

# both spend more forward passes. the model and its weights never change.

# (1) DEEPER: one long chain. the model writes its own working memory.
answer = model.generate(question, thinking_budget=8000)     # long CoT

# (2) WIDER: many independent chains, then aggregate them.
tries = [model.generate(question) for _ in range(N)]          # N samples
answer = majority_vote([t.final for t in tries])           # self-consistency
answer = max(tries, key=lambda t: verifier(t))           # or best-of-N, if you can check

# bigger budget or bigger N -> higher odds a correct path exists and is picked.

The gains do not go on forever, and the reason also follows from first principles. The cheap, common reasoning paths solve the easy problems first. What remains are problems the model cannot do at all, where more samples repeat the same blind spot. There are also very long chains, where a small error rate at each step adds up to a wrong final line. Coverage saturates: each doubling of the budget catches fewer new answers than the one before. So accuracy climbs steeply, then flattens. That curve is the dial you will drag in the next section.

How long a model thinks is now a dial you can set.

This changes how you pay for a smarter model. For years you made a model smarter by making it bigger and training it longer, a cost paid once, up front. The thinking budget from the last section moves that spending onto a second axis, test-time compute, and that bill is paid per query, every time you let the model think.

That is why providers offer a reasoning-effort knob. Turning it up trades money and waiting time for correctness. The returns are real, but they follow the same curve: steep at first, then flat. The saturation from the last section shows up here as a price.

The curve is illustrative, but the shape is real. OpenAI reported that o1's accuracy rose smoothly with test-time compute, and every reasoning model since shows the same bend of diminishing returns. Past a point, you pay a lot of tokens for a little more correctness.

Under the hood, the thinking is ordinary tokens, written into a delimited block (a section with a start marker and an end marker) before the answer. Open models show this directly. DeepSeek-R1 wraps its reasoning in <think> ... </think>, and the model itself decides when to close the tag and start answering. Left alone, a reasoning model emits an end-of-thinking token when it decides it is done. The effort knob overrides that decision with a budget.

It works in two directions, and both are crude. To make the model think less, the server caps the block. When the budget runs out, it inserts the closing delimiter, so the model has to stop reasoning and answer with whatever it has. Set the budget to zero and you are back to a plain LLM. To make it think more, you do the opposite. You block the end-of-thinking token and insert a word like Wait, so the model keeps going instead of wrapping up. Researchers call this trick budget forcing, and it works surprisingly well for something so crude.

Now connect this back to the datacenter. Every extra thinking token is one more memory-bound decode step on a chip somewhere. So "make the model reason more" and "buy more inference chips" mean the same thing. That is why the industry's compute spending is shifting from training toward inference, and why low-latency chips now matter as much as big ones.

A reasoning model is the same model, given room to think.

A reasoning model is the same decoder-only transformer, predicting one token at a time. It is allowed to write for much longer, and its training rewarded it for using that room to think out loud, check itself, and try again.

The rest follows from that. The thinking is made of tokens. Tokens come from the memory-bound decode loop. The decode loop runs on chips where bandwidth and latency matter more than raw FLOPs. And how long you let that loop run is now a setting with a price. If you understand that loop, you understand the rest, from the reward function to the HBM stack.

The interactive diagrams here are schematic. They are meant to build intuition, not to show exact numbers. Bandwidth figures come from vendor specifications. The accuracy curve shows the diminishing-returns shape reported for test-time scaling; it is not a measured benchmark.