The transformer never sees a word.

Every modality is a different door to one room.

Picture a single, shared vector space: an enormous coordinate system where a point can mean the word "dog," the sound of a bark, or a photo of a spaniel, and if they mean the same thing, they sit near each other. The transformer lives inside this room and reasons over whatever points it is handed.

Each modality gets its own encoder, a piece of network whose only job is to take raw input, pixels, audio samples, a rendered page, and turn it into points in that shared room. The encoder is the border crossing. Text has a trivial one, a lookup table. Images and audio need real neural networks. But once anything is through the door, it is just vectors on the same footing as the words, and attention mixes them all together in one sequence.

# one model, many inputs. the encoder is the only modality-specific part.
def forward(inputs):                 # inputs: text, an image, some audio, a page...
    seq = []
    for x in inputs:
        enc = ENCODERS[x.modality]      # pick the door for this modality
        seq += enc(x)                   # -> vectors in ONE shared space
    return transformer(seq)            # same attention, same next-token loop
# pixels instead of words changes ENCODERS, not the transformer.

What an embedding actually looks like.

"Turned into a vector" stays abstract until you watch it happen once. So here is one worked example per door, with real shapes. The width below is 4,096 numbers, the embedding dimension of a Llama-3-8B-class model. Bigger models use wider vectors, but the story never changes.

# the same LLM width (4,096 floats), reached three ways
"dog"                             # text: a table lookup, no computation
  → id 5679 → E[5679]
  → [ 0.013, -0.208,  0.091, ...,  0.044]

patch(row 3, col 7)               # image: one matrix multiply
  → 768 pixel values → W @ px
  → [-0.171,  0.062, -0.055, ...,  0.238]

frame @ t=1.20s                   # audio: conv + attention, then a connector
  → 80 mel energies → encode
  → [ 0.087, -0.114,  0.302, ..., -0.201]
# three origins, one shape. downstream, nothing can tell them apart.

An image is worth sixteen-by-sixteen words.

A sentence is already a sequence, so tokenizing it is natural. An image is a grid of a million pixels, with no obvious "first" or "next." The Vision Transformer solved this with an idea so blunt it is almost funny: chop the image into a grid of small squares and call each square a token. A patch of, say, 16×16 pixels becomes one item in a sequence, exactly like a word.

Each patch is flattened into a line of numbers and pushed through one small linear layer, a patch embedding, which turns its raw pixels into a vector in the shared space. Then a positional embedding is added so the model knows where the patch sat. Now the picture is a sequence of vectors, and it can walk into the same transformer the words use.

Attention has no idea what came first.

Why did that stepper need an "add position" station at all? Look at what attention literally computes. Each token derives three small vectors, a query, a key and a value. The score between two tokens is a dot product of a query with a key. The scores are softmaxed into weights, and the output is a weighted average of the values. Every one of those operations acts on a set. No index, no "previous", no "next" appears anywhere in the formula.

# attention, for one token A among {A, B, C}
w = softmax([q_A·k_A, q_A·k_B, q_A·k_C])   # who should A look at?
out_A = w[0]*v_A + w[1]*v_B + w[2]*v_C      # a weighted average. that is all.
# hand it B and C in swapped order: the same numbers come out.
# without position, "dog bites man" and "man bites dog" are one bag of words.

So order has to be smuggled in as data: before attention runs, a position vector is added to each token's embedding. After that, two queries and keys can "feel" how far apart their tokens sit, because the position signal is baked into the dot products.

What does that position vector look like? The original 2017 transformer used sine waves: each dimension oscillates at its own frequency, fast ones ticking every position, slow ones drifting across thousands, like the second, minute and hour hands of a clock. A Vision Transformer instead keeps a plain learned table, one free vector per grid slot, random at the start, shaped by training. And most modern LLMs use RoPE, which rotates the query and key vectors by an angle proportional to position instead of adding anything at all. Three flavors, one job: make identical tokens at different slots look different.

Teaching pixels and words to share an address.

Patchifying gives you image vectors, but they start out in the vision encoder's own private space, not the language model's. Two moves connect them.

First, alignment. A model called CLIP is trained on hundreds of millions of image-caption pairs to place a photo and its description at the same spot. After that, "a photo of a dog" and an actual dog photo are neighbors. The vision encoder now speaks a language the text model can understand.

Second, the bridge. Most vision-language models keep a frozen CLIP-style encoder and bolt on a tiny connector, often a two-layer network, that projects image vectors precisely into the LLM's token space. This is the recipe behind LLaVA and most open vision models: a photo enters as a few hundred visual tokens, slotted into the prompt right next to the words.

# a CLIP alignment pair (one of ~400,000,000)
{ "image": beach_dog_0412.jpg,      # just pixels, nothing labeled inside
  "text":  "a golden retriever running on a beach" }
# batched 32,768 at a time: pull each true pair together,
# push the 32,767 mismatches apart. that is the whole recipe.

# a LLaVA connector-tuning example (one of ~600,000)
{ "image": "coco/000000215677.jpg",
  "conversations": [
    { "from": "human", "value": "<image>\nWhat is unusual about this image?" },
    { "from": "gpt",   "value": "The man is ironing clothes on a board
                              attached to the roof of a moving taxi." } ] }
# at train time the <image> slot is replaced by ~576 vectors from the
# frozen vision encoder. only the tiny connector learns to make them legible.

That "few hundred tokens" is why images cost what they do. Providers slice a picture into tiles and charge per tile. It is the same accounting as text: more visual detail means more tokens means more compute.

# how many tokens is an image? (OpenAI high-detail accounting, 2026)
def image_tokens(w, h, tile=512):
    w, h = fit_inside(w, h, box=2048)      # shrink to fit a 2048 box
    w, h = scale_short_side(w, h, to=768)   # then short side -> 768px
    tiles = ceil(w/tile) * ceil(h/tile)     # count the 512px tiles
    return 85 + 170 * tiles                # base + per-tile
image_tokens(1024, 1024)                  # -> ~765 tokens for one picture
# Claude and Gemini use different formulas, same idea: detail = tokens.

Sound becomes a picture first.

Raw audio is brutal to model directly: a single second is tens of thousands of samples, far too many to attend over. So almost every audio model does something sneaky. It converts the waveform into a spectrogram, a picture with time across the bottom and pitch up the side, and then treats that picture almost exactly like the image case.

This is what Whisper does: waveform to a log-mel spectrogram, then a convolutional stem and a transformer encoder that emit a sequence of vectors, one per short slice of time. From there it is the shared-space story again: those vectors can feed a decoder that writes text, or be projected into an LLM so it can "hear" your voice note.

Generating audio needs the reverse. A neural codec compresses sound into a stream of discrete audio tokens using residual vector quantization, the model predicts those tokens like any other sequence, and the codec's decoder turns them back into a waveform. Understanding uses the spectrogram door; generation uses the codec door.

# whisper-large's front door, concretely
samples = 30 * 16_000            # 480,000 amplitude values
mel     = log_mel(samples)        # (80, 3000) · 80 pitches x 10 ms frames
h       = conv_stem(mel)          # (1500, 1280) · stride-2 conv halves time
vectors = encoder(h)              # 1,500 vectors · one per 20 ms of sound
# 480,000 samples in, 1,500 vectors out: a 320x squeeze.
# your voice costs ~50 vectors per second, the price of dense prose.

A PDF is not special. It's a photo.

This is the one that surprises people. A modern model usually does not parse a PDF's internal structure at all. There are two doors, and the interesting one is almost aggressively simple.

That second door is the punchline. To read your PDF, the model takes a screenshot of each page and patchifies it like any other image. It does not know it is a "document." It is looking at a picture that happens to be full of words and lines, and reading it with the same vision machinery it uses for a photo of a beach.

So who reads documents best right now? The surprise of the past year: the generalists won. On public OCR leaderboards in early 2026 the top of the table is not a document specialist at all. Gemini 3, Claude Opus 4.6 and GPT-5.2, ordinary frontier multimodal models walking through the render-and-look door, beat dedicated OCR pipelines on messy scans, receipts and handwriting, with roughly 3-4× fewer character errors on noisy pages.

The research edge runs the logic backwards. DeepSeek-OCR showed that a page-as-image can be the cheap option, not the expensive one: one vision token can carry roughly ten text tokens' worth of content at ~97% fidelity, so a dense page fits in a few hundred vision tokens where a raw text dump needs thousands. Its January 2026 successor reads a complex page in 256 to 1,120 vision tokens. The door we picked for faithfulness is turning into a compression scheme.

The honest caveat: VLM readers run 5-10× slower than classic OCR engines. And what happens when they misread is the subject of the next block.

If a page is just patch vectors, a fair worry follows. Patchifying is lossy compression, and a next-token engine never says "unreadable." Show it a smudged 7 and it will happily emit the most plausible digit instead, with full confidence. Nothing in the mechanism guarantees a number survives the trip. So where does the accuracy actually come from? Four places, none of them magic.

# both doors at once, then verify (Claude-style ingestion)
for page in pdf:
    seq += vision_tokens(render(page, dpi=200))   # layout, charts, stamps
    seq += text_tokens(extract_text(page))        # the exact characters
answer = llm(seq + question)
# digits anchor to the text stream, positions to the pixels.
# high stakes? cross-check with classic OCR, or require a
# page + bounding-box citation before trusting a number.

Three ways to let a modality in.

Once you can turn an image or a sound into vectors, one question remains: how do those vectors actually meet the language model? The field has tried three answers, and they line up as a clear progression from bolting-on to building-in.

How the shared space is learned.

A shared vector space is not free. Something has to teach a dog photo and the word "dog" to land in the same place. Three techniques stack up, matching the three fusion styles.

Contrastive alignment is where it starts. Show the model a batch of image-caption pairs, and train it to pull each true pair together while pushing every mismatch apart. Do this across hundreds of millions of pairs and the two encoders converge on one geometry. That is CLIP, and it is the reason any of this works.

Connector tuning comes next. Freeze the vision encoder and the LLM, and train only the little bridge between them on image-and-caption data, then lightly instruction-tune. Cheap, fast, and enough to give a text model working eyes. Native pretraining is the heavy option: mix every modality into one token stream and train the whole model on next-token prediction over all of it, from the first step.

# CLIP: pull matching image+text together, push mismatches apart.
img = image_encoder(images)               # N images   -> N vectors
txt = text_encoder(captions)              # N captions -> N vectors
scores = img @ txt.T                       # N x N grid of similarities
loss = cross_entropy(scores, diagonal)    # the diagonal is the true pairing
# minimize this and "a dog" lands exactly where a dog photo lands.

The detailed pipeline, pretraining then post-training, gets its own teardown in the LLM training explainer. The multimodal twist is only this: the data stream now carries pictures and sound, not just words.

Vectors back into the world.

Everything so far was input: senses into vectors. Output runs the arrow backwards, vectors into pixels or sound. There are two dominant ways to make an image, and they mirror the fusion debate.

Audio output is the same story with a different codec: predict discrete audio tokens, then let the neural codec's decoder rebuild the waveform. Put input and output together and you get an any-to-any model, one network that reads a screenshot, hears a question, and answers in speech or a fresh picture. Google's Gemini Omni, announced in May 2026, pushes this furthest: a single backbone taking any mix of text, image, audio and video and generating video out.

Attention is blind to where a vector came from.

Step down one level and the reason all of this holds becomes almost inevitable. Look at what attention physically does: it takes a set of vectors and computes weighted averages of them, based only on how they point relative to each other. It never inspects an origin tag. A vector from the word "cat" and a vector from a 16×16 patch of fur are, to the attention math, indistinguishable kinds of thing. They are just points.

And meaning, in these models, is geometric. A vector means what its neighbors mean; the content is the position, not the label. So if your encoder does its job, if it lands "a dog barking" near the word "dog" and near a photo of a dog, then reasoning about the sound and reasoning about the sentence are literally the same computation. The transformer does not translate between modalities. There is nothing to translate once everyone is a point in one space.

That is why the modality gap is an encoding boundary, not a reasoning boundary. Adding a new sense to a model, touch, smell, a stream of sensor data, has never required a smarter transformer. It requires one thing: a good encoder that maps the new signal into the shared space. The intelligence is already modality-agnostic. It was waiting for the door.

Same engine, more senses.

A multimodal model is not a different species from the LLM you already understand. It is that exact model, handed a wider set of doors. The transformer still never sees a word, or a pixel, or a soundwave. It sees vectors, mixes them with attention, and predicts the next one.

Everything followed from one move: turn each modality into a sequence of vectors in one shared space. Patchify an image, picture a sound as a spectrogram, screenshot a page. Align them with contrastive training, bridge them with a small connector or grow them together from scratch. Run the arrow backwards through a decoder and the model speaks in pictures and sound too. Learn the border crossing, and the whole thing lines up behind it.

The interactive diagrams are schematic, built to carry the intuition rather than exact tensor shapes. Patch counts, token costs and codebook sizes are representative figures from the cited papers and provider docs, and vary by model and configuration.