The transformer only ever sees lists of numbers.

Each kind of input has its own door into one shared space.

Picture a single, shared vector space. It is a huge coordinate system where every input becomes a point. A point can stand for the word "dog," the sound of a bark, or a photo of a spaniel. If they mean the same thing, they sit near each other. The transformer works inside this space and reasons over whatever points it is handed.

Each modality gets its own encoder. An encoder is a piece of network with one job: take raw input (pixels, audio samples, a rendered page) and turn it into points in the shared space. It is that input's door into the space. For text the encoder is simple, a lookup table. Images and audio need real neural networks. Once an input is through its door, it is a list of vectors like the words are, and attention mixes them all together in one sequence.

# one model, many inputs. the encoder is the only modality-specific part.
def forward(inputs):                 # inputs: text, an image, some audio, a page...
    seq = []
    for x in inputs:
        enc = ENCODERS[x.modality]      # pick the door for this modality
        seq += enc(x)                   # -> vectors in ONE shared space
    return transformer(seq)            # same attention, same next-token loop
# pixels instead of words changes ENCODERS, not the transformer.

A word, a patch and a sound all become 4,096 numbers.

"Turned into a vector" is hard to picture until you see it done once. So here is one worked example for each door, with real shapes. The vectors below are 4,096 numbers wide. That is the embedding dimension of a Llama-3-8B-class model. Bigger models use wider vectors, but the steps are the same.

# the same LLM width (4,096 floats), reached three ways
"dog"                             # text: a table lookup, no computation
  → id 5679 → E[5679]
  → [ 0.013, -0.208,  0.091, ...,  0.044]

patch(row 3, col 7)               # image: one matrix multiply
  → 768 pixel values → W @ px
  → [-0.171,  0.062, -0.055, ...,  0.238]

frame @ t=1.20s                   # audio: conv + attention, then a connector
  → 80 mel energies → encode
  → [ 0.087, -0.114,  0.302, ..., -0.201]
# three origins, one shape. downstream, nothing can tell them apart.

Cut the image into squares and treat each square as a word.

A sentence is already a sequence, so splitting it into tokens is natural. An image is a grid of a million pixels, with no obvious "first" or "next." The Vision Transformer solved this with a very simple idea. Chop the image into a grid of small squares and call each square a token. A patch of, say, 16×16 pixels becomes one item in a sequence, the same way a word does.

Each patch is flattened into one long line of numbers and passed through one small linear layer, the patch embedding, which turns its raw pixels into a vector in the shared space. Then a positional embedding is added so the model knows where the patch sat in the grid. Now the picture is a sequence of vectors, and it can go into the same transformer the words use.

Attention has no idea what came first.

Why did that stepper need an "add position" station at all? Look at what attention computes. Each token makes three small vectors from its embedding: a query, a key and a value. The score between two tokens is the dot product of one token's query with the other's key. A softmax turns the scores into weights that add up to 1, and the output is a weighted average of the values. Each of these operations treats the tokens as a set, with no order. No index, no "previous" and no "next" appear anywhere in the formula.

# attention, for one token A among {A, B, C}
w = softmax([q_A·k_A, q_A·k_B, q_A·k_C])   # who should A look at?
out_A = w[0]*v_A + w[1]*v_B + w[2]*v_C      # a weighted average. that is all.
# hand it B and C in swapped order: the same numbers come out.
# without position, "dog bites man" and "man bites dog" are one bag of words.

So order has to be put in as data. Before attention runs, a position vector is added to each token's embedding. Now position is part of every query and key, so the dot product between two tokens changes with how far apart they sit.

What does that position vector look like? The original 2017 transformer used sine waves. Each dimension goes up and down at its own frequency. Fast ones change at every position. Slow ones drift across thousands of positions, like the second, minute and hour hands of a clock. A Vision Transformer instead keeps a plain learned table: one vector per grid slot, random at the start and shaped by training. And most modern LLMs use RoPE, which rotates the query and key vectors by an angle proportional to position instead of adding anything. All three do the same job: they make the same token look different at different positions.

Getting image vectors into the language model's space.

Patchifying gives you image vectors, but they start out in the vision encoder's own space, which is different from the language model's. Two steps connect them.

First, alignment. A model called CLIP is trained on hundreds of millions of image-caption pairs to place a photo and its description at the same spot. After training, the text "a photo of a dog" and an actual dog photo land near each other. So the image vectors now carry meaning in a form that lines up with text.

Second, the bridge. Most vision-language models keep a frozen CLIP-style encoder (its weights are not trained any further) and add a tiny connector, often a two-layer network, that maps image vectors into the LLM's token space. This is the recipe behind LLaVA and most open vision models. A photo goes in as a few hundred visual tokens, placed in the prompt right next to the words.

# a CLIP alignment pair (one of ~400,000,000)
{ "image": beach_dog_0412.jpg,      # just pixels, nothing labeled inside
  "text":  "a golden retriever running on a beach" }
# batched 32,768 at a time: pull each true pair together,
# push the 32,767 mismatches apart. that is the whole recipe.

# a LLaVA connector-tuning example (one of ~600,000)
{ "image": "coco/000000215677.jpg",
  "conversations": [
    { "from": "human", "value": "<image>\nWhat is unusual about this image?" },
    { "from": "gpt",   "value": "The man is ironing clothes on a board
                              attached to the roof of a moving taxi." } ] }
# at train time the <image> slot is replaced by ~576 vectors from the
# frozen vision encoder. only the tiny connector learns to make them legible.

Those few hundred tokens are why images cost what they do. Providers cut a picture into tiles and charge per tile. It works like text pricing: more visual detail means more tokens, and more tokens means more compute.

# how many tokens is an image? (OpenAI high-detail accounting, 2026)
def image_tokens(w, h, tile=512):
    w, h = fit_inside(w, h, box=2048)      # shrink to fit a 2048 box
    w, h = scale_short_side(w, h, to=768)   # then short side -> 768px
    tiles = ceil(w/tile) * ceil(h/tile)     # count the 512px tiles
    return 85 + 170 * tiles                # base + per-tile
image_tokens(1024, 1024)                  # -> ~765 tokens for one picture
# Claude and Gemini use different formulas, same idea: detail = tokens.

Sound becomes a picture first.

Raw audio is hard to model directly. One second of sound is tens of thousands of samples, far too many for attention to handle. So almost every audio model first converts the waveform into a spectrogram, a picture with time across the bottom and pitch up the side. Then it handles that picture almost the same way as an image.

This is what Whisper does. It turns the waveform into a log-mel spectrogram. Then a few convolution layers and a transformer encoder turn that into a sequence of vectors, one per short slice of time. From there, it works like every other input. Those vectors can feed a decoder that writes text, or be projected into an LLM so it can "hear" your voice note.

Generating audio needs the reverse. A neural codec compresses sound into a stream of discrete audio tokens using residual vector quantization. The model predicts those tokens like any other sequence, and the codec's decoder turns them back into a waveform. So to understand audio, a model uses the spectrogram door. To produce audio, it uses the codec door.

# whisper-large's front door, concretely
samples = 30 * 16_000            # 480,000 amplitude values
mel     = log_mel(samples)        # (80, 3000) · 80 pitches x 10 ms frames
h       = conv_stem(mel)          # (1500, 1280) · stride-2 conv halves time
vectors = encoder(h)              # 1,500 vectors · one per 20 ms of sound
# 480,000 samples in, 1,500 vectors out: a 320x squeeze.
# your voice costs ~50 vectors per second, the price of dense prose.

A model often reads a PDF as a picture of each page.

This one surprises people. A modern model usually does not parse a PDF's internal file structure at all. There are two doors, and the more interesting one is also the simpler one.

Look at what the second door means. To read your PDF, the model takes a screenshot of each page and cuts it into patches like any other image. Nothing tells it this is a "document." It sees a picture that happens to be full of words and lines, and reads it with the same vision machinery it uses for a photo of a beach.

So which models read documents best right now? The surprise of the past year is that the generalists won. On public OCR leaderboards in early 2026, the top of the table is held by general-purpose models, not document specialists. Gemini 3, Claude Opus 4.6 and GPT-5.2 are ordinary frontier multimodal models that read pages through the render-and-look door. They beat dedicated OCR pipelines on messy scans, receipts and handwriting, with roughly 3-4× fewer character errors on noisy pages.

Recent research turns the cost argument around. DeepSeek-OCR showed that reading a page as an image can be the cheap option. One vision token can carry roughly ten text tokens' worth of content at ~97% fidelity. So a dense page fits in a few hundred vision tokens, where a raw text dump of the same page needs thousands. Its January 2026 successor reads a complex page in 256 to 1,120 vision tokens. The image door was first chosen because it keeps the page faithful. Now it is also becoming a way to compress text.

One caveat: VLM readers run 5-10× slower than classic OCR engines. What happens when they misread is covered next.

If a page becomes patch vectors, there is a fair worry. Patchifying is lossy compression, so some detail is thrown away. And a next-token predictor never says "unreadable." Show it a smudged 7 and it will output the most plausible digit, with full confidence. Nothing in the mechanism guarantees that a number comes through correctly. So where does the accuracy come from? It comes from four places.

# both doors at once, then verify (Claude-style ingestion)
for page in pdf:
    seq += vision_tokens(render(page, dpi=200))   # layout, charts, stamps
    seq += text_tokens(extract_text(page))        # the exact characters
answer = llm(seq + question)
# digits anchor to the text stream, positions to the pixels.
# high stakes? cross-check with classic OCR, or require a
# page + bounding-box citation before trusting a number.

Three ways to feed a new modality to the LLM.

Once you can turn an image or a sound into vectors, one question is left: how do those vectors get into the language model? The field has tried three answers. In order, they go from attaching vision to an existing model to building it in from the start.

How the shared space is learned.

A shared vector space has to be learned. Something has to train the model so that a dog photo and the word "dog" land in the same place. There are three techniques, matching the three fusion styles.

Contrastive alignment comes first. Show the model a batch of image-caption pairs. Train it to pull each true pair closer together and push every mismatched pair apart. Do this across hundreds of millions of pairs and the image encoder and the text encoder end up placing matching things in the same spots. That is CLIP, and it is the reason any of this works.

Connector tuning comes next. Freeze the vision encoder and the LLM, and train only the small connector between them on image-and-caption data. Then do a light round of instruction tuning. This is cheap and fast, and it is enough to let a text model see images. Native pretraining is the expensive option. Mix every modality into one token stream and train the whole model on next-token prediction over all of it, from the first step.

# CLIP: pull matching image+text together, push mismatches apart.
img = image_encoder(images)               # N images   -> N vectors
txt = text_encoder(captions)              # N captions -> N vectors
scores = img @ txt.T                       # N x N grid of similarities
loss = cross_entropy(scores, diagonal)    # the diagonal is the true pairing
# minimize this and "a dog" lands exactly where a dog photo lands.

The full pipeline, pretraining then post-training, is covered step by step in the LLM training explainer. For a multimodal model, the only difference is that the training data now includes pictures and sound as well as words.

Turning vectors back into pictures and sound.

Everything so far was about input: turning images and sound into vectors. Output goes the other way, from vectors to pixels or sound. There are two main ways to make an image, and they mirror the fusion choices above.

Audio output works the same way, with an audio codec in place of the image codebook. The model predicts discrete audio tokens, and the neural codec's decoder rebuilds the waveform. Put input and output together and you get an any-to-any model: one network that can read a screenshot, hear a question, and answer in speech or with a new picture. Google's Gemini Omni, announced in May 2026, goes furthest. It is a single backbone that takes any mix of text, image, audio and video and can generate video as output.

Attention cannot tell where a vector came from.

Why does all of this work? Look at what attention does. It takes a set of vectors and computes weighted averages of them, based only on how they point relative to each other. No tag says where a vector came from, and attention never looks for one. To the attention math, a vector from the word "cat" and a vector from a 16×16 patch of fur are the same kind of thing: a point in the space.

In these models, meaning is geometric. A vector's meaning comes from where it sits and what it sits near. So suppose your encoder does its job and places the sound of a dog barking near the word "dog" and near a photo of a dog. Then reasoning about the sound and reasoning about the sentence are the same computation. The transformer does not translate between modalities. Once every input is a point in one space, there is nothing to translate.

That is why the gap between modalities is a problem of encoding, not reasoning. Adding a new sense to a model (touch, smell, a stream of sensor data) has never required a smarter transformer. It requires a good encoder that maps the new signal into the shared space. The transformer's reasoning already works on any input placed in that space.

The same LLM, with more ways in.

A multimodal model is the same LLM you already understand, with more doors. The transformer still never sees a word, a pixel or a sound wave. It sees vectors, mixes them with attention, and predicts the next one.

Everything on this page came from one idea: turn each modality into a sequence of vectors in one shared space. Cut an image into patches, turn a sound into a spectrogram, take a screenshot of a page. Align them with contrastive training. Then connect them with a small connector, or train everything together from scratch. Run the process backwards through a decoder and the model can produce pictures and sound too. Understand the encoders, and the rest is the LLM you already know.

The interactive diagrams are simplified sketches meant to build intuition, not to show exact tensor shapes. Patch counts, token costs and codebook sizes are typical figures from the cited papers and provider docs. They vary by model and configuration.