Backporch

Research · architectures

The Architecture of Intelligence

The mathematics under modern AI, built the way the rest of this research is — from the pieces up. A model never sees a word; it sees a vector. Everything after is linear algebra, probability, and one idea used everywhere: attention, a soft and differentiable lookup that turns out to be ordinary kernel regression. Below: the token, the attention mechanism (with a live lab), the transformer block, the training loss, diffusion, and the shape of what comes next.

1 · Tokens & embeddings — meaning as geometry

Text is first chopped into tokens (word-pieces), and an embedding table maps each token to a point in a high-dimensional space, $E : token \mapsto R^{d}$ . Those points are learned so that meaning becomes geometry: related tokens sit close, and directions carry sense — the canonical $king - man + woman \approx queen$ . Similarity is just a dot product, and cosine is that dot product normalized:

cos θ = \frac{⟨ u , v ⟩}{∥ u ∥ ∥ v ∥}

That is the same correlation-is-a-cosine move from the derivations — here it is the entire notion of “how related are these two tokens.” Every operation that follows is arithmetic on these vectors.

2 · Attention — a soft, differentiable lookup

The central move. A token needs to gather information from other tokens — but which ones? Attention answers softly. From each token form a query $q$ ; from every token a key $k$ and a value $v$ . Score the query against every key, softmax the scores into weights, and read out a weighted average of the values:

Attention (Q, K, V) = softmax (\frac{Q K ^{⊤}}{d}) V

The $d$ keeps the dot products from growing with dimension and saturating the softmax. The punchline is in the lab: this is kernel regression — Nadaraya–Watson — with the dot product as its similarity kernel. Drag the query and cool the temperature to watch a soft blend collapse into a hard nearest-key lookup.

Attention is a soft lookup. Each query pulls a weighted average of the values — weights are a softmax over similarity to the keys. It is kernel regression.

similarity

query q = 0.50temperature τ = 0.120

output ŷ(q) = Σᵢ wᵢ vᵢ = 0.567 · sharpest weight 38%. Warm: a smooth blend of the nearby keys.

The same move, as a matrix. Scaled dot-product self-attention over a toy sequence: row i is how token i attends to every token. A = softmax(QKᵀ/√d).

the

cat

sat

the

mat

the

cat

sat

the

mat

cat attends most to cat — the rhyme-and-noun pair light up because their embeddings align. No training here; the pattern is pure geometry of the vectors.

Multi-head, self, and cross

One head sees one kind of relation. Multi-head attention runs $h$ of these in parallel, each with its own learned projections $W_{Q}, W_{K}, W_{V}$ , so different heads specialize — syntax, coreference, position — then concatenate. Self-attention has every token attend to the whole sequence; cross-attention lets one sequence attend to another (a decoder reading an encoder). The cost is $O (n^{2})$ in sequence length $n$ — the bill the frontier is trying to cut.

3 · Softmax & sampling — living on the simplex

Softmax is the map from arbitrary scores to a probability distribution — a point on the simplex. A temperature $T$ divides the logits before exponentiating and sets how peaked the result is:

p_{i} = \frac{e ^{z_{i} / T}}{\sum _{j} e ^{z_{j} / T}}, T \to 0 : argmax (greedy), T \to \infty : uniform

A language model's last layer emits one logit per vocabulary token; softmax makes them probabilities; and the next token is sampled — with temperature, and usually a top- $k$ or nucleus (top- $p$ ) truncation to cut the long tail of nonsense. Low temperature is careful and repetitive; high temperature is creative and risky. It is the same softmax that weighted the attention above — the model's one nonlinearity for turning scores into choices.

Scores become choices. Softmax turns the model's logits into a distribution over the next token. Temperature sets how peaked it is; top-p keeps only the smallest set of tokens that covers a probability mass; then you sample.

temperature T = 1.00nucleus top-p = 1.00

mat

50.0%

floor

22.5%

sofa

13.6%

roof

6.1%

ledge

4.1%

keyboard

2.3%

moon

1.0%

idea

0.5%

greedy pick mat · entropy 1.41 nats ≈ 4.1 effective choices · nucleus keeps 8 of 8. Balanced.

4 · The transformer block — a stream of small edits

A transformer is one block, stacked dozens of times. Picture a residual stream — a running representation — that every sublayer reads from and adds back to. Each block does two things: mix information across tokens with attention, then mix it across features with a position-wise MLP (typically 4× wider, with a GELU). LayerNorm keeps the scales in check.

x \leftarrow x + MHA (LN (x)), x \leftarrow x + MLP (LN (x))

The residual $x + \dots$ is the quiet hero: it makes the network a sequence of small editsto a shared representation, so gradients flow cleanly through great depth (the same reason the gradient doesn't vanish) and adding layers helps instead of hurts. Everything a model “knows” is written into and read out of this stream.

hovering

Multi-head self-attention

x \leftarrow x + MHA (LN (x))

The only place tokens exchange information — several attention heads in parallel, each mixing ACROSS the sequence.

A transformer is this block, stacked — read the residual stream bottom to top, the two sublayers as small edits onto it.

5 · Autoregression & the loss — prediction is compression

A language model is a next-token predictor. The probability of a whole sequence factors by the chain rule of probability, one token at a time:

p (x_{1}, \dots, x_{T}) = t = 1 \prod T p (x_{t} ∣ x_{< t})

Training maximizes the log-likelihood of real text, which is minimizing the cross-entropy between the predicted distribution and the token that actually came next:

L = - t = 1 \sum T lo g p (x_{t} ∣ x_{< t}), perplexity = e^{L / T}

Perplexity is the effective branching factor — how many equally-likely next tokens the model is torn between. A causal mask zeroes attention to future positions so the prediction at $t$ can only use $x_{< t}$ . Nothing in here knows grammar or facts a priori; it is compression of the corpus, and the structure falls out because predicting well requires it.

6 · Diffusion — learning to reverse noise

The other generative paradigm, and the one behind most image and video models. Instead of predicting the next token, learn to undo noise. A fixed forward process adds a little Gaussian noise at each of many steps until the data is indistinguishable from static:

x_{t} = \overset{α}{ˉ}_{t} x_{0} + 1 - \overset{α}{ˉ}_{t} ε, ε \sim N (0, I)

A network is trained to predict that noise — equivalently, to estimate the score $\nabla_{x} lo g p (x)$ , the direction toward denser data. Generation runs the process backwards: start from pure noise and integrate the reverse SDE (or its deterministic ODE twin) down to a clean sample. It is the Gaussian and its geometry all the way down. Drag the lab below to dissolve a two-moons distribution into static, then Sample to watch a shape condense out of noise — and note the reverse step's denoiser is a softmax over the data: the same attention from chapter 2.

Forward, then back. Drag the noise level to dissolve the two-moons data into Gaussian static — then Sample to run the reverse process and watch a shape condense out of pure noise. The denoiser's guess for the clean point is a softmax-weighted average of the data: attention, again.

t = 0.00 · clean data

noise level t = 0.00

target dataparticles

Forward: x_t = √ᾱ_t x₀ + √(1−ᾱ_t) ε. Reverse: predict x̂₀ = Σ wᵢ x₀⁽ⁱ⁾ (a softmax over the data), take a DDIM step toward lower noise, repeat.

7 · The frontier — beyond quadratic attention

Attention's $O (n^{2})$ cost is the wall. The active mathematics is trading that quadratic for structure. State-space models (S4, Mamba) replace attention with a linear recurrence you can run as a convolution — $O (n)$ and streaming. Mixture-of-experts routes each token to a few of many MLPs, growing capacity without growing compute per token. And the scaling laws say the loss falls as a clean power law in parameters, data, and compute — an empirical regularity whose spectral and statistical explanation is still being written. Where this goes next is an open, and deeply mathematical, question.

Built from first principles, with live labs at each mechanism — attention, softmax, the transformer block, and diffusion. It is the applied-math spine of the research program, aimed squarely at how modern models actually work; the frontier chapter is where it keeps growing.