Transformers

Transformers are neural network architectures built around attention. They were first popularized for sequence modeling, but the same core idea now appears in language, vision, audio, robotics, and multimodal systems.

The central move is simple: instead of processing a sequence strictly left to right, let each token decide which other tokens are relevant.

Tokens and embeddings

Text is first split into tokens. Each token is mapped to a vector:

"gradient" -> token_id -> embedding vector

The model does not directly understand words. It manipulates vectors, and training shapes those vectors so useful relationships become available to the network.

Self-attention

Self-attention creates three vectors for each token:

Attention scores are computed by comparing queries with keys:

$$ \operatorname{Attention}(Q, K, V) = \operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V $$

The softmax turns scores into weights. The output is a weighted combination of value vectors.

Tiny attention sketch

import numpy as np

def softmax(x):
    x = x - np.max(x, axis=-1, keepdims=True)
    exp = np.exp(x)
    return exp / exp.sum(axis=-1, keepdims=True)

def attention(Q, K, V):
    d_k = Q.shape[-1]
    scores = Q @ K.T / np.sqrt(d_k)
    weights = softmax(scores)
    return weights @ V

This is not a complete transformer, but it captures the main mechanism.

The transformer block

A typical block has:

1. Multi-head self-attention.
2. A residual connection.
3. Normalization.
4. A feed-forward network.
5. Another residual connection.
6. Another normalization.

Stack many blocks and the model can build increasingly abstract representations.

Why multi-head attention?

One attention head can focus on one pattern. Multiple heads let the model attend to different relationships at the same time.

For language, one head might track syntax, another might connect pronouns to nouns, and another might focus on local phrase structure. The model is not guaranteed to divide labor so cleanly, but that is the useful intuition.

Causal masking

For generation, the model should not see future tokens. A causal mask blocks attention to later positions:

token 1 can see: token 1
token 2 can see: token 1, token 2
token 3 can see: token 1, token 2, token 3

That constraint lets the model predict the next token without cheating.

Why transformers scaled

Transformers became dominant because they combine several practical advantages:

• Attention can model long-range dependencies.
• Blocks are highly parallelizable during training.
• The architecture scales predictably with data and compute.
• The same template works across many modalities.

The tradeoff

Attention is powerful, but it can be expensive because full attention compares every token with every other token.

$$ \text{full attention cost} \propto \text{sequence length}^2 $$

That is why long-context modeling, sparse attention, retrieval, recurrence, and state-space alternatives remain active areas of engineering and research.

Mental model

A transformer repeatedly asks:

For each token, what other tokens matter right now, and how should their information update this representation?

That repeated question is enough to build surprisingly rich behavior when trained at scale.