3. AI and ComputingNo. 03.06
Transformer
A transformer is a neural network first described in 2017, for translating text.
Handling
Tap it to watch a prepared next-word example. Try the Encoder–decoder diagram in the Splash tab.
Its tag
- Shelf
- 3. AI and Computing
- Kind
- Built from a recipe
- Made from
- Encoder–decoder key: ◉ attention · ◒ masked attention · + add & norm · » feed forward · ⠿ embedding · ∿ positional encoding · ╱ linear · ▁▃▇ softmax · N× repeated N times. HELLO WORLD goes in, and the prepared decoder example, given START HOLA, ends with MUNDO. Its words and attention strengths are stored.
- Sound
- Yes
- Licenses
- Own work (made in the page)
On the wall it weighs 6.1 KB: 504 splats, fitted to pictures of the full Splash, which opens with 500,003.
Its controls
| Control | Kind | Range | Start |
|---|---|---|---|
| Predict the next word | Tap | ||
| Predict | Button | ||
| Diagram | List | 2 choices | Token flow |
| View | List | 2 choices | Poster (2D) |
About it
A transformer is a neural network first described in 2017, for translating text. It splits text into tokens, and in each layer every token uses attention: it looks at all the others and weighs how much each one matters to it. Language models built this way predict the next token.
Tap it: arcs jump between THE CAT SAT ON, thicker where attention is stronger, one color for each of two heads; the tiles rise through the layers, and MAT drops in. The Encoder–decoder diagram in the Splash tab is the classic design. Its key: an eye is attention; a lidded eye, masked attention (it only looks back); +, add and norm; », feed forward; a vector, embedding; a wave, positional encoding; a slash, linear; bars, softmax; N×, repeated N times. HELLO WORLD goes in, and given START HOLA, the prepared example ends with MUNDO. The words and attention strengths are prepared; this view does not calculate attention or predict text.
Details
- Splats
- Opens at 200,000 at Mid Detail, 120,002 at Low, and up to 720,000 at Max
Is it right?
The classic encoder-decoder diagram names the main parts of the 2017 transformer correctly. The words and attention arcs are prepared examples; no attention matrix, translation, or next-word inference is calculated.
Correct
The classic diagram follows the 2017 encoder-decoder architecture.1
The box labels include embedding, position, masked self-attention, encoder attention, add and norm, feed-forward, linear, and softmax.
Simplified
The splash presents prepared next-word and translation examples.1
MAT and the MUNDO tile label are literal outputs, and arc strengths are stored. The How-to and About now say that these are prepared examples.
Simplified
The first attention picture is schematic rather than a causal decoder calculation.1
Some arcs lead to later token positions; there is no QK product, mask, normalization, or learned attention value.
- 1Attention Is All You Need, Vaswani and coauthors, 2017
- Simplified
- MAT and MUNDO are stored outputs. Arc strengths are fixed, not attention probabilities. The first picture permits links to later tokens, so it should not be read as causal decoder attention.
Sources and credits
Made in Splashery. Its code is MIT.
