Deep learning · /transformer

Transformer

A toy causal transformer with every intermediate on display. This mode is a hand-wired induction circuit: a previous-token head feeding an induction head. Give it a repeated pattern and it really completes it — no training involved, just the two attention heads doing their textbook job.

Sequence

⟨s⟩moonstartreemoonstar
τ0.50

Next-token distribution

tree
99.4%
star
0.2%
moon
0.0%
⟨s⟩
0.0%
rain
0.0%
fish
0.0%
bird
0.0%
rock
0.0%

Model

modeseedd = 56 · 2 layers · 2 heads · causal

layer 1 · head 1 — rows attend to columns; click a cell to bind the formula

layer 2 · head 1 — rows attend to columns; click a cell to bind the formula

What's simplified here
  • No LayerNorm and no MLP/feed-forward blocks — attention-only, so the circuit stays legible.
  • Mechanics mode is untrained (seeded random weights): real computation, meaningless predictions — that's the point.
  • Induction mode is hand-wired, not learned: orthonormal token/position bases and two exact heads (previous-token → induction), with a BOS token absorbing the position-0 artifact.
  • Word-level vocabulary of 12 tokens; max sequence length 24.