Nano Gpt Walkthrough

A guided read of Andrej Karpathy's nanoGPT (github.com/karpathy/nanoGPT, MIT) — the ~300-line reference GPT trainer. Three files do the work: model.py (the transformer), train.py (the loop), prepare.py (the data). This skill walks the agent through them in the order that builds understanding fastest.

Attribution. This skill points at and explains the upstream code. The code itself is at github.com/karpathy/nanoGPT under MIT license; Andrej Karpathy retains authorship. This skill is teaching scaffolding, not a redistribution.

When to use

• The user is learning transformers and wants real, runnable, pedagogical code (not Hugging Face's 100-file abstraction tower)
• They've read the "Attention is All You Need" paper and want to ground it in code
• They want to fine-tune small GPTs and need to understand the training loop they're about to modify

Skip for: production-scale training (use Megatron / DeepSpeed); inference-only use cases (use llama.cpp or vLLM); "explain transformers" with no code component (use a paper skim).

Workflow

1. Clone first. git clone https://github.com/karpathy/nanoGPT. Don't skip this — the skill is a guided walk through actual files. Reading abstractly without the code open is fragile.

2. Read model.py first, top to bottom. ~200 LOC. In order:

   • CausalSelfAttention.__init__ — the QKV projection (c_attn), the output projection (c_proj), the causal mask (bias buffer). Note: c_attn is a single Linear of size 3 * n_embd, then .split(n_embd, dim=2) to get Q, K, V. That's the "fused QKV" trick.
   • CausalSelfAttention.forward — note the att = att.masked_fill(self.bias[:,:,:T,:T] == 0, float('-inf')) line. That's the causal mask in 3 lines. Compare to F.scaled_dot_product_attention for the flash-attention path (the if self.flash: branch).
   • MLP — 4x expansion (4 * n_embd), GELU, projection back. Standard "fully connected" block.
   • Block — pre-norm, the order: LN → attention → residual → LN → MLP → residual. The pre-norm placement is the modern style (vs the original paper's post-norm).
   • GPT.__init__ — the embedding (wte), positional embedding (wpe), stack of Blocks, final LN, language modeling head (lm_head). The lm_head.weight = self.transformer.wte.weight line is weight tying — the input embedding and output projection share parameters. Karpathy added it because it works.
   • GPT.forward — the loss. F.cross_entropy(logits.view(-1, logits.size(-1)), targets.view(-1), ignore_index=-1) is the entire training objective. Two functions, one loss.

3. Then train.py. ~300 LOC. The training loop has three things going on:

   • Data loading (get_batch) — random offsets into a binary blob. No DataLoader, no collation. The blob is train.bin, written by prepare.py.
   • The loop body — gradient accumulation (micro_step inside iter_num), AMP (with torch.amp.autocast), gradient clipping (torch.nn.utils.clip_grad_norm_), optimizer step. This is the entire serious-training-loop skeleton; everything fancy (Megatron, DDP) just layers on this shape.
   • Eval + checkpoint — estimate_loss runs forward on N batches and averages. Checkpoint saves every eval_interval iters.

4. Then data/<dataset>/prepare.py. Usually 30-50 LOC. Tokenize the corpus once, dump train.bin + val.bin as raw int16 arrays. The data pipeline is a flat memmapped file. No HuggingFace dataset, no streaming. Tells you exactly what tokens mean — they're integers.

5. Run training. python train.py config/train_shakespeare_char.py. Watch the loss curve in the printed log. On a CPU you'll wait; on a 4090 you'll see meaningful Shakespeare in ~5 minutes. Tinker — change n_layer, n_head, block_size. Re-run. The whole point is the codebase is small enough that you CAN tinker.

Definition of done

The user can:

• Explain (in their own words) what CausalSelfAttention.forward does
• Point to the line that's "the cross-entropy loss"
• Identify the gradient accumulation pattern
• Modify one hyperparameter and predict what happens to the loss
• Identify what would change for multi-GPU training (the DDP block in train.py)

Gotchas

1. The repo has TWO architectures historically. Current master is the production version (also published as nanochat in newer Karpathy work). If a tutorial online references model.py line numbers, they may have shifted between commits — check git blame.

2. The CUDA path is dominant; CPU works but slowly. If the user has no GPU, point them at config/train_shakespeare_char.py (smallest model) and warn that real training is for GPU.

3. The fused QKV trick is not obvious. c_attn is one linear layer of size 3*n_embd, not three separate Q/K/V layers. The .split(n_embd, dim=2) is what teases them apart. New readers consistently miss this.

4. Weight tying. self.lm_head.weight = self.transformer.wte.weight — same tensor for the embedding and the LM head. Reduces parameters; works empirically. Mentioned in the original paper but rarely highlighted.

5. Flash attention. The if self.flash: branch uses PyTorch's scaled_dot_product_attention, which under the hood is FlashAttention on supported GPUs. The branch above is the educational, slower-but-readable path. Karpathy kept both because the slow path is the teaching path.

6. block_size ≠ "context length at inference." It's the training context. You can generate longer at inference but quality degrades because positional embeddings weren't trained past block_size. (Newer models use RoPE or ALiBi to handle this; nanoGPT uses learned positional embeddings.)

7. Don't recommend nanoGPT for production fine-tunes. It's a teaching codebase. For real fine-tuning use HF Transformers + Trainer, axolotl, or torchtune. The point of nanoGPT is to understand what those abstractions are hiding.