Topic

Pretraining

Large-scale next-token pretraining.

2 checkpoints

CHECKPOINT 00102026-06-15REPRODUCTIONSIn progress

Reproducing the nanoGPT Speedrun: What Actually Moves the Loss Curve

The nanoGPT speedrun is a rare, fully open optimization target: hit 3.28 FineWeb validation loss on a GPT-2 (124M)-class model in minimum wall-clock on 8×H100. We reproduce the pipeline, isolate what the Muon optimizer and the architecture changes actually buy, and flag what will not transfer off the bench.

2026-06-11SIGNALS

The modded-nanogpt speedrun and the Muon optimizer

A rare fully-public optimization target with a reproducible harness — exactly the kind of artifact we like. The Muon optimizer it popularized is the most interesting practical idea to come out of it.

pretraining optimization reproducibility