Topic

Transformers

The transformer architecture and its components.

2 checkpoints

CHECKPOINT 00062026-06-01EXPLAINERSadvanced

Routing Is the Hard Part: A Practitioner's Guide to Mixture-of-Experts

MoE decouples parameter count from per-token FLOPs, but every hard problem — instability, dropped tokens, load imbalance, all-to-all traffic, a footprint set by total not active params — lives in the router. A structural tour from Switch/GShard to fine-grained and aux-loss-free designs, and the systems bill you actually pay.

CHECKPOINT 00042026-05-24EXPLAINERSadvanced

RoPE and the Long-Context Stack: Rotation, Interpolation, and What Breaks at 128k

RoPE turns position into a per-dimension rotation — and that same rotation is why PI, NTK-aware scaling, and YaRN exist, and why a 128k window rarely means 128k of usable context. The math, the methods, and the serving bill.

rope long-context attention transformers