Topic

Long Context

Extending and stabilizing context windows past native length.

2 checkpoints

2026-06-20SIGNALS

Mamba and the selective-state-space line

Worth understanding even if you ship transformers: SSMs change the asymptotics (linear in sequence length, constant state at inference) and the failure modes. The interesting deployments are hybrids, not pure-SSM.

CHECKPOINT 00042026-05-24EXPLAINERSadvanced

RoPE and the Long-Context Stack: Rotation, Interpolation, and What Breaks at 128k

RoPE turns position into a per-dimension rotation — and that same rotation is why PI, NTK-aware scaling, and YaRN exist, and why a 128k window rarely means 128k of usable context. The math, the methods, and the serving bill.

rope long-context attention transformers