Topic

FlashAttention

IO-aware, tiled attention kernels.

2 checkpoints

2026-06-24SIGNALS

FlashAttention-3: async, low-precision, Hopper-native

The headline is hardware-specific: FA3 is a Hopper story (async copy/MMA overlap, FP8 paths). The portable lesson from the FA line is still the one that matters — attention is bandwidth-bound, and the win is in HBM traffic, not FLOPs.

CHECKPOINT 00052026-05-28RECREATIONSscope: minimal

Recreating FlashAttention: A Tiled, IO-Aware Attention Kernel from Scratch

FlashAttention is exact attention restructured for the memory hierarchy, not an approximation. We implement the tiled forward and recompute backward in Triton, validate exactness against a reference, and separate what a tutorial actually reproduces from what needs CUTLASS-grade engineering.

flash-attention kernels attention gpu-memory