al-folio

a simple whitespace theme for academics

Attention Residuals: Making the "+" Learned

Part 2 — replacing the fixed `+` with a softmax over prior block outputs.

9 min read · May 17, 2026

2026 · transformers attention adaptive-computation · ml
Attention, the Residual Stream, and What's Hiding in the "+"

Part 1 — what the residual `+` in a transformer is actually doing, and why we should care.

8 min read · May 17, 2026

2026 · transformers attention residuals · ml
AR-Retrofit and ReSkip: Attention Residuals on a Frozen Backbone

Part 3 — retrofitting attention residuals into a pretrained decoder, and using the routing weights to skip blocks at inference.

16 min read · May 17, 2026

2026 · transformers attention adaptive-computation peft · ml
Some Nonsense About Bitcoin

Notes on the strong correlation between bitcoin price and mining difficulty.

9 min read · January 24, 2018

2018 · bitcoin economics · archive