Simo Wu
Notes on machine learning, robotics, and other things worth writing down.
This is the personal site of Simo Wu. I work on machine learning and robotics. This site collects notes, blog posts, and the occasional longer-form writeup.
The current top of the stack is a three-part series on the residual + in transformers — what it does, what happens when you replace it with a learned attention over depth, and how to retrofit that into a pretrained model:
- Attention, the Residual Stream, and What’s Hiding in the “+”
- Attention Residuals: Making the “+” Learned
- AR-Retrofit and ReSkip: Attention Residuals on a Frozen Backbone
For everything else, see the blog.