Essays, notes, papers, and resources by Ludwig Abap.

Essays, notes, papers, and resources by Ludwig Abap.

Many new Transformer architecture improvements have been proposed since my last post on “The Transformer Family” about three years ago. Here I did a big refactoring and enrichment of that 2020 post — restructure the hierarchy of sections and improve many sections with more recent papers. Version 2.0 is a superset of the old version, about twice the length. Notations Symbol Meaning $d$ The model size / hidden state dimension / positional encoding size.

I wanted to make another cube drawing so I could have it in the same orientation as the other CMYK machine and hand drawings which will make up a selection for art of my exhibition. I also wanted t…

In the previous post I defined the notion of “learning from scratch” algorithms—a broad category that includes, among other things, any randomly-initialized machine learning algorithm (no matter how complicated), and any memory system that starts out empty. I then proposed a division of the brain into two parts based on whether or not they learn from scratch. Now I’m giving them names: