I received a DM earlier this week that was generally asking about how does one “get into compilers and ML compilers specifically”. This is what I answered:
re: trying to break into ML compilers specifically, idk if I have any good advice outside of:
- reading the codebases of smaller ML compilers
- tinkering or hacking on your own very small ML compiler from intuition and then comapre to the state of the art and see where you differed and why
- you can also take a look at things like Tinygrad and overall docs like Tinygrad with the codebase side to side, and explore, etc….
- there’s also lots of things to do with things like pytorch, they have tons of things to introspect how a given piece of pytorch gets “compiled” by torch.compile, you can inspect nearly any parts of that process
- familiarizing yourself with MLIR and how it’s used by ML compilers, being comfortable with how it fits into the pipeline of “some model defined in some ML python library” -> convert it to a graph -> optimize that graph -> assemble the kernels and ops that will execute the computation laid out by the model
- there are a bunch of toy ml compilers you can take a look at and some excellent ressources for MLIR (including their own docs and some great writeups by Jeremy Kun, Stephen Diehl etc)