AI research interviews

Bas van Opheusden, recent addition to the technical staff at OpenAI, shares his advice for preparing for AI research lab interviews. It pains me that we occupy an equilibrium where leetcoding is broadly seen as suboptimal for identifying talent yet is the dominant form of technical interview. Classic streetlight effect.

These guides can often feel like advice to “know everything about everything” (I should know, I wrote one myself some years ago) but there are some good nuggets:

Concrete interview topics to prepare:

• Debugging transformers. This is a classic in which you get a botched implementation of a self-attention block and have to debug it. Make sure you’ve practiced debugging tensor shapes, and pay special attention to the causal attention mask - that’s where it gets most tricky. For reference, check out nanoGPT.
• Top-k/knn. The problem of “picking the k largest items” comes up in ML in various places and makes for a nice interview problem, particularly because the solution (a heap) is not something you can invent on-the-fly. Make sure you know what heaps are.
• Implementing BPE. Tokenizers are the worst part of LLMs, and implementing BPE without errors is tricky. This is somewhat popular.
• Backpropagation from scratch. Implementing a basic version of auto-diff, the chain rule etc. Lots of opportunities for indexing errors.
• KV Cache. This essentially amounts to building a matrix, but if you haven’t seen it before, you might do something convoluted
• Binary search, backtracking, dijkstra, ...