AI safety and LLM evaluation
I work on AI safety and AI evaluation. I'm a Principal TPM for Agentic Engineering in the Office of the CTO at Microsoft. Before that I was head of Responsible Technology at Indeed, where my team owned pre-deployment bias detection and evaluation for LLMs, LLM guardrails, and red-teaming. This page gathers what I've written on making AI systems safe to ship and the notes I've kept on evals, hallucination, misalignment, and the safety teams at the labs.
Essays
- Solving LLM hallucinations is (mostly) a UX problemWhy LLM hallucinations are primarily a UX challenge rather than an algorithmic one, and how design improvements can mitigate their impact on users.
- Quantification biasQuantification bias: an AI system whose errors can be measured is judged more harshly than the human process it replaces, even when it performs better.
- Skeptical optimists are the key to an abundant AI futureBeyond doomers and accelerationists: why skeptical optimists who believe AI problems are solvable with effort are essential for building an abundant AI future.
- Building Successful Responsible AI TeamsA practical guide to building responsible AI teams that succeed by aligning with business goals, hiring builders over framework-creators, and focusing on empirical risk management.
- Responsible AI is dying. Long live responsible AI.How you, yes you, can help build a better future with AI.
Notes on AI evals
Notes on hallucination, misalignment and model behavior
Notes on safety teams and the labs
More
Responsible AI and LLM evaluation gathers the writing on responsible AI teams. Talks, podcasts and press. Notes tagged evals and alignment. To talk about AI safety or AI evals work, get in touch or book a call.