Responsible AI and LLM evaluation
I was head of Responsible Technology at Indeed, where I led Responsible AI. I built the responsible AI function from one data scientist into a cross-functional team of data scientists, engineers, researchers, and product managers. My team owned pre-deployment bias detection and evaluation for LLMs, LLM guardrails, and red-teaming. This page gathers what I've written on that work and the notes I've kept on AI evals.
Essays
- Building Successful Responsible AI TeamsA practical guide to building responsible AI teams that succeed by aligning with business goals, hiring builders over framework-creators, and focusing on empirical risk management.
- Responsible AI is dying. Long live responsible AI.How you, yes you, can help build a better future with AI.
- Quantification biasQuantification bias: an AI system whose errors can be measured is judged more harshly than the human process it replaces, even when it performs better.
- Solving LLM hallucinations is (mostly) a UX problemWhy LLM hallucinations are primarily a UX challenge rather than an algorithmic one, and how design improvements can mitigate their impact on users.
- Skeptical optimists are the key to an abundant AI futureBeyond doomers and accelerationists: why skeptical optimists who believe AI problems are solvable with effort are essential for building an abundant AI future.
Notes on AI evals
Notes on transparency and critique
More
AI safety and LLM evaluation gathers the writing on AI safety, hallucination and misalignment. Talks, podcasts and press on responsible AI and LLM evaluation. Notes tagged evals and alignment. To talk about responsible AI or AI evals work, get in touch or book a call.