With search, hallucinations are now missing context rather than blatantly incorrect content. Language models are nearly-perfect at copying content and similarly solid at referencing it, but they’re still very flawed at long-context understanding. Hallucinations still matter, but it’s a very different chapter of the story and will be studied differently depending on if it is for reasoning or non-reasoning language models.
Subscribe for new posts
Blog posts only. No commonplace entries. Never sold or shared.
Before these models were released, models were effectively limited to “just” next-token prediction based on the information in their pretraining data and with a hard knowledge cut-off date.
While base models still predict next tokens and knowledge cut-off dates are still very much a real thing, reasoning models can search and obtain new information needed to complete tasks.
Lambert breaks down reasoning models into three primitives that distinguish them from their static predecessors: thinking (i.e., dynamically using more tokens to arrive at an answer), searching, and acting / executing.