AI’s Swiss Cheese Capabilities
Another gem from Andrej Karpathy's Deep Dive into LLMs like ChatGPT is his likening of AI capabilities to Swiss cheese: Delicious all around but with big holes in it.
Imagine conversing with someone for the first time at a social event. You discuss some current affairs topics, and the person comes across as genuinely knowledgeable and thoughtful. Then you switch topics to sports, and not only do they not know what soccer is, they don't even know what a ball is. This is quite unthinkable for humans, but commonplace with AI.
Now it's true that people can be brilliant in certain areas and lacking in other, yet we have built up an intuition of what we can expect from each other based on demonstrated capabilities.
The "Swiss Cheese" model of AI capabilities tells us that, with generative AI, we cannot rely on that intuition. These models can stun us with super-human performance in one domain and then frustrate us with basic mistakes and gaps in another.
The takeaway for running a successful AI initiative is that each new task and application of AI requires its own set of evaluations. You cannot rely on simple interpolation: "If it knows X and Z, then clearly it must know Y".
This is true both at the macroscopic and microscopic level:
Just because an AI model is good at writing code, which requires sound structured thinking, doesn't mean it'll be good at other tasks that require sound structured thinking, such as writing legal documents.
Just because the AI can do specific coding tasks surprisingly well doesn't mean it won't make mindbogglingly basic mistakes in others.
While these hilarious mistakes sometimes make for viral tweets, they'll also erode trust in the AI tools that rely on them. Better to check with solid evals and good testing how well AI performs on your tasks before blindly trusting that it should know what it's doing.
