Your AI Isn't “Wrong”; It Never Knew.
This is part one of three posts on using AI for research-heavy knowledge work. Stay tuned!
A friend of mine, a theoretical physicist, recently told me that when he asks AI about topics he's familiar with, it "often makes conceptual errors and presents them with complete confidence." And when he asks about topics he's not familiar with, it gives great-sounding answers where the errors are harder to catch.
He's not wrong. But his framing reveals a common trap: the assumption that the AI knows things and occasionally gets them wrong. That's not what's happening.
Here's a better mental model. Imagine a student who didn't do the assigned reading. The professor asks a question. The student doesn't say "I don't know." Instead, they cobble together a response from half-remembered lecture fragments and the general shape of what a good answer sounds like. Sometimes they nail it. Sometimes they don't. But they have no idea which is which.
That's your AI.
Large language models don't retrieve facts from a knowledge base.* They predict what words are likely to follow other words, based on statistical patterns in their training data. When the question sits squarely within well-trodden territory, the statistically likely answer and the correct answer tend to overlap. When it doesn't, like when you're asking about niche research, recent developments, or subtle conceptual distinctions, the model keeps generating plausible-sounding text. It just stops being right.
This is why the term "hallucination" bugs me. It implies a malfunction: the AI was operating normally, and then briefly lost its grip on reality. But that's not what’s happening. The AI never had a grip on reality. It produces plausible text. Sometimes, plausible text happens to be true. The architecture is the same either way.
The practical upshot? Don’t treat AI like an oracle that occasionally glitches, and treat it like a fast, articulate collaborator that has read a lot but understood less than you think.
If you're an expert in the field you're asking about, you have a built-in fact-checker: your own knowledge. Use AI to draft, rephrase, and explore, but verify anything that matters. If you're not an expert, that's exactly where the danger lies. The output reads just as confidently whether it's right or wrong, and you don't have the filter to tell the difference.
The confidence isn't a feature or a bug. It's the only mode the machine has.
*If they do, it’s because extra engineering work has gone into appropriate retrieval systems.
