Fluency is not a truth signal
Every wrong answer you've ever read from a model arrived in the same tone as every right one. No hedge, no typo, no stumble. There's no tell.
Humans have spent our entire history using fluency and confidence as a proxy for competence — and until very recently it was one. Producing confident, articulate, specific prose required actually knowing things. That link is now broken, and your instincts haven't been notified.
The trap that names this topic
Plot your ability to catch a mistake against your expertise, and it isn't a straight line:
- Your own domain — a wrong claim jars instantly. You catch it. Low risk.
- Total ignorance — you know you can't evaluate it, so you go and verify. Low risk.
- Adjacent competence — you know enough to follow along, not enough to catch a subtle error. Maximum risk.
Almost all real damage lives in that middle band. The analyst reading legal language. The engineer reading the marketing claim. The manager reading a technical spec. Not ignorance — partial knowledge, which feels exactly like knowledge from the inside.
There's a name for the intuition: the Gell-Mann amnesia effect. You read an article about your own field and see it's riddled with errors. You turn the page and believe the next article completely.
Now do that with a model, fifty times a day.
The counter is a rule, not a feeling:
Your confidence in an output should track your ability to check it — not how good it sounds.