When a fact appears only once
Training corpora follow a power law. A handful of entities are repeated constantly, most are sparse, and many arbitrary facts (birthdates, patent holders, minor events) appear only once, a singleton, or as Kalai and Vempala term it, a monofact.[6] Accuracy tracks frequency almost directly, since how often a fact showed up in training predicts, nearly linearly, how often the model gets it right.[7] For a calibrated model, that relationship has a floor. Kalai and Vempala show its hallucination rate on such arbitrary facts is close to its monofact rate (the share it saw exactly once) minus how miscalibrated it actually is:
One of OpenAI's SimpleQA questions makes the mechanism concrete:[29] "Which hill station is known as 'The Gateway to Ladakh'?" The correct answer, Dras, is obscure enough that Gemini gets it wrong when asked directly, confidently naming Kargil instead, a real, well-known town just down the road from Dras, not an invented name.
The empirical result behind this section applies to exactly this kind of fact, one that's rare, with little room in a real corpus to begin with:
Accuracy vs. document frequency
BLOOM-176B on TriviaQA, exact values as reported.
QA accuracy “jumps from 25% to above 55%” as relevant pretraining document count rises from 10¹ to 10⁴.[7]
A second, independent paper finds the same pattern a different way, tracking real-world popularity instead of raw document counts, on a different model:
Accuracy vs. entity popularity
GPT-3 on PopQA, exact values as reported.
“GPT-3 achieves 35% accuracy” overall on PopQA, but on the 4,000 questions about its least-popular subject entities, “GPT-3 davinci-003 has 19% accuracy.”[8]
Why the wrong answers weren't random
Wrong answers cluster around close neighbors (Gemini's own wrong guess above names Kargil, a real town a short drive from the correct answer, not a random invention) because parameters compress the corpus at fixed size, not one slot per fact; memorization scales log-linearly with duplication count and model size.[24] A fact seen once gets folded into the nearest better-supported pattern, and more parameters only lower that boundary, never remove it. Two mechanisms, one failure, and they compound: a fact too rare to commit to, or one compressed into a blurred neighborhood once seen.