Plausible ≠ True

What training rewards

Next-token pretraining predicts the next token, compares it to the observed one, and adjusts parameters accordingly, repeated at scale. The objective never checks whether a prediction is true, only whether it matches. For facts occurring once or twice in the corpus, this has a proven consequence: a model whose probabilities track training frequency must assign nonzero probability to false completions of those sparse facts, and will therefore sometimes generate them.^[6]

Why it gets so much right

Plausibility and truth correlate because accurate claims are repeated more consistently than any single error, so predicting the most frequent continuation usually predicts the correct one; accuracy on a fact tracks its training frequency, nearly linearly.^[7] This correlation, not a guarantee, is the primary reason the models are useful, since the objective itself does not distinguish a well-supported claim from a confidently wrong one.

Familiarity is not verification

Familiarity and truth are separate signals. A claim can be repeated frequently and still be false, or true and rarely stated. Sort these five claims before checking any of them:

When the two come apart

Plausibility and truth diverge wherever a false claim occurs more often in text than its correction: outdated facts that outnumber their updates, misconceptions that outnumber their corrections, fiction and speculation stored alongside reference text with no reliable label distinguishing them. Models measurably absorb these imbalances, and larger models reproduce them with more confidence, since scale improves imitation rather than correction.^[9] This is not deception. The model applies its training objective in a case where the frequent continuation and the correct one no longer coincide.