Why one wrong token becomes a paragraph
Training teaches next-token prediction against a ground-truth prefix, where every step conditions on what actually came next. Generation gets no such luxury. Each token conditions on whatever the model itself produced a moment ago, right or wrong, a train/test mismatch called exposure bias.[27] Nothing in that loop distinguishes trustworthy context from invented context, so a false claim, once generated, becomes as much a fact to build on as a true one, since the same fluency-seeking objective that makes prose coherent also makes it consistent with whatever's already there. One bad token doesn't stay one token. Every later token conditions on it, and staying coherent with an error reads more fluent than contradicting it. Watch it happen:
Pick which false name to inject in place of the true one, and two live continuations run from otherwise-identical prefixes (clean and poisoned) so anything that differs traces back to that one swapped word.
Checkpoint
The theory of evolution by natural selection was developed by Charles…
The model is told not to correct the false premise: a real generation can never revise a token once it’s committed, and a well-aligned model would otherwise just catch and fix an obvious error. The dependent-claim labels are its own follow-up classification, not verified fact.
Ask again, separately, and it often knows
That gap between generating and checking is measurable. Across several question-answering tests, models that stated a wrong answer and then justified it (inventing a false supporting claim along the way) were asked afterward, separately, whether that same claim was true on its own. GPT-3.5 caught its own invented claim 67% of the time; GPT-4, 87%; LLaMA2-70B-chat, 94%.[28] The authors read this as evidence against a knowledge gap. The model has the fact, but next-token prediction only asks a token to follow the ones before it, never to check them, so staying consistent with an earlier claim wins out over contradicting it. Prompting the model to reason before answering helps on easier questions, but the effect doesn't transfer to every model or to harder questions: a fix inside the same generation isn't reliable, so verification has to come from outside the sentence being produced.