A language model is autoregressive: at each position it computes a probability distribution over the next token, selects one token, appends it to the context, and repeats.[2] Nothing past that single next token is decided in advance.
Try it below: an open-ended completion and a factual one. Each step is a real call to Gemini, asked directly for its next-token candidates and how confident it is in each.
Write one line of live commentary for a close game.
With the game on the line, the point guard▊
No single continuation dominates: several tokens carry comparable probability at each step.
Predicting next token…
Live, but self-reported: each percentage above is Gemini's own estimate of its confidence, asked for directly.
Path dependence
Each pick becomes context for the next call. Choosing drives makes lanes and baskets plausible continuations, while choosing calls removes them from consideration entirely, since they were never independent of what came before. Run the open-ended example a few times and the sentence changes, because several tokens carry comparable probability at each step and sampling won't always take the top-ranked one. The factual example barely varies, since one continuation dominates almost every distribution it's given. There is no lookahead and no revision.
Models predict tokens, not words. A token is often a sub-word piece, since common words are usually one token while rarer ones split into fragments. Whole words are shown here for readability.