Rolling the Dice

Turning a ranked list of candidate tokens into one actual token is sampling. How boldly it plays the odds is governed by a single parameter: temperature.

Temperature

Temperature rescales the gaps between candidate scores before sampling, dividing every logit by $T$, then taking softmax. As $T \to 0$, probability mass collapses onto the single highest-scoring candidate; as $T$ grows past 1, the gaps shrink and more of the list becomes reachable.

The formula

$z_i$ is candidate $i$'s raw score. Temperature $T$ changes the contrast between scores before softmax. A lower $T$ enlarges the gaps, making the leading candidate more dominant, while a higher $T$ compresses them, giving alternatives more probability. The exponential converts the adjusted scores into positive weights, and the denominator scales those weights so the final probabilities sum to 1.

Temperature isn't the only rule

Two more parameters trim the candidate list before sampling, rather than reweighting it. Top-k keeps only the $k$ most likely candidates.^[4] Top-p (nucleus sampling) keeps the smallest set whose probabilities sum to $p$, so the cutoff shrinks when the model is confident and widens when it's unsure.^[5]

A fixed $k$ can't tell confidence from a toss-up, so it drags along implausible candidates when the model is certain and excludes near-equal options when it isn't. Top-p adapts to how peaked the distribution actually is at each step.

The limit of decoding

Temperature, top-k, and top-p all reshape a distribution the model already computed. None can introduce a candidate that wasn't already assigned probability, or correct a distribution that was wrong to begin with, since decoding only selects among existing candidates and never evaluates whether any of them are true.