Scored like an exam
A model choosing whether to answer, guess, or abstain is playing a scoring game, whether or not anyone designed it that way. Score a correct answer +1, a wrong one 0, and "I don't know" also 0, and the arithmetic is unambiguous. Any nonzero chance of being right has positive expected value, so the rational move is to always guess, the same trade-off as a multiple-choice test with no penalty for a wrong answer. Change the wrong-answer score to −1 and the arithmetic flips. Guessing only pays once confidence clears 50%, and abstaining below that line scores higher. Layer a second pressure on top, entirely outside the scoring rule: a user who insists on a definite answer. Run ten questions below and compare two copies of the same uncertain model (one neutral, one pressured) drawing the line for answering in different places:
Neutral
Answers only once confidence clears the rational threshold for this rule: 0%.
Pressured
Same rule, but answers down to 0% because a user pushed back: “Please give me a definite answer, don’t say you’re unsure.”
Click Run to simulate all 10 questions under both policies at once.
10 real questions from SimpleQA[29], with fixed confidence and a fixed correct/wrong outcome if answered; only the threshold for answering differs between policies.
None of this is hypothetical
RLHF fine-tunes a model on human rankings of paired outputs.[25] Raters can't verify an answer on the spot, so they reward confidence and completeness, not correctness. As a result, GPT-4 hallucinates less after RLHF, but its stated confidence stops tracking its actual accuracy.[6] User pressure does the same thing: models shift answers toward a user's stated belief or pushback, trading accuracy for agreement, across every major assistant tested.[26]
The −1 rule above isn't hypothetical either. Kalai, Nachum, Vempala, and Zhang propose it for the benchmarks that make up most leaderboards.[46] Almost all of them still grade +1/0/0, rewarding a guess over an admission of uncertainty. Their fix: state an explicit confidence target in the grading instructions, and penalize a wrong answer points. At , that's exactly the −1 above, applied to the scores post-training already optimizes against, so bluffing stops paying.