Ask the same question five times
Ask the same question several times. Stable facts tend to recur; uncertain details tend to drift.[35]
SelfCheckGPT turns that into a check. Take one answer a model already gave, sample several fresh answers to the same prompt, and see which of the original answer's claims the fresh answers back up and which they don't.[35] No access to the model's internals or any outside source is required, so it works even on a fully black-box system. Try it below.
Prompt
“Write a short biography of Giuseppe Mariani.”
Answer under review
Giuseppe Mariani was an Italian violinist, born in Milan, who died in Rome in 1987.
Profession: violinistBirthplace: MilanPlace of death: Rome
Agreement is a plain string match against each sample’s claim, simplified from SelfCheckGPT’s real NLI/prompt-based scoring. “Flag for verification” means a claim didn’t reproduce reliably, not that it’s confirmed false.
Agreement can still be wrong
Prompt: “How tall was Napoleon?”, asked five times
Sample 1
Napoleon Bonaparte was notoriously short, only around 5 feet 2 inches tall.
Sample 2
Napoleon's short stature — roughly 5'2" — became legendary.
Sample 3
Standing just 5 feet 2, Napoleon was famously short for his time.
Sample 4
Napoleon was a short man, about 5'2", which fed the ‘Napoleon complex.’
Sample 5
At only 5'2", Napoleon Bonaparte was remarkably short.
Agreement measures stability, not truth. A familiar misconception can be repeated consistently.[9]
What the paper found
Detecting non-factual sentences
AUC-PR: how well a detector ranks non-factual sentences above factual ones. Higher is better; 100 is perfect.
| Method | AUC-PR |
|---|---|
| GPT-3 Avg(-log p)token-probability baseline | 83.2 |
| SelfCheckGPT-NLIresample & compare | 92.5 |
| SelfCheckGPT-Promptresample & compare | 93.4 |
Sampling and comparing answers beat relying on token probabilities alone.
238 GPT-3-generated WikiBio passages, human-annotated.[35]
SelfCheckGPT treats disagreement as an uncertainty signal. It works with black-box models, but repeated sampling adds generation cost and consistency is not proof of truth. Disagreement is a reason to verify; agreement only shows that the model is reproducing the same answer.