GPQA Diamond Benchmark Explained: 198 Questions and Why Small Gaps Mislead

GPQA Diamond is a 198-question, four-option multiple-choice set that keeps only questions where domain experts agree on the answer and most non-experts with web access get it wrong. Artificial Analysis reports, as of 2026-10-06, 65% accuracy for experts on the wider GPQA set, 74% after discounting mistakes the experts identified later, and 34% for skilled non-experts with unrestricted web access. At 198 items, one question is worth half a point, so a single score carries a band of several points and small gaps between models say less than they appear to.

What does GPQA Diamond measure?

GPQA Diamond measures whether a model can answer graduate-level science questions that resist a quick search. The name comes from the paper title, "GPQA: A Graduate-Level Google-Proof Q&A Benchmark," whose citation appears in the project repository. The questions cover biology, physics and chemistry, and each has four answer options, so chance is 25%.

Diamond is a filtered subset of the wider set. Ultralytics describes it as a more selectively filtered, 198-question subset of the main GPQA set, which has 448 questions. Any score or human figure should name which of the two it refers to.

How were the 198 questions selected?

Selection needs two conditions. Both domain expert annotators must answer correctly, and most non-domain experts must answer incorrectly, as Epoch AI describes the subset. A paper record on OpenReview adds that humans later validated the difficulty and objectivity of the Diamond subset in a follow-up analysis.

This filter is the benchmark's main validity argument. It selects for items where experts agree and the crowd fails, which makes the questions hard to guess by search. Agreement between two annotators is still a check, not a proof: two annotators can share an error, and the filter does not show that every answer key is right.

What do humans score on GPQA?

Human baselines give the benchmark its reference points. The table below uses the same columns for each group, so the rows can be compared directly.

The expert and non-expert rows describe the wider GPQA set, not Diamond alone. The guessing row is the only one that applies to Diamond directly. None of these rows are lab-reported model scores. They are human results that Artificial Analysis publishes on its own evaluation page.

What does 198 questions do to a score?

Each question is worth 0.5 points, because 1 divided by 198 is about 0.505%. For a score near 80%, the standard error from the binomial formula, the square root of p(1 − p) divided by n, is about 2.8 points. The 95% interval is roughly ±5.6 points, or 74.4% to 85.6%.

This is our calculation from the item count, not a figure from a source. It treats the 198 items as a random sample of questions like these. The set is fixed and was selected for difficulty, so the true band could be wider or narrower than this estimate.

A three-point gap is six questions. Whether six questions is signal depends on how often the two models disagree on the same items. A published single score does not show that, so a reader has to ask for the per-question results or treat the gap as unresolved.

What are the limits of a four-option format?

Scores near 25% say little, because that is what guessing produces. At 198 items, the standard error at chance is about 3.1 points, so a score of 30% sits about 1.6 standard errors above chance. That is not clearly distinguishable from guessing on this set alone.

A correct choice also does not show how the model got there. Multiple-choice items let a model eliminate options, and real research and engineering work rarely arrives as four labelled answers. A high Diamond score tells you the model picked the right option, not that its reasoning would hold up on an open-ended task.

What should you check before quoting a GPQA score?

Five checks prevent most misreadings:

1. Which set? Diamond (198 questions) and the wider GPQA set (448 in the main set) produce different numbers and should never share a column.
2. Who ran it? A lab-reported score and an independent run can differ for the same model. Label which one you are quoting.
3. As of when? Model versions and harnesses change, so attach the capture date to every score.
4. What is the interval? At 198 items, a score carries a band of several points. Quote the band or say it is unknown.
5. Is the comparison paired? A gap between two models needs both models run on the same items, with per-question results where possible.

These checks apply to any benchmark. Our saturation post covers the broader pattern of tests nearing their ceiling, and how to read AI leaderboards covers intervals on any ranking.

GPQA Diamond also covers only biology, physics and chemistry. It says nothing about the coding, agentic or voice tasks that most products need, so a strong Diamond score is one input, not a purchase decision.

What's still unclear?

• Human figures for Diamond. The expert and non-expert baselines above come from the GPQA paper's abstract, which describes the 448-question set. Artificial Analysis heads the same figures with the 198-question Diamond set, so the set attribution is unresolved. Epoch AI cites an OpenAI study that reports 69.7% for PhD-level experts on Diamond. That is a lab-reported figure, so any expert-to-model gap on Diamond needs that caveat.
• Answer-key accuracy. Expert agreement filters for ambiguous items. It does not measure how many answer keys are wrong, so the error rate of the key itself is unknown from these sources.
• Gap uncertainty. The uncertainty of a gap between two models depends on their per-question results, which this post does not have.
• Weakest point of the argument. The ±5.6-point band assumes the 198 items are a random sample from a larger pool of similar questions. The set is fixed and selected for difficulty, so the band is an approximation.
• Authors' affiliations and funding. The sources cited here do not state them, so this post does not attribute any institutional interest to the authors.