Gemini and Claude on the same gold set
8 synthetic pulmonology case reports, the same prompt (v1) for every model, and a verbatim source quote required for every extracted fact. It is a smoke test, not a benchmark: read its limits.
Live rungenerated 2026-10-014 modelslabels LLM-drafted, hand-checked
Model comparison
| Model | Macro-F1 | Diagnosis | Schema valid | Quotes found | Hallucinated | Negations | Latency p50 / p95 | Cost / case |
|---|---|---|---|---|---|---|---|---|
| Gemini 3.8 Flashgemini · gemini-3.8-flash | 0.93 | 100%of 8 cases | 100% first try100% after repair | 100% | 0.0% | 0 | 7.9 s / 18.0 s | $0.00712,500 in · 1,400 out |
| Claude Sonnet 5claude · claude-sonnet-5 | 0.95 | 100%of 8 cases | 100% first try100% after repair | 100% | 0.0% | 7 | 10.7 s / 13.0 s | $0.0233,920 in · 1,482 out |
| Claude Opus 4.8in productionclaude · claude-opus-4-8 | 0.96 | 100%of 8 cases | 100% first try100% after repair | 100% | 0.0% | 3 | 14.1 s / 16.6 s | $0.0553,920 in · 1,415 out |
| Claude Opus 5.5claude · claude-opus-5-5 | 0.97best | 100%of 8 cases | 100% first try100% after repair | 100% | 0.0% | 2 | 12.2 s / 15.4 s | $0.0443,922 in · 1,416 out |
Your production model vs. the next one
On this gold set Claude Opus 5.5 is about as accurate (+0.01 macro-F1, within the noise of 8 cases), 20% cheaper per case and 1.9 s faster at the median. Worth a shadow run on your own cases before switching.
Claude Opus 4.8 is the production model named in Eximion's public AI disclosure. Claude Opus 5.5 always thinks adaptively (effort is the only lever, set to low here), which is where its extra latency comes from. Same prompt, same 8 cases, same scoring.
| Metric | nowOpus 4.8 | nextOpus 5.5 | Change |
|---|---|---|---|
| Macro-F1 | 0.96 | 0.97 | +0.01 (better) |
| Hallucinated | 0.0% | 0.0% | +0.0 pt |
| Latency p50 | 14.1 s | 12.2 s | −1.9 s (better) |
| Cost per case | $0.055 | $0.044 | −20% (better) |
| Per 1,000 cases | $55 | $44 |
Accuracy by field
Where the models differ is in the long lists: findings and measurements. Demographics and the diagnosis are nearly solved; recall on findings is where a model leaves facts behind.
| Field | Gemini 3.8 Flash | Claude Sonnet 5 | Claude Opus 4.8 | Claude Opus 5.5 |
|---|---|---|---|---|
| Patient | ||||
| Patient age | 1.00 | 1.00 | 1.00 | 1.00 |
| Patient sex | 1.00 | 1.00 | 1.00 | 1.00 |
| Presentation | ||||
| Chief complaint | 0.88 | 1.00 (best) | 1.00 (best) | 1.00 (best) |
| Final diagnosis | 1.00 | 1.00 | 1.00 | 1.00 |
| Findings | ||||
| precision | 0.69 | 0.73 | 0.78 | 0.82 (best) |
| recall | 0.68 | 0.75 | 0.78 | 0.82 (best) |
| F1 | 0.68 | 0.74 | 0.78 | 0.82 (best) |
| Measurements | ||||
| precision | 1.00 (best) | 0.97 | 1.00 (best) | 1.00 (best) |
| recall | 1.00 (best) | 0.98 | 0.98 | 1.00 (best) |
| F1 | 1.00 (best) | 0.97 | 0.99 | 1.00 (best) |
Read this before the numbers
- The gold set is synthetic: 8 pulmonology case reports written for this repository; no real patient is described.
- Labels were drafted with an LLM before each text was written, then checked by hand and by automated consistency tests (every quote is found in the de-identified text; the labels score perfectly against themselves).
- 8 cases is a smoke-level harness: it catches regressions and large gaps between models; it is not a benchmark, and no confidence intervals are reported.
- Claude helped draft the labels and is also evaluated: a known bias in its favour.
- Findings count only on near-identical wording (token_sort_ratio >= 85, any word order), so a paraphrase that drops a qualifier is a miss; every model is scored the same way.
- The diagnosis rule was relaxed after the first live run: all four models were marked wrong on the same two cases while naming the right diagnosis more specifically (e.g. 'Sarcoidosis, Scadding stage II' for 'Sarcoidosis'). Naming the gold diagnosis as a phrase now counts; a hedge ('X or Y', 'X vs Y') still does not. All models were re-scored from the same answers.
- Route: Gemini runs on Vertex AI; Claude runs on the direct Anthropic API because Vertex AI granted this new project no Claude quota (NOT_ENOUGH_USAGE_HISTORY). Both go through the same provider interface with the same prompts, schema and scoring.
How to read this. Macro-F1 averages six per-case scores (age, sex, chief complaint, diagnosis, findings F1, measurements F1). Quotes found means the model's verbatim quote was located in the source text. A hallucination is an extracted item that matches no labelled fact and whose quote is not in the source. Cost uses list prices per million tokens; thinking tokens count as output.