Skip to content
Case Lab
Take-home · three tasks, one flow

Clinical cases, end to end

A clinical note becomes a structured case with a verbatim source quote behind every fact. A physician reviews the AI draft, peers play it stage by stage, and the server scores every decision, opens the debrief and shows where the cohort turned.

Rare diseases take years to diagnose because the first explanation usually fits well enough. The featured case is built to test the second look.

FastAPI · PostgreSQL 18 · Next.js 16 · Gemini and Claude on Vertex AI · Cloud Run

The case most physicians miss

Case 01 · rare disease~8 min

A second collapsed lung in a young non-smoker

32-year-old woman · hard

Sudden left-sided chest pain and breathlessness

  • Ask the questions that matterthe patient answers
  • Open the differentialkeep it wide
  • Order the workupwhere it turns
  • Commit to a diagnosis and a planrate confidence
The take-home

Three tasks, wired into one product

Each part runs live. Open a card to see it working.

  • 01Backend

    FastAPI and PostgreSQL

    Cases arrive as JSON, land in a normalized schema through Alembic migrations, and every attempt is scored on the server.

    • Idempotent ingest keyed by a content hash
    • Answers never reach the browser before the case closes
    • Select-all and hedged diagnoses earn nothing
  • 02Frontend

    Next.js App Router

    Server Components render the case, a client stepper plays it, and a Server Action submits the answers. Types come from the API's OpenAPI schema.

    • Given stages are server-rendered and slotted into the client player
    • The debrief marks every decision right, wrong or missed
    • One generated types file behind every API call
  • 03LLM pipeline

    Gemini and Claude on Vertex AI

    Text is de-identified, extracted against a JSON schema with a verbatim quote for every fact, checked against the source, then authored into decisions.

    • Validate-and-repair loop on schema errors
    • Every quote verified against the source text
    • Eval harness: accuracy per field, hallucinations, latency, cost
The flow

From a note to a debrief

  1. 01

    Raw note

    pasted or sampled

  2. 02

    De-identify

    names, dates, IDs masked

  3. 03

    Extract and ground

    every fact quoted, every quote checked

  4. 04

    Physician review

    flagged items, not the whole case

  5. 05

    Play

    nine stages, scored on the server

  6. 06

    Debrief

    percentile, AI markers, sponsor view

Nothing is marked until the case closes. The player never shows right or wrong mid-case, and the answers stay on the server until you submit. Points come only from the diagnosis and the treatment plan.

What else is here

Three things beyond the brief

Built after the three tasks were done and tested, around how a case platform actually runs.

  • AI players

    Models on the same curve as physicians

    Gemini and Claude play every showcase case blinded, through the same scoring code. They show up as labelled markers on your percentile curve and never enter the cohort.

  • Review gate

    AI-assisted, not AI-generated

    Studio output is only a draft. A reviewer sees the answer key and a server-computed checklist, checks the few flagged items, and signs off before the case counts.

  • Sponsor view

    Where the cohort turned

    After you close a case: pick rates at every decision point, the right steps most physicians skipped, and how ordering the key test changed the diagnosis rate.

The numbers

Four models, one gold set

Gemini and Claude extract the same eight synthetic pulmonology notes; every fact must quote the source. A smoke test, not a benchmark, and the page says why.

Your production model vs. the next one

On this gold set Claude Opus 5.5 is about as accurate (+0.01 macro-F1, within the noise of 8 cases), 20% cheaper per case and 1.9 s faster at the median. Worth a shadow run on your own cases before switching.

Extraction accuracy

8-case gold set
Macro-F1
0.97
Quotes found
100%
Hallucinated
0.0%
Cost per case
$0.044

Claude Opus 5.5, best of 4 models on the same notes. Full comparison

Read the code, the API and the numbers

Source on GitHub API docs Eval report