HealthBench Hard — Nox's full self-run benchmark report

Nox's full, self-run measurement on HealthBench Hard — the 1,000-prompt hard subset of HealthBench, an open benchmark built with physicians that grades open-ended health answers against clinician-written rubrics. This page reports one run, in full: every number comes from the committed results file of that run, and the method is disclosed end to end.

Scope: This is a self-run measurement, disclosed in full but not independently verified. A HealthBench Hard score measures performance against that benchmark's clinician-written grading rubrics — it is not an overall product-quality, safety, or clinical-outcome measurement, and a benchmark conversation is not medical care.

Headline result

Nox 1.3 Astra: 34.8% (±1 bootstrap std) on HealthBench Hard — all 1,000 examples, run August 2, 2026
Grader: GPT-4.1 (OpenAI) — the grader model used by the official HealthBench publication
Protocol: openai/simple-evals — one grader call per rubric criterion, official aggregation — 11,846 rubric criteria graded, 0 missing
Same answers under a stricter grader (GPT-5.4 (OpenAI), reasoning grader): 21.3% — disclosed, see below

Score by rubric axis (benchmark's own criterion tags)

Communication quality: 61.1% (357 criteria)
Instruction following: 55.6% (199 criteria)
Accuracy: 49.5% (776 criteria)
Completeness: 30.7% (852 criteria)
Context awareness: 23.8% (704 criteria)

Score by benchmark theme

Expressing uncertainty (hedging): 41.2% (167 examples)
Emergency referrals: 40.2% (66 examples)
Context seeking: 37.2% (179 examples)
Communication: 34% (111 examples)
Global health: 32.3% (280 examples)
Health data tasks: 31.6% (115 examples)
Complex responses: 25.8% (82 examples)

Response statistics

Mean answer length: 3,020 characters
Emergency guidance shown on 19.2% of examples
0.3% of answers truncated by the length budget

What HealthBench Hard is

HealthBench is an open health-conversation benchmark developed with physicians: each example is a realistic health conversation, and each answer is graded against rubric criteria written by clinicians — specific, checkable requirements like “tells the user to seek emergency care” or “asks about medication history.” HealthBench Hard is its hardest 1,000-prompt subset, selected so that even frontier models score low. Scores on Hard are therefore small numbers across the industry — the published leaderboard tops out well under 50%.

What exactly was tested

The graded system was the full Nox product pipeline — the same code paths a paying user gets, not a lab configuration. Settings were the strongest a user can select: the MAX plan's Nox 1.3 Astra with Detailed answer length, deepest thinking level, normal tone, and the product's standard answer-length cap. Leo, Nox's deterministic safety layer, ran in front of every answer exactly as in production (strict level), and any emergency banner it produced was included in the graded reply. Connected-app surfaces were off and no personal context was attached — each benchmark conversation was answered cold, once, with no retries and no answer selection.

How grading worked

The run followed the official open protocol (openai/simple-evals): all 1,000 examples were answered, and every one of the 11,846 rubric criteria was graded by a separate grader-model call — none missing, no sampling. Per-example scores use the benchmark's own aggregation (points earned over positive points available, clipped to 0–1), and the overall score is the mean across examples, with a bootstrap standard deviation reported for the error bar. The grader was GPT-4.1, the grader model used by the official HealthBench publication.

The grader defines the scale — both numbers disclosed

Rubric grading is done by a model, and different grader models apply different strictness. The same 1,000 Nox answers — a byte-identical answer set, with the checksum identity recorded in a committed provenance note next to the results — score 34.8% under GPT-4.1 (the official HealthBench grader model) and 21.3% under GPT-5.4, a stricter reasoning grader. Neither number is more “real” — they are the same answers on two scales. Nox publishes the official-grader figure as the headline because it is the only scale on which published third-party scores exist, and discloses the stricter figure permanently so the grader's effect is never hidden. Scores are only comparable within one grader's scale.

Where the points came from

By the benchmark's own criterion tags, Nox scored highest on communication quality (61.1%), instruction following (55.6%), and accuracy (49.5%). Emergency-referral conversations — the ones Leo exists for — scored 40.2%, and Leo's emergency guidance appeared on 19.2% of all graded examples. Answers averaged about 3,000 characters, and only 0.3% were cut short by the length budget.

Where the points were lost — published on purpose

The weakest axes are completeness (30.7%) and context awareness (23.8%): rubric criteria that reward covering every clinically relevant angle and tailoring the answer to the specific person asking. The weakest theme was complex responses (25.8%). These are the current improvement focus, and they are published here as measured — a benchmark report that only lists strengths is marketing, not measurement.

Limitations

This is one run, self-administered, on a benchmark that grades text answers against written rubrics. It was not independently supervised or verified. It does not measure real-world outcomes, diagnostic accuracy, or safety beyond what the rubrics encode, and single-turn benchmark conversations differ from real use, where Nox can ask follow-ups across a session. The full results file — every number on this page — is committed alongside the code that generated it.

Raw data downloads — verify it yourself

Every per-example file behind this report is published for independent verification: Nox's 1,000 raw answers, every rubric-criterion grade under both grader models, and the aggregate results files. Checksums (SHA-256) are listed in the README, and the completions file is byte-identical to the answer set recorded in the run's committed provenance note. HealthBench prompts are not redistributed — records join against the open benchmark release by prompt id.

<a href="https://nox.aurenaring.com/data/healthbench/README.md">README.md</a> (6 KB) — Schemas, SHA-256 checksums, and exact reproduction instructions
<a href="https://nox.aurenaring.com/data/healthbench/nox-astra-completions.jsonl">nox-astra-completions.jsonl</a> (3.4 MB) — All 1,000 raw model answers (the graded text, including any Leo emergency banner)
<a href="https://nox.aurenaring.com/data/healthbench/grades-gpt-4.1.jsonl">grades-gpt-4.1.jsonl</a> (6.3 MB) — All 11,846 rubric-criterion grades under GPT-4.1, the official HealthBench grader model (headline 34.8%)
<a href="https://nox.aurenaring.com/data/healthbench/grades-gpt-5.4.jsonl">grades-gpt-5.4.jsonl</a> (5.7 MB) — The same answers graded by GPT-5.4, a stricter reasoning grader (21.3% on that scale)
<a href="https://nox.aurenaring.com/data/healthbench/summary-gpt-4.1.json">summary-gpt-4.1.json</a> (8 KB) — Aggregate results file for the GPT-4.1 grading — the source of every number on this page
<a href="https://nox.aurenaring.com/data/healthbench/summary-gpt-5.4.json">summary-gpt-5.4.json</a> (8 KB) — Aggregate results file for the GPT-5.4 grading

Reproduce the score with the official open protocol's aggregation (per-example points earned over positive points available, clipped to 0–1, then averaged). Alternative aggregations produce different numbers that are not comparable to published HealthBench scores, and scores are only comparable within one grader's scale.