LLM Index for Scientific Computing

Generated 2026-10-07 · 214 models scored from 696 measured by Artificial Analysis · raw cost data only

Each point is a model: vertical position is the SoftMinZ performance index, horizontal position the raw cost of one agentic task (log scale). The dashed line is the Pareto frontier — models for which no cheaper model is at least as good. Frontier members are colour- and shape-coded; everything else is grey.

Full ranking

Every scored model. Bold rows are Pareto frontier members. Use the two buttons to switch the ranking — they need no JavaScript. In cost view, models with no published cost data sort last.

What SoftMinZ measures

Seven public benchmarks are scored per model. Each benchmark score is first converted to a z-score across the scored cohort, so all benchmarks contribute in comparable units regardless of their raw scale:

The index is then the negative natural logarithm of the mean of e^−z over the model's benchmarks:

This is a smooth soft-minimum of the z-scores: it always sits between the model's worst z-score and its mean z-score, and it slides toward the worst end as the profile becomes more uneven. The practical consequences:

• A weak benchmark cannot be averaged away. The exponential weights e^−z concentrate on the model's worst results. The discount is not cosmetic: a model with six benchmarks at z = +1.5 and one at z = −0.8 scores 0.67 — below a model that is merely even at z = +1.0 everywhere, despite the first model's higher mean (1.17 vs 1.0), because the soft-minimum punishes the imbalance by more than the mean gap.
• Balanced excellence is rewarded; lopsided excellence is discounted.
• Every score covers the same ground. Because only fully-benched models are scored, n is the same battery size for every model — no model's index can look better simply by having been measured on fewer (or easier-to-average) benchmarks.

A model is scored only if it has been evaluated on every benchmark in the battery — no partial profiles. A model missing even one measurement receives no index (rather than a gamed low one) and, just as important, contributes to no benchmark's z-statistics: the mean and spread of each benchmark are computed over fully-benched models only. The scored cohort is therefore exactly the set of models Artificial Analysis has completely evaluated — smaller, but representative.

Why these seven benchmarks

The performance battery is chosen for relevance to scientific computing: science-adjacent reasoning and knowledge (CritPt, Humanity's Last Exam, AA-Omniscience), the code-execution skills real computational work depends on (SciCode, Terminal-Bench v4.0), and the trust dimensions that decide whether a model's output can be believed without full re-verification (non-hallucination rate, AA-LCR long-context reasoning).

GPQA Diamond was removed from the battery on 2026-09-07: Artificial Analysis dropped it from their Intelligence Index in the v4.2 revision, and a benchmark the index has abandoned no longer earns a place here. τ³-Banking is excluded from the performance calculation because it simulates fintech customer support — a domain-specific agent task whose skill profile says nothing about scientific capability; it too left the Intelligence Index in AA's v4.3 revision and is now excluded everywhere on this page. GDPval-AA v2.1, AutomationBench-AA and AA-Briefcase v1.1 are likewise excluded from scoring: they measure professional deliverables and workflow automation, not scientific computing — though they remain in the cost model below as agentic workload.

How the cost is computed

The cost benchmarks are chosen for relevance to agentic workflow costs. Artificial Analysis publishes, for each benchmark in its Intelligence Index, a weighted cost per task; the per-benchmark split comes from the priced-models table embedded in AA's model pages (the leaderboard payload carries only the aggregate total, which we use as a cross-check: the split must sum to the published total or the run fails). Dividing that by the benchmark's Intelligence Index weight recovers the unweighted per-benchmark cost C_b; the five benchmarks that represent agentic workload — GDPval-AA v2.1, AutomationBench-AA, Terminal-Bench v4.0, AA-LCR v1.1 and AA-Briefcase v1.1 — are then averaged weighted by task count:

This is the total cost of running these 5 benchmarks divided by the number of tasks across all 5 benchmarks.

Cheap high-volume knowledge benchmarks (AA-Omniscience, HLE, CritPt, SciCode) are excluded from the cost average because they would otherwise dominate the task count while representing almost no real agentic spend. The result is a raw USD figure per agentic task — list prices only, no discounts. Cost data exists only where Artificial Analysis publishes it: 178 of the 214 scored models carry a price today. The rest (mostly models AA does not track pricing for) show — in the table and are excluded from the cost axis and the Pareto frontier; that is a data-availability gap, not a zero price.

The Artificial Analysis Intelligence Index, and why we don't use it

Artificial Analysis' own headline metric, the Intelligence Index, is a plain weighted average of normalised benchmark scores:

with category weights Agents 30%, Coding 20%, Scientific Reasoning 20%, General 30%, and per-benchmark weights AA-Briefcase v1.1 15%, GDPval-AA v2.1 10%, AutomationBench-AA 5%, Terminal-Bench v4.0 10%, SciCode 10%, HLE 10%, GDP.pdf 10%, CritPt 10%, AA-Omniscience 15% (split: 10% accuracy + 5% non-hallucination) and AA-LCR v1.1 5% (per the live-parsed AA methodology page, Intelligence Index v4.3.2, September 2026). The v4.3 revision replaced τ³-Banking with AutomationBench-AA (5%) and upgraded Terminal-Bench v2.1 to v4.0 (a table reset in progress — v4.0 scores exist for only a fraction of models so far, which shrinks every fully-benched cohort on this page until AA finishes re-running models); the v4.3.2 revision re-anchored the two Elo benchmarks (GDPval-AA v2.1 to DeepSeek V4.1 Flash at 1600, AA-Briefcase v1.1 Crowd-BT fits) and refreshed judge panels; the numbers below track the current revision on every run.

It is a useful headline, but for scientific work it has real flaws:

• A linear average hides weakness. A model that is excellent on agentic benchmarks but hallucinates heavily still posts a high II, because one strong benchmark linearly cancels one weak one. For science, a hallucinated citation or a confidently wrong intermediate result can invalidate the whole task — the failure mode a soft-minimum punishes and an average does not.
• Hallucination is nearly an afterthought. Non-hallucination carries only 5% of the total weight inside AA-Omniscience's 15% share. Here it is a mandatory benchmark and a dominant term whenever it is bad.
• The weighting emphasises generalist agentic work, not science. Agents plus the generalist members — AA-Briefcase v1.1 (15%), GDP.pdf (10%) and now AutomationBench-AA (5%) — put roughly half the Index on knowledge-work and business deliverables, while GPQA Diamond, the one graduate-science benchmark, has been gone since the v4.2 revision. The skill mix measures office work, not scientific capability.
• Ad hoc normalisations. The GDPval clamp((Elo − 500)/2000) mapping and the frozen Elo reference points are parameters AA itself may adjust over time, so index values are not perfectly comparable across model vintages.

Why the GPT-5.6 lineup scores below its reputation

Across the 17 GPT-5.6 variants scored today, the median non-hallucination rate is 9.2% against a field mean of 28.4% (population σ = 22.4 points) — a z-score of roughly -0.85 on the trust benchmark alone. Top-ranked models sit far higher: the five best in the table carry non-hallucination rates of 41–85%. Because SoftMinZ is a soft-minimum, that weak trust score drags every GPT-5.6 variant down even where CritPt, HLE and SciCode are strong — the best-placed variant ranks #72 of 214.

Reproducibility

Every scraped score is cross-validated against the JSON-LD metadata on its source page; a benchmark whose parsed values disagree is rejected rather than published. Intelligence Index weights and task counts used in the cost model are parsed live from the AA methodology page on every run (today: live (AA methodology page, parsed this run)); if that page is unreachable or its table cannot be parsed, the run aborts rather than publish costs computed from stale weights. Today's scrape: artificial-analysis-intelligence-index: validated (0 checked); artificial-analysis-long-context-reasoning: validated (0 checked); critpt: validated (0 checked); humanitys-last-exam: validated (0 checked); leaderboards/models: 696 records — hle:659, scicode:226, critpt:574, omniscience:571, lcr:570, terminalbench_v40:215, intelligence_index_v4_1:687, analystAgent:38; models/QwQ-32B-Preview: cost split (185 priced, 0 totals cross-checked); omniscience: validated (0 checked); scicode: validated (0 checked); terminalbench-v4-0: validated (0 checked). One script generates this page — see the repository.