LLM Index for Scientific Computing

Generated 2026-08-14 · 478 models scored from 579 measured by Artificial Analysis · raw cost data only

Each point is a model: vertical position is the SoftMinZ performance index, horizontal position the raw cost of one agentic task (log scale). The dashed line is the Pareto frontier — models for which no cheaper model is at least as good. Frontier members are colour- and shape-coded; everything else is grey.

Full ranking

Every scored model. Bold rows are Pareto frontier members. Use the two buttons to switch the ranking — they need no JavaScript. In cost view, models with no published cost data sort last.

What SoftMinZ measures

Nine public benchmarks are scored per model. Each benchmark score is first converted to a z-score across every model measured on that benchmark, so all benchmarks contribute in comparable units regardless of their raw scale:

The index is then the negative natural logarithm of the mean of e^−z over the model's benchmarks:

This is a smooth soft-minimum of the z-scores: it always sits between the model's worst z-score and its mean z-score, and it slides toward the worst end as the profile becomes more uneven. The practical consequences:

• A weak benchmark cannot be averaged away. The exponential weights e^−z concentrate on the model's worst results. The discount is not cosmetic: a model with eight benchmarks at z = +1.5 and one at z = −0.8 scores 0.81 — below a model that is merely even at z = +1.0 everywhere, despite the first model's higher mean (1.24 vs 1.0), because the soft-minimum punishes the imbalance by more than the mean gap.
• Balanced excellence is rewarded; lopsided excellence is discounted.
• Coverage does not inflate the score. The 1/n normalisation makes the index intensive: being measured on more benchmarks moves the score only through the quality of the new result, never through the count itself.

A model is excluded from scoring entirely (no index, rather than a low one) if it is measured on fewer than 7 of the 9 battery benchmarks, or if either of two mandatory measurements is missing: CritPt (physics) and the non-hallucination rate. A model with no physics evaluation or no hallucination measurement is simply not evaluated here.

Why these nine benchmarks

The performance battery is chosen for relevance to scientific computing: science-adjacent reasoning and knowledge (GPQA Diamond, CritPt, Humanity's Last Exam, AA-Omniscience), the code-execution skills real computational work depends on (SciCode, LiveCodeBench, Terminal-Bench Hard), and the trust dimensions that decide whether a model's output can be believed without full re-verification (non-hallucination rate, AA-LCR long-context reasoning).

τ³-Banking is excluded from the performance calculation because it simulates fintech customer support — a domain-specific agent task whose skill profile says nothing about scientific capability. (It does remain in the cost calculation below, where it represents real agentic workflow spend.) GDPval-AA v2 is likewise excluded: its tasks are real-world professional deliverables (analyst memos, marketing plans and the like), not scientific computing, so it has no place in a performance battery for science either — though it too remains in the cost model as agentic workload.

How the cost is computed

The cost benchmarks are chosen for relevance to agentic workflow costs. Artificial Analysis publishes, for each benchmark in its Intelligence Index, a weighted cost per task. Dividing that by the benchmark's Intelligence Index weight recovers the unweighted per-benchmark cost C_i; the four benchmarks that represent agentic workload — GDPval-AA v2, τ³-Banking, Terminal-Bench v2.1 and AA-LCR — are then averaged weighted by task count:

Cheap high-volume knowledge benchmarks (AA-Omniscience, HLE, GPQA Diamond, CritPt, SciCode) are excluded from the cost average because they would otherwise dominate the task count while representing almost no real agentic spend. The result is a raw USD figure per agentic task — list prices only, no discounts. A handful of open-weight models are served by free hosting endpoints and report a $0 cost to AA; they are excluded from the cost axis (and the frontier) rather than treated as genuinely free products, but remain scored and ranked in the table.

The Artificial Analysis Intelligence Index, and why we don't use it

Artificial Analysis' own headline metric, the Intelligence Index v4.1.1, is a plain weighted average of normalised benchmark scores:

with category weights Agents 34%, Coding 24%, Scientific Reasoning 24%, General 18%, and per-benchmark weights GDPval-AA v2 20%, τ³-Banking 14%, Terminal-Bench v2.1 16%, SciCode 8%, HLE 12%, GPQA Diamond 6%, CritPt 6%, AA-Omniscience 12% (itself split: 8% accuracy + 4% non-hallucination) and AA-LCR 6%. GDPval-AA v2 Elo scores are folded in as clamp((Elo − 500) / 2000). (Formula per the AA methodology page.)

It is a useful headline, but for scientific work it has real flaws:

• A linear average hides weakness. A model that is excellent on agentic benchmarks but hallucinates heavily still posts a high II, because one strong benchmark linearly cancels one weak one. For science, a hallucinated citation or a confidently wrong intermediate result can invalidate the whole task — the failure mode a soft-minimum punishes and an average does not.
• Hallucination is nearly an afterthought. Non-hallucination carries only 4% of the total weight inside AA-Omniscience's 12% share. Here it is a mandatory benchmark and a dominant term whenever it is bad.
• The weighting emphasises generalist agentic work, not science. 34% Agents includes τ³-Banking (14%), a fintech customer-support simulation — a large weight on a skill orthogonal to scientific capability.
• Ad hoc normalisations. The GDPval clamp((Elo − 500)/2000) mapping and the frozen Elo reference points are parameters AA itself may adjust over time, so index values are not perfectly comparable across model vintages.

Why the GPT-5.6 lineup scores below its reputation

Across the 18 GPT-5.6 variants scored today, the median non-hallucination rate is 9.2% against a field mean of 24.5% (population σ = 20.8 points) — a z-score of roughly -0.74 on the trust benchmark alone. Frontier models sit far higher: the top of the table carries non-hallucination rates in the 45–70% range. Because SoftMinZ is a soft-minimum, that weak trust score receives the largest Boltzmann weight in the average and drags every GPT-5.6 variant down even where GPQA, HLE and SciCode are strong — the best-placed variant ranks #51 of 478.

Reproducibility

Every scraped score is cross-validated against the JSON-LD metadata on its source page; a benchmark whose parsed values disagree is rejected rather than published. Intelligence Index weights and task counts used in the cost model are parsed live from the AA methodology page on every run (today: live (AA methodology page, parsed this run)); if that page is unreachable, a frozen snapshot of the weights is used and noted here. Today's scrape: agentic cost per task: 132 models; artificial-analysis-intelligence-index: 153 models (validated 20); artificial-analysis-long-context-reasoning: 500 models (validated 20); critpt: 482 models (validated 20); gpqa-diamond: 575 models (validated 20); humanitys-last-exam: 567 models (validated 20); livecodebench: 343 models (validated 7); non-hallucination rate: 479 models; omniscience: 479 models (validated 20); scicode: 567 models (validated 20); terminalbench-hard: 432 models (validated 14). One script generates this page — see the repository.