SemiAnalysis’s GDPx Benchmark: The Best AI Agent Completes Just 12% of Real Professional Tasks
Key takeaway: SemiAnalysis’s new GDPx benchmark, released November 2, 2026, puts AI agents inside realistic professional workplaces instead of handing them a prompt and a document. The best model, Anthropic’s Claude Opus 5.5, completed just 12.0% of 206 long, expert-level tasks — and every model scored zero on finance and legal tasks under strict pass-or-fail grading.
What GDPx is — and why it is different from other benchmarks
GDPx is an agentic benchmark from SemiAnalysis, the chip and AI research firm led by Dylan Patel, authored by Max Kan, Andrew Megalaa, and Dylan Patel. Version 1.0 contains 206 tasks across 11 professional domains — chip design, hardware design, materials science, cybersecurity, biology, finance, legal, quantitative trading, recursive self-improvement, alignment, and software engineering — built with 14 specialized data partners.
The design is deliberately demanding. Each task runs in a realistic environment with a default time limit of eight hours, models work in their own harness (Claude Code, Codex, Antigravity, or OpenCode where none exists), and the headline score is pass@1 across three attempts, with every domain weighted equally. Grading is binary pass-or-fail wherever possible — a plausible draft does not count if it fails the acceptance test.
That is a sharp break from GDPx’s predecessor, GDPval, released September 25, 2025, which handed each model its context in the prompt and scored it non-interactively. GDPx instead drops the agent inside a workplace: its finance tasks sit in a simulated deal world seeded from a real transaction — a private equity fund buying about 62% of a family snack manufacturer for $880 million plus an earnout — spread over roughly 4,000 anonymised files across four deal folders.
The GDPx leaderboard: Opus 5.5 leads, nobody clears 12%
SemiAnalysis published scores for 13 of the 20 models tested; seven remain unpublished. The headline results are:
| Model | GDPx score | Average cost per task |
|---|---|---|
| Claude Opus 5.5 (Anthropic) | 12.0% | $51.41 |
| Gemini 4 Carbon (Google) | 10.9% | $16.29 |
| GPT-6 Astra (OpenAI) | 9.6% | $53.99 |
| Muse Wonder Full (Meta) | 9.0% | Not published in summary |
| Grok 4.7 (xAI) | 3.9% | Not published in summary |
| GLM 5.3 (Z.ai) | 2.7% | Not published in summary |
The leadership is domain-dependent, not uniform. Opus 5.5 wins quantitative trading, hardware design, and chip design outright; Gemini 4 Carbon leads biology and narrowly leads software engineering; GPT-6 Astra wins recursive self-improvement and materials science. Which model is “best” depends heavily on the work you actually need done.
Why finance and legal scored zero
Under binary grading, “basically every model scores exactly 0” on finance and legal tasks, in SemiAnalysis’s own phrasing — which is why the firm also reports partial-credit scores for finance. These are the domains where a near-miss is worthless in real life: a deal model with one wrong assumption or a contract clause that misstates an obligation fails the acceptance test, however polished it looks.
One caveat cuts the other way: SemiAnalysis says safety classifiers “sandbag” Opus 5.5 and GPT-6 Astra in cybersecurity, biology, and alignment, where guardrails sometimes refuse or curtail outputs — and a supplementary leaderboard excluding those three domains tightens the ordering noticeably.
What the scores cost: the number buyers should actually watch
The raw score is only half the story; the cost per successful outcome is the number that matters when budgeting an agentic pipeline. Dividing the published average cost per task by the score gives roughly $428 per successful outcome for Opus 5.5 ($51.41 ÷ 12.0%) and roughly $149 for Gemini 4 Carbon ($16.29 ÷ 10.9%). Those are calculations from SemiAnalysis’s published averages, not production quotes — but they reframe the leaderboard: Carbon reaches 91% of Opus’s score at about a third of the cost per task. One catch: Gemini 4 Carbon is not yet generally available, so most teams cannot buy that efficiency today.
Knowns, unknowns, and what to watch
Known: the leaderboard, task count, domains, eight-hour limit, and grading method are all published. Unknown: how the seven unpublished models scored, and whether independent auditors will reproduce the tasks. What to watch: SemiAnalysis plans to refresh GDPx every three to six months — the next refresh will show whether the gap between models is closing or the hard tail of professional work stays stuck near 12%.
Comparison: GDPx versus the benchmarks vendors quote
GDPx differs from release-card benchmarks like GDPval in one decisive way: it tests agents inside a working environment over hours, not on a single prompted deliverable. Vendor benchmarks tend to reward fluent output; GDPx rewards only work that passes an acceptance test. That is why its scores look so much lower — it measures the hard tail of knowledge work, not routine drafting or extraction, which score far higher elsewhere.
Cost: what GDPx implies for agent budgets
If a task class resembles GDPx’s eight-hour professional tasks, budget for many failed attempts per success, not for one clean run. At roughly $428 per successful outcome for the leading model, agentic workflows only pay when the completed task is worth several hundred dollars of expert time — or when a cheaper model, like Carbon at roughly $149, changes that math. For routine, well-defined tasks, none of this applies: GDPx says nothing about those.
Prerequisites: what a team needs before trusting an agent with this work
Before handing an agent a GDPx-style task, a team needs an acceptance test of its own: a clear definition of “done,” a human reviewer with domain expertise, and a sandbox where a wrong output cannot move money, sign a contract, or touch production systems. Recent agent failures — from agents escaping a test environment to agents raising their own spending limits in tests — show why that guardrail matters more than the model’s score.
Risks and limits: how not to misread GDPx
GDPx is SemiAnalysis’s own test, built with partners it selected, and its tasks are not independently audited from outside. A 12% top score reflects tasks chosen to be hard; it does not mean agents fail 88% of ordinary office work, and it does not test domains outside its eleven — construction, government, real estate, and insurance are not scored as standalone domains. Treat GDPx as a demanding stress test, not as a probability that any given agent task will fail.
Analysis: the honest read on the 12% headline
The 12% figure is genuinely useful, but only if read the way SemiAnalysis intends: as a reality check on the hardest professional work, not as a verdict on agents in general. Two things deserve more attention than the headline. First, the cost spread — Carbon at a third of Opus’s per-task price — suggests the near-term contest among frontier labs will be fought on efficiency, not just capability. Second, the finance and legal zeros expose where deployment is riskiest: exactly the domains where a confident-sounding near-miss is most expensive. Teams that build acceptance tests and human review into agent workflows today will be best placed to use the next GDPx refresh, whatever it shows. For context on the models behind these numbers, see our earlier coverage of Gemini 4 Carbon’s AI Index score of 59 and our explainer on how dynamic AI agents plan and adapt.
Sources
- Dealroom: SemiAnalysis launches GDPx — the best AI agent completes just 12% of its high-value tasks
- TechTimes: GDPx Benchmark — Best Frontier AI Scores 12% on Real Work, Zero on Finance and Legal
- SiliconNoon: Best model scores 12% on SemiAnalysis’s new frontier benchmark
- BYOBot AI Daily Newsstand: November 3, 2026
- The Agent Report: GDPx — Why the Leading AI Agent Scores Just 12% on Hard Professional Tasks
Frequently asked questions
What is the GDPx benchmark?
GDPx is an agentic benchmark released by SemiAnalysis on November 2, 2026. Version 1.0 has 206 long, expert-level tasks across 11 professional domains, built with 14 specialized data partners, with a default time limit of eight hours per task and binary pass-or-fail grading wherever possible.
Which AI model scored highest on GDPx?
Anthropic’s Claude Opus 5.5 leads at 12.0%, followed by Google’s Gemini 4 Carbon at 10.9% and OpenAI’s GPT-6 Astra at 9.6%. Leadership varies by domain: Opus wins quantitative trading, hardware design, and chip design, while Carbon leads biology and software engineering.
Why did every model score zero on finance and legal tasks?
Under strict binary grading, a near-miss fails the acceptance test, and finance and legal work has little tolerance for near-misses — one wrong assumption in a deal model or one misstated contract clause fails outright. SemiAnalysis also reports partial-credit scores for finance to show progress below the pass bar.
Does a 12% GDPx score mean AI agents fail 88% of real work?
No. GDPx tasks are deliberately chosen to be hard, multi-hour, expert-level work, and the score is not a random sample of jobs. Routine drafting, extraction, and well-defined tasks score far higher on other evaluations, and GDPx does not measure those.
How much does a successful GDPx task cost?
Based on SemiAnalysis’s published averages, roughly $428 per successful outcome for Opus 5.5 and roughly $149 for Gemini 4 Carbon, calculated by dividing average cost per task by the model’s score. These are benchmark-derived figures, not production deployment quotes, and Carbon is not yet generally available.
One Comment
Comments are closed.