The Best AI in the World Just Scored 12% on a Real Expert Test
SemiAnalysis dropped a new benchmark for agentic AI. The top model failed 88% of the time, proving AI is still just an assistant.
Research · Source: Unite.AI
What happened
SemiAnalysis just released GDPx on November 2, 2026. It is a new benchmark testing AI agents on real professional work. The test covers 206 tasks across 11 high-value domains like chip design, finance, and biology. SemiAnalysis worked with 14 specialized data partners to build the challenges. Models get a default time limit of eight hours to complete multi-step goals. They run in their native environments like Claude Code or Codex. The results are brutal.
Anthropic took the crown. Claude Opus 5.5 scored exactly 12.0 percent. It costs an average of 51.41 dollars per task and uses 548K output tokens. Google Gemini 4 Carbon hit 10.9 percent at a much cheaper 16.29 dollars per task. OpenAI GPT-6 Astra landed at 9.6 percent. Meta Muse Wonder Full took fourth with 9.0 percent. SemiAnalysis noted that safety classifiers sandbagged the scores for Opus and Astra in cyber and biology. Even so, the top score proves that frontier models still struggle with deep expert work.
The benchmark exposes massive gaps in reasoning. SemiAnalysis built GDPx to replace OpenAI GDPval, which spoon-fed context to models. Real work requires digging. In finance and legal domains, every single model scored a flat zero under strict binary grading. The finance test asked models to process a real private equity buyout using 4,000 anonymized files. They all failed. In quant trading, 70 out of 111 runs performed worse than a basic reference strategy. In alignment tasks, 196 out of 200 runs broke at least one behavior rule.
Key facts
- 12.0% — Claude Opus 5.5 top score on the GDPx benchmark
- 206 — Number of tasks in the first version of GDPx
- $51.41 — Average cost per task for Claude Opus 5.5
- 4,000 — Anonymized files used in the finance buyout task
- 0 — Score every model received in finance and legal binary grading
Why it matters
Founders need a massive reality check on AI capabilities. Chatbots look brilliant when you feed them perfect context in a prompt. Real work is messy and unstructured. When you ask an AI agent to navigate thousands of files for an acquisition, it breaks down completely. You cannot replace your senior engineers, quants, or lawyers yet. The technology simply is not there. You are building tools to make human experts faster. You are not building tools to fire them. Anyone selling fully autonomous enterprise workers is lying to their customers.
The first trillion dollars in AI revenue will not come from generic consumer tasks. SemiAnalysis built this test around sectors with the highest return on investment. They intentionally ignored government, real estate, and insurance. If you are building AI products, you must target these high-value niches where companies have budget. But you must also design for human-in-the-loop workflows. The domain results show extreme fragmentation. Gemini wins in biology. GPT-6 Astra wins in recursive self-improvement. Claude wins in hardware design. Builders will need to route tasks to specific models rather than relying on one provider.
For builders
Build specialized tools for high-value niches
General AI fails at deep expert work. Founders who build narrow workflows for finance, biology, or chip design will win. Enterprises will pay massive premiums for tools that solve even a small fraction of these complex tasks. Stop building wrappers and start solving hard domain problems.
Optimize for cost and specific domain strengths
Claude Opus 5.5 costs over 51 dollars per task. Gemini 4 Carbon costs 16 dollars and actually wins in biology and software engineering. Builders must route tasks to the right model. You will lose money if you blindly send every prompt to the most expensive model on the market.
Do not trust models with compliance
In alignment tasks, 196 out of 200 runs broke a behavior rule. Models cannot be trusted with strict legal or safety boundaries yet. Startups that build robust guardrails and compliance layers will capture the enterprise market. Corporate buyers will pay heavily to avoid AI disasters.
My take
I see everyone selling the dream of autonomous AI workers, but the hard data says otherwise. A 12 percent success rate means AI is a junior intern, not a senior expert. We need to stop building products that promise total automation and start building tools that help smart people work faster.
Original reporting: Unite.AI. This is my rewrite and opinion.