AI Models Fail Real Enterprise Code in New Real-SWE Benchmark
Specific Labs tested top AI models on private enterprise codebases, and even the best model could only solve 38.8 percent of the tasks.
Models · Source: Hacker News
What happened
Specific Labs just dropped Real-SWE. It is a new benchmark that tests frontier AI models on private enterprise codebases. Authors Snagnik Das, Siddhant Paliwal, and Janak Sunil licensed real tasks from actual companies. The tasks come from production environments with substantial usage. Examples include a consumer fintech platform processing over one hundred thousand bank statements and an enterprise AI sales platform. This is not a synthetic test. These are real problems that human engineers get paid salaries to solve.
The results are brutal. Fable 5.1 using Claude Code took first place with just a 38.8 percent resolution rate. GPT-6 Astra using Codex CLI followed at 33.8 percent. Grok 4.6 hit 32.5 percent. Gemini 3.8 Flash reached 31.2 percent. GLM 5.3 scored 28.8 percent. Muse Spark 1.3 and Kimi K3 performed even worse. GPT-5.6 Sol came in dead last at 16.2 percent. No model could solve every task.
The tasks are complex and natively out of distribution. Private code means models cannot cheat by memorizing public GitHub repositories. Ninety-nine percent of tokens in real-world enterprises are hidden from frontier models. Agents had to navigate proprietary business logic, fix billing systems, and manage tax jurisdictions across multiple services. They had to use tools like Docker, Kubernetes, AWS emulators, and PostgreSQL. A typical instruction was over seventeen hundred characters long.
Key facts
- 38.8% — Resolution rate of the top-performing model Fable 5.1.
- 11 — Median number of files edited in Real-SWE tasks compared to 6 in older benchmarks.
- 71.4% — Failure rate of model rollouts that lasted under 10 minutes.
- 0.0% — Pass rate for the analytics stream reducer task across all models.
Why it matters
This proves coding agents are not ready to replace human engineers in complex systems. If you build AI coding tools, public benchmarks like FrontierCode or DeepSWE are no longer enough. You need to test against messy and undocumented company standards. The benchmark shows that agents fail mostly because they miss requirements or make unverified assumptions about the system. They guess how the code works instead of checking the workspace. This is a fatal flaw in production environments where a wrong guess breaks billing or customer data.
The gap between synthetic benchmarks and real-world utility is massive. Enterprises care about strict coding patterns and cross-functional changes. Models that look like geniuses on standard tests look like juniors when forced to edit eleven files to fix a billing schedule. Six out of ten tasks had resolution rates below fifteen percent. The analytics stream reducer task had a zero percent pass rate across every single model. This creates a massive market for context-aware developer tools. The winners will be founders who build infrastructure that forces agents to understand existing architecture before writing a single line of code.
For builders
Build better context retrieval systems
Models fail because they guess instead of checking the workspace. GPT-5.6 Sol failed 43.3 percent of the time due to unverified assumptions. Founders who build tools that force agents to verify assumptions before writing code will win lucrative enterprise contracts.
Target niche enterprise workflows
Six out of ten tasks had resolution rates below fifteen percent. Complex tasks like analytics stream reducers had a zero percent pass rate. Startups that solve these specific and hard enterprise problems can charge premium prices to engineering teams.
Focus on cross-file code generation
The median Real-SWE task requires editing eleven different files. Most current agents struggle to maintain context across that many changes. If you can build an agent that reliably executes cross-functional changes without regressions, you have a massive competitive advantage.
My take
I am tired of seeing models ace synthetic coding tests only to break my production builds. Real-SWE finally proves what every builder already knew. We do not need smarter models. We need agents that actually read the damn codebase before they start typing.
Original reporting: Hacker News. This is my rewrite and opinion.