AI Benchmarks Ignore 90 Percent of the Work Americans Actually Do
Existing AI benchmarks heavily favor high-wage tech and legal jobs, leaving massive blind spots in what models can actually do.
Research · Source: Hacker News
What happened
Mercor Research mapped 29 popular AI benchmarks against the actual US economy. They used the Department of Labor taxonomy, which breaks the workforce into 923 distinct occupations and nearly 19,000 specific tasks. The results show a massive disconnect between AI testing and real-world labor. Existing benchmarks cover only a tiny fraction of actual jobs.
Researchers found that 90 percent of American working hours are spent on tasks with almost zero benchmark coverage. Only 208 out of 923 occupations have a single task covered by at least five benchmark prompts. Entire sectors of the economy are completely ignored by the AI community. Six out of 22 major job families have zero occupations covered by these tests.
Benchmark creators are clearly chasing the money. Coverage heavily skews toward high-wage, highly AI-exposed roles like software engineering, financial analysis, and corporate law. Meanwhile, middle-income knowledge workers like legal secretaries, court clerks, and administrative staff are left out entirely. The data proves that we have a massive blind spot in understanding what AI models can actually do in the real economy.
Key facts
- 90 percent — Share of US working hours spent on tasks with little to no AI benchmark coverage
- 208 — Number of occupations out of 923 that have a task with at least five benchmark prompts
- 6 — Number of major job families out of 22 with zero covered occupations
- 18,797 — Total number of tasks analyzed across the US workforce
Why it matters
You cannot trust generic benchmark scores to tell you if an AI model will work for your specific enterprise customer. If you are building tools for logistics, administration, or middle-market services, frontier model scores are essentially useless. The current benchmarks are designed to measure theoretical AI progress, not practical economic utility. You have to build your own evaluation pipelines from scratch to prove return on investment to buyers. Without custom benchmarks, you are flying blind in the enterprise market.
This creates a massive wedge between perceived AI progress and actual economic automation. Policymakers and executives are making sweeping decisions based on benchmarks that only represent a tiny elite slice of the workforce. They assume AI is ready to automate everything because it passes a bar exam or writes Python code. As a result, venture capital will over-index on automating high-end knowledge work while ignoring highly profitable, unsexy middle-market automation opportunities. The real economic impact of AI will be delayed because we are not even measuring the right things.
For builders
Build niche evaluation datasets
General benchmarks fail completely outside of tech and finance. Founders who build proprietary, task-specific evaluation sets for unsexy industries can charge a massive premium. Enterprise buyers will pay handsomely for concrete proof that a model actually works for their specific daily workflows.
Target middle-income knowledge gaps
The AI community is obsessed with replacing software engineers and corporate lawyers. Middle-income roles like court clerks, data entry clerks, and administrative staff are completely ignored by researchers. Builders who target these overlooked verticals face zero competition from baseline model benchmarks and can dominate the market.
Ignore academic leaderboards entirely
Stop using public leaderboards to sell your product to traditional businesses. Customers in healthcare, manufacturing, and local government do not care about these scores. You will lose deals if you rely on generic benchmarks instead of showing direct automation of their specific tasks.
My take
I am tired of seeing founders brag about models beating benchmarks that have absolutely nothing to do with reality. If you want to build a real business, stop optimizing for academic leaderboards and start measuring how well your AI does the actual grunt work your customers pay for. The biggest opportunities right now are in the 90 percent of the economy that researchers are too lazy to measure.
Original reporting: Hacker News. This is my rewrite and opinion.