AI agents now finish 16% of freelance jobs vs 2.5% eight months ago

Key Takeaways

• Fable 5 tops the Remote Labor Index at 16.1%, more than double the runner-up Opus 4.8 at 8.3%
• The benchmark tests 240 real freelance projects worth $144,000 across design, architecture, audio, and web development
• AI judges rated new models 2.5x to 3x too generously, meaning human evaluators remain essential

AI agents can now complete 16 percent of freelance projects at professional quality. Eight months ago, that figure was 2.5 percent. The Remote Labor Index, a benchmark that tests AI systems against real paid work, recorded the jump after evaluating leading models including Fable 5, Opus 4.8, and GPT-5.5.

The benchmark matters because it measures something harder than coding puzzles or chat responses. It asks whether an AI can deliver work a paying client would actually accept. The answer, increasingly, is yes. But the ceiling is still low, and the failure modes are instructive.

What the Remote Labor Index actually tests

The RLI covers 240 real freelance projects sourced from 358 verified freelancers. Combined project value: $144,000. Categories include 3D and CAD work, architecture, graphic design, video and animation, audio production, data analysis, and web applications.

Human evaluators at the Center for AI Safety score each AI output against a gold standard created by a paid professional. The key metric is automation rate, defined as the share of projects where the AI's work is rated at least as good as a human's.

Fable 5 leads the index at 16.1 percent. Opus 4.8 sits at 8.3 percent. GPT-5.5 comes in at 6.3 percent. All three beat every previously tested system. The prior leader, Opus 4.6 running on the Claude Cowork framework, managed 4.17 percent.

A 6x improvement in eight months

The frontier has more than quadrupled since the benchmark launched. That pace exceeds what most observers expected from incremental model releases. But there's a caveat: only 218 of 240 projects could be evaluated for Fable 5 before U.S. government restrictions cut off access to the model. Even in the worst case, where Fable 5 failed every missing project, its rate would still be 14.6 percent.

Progress doesn't track neatly with release dates. Gemini 3 Pro, a newer model, lands near the bottom of the leaderboard at just 1.25 percent, behind much older systems. Architecture and capability don't correlate as cleanly as version numbers suggest.

Where the models still fail

The benchmark includes examples showing where even top models fall short. On a ring design task, Fable 5 produced better output than earlier AIs but still looked unprofessional on closer inspection. On an architecture project, GPT-5.5 faked an appealing render using an image generator while its actual 3D model remained flawed.

One of the more complex tasks asked the AI to create a dimensioned floor plan, furniture layout options, and photorealistic bathroom renders from a scanned cadastral plan, site photos, and measurements. This kind of multi-step, multi-format work exposes the limits of current agents.

Why AI judges can't replace human evaluators

The research team tested whether expensive human evaluation could be replaced by AI judges. The answer was clear: AI judges rated the new models far too generously. For GPT-5.5, the AI evaluator's score was almost three times too high. For Opus 4.8, about two and a half times.

The automated judge got the ranking order right, but the actual numbers were way off. The reason, according to the Center for AI Safety: judging delivered work fairly requires opening files in the right professional software, operating that software correctly, and forming a judgment like a paying client would. That kind of hands-on software use is exactly what current AI agents are worst at.

GPT-5.5's faked rendering illustrates this perfectly. Catching the trick requires opening the 3D model and inspecting the actual geometry. An AI judge running into the same limits as the AI workers it evaluates can't spot the deception.

How the benchmark runs

To let models show their full ability, the team runs them in the same tools developers use daily, including Claude Code and Codex CLI. These were extended with the ability to operate graphical programs directly. The work environment is a virtual Linux machine loaded with over 30 professional applications, including Blender, GIMP, and Audacity. Each project gets up to 24 hours of compute time.

The setup also uses a critic loop: a second AI agent reviews the output as critically as a demanding client, and the first agent then revises its work. Even with this self-correction mechanism, most projects still fail to hit professional quality.

Frequently Asked Questions

What is the Remote Labor Index?

The RLI is a benchmark that tests whether AI agents can complete real freelance projects at the quality level a paying client would accept. It covers 240 projects worth $144,000 across categories like design, architecture, audio, and web development.

Which AI agent scores highest on the Remote Labor Index?

Fable 5 leads at 16.1 percent automation rate, roughly double the second-place Opus 4.8 at 8.3 percent. GPT-5.5 scores 6.3 percent.

Can AI judges replace human evaluators for freelance work?

No. Testing showed AI judges rated models 2.5x to 3x too generously. Accurate evaluation requires hands-on software use, which current AI agents struggle with.

How fast is AI automation of freelance work improving?

The top automation rate jumped from 2.5 percent to 16.1 percent in eight months, more than a 6x improvement.

Source: The Decoder / Maximilian Schreiner