Coding agents skip open source guidelines, study finds

AI coding agents regularly bypass contribution guidelines when submitting to open source projects, according to new research. The finding exposes a gap between how these tools perform on benchmarks and how they behave in real repositories with established norms.

The study, reported by The New Stack, examined how AI agents interact with CONTRIBUTING.md files, code style guides, and other project-specific requirements. Instead of reading and following these documents, agents frequently generate code that violates formatting rules, skips required tests, or ignores commit message conventions.

Why agents fail at real-world contributions

Open source projects encode years of accumulated decisions into their contribution guidelines. A CONTRIBUTING.md might require signed commits, specific branch naming, or mandatory CI checks. Style guides enforce tab versus space, import ordering, documentation standards. These rules exist because maintainers learned, often painfully, what keeps a codebase healthy.

Coding agents trained on generic code patterns don't prioritize these project-specific constraints. They optimize for syntactically correct, functionally plausible code. Whether that code matches the project's established patterns is a secondary concern, if it registers at all.

The numbers bear this out. Research from METR (Model Evaluation & Threat Research) found that AI agents achieved only 7.5% autonomous success on real GitHub issues. With human guidance, that figure climbed to 41%. The gap suggests agents can produce working code but lack the contextual awareness to produce acceptable contributions.

The benchmark problem

Benchmark environments strip away the messy realities of real repositories. There's no CONTRIBUTING.md to read, no CI pipeline to satisfy, no maintainer preferences to intuit. An agent that scores well on HumanEval or SWE-bench may still produce pull requests that get rejected on sight.

METR's research showed agents performing 25-50% worse on real-world open source tasks compared to controlled environments. That's not a minor calibration error. It's a structural mismatch between what agents are trained to do and what maintainers actually need.

What this means for engineering teams

For teams using AI coding assistants internally, the implications are manageable. You control the environment, enforce standards through CI, and can train developers to review agent output against your own guidelines.

For open source maintainers, the future looks noisier. As more developers delegate contribution tasks to agents, maintainers will see more PRs that ignore documented requirements. The burden shifts to reviewers who must either reject mechanically or spend time explaining norms the contributor's agent could have read.

Some projects already flag agent-generated PRs for additional scrutiny. Others have started requiring explicit acknowledgment that the contributor read the guidelines. Neither solution scales gracefully.

Can agents learn project norms?

In theory, yes. An agent could be instructed to read CONTRIBUTING.md first, parse style guides, and check its output against those requirements before submitting. Some enterprise deployments already implement this workflow.

The harder question is whether agents can infer norms that aren't explicitly documented. Experienced contributors pick up on patterns: this project prefers small PRs; that maintainer wants verbose commit messages; another expects you to open an issue before submitting code. These conventions live in git history and issue threads, not README files.

Teaching agents to read social context, not just code context, remains an open research problem. Until it's solved, human judgment stays in the loop.

Source: The New Stack / Meredith Shubel