Study Finds AI Coding Assistants Deliver Marginal Returns

Compiled by the editorial desk with reference to the Bain & Company report, the Model Evaluation & Threat Research study, Stack Overflow survey, and Apiiro security data.

Generative AI's promise to supercharge software development has hit a reality check, according to a new report from management consultancy Bain & Company. Despite being one of the first enterprise areas to adopt the technology, the study found that productivity savings have been unremarkable, with developer adoption remaining low even among companies that have rolled out AI tools.

The report, which analyzed early deployments, noted that while some coding assistants achieved ten to fifteen percent productivity boosts, these gains usually didn't translate into positive returns. "Generative AI arrived on the scene with sky-high expectations, and many companies rushed into pilot projects," the report states. "Yet the results haven't lived up to the hype."

Mixed Results and Slower Developers

The findings align with a July study from the nonprofit Model Evaluation & Threat Research, which found that AI tools could actually slow developers down. "Surprisingly, we find that when developers use AI tools, they take 19 percent longer than without — AI makes them slower," the researchers wrote. The slowdown was attributed to hallucinations that forced developers to spend extra time cleaning up and reviewing code, with the tools also struggling on large and complex code repositories.

Developer sentiment appears to be souring as well. A survey by programming website Stack Overflow earlier this year found that while more developers are using AI tools, trust in them has declined. Erin Yepis, senior analyst for market research and insights at Stack Overflow, told VentureBeat in July that the shift was surprising given the industry's investment in AI. "This response is surprising because with all of the investment in and focus on AI in tech news, I would expect that the trust would grow as the technology gets better," she said.

Security Concerns and Measurement Gaps

Beyond productivity, security risks are emerging. A recent report from security firm Apiiro found that developers using AI produce ten times more security problems than those who don't. This adds to concerns about the technology's reliability, especially as companies pour billions into AI development amid fears of an industry bubble.

The Bain report also highlights a lack of agreed-upon metrics for tracking productivity gains, a glaring omission given the scale of investment. To realize promised benefits, the consultancy argues that companies must commit to broader changes. "Real value comes from applying generative AI across the entire software development life cycle, not just coding," the report reads. "Nearly every phase can benefit, from the earlier discovery and requirements stages, through planning and design, to testing, deployment, and maintenance."

"Broad adoption, however, requires process changes," the consultancy added. "If AI speeds up coding, then code review, integration, and release must speed up as well to avoid bottlenecks."

Whether the next wave of "agentic AI" — systems designed to autonomously execute tasks — will change the outlook remains uncertain. The Bain report suggests that only companies that act decisively, redesigning architecture and workflows, will unlock value. For now, the evidence points to a technology that, in its current form, often falls short of its billing.