Devin and AI Software Engineers: What the Demos Say About Developer Jobs

Devin shows that AI agents can solve narrowly defined programming tasks on their own in part, but only a small share of tasks succeed without human rework. For decision-makers this means the demos prove a new capability and do not yet prove a replacement for development teams. Whether such an agent saves time depends on the success rate as well as the review effort, and both can be measured on your own code before any investment.

This article places the Devin announcement in context, calculates the economics of an AI software engineer and gives decision-makers a checklist for a pilot. The pressure is real: the demo videos spread quickly, and with them the question whether your own engineering function is set up correctly. It deserves an answer with numbers.

What did Devin show, and what is proven?

The company Cognition presented Devin on 12 March 2024 and called it the first AI software engineer. The agent works in its own environment with a shell, a code editor and a browser. It plans a task in steps, writes code, runs tests, then corrects errors. In the demonstrations it handles jobs such as setting up an application, fixing a bug in an open-source project and working through a freelance platform listing. So far only selected applicants have been given access.

What is documented is a result on a public test. The SWE-bench benchmark, introduced in 2023 by a research group around Carlos Jimenez, comprises 2,294 real issues and change requests from twelve widely used Python projects. A system counts as successful if its change passes the associated tests. According to Cognition's technical report of 15 March 2024, Devin solved 79 of 570 randomly drawn tasks, a rate of 13.86 per cent, without being told which files were affected. The best previously published result without such help was 1.96 per cent. That is a large jump, and it also means that more than 86 per cent of the tasks remained unsolved.

Two points matter for interpretation. The measurement comes from the vendor and has not yet been independently repeated. And the test tasks come from twelve open-source Python projects and each brings test cases with it, which is rarely the case in grown enterprise systems. How decision-makers handle success stories of this kind is described in the article on checking AI success stories.

Will an AI software engineer replace developers?

On the available evidence an AI software engineer does not replace development teams, because software development consists of more than working through prepared tickets. A team's tasks include clarifying requirements with business units as well as designing and weighing the architecture, coordinating with other teams and running the live system including fault-finding. The demos show progress on one part of this: the execution of narrowly defined tasks in familiar code.

For that part there is evidence that AI tools speed developers up. In a controlled experiment by researchers at Microsoft Research, GitHub and MIT, developers using the GitHub Copilot assistant completed a programming task, writing a web server in JavaScript, 55.8 per cent faster on average than the comparison group. The study by Peng and colleagues appeared in 2023 and measures one defined task, not a team's working day. It does show where things are heading: the work shifts from writing to specifying, reviewing and deciding.

This has consequences for roles. When code is produced faster, the bottleneck becomes review: who understands whether a change is right, and who answers for it? The share of time that experienced people spend on reviews and specification as well as architecture grows. Experience gains in value as a result. Consider also the effect on the codebase: whoever produces code in volume produces debt in volume, as the article on technical debt shows.

How does an AI agent for software pay off?

An AI agent saves time when the success rate times the task effort exceeds the review effort per attempt. The calculation is simple: the expected saving per attempt is the success rate times the hours a person would need for the task, less the hours a person needs to review the result. Review occurs on every attempt, including failed ones. The following example is illustrative.

Break-even is review effort divided by task effort: 0.5 divided by 4 is 12.5 per cent. A success rate at the level of the published benchmark therefore sits at this threshold, and every improvement in the rate or reduction in review effort shifts the result markedly. More important than the rate on someone else's benchmark is your own rate on your own tasks. There, test coverage and documentation as well as the way tasks are cut decide the outcome. The value of the time saved also rests on a question that the article on Sora and generative video asks of AI announcements in general: what would the benefit be if the announcement holds, and what does it cost to act too early or too late?

How do you test an AI software engineer in a pilot?

Test an AI software engineer with a limited pilot on real tasks from your own code, with measures fixed in advance. Select 30 to 50 completed tasks with test cases from the past year, let the agent work on them and measure the success rate and the review effort. You then enter the figures in the calculation.

The maturity levels cannot be skipped. A team that still uses assistants without rules has neither measurement nor a review path for agents. How to tell hype from progress is described in the article on the AI hype cycle.

What does this mean for developer jobs?

For developer jobs it means a shift of emphasis and no end of the profession: repeatable implementation work gets faster, while specification and review as well as architecture gain weight. The demands on experience rise, because somebody has to judge whether generated code is right. For managers the question arises of how newcomers learn when simple tasks disappear. It pays to create early on the routes by which junior staff practise reviewing and understanding.

Capacity planning should rest on measured values. Whoever cuts posts because a video is convincing decides without measurement. Whoever runs a pilot has figures for their own organisation within a quarter.

Devin and AI software engineers with Niehues Consulting

Niehues Consulting is a consultancy for AI management and organisational design from the delivery perspective, based in Düsseldorf. Founder Marin Niehues brings AI into operations, in projects from VC-funded scale-ups to DAX-40 corporations and in more than 20 countries on six continents. In the service area of AI management, the work covers business analysis, business case and the selection and sequence of applications; for prototypes and rollout it covers the enablement of users and responsibility in operations.

The approach follows the Niehues Delivery Model. For AI in software development, three of its check questions matter: What is the baseline, and how do we measure success? Does the prototype prove the benefit in the real process with real data? Is the business case confirmed with real values? The scope and terms of an engagement are discussed in the free initial consultation, which lasts 30 minutes and can be booked through the contact page.