# Devin and AI Software Engineers: What the Demos Say About Developer Jobs

Author: Marin Niehues, Niehues Consulting · Published: 2024-03-18 · AI Strategy & Business Case
Source: https://www.niehuesconsulting.de/en/blog/devin-ai-software-engineer/

Devin shows that AI agents can solve narrowly defined programming tasks on their own in part, but only a small share of tasks succeed without human rework. For decision-makers this means the demos prove a new capability and do not yet prove a replacement for development teams. Whether such an agent saves time depends on the success rate as well as the review effort, and both can be measured on your own code before any investment.

This article places the Devin announcement in context, calculates the economics of an AI software engineer and gives decision-makers a checklist for a pilot. The pressure is real: the demo videos spread quickly, and with them the question whether your own engineering function is set up correctly. It deserves an answer with numbers.

**An AI agent for software saves time when its success rate exceeds the ratio of review effort to task effort. With a four-hour task and half an hour of review, that threshold sits at just over twelve per cent.**

## What did Devin show, and what is proven?

The company Cognition presented Devin on 12 March 2024 and called it the first AI software engineer. The agent works in its own environment with a shell, a code editor and a browser. It plans a task in steps, writes code, runs tests, then corrects errors. In the demonstrations it handles jobs such as setting up an application, fixing a bug in an open-source project and working through a freelance platform listing. So far only selected applicants have been given access.

What is documented is a result on a public test. The SWE-bench benchmark, introduced in 2023 by a research group around Carlos Jimenez, comprises 2,294 real issues and change requests from twelve widely used Python projects. A system counts as successful if its change passes the associated tests. According to Cognition's technical report of 15 March 2024, Devin solved 79 of 570 randomly drawn tasks, a rate of 13.86 per cent, without being told which files were affected. The best previously published result without such help was 1.96 per cent. That is a large jump, and it also means that more than 86 per cent of the tasks remained unsolved.

Two points matter for interpretation. The measurement comes from the vendor and has not yet been independently repeated. And the test tasks come from twelve open-source Python projects and each brings test cases with it, which is rarely the case in grown enterprise systems. How decision-makers handle success stories of this kind is described in the article on [checking AI success stories](/en/blog/ai-case-studies-check/).

## Will an AI software engineer replace developers?

On the available evidence an AI software engineer does not replace development teams, because software development consists of more than working through prepared tickets. A team's tasks include clarifying requirements with business units as well as designing and weighing the architecture, coordinating with other teams and running the live system including fault-finding. The demos show progress on one part of this: the execution of narrowly defined tasks in familiar code.

For that part there is evidence that AI tools speed developers up. In a controlled experiment by researchers at Microsoft Research, GitHub and MIT, developers using the GitHub Copilot assistant completed a programming task, writing a web server in JavaScript, 55.8 per cent faster on average than the comparison group. The study by Peng and colleagues appeared in 2023 and measures one defined task, not a team's working day. It does show where things are heading: the work shifts from writing to specifying, reviewing and deciding.

This has consequences for roles. When code is produced faster, the bottleneck becomes review: who understands whether a change is right, and who answers for it? The share of time that experienced people spend on reviews and specification as well as architecture grows. Experience gains in value as a result. Consider also the effect on the codebase: whoever produces code in volume produces debt in volume, as the article on [technical debt](/en/blog/technical-debt-management/) shows.

## How does an AI agent for software pay off?

An AI agent saves time when the success rate times the task effort exceeds the review effort per attempt. The calculation is simple: the expected saving per attempt is the success rate times the hours a person would need for the task, less the hours a person needs to review the result. Review occurs on every attempt, including failed ones. The following example is illustrative.

**Calculation for an AI agent in development**
Criterion | Success rate 14 per cent | Success rate 30 per cent | Success rate 60 per cent
Effort of one task without agent | 4 hours | 4 hours | 4 hours
Review effort per attempt | 0.5 hours | 0.5 hours | 0.5 hours
Saving on success | 4 hours | 4 hours | 4 hours
Expected saving per attempt | 0.56 hours | 1.2 hours | 2.4 hours
Less review effort | minus 0.5 hours | minus 0.5 hours | minus 0.5 hours
Net result per attempt | 0.06 hours | 0.7 hours | 1.9 hours
Assessment | At the threshold, no gain | Clear gain | High gain

Break-even is review effort divided by task effort: 0.5 divided by 4 is 12.5 per cent. A success rate at the level of the published benchmark therefore sits at this threshold, and every improvement in the rate or reduction in review effort shifts the result markedly. More important than the rate on someone else's benchmark is your own rate on your own tasks. There, test coverage and documentation as well as the way tasks are cut decide the outcome. The value of the time saved also rests on a question that the article on [Sora and generative video](/en/blog/sora-generative-video/) asks of AI announcements in general: what would the benefit be if the announcement holds, and what does it cost to act too early or too late?

## How do you test an AI software engineer in a pilot?

Test an AI software engineer with a limited pilot on real tasks from your own code, with measures fixed in advance. Select 30 to 50 completed tasks with test cases from the past year, let the agent work on them and measure the success rate and the review effort. You then enter the figures in the calculation.

**Pilot checklist**
1. Which task types should be covered, and how many completed reference tasks with tests are available?
2. What is the success rate without rework, and what is it with rework?
3. How many minutes does an experienced developer need on average to review a result?
4. Which repositories and data as well as systems can the agent access, and who approved that?
5. Which quality rules apply to the generated code, and how do they feed into the definition of done?
6. Who is responsible for the changes in operation?

**Maturity levels in the use of AI in development**
Criterion | Level 1: Individuals use assistants | Level 2: Team uses assistants with rules | Level 3: Agents work on tasks
Control | None | Rules for review and data access | Task scoping, review path, measures
Measurement | Perceived benefit | Lead time and defect rate in the team | Success rate and review effort per task type
Prerequisite | Access | Shared quality rules | Good test coverage, clear responsibility
With Niehues Consulting | Inventory of usage | Business case and rollout | Pilot with stop criteria and operational ownership

The maturity levels cannot be skipped. A team that still uses assistants without rules has neither measurement nor a review path for agents. How to tell hype from progress is described in the article on the [AI hype cycle](/en/blog/ai-hype-cycle-progress-versus-announcement/).

## What does this mean for developer jobs?

For developer jobs it means a shift of emphasis and no end of the profession: repeatable implementation work gets faster, while specification and review as well as architecture gain weight. The demands on experience rise, because somebody has to judge whether generated code is right. For managers the question arises of how newcomers learn when simple tasks disappear. It pays to create early on the routes by which junior staff practise reviewing and understanding.

Capacity planning should rest on measured values. Whoever cuts posts because a video is convincing decides without measurement. Whoever runs a pilot has figures for their own organisation within a quarter.

## Devin and AI software engineers with Niehues Consulting

[Niehues Consulting](/en/) is a consultancy for AI management and organisational design from the delivery perspective, based in Düsseldorf. Founder Marin Niehues brings AI into operations, in projects from VC-funded scale-ups to DAX-40 corporations and in more than 20 countries on six continents. In the service area of AI management, the work covers business analysis, business case and the selection and sequence of applications; for prototypes and rollout it covers the enablement of users and responsibility in operations.

The approach follows the Niehues Delivery Model. For AI in software development, three of its check questions matter: What is the baseline, and how do we measure success? Does the prototype prove the benefit in the real process with real data? Is the business case confirmed with real values? The scope and terms of an engagement are discussed in the free initial consultation, which lasts 30 minutes and can be booked through the [contact page](/en/#kontakt).

**Conclusion**
Devin shows a new capability and not yet a replacement for development teams: fewer than one in seven benchmark tasks was solved, and the result comes from the vendor. Whether an AI agent pays off is decided in the end by your own success rate against the review effort. A pilot with 30 to 50 of your own tasks delivers that answer within a few weeks.

**Sources**
- Cognition AI: Introducing Devin, the first AI software engineer, company announcement, 12 March 2024
- Cognition AI: SWE-bench technical report, vendor report, 15 March 2024
- Jimenez et al.: SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, research paper, 2023
- Peng, Kalliamvakou, Cihon, Demirer: The Impact of AI on Developer Productivity: Evidence from GitHub Copilot, research paper, 2023

## Frequently asked questions

### What is Devin?

Devin is an AI agent from the company Cognition that works on programming tasks in its own environment with a shell and an editor as well as a browser. It plans steps, writes code, runs tests and then corrects errors. Cognition presented it on 12 March 2024, and access is so far limited.

### How good is Devin compared with other AI coding tools?

According to the vendor, Devin solved 13.86 per cent of the tasks in a part of the SWE-bench benchmark, while the best earlier result without help was 1.96 per cent. The comparison concerns a different field of use from assistants such as GitHub Copilot, which support developers while they write. An independent repetition of the measurement does not yet exist.

### Can AI replace software engineers?

AI can take over parts of the work but does not replace development teams. The demos show the execution of narrowly defined tasks, while requirements and architecture as well as coordination and operations stay with people. The role shifts towards specification and review.

### What is SWE-bench?

SWE-bench is a test with 2,294 real issues and change requests from twelve Python projects, introduced in 2023 by Jimenez and colleagues. A system passes a task if its change satisfies the associated tests. The test measures the solution of defined tasks, not whole development work.

### How much faster does AI make developers?

In a controlled experiment by Peng and colleagues (2023), developers using GitHub Copilot completed a defined task 55.8 per cent faster on average. The result applies to that task and not to a whole working day. Measurements in your own team show the real effect.

### Is an AI software engineer worth the money?

It is worth it when the success rate exceeds the ratio of review effort to task effort. With a four-hour task and half an hour of review, the threshold is 12.5 per cent. You have to measure the rate on your own tasks.

### How do you start a pilot with an AI agent?

Select 30 to 50 completed tasks with test cases, let the agent work on them and measure success rate and review effort. Set stop criteria and access rights in advance. The result goes into the calculation and into the decision on use.

### What are the risks of AI-generated code?

The greatest risk is unreviewed code that brings technical debt and defects into operations. Further risks concern the agent's access rights to code and data. A definition of done with tests and review as well as named responsibility limits both.