Checking AI Success Stories: What Stands Behind Your Competitors' Case Studies

You check AI success stories with eight questions that run from baseline and measurement period through comparison group and operating costs to the source of the figure and its transferability. A case study that leaves three of them unanswered shows a possibility, but not yet evidence. This article explains why competitors appear so much more successful with AI than they are, and how decision makers turn other people's figures into a calculation of their own.

The pressure often arises from a single slide at a conference, where markedly shorter processing times or savings running into millions are mentioned. Management asks why its own company does not stand there. That is a fair question, but it needs a checked answer before budget flows. The article on the AI hype cycle shows how announcements and progress can be told apart. This one concerns the individual document: the case study.

Why do AI success stories look better than reality?

AI success stories look better than reality because only the successful cases are published and because the figures usually come from the people involved. Four mechanisms stand behind this, and decision makers should know them.

The first is selection. Companies publish what worked and leave out projects that ended at pilot stage. Whoever sees only the wins overestimates the success rate. The path from pilot to operation is long, as the 2015 paper by Sculley and colleagues on technical debt in machine learning describes: the model code is only a small part of the system, and the rest consists of data pipelines and configuration as well as monitoring and operation.

The second mechanism is self-reporting. A figure such as “40 per cent faster” often comes from a survey of users or from the project team's estimate. Those who introduced a new tool themselves tend to rate it generously. The third is the before-and-after comparison without a control group: when order volume and staffing change in the same quarter, and the process with them, the effect of the AI cannot be isolated. The fourth is the cost side. Case studies state the benefit in the pilot and leave out operation and data maintenance, as well as training and rechecking.

Which eight questions belong on every case study?

Eight questions cover the typical gaps in a case study. Each has an answer that a reliable case supplies and a warning sign that points to a claim.

The questions can be put to any document within an hour. Whoever answers them before a supplier conversation sees which figures are reliable and which the supplier must substantiate.

What does reliable evidence look like?

Reliable evidence has a comparison group, a precise measurement and an open spread. Three scientific papers from 2023 show what that looks like and what comes out of it.

A field study by the research team around Erik Brynjolfsson examined around 5,000 customer support agents working with an AI assistant. Issues resolved per hour rose by 14 per cent on average and by 34 per cent for novices, but hardly at all for experienced agents. The average alone hides that the benefit rises and falls with experience. A company with mostly experienced staff therefore must not adopt the average value.

A randomised study by Noy and Zhang with 453 professionals found, for writing tasks, a 40 per cent shorter completion time and an 18 per cent better quality rating from independent assessors. These were short, self-contained experimental tasks, so the time saving may be smaller in a running business, where checking the output and phrasing the instructions take time.

A third study by Dell'Acqua and colleagues with 758 consultants shows the limit. For tasks within the capabilities of the model, participants with AI completed on average 12.2 per cent more tasks and about a quarter faster. For a task outside those capabilities, the share of correct solutions with AI was 19 percentage points lower than without. A reader of a success story should therefore ask whether the task in their own business lies inside or outside this frontier.

All three papers have comparison groups and state their samples. They also report differences by experience or type of task. Marketing case studies rarely do. The articles on ChatGPT in the enterprise and on the AI year in review 2023 put into context what companies had actually implemented by the end of the year.

How do you transfer someone else's figure to your own business?

You transfer another party's figure by multiplying the measured effect by the share of employees, the time share of the task and the part of the time gain that can actually be used. The result is an estimate of your own, which you check in a prototype with real data.

The calculation shows two things. The published percentage shrinks considerably on the way to operation, and every factor can be checked before the decision. How you assess your own position against competitors is shown in the article on competitive pressure on AI.

What if a competitor really is ahead?

If a competitor demonstrably has a lead, your own proof is faster than copying their initiative. A prototype in the real process with real data answers within a few weeks whether the effect arises in your business. The initiative is placed by value and feasibility, and management decides on its own data. A company that rebuilds other people's cases also takes over their assumptions about data and processes, which need not hold for its own business.

Checking success stories with Niehues Consulting

Niehues Consulting is a consultancy for AI management and organisation design from a delivery perspective, based in Düsseldorf. Founder Marin Niehues brings AI into operations, in projects ranging from VC-funded scale-ups to DAX 40 groups and in more than 20 countries on six continents. In the service area of AI management, business analysis and the business case belong to the work, as does the selection of applications, which are exactly the steps that turn another party's figure into a decision of your own.

The approach follows the Niehues Delivery Model. Three checking questions matter most for case studies: What is the baseline, and how do we measure success? Does the prototype prove the benefit in the real process with real data? Is the business case confirmed with real values? The entry point is a free initial consultation of 30 minutes, which you can book through the contact form.