# Checking AI Success Stories: What Stands Behind Your Competitors' Case Studies

Author: Marin Niehues, Niehues Consulting · Published: 2024-02-15 · AI Strategy & Business Case
Source: https://www.niehuesconsulting.de/en/blog/ai-case-studies-check/

You check AI success stories with eight questions that run from baseline and measurement period through comparison group and operating costs to the source of the figure and its transferability. A case study that leaves three of them unanswered shows a possibility, but not yet evidence. This article explains why competitors appear so much more successful with AI than they are, and how decision makers turn other people's figures into a calculation of their own.

The pressure often arises from a single slide at a conference, where markedly shorter processing times or savings running into millions are mentioned. Management asks why its own company does not stand there. That is a fair question, but it needs a checked answer before budget flows. The article on the [AI hype cycle](/en/blog/ai-hype-cycle-progress-versus-announcement/) shows how announcements and progress can be told apart. This one concerns the individual document: the case study.

**A success story proves something when the baseline, the comparison and the net benefit are open to inspection. If one of them is missing, it remains a claim to be used as inspiration and not as a benchmark.**

## Why do AI success stories look better than reality?

AI success stories look better than reality because only the successful cases are published and because the figures usually come from the people involved. Four mechanisms stand behind this, and decision makers should know them.

The first is selection. Companies publish what worked and leave out projects that ended at pilot stage. Whoever sees only the wins overestimates the success rate. The path from pilot to operation is long, as the 2015 paper by Sculley and colleagues on technical debt in machine learning describes: the model code is only a small part of the system, and the rest consists of data pipelines and configuration as well as monitoring and operation.

The second mechanism is self-reporting. A figure such as "40 per cent faster" often comes from a survey of users or from the project team's estimate. Those who introduced a new tool themselves tend to rate it generously. The third is the before-and-after comparison without a control group: when order volume and staffing change in the same quarter, and the process with them, the effect of the AI cannot be isolated. The fourth is the cost side. Case studies state the benefit in the pilot and leave out operation and data maintenance, as well as training and rechecking.

## Which eight questions belong on every case study?

Eight questions cover the typical gaps in a case study. Each has an answer that a reliable case supplies and a warning sign that points to a claim.

**Checking grid for AI case studies**
Question | Warning sign | Answer in a reliable case
1 Baseline | Only the value after introduction is stated | Measured value before introduction, with source
2 Measurement period | "In the pilot" or "in the first weeks" | Several months in regular operation
3 Comparison group | Before and after without control | Simultaneous comparison group without AI or random assignment
4 Gross and net | Benefit without costs | Benefit minus operation and upkeep as well as training
5 Indicator | Percentage without a reference base | Absolute values and definition of the measurement
6 Source | Self-report by the vendor or the users | Independent measurement or data from the process
7 Spread | Only the average | Distribution by user experience and type of task
8 Transferability | Industry and data situation unnamed | Description of the data and the process as well as the organisation
With Niehues Consulting | Other people's figures become the starting point of a calculation of your own | Own measurement in the real process with real data

The questions can be put to any document within an hour. Whoever answers them before a supplier conversation sees which figures are reliable and which the supplier must substantiate.

## What does reliable evidence look like?

Reliable evidence has a comparison group, a precise measurement and an open spread. Three scientific papers from 2023 show what that looks like and what comes out of it.

A field study by the research team around Erik Brynjolfsson examined around 5,000 customer support agents working with an AI assistant. Issues resolved per hour rose by 14 per cent on average and by 34 per cent for novices, but hardly at all for experienced agents. The average alone hides that the benefit rises and falls with experience. A company with mostly experienced staff therefore must not adopt the average value.

A randomised study by Noy and Zhang with 453 professionals found, for writing tasks, a 40 per cent shorter completion time and an 18 per cent better quality rating from independent assessors. These were short, self-contained experimental tasks, so the time saving may be smaller in a running business, where checking the output and phrasing the instructions take time.

A third study by Dell'Acqua and colleagues with 758 consultants shows the limit. For tasks within the capabilities of the model, participants with AI completed on average 12.2 per cent more tasks and about a quarter faster. For a task outside those capabilities, the share of correct solutions with AI was 19 percentage points lower than without. A reader of a success story should therefore ask whether the task in their own business lies inside or outside this frontier.

All three papers have comparison groups and state their samples. They also report differences by experience or type of task. Marketing case studies rarely do. The articles on [ChatGPT in the enterprise](/en/blog/chatgpt-in-business-sober-review/) and on the [AI year in review 2023](/en/blog/ai-review-2023/) put into context what companies had actually implemented by the end of the year.

## How do you transfer someone else's figure to your own business?

You transfer another party's figure by multiplying the measured effect by the share of employees, the time share of the task and the part of the time gain that can actually be used. The result is an estimate of your own, which you check in a prototype with real data.

**Calculation with assumed values**
Step | Calculation | Example value
Effect according to the study | More issues resolved per hour on average | 14 per cent
Employees affected | Service staff who use the tool (assumed) | 40
Time share of the task | Share of working time spent on the studied task (assumed) | 60 per cent
Arithmetical gain | 40 times 0.6 times 0.14 | 3.4 full-time posts
Usable share | Share that becomes capacity and not idle time (assumed) | 50 per cent
Gain after realisation | 3.4 times 0.5 | 1.7 full-time posts
Experience discount | A high share of experienced staff lowers the effect (assumed) | Check for a further reduction

The calculation shows two things. The published percentage shrinks considerably on the way to operation, and every factor can be checked before the decision. How you assess your own position against competitors is shown in the article on [competitive pressure on AI](/en/blog/ai-competitive-pressure-falling-behind/).

**Five questions for the sender of a case study**
- Which baseline existed before introduction, and who measured it?
- How long did the measurement run, and did it run in regular operation?
- Was there a comparison group without the tool?
- Which costs of operation and upkeep are deducted from the benefit?
- Which data were used, and how similar is the process to ours?

## What if a competitor really is ahead?

If a competitor demonstrably has a lead, your own proof is faster than copying their initiative. A prototype in the real process with real data answers within a few weeks whether the effect arises in your business. The initiative is placed by value and feasibility, and management decides on its own data. A company that rebuilds other people's cases also takes over their assumptions about data and processes, which need not hold for its own business.

## Checking success stories with Niehues Consulting

[Niehues Consulting](/en/) is a consultancy for AI management and organisation design from a delivery perspective, based in Düsseldorf. Founder Marin Niehues brings AI into operations, in projects ranging from VC-funded scale-ups to DAX 40 groups and in more than 20 countries on six continents. In the service area of AI management, business analysis and the business case belong to the work, as does the selection of applications, which are exactly the steps that turn another party's figure into a decision of your own.

The approach follows the Niehues Delivery Model. Three checking questions matter most for case studies: What is the baseline, and how do we measure success? Does the prototype prove the benefit in the real process with real data? Is the business case confirmed with real values? The entry point is a free initial consultation of 30 minutes, which you can book through the [contact form](/en/#kontakt).

**Conclusion**
An AI success story is a reason to check and not a benchmark. Eight questions separate evidence from claim, and the calculation transfers the foreign effect to your own business. The road to a decision runs through a prototype with your own data and a business case with real values.

**Sources**
- Brynjolfsson, Li, Raymond: Generative AI at Work, NBER Working Paper 31161, 2023
- Noy, Zhang: Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence, Science, 2023
- Dell'Acqua et al.: Navigating the Jagged Technological Frontier, Harvard Business School Working Paper, 2023
- Sculley et al.: Hidden Technical Debt in Machine Learning Systems, NeurIPS conference paper, 2015

## Frequently asked questions

### How do you recognise a reliable AI case study?

A reliable case study states the baseline, the measurement period and a comparison group. It reports the benefit minus operating costs and describes data and process. If several of these details are missing, treat the figure as a claim.

### Why do companies publish only successes?

Companies publish successes because case studies are marketing and failed projects are rarely told. This distorts the picture, because the failures are missing. Anyone estimating the success rate has to think of the invisible projects as well.

### Can productivity figures from AI studies be applied to my company?

Productivity figures can be applied when task, user experience and data situation are similar. The study on support agents shows that the effect was clearly higher for novices than for experienced agents. The calculation in this article converts the value to your own business.

### What are typical mistakes in AI case studies?

Typical mistakes are a missing baseline, a before-and-after comparison without a control group and benefit figures without operating costs. The numbers often come from self-reporting by users. Averages that hide differences between user groups add to this.

### How do you calculate the benefit of an AI application for your own company?

You multiply the measured effect by the number of users, the time share of the task and the usable part of the time gain. Then you deduct the operating costs. You check the result in a prototype with real data.

### Which studies exist on the benefit of AI?

There are field studies on support agents, experiments on writing tasks and investigations into knowledge work, each working with comparison groups. They show double-digit productivity gains on suitable tasks. For tasks outside the capabilities of the model, quality fell.

### What does it cost to rebuild an AI success story?

The cost follows the scope of the initiative, and the case study usually does not state it in full. Besides development, plan for data maintenance and operation as well as training. Scope and terms of an engagement are discussed in the free initial consultation.

### Should a company follow a competitor on AI immediately?

A company should follow a competitor when the lead is proven and its own prototype confirms it. An early start without checking ties up budget without benefit. A prototype with your own data provides the basis for the decision within a few weeks.