# AI Strategy After the Hype: What Remains of the First Wave

Author: Marin Niehues, Niehues Consulting · Published: 2024-09-02 · AI Strategy & Business Case
Source: https://www.niehuesconsulting.de/en/blog/ai-strategy-after-the-hype/

After the first year of hype, what remains of the AI wave are applications that measurably gain time or quality in a clearly bounded work step. A strategy built on this rates every pilot by its result in operations and gives budget only to initiatives with an owner and a metric. The experience of your own teams provides the basis.

This article is written for managing directors and heads of function who have built pilots, licences and working groups since autumn 2022 and now decide what continues. It summarises what is evidenced and offers a decision procedure for the portfolio of running initiatives.

**What remains of the first wave is what runs in a specific work step with an owner and a metric. Everything else was a learning phase, and it is worth something only if the company turns its results into a decision.**

## What remains of the AI wave?

What remains are applications that process text, data and knowledge in everyday work steps and demonstrably save time. Three academic studies from 2023 point the same way. A field study of more than 5,000 customer support agents found, on average, 14 per cent more issues resolved per hour, and 34 per cent for novices. An experiment with 453 professionals in writing occupations, published in the journal Science, found about 40 per cent shorter working time on writing tasks with higher rated quality. A field experiment with 758 consultants, led by Harvard Business School researchers, showed the limit at the same time: for tasks within the capability of the AI, quality rose clearly, while for one task outside that frontier participants with AI help were more often wrong than those without it.

A clear rule for strategy follows. AI pays off where the work step is bounded, the results can be checked and the users can judge the answer on its merits. It does not pay off where nobody can judge the result. The studies report no effect at company level, because a faster work step only becomes a better result when the process around it is adjusted.

A second finding concerns adoption. According to Eurostat, 8 per cent of enterprises in the European Union with ten or more employees used AI technologies in 2023. Broad everyday use therefore takes place in a minority of firms. Anyone who is only now assessing their pilots is at the same point as the majority.

## What was hype and what was insight?

The hype was the expectation that general AI assistants would transform entire functions without adaptation. The insight was that the benefit is set by the task and the workflow. Broad licences for all employees achieve little as long as nobody has settled how to use the tool for their own task, and usage flattens after the first weeks. The article [AI usage without results](/en/blog/ai-usage-without-results/) describes how to recognise this.

One insight concerns the technology. The effort rarely lies in the model. As early as 2015, researchers at Google showed in the paper "Hidden Technical Debt in Machine Learning Systems" that model code makes up only a small part of a production system, while data management, configuration, monitoring and interfaces make up the larger part. This holds for generative AI as well: the connection to documents, permissions and processes decides the benefit, not the choice of model. With several capable vendors on the market, the model also becomes more interchangeable, while your own data base and your own workflow do not.

A second insight concerns the organisation. An application without an owner in the business function is not maintained and is no longer used after six months. Introduction is therefore an initiative of the line, even when IT runs it technically. How budgets seeped away in the first year and how to spot it early is shown in the article [AI investments without a plan](/en/blog/ai-investment-without-a-plan/).

## How do you assess running AI pilots after the hype?

You assess running pilots with a decision table that assigns each initiative one of four answers: scale up, rebuild, observe or end. The assignment follows four questions about each pilot. Has a metric changed, and has the effect been measured? Does the application have an owner in the business function? Do the intended users still use it regularly after 90 days? Is the cost per use known?

**Decision table for running AI pilots**
Finding | Measurable effect | Owner in the business | Usage after 90 days | Decision
Pilot A | Yes, lead time measured | Yes | Regular | Scale up: plan operations and rollout
Pilot B | Yes | No | Regular | Rebuild: name an owner, then scale up
Pilot C | No, impression only | Yes | Occasional | Observe: fix a metric, decide again after a quarter
Pilot D | No | No | Rare | End: release licence and capacity

The table forces a decision on every pilot, including those nobody wants to switch off. Ending belongs to strategy: every pilot that ends frees attention and budget for the initiatives that work. A group of pilots that trigger no decision has caused costs after twelve months and delivered no insight.

A value comparison helps to rank the remaining initiatives. Set the annual relief against the annual cost of the application. An example with illustrative figures: a sales team of 20 people saves two hours per person per week through a proposal assistant. Over 44 working weeks that gives 20 times 2 times 44, or 1,760 hours a year. At an internal hourly rate of EUR 60 this equals EUR 105,600. If licences, operation and maintenance cost EUR 40,000 a year, a calculated benefit of EUR 65,600 remains. It becomes real benefit when the time gained is used for additional customer meetings.

## Which questions must an AI strategy answer now?

An AI strategy after the hype answers five questions. Which business outcomes should AI change measurably in the next twelve months? Which applications contribute, and in what order? Who is responsible for each application in operations? Which data and interfaces must be ready? How much capacity is available, and what ends to make room?

The first question sets the frame: without a target value there is no priority. The second leads to a portfolio with few concurrent initiatives. One to three initiatives per function is a realistic value in delivery practice. The third and fourth decide whether the introduction succeeds. The fifth question is the most uncomfortable, because it demands sacrifice. A strategy without sacrifice is a wish list.

**Checklist for the strategy review**
- Is every running pilot assigned to one of the four decisions?
- Does every continuing initiative have an owner in the business and a metric with a baseline?
- Are the annual costs per application known, including operation and maintenance?
- Is the number of concurrent initiatives per function limited?
- Is there a date each quarter on which management reviews the results?
- Is it settled which activities end for the new priority?

## What happens after the review?

After the review comes a rhythm of quarterly decisions. Each quarter, management rates the running initiatives by their metrics, takes in new applications and ends whatever misses its targets. The rhythm prevents two errors that often follow a hype: clinging to pilots out of habit and restarting in the next wave whenever a new model appears. New models and features, such as those several vendors released in spring and summer 2024, belong in the portfolio when they solve an existing bottleneck. Their release alone justifies no new initiative.

For an overview of the market, the article on the [first half of 2024 in AI](/en/blog/ai-first-half-2024/) shows which announcements reached everyday use. How stock market movements relate to the decision in your own company is covered in the article [AI bubble or turning point](/en/blog/ai-bubble-or-turning-point/).

**Three stances after the hype**
Criterion | Carry on as before | Pause until things are clear | Strategy with quarterly decisions
Pilots | Continue without a decision | Are frozen | Each is assigned one of four decisions
Budget | Spread across many initiatives | Withheld | Concentrated on a few initiatives with a metric
Learning | Random | Little | Fixed in a quarterly rhythm
Risk | Costs without effect | Falling behind in operations | A verifiable order of priority

## AI strategy with Niehues Consulting

[Niehues Consulting](/en/) is a consultancy for AI management and organisation design from a delivery perspective, based in Düsseldorf. Founder Marin Niehues brings AI into operations, in projects ranging from VC-funded scale-ups to DAX 40 corporations and in more than 20 countries on six continents. In the service area of AI management, the work ranges from business analysis through the business case to the selection and sequencing of applications and the steering of the portfolio.

The approach follows the Niehues Delivery Model with its three phases. Three of its checking questions matter for reviewing a strategy after the hype: Which business outcome should change measurably? What is the baseline, and how do we measure success? Which metric shows after 90 days that the solution is landing? The entry point is a free initial consultation of 30 minutes, which you can book via the [contact form](/en/#kontakt).

**Conclusion**
What remains of the first AI wave are applications with a clearly bounded task, a named owner and a measured result. Management regains control by assigning each pilot a decision, limiting the number of initiatives and reviewing each quarter. The pressure of the hype year thus becomes a portfolio that can be justified to shareholders and employees.

**Sources**
- Brynjolfsson, Li, Raymond: Generative AI at Work, NBER Working Paper, 2023
- Noy, Zhang: Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence, Science, 2023
- Dell'Acqua et al.: Navigating the Jagged Technological Frontier, Harvard Business School Working Paper, 2023
- Eurostat: 8 % of EU enterprises used AI technologies in 2023, Eurostat News, 2024
- Sculley et al.: Hidden Technical Debt in Machine Learning Systems, NeurIPS conference paper, 2015

## Frequently asked questions

### Is the AI hype over?

Expectations of general AI assistants have become more realistic, and usage keeps growing. Productivity gains are evidenced in clearly bounded tasks, such as customer support and writing. What matters for a company is whether its own initiatives meet these conditions.

### What does AI demonstrably deliver in business?

AI demonstrably delivers time savings and better quality in tasks involving text and knowledge, provided results can be checked. A field study in customer support showed 14 per cent more issues resolved per hour on average. Effects are larger for less experienced staff.

### How many companies use AI?

According to Eurostat, 8 per cent of enterprises in the European Union with ten or more employees used AI technologies in 2023. The figure differs by country, sector and company size. Large companies use AI considerably more often than small ones.

### Should we end our AI pilots?

End the pilots that have no measurable effect, no owner and no regular use. Scale up the pilots with an effect and an owner. The decision table in this article assigns each pilot one of four answers.

### How many AI initiatives should a company pursue at once?

One to three initiatives per function is a good value to start with. Each initiative ties up attention in the business function, and several parallel rollouts delay each other. More follow once the first runs in regular operation.

### What belongs in an AI strategy?

An AI strategy contains target values for business outcomes, the order of applications, owners in operations, requirements for data and interfaces, and capacity. It also states which initiatives end to make room. A quarterly rhythm keeps it current.

### Do we have to assess every new AI model?

No, a new model deserves assessment when it solves a known bottleneck of a running initiative. A model change then affects one component, not the strategy. The more clearly task and data are defined, the easier a model is to swap.

### How do you calculate the benefit of an AI application?

The benefit is the working time saved multiplied by the hourly rate, minus the cost of licences, operation and maintenance. It becomes real when the time gained is used differently. Check the assumptions together with the business function.