AI Strategy After the Hype: What Remains of the First Wave

After the first year of hype, what remains of the AI wave are applications that measurably gain time or quality in a clearly bounded work step. A strategy built on this rates every pilot by its result in operations and gives budget only to initiatives with an owner and a metric. The experience of your own teams provides the basis.

This article is written for managing directors and heads of function who have built pilots, licences and working groups since autumn 2022 and now decide what continues. It summarises what is evidenced and offers a decision procedure for the portfolio of running initiatives.

What remains of the AI wave?

What remains are applications that process text, data and knowledge in everyday work steps and demonstrably save time. Three academic studies from 2023 point the same way. A field study of more than 5,000 customer support agents found, on average, 14 per cent more issues resolved per hour, and 34 per cent for novices. An experiment with 453 professionals in writing occupations, published in the journal Science, found about 40 per cent shorter working time on writing tasks with higher rated quality. A field experiment with 758 consultants, led by Harvard Business School researchers, showed the limit at the same time: for tasks within the capability of the AI, quality rose clearly, while for one task outside that frontier participants with AI help were more often wrong than those without it.

A clear rule for strategy follows. AI pays off where the work step is bounded, the results can be checked and the users can judge the answer on its merits. It does not pay off where nobody can judge the result. The studies report no effect at company level, because a faster work step only becomes a better result when the process around it is adjusted.

A second finding concerns adoption. According to Eurostat, 8 per cent of enterprises in the European Union with ten or more employees used AI technologies in 2023. Broad everyday use therefore takes place in a minority of firms. Anyone who is only now assessing their pilots is at the same point as the majority.

What was hype and what was insight?

The hype was the expectation that general AI assistants would transform entire functions without adaptation. The insight was that the benefit is set by the task and the workflow. Broad licences for all employees achieve little as long as nobody has settled how to use the tool for their own task, and usage flattens after the first weeks. The article AI usage without results describes how to recognise this.

One insight concerns the technology. The effort rarely lies in the model. As early as 2015, researchers at Google showed in the paper “Hidden Technical Debt in Machine Learning Systems” that model code makes up only a small part of a production system, while data management, configuration, monitoring and interfaces make up the larger part. This holds for generative AI as well: the connection to documents, permissions and processes decides the benefit, not the choice of model. With several capable vendors on the market, the model also becomes more interchangeable, while your own data base and your own workflow do not.

A second insight concerns the organisation. An application without an owner in the business function is not maintained and is no longer used after six months. Introduction is therefore an initiative of the line, even when IT runs it technically. How budgets seeped away in the first year and how to spot it early is shown in the article AI investments without a plan.

How do you assess running AI pilots after the hype?

You assess running pilots with a decision table that assigns each initiative one of four answers: scale up, rebuild, observe or end. The assignment follows four questions about each pilot. Has a metric changed, and has the effect been measured? Does the application have an owner in the business function? Do the intended users still use it regularly after 90 days? Is the cost per use known?

The table forces a decision on every pilot, including those nobody wants to switch off. Ending belongs to strategy: every pilot that ends frees attention and budget for the initiatives that work. A group of pilots that trigger no decision has caused costs after twelve months and delivered no insight.

A value comparison helps to rank the remaining initiatives. Set the annual relief against the annual cost of the application. An example with illustrative figures: a sales team of 20 people saves two hours per person per week through a proposal assistant. Over 44 working weeks that gives 20 times 2 times 44, or 1,760 hours a year. At an internal hourly rate of EUR 60 this equals EUR 105,600. If licences, operation and maintenance cost EUR 40,000 a year, a calculated benefit of EUR 65,600 remains. It becomes real benefit when the time gained is used for additional customer meetings.

Which questions must an AI strategy answer now?

An AI strategy after the hype answers five questions. Which business outcomes should AI change measurably in the next twelve months? Which applications contribute, and in what order? Who is responsible for each application in operations? Which data and interfaces must be ready? How much capacity is available, and what ends to make room?

The first question sets the frame: without a target value there is no priority. The second leads to a portfolio with few concurrent initiatives. One to three initiatives per function is a realistic value in delivery practice. The third and fourth decide whether the introduction succeeds. The fifth question is the most uncomfortable, because it demands sacrifice. A strategy without sacrifice is a wish list.

What happens after the review?

After the review comes a rhythm of quarterly decisions. Each quarter, management rates the running initiatives by their metrics, takes in new applications and ends whatever misses its targets. The rhythm prevents two errors that often follow a hype: clinging to pilots out of habit and restarting in the next wave whenever a new model appears. New models and features, such as those several vendors released in spring and summer 2024, belong in the portfolio when they solve an existing bottleneck. Their release alone justifies no new initiative.

For an overview of the market, the article on the first half of 2024 in AI shows which announcements reached everyday use. How stock market movements relate to the decision in your own company is covered in the article AI bubble or turning point.

AI strategy with Niehues Consulting

Niehues Consulting is a consultancy for AI management and organisation design from a delivery perspective, based in Düsseldorf. Founder Marin Niehues brings AI into operations, in projects ranging from VC-funded scale-ups to DAX 40 corporations and in more than 20 countries on six continents. In the service area of AI management, the work ranges from business analysis through the business case to the selection and sequencing of applications and the steering of the portfolio.

The approach follows the Niehues Delivery Model with its three phases. Three of its checking questions matter for reviewing a strategy after the hype: Which business outcome should change measurably? What is the baseline, and how do we measure success? Which metric shows after 90 days that the solution is landing? The entry point is a free initial consultation of 30 minutes, which you can book via the contact form.