# AI Usage Without Results: Why Employees Use Tools and the Benefit Fails to Appear

Author: Marin Niehues, Niehues Consulting · Published: 2024-07-18 · AI Strategy & Business Case
Source: https://www.niehuesconsulting.de/en/blog/ai-usage-without-results/

AI usage stays without results when employees use a tool but neither the task nor the process nor the measurement has been set up for it. Individuals then save minutes, and nothing changes in the result of the department. The causes lie in four gaps in the organisation: the time saved is not put to use, the tasks do not suit the tool, the process stays as it was, and nobody measures the effect.

This article sets out what the available studies show about the effect of generative AI at work. It explains why usage and benefit diverge, and how managers can find out where their organisation stands with a maturity model and a checklist. The pressure to show visible progress on AI is real, and that is exactly why a close look at what usage achieves is worthwhile.

**Usage is an input, impact is an output. Between the two lie the choice of tasks, the adaptation of the process and measurement, and where one of them is missing, the benefit fails to appear even if more employees open the tool every week.**

## What do studies show about the effect of AI at work?

Studies show that generative AI helps measurably on suitable tasks and does harm on unsuitable ones. A field study with more than 5,000 customer service agents found that access to an AI assistant raised the number of issues resolved per hour by 14 per cent on average. For new and less experienced staff the gain was greatest, while for experienced agents the effect was small. An experiment with several hundred college-educated professionals on writing tasks cut the time required by around 40 per cent and raised the rated quality by 18 per cent.

The most instructive result comes from an experiment with 758 consultants at a management consultancy. On tasks inside the "frontier" of AI capability, participants using AI completed 12.2 per cent more tasks, 25.1 per cent faster and at markedly higher quality. On a task outside that boundary that looked similar at first glance, participants using AI were 19 percentage points less likely to be correct than the control group. The authors speak of a "jagged frontier": where it runs cannot be seen from the outside.

**Findings from studies on AI at work**
14 % | more issues resolved per hour in customer service, strongest for newcomers
40 % | less time on writing tasks, with 18 % higher quality
25.1 % | faster completion of tasks inside AI capability
19 points | lower accuracy on a task outside AI capability

Two things follow for decision makers. First, the benefit arises at the level of the individual task. Second, someone has to select the tasks where the tool helps and guide employees on where it harms. Employees rarely take on either by themselves.

## Why does the benefit fail to appear although employees use AI?

The benefit fails to appear because usage satisfies only the first of four conditions. Four gaps explain the difference, and each has its own remedy.

The first gap is time saved that is never put to use. If a person saves two hours a week, the department sees an effect only when someone schedules that time for something measurable, such as more customer contacts or shorter lead times. Without that decision, the time gained spreads across other tasks and nothing shows in the accounts.

The second gap is wrongly chosen tasks. Employees use AI where they are curious, and the greatest lever often lies elsewhere. Sometimes they use the tool outside its strengths, as the jagged-frontier experiment shows, and have to check or correct the results at length. Checking then costs more time than creating saved.

The third gap is a missing link to the process. Individual use sits beside the workflow: a draft appears faster, but the approval route stays the same, and the bottleneck remains in coordination. Impact arises when the process is tailored to the tool, with adapted steps and roles as well as handovers. The effect of uncontrolled tools and subscriptions is described in the article on [AI subscription sprawl](/en/blog/ai-subscription-sprawl/).

The fourth gap is missing measurement. Usage figures such as log-ins or queries say something about spread and nothing about results. Without a baseline and an indicator, neither a success can be proved nor a failure recognised, and decisions on further funds follow impressions. The patterns that arose in budgets during the first hype phase are shown in the article on [AI investments without a plan](/en/blog/ai-investment-without-a-plan/).

## How much of the time saved turns into results?

Time saved turns into results when it exceeds the effort of checking and rework and finds a new use in the planning. The calculation has four steps. First, the time saved per person and week is measured, for example through sampling. Then the time for checking and correction is deducted. What remains is the net gain. Finally the business unit decides which share of it flows into which activity and values that activity with an indicator.

A worked example with assumed values shows the difference. In one department 100 people use an AI tool. By their own account each saves two hours a week. A sample shows that checking and correction take half an hour a week, so the net gain is 1.5 hours per person and week, 150 hours in total. If 60 per cent of that goes into additional customer appointments, 90 hours of customer time arise each week, about 2.3 full-time positions. Without a decision on its use, the 150 hours remain invisible as an effect, and the self-reported two hours overstate the effect by a third.

## Which maturity levels exist for AI usage?

Four maturity levels describe how far an organisation has come from usage to impact. At level one, individual employees use tools at their own discretion and the effect is unknown. At level two there is a common selection of tools and first rules, but no list of tasks and no indicator. At level three, tasks with high leverage are named and processes adapted, and baselines have been collected. At level four the business unit measures the effect continuously, uses the time gained deliberately and reviews each quarter which applications stay.

**Maturity levels of AI usage**
Criterion | Levels 1 and 2: Use at discretion | Levels 3 and 4: Managed use | With Niehues Consulting
Choice of tasks | By curiosity and chance | List of tasks with high leverage | Prioritisation by value and feasibility
Process | Unchanged | Adapted to the tool | Roles and handovers as well as approvals in the workflow
Measurement | Usage figures | Baseline and indicator per task, continuous from level 4 | Indicator after 90 days with an owner
Time gained | Spreads out | Use fixed in the plan | Net gain after checking effort
Responsibility | None, or only for tools and rules | Owner in the business unit | Operations settled from day one

## How does an organisation get from usage to impact?

An organisation gets from usage to impact in five steps. The first step surveys which tasks employees already do with AI and sorts them by leverage. The second names a baseline and an indicator for two or three tasks per unit. The third adapts the process: who checks, who approves, where does the result go? The fourth trains employees on these tasks, including the question of where the tool fails. The fifth measures the net gain after 90 days and decides how the time gained is used.

**Checklist: Is usage effective in our organisation?**
- Is there a list of tasks for which AI is meant to be used, with reasons?
- Is there a baseline per task that was collected before introduction?
- Has the effort of checking and correction been measured and deducted?
- Has the business unit decided what the time gained is used for?
- Is there an owner who presents the indicator after 90 days?

Readers who want to go further into hype and reality will find a [sober assessment](/en/blog/chatgpt-in-business-sober-review/) of expectations in the article on the year after the launch of ChatGPT.

## AI usage with Niehues Consulting

[Niehues Consulting](/en/) is a consultancy for AI management and organisation design from a delivery perspective, based in Düsseldorf. Founder Marin Niehues brings AI into operations, in projects from VC-funded scale-ups to DAX 40 corporations and in more than 20 countries on six continents. In the service area of AI management this includes business analysis, the business case and the selection and sequencing of applications, and in the service area of rollout it includes enabling users and adapting processes.

The approach follows the Niehues Delivery Model with three phases. For the gap between usage and benefit, three of its test questions matter: What is the baseline, and how do we measure success? Are users enabled and processes adapted? Which indicator shows after 90 days that the solution has landed? The entry point is a free 30-minute initial consultation, which you can arrange through the [contact form](/en/#kontakt). The scope and terms of an engagement are discussed in the free initial consultation.

**Conclusion**
Usage and benefit diverge when time gained goes unused, tasks do not suit the tool, the process stays unchanged and nobody measures. A maturity model with four levels and a checklist show where the organisation stands. Impact arises at the level of the individual task, with a baseline, an indicator and an owner.

**Sources**
- Brynjolfsson, Li, Raymond: Generative AI at Work, NBER Working Paper 31161, 2023
- Noy, Zhang: Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence, Science, 2023
- Dell'Acqua et al.: Navigating the Jagged Technological Frontier, Harvard Business School Working Paper, 2023

## Frequently asked questions

### Why is AI not delivering measurable benefit in my company?

AI delivers no measurable benefit when task, process and measurement have not been set up for the tool. The most frequent causes are time saved without a use, unsuitable tasks, an unchanged workflow and missing indicators. A check of these four points shows where the gap lies.

### How do you measure the benefit of AI in a company?

You measure the benefit with an indicator per task that was collected as a baseline before introduction. What counts is the net gain after deducting checking and correction effort, and the number of users or queries provides no evidence of it. After 90 days a comparison with the baseline shows whether the solution has landed.

### How much productivity does generative AI add?

Studies show gains of 14 per cent in a customer service operation and around 40 per cent time savings on writing tasks for suitable tasks. On tasks outside AI capability, accuracy fell by 19 percentage points. The effect arises at the level of the task, and it has to be measured in your own company.

### Why do employees use AI but the benefit fails to appear?

Employees use AI out of curiosity or time pressure on tasks that do not have the greatest leverage, and the process around them stays the same. The time gained then spreads across other activities. Impact arises when the business unit selects tasks and plans the time gained.

### How much time does AI really save?

After deducting checking and correction, the gain is often much smaller than self-reports suggest. In this article's worked example, two hours saved per week shrink to 1.5 hours net. Measure the time in a sample before you extrapolate.

### What is a maturity model for AI usage?

A maturity model for AI usage describes in four levels the path from individual use via common rules and managed use to continuous measurement of impact. Each level has criteria for choice of tasks, process, measurement and responsibility. It shows which step comes next.

### Who should be responsible for the benefit of AI in a business unit?

The owner of the indicator in the business unit is responsible, usually the head of the unit. IT supplies technology and operations, while the business unit decides on tasks and on the use of time gained. Without this owner, usage remains a collection of individual initiatives.

### How long does it take for AI usage to show an effect?

A first statement on impact is possible after 90 days if baseline and indicator were fixed before the start. The period applies to one bounded task in one unit. Further units follow once the first application runs in regular operation.