ChatGPT in Business: A Sober Review Almost a Year After Launch

ChatGPT delivers measurable benefit in business on clearly defined writing tasks with human review, and it is unsuitable as a source of facts without a cross-check. Studies from 2023 show efficiency gains of 14 to 40 per cent, yet the same studies show that errors outside the model's capability frontier sound convincing. Executives therefore need a grid that sorts tasks into use classes and ties approval to how well the result can be checked.

This article takes stock after almost a year. It sets out the solid evidence, classifies the available offerings and contains a grid of four use classes for approving applications in everyday business. It provides an overview and does not constitute legal advice.

What does ChatGPT demonstrably achieve in business?

Generative AI assistants such as ChatGPT raise productivity on writing tasks by 14 to 40 per cent in studies, and the effect is largest for less experienced staff. A field study by Brynjolfsson, Li and Raymond of more than 5,000 customer support agents at a software company, which examined an assistant built on GPT models, found an average rise of 14 per cent in issues resolved per hour, and 34 per cent for novices. An experiment by Noy and Zhang with 453 professionals found working time cut by around 40 per cent with higher-rated quality. The effect partly evens out performance differences between employees, because the tool makes the approach of strong colleagues available to others.

A third study from September 2023 limits this finding. In a study of 758 consultants at an international consulting firm, participants with access to GPT-4 solved tasks inside the model's capability frontier around 25 per cent faster and at higher rated quality. On a task outside that frontier they produced 19 percentage points fewer correct solutions than the group without AI. The authors speak of a “jagged” frontier: which task lies on which side is hard to see from outside. For a company this means approval should follow the checkability of the result, while the impression a reply makes says nothing about whether it is right.

Where are the limits of ChatGPT in business?

The limits of ChatGPT lie in invented content, in a knowledge base with a cut-off date and in uneven quality across tasks. Research calls invented but plausible-sounding statements hallucinations, and a survey by Ji and colleagues (2023) classifies causes and countermeasures. The model computes likely sequences of words. It does not check whether a statement is true.

Three rules for everyday use follow. First, figures, names and source references from model replies never go unchecked into decision papers or customer letters. Second, every application needs a person who can judge the result on the merits: whoever cannot judge a task cannot check its solution either. Third, confidential company data should be processed only in offerings whose terms of use govern how inputs are used. For ChatGPT Enterprise, which the provider introduced at the end of August 2023, the provider states that inputs are not used to train its models. A further route is access through the Azure OpenAI Service, generally available since January 2023, whose terms companies agree with the provider individually.

Which offerings exist, and what do they cost?

In October 2023 four routes to access are available for everyday business: the free version of ChatGPT, the paid offering ChatGPT Plus, ChatGPT Enterprise and access through programming interfaces such as the Azure OpenAI Service. The provider quotes a price of 20 US dollars per user per month for ChatGPT Plus. No public price list exists yet for ChatGPT Enterprise, and terms are offered individually.

A rough calculation shows when a paid account pays off. At a full cost of 60 euros per working hour, around 20 euros a month corresponds to about 20 minutes of working time. If a person saves more than 20 minutes a month through regular use, the licence pays for itself on paper. The caveat lies in the words “on paper”: saved time has value only if the business unit plans it for other tasks. How quickly licence costs grow without usage is shown in the article on AI FOMO: with 20 per cent active users, the cost per user rises fivefold.

Which four use classes are there?

Four use classes sort ChatGPT applications by how well the result can be checked and by the damage a mistake would cause. The grid answers whether an application is approved, accompanied or put on hold.

Most applications in office and administrative work fall into classes A and B and can be approved with little effort. Class C requires working instructions that say who checks and with what. Class D begins where the reply reaches customers or business processes without human review, and it presupposes a quality assurance that still has to be built for a language model.

How do companies introduce ChatGPT in an orderly way?

Companies introduce ChatGPT in an orderly way by starting with one to three applications from classes A and B, fixing an owner and a metric, and deciding after 90 days. The sequence follows four steps. First, business units record what employees already use the tool for, because use often begins before the company regulates it. Next, the management defines the use classes and states which data belong in which offering. In the third step individual teams start with clearly defined tasks, such as drafting quotations or preparing meeting notes, and measure processing time and quality against the baseline. In the fourth step, leadership decides after 90 days which applications to extend.

Enablement matters as much as the licence. The studies show the largest gains among employees who use the tool with clear tasks and feedback. Staff need examples of good prompts, a rule for checking results and a contact person for questions.

ChatGPT in business with Niehues Consulting

Niehues Consulting is a consultancy for AI management and organisation design from a delivery perspective, based in Düsseldorf. Founder Marin Niehues brings AI into operations, in projects ranging from VC-funded scale-ups to DAX 40 corporations and in more than 20 countries across six continents. In the AI management field the work runs from business analysis and requirements through the business case to the selection and sequencing of applications.

The approach follows the Niehues Delivery Model. Three checks matter most for ChatGPT: Which business result should change measurably? Does the prototype prove the benefit in the real workflow with real data? Are users enabled and processes adapted? For language models the third question decides whether licences turn into use. Scope and terms of an engagement are discussed in the free initial consultation, which you can arrange via the contact form.