# GPT-4o in Customer Service: What Talking AI Changes for Customer Contact

Author: Marin Niehues, Niehues Consulting · Published: 2024-05-16 · AI Strategy & Business Case
Source: https://www.niehuesconsulting.de/en/blog/gpt-4o-voice-ai-customer-service/

GPT-4o makes voice dialogue possible at conversational speed, and so moves the question in customer service from technology to the choice of enquiries an AI may handle. Latency was one of the main obstacles to natural phone calls with machines. With it gone, one excuse falls away, and the task remains of defining enquiries and quality measures as well as the handover to people with care.

This article puts the announcement of 13 May 2024 into context for customer service leaders, rates enquiry types by suitability and describes an approach for a pilot that survives contact with operations. It also touches on the recording of calls, which needs legal review. This article provides an overview and does not constitute legal advice.

**Voice AI pays off in customer service where enquiries are frequent and unambiguous, and where clean data can solve them. For everything else, the handover to a human decides the quality, and that handover has to be built before the first call.**

## What is GPT-4o and what can it do in a conversation?

GPT-4o is an AI model from OpenAI that processes text and images as well as audio in a single model and understands and produces speech directly. The "o" stands for "omni". Until now, ChatGPT's voice mode consisted of three steps in a chain: one model converted speech to text, a second wrote the answer and a third read it aloud. According to the vendor, average response times were 2.8 seconds with GPT-3.5 and 5.4 seconds with GPT-4. Intonation and pauses were lost along the way, as was background sound. For GPT-4o the vendor quotes response times from 232 milliseconds, with an average of 320 milliseconds.

The figure gains weight when set against human conversation. A widely cited study across ten languages found that the pause between two speakers' turns averages around 200 milliseconds. A system that answers in that range no longer feels like a machine with a thinking pause on the phone. It can be interrupted, it reacts to tone of voice and it replies in the caller's language.

At the time of the announcement, text functions and image functions are available, the new voice functions are due to roll out gradually over the coming weeks, and the audio interface for developers is open at first to a small group of partners. Anyone planning voice contact with customers can therefore describe the scenario and prepare the assessment now, but cannot yet set up a robust operational test with their own telephony. The time until release is enough for the work that decides the outcome anyway.

## What changes in customer service in practice?

What changes in customer service is the obstacle on which voice automation has failed so far: the feeling customers have of fighting a machine. Classic voice menus with keypad selection and rigid speech-recognition dialogues resolve many enquiries, but they force callers down predefined paths. A language model understands sentences such as "My delivery from Tuesday hasn't arrived, and I'm away from tomorrow", can query the status and offer alternatives, and then summarises the conversation.

Value arises from two sources. The first is relief for the team through completed standard contacts. The second is quality in the contacts that stay with people. Here lies the most robust research material: a field study of more than 5,000 customer support agents in a chat setting found that an AI assistant offering suggested replies raised the number of issues resolved per hour by 14 per cent on average, and by 34 per cent for novices. The study examined assistance for employees in text. It nevertheless shows where part of the leverage lies: knowledge that used to sit with experienced agents becomes available to everyone on the team.

The two uses differ in risk. An assistant for employees always has a person in front of the customer who catches errors. A voice agent speaking directly to customers delivers its answer with no corrective in between. The order of introduction follows from this: assistance in the background first, then self-service for the clearest enquiries, and more complex conversations last.

## Which customer enquiries suit voice AI?

Enquiries suit voice AI when they come in high volume, have an unambiguous solution and draw on reliable data in the background. The table below rates typical enquiries by suitability for direct customer contact and names the precondition for starting.

**Suitability of typical enquiries for voice AI**
Criterion | Standard information and status | Appointments and changes | Complaints and escalations | Advice requiring depth | With Niehues Consulting
Volume and repetition | High | High | Medium | Low | Analysis of contact reasons from recent months
Solvable with system data | Yes, by lookup | Yes, via calendar or order system | In part, decision rests with a person | Rarely | Check of the integration before selection
Consequence of error | Low | Medium, double booking possible | High, customer retention affected | High | Threshold per enquiry type with sign-off by the owners
Recommendation | Automate directly | Automate with confirmation by message | Detect the case, then hand over to a person with a summary | Support employees with assistance | Sequence with a metric per enquiry type

The table shows a pattern: the higher the consequences of an error and the more emotional the caller, the earlier a person must take over. A good voice AI recognises that situation from content and tone and passes the conversation on with a summary, so that the customer does not have to repeat anything.

## How do you test whether a pilot has a chance of reaching operations?

A pilot has a chance of reaching operations when six questions are answered with yes. They capture the points at which voice projects fail in daily work even though the demonstration was convincing.

1. Are the ten most frequent contact reasons available with their share of total volume, and are at least two of them unambiguously solvable?
2. For each of these enquiries, is it defined which systems supply the answer and who is responsible for their data quality?
3. Is there a baseline for each enquiry type, such as handling time, first-contact resolution rate and customer satisfaction?
4. Is the handover to a person described, with triggers and availability as well as a summary?
5. Has the system been tested with real calls, including dialect and background noise as well as interruptions, rather than with sample sentences from the office?
6. Has the head of service named an owner for operations from day one?

The business case follows from these answers. One way to calculate it: the number of calls per month, times the share of the enquiry type, times the rate that is resolved conclusively, gives the number of contacts relieved. Multiply that by the full cost of a contact handled by a person and deduct the running costs of the solution, meaning usage, operation, upkeep of the knowledge base and quality review. With 10,000 calls a month, a 20 per cent share for the enquiry type and a resolution rate of 60 per cent, that is 1,200 contacts relieved. These figures are a worked example for illustration. What matters is that you take the resolution rate from a test with real calls, since an estimated rate makes any business case vulnerable. How such assumptions can be evidenced is covered in the article on [AI success stories](/en/blog/ai-case-studies-check/), and the article on [AI washing](/en/blog/ai-washing-spot-products-claiming-ai/) offers guidance on inflated product claims.

**Quality measures for voice agents**
- Resolution rate without handover, measured per enquiry type
- Share of repeat contacts on the same issue within seven days
- Customer satisfaction compared with contacts handled by a person
- Share of handovers with a complete summary

## Which risks need clearing before the first call?

Three risks need clearing before the first call: wrong information, lack of transparency and uncontrolled costs. Language models phrase answers fluently even when the answer is wrong. For information with financial consequences, such as prices and commitments, the agent should reproduce only content from approved sources and hand over when uncertain. Customers should be told at the start that they are speaking to an AI. For recording and processing of call data, specialists in law and data protection settle the framework before the test. On cost, it is worth looking at the minute of conversation: the vendor quotes an API price for GPT-4o half that of its predecessor GPT-4 Turbo, but voice conversations consume many units, and a calculation per resolved contact says more than a price per unit.

The organisation also demands attention. Where knowledge used to live in people's heads, it must now exist as a maintained knowledge base. Every price change and every new rule then belongs in the maintenance process. Anyone who assigns this task to no role will see outdated answers after three months. Concern about job losses eases when senior management says early which tasks fall away and which grow.

## The approach in three steps

The approach starts with the choice of enquiries, sets up a small pilot with handover and ends with a decision based on the metrics.

1. Selection: from the contact reasons of the last six months, you choose one or two enquiries with high volume, an unambiguous solution and a low consequence of error. The service department sets baselines.
2. Pilot: the enquiry first runs in the background, with the system proposing replies and employees approving them. Then it takes a small share of calls directly, with handover to the team, a record of the reason for every handover and a weekly review.
3. Decision: after the agreed period, the head of the business unit decides on the basis of the metrics whether the share rises, the enquiry type is adjusted or the trial ends. Termination criteria are fixed beforehand.

## Working with Niehues Consulting towards operations

[Niehues Consulting](/en/) is a consultancy for AI management and organisation design from a delivery perspective, based in Düsseldorf. Founder Marin Niehues brings AI into operations, in projects from VC-funded scale-ups to DAX 40 groups and in more than 20 countries on six continents. For voice AI in customer service, the work runs from analysing contact reasons and the business case through prototypes with real calls to introduction in the service operation.

The approach follows the Niehues Delivery Model with its three phases. Three of its test questions matter most here: which business outcome should change measurably? What is the baseline, and how do we measure success? Who is responsible for operations from day one? The scope and terms of an engagement are discussed in the free initial consultation, which you can arrange in 30 minutes via the [contact form](/en/#kontakt).

**Conclusion**
GPT-4o lowers the technical barrier to voice dialogue, but operations are decided by the choice of enquiries, the quality of data and the handover to people. Customer service leaders gain most when they use the time until broad availability for contact analysis and baselines as well as the handover process.

**Sources**
- OpenAI: Hello GPT-4o, product announcement, 13 May 2024 (vendor statements on response times and price)
- Stivers et al.: Universals and cultural variation in turn-taking in conversation, PNAS, 2009
- Brynjolfsson, Li, Raymond: Generative AI at Work, NBER Working Paper, 2023

## Frequently asked questions

### What is GPT-4o?

GPT-4o is an AI model from OpenAI that processes text and images as well as speech in one model. It was presented on 13 May 2024 and, according to the vendor, answers in voice dialogue in 320 milliseconds on average. The voice functions are being released gradually.

### Can GPT-4o handle customer phone calls?

GPT-4o can hold conversations at conversational speed, but the connection to telephony and customer systems still has to be built. At the time of the announcement, the audio interface for developers is not yet generally open. Until then, plan the enquiry types and the data as well as the handover.

### Which customer enquiries can voice AI take over?

Voice AI works best on frequent, unambiguous enquiries backed by clear system data, such as status information and appointment changes. It recognises complaints and escalations and hands them to people with a summary. Suitability rises with volume and falls with the consequence of error.

### Will voice AI replace customer service staff?

Voice AI takes over standard contacts and changes the tasks in the team, but it does not replace the team entirely. The remaining contacts are more demanding, and maintaining the knowledge base and quality needs additional roles. A field study also shows productivity gains for employees who work with AI assistance.

### How much does voice AI for customer service cost?

The costs consist of usage, integration, operation and upkeep, and the figure that matters is the cost per resolved contact. The vendor quotes half the API price for GPT-4o compared with GPT-4 Turbo. The scope and terms of an engagement are discussed in the free initial consultation.

### Do customers have to be told they are talking to an AI?

Customers should be told at the start of the call that they are speaking to an AI, and they should be able to ask for a person at any time. That builds trust and avoids irritation with complex enquiries. Legal and data protection specialists settle the exact wording and the rules on recording.

### When does voice AI pay off in customer service?

Voice AI pays off when an enquiry type has high volume, can be solved unambiguously and shows a convincing resolution rate in a test with real calls. Without an evidenced rate, the business case remains an estimate. Begin with the ten most frequent contact reasons.