Gemini vs GPT-4: Why the Model Race Is Secondary for Companies

Gemini was announced this week, but what decides the benefit of AI in a company is mainly the workflow in which the model operates. The quality gap between leading models changes within months, while the cost of introduction and of changing the way people work remains. A company that selects a model against six criteria, tests it on its own tasks and keeps switching open makes the decision cleanly once and does not have to reopen it with every announcement.

On 6 December Google presented its Gemini model in three sizes for different places of use: Ultra for demanding tasks, Pro for broad applications and Nano for devices such as smartphones. In its announcement Google cites better scores than GPT-4 for Ultra in common benchmark tests and stresses that the model processes text, images, audio, video as well as code together from the outset. The largest version, Ultra, is to follow next year, and Pro is already part of the Bard chat assistant. For executives this raises the question whether they must reconsider a decision for GPT-4 or another provider. This article answers it with criteria and with a test you can run yourself.

Is Gemini better than GPT-4?

Whether Gemini is better than GPT-4 is evidenced at the time of the announcement only by the benchmark tests Google has published, and for your tasks there is no result yet. The figures come from the provider itself, the largest version is not available until next year, and a benchmark measures general abilities, such as knowledge questions or arithmetic, that have little in common with preparing a quotation or handling a complaint. A comparison becomes meaningful only when both models have to solve the same tasks from your business.

The speed of the field adds to this. GPT-4 appeared in March this year, in November OpenAI announced a faster variant with a much larger context window, and other providers are following with models of their own. Each of these announcements shifts the ranking for a few months. A decision that hangs on today's top position is out of date after a short time. How to place such announcements is described in the article on the AI hype cycle.

Which criteria matter when choosing a model?

Six criteria matter when choosing a model, and the ranking in benchmark tests covers only one of them. The table summarises them and shows which question to ask for each.

Measured by outcome, the first criterion carries the greatest weight, but the other five decide whether the application stays in operation. A model of outstanding quality that cannot be integrated into the existing platform, or whose running cost exceeds the savings potential, fails on criteria that appear in no announcement.

How do you test models on your own tasks?

You test models by collecting 30 to 50 real tasks from one workflow, letting each model solve the same tasks and having the business unit rate the results blind. Blind means the raters do not know which model produced which answer. Depending on the task, the test takes a few days and yields three figures: the share of answers usable without correction, the share with minor correction and the share that is discarded.

The test delivers more than a model comparison. It shows how much rework remains in the workflow, and so the figure the business case needs. An example with illustrative values: of 40 tasks, model A delivers 24 usable answers, 10 with minor correction as well as 6 unusable. Model B scores 26, 9 as well as 5. The difference of two tasks is of the order that a different wording of the instruction can already produce. More interesting is that with both models roughly one task in four needs rework, and that implementation must turn this rework into a review step in the workflow.

What does AI actually deliver in operations?

The benefit arises in the workflow, and it is documented when a tool is embedded in a concrete process. A field study of more than 5,000 customer service agents shows that an AI assistant built on a language model, which suggests replies, raised the number of issues resolved per hour by 14 per cent on average. For novices and less experienced staff the gain was 34 per cent. The study examined an assistant built into the agents' working interface, and it did not examine a freely available chat window.

The lesson for decision-makers: the effect came from the interplay of model, workflow as well as the people who work with it. The model was one component among several. A company that debates models for months but defines no workflow in which the AI takes over a task gains no benefit, whatever the ranking looks like. How a year of experience with ChatGPT confirms this is shown in the article on ChatGPT in the enterprise, and the additional functions arriving in office applications are described in the article on Microsoft 365 Copilot.

How should a company deal with the model race?

A company should watch the model race but build so that switching stays possible. In concrete terms: applications treat the model as a replaceable component, the instructions and test tasks belong to the company, and after an announcement the business unit can check within days whether another model performs better. A decision for one provider is then a decision for a period, confirmed or changed at each review of the portfolio.

At the same time, management should resist the pressure to answer every announcement with a project. How such pressure can be turned into a plan is described in the article on shareholders' questions to management. The eight questions listed there apply regardless of the model.

Model choice and implementation with Niehues Consulting

Niehues Consulting is a consultancy for AI management and organisation design from a delivery perspective, based in Düsseldorf. Founder Marin Niehues brings AI into operations, on projects ranging from VC-funded scale-ups to DAX 40 groups and in more than 20 countries on six continents. In the service line AI prototypes, operationalisation and implementation, Niehues Consulting builds applications so that the model remains a replaceable component and the benefit arises in the workflow.

The approach follows the Niehues Delivery Model with three phases. Three test questions count for the choice of model: does the prototype prove the benefit in the real workflow with real data? Who is accountable for operation from day one? Which metric shows after 90 days that the solution is landing? The answers are independent of which model is finally chosen. The entry point is a free initial consultation of 30 minutes, which you can arrange via the contact form.