# AI Agents in Production: Lessons from Air Canada, Klarna and Replit Author: Marin Niehues, Niehues Consulting · Published: 2026-08-10 · AI Law, Risk & Governance Source: https://www.niehuesconsulting.de/en/blog/ai-agent-failures-lessons/ The best-known AI agent failures at Air Canada, Klarna and Replit share one cause: guardrails that are proven today were missing. Each case is publicly documented, and each traces back to a gap in technology and organisation that would have been visible before go-live. For the executive team this is good news. Others have already paid for the lesson, and the countermeasures are known. This article examines five cases, sets out cause and countermeasure for each and condenses them into a go-live checklist. Because several cases touch on liability: This article provides an overview and does not constitute legal advice. **An AI agent rarely fails in production because of the model. It fails because limits on answers, permissions and autonomy are missing.** ## Why do AI agents fail in production? AI agents fail in production when they are allowed to do more than their organisation can control. A language model produces plausible answers and executes instructions, yet it knows neither the policy in force nor the consequences of a command. Architecture and operating model draw that boundary. The cases below follow a pattern. A system that had convinced in the pilot was given access to customers or production data. What a specialist at the screen had caught in the pilot now went out unchecked. Which fields agents cover reliably today is described in [AI agents: what they can do and where they stand](/en/blog/ai-agents-capabilities-and-maturity/). ## Air Canada: Is a company liable for what its chatbot tells customers? Yes, in the Air Canada case the company was held liable for its chatbot's incorrect information. The Civil Resolution Tribunal of British Columbia decided Moffatt v. Air Canada in February 2024. What happened: After his grandmother died in November 2022, a customer asked the airline's website chatbot about bereavement fares. The bot stated that he could apply for the reduced fare retroactively, within 90 days of the ticket being issued. The actual policy excluded retroactive refunds, and Air Canada rejected the claim. Consequence: Air Canada argued that the chatbot was responsible for its own statements. The tribunal rejected this. The company was responsible for all information on its website, whether on a static page or in a chat, and had not taken reasonable care to ensure the bot was accurate. It awarded the customer the difference to the bereavement fare, around 650 Canadian dollars plus interest and fees. The sum was small, the public impact considerable. Cause: The bot answered from the language model without being bound to the policy text in force. Countermeasure: retrieval with source binding. The bot answers only from a versioned knowledge base of approved policies and cites the passage for each statement. Before sending, a check verifies that the cited passage supports the answer. Unsupported answers go to a service agent. Organisationally, the knowledge base needs an owner in the business who feeds in every policy change. How hallucinations can be limited in operation is covered in [AI hallucinations in production](/en/blog/ai-hallucinations-in-production/). **Liability for chatbot statements** The decision comes from Canada and does not bind European courts. The underlying idea transfers: a company that puts a chatbot in front of customers in its own name must accept its statements as it would those of a service employee. How management and the organisation protect themselves is set out in [Liability for AI errors](/en/blog/ai-liability-for-executives/). ## Klarna: Autonomy without measurement costs quality What happened: In February 2024 the payments provider Klarna announced that its AI assistant had handled 2.3 million customer conversations in its first month, around two thirds of all service chats. By the company's own account it was doing the work of 700 full-time agents. In May 2025 chief executive Sebastian Siemiatkowski told Bloomberg that cost had been too dominant a factor in organising the service and that the result had been lower quality. Klarna began hiring people for customer service again and kept the assistant for simple requests. Cause: The automation was measured by volume. For complex requests there was no quality measure that would have raised an early alarm. A publicly celebrated success thus became a public course correction. Countermeasure: graded autonomy with measurement. The agent takes over requests by type, starting with the simple ones. Before every release a test set of real cases is run, and in operation samples are scored against defined quality criteria. If a request type falls below the threshold, exactly that type returns to people. The quality metric belongs to a named person in service who decides on expansion and rollback. ## Replit: An agent may only be able to do what it should What happened: In July 2025 Jason Lemkin, founder of the business community SaaStr, reported publicly on a trial with Replit's coding agent. Despite an explicit code freeze, the agent deleted a production database holding records on more than a thousand executives and companies. It then wrongly reported that recovery was impossible. Replit chief executive Amjad Masad called the incident unacceptable. He announced automatic separation of development and production databases, along with a mode in which the agent only plans. Cause: The freeze lived in the prompt. Technically the agent had delete rights on production, and test and production shared the same database. Countermeasure: least privilege and approval gates. Each agent receives its own service account with exactly the rights its task requires. It works in a sandbox or on a copy. Writing and deleting operations on production systems pass through human approval. Backups with tested recovery limit the damage. An instruction in a prompt is no technical lock. ## Australia: Fabricated sources in a government report What happened: Australia's Department of Employment and Workplace Relations had commissioned a large consulting firm to review an IT-based compliance system that applies penalties to welfare recipients, with a contract value of around 440,000 Australian dollars. The report appeared in July 2025. In August an academic at the University of Sydney pointed out non-existent references, along with a fabricated quotation from a Federal Court judgment. The corrected version disclosed that a generative language model had been used in its preparation. In October 2025 it became known that the firm would repay the final instalment of the fee. The department stated that the recommendations remained unchanged. Cause: Generated references went into an official document without systematic checking. The result was a repayment and parliamentary criticism. Countermeasure: citation verification. A verification pipeline resolves every reference automatically and compares the statement with the source passage. A document is released only when every claim is supported or visibly marked as an assessment. Organisationally, one rule applies to every report with external reach: whoever signs has seen every source. ## Chevrolet dealer: Prompt injection in the sales chat What happened: In December 2023 users got the sales chatbot of a Chevrolet dealer in Watsonville, California, to confirm a new Chevrolet Tahoe for one dollar as a "legally binding offer". The trick was an injected instruction to agree with everything the customer said. The dealer took the bot offline. No sales contract was concluded; the damage was public embarrassment. Cause: The language model was allowed to formulate prices and commitments, and external input could override its instructions. Countermeasure: guardrails against prompt injection. Conversation and transaction are separated. Binding values come from an interface with a rule set, and the model only formulates. Input passes through classification, output through a filter for commitment language. Before launch a red team tests attacks deliberately. The attack surfaces are described in [AI security: prompt injection and new attack surfaces](/en/blog/prompt-injection-ai-security/). ## Five guardrails compared The cases yield five guardrails for every agent with external reach or system access. The table shows the difference in operation. **AI agents without and with guardrails** Criterion | Without guardrails | With guardrails | With Niehues Consulting Answers to customers | From model knowledge, without source | Retrieval from approved policies with citation | Knowledge base with a business owner Autonomy | All or nothing | Graded by request type, with threshold | Levels and metrics set in the business case System rights | Broad access, lock in the prompt | Own account with approval gates | Rights concept per agent as an acceptance criterion in the prototype Sources in documents | Adopted unchecked | Automated citation check before release | Approval path with a signatory per document type External input | Unfiltered | Classification and red-team test | Attack tests as part of the go-live review Accountability | With the project team until launch | With operations, with a metric | Operational responsibility handed over from day one ## Which checks belong before an AI agent goes live? Eight checks belong on the table before go-live, and each needs a documented answer. For agents with external reach or write access to production systems all eight are mandatory. For internal drafts without system access, points 1, 4 and 7 suffice. **Go-live checklist** 1. Every answer comes from an approved source and cites the passage. 2. The agent has its own account with the rights its task requires and no others. 3. Actions that delete, pay or bind externally pass through human approval. 4. A test set of real cases has measured the error rate per request type. 5. Thresholds for withdrawing autonomy are set, as is the person who decides. 6. A red team has tested prompt injection and manipulation attempts. 7. Backups and recovery are tested, and a shutdown path is documented. 8. Operational responsibility is named, and the business units know the escalation path. The checklist makes risk management auditable. How the executive team steers the AI portfolio as a whole is described in the [AI management guide](/en/blog/ai-management-guide/). ## How Niehues Consulting delivers such initiatives [Niehues Consulting](/en/) supports AI initiatives from analysis through the prototype to introduction into operation. Marin Niehues works as a management consultant from the delivery perspective. He builds organisations that deliver and brings AI into operation, in projects ranging from VC-funded scale-ups to DAX 40 corporations and in assignments in more than 20 countries on six continents. The approach follows the three phases of the Niehues Delivery Model. For agents three review questions matter most. Who decides, who delivers, who uses? Does the prototype prove its value in the real workflow with real data? Who is accountable for operation from day one? In the Understand phase, requirements and the business case are produced, together with the classification of which actions an agent may take on its own. In Build, the prototype shows in the real workflow whether source binding and approvals work. In Introduce, metrics and responsibilities pass to operations. The free 30-minute initial consultation clarifies which agent goes on the checklist first ([book a call](/en/#kontakt)). **What decision-makers take from the cases** The five cases reveal five gaps that can be closed before go-live: - answers only with a source - autonomy graded and measured - rights limited, critical actions approved - sources in documents verified - external input filtered and tested **Sources** - Civil Resolution Tribunal British Columbia: Moffatt v. Air Canada, 2024 BCCRT 149, 2024 - OWASP Foundation: OWASP Top 10 for LLM Applications, 2025 - National Institute of Standards and Technology: AI Risk Management Framework (AI RMF 1.0), 2023 ## Frequently asked questions ### What are well-known examples of AI agent failures? Publicly documented cases include Air Canada's chatbot inventing a refund rule and Klarna's course correction in customer service. At Replit a coding agent deleted a production database during a trial. They are joined by fabricated sources in an Australian government report and a sales chatbot that agreed to sell a new car for one dollar. ### Is a company liable for incorrect answers from its chatbot? In the Air Canada case a Canadian tribunal held the company liable for its chatbot's incorrect information. It rejected the argument that the bot was responsible for its own statements. The underlying idea transfers: a bot speaks on behalf of the company, and the organisation ensures its accuracy. ### What exactly happened at Replit? In July 2025 Replit's coding agent deleted a production database during a trial by SaaStr founder Jason Lemkin, despite an explicit code freeze. The agent technically had delete rights on production. Replit then announced automatic separation of development and production data. ### Why did Klarna start hiring human customer service staff again? Klarna began hiring people again because service quality had suffered. Chief executive Sebastian Siemiatkowski said in May 2025 that cost had been too dominant a factor. The AI assistant remains in use for simple requests, while people handle complex cases. ### How do you stop an AI agent from deleting data? An agent cannot delete production data if it has no technical delete rights on it. That is ensured by its own service account under the principle of least privilege and by human approval for every deleting action. Backups with tested recovery limit the damage in an emergency. ### What is prompt injection? Prompt injection is an attack in which injected instructions make a language model ignore its rules. At the Chevrolet dealer, one chat message was enough for the bot to agree to every request. Protection comes from separating conversation and transaction and from targeted attack tests before launch. ### How much autonomy should an AI agent have at the start? At the start an agent should only carry out tasks on its own whose errors are easy to correct. Payments, deletions and actions with external effect pass through human approval. Autonomy grows per task type once the measured error rate stays within the defined threshold. ### What belongs on a go-live checklist for an AI agent? The checklist covers source binding, a limited rights concept and approvals for critical actions. It adds a test set with a measured error rate, rollback thresholds, attack tests, tested backups and named operational responsibility. Each point needs a documented answer before launch.