Why an AI pilot fails and how to prevent it

The demo worked. Everyone was enthusiastic. And three months later, nothing is running in production. You see this pattern more often than a failed pilot, because a stalled pilot doesn't fail: it simply stays where it was, until nobody asks about it any more.

An AI pilot rarely fails because of the technology. It fails because nobody agreed up front when it counts as a success, who owns it afterwards and how it will run in the real environment. If there is one thing I see in AI projects that stall, it is that someone starts with the technology instead of the work. Below you will find the five causes that keep coming back, the difference between a proof of concept and a pilot, how to keep a pilot small and short, and when to stop.

What does a stalled pilot look like?

Not like an outage. There's no moment when someone decides to stop. There's just a demo that went well, a list of comments still to be dealt with, and a calendar filling up with other work.

The tricky part is that something was delivered. The technology works, the results looked good, and still nothing changes in the way people work. That's more expensive than a pilot that fails straight away, because you've spent the money and not learned the lesson.

What is an AI pilot, and how is it different from a proof of concept?

A proof of concept shows that something is technically possible, usually with test data and in a few days. An AI pilot tests whether it works in your business: with real data, real users and a success criterion you choose in advance. The proof of concept answers "can it be done?", the pilot "does it pay off?".

My view: for most tasks you no longer need an AI proof of concept. Whether a language model can read an invoice or an email was proven long ago. The real question is whether it reads your invoices, with your exceptions, in your process. Only a pilot answers that.

Sometimes a proof of concept does make sense. For example, when your data sits in an old system and nobody knows whether you can get at it, or when the task is genuinely new. Keep it short and small, because it only proves that it can be done. It is similar to an MVP, but without users. You can read more about that difference in what an MVP is.

Where it often goes wrong: a team celebrates a successful proof of concept as if the pilot had already succeeded. In July 2024 Gartner predicted that by the end of 2025 companies would abandon at least 30% of their generative AI projects after the proof of concept. The reasons Gartner gave were poor data quality, inadequate risk controls, escalating costs and unclear business value (Gartner, 29 July 2024). You can solve that last reason before you start.

Reason 1: there was no baseline

Without a figure beforehand, there's no conversation afterwards. Does it work? That depends on what you compare it with. And if nobody knows how much time the old way took, the old way always wins, because it's familiar.

So measure before you start: how many cases a week, how many minutes per case, how often it goes wrong and what fixing it costs. How to set that up is covered in calculating the business case, and the calculator in the cost of manual work takes you through it step by step.

Reason 2: there was no owner after the demo

During a pilot, the owner is usually whoever asked for it. Once it goes into production, someone has to decide on exceptions, on changes and on whether it can stay switched on. That person is rarely appointed.

Appoint them before you start, by name and with time in their calendar. An owner without hours isn't an owner. And make it someone from the team that does the work, not someone from IT who only knows the process from a diagram.

Reason 3: the pilot ran on a copy

A pilot on an exported data set shows whether the idea can work. It doesn't show whether it can work in your environment. That's usually where the real work is: access, permissions, an integration that doesn't exist yet, and a system that only an administrator can open up.

So bring that step forward. What's involved is covered in connecting an AI agent to your existing systems. If the pilot can't reach the real source, that isn't a detail. It's your most important finding.

Reason 4: maintenance was nobody's job

What follows a pilot is maintenance: a model that changes, a supplier that changes its layout, a new field, a token that expires. If nobody has time for that, quality slips without anyone noticing.

So agree who checks, how often and on what. A weekly look at the cases that were rejected is usually enough. What an implementation needs here is covered under AI implementation.

Reason 5: the exceptions only showed up in production

In a pilot you often pick the tidy cases. In production, the rest arrive: the customer who asks three things at once, the document that was half scanned, the order that gets changed afterwards.

So include the messy cases in the pilot on purpose. Run a hundred real cases from the past month through the set-up, the awkward ones included, and count how many get stuck. That number is your real result.

How do you start an AI pilot without risk?

You start without risk by keeping the pilot small and short: one task, one team, four to six weeks. You decide in advance when it counts as a success, test with real data in your own environment and put the date of the decision point in the diary straight away. That way everyone knows from day one where it ends.

My advice, in five steps:

1. Scope. Choose one task with volume and an owner, with a clear start and finish. So not "customer service", but "answering return requests that come in through the web shop". Don't pick the process that sounds most interesting. Which tasks suit this is set out task by task under AI employees.
2. Duration: four to six weeks. Shorter gives too little time to get used to it and to measure fairly. Longer, and the pilot turns into a project without an owner.
3. Success criterion up front. Write down the number you want to see, for example "at least half of the return requests answered correctly without correction". Measure the current situation first, in numbers and in minutes. The calculator in what manual work costs helps with that.
4. Real data in the real environment. Test data tells you little about your own exceptions. So work with the real source, with only the access the pilot needs, and run the outcome alongside the existing work so you can compare. Agree up front what happens to personal data; the rules are in EU AI Act and GDPR: what to sort out. With an AI assistant we built for an organisation in the staffing sector, the model sometimes gave values in the first trial versions that were not in the documents anywhere. Looking back, I would test those trial versions with real documents even earlier.
5. Decision point. At the end, one person decides: go to production, adjust or stop. That person is also there at the start. How to put that on half a page is explained below.

What you often see without a criterion: after six weeks one person thinks the pilot succeeded and another does not, and then whoever talks loudest wins. If the number is fixed in advance, the decision moment is short and businesslike.

What do you agree up front?

Four things, on half a page. The goal as a number. The threshold above which you continue. The end date. And who decides. Without those four, the evaluation becomes a conversation about impressions.

Also record what happens with the outcome if it works. Who gets the budget, who does the roll-out, and by when. Otherwise the next step is once again a decision nobody takes. An AI workshop is a quick way to get those four points clear with your team.

When do you stop an AI pilot?

You stop a pilot when it stays below your threshold at the decision point and you see no cause you can fix quickly. Also stop when the process itself turns out to be the problem, when the data isn't there or when you are not allowed to use it. Those situations are covered in when not to use AI.

There are also signs during the pilot that it is time to stop:

• Someone adjusts the success criterion halfway, so that it is "met after all".
• The team quietly keeps doing the work by hand, just to be safe.
• The cost of maintenance and licences in production is higher than what the pilot saves.
• Nobody wants to own it once the pilot is over.

What I would not do: extend the pilot "to have another look". You are putting off a decision and paying in the meantime. Two extra weeks is fine, but only with one concrete change and the same yardstick. If the team works around the solution, also check whether the users were involved from the start. With an AI platform for TV production, feedback from editors and story producers helped decide what we built from the very beginning.

A common scenario: halfway through, the data the pilot needs turns out not to be in one system, but spread across mailboxes and spreadsheets. Then you stop, sort out the data first and start again afterwards.

Stopping for a reason is a good outcome. You know what it isn't, and you have made the next pilot cheaper. If the pilot does meet the criterion, the next step is not "roll it out to everyone". First you set up maintenance, appoint an owner and agree who handles the exceptions. Then the pilot becomes an AI employee that runs alongside your team every day. If you want to know beforehand which process has the best chance, that is what the AI Opportunity Audit finds out. Indicative prices are in what an AI solution costs.