Insights
When should an AI experiment become a real implementation?
Most scale-ups I meet are past the start line on AI. They have a list of experiments, several of which "work", and a board that has started asking what any of it did to the numbers.
The published evidence points to where that gap comes from. A pilot can prove a tool does a task well, and the measured value arrives later, when someone changes the work around the tool.
How do you decide when an AI experiment should become a real implementation?
An AI experiment should become a real implementation when the pilot has beaten a baseline on a number agreed before it started, and the company is ready to redesign the workflow around the AI, because published studies find that the measurable value comes from that redesign.
Most decision checklists cover the first half well: thresholds set in advance, risk, cost, someone to support it. The second half gets less attention. In McKinsey's 2025 analysis of 31 factors, workflow redesign made "one of the strongest contributions" to business impact, and clear rules for when a person checks the AI's output were among the factors that set the best companies apart (McKinsey, 2025).
Why do successful AI pilots often show no business impact?
Successful AI pilots often show no business impact because the time they save on a task is absorbed into other work unless someone changes the workflow, and large studies find almost no company-level gain two years after adoption.
Field studies find task-level gains of 10% to 140%. Yet Danish records covering about 25,000 workers and 7,000 workplaces rule out effects on earnings or hours above 2% two years after adoption, with reported time savings averaging about 3% (BFI, 2025).
In a Microsoft 365 Copilot study across 66 large firms and 7,137 workers, people spent about three hours a week less on email, and their meeting time did not change at all (NBER w33795). Microsoft co-authored the study, and its versions differ on the email figure. The UK Department for Business and Trade ran its own Copilot pilot with 1,000 licences and found no robust evidence that the time saved improved productivity, though the pilot had no proper control group (GOV.UK).
At company level, IBM found that 25% of AI initiatives delivered the return expected and 16% scaled across the enterprise (IBM, 2025). Economists call the pattern a productivity J-curve. Firms spend first on redesigning their processes, which pulls measured productivity down before it lifts it (NBER w25148).
I saw a small version of this in a survey I ran in April with founders across one VC firm's portfolio. Of the roughly 10 who replied, about half had AI pilots running, and measuring whether AI was working came last on their list of priorities.
What does it mean to redesign a workflow around AI?
Redesigning a workflow around AI means changing who does each step, which steps the AI takes over and when a person checks the AI's output, so the time the AI saves turns into more output or lower cost.
I start by mapping how the work runs today, with the people who do it, before anything gets automated. It is the same habit as starting from the job the work is meant to get done. Then each step gets an owner: the AI, a person, or the AI with a person approving. Handovers change, and so does how new starters learn the job.
The best-documented example comes from customer support. In a study of 5,172 agents, new agents with two months of AI help performed like agents with more than six months' experience working without it (NBER w31161). That gain pays only if the company changes how it hires and trains for the desk, and staffs it to match.
What do companies that get value from AI do differently?
Companies that get value from AI redesign whole workflows and measure the result: in McKinsey's 2026 survey, nearly three-quarters of AI high performers had fundamentally redesigned workflows, against a quarter of other companies.
Those high performers are about 6% of the 1,719 companies surveyed, and they are twice as likely to have a defined process for measuring impact (McKinsey, 2026). BCG's equivalent group, the 5% it calls future-built, captures 70% of its AI value in core functions (BCG, 2025). These are large companies. In a 60-person scale-up, the same discipline applies to one workflow at a time.
How do you test whether an AI pilot is ready to scale?
An AI pilot is ready to scale when it has beaten a baseline on a measured number agreed in advance, the workflow change is written down, a named person owns the system after launch and the value covers the full running cost.
These are the five questions I'd put to a pilot before it scales:
- Did it beat the baseline on the number we agreed before it started?
- Was that number measured, or reported by the people using the tool?
- Can we write down who does what differently once it is live?
- Who owns it after launch, including the budget and the decision to stop it?
- Does the value still hold once we add human review, monitoring and model costs?
Each candidate gets a go or no-go with the reasoning written down, on a date set at the start. Boards need that record, since 82% of directors rate the information they get linking AI to business performance as fair, poor or missing (PwC, 2026).
Which AI pilot results can you trust?
Trust AI pilot results that come from a measured metric with a baseline, and treat time savings people report themselves with caution: in one randomised study, experienced developers using AI were 19% slower yet believed AI had made them about 20% faster (METR). The study's sample size comes from secondary reports.
With tens of people in a team, a scale-up's AI pilot is too small to detect the 1% to 5% changes the large studies measure. Pick a metric where you expect a big change, such as handling time, throughput or how fast a new starter gets up to speed. Support and funnel metrics move within 2 to 8 weeks. Finance and engineering metrics take 2 to 8 months, and churn or retention needs at least one renewal cycle.
When should you stop an AI pilot?
Stop an AI pilot when the number agreed at the start did not move, when the only evidence is self-reported time saved, or when nobody is willing to change the workflow around it, because in each case scaling it adds cost with no measured return.
Mixed results are a third option. If the AI handles some cases well and others badly, restrict it to the cases that work and keep a person on the rest.
At Forward Operating Partners, this is how we approach the first weeks with a founder: decide on the company's own data which AI changes to make, set the number before the work starts, and put a go or no-go on each one.
How many of your AI pilots have a decision date?
FAQ
What is the difference between an AI pilot and an AI implementation?
An AI pilot tests whether a tool improves a task on real data. An AI implementation changes the workflow, roles and measures around the tool so the gain shows up in a company number.
Why do most AI pilots fail to reach production?
Most AI pilots fail to reach production because they have no measured result, no owner after launch and no change to the work around them. IBM found that 16% of AI initiatives scaled across the enterprise and 25% delivered the expected return (IBM, 2025).
How long should an AI pilot run?
An AI pilot should run long enough for its chosen metric to move: 2 to 8 weeks for support and funnel metrics, 2 to 8 months for finance and engineering, and at least one renewal cycle for churn or retention.
What metrics should decide whether an AI pilot scales?
The decision to scale an AI pilot should rest on a measured number agreed before the pilot started, on a line the board already reviews, with the full running cost included. Self-reported time savings are not enough.
Why isn't AI showing up in our P&L?
AI time savings reach the P&L only when the workflow is redesigned to use them. Large studies find task-level gains of 10% to 140% and almost no company-level gain two years after adoption.
Who should own an AI system after the pilot?
A named person in the team that runs the workflow should own the AI system after the pilot, with the budget, the monitoring and the decision to change or stop it.
How do you avoid AI pilot purgatory?
To avoid AI pilot purgatory, set a decision date, the success threshold and the kill criteria before the pilot starts, and make the go or no-go call on that date.
Do small companies need a different approach to scaling AI pilots?
Small companies need AI pilot metrics with large expected effects, such as handling time or new-starter ramp-up, because their pilots are too small to detect the 1% to 5% changes that large studies measure.