Delivery Forecasting with Monte Carlo Simulation: Committing to Dates with Probabilities

A Monte Carlo simulation forecasts delivery dates by recombining a team's recent weekly throughput at random thousands of times and deriving a distribution of possible completion dates. Instead of one date you get a statement such as “85 per cent likely by the end of July”. The method needs no estimates in hours or points, only the number of items a team actually finishes each week.

This article explains the method through a worked example, shows which data you need, and describes how to present the results to the executive board and to customers without promising a date that nobody can keep.

What is a Monte Carlo simulation in delivery forecasting?

A Monte Carlo simulation plays through the future of an undertaking many thousands of times, each time with values drawn at random from the past. The name goes back to the method that Nicholas Metropolis and Stanislaw Ulam described in a 1949 paper on the Monte Carlo method. In software development and project work it was popularised by, among others, Troy Magennis and Daniel Vacanti, whose book “Actionable Agile Metrics for Predictability” appeared in 2015.

The simulation works like this. You know your team's throughput, meaning the number of items completed per week, for the past twelve weeks. For each simulated run, the computer picks one of those twelve values at random for every coming week, adds the weeks up until the backlog is cleared, and records the number of weeks needed. It repeats this ten thousand times. The ten thousand results form a distribution: so many runs finished after nine weeks, so many after ten.

You read the result as percentiles. The 50th percentile gives the number of weeks that sufficed in half of the runs, and the 85th the number that sufficed in 85 of 100 runs. For commitments to third parties, teams usually use the 85th or 95th percentile.

Why are averages dangerous for deadlines?

Averages are dangerous because a date based on the average is missed in about half of all cases. Sam Savage, a consulting professor at Stanford University, described this in his 2009 book “The Flaw of Averages”: plans based on average values go wrong on average.

An example shows the scale. A team completes 4.75 items a week on average, and the backlog contains 40 items. Dividing gives 8.4 weeks, and the team promises eight. The spread of the weekly values is about 1.45 items. After eight weeks the team has on average completed 38 items, and the probability of reaching 40 or more is just under one in three. Two out of three deadlines of this kind are therefore missed, even though each individual calculation looked correct.

Estimates in hours or story points also tend to fall short of what actually happens, because they leave out waiting time, interruptions and rework. Throughput measures what was actually finished after all those influences.

How do you calculate a delivery forecast?

You calculate a delivery forecast in five steps, from recording completions to a commitment with a probability attached.

1. Record throughput: count, for each of the last ten to twenty weeks, how many items the team completed. An item counts when it meets the definition of done.
2. Count the backlog: establish how many items are missing for the goal. Slice them to a similar size so that counting is meaningful.
3. Allow for growth: if a backlog tends to grow because items split during work, include a growth rate. If the list typically grows by a fifth, calculate with 48 items instead of 40.
4. Run the simulation: a spreadsheet is enough for the calculation, as is a small script or one of the freely available community tools.
5. Read off the percentiles: state the 50th, 85th and 95th percentile together with their probabilities.

The values are an approximation using the mean and spread of the twelve weeks in the example. A real simulation over the raw data gives results of a similar magnitude, plus the shape of the distribution, which differs when there are rare outliers. Growth from 40 to 48 items moves the commitment by two weeks, an indication of how much weight the quality of the count carries.

When is a Monte Carlo forecast reliable?

A Monte Carlo forecast is reliable when four conditions are met: past throughput also holds for the future, items are of similar size, the team remains largely stable, and the number of items in progress at the same time does not fluctuate widely. John Little described the link between lead time, throughput and work started in 1961 in the law that bears his name: the average number of items in the system equals throughput times average lead time. Whoever limits work in progress makes the forecast more stable. The article on limiting work in progress shows how.

The forecast loses its value when circumstances change fundamentally: a team grows or shrinks, the type of work changes, or an external dependency delays completion. In these cases you reset the data and calculate with the weeks since the change.

Monte Carlo or estimation: which fits which situation?

The simulation fits teams that deliver steadily and have a history, while estimation fits new undertakings without a data basis. The two methods do not exclude each other: for a new undertaking you begin with a rough estimate of the number of items and replace it with measured values once the team has delivered for a few weeks.

How do you present forecasts to the board and to customers?

You present a forecast by making the probability the basis of the decision: the board chooses the risk, and the team names the date that goes with it. A commitment at the 85th percentile means that in 15 of 100 comparable cases more time would be needed. Whoever states this number openly moves the discussion from the date to the risk, and to the question of whether scope or budget can be adjusted if an earlier date is demanded.

This also helps with contracts and fixed dates. For an undertaking with high costs of delay you choose the 95th percentile and keep scope ready as a variable element, as the article on agile fixed-price projects describes. The forecast complements planning across many teams: for dates that depend on several teams, also read the article on release planning.

Which mistakes occur most often in forecasting?

Four mistakes occur most often. First, teams use too little history, for example only three weeks, and the distribution does not reflect the pattern. Second, they count items of different sizes as equal without splitting the largest. Third, work that is later discarded flows into the figures, so throughput looks higher than the benefit. Fourth, the forecast is made once and never updated, although new weeks would improve the statement. An update after each week takes little effort, and the gap between forecast and reality shows early whether an undertaking is falling out of rhythm. As an early warning system, flow metrics supply the measures.

Steering forecasts with Niehues Consulting

Niehues Consulting is a consultancy for AI management and organisation design from a delivery perspective, based in Düsseldorf. Founder Marin Niehues builds organisations that deliver, in projects ranging from VC-funded scale-ups to DAX 40 corporations and in more than 20 countries on six continents. In the service area of delivery management and organisation design he steers delivery as delivery manager, project manager or product owner.

The approach follows the Niehues Delivery Model. Two of its test questions matter most for delivery forecasts: What is the baseline, and how do we measure success? Who is responsible for operations from day one? It follows that data capture and forecasting belong in the team's ongoing steering, where they are refreshed every week. You start with a free initial consultation of 30 minutes, which you can book via the contact form. The scope and terms of an engagement are discussed in the free initial consultation.