Measuring Agile Transformation: How to Recognise Progress Beyond Certificates

Progress in an agile transformation is measured on three levels: the delivery of the teams, the collaboration between teams and the decision behaviour of leadership. Certificates and training rates, like the number of teams applying Scrum, show activity and say little about impact. A measurement system with six to eight measures, a baseline and a quarterly rhythm gives the executive board the answer that a certificate report cannot.

This article is written for managing directors and heads of department who want to know, after one or two years of agile working, whether anything has changed. It shows which measures are meaningful at which level, places them in a maturity model and provides a checklist for the first measurement cycle.

Why do certificates say little about the progress of an agile transformation?

Certificates show that a person knows a method, and they say nothing about whether the work has changed. A Scrum Master certificate shows that someone has understood the Scrum Guide and passed an exam. Whether the team afterwards delivers faster, whether decisions fall earlier and whether customers see results sooner cannot be read from it. The same applies to training rates and to the number of teams working in Scrum. Such figures are easy to collect and rise in every programme, which is why they appear in almost every transformation report.

The reason lies in measurement logic. Input measures such as trainings or ceremonies count what the organisation puts into the programme. Output measures count what arrives with the customer and in the organisation. The Scrum Guide of 2020 demands empiricism, that is, decisions based on observation, and the same logic applies to the transformation itself: anyone building an agile organisation should steer it by observed impact.

On which three levels can progress be measured?

Progress can be measured at the level of the teams, of collaboration between teams and of leadership, and each level has its own measures. The division follows the three Flight Levels described by Klaus Leopold: the first covers the operational delivery of the teams, the second the coordination between them and the third the strategic steering of initiatives. Anyone who measures at all three levels sees where a transformation stalls. How the levels connect is described in the article on Flight Levels.

At team level the flow metrics count: cycle time and throughput as well as the age of work in progress. They show whether teams deliver faster and more predictably. The article on metrics for agile teams explains which of them suit management. In addition, the stability of delivery, for example the share of releases that had to be reworked, shows whether speed is bought at the expense of quality. These measures come from the research behind “Accelerate” by Forsgren, Humble and Kim from 2018, where they mark high-performing organisations.

At the level of collaboration it becomes clear whether the teams work as a network. Meaningful measures are the number of dependencies a team has on other teams, the average wait for a contribution from another team and the share of work that leaves a team without a handover. Falling values show that the way teams are cut fits the work better, as the article on Team Topologies describes.

At leadership level you measure how quickly decisions are made and how often leadership changes priorities. One measure is the time from a team's request for a decision to the answer. A second is the number of initiatives running at the same time relative to the number of teams. If leadership does not change its own work, the progress of the teams remains limited, because delays then arise at the interfaces with leadership.

What does a maturity model with measures look like?

A usable maturity model describes four stages and assigns each stage an observable measure, so that the rating rests on data. The stages are Apply, Stabilise, Connect and Steer. The model is a working version of this article and not a published standard. It can be adapted to your own organisation. The rating is made per business area, because areas are at different points.

The stages build on one another, and an area can leave a stage only when its measure has improved. Many transformations stay on the first stage because it is easy to count there: the more teams adopt roles and ceremonies, the more successful the programme looks in the report. The move to the second stage requires cycle times to be collected, and that is the moment when programmes see uncomfortable figures for the first time.

Which baselines does a measurement need?

Every measurement needs a baseline from the time before the transformation or from its first weeks, because without it no change can be evidenced. Three measures per area suffice for the start: the cycle time of a typical unit of work, the time from a request for a decision to the answer and the number of initiatives running at once. If the history was not recorded, collect the values retrospectively from tickets and calendars as well as approval documents for the last twelve weeks. The back-calculation is imprecise, yet it provides an order of magnitude that is enough for the comparison after one quarter.

A short survey of those involved is also worthwhile, for instance with five statements on clarity of goals, reliability of collaboration and room for manoeuvre, answered on a scale of one to five. Amy Edmondson's research on psychological safety in teams from 1999 shows that teams learn better where members can raise mistakes. A survey makes this precondition visible. It complements the process data and does not replace it, because sentiment scores are easier to influence than cycle times.

Which measurement traps exist?

Three traps recur in transformation programmes. The first is linking metrics to targets or pay. As soon as a measure is declared a target, behaviour changes and the measure loses its meaning; this observation is known as Goodhart's law. The second trap is comparing teams on measures that only mean something within the team, such as story points. The third trap is measuring without consequences. A metric with no follow-up decision costs collection effort and breeds distrust.

John Kotter's model of change from 1996 treats short-term wins as a step in their own right. The point is that progress must become visible early so that the organisation follows the effort. A measurement that shows a shorter cycle time in two pilot areas after three months achieves this better than a report on the number of trained employees. It needs data that a business unit can follow in conversation.

Checklist for the first measurement cycle

The checklist shows whether a measurement system is in place. Each question can be answered with yes or no.

1. Does each area have a baseline that predates the programme or comes from its first weeks?
2. Do the measures capture impact, meaning lead time, predictability or decision time, rather than activity?
3. Does at least one measure concern leadership itself?
4. Does each measure have an owner who collects and explains it?
5. Is it settled that no measure enters target agreements or pay?
6. Is a quarterly rhythm set in which the executive board discusses the figures and takes a decision?

The scoring is a rule of thumb of this article: with six yes answers the measurement system is ready for use. With four or five, the gaps should be closed before the first report, and with three or fewer the programme is so far under way without steering.

When is a transformation successfully complete?

A transformation is complete when the organisation adapts its way of working under its own power, and this shows when areas start improvements without a programme. The measurement system then stays and becomes part of normal steering, while the programme as a separate structure winds down. Anyone aiming for this hands the metrics to the line early instead of keeping them in the transformation office. How a stalled programme is set up afresh is described in the article on restarting an agile transformation.

Measurement with Niehues Consulting

Niehues Consulting is a consultancy for AI management and organisational design from a delivery perspective, based in Düsseldorf. Founder Marin Niehues builds organisations that deliver and works as delivery manager, project manager and product owner on projects ranging from VC-funded scale-ups to DAX 40 groups. In the service area of delivery management and organisational design, this includes recording baselines, agreeing measures with the business areas and turning the results into leadership decisions.

The approach follows the Niehues Delivery Model. Three of its test questions matter for measuring a transformation: Which business result should change measurably? What is the baseline, and how do we measure success? Which metric shows after 90 days that the solution has landed? The entry point is a free initial consultation of 30 minutes, which you can arrange via contact. Scope and terms of an engagement are discussed there.