Data gathered by Scrum Foundation founders Sutherland, renowned software productivity researcher Caper Jones from SPR and leading Scrum Consultants have measured gains ranging from 450-750% by Scrum teams over the velocity of waterfall projects. However over 90% of the Scrum teams are not able to deliver the velocity seen in high performance Scrum teams.
Scrum teams have trouble measuring performance. Teams do not know their velocity of production and ways to improve and measure this rate. Management teams too are finding it difficult to compare the performance of their Scrum teams. Scrum teams need a set of reliable lightweight metrics so that Teams can monitor their performance easily and quickly correct if problems arise. The lack of appropriate metrics can prevent Scrum teams from systematically improving and reaching productivity significantly (~300-400%) better than the average waterfall team.
Scrum & Management teams require understanding of the following ten vital aspects to understand the progress made and the issues requiring correction
- Are we increasing/maintaining the quantity of work delivered?
- What’s the amount of all work reported in a sprint?
- What percentage of the team’s work results in finished stories?
- What percentage of the team’s work is spent on additional work discovered to complete the Sprint backlog?
- What percentage of the work in the sprint is brought forward from the product backlog?
- Are we conservative/optimistic in estimating the work to be taken up in the sprint?
- Are we able to deliver the Story Points we committed to for a Sprint?
- Is the Team’s effort aligned with the business?
- What is the maximum size of Sprint Backlog items the team is able to deliver successfully?
- What is the degree to which the team is able to meet their commitment to the customer?
Here are ten metrics that can help Scrum & Management teams to answer the above questions to develop and sustain high performance—Velocity Increase, Work Capacity, Focus Factor, Percentage of Adopted Work, Percentage of Found Work, Accuracy of Estimation, Accuracy of Forecast, Total Business Value, Success at Scale, and the Ratio of Successful Sprints of the Team.
| Metrics | Formula | Average Range | Interpretation |
| Velocity Increase | Current Sprint’s Velocity ÷ Original Velocity | 300% – 400% for Teams properly coached on how to achieve performance | This should increase gradually and stabilize by about 10 sprints Velocity – Velocity is defined as sum of original estimates of all accepted work |
| Focus Factor | Velocity ÷ Work Capacity | ≥ 80% | Focus Factor should remain ≥ 80% on average for a healthy team Data points below 80% indicate a team that is disrupted by external events or otherwise incapable of turning their Forecast work into Accepted work Work Capacity – The sum of all work reported during the Sprint, whether the Sprint Backlog Item toward which the work was applied finished or not. |
| Percentage of Adopted Work | Σ(Original Estimates of Adopted Work) ÷ (Original Forecast for the Sprint) | ~5-15% | % of Adopted Work + Found Work should not exceed 20% of the Original Forecast Adopted Work is work that is brought forward from the Product Backlog at any point during the Sprint because the Team has completed their original Forecast Sprint Backlog. |
| Percentage of Found Work | Σ(Original Estimates of Found Work) ÷ (Original Forecast for the Sprint) | ~5-15% | % of Adopted Work + Found Work should not exceed 20% of the Original Forecast Found Work is work associated with a piece of Sprint Backlog Item which is above and beyond what was initially expected but which must be completed to deliver the original Backlog Item. |
| Accuracy of Estimation | 1-(Σ(Estimate Deltas) ÷ Total Forecast) | ~80% | Above 88%, it is likely that the Team is being overly conservative and possibly spending an inordinate amount of time planning Below 72%, the Scrum Master should investigate the pressures on the Team. |
| Accuracy of Forecast | (Σ Original Estimates) ÷ Σ (Σ Original Estimates + Σ Adopted Work + Σ Found Work) | ~80% | Above 90%, the Scrum Master needs to evaluate the environment of the Team to be sure that they feel safe making a good faith effort at more work Below 75-80%, it is generally because the Team is heavily randomized or is not being adequately protected by the Scrum Master during the Sprint. |
| Total Business Value Earned | Σ Business Value that was earned for accepted work during a sprint | Business Value should be proportional to the Velocity increase | Velocity increase should be correlated with Business Value Earned in each sprint. The correlation will help assess whether Velocity increase is due to the fact that the team was delivering items of little Business Value Use an estimation technique similar to the one used for Effort Estimation to associate a number of Value Points to a work item, for example using the Fibonacci range. Business Value should be estimated by the representatives of the Business, i.e. the Product Owner (PO) and business stakeholders, and not the development team. |
| Success at Scale | For each Point on the Fibonacci Scale (Fp), the formula is:(Σ No. Accepted Attempts of scale Fp) ÷ (No. of All Attempts of scale Fp) | ~80%+ | This gives Teams a sense of confidence during Sprint Planning Meetings, as well as helps them select the granularity to which Sprint Backlog Items should be itemized Unwise to accept a single Sprint Backlog Item larger than 50% of the teams Velocity |
| Ratio of Successful Sprints | No. of successful Sprints ÷ Total No. of Sprints | >90% | Scrum Master should investigate causes for Sprint Loss Each Sprint is a Win only if:a) A minimum of 80% of the Original Forecast is Accepted) Found + Adopted Work During the Sprint remains at 20% or less of the Original Forecast. |
Scrum teams should be encouraged to apply metrics to their Scrum processes, after all, the numbers tell the tale. In this whitepaper, we provided a lightweight and understandable set of metrics that can be used to track a Scrum team’s performance. This simple set of metrics are easy to implement and has a powerful effect on the performance of Scrum Teams.
References
- Scott Downey& Jeff Sutherland, “Scrum Metrics for Hyperproductive Teams: How They Fly like Fighter Aircraft”
- G. Benefield, “Rolling Out Agile at a Large Enterprise” in HICSS’41, Hawaii International Conference on Software Systems
- C. Jones, “Software Engineering Best Practices: Lessons from Successful Projects in the Top Companies” McGraw Hill, 2010.