Quality is one of the four performance measures that you should consider when building an Agile Balanced Scorecard. The other three are Value, Velocity and Reliability.

The Key Performance Indicators for Quality include the following:

• Number of beta defects – how many bugs are raised (and fixed) prior to go-live?
• Number of live defects – how many bugs are getting through to the live environment?
• How many stories are rejected at the end of each sprint? why?

If you are concerned about quality then ask yourself the following questions:

• Have you integrated testing throughout the development cycle?
• Does the development team help to test their software?
• Do you the developers demonstrate their own output (to stakeholders?) at the end of each sprint?
• Do your effort estimations include testing? They should.

Testing should be integrated throughout the development cycle.

Start new task> Define>Build>Test(fix)>Accept>Done > Start new task> etc…

You should encourage the team to see a task through to completion and in doing so, make sure one thing works before starting another.

You can introduce Work in Progress limits on different steps e.g. only 3 things are allowed to be ‘In Progress’ or the ‘QA’ column at any given point in time. This will help to make sure that you don’t bottleneck at one stage – which often seems to be the testing/fixing stage.

It may be that the developer(s) has to sit with a Test Analyst or a stakeholder to ensure that the task is acceptable before moving on. This encourages accountability and also gives the team a chance to show off their work.

Another thing to consider is your definition of ‘Done’. Does ‘Done’ mean:

• developed and awaiting QA?
• Tested, Passed and Accepted?
• Released?

Make sure your expectations are clear so the team knows what they have to achieve in a sprint. Remember, if it’s not Done, it’s not counted towards your velocity! ‘Done’ should at the very least mean accepted by the product owner.

The satisfaction of the product owner and your general stakeholders is very important. Ask them what they think about the quality of the output.

Although you only count accepted Story Points into your velocity, there is still a range here – are they scraping past or are they ‘Wow-ing’ the Product Owner/Stakeholders? As satisfaction is an intangible measure, you need to use proxy-measures to ensure that everything is going to plan – e.g. a positive email, good feedback or you could circulate a survey on a quarterly / bi-annual basis.

Another thing you could do is track live defects and beta defects. The number of beta defects detected give some indication of the quality of QA and of your development.

The number of live defects will be a lead indicator for stakeholder dissatisfaction – more live defects = fewer happy customers.

We use an Excel Template to track the number of live defects, beta defects and number of patch releases on a sprint-by-sprint basis. It is a simple metric that requires no additional effort to extract assuming you use a bug tracking tool like Elementool or Fogbugz.

Feel free to download a sample of this template below.

Subscribe to the to be notified when I upload new Articles, Videos, Templates and Tips!

Related Articles:

• Agile Balanced Scorecard – Measuring the effectiveness of an Agile Team (agile101)
• The Agile Balanced Scorecard – Tracking the Reliability of an Agile Software Development Team (agile101)
• Using Velocity to measure the Productivity of an Agile Development Team (agile101)
• Minimising the impact of Sprint disruptions (agile101)
• What’s my team velocity? (agile101)
• Work-In-Play Limits in Agile Software Development (agile-software-development.com)