Metrics for Continous Improvement

Problem Domain
What are the best metrics to use to measure and encourage Continuous Improvement on Agile development teams. There are lots of metrics out there, which ones work best for determining if a team is getting better, and which ones are more effective for causing teams to think about things that they could do to improve. This problem definition assumes that different metrics are exclusive to one purpose over the other, however there may be some metrics that serve both purposes equally well.

Discussion
Key points raised during the discussion:

• Having a goal when using metrics is important. You need a goal to use metrics effectively.
• Should the team determine success, as opposed to the definition of success being imposed on the team (likely by management)? What's a good technique to encourage teams to determine their own success measures?
• A good metric is easy - easy to interpret and easy to gather.
• You get what you measure!
• Everything can be gamed.
• No single metric is a silver bullet. You need multiple metrics to cover the multiple facets of the process you're trying to learn about and improve.
• Metrics need their own meta process - that is, for your metrics process you need to have checkpoints, retrospectives and an improvement stream so your metrics process can evolve and improve.
• Value - how do you use metrics to measure the full value chain? Did we create value?
• Metrics should include not only objective but also subjective measures. For example, how do you measure that the team is happy? Team satisfaction can be a measure of success.
• Must keep in mind that metrics just surface information, forced compliance with metrics is bad as metrics can be gamed.
• Metrics are a tool for helping teams identify concrete actions to improve.
• There should be two primary audiences for metrics, for management and for the team.
• Metrics don't replace management and leadership.
• There's a minimum amount of rigour needed for metrics to make sense.
• Trending is extremely important to consider.
• There's a difference between metrics and tracking. Metrics are most useful when they are used to learn and improve.
• A medical analogy is useful when thinking about metrics for teams. A doctor uses metrics such as blood pressure, temperature, etc. as indicators of a potential problem but they don't necessarily indicate the problem. The actual problem is usually determined through follow-up investigations with a specialist. In the same way, a suite of metrics can indicate a potential problem but the actual problem usually needs additional investigating to define.
• Most companies don't have long-terms visions right now that would be useful for a team to use metrics to achieve.
• Ideal metrics process:
  • Establish a goal
  • Determine metrics - measures, norms, trends
  • Attempt to game the metric yourself
  • Implement the metric
  • Review how successful the metric has been, identify improvements and go back to Step 1
• Team vital signs to consider:
  • Team cohesiveness
  • Time stories are in QA
  • Number of bugs in Production
  • Burn rate
  • Bug fix time
  • Cost vs. revenue