What are meaningful Testing KPIs?

KPI… Key Performance Indicator. Such a popular shortening, used in almost every organisation, talked in every 2nd meeting at and traced by millions of brains around the world. Be it IT, be it production factory, be it delivery company. We all are measuring our output and performance against our target goals.

We as humans and we as companies, we all are heading somewhere. Unfortunately, it is hard to judge objectively using just common sense, if we are heading for the right direction. KPIs, value and data-based decisions coming from them are closest to what we, imperfect creatures, could use to avoid subjectivity.

In the last 10 years, I was working and developing my know-how in IT Quality Assurance area. I was lucky to experience different working and testing cultures in East Europe, Scandinavia and now Switzerland. During 10 years I saw successful and not so successful deliveries in Agile, Kanban, Waterfall, Nexus, SAFe or we just do it methods. Different managers made me aware of different stakeholders expectations. Moving to different companies placed me in the shoe of a software vendor, IT consultant, government project participant, Investment and Wealth Management banker, internal software development company and so on.

And in each project Test Manager has to answer the same question: How to provide information for Production release in a timely, valuable, human-understandable value. Even more, how we track if testing, processes and product quality are getting better? As you might guess and even know, just passing a certification exam will not get you the answers.

I was lucky, at least this is how I see my experience now, to have demanding people around me, questioning data I am providing, which helps to measure Quality. Today I would like to share my experience, which meaningful KPIs your company or your team could implement in this area.

So what KPIs to use?

I am not going to tell you about counting defects and test cases. It is straight forward exercise, usually leaving everyone in the meeting clueless about what it really means. I suggest trying to track the following information:

• Defect ration in each testing phase (ie Agile Sprint defects / Pre-prod regression defects). This KPI, and especially trend over time, would allow gathering information about early testing availability. Late defects in development cycle could mean non-testable features, lack of integrated test environment, bad quality of requirements, not stable code etc. In other words, used wisely, it could give valuable insights into the whole development chain.
• Technical debt increase/decrease per release. Is your product getting better after a few months or weeks of development? Will your users be happy? If you could be even more specific, which areas are affected by problem accumulation, even better!
• How much time test case has to be executed and in which areas. None likes repeating test scripts, especially if this is done manually. However, information about test repeats could allow planning of the next testing phase, identify automatable areas or features, where tests fail (due to any reasons, including defects).
• The ratio of automatable tests versus automated tests. Measure your progress against the plan, not just plane number.
• Exploratory Session time pro release. I could write a whole new essay on how exploratory/session testing wins against scripted testing. Knowing that team of professional test specialists had a chance to inspect system without predefined scenarios, and thus identify new issues, questions or even requirements is the best quality assurance and security net you could get for any product.
• End-to-End environment testability time versus planned testing time pro r release. Handy KPI to measure, if with new development methods (Agile, SAFe) the system is enough time under test. I know systems who could be E2E tested in a few hours, and I know systems which even with automated checks needs a week or two for one regression test set.
• How buggy is your business flow. Do not track defects in total. Track business flows in colours, with defect number nearby. You get the attention of each stakeholder, who will see a red colour near her area.

Rules applying KPIs

I would allow myself to be laconic and name just a few:

• Tracking this information should not be done manually, but generated automatically using tools you have.
• Information should be available online every time with life data
• Link to KPIs is sent to all involved stakeholders periodically, making everyone get used to seeing that numbers. Be sure you have an intro meeting and that your target group understands how to read the data. As the old saying says — there is nothing worse than the CEO with statistics.
• KPIs measure processes, not people. And KPIs measure processes, that are important, not just because data are available.

I would be more than happy to hear your experience — have you seen those KPIs applied? Have you effectively used other ones? Let me know!