I've been developing software with passion for over ten years and for many companies, deeply immersed in the challenging domain of finance. I'm always experimenting on how to improve our craft, in particular with Domain-Driven Design, patterns and agile principles. cyrille is a DZone MVB and is not an employee of DZone and has posted 13 posts at DZone. You can read more from them at their website. View Full User Profile

Visualizing Quality – keynote by Gojko Adzic at BDD Exchange 2011

12.10.2011
| 3005 views |
  • submit to reddit

How do you measure quality?  Number of defects? Customer happiness? Money earned? Developer smiles? That’s the question raised by @gojkoadzic in his keynote at the recent BDD and Agile Testing Exchange in London, to make us think and propose some elements of response.

We tend to ignore information

We are used to ignore automatic alarms, even fire alarms:  we just don’t care, but we care when a person tells us about an issue or is shouting « fire in the building! ».

Security Poster in a mine

As in the story of the Gloves on the Boardroom Table,  making the problem tangible and visible help trigger reactions. The book Switch gives examples of such surprising techniques. Sharing information is not enough, a company is like an elephant, you don’t really drive it, you motivate it to go where you’d like to go. So visualization can help there.

Best practices change over time, as the techniques in athleticism replace each other. For example at one company they disable most of their functional tests after a feature is developed, because they impede the frequency of their release cycle; on the other hand, they closely monitor the conversion rate of the website, which is for them the best metrics since it encompasses both the technical bugs and the business bugs.

This does not mean you should disable your functional tests, but it illustrate that what may seem nonsense today may be the next best practice tomorrow.

Absence of bugs ≠ Presence of quality

The absence of bugs is completely different than the presence of quality. Twitter has lots of bugs but most people are happy with it so it has quality; on the other hand, Nokia phones have no bug, yet if nobody likes them they have no quality either.

Onboard screens in a plane all showing a computer crash

Do we have quality here?

So how to measure quality? If you put 20 people in a room with 3 post-it’s each and ask them to write what attributes of the product they think are the most important, and if you get a small enough group of common answers then they are candidates of things you need to measure.

Usually quality c an be measured at 3 levels (the 3 P’s):
  1. Process effectiveness
  2. Product status
  3. Production performance

Here I have the feeling that each of the first two is just a proxy for the next, and only the last is the real one, though I’m not sure.

Test coverage is crap

Taking an example of a UI, a window with 5 buttons on it it will likely requires 5 time more testing than one with only one button, even if the later brings a lot more value, which suggests that test coverage is crap.

If test coverage itself is crap, you may however monitor your risk coverage, that is the amount of testing compared to the associated risk. This metrics helps identify where you need to test more.

Visualize to make information actionable

Once you know what to measure, you want to visualize it. James Bach proposes his technique of low-tech testing dashboard. You divide your system into high-level subsystems, and for each you monitor the « progress of testing »: not yet started, half done or done, and the « level of confidence so far » with 3 smileys from happy to anxious. That’s better than a bug-tracking tool with 500 tickets mixed altogether.

Another visualization mode is the use of effects maps, or ACC matrix. You may also monitor in realtime the feed of sentiment of every Twitter message about your company or product as they do at FINN Technology in Oslo.

As a conclusion, Gojko introduces the brand new visualisingquality.org website where you can find many ways to visualize your measures of quality and where you can contribute additional ones. Since making things visible is so essential for action, let’s try ideas and share them through this website!

References:

The podcast of the full talk is available here: http://skillsmatter.com/event/agile-testing/agile-testing-bdd-exchange-2011/js-2990

 

From http://cyrille.martraire.com/2011/11/visualizing-quality-keynote-by-gojko-adzic-at-bdd-exchange-2011/

Published at DZone with permission of cyrille martraire, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Afandi Merathi replied on Sat, 2012/03/24 - 8:14am

Actually, that suggests that that model of test coverage is crap. The idea that one button requires a certain amount of testing, and that five buttons naturally requires five times as much is terrible measurement and worse testing. Are there interdependencies between those buttons? What do those buttons do? Does each do the same thing? Are we talking about an online dating service or a nuclear power plant, here?

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.