Nicolas Frankel is an IT consultant with 10 years experience in Java / JEE environments. He likes his job so much he writes technical articles on his blog and reviews technical books in his spare time. He also tries to find other geeks like him in universities, as a part-time lecturer. Nicolas is a DZone MVB and is not an employee of DZone and has posted 226 posts at DZone. You can read more from them at their website. View Full User Profile

100% Code Coverage!

04.18.2011
| 11622 views |
  • submit to reddit

The basis of this article is a sequences of tweets betwen me and M. Robert Martin, on April 8th 2011:

  1. If you have 100% coverage you don’t know if your system works, but you _do_ know that every line you wrote does what you thought it should.
  2. @unclebobmartin 100% code coverage doesn’t achieve anything, save making you safer while nothing could be further from the truth.
  3. @nicolas_frankel 100% code coverage isn’t an achievement, it’s a minimum requirement. If you write a line of code, you’d better test it.
  4. @unclebobmartin I can get 100% code coverage and test nothing because if have no assert. Stop making coverage a goal, it’s only a way!

This left me a little speechless, even more coming from someone as respected as M. Martin. Since the size of my arguments is well beyond the 140-characters limit, a full-fledged article is in order.

Code coverage doesn’t test anything

Let’s begin with the base assertion of M. Martin, namely that 100% code coverage ensures that every code line behaves as intended. Nothing could be further from the truth!

Code coverage is simply a way to trace where the flow passed during the execution of a test. Since coverage is achieved through instrumentation, code coverage could also be measured during the execution of standard applications.

Testing, however, is to execute some code by providing input and verifying the output is the desired one. In Java applications, this is done at the unit level with the help of frameworks like JUnit and TestNG. Both use the assert() method to check output.

Thus, code coverage and test are completely different things. Taken to the extreme, we could code-cover the entire application, that is achieve 100%, and not test a single line because we have no assertion!

100% coverage means test everything

Not all of our code is neither critical nor complex. In fact, some can even seen as downright trivial. Any doubt? Think about getters and setter then. Should they be tested, even though any IDE worth his salt can generate them faithfully for us? If your answer is yes, you should also consider testing the frameworks you use, because they are a bigger source of bugs than generated getters and setters.

Confusion between the goal and the way to achieve it

The second assertion of M. Martin is that 100% of code coverage is a requirement. I beg to differ, but I use the word ‘requirement’ with a different meaning. Business analysts translate business needs into functional requirements whereas architects translate them into non-functional requirements.

Code coverage and unit testing are just a way to decrease the likehood of bugs, they are not requirements. In fact, users couldn’t care less about code average, as long as the software meet their needs. Only IT, and only in the case of long lived applications, may have an interest in high code coverage, and even then not 100%.

Cost effectiveness

I understand that in certain particular contexts, software cannot fail: in chirurgy or aeronautics, lives are at stake whereas in financial applications, a single failure can cost millions or even billions.

However, what my meager experience taught me so far, it’s that money reigns, whether we want or not. The equation is very simple: what’s the cost of the test, what’s the cost of the bug and what’s the likehood of it. If the cost of the bug is a human life and the likehood is high, the application better be tested like hell. On the contrary, if the cost of the bug is half a man-day and the likehood low, why should we take 1 man-day to correct it in advance? The technical debt point of view help us to answer this question. Moreover, it’s a decision managers have to make, with the help of IT.

Point is, achieving 100% testing (and not 100% coverage) is overkill in most software.

For the sake of it

Last but not least, I’ve seen in my line of work some interesting deviant behaviours.

The first is being, quality for quality’s sake. Me being a technical person and all, I must admit I fell for it: “Wow, if I just test this, I can gain 1% code coverage!” Was it necessary? More importantly, did it increase maintainability or decrease the likehood of bugs? In all cases, you should ask yourself these questions. If both answers are no, forget about it.

A slightly different case is the challenge between teams. Whereas challenges are good in that they create an emulation and can make everyone better, the object of the challenge should bring some added-value, something the raw code coverage percentage doesn’t.

Finally, I already rambled about it, but some egos there just do things to write on their CV, 100% coverage is just one of them. This is bad and if you have any influence over such individuals, you should strive to orient them toward more project-oriented goal (such as 100% OK acceptance tests).

Conclusion

With the above arguments, I proved beyond doubt that assertions like ‘we must achieve 100% code coverage’ or ‘it’s a requirement’ cannot be taken as a general rule and is utter nonsense without some proper context.

As for myself, before beginning a project, one of the thing on my todo list is to agree on a certain level of code coverage with all concerned team members (mainly QA, PM and developers). Then, I take great care to explain that this metrics is not a hard-and-fast one and I list getters/setters and no assert as ways of artificially increasing it. The more critical and the more complex the code, the higher the coverage it should have. If offered a choice, I prefer not reaching the agreed upon number and yet having both firmly tested.

For teams to embrace quality is a worthy goal. It’s even more attainable if we set SMART objectives: 100% code coverage is only Measurable, which is why it’s better to forget it, the sooner the better, and focus on more business-oriented targets.

 

From http://blog.frankel.ch/100-code-coverage

Published at DZone with permission of Nicolas Frankel, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Tags:

Comments

Sylvain Brocard replied on Thu, 2011/04/21 - 6:58am

I think, you simply see the problem from different point of view.

If I were a craftsman, it seems to be right to test all the code I write (I won't enter in the issue about coverage and test).

If I were a project manager, it also seems right to reach the quality that will grant the best ROI (see for example "Cost/Benefit-Aspects of Software Quality Assurance" of Marc Giombetti).

Matt Avery replied on Tue, 2011/04/19 - 5:07am

I've done a few projects now where I've achieved 100% code coverage (line coverage, not path coverage), or very close.  Those experiences changed the way I program and now I'm writing a book entitled Increase Your Test Coverage.  A couple of excerpts from the first draft:

"The intent of this book is to encourage ever increasing test coverage throughout the life of a project, but the ultimate goal is to improve code quality by improving the object oriented design.

In point of fact, I want to discourage you from setting a coverage goal. The starting point is to begin measuring coverage. Once you begin measuring coverage, the logical next step is to increase your test coverage."

 and later on

"Yes, we're going to find out how to increase test coverage in a number of ways, but many of these techniques are going to prompt you to make design changes in your code and that is the whole point. It's all about OO design. This whole test coverage issue is just to facilitate making code changes. The fact that you will end up with a test suite that is going to be a great help in preventing regression bugs and serve as the ultimate in project documentation are secondary benefits."

 I don't want to come off as too adversarial here, but your comments on "cost effectiveness" are plainly a straw-man argument.  It is impossible to know how much the bug is going to cost ahead of time and often difficult to determine the likelihood of its occurance. The "cost" often has nothing to do with man-days.  If my manager gets a black eye in some high level meeting because our team let a small bug get out, it can be extremely costly to our team!

Again, the "for the sake of it" section regarding egos and CV entries is upside down.  In my hard-won experience, these types of comments invariably come from someone who has not even attempted to achieve 100% code coverage on a project.  I'm not saying that is the case here, but if you haven't done it, I strongly encourage you to go through the exercise of achieving 100% code coverage on a small project of your own and see if it changes the way you program.

 

Artur Biesiadowski replied on Tue, 2011/04/19 - 7:12am

Question for 100% code coverage people:

What is the biggest project size in LOC (excluding tests) on which you have managed to achieve 95%+ code coverage? Anybody getting it on something in range of 1-10 millions LOC ?

Matt Avery replied on Tue, 2011/04/19 - 9:08am in response to: Artur Biesiadowski

The largest for me was 80K instructions.  Coverage was around 99.6%.  I don't know what the line count was, but in terms of coverage, I use the EclEmma Eclipse plugin and it reports the number of instructions covered.  I honestly don't know how LOC correlates to number of instructions.  Just looking at a few classes, I see that some classes have more instructions than LOC and others are the reverse, so perhaps it is close to a one-to-one ratio?

That project made use of many other in-house modules.  We have about 14 million LOC of Java code under maintenance total.  I imagine the project I worked to get the code coverage on made use of a small fraction of that, but certainly far more than 80K LOC in total. 

Zqudlyba Navis replied on Tue, 2011/04/19 - 5:54pm

What is the biggest project size in LOC (excluding tests) on which you have managed to achieve 95%+ code coverage? Anybody getting it on something in range of 1-10 millions LOC ?

I worked on a USD$200 million dollar project, with 40+ developers and 2 million+ Lines Of Codes (excluding junits). In 3 years, we only managed 80% junit code coverage. Our selenium tests didn't cover all UIs, but only UIs that are heavily used (and cannot live without), by customers. It's very easy to achieve 100% code coverage on toy projects or projects that cost less that USD$1 million, but for a USD$200 million+ project, we find acceptance tests (on critical screens only) to be more useful.

Jose Maria Arranz replied on Wed, 2011/04/20 - 2:20am in response to: Zqudlyba Navis

"It's very easy to achieve 100% code coverage on toy projects or projects that cost less that USD$1 million, but for a USD$200 million+ project, we find acceptance tests (on critical screens only) to be more useful "

I agree, this is the HARD truth that nobody wants to talk about when talking about testing because "sounds bad".

In pet projects your favourite RAD tool, your favourite dynamic language and your perfect desires of quality based on extensive testing, all of them shine...

When things get complicated with lots of complex code...

- Your favourite RAD tool becomes too intrusive, simplistic and rigid

- Your favourite dynamic language fools you because of the lack of a compiler doing tons of tests for free, lack of refactoring, code browsing (Find Usages at NetBeans, References at Eclipse), dependency analysis and static code analysis, and understanding of code written by others becomes a challenge.

- 100% code coverage becomes a very very expensive achievement only affordable in NASA (and I'm not sure because of recent cost cuttings)

- Extensive real unit testing becomes a myth because of the tons of tests needed and how hard is mocking a system with thousands of classes.

 

Philopator Ptolemy replied on Wed, 2011/04/20 - 10:03am

I saw those tweets and also blogged on them. Essentially i was saying the same thing about ROI of extra testing and fixing minor bugs.

 

Robert C. Martin replied on Wed, 2011/04/20 - 12:35pm

Yes, it's quite true, that you can get 100% code coverage without asserting anything. And if a manager were to demand 100% code coverage that's likely what he'd get. But that's not what I'm talking about. I'm not talking about a manager making demands. I'm not talking about some corporate goal of 100% code coverage. None of that matters at all. What I'm talking about is a _personal_ discipline of testing every line of code you write. Over and out. Bar none. No excuses. Every bloody line. If you are the type of person who would write tests without asserts to convince yourself that you'd achieved 100% code coverage, then I'm sorry for you. I imagine you also loosen the springs in your bathroom scale to convince yourself you've lost weight. I'm not talking about the $200M project with 5M LOC. I'm talking about _your_ part of that project. Did _you_ test every single line of code you wrote; and can you prove it? If not, why not? What's your reason? Doesn't that code have to work? If it doesn't have to work, why did you write it?

Nicolas Frankel replied on Thu, 2011/04/21 - 6:56am in response to: Robert C. Martin

Thanks for your answer.

In order for this debate to be fruitful, it would also be nice to hear your position on all the points I raised.

Matt Avery replied on Thu, 2011/04/21 - 9:48am in response to: Nicolas Frankel

Nicolas, the goal I am encouraging for everyone is to increase test coverage in your own code continuously.  In many (most?) cases I am starting with someone elses code and it does not have unit tests, so I begin adding them along the way.  I am finding that even though I have much higher test coverage on the code I write than the other developers on my team, my productivity is far higher than average.  A manager could see that and take it to the logical extreme... "hmm, higher coverage appears to correlate with higher productivity, so I'll mandate 100% code coverage to send my team's productivity through the roof and be a hero".

 I think we are declining to engage the points you raised because they are, in truth, beyond our control.  If my manager mandates some code coverage goal, so what?  My internal compass is telling me to continue moving toward 100% code coverage anyway.  If other developers on the team are not writing unit tests, I'm not helping anyone by pointing out that we all greed to some code coverage goal at the beginning of the project and beating them over the head with it.  The only way I can help them is to jump over and offer to write some tests with them, pair program, brain-storm, mentor, etc.  I truly do not want to be too harsh here, but most of the other points appear to simply be excuses from the standpoint of a person who once considered all of those same arguments and is now thoroughly convinced that 100% code coverage is at least a good starting point for unit test coverage.

I need to say one last thing here... I *love* programming!  I prayed to the Lord to make my "kung-fu" awesome.  He answered my prayer by giving me a project where my manager stipulated 100% code coverage on a project that had ~80K instructions and perhaps 4% code coverage.  It changed the way I program.  Later I read Bob Martin's Clean Code book and now I'm writing my own book about increasing test coverage which is essentially Avery's version of Clean Code with some testing techniques from hard-won testing experience. 

Shoaib Almas replied on Sat, 2012/08/25 - 5:52am

100% in code coverage is easy. 100% in test is impossible, there’s some thesis on the subject. Some methods allow to limits the risks, like formal methods, which are often expensive and so not always relevant.
To conclude I think that code coverage is just a measure that means nothing, As R.Martin say, it also depends on the quality of the test. Even more I really prefer a 15% code coverage, only on critical parts of the code, with the asserts and the relevant methods input…

Java Forum

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.