Good Unit Test CheckList
Here are a few items I try to respect every time I write unit tests.
- My test class is testing one and only one class
- My methods are testing one and only one method at a time
- My variables and methods names are explicit
- My test cases are easy to read by human
- My tests are also testing expected exception with @Test(expected=MyException.class)
- My tests don’t need access to database
- My tests don’t need access to network resources
- My tests respect the usual clean code standards (lenght of lines, cyclomatic complexity,…)
- My tests control side effects, limit values (max, min) and null variables (even if it throws an exception)
- My tests can be run any time on any place without needing configuration
- My tests are concrete (ex. dates are hardwired, not computed every time, strings too…)
- My tests use mock to simulate/stub complex class structure or methods
From http://jbrieu.info/leblog/2011/good-unit-test-checklist/
(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)
Comments
Tom Gardner replied on Sat, 2011/08/20 - 3:28am
Most of your points are good and valuable, but...
It is very rare to be able to only test one class at a time: in the vast majority of the cases you are testing the integration of your class with one or more libraries. Just look and see whether your unit-under-test has any import statements! If you think libraries are error-free, then I have a bridge I'd like to sell you :)
Testing only one method at a time is, of itself, not a benefit. I have frequently seen this principle used to justify reducing the quality of the unit-under-test. For example, unnecessarily making getter/setter methods public solely so they can be unit tested.
Liam Knox replied on Sun, 2011/08/21 - 7:01am
Configuration is also contentious. Spring has great support for this which often is easier to use and covers more that trying to use pure programmatic idioms and mocks.
And probably the most contentious of all is the one 2 one method. I doubt you will ever find this is followed 100% in any large team. I don't think its a good rule. If you express tests that cover business scenarios that exercise many classes/methods these often have more meaning and more proveable coverage and correctness.
Chad Hahn replied on Sun, 2011/08/21 - 7:26am
in response to:
Tom Gardner
This is really where DI comes into play. When you inject the needed dependencies into a class, you're free to use mocks or stubs to NOT test the libraries. That's what unit testing is, testing things in isolation. If you want to test the interaction between other classes and libraries, you'd want to do integration testing.
It's somewhat common to increase scope in order to satisfy testing, but you would never go from private to public. You'd go from private to friendly. However, if you're doing it correctly, your private methods should be tested in the course of testing the public interface, otherwise the private methods aren't being used.
Tom Gardner replied on Sun, 2011/08/21 - 11:45am
in response to:
Liam Knox
"If you express tests that cover business scenarios that exercise many classes/methods these often have more meaning and more proveable coverage and correctness."
Precisely.
As I wish more people realised, there has to be a stated reason for doing a test. If there isn't, then the testing is just a fetish that is being used to ward off Unspecified Bad Things
Tom Gardner replied on Sun, 2011/08/21 - 11:55am
in response to:
Chad Hahn
"This is really where DI comes into play. When you inject the needed dependencies into a class, you're free to use mocks or stubs to NOT test the libraries. That's what unit testing is, testing things in isolation. If you want to test the interaction between other classes and libraries, you'd want to do integration testing."
- Do you really use DI to mock the standard library? Or any other non-standard but widely-used library? If so, what benefit do you get from it?
- Who cares whether your class/method works with other classes/methods written by you? Your class/method and the library class/method may each be correct in isolation but faulty when used together. Or it is quite possible that your mock exhibits significantly different behaviour to the "real" library.
Yes, that does mean that either almost all non-trivial testing is integration testing, or that "unit" has to be defined to be larger than a single class.Tom Gardner replied on Sun, 2011/08/21 - 12:03pm
in response to:
Chad Hahn
"It's somewhat common to increase scope in order to satisfy testing, but you would never go from private to public. You'd go from private to friendly. However, if you're doing it correctly, your private methods should be tested in the course of testing the public interface, otherwise the private methods aren't being used."
I've seen it done, repeatedly. Sorry, I can't fix the world by removing all bad developers.
Black boxes are better than transparent boxes because you know you don't have to understand how they work.
Private methods are not part of the public interface/signature that has to be maintained/preserved as the software evolves, whereas public and friendly methods (and classes) are. Hence making something unnecessarily non-private reduces the software quality by:
Chad Hahn replied on Sun, 2011/08/21 - 1:59pm
in response to:
Tom Gardner
I think it would be helpful to get yourself a copy of JUnit in Action or a similar book and read it carefully. It'll give you a better understanding of what a unit test is, why they are important, and why testing in isolation is a goal to strive for.
Heck, grab yourself a copy of Pragmatic Unit Testing by Hunt and Thomas from 2003. They had it correct back then when they wrote "A unit test is a piece of code written by a developer that exercises a very small, specific area of functionality of the code being tested." They go on to explain mocking, independent tests (isolation), and some pitfalls around improper testing.
This isn't rocket surgery, but, among the teams I've worked with, it's surprising how immature some of them have been when it comes to proper testing. I think that's because people refuse to acknowledge the body of work already done in this area. We really shouldn't have to be spending time trying to redefine "unit" as "larger than a single class" testing.
Tom Gardner replied on Mon, 2011/08/22 - 4:52pm
You have ignored the point I was making and "answered" (and I use that word loosely) with a diatribe.
I've been using JUnit (and TestNG) for the past decade and have introduced its use to more than one company, so I'm well aware of its advantages. And also of the limitations to unit testing.
Having developed systems where
I am well aware that unit testing is limited.
There's an old engineering aphorism: "you can't inspect or test quality into a product". I suggest you ponder the wisdom in that truism
Chad Hahn replied on Mon, 2011/08/22 - 10:14pm
in response to:
Tom Gardner
I was trying to point out that you're basing your argument on a principle that is generally considered wrong. You said "It is very rare to be able to only test one class at a time." That statement goes against what is commonly accepted as unit testing. If you want to complain about the OP, you need to understand the terms he's using. Nothing more, nothing less.
Tom Gardner replied on Tue, 2011/08/23 - 2:07am
I suggest you read what I actually wrote in its entirety, and not base your point on a partial quote. I'll try to spell it out.
How many of your classes don't have a single import statement? If they have one or more import statements then you aren't testing a single class in isolation: you are testing one class and its integration with the classes named in the import statements. It really is that simple.
Yes, that does mean that the whole concept of "test one class on its own" is so simplistic as to be misleading.
It doesn't mean that "unit testing" is a misnomer, merely that the "unit" is almost always larger than a single class, and should be recognised as such.
Chad Hahn replied on Tue, 2011/08/23 - 7:03am
in response to:
Tom Gardner
Andrew Spencer replied on Tue, 2011/08/23 - 7:50am
Tom,a decade ago it would have been reasonable to argue that you can't test a class in isolation. But these days, at least in my field, developers are expected to know how avoid hardwiring dependencies.
I totally agree that testing a class in isolation isn't enough - even in the non critical systems I develop - and that you also need to test how classes interact. I call that integration testing, though the term's too broad and I'd be happy to accept a more precise taxonomy.
There's a danger here of confusion between a discussion about what kinds testing one should do, with a discussion about what one means by a term such as unit testing. This has led previous commenters to talk at cross purposes. Stating that unit testing means testing only one class at a time, is not the same as arguing that integration tests are superfluous. Are you certain you're not tilting at windmills?
Tom Gardner replied on Thu, 2011/08/25 - 4:24am
Thank you, Andrew, for a thoughtful post. My principal argument is with those people that make statements that are so wideranging that they become wrong - even though the statements were inspired by valid principles. (That's why the GoF Design Patterns book containd the "do not use if..." sections explaining when a design pattern shouldn't be used)
Hardwiring dependencies is an interesting topic, and I suspect our views are actually closely aligned in this area. There are many cases in which removing a dependency that has been lazily, unnecessarily and harmfully introduced is clearly The Right Thing To Do. And, unfortunately I've had to spend too much of my professional lifeteaching and mentoring people in such techniques.
However, I'm sure you'd agree that it is impractical and not beneficial to remove all dependencies. Obvious examples of the former: the standard String, Set, Integer classes. Obvious examples of the latter would be trying to remove a dependency on, say, ThreadedExecutor or BoundedBuffer in which the correctness of your class relies on the properties of those specific library classes and not their relatives.
I've also seen people take generally useful rules of thumb and relentlessly apply them without understanding when they aren't beneficial. Such religious cargo-cult application of techniques is intensely annoying.
As you imply, the boundary between unit testing and integration testing is a gray area. Some poster's comments don't indicate that they realise that.
As for "tilting at windmills"? Certainly not :) Because, unlike Don Quixote, I'm right.
Tom Gardner replied on Thu, 2011/08/25 - 4:20am
Chris, your points, while they have some academic merit, are so wideranging as to be invalid in many real-world contexts.
For example, if my class's correctness is predicated on its using a DirectExecutor rather than a ThreadedExecutor, then it would simply be bad engineering to allow the possibility of accepting a ThreadedExecutor in my classes' constructor. And attempting to test my class without using the DirectExecutor would be merely an exercise in intellectual onanism.
I agree that "import statements don't imply tight coupling". But they do imply coupling, and that is all that is needed to destroy the concept that is it possible/practical/beneficial to test such a class on its own.
As for refactoring: in me experience I've rarely seen it possible to refactor a single class onits own - except when I've been merely tweaking a business rule of some sort that is dropped into a framework. Almost always refactoring involves changing a set of highly cohesive classes.
There's good evidence that is the normal case for everybody else: the way refactoring editors/IDEs track inter-class dependencies. And the converse is true: if it wasn't the case then it would be easy and sufficient to use traditional editors in the old-fashioned way e.g. emacs + grep.
Chad Hahn replied on Thu, 2011/08/25 - 8:46am
Tom Gardner replied on Fri, 2011/08/26 - 2:00am
If by "Ted" you mean me then I am well aware that, within the limited subset of coding exemplified by JEE (or JAIN etc), unit testing is practical and beneficial. I alluded to that when I wrote "...except when I've been merely tweaking a business rule of some sort that is dropped into a framework". Maybe I shouhld have been more specific.
But that is a very limited set of programming envionments and conditions, as I alluded to when I mentioned some of concurrency classes that you would never use inside a JEE component. If your professional activity is limited to such environments I can understand your attitude to unit programming.
However, programming is much more wideranging and varied than "inside" JEE/JAIN/etc environments - obvious examples being real-time programming (where being 1ms too late with the "correct" answer means that answer is actually wrong) or high performance computing (where heat dissipation, the speed of light and buggy compilers/languages (e.g. C/C++) limit your ability to get an answer within a useful timeframe).
Hence your statements about unit testing are too wideranging. There's a lot you haven't experienced- don't presume that your limited experience is universally applicable.
Sorry for adding the emphasis, but it looks like you've been skimming what I wrote without thinking about it.
Chad Hahn replied on Thu, 2011/10/27 - 6:10pm
in response to:
Tom Gardner