Abby Fichtner writes The Hacker Chick Blog, co-organizes Boston's 2500 member Lean Startup Circle, and was Microsoft's Evangelist for Startups where she helped hundreds of startups as they build out the next generation of software. Her background is a mixture of developing bleeding-edge technology for startups and coaching teams on how to improve how they develop software. She's extremely passionate about building communities where innovation thrives and in helping others to push the edge on what’s possible because she believes that each & every one of us is capable of changing the world. Abby is a DZone MVB and is not an employee of DZone and has posted 25 posts at DZone. You can read more from them at their website. View Full User Profile

My Unit Tests Are Purer Than Yours

07.02.2008
| 6035 views |
  • submit to reddit
There is a hot debate on my project about whether or not our JUnit tests are pure unit tests. What the heck does that mean, pure unit tests? Our tests are JUnit tests. Doesn't that, by definition, make them unit tests?


Actually, no. Unit testing does actually refer to a very specific type of test - one whose sole purpose in life is to:

isolate an individual unit and validate its correctness.

In other words, a unit test should only test an individual unit (a class or a method). If executing a test crosses that class's boundaries to say, access the database or call a method in another class, you're no longer doing unit testing, you're doing integration testing.

Okay, that's really nice and now I can feel like a big, important software engineer for knowing the true definition of unit test. But, why do I care?

That's a good question. The reality is that most "unit tests" - the ones on my project included - do tend to go a little beyond just an individual class. We've got a J2EE application with business objects that are hooked up to the database via hibernate DAOs. And, of course, our classes have relations to other classes. So, when you call one of our class's methods, those methods quite frequently access the database and/or execute calls to other classes. Sometimes we even purposefully test scenarios that span several classes. For example, we're developing rules to determine how to process health insurance claims. These rules exist within a complete rules framework. So, we want to test not only that the rule is right, but that it operates as expected within the framework it will be used within.

Okay... so we're clearly getting beyond the strict definition of unit tests here. But again, I'll ask, who cares?

Let's stick with our rules example. The rules are, for the most, part very simple. "If A is true, make sure B is also true", "If a claim is for Dental, make sure the patient has dental coverage." Where the real complexity comes in is that there are over 700 of these rules, they all have to be run in a very specific order, and only under certain conditions, yadda yadda. You get the idea.

And so, if we heed the saying "a good test is one that finds a defect", then where are we going to get the most bang for our buck? Testing very simple rules in isolation? Or testing this extremely complex interaction of rules within the framework. Obviously the latter is exponentially more likely to turn up issues and thus make for the better test!

Okay, great - so I go off and run our unit/integration tests and what should happen but I get 203 failures(!@). It's a known issue but nobody has time to track down the problems. So, now not only is this a problem because we have no way of knowing if new code breaks the existing rules (if the test fails - what does that mean? Is it something we broke or was it already broken?) but for similar reasons we have no way to validate if the new rules we're writing work. And nobody really has any idea where in those 700+ classes the problems fall, so we don't know how to fix it.

Which of course is the age old problem with software integration and the very reason, actually, that it was determined that we need to do things like unit tests to test individual modules separately in order to validate that they work on their own before we try to put them all together.

The reality is that we need both. We need unit tests to ensure each unit works in isolation, and then we need higher level functional or integration tests to ensure the overall system works as expected. Do we really need our unit tests to be 100% pure? I think probably not. Just how pure do they need to be? Well, as with most things, my answer will be to try it out iteratively. Obviously, the current solution ain't working, so let's try to isolate them a little further and see if we can find those 203 problems. If it's still impossible, then make 'em a little more pure. Rinse, lather, repeat.

Stay tuned and I'll cover some of the issues that come up around "purifying" our unit tests - such as how to prevent our tests from accessing the database and what to do about testing private methods.

by abby fichtner,
reposted from: The Hacker Chick Blog
Published at DZone with permission of Abby Fichtner, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Florian Kammermann replied on Wed, 2008/07/02 - 3:29am

Very interesting, i pose me exactly the same question the moment. We must also do unit-tests for our business-logic and the point is, we need goot testdata.

In reality the data lies in a database, you have now two possibilities:

- write a test-data-generator, which is extrawork and have no value for production

- use dbunit in conjunction with the db, here you have dependency to the db, what is ugly. But you can use an in-memory-db like hsqlb.

 

We choosen now the second way an use dbunit-hsqldb. A junit-purist would cry, when he see this. But I hate ideology and I think only writing pure junit-tests is pure ideology and don't meets real-life requirenments. 

 It's like you told, you must find out how pure you need youre junit-tests. 

 What is also very important for me, is that a new developer can check-out the code and run the junit-test without configuration of the environment (db, jms-queues, jndi etc.). This I really don't like if junit-tests depend on such environments. In my opinion this should be tested in another scope.

 

Ricky Clarkson replied on Wed, 2008/07/02 - 3:36am

I think there's something wrong with this notion of purity.  It seems to be about having 0 dependencies.  Well, as long as your code runs, it has dependencies on the language and runtime it uses, and probably a few basic libraries at least. 

Suppose I write some code that uses java.util.ArrayList - that's probably deemed ok because java.util is standard, so my test is still pure.  Then I change it to use my.util.ArrayList and now somehow my test is no longer pure.  The division made here between pure and impure is quite arbitrary.  However, I think you can change the division a little and have something useful (though I wouldn't call it pure):

If X uses Y, Y should have its own test.  There should be no dependency loops.

That's short, possibly too short, so I'll explain a little.  In my above case you can individually test my.util.ArrayList, and then assuming your tests are good enough you can test the code that uses my.util.ArrayList without having to worry about my.util.ArrayList breaking.  Well, technically you still have that worry because unit tests are rarely proofs, but hey, you're using unit tests instead of static typing.

I think you should only call something pure if it exposes no side effects.  E.g., an immutable (or lazily initialised) object, or a method that always returns the same value given the same inputs.

Brian Sayatovic replied on Wed, 2008/07/02 - 9:27am in response to: Ricky Clarkson

The concerns about dependencies isn't the quantity, but the quality.

 Dpeending on your language and runtime is pretty much as given.  But what you don't want to depend on are "akward collaborators": things that are slow, volatile or shared.  These types of dependencies are also often unavailable in the runtime context of unit tests (e.g. test runner in your IDE, step in a build process) lest you deploy and run in your tests in your fully-configured container.

A database, for example, is generally all three of those things.  It is slow (compared to in-memory data and code), it is volatile (anyone can change the data your test is depending upon) and it is shared (one test could affect another).  There are ways to try and mediate these problems with depending on a database (e.g. in-memory database, separate instance per test, test sets up its own data); however, the point goes further...

When you have slow dependencies, your tests are slow.  Tests should be something that you can run frequently while you are coding, not when you think you're done.  Well, this is part of the spirit of TDD, anyways.  As your tests become too slow, it stops being an active part of your coding mental stream but instead an annoying step that breaks your train of thought.  If you think that's not important, think again.

If your dependencies are shared (e.g. singleton, database, etc.) then there is a high risk that the dependency might be affected by one test, changing the initial state for the next test.  Tests shoudl be independent, but since most testing frameworks run all tests in a single runtime, shared dependencies breaks that independence.  Worst case scenario: hard-to-track-down false negatives.

If your dependencies are volatile (e.g. pre-filled database, time) then your tests may also give false negatives.  A volatile dependency is subject to change beyond the control of your test (and even outside of its lifecycle).

Ricky Clarkson replied on Wed, 2008/07/02 - 9:48am in response to: Brian Sayatovic

So that's not purity but just speed, which doesn't seem to be what the original poster was talking about.  Keeping unit tests fast is quite valid.  Some of my tests involve automatically generated data; so I can just change a single number to make the tests run overnight, or in a minute.

For your point about tests interfering with each other, the easy answer is for all data to be either constructed by the test or immutable.  If that turns out to be impractical, *the test* should verify its assumptions about the outside world.

Mark Thornton replied on Wed, 2008/07/02 - 10:53am

Many of my tests involve quite complex data which I still have no satisfactory way of providing in a test environment.

1. Generate in the test itself. A DSL approach helps, but it is still very tedious setting up the test data by hand. Only practical for the very smallest tests (which usually don't get near the interesting behaviour).

2. Using a database. Heavy weight and hard to ensure that it is available and not changed by others.

3. Written in a data file packaged with the test. This leads to the format question. Serialization is fragile. Formats which aren't fragile need quite a bit of supporting code which is itself subject to breakage.

Ricky Clarkson replied on Wed, 2008/07/02 - 12:34pm

Mark,

I generate arbitrary data for testing just by having a static method in each relevant class - class Foo { public static Iterable[Foo] randomFoos(Random random){} }.  I find it quite easy to build this up so that I can have random quite complex data structures.  If repeatability is an issue control the seed of your Random.

Mark Thornton replied on Wed, 2008/07/02 - 12:53pm

I forgot that one --- I do use random data for some purposes. It is less suitable for regression testing where you want to test the specific case that failed, or where the failing case is very rare in random data.

Ricky Clarkson replied on Wed, 2008/07/02 - 1:21pm

You can tune random data to make it more likely to produce interesting results.  E.g., I'd make sure to include 16777216 in random floats, the first float for which x+1==x.

Anyway, take a look at Reductio, reductiotest.org for more disciplined approaches to testing based on randomised data.

Jeroen Wenting replied on Wed, 2008/07/09 - 12:27am in response to: Florian Kammermann

[quote=haschibaschi]

Very interesting, i pose me exactly the same question the moment. We must also do unit-tests for our business-logic and the point is, we need goot testdata.

In reality the data lies in a database, you have now two possibilities:

- write a test-data-generator, which is extrawork and have no value for production

- use dbunit in conjunction with the db, here you have dependency to the db, what is ugly. But you can use an in-memory-db like hsqlb.

[/quote]

it gets worse. What's to guarantee your test data generator will work correctly?
It'll need its own series of tests to ensure that...

I've seen the results of flawed test data generators firsthand. In that scenario no actual data could be made available due to privacy considerations as well as practical reasons, so ALL tests were of necessity performed with data from a generator (which replaced a complete telephone exchange).
Problem was, it generated data that while correct according to the specs was incorrect according to the actual data supplied by the actual telephone exchange. The application to handle the data passed the tests with flying colours, but of course failed completely as soon as it was hooked up to some real data providers, causing millions of dollars worth of phonecalls to be rejected for billing because the files from the exchange were invalid.
And that's not the only time I've seen such results (both from failures in the generator where it would provide bad data that noone noticed and from generators that provided technically correct data that didn't match the real world).

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.