Nicolas Frankel is an IT consultant with 10 years experience in Java / JEE environments. He likes his job so much he writes technical articles on his blog and reviews technical books in his spare time. He also tries to find other geeks like him in universities, as a part-time lecturer. Nicolas is a DZone MVB and is not an employee of DZone and has posted 231 posts at DZone. You can read more from them at their website. View Full User Profile

Cons of Custom Assertion Matchers

04.12.2011
| 3906 views |
  • submit to reddit

Ten years ago, the only tests we knew of were users acceptance tests. The last decade saw a gigantic leap forward: it brought unit testing. Unit testing was made popular with JUnit. In turn, TestNG added annotations to the test classes, making them ever easier. Then, EasyMock provided the means to mock our class dependencies in a test context while Mockito streamlined the process of doing so. One could think everything has been said and done for unit testing and we should move forward to a more worthy goal of our workforce.

I think nothing could be further from the truth. I do think that the next step will be making tests more maintainable. During my career, I had to maintain software that had no unit tests and that was bad. However, when I had to work on software with tests I had difficulties understanding, I cannot say for sure if it was better. Taking time making sure there's no regression nor side-effect is one thing, taking time to debug test intricacies is another, and hardly expected by the management at that. One particular obstacle is in the assertion statement. I understand it may seem strange at first. How in the world could the following assertion be undecipherable?

Assert.assertFalse(myList.isEmpty());

Well, I infer that some people must have had enough of always writing the same assertions and provided some common API to test them. FEST and Hamcrest are example of such testing commodities that let us boost our productivity through the use of a DSL where the domain is assertion. Left it at that, I couldn't agree more with their use and even promote it in my teams. However, Hamcrest also provides an API to create our own custom matcher: while it's true that creating such matcher may be a help at first glance, I'll try to raise some points against it.

First, what about maintainability? If a test fails after I made updates, is it because of my code, the matcher or a combination of both? I'll have to either ask the original coder, but most of the times, I'll be on my own to decide such things. This will likely involve sweat, toil and much debugging. Taken individually, the cost is not much, but it's exponentially proportional to the number of matchers.

Then, in order to decrease the probability of a bug in a matcher, it will have to be tested on its own! Will their respective tests need assertion matchers? It seems like it will be turtles down to the bottom. Again, costs are bound to rise.

Moreover, we are taught in school that code duplication is bad (many tools provide this metric) and should be avoided at all costs. I can only support the first axiom, just as I have to condemn the second, and more particularly the 'at all costs' part. This isn't true! Duplication is the first level of reuse, and I hope the previous arguments made my point regarding the cost of writing and maintaining custom matchers. For example, if we duplicate two lines three times, that's not enough in my eyes to justify the added cost of a matcher!

Finally, it may seem strange but testing is not the goal of software, only a mean to an end (something I will have to write about in a next post). We're not tasked to make our CV shine brighter, nor to partake in the highest code coverage challenge. We're just paid to deliver software to our customers, while respecting the holy trinity of costs, delay and quality. If writing custom matchers can help us reach this goal, fine but whether it's effectively the case is very contextual.

At the present, it seems advantageous to me to write custom matchers under the following circumstances:

  • There's a real advantage in doing so. Search how many duplicated lines exist and whether it's cost effective to factorize them in a matcher.
  • It won't impact delivery. What happens if we're in the middle of writing the best matcher ever and we're ordered to ship the product? The customer doesn't care about matchers.
  • Matchers shouldn't ever be owned by a single developer but by the whole team. This prevents, or at least decreases the risk of having to debug a matcher to understand what its goal was.

In conclusion, I hope this article made you think about the whole custom matchers stuff. Next time you are about to create a matcher, think about it: it's perhaps a good thing to do, but it's not necessarily the case.

From http://blog.frankel.ch/cons-of-custom-assertion-matchers

Published at DZone with permission of Nicolas Frankel, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Tags:

Comments

Shoaib Almas replied on Sat, 2012/08/25 - 5:53am

Sorry I get what you’re getting at but: “It won’t impact delivery. What happens if we’re in the middle of writing the best matcher ever and we’re ordered to ship the product? The customer doesn’t care about matchers.” isn’t a valid argument – customers care about the quality of the product, that quality is in part ensured by well tested code, ergo if writing the matcher improves your test quality it increases your product quality and the customer is happy.

Java Forum

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.