Khoo has posted 3 posts at DZone. View Full User Profile

Deliver a Good Return of Investment Unit Testing Code

07.21.2008
| 6258 views |
  • submit to reddit
It has always been my belief that comprehensive integration/functional testing of code is far far more important/productive than to blindly try to achieve 100% unit testing code coverage. I believe:
a) Good integration/functional tested code will always deliver more test coverage then unit testing code.
b) Functional/Integration testing code allows us to capture our programming errors as early as possible, and avoid “big bang” integration errors at the end.

Thus, I always have “healthy and interesting” discussion with developers where they always believe:

  1. Achieve high percentage of unit testing code coverage is very important.
  2. Every class delivered must be “unit testable”, all dependency classes/resources should be “mocked”, even though it sometimes is very difficult  and time consuming to setup such mock objects.
  3. If our code is not testable, that must be something wrong with our design. We have to change our code to make it testable. Thus making our class final, or our class not implementing an interface, or using static methods, or any things that make our class not "unit testing friendly" are bad.

Don't get me wrong, I am not 100% against the points above, and I do understand the importance of unit testing. However, I will not go to extremes spending extra effort to achieve 100% unit test code coverage. Thus, as Software Development Manager, it's my belief and I  encourage my team to deliver a good Return of Investment (ROI) testing code. As Java resources are expansive and we need to take of maintaining non productive unit testing code we deliver into consideration.

 So, what is considered a good ROI code:

  • It's should be “black box” test, where developers pass certains parameters to a method, execute the method and verify the return value against the expected result, we should not care how the method is executed, and what underlying resources being called.
  • The testing code should be easily promoted to functional/integration testing code, why reinvent the wheel?
  • The testing code should design around your use cases, and right back to your application's features. Thus, using a HR application as an example, to unit test your annual leave application is always approved by your manager would produce good ROI value. But to unit test simple getter or setter of your data object, or to mock a DAO on returning an employee object, or make utilities to read properties files or parsing a mock XML file does not produce good ROI value.

To explain this further, assuming I have a business delegate class, called OrderManager, which has a dependency on InventoryManager, AccountManager, and ShipmentManager.There is a method to check if a order should be processed or not, as code shown below: 

Public class OrderManager {
     public boolean proceedOrder(OrderVO anOrder) {
             return ( inventoryManager.hasStock(anOrder) &&
                        accountManager.customerGoodCreadit(anOrder.getCustomer()) &&
                        shipmentManager.shipmentDateOK(anOder.getPreferShipmentDate()
                      );

}

To unit test proceedOrder() method using Mock:

  1. First, create InventoryManager, AccountManager, and ShipmentManager mocks
  2. Instruct our mock framework on mock methods called, and tell the mock the expected sequence of the call, and set return for each mock call.
  3. And finally, we test return value with our expected result, and ensure mock methods being called.

Does the above code give us good ROI? 

  • It's breaking the black box testing approach, we are telling our mock framework step-by-step what mock methods will be call, and specifically ask them to return the value we set, and test method proceedOrder() returns a value with our expected result, when we already know what it will return in the first hand.
  • What happens if we want to change the implementation, like changing the calling sequence, or introduce new business logic, such as check if customer is VIP customer, who might have priority on getting his/her preferred shipment slot, do we have to change our testing code?
  • Even if our unit code coverage report shows that we have cover this method 100%, does that give us any value at all?

As an example, A good ROI testing code for the above use case:
1. Set a InventoryManager product (say product “A”) quantities to a specific amount (say 50)

2. Test the proceedOrder() method by placing product A with various quantities
3. Verify the result against expectedResult

So, to conclude, I believe that if unit testing code does not bring any good ROI, do spend more time on delivering functional/integration testing code, and delivering good "ROI" usable code coverage

Apply my points to Automobile industries, we do not hear of engineers that create mock car tires, or create a mock for road surface to test a car braking system, he/she will actually “integrate” the braking system with a set of tires (made of different rubbers), and conduct braking test on various surface (maybe on a rolling board with different surfaces), right?

Do share your views on this.

Published at DZone with permission of its author, Khoo Chen Shiang.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Tags:

Comments

Alex Kizub replied on Mon, 2008/07/21 - 12:27pm

In your Automobile industries example Unit test would be does break slow spinning or not, is it compatible with axe, how much power it requires. NOBODY install breaking system to the car and then start to test it.

I cannot imagine any system without Unit tests. You are talknig about ROI when Unit test is about to be alive. You must be alive before you even start to think what are you living for.

Serge Bureau replied on Mon, 2008/07/21 - 1:44pm

I am in total agreement with you Khoo,

 

Unit testing at all cost looks more like a religion than  a technique.

It is a valuable tool, but only when used sensibly. 

Nicholas Pellow replied on Mon, 2008/07/21 - 6:53pm

To " blindly try to achieve" anything is always a bad idea.

 Also, the implied belief contained in your assertion:

"Good integration/functional tested code will always deliver more test coverage then unit testing code."

that code coverage == test quality is fundamentally flawed. It is extremely easy to cover large swathes of a code base with a few functional/integration tests. Code coverage has limited value in such instances because you can no longer be certain of spotting parts of the code base which are untested - very different to uncovered.  

 

Khoo Chen Shiang replied on Mon, 2008/07/21 - 8:20pm

Hi All

Thanks for comments

Alex, yep, agree with u 100%, I will not put myself in a car to test a breaking system that didn't pass some fundamental tests, I still enjoy my life now...hahahaha...

In my example, I do stated that the engineer do integrate the breaking system with some real tires and tested on real surface on rolling board, all this been carry out without test on real car..else our Auto industries will out of business very soon...haha And for me, this is more like functional testing, and I agree with u, the engineer do needs to perform some soft of “unit testing” on the breaking system, (like to test the wiring, the air pressure, and etc) prior exec any functional testing, and hence in my article, I didn't discount the importance of unit testing..

npellow, yep agree, unit and functional testing are equally important.

 

Cheers

Roland Carlsson replied on Tue, 2008/07/22 - 3:34am

Testing is an insurance. It might not cover your loss but it covers a lot of situations. Being ignorant and think that your caulculation of the "right" insurancefee is the correct for everyone is not sensible.

 As with every insurance you have to find out how much damage you can handle without going out of business and how large your insurancefee you can afford and then finding a resonable compromize. 

 Some orgainisations can't afford any damage and therefore have to pay a very large fee (100% testcoverage). Some orgainizations can't afford the fee and therefore have to handle the damages (when and if the occur). It's plain economics and not programming.

 But only using unit-test for finding error is a waste in my opinion. Refactoring is the larger benefit in my opinion. Finding errors is second or perhaps third priority. 

Artur Biesiadowski replied on Tue, 2008/07/22 - 4:15am in response to: Roland Carlsson

[quote=rc83302]

Some orgainisations can't afford any damage and therefore have to pay a very large fee (100% testcoverage). Some orgainizations can't afford the fee and therefore have to handle the damages (when and if the occur). It's plain economics and not programming.

[/quote]

There are also companies which have computed the costs of full covarage and realized that occasional failures are cheaper. For every application in the world, there will be a cutoff point where extra testing will cost more than expected failures. There is no such thing as 100% test coverage. You can maybe do unit tests executing 100% of code paths, but this has nothing to do with testing full application.

Now, only question is where you put the cut point. It will be for sure different for database frontend for intranet in small store, different for investment banking front office applications and different for life support system in hospital. But there is only so much testing you can do for software - at some point, you have to start caring about cosmic rays and mobile phone interference as being higher risk than missing unit test on 1+1=2.

On top of all that, I still have to see any reasonable way to test heavily multithreaded code. At this moment, multi-eyes code review is only way I see.

Jay Ceron replied on Tue, 2008/07/22 - 7:06am

While I agree that setting up property files and XML files is not good ROI, I disagree that mock objects are not.  If you design a service oriented architecture swapping mock and live object is simple.  And so what if these return data that we've previously defined for non-calculating methods?  That gives more control over testing.  The only thing to note is that mock objects should be completely configurable, with all members public.

Roland Carlsson replied on Tue, 2008/07/22 - 5:29pm in response to: Artur Biesiadowski

There are also companies which have computed the costs of full covarage and realized that occasional failures are cheaper

 Some compaines can afford failure, some might not. Saying that because some compainies can do doesn't say anything about company X needs. Of course the amount of money in the world sets a few limits on how much effort that can be spent on insurances.

 

There is no such thing as 100% test coverage. You can maybe do unit tests executing 100% of code paths, but this has nothing to do with testing full application.

You can only test for what you know something about. 100% code coverage is about the contract the methods should fullfill. Testing that contract should be strait forward in most curcumstances.  If the contract doesn't talk about cosmic rays then don't test for it. Perhaps common sense about what the contract should contain is the problem and not unit test?

 Now, only question is where you put the cut point.

This is the intresting point. And noone on this earth can give you an answer for your situation. Writing the tests before the acctuall code is to me a good guideline about what I should solve. Trying to keep the code testable to be able to introduce tests without large refactorings seems like a sensible idea to me. If I write code that I belive is subject to change I think that being able to change the tests is a very good tool to find what needs to be changed in the code. But I could be coloured by the fact that since I started to use unit-tests I have been working with maintence on an existing code base that I had no experiance with and that are poorly documented.Changing in code without knowing if there are unexpected conseqenses could perhaps make anyone a fan of the idea of 100% test coverage. :-)

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.