As an Agile Coach, Miško is responsible for teaching his co-workers to maintain the highest level of automated testing culture, allowing frequent releases of applications with high quality. He is very involved in Open Source community and an author of several open source projects. Recently his interest in Test Driven Developement turned into http://TestabilityExplorer.org with which he hopes will change the testing culture of the open source community. Misko is a DZone MVB and is not an employee of DZone and has posted 38 posts at DZone. You can read more from them at their website. View Full User Profile

Cost of Testing

10.02.2009
| 4917 views |
  • submit to reddit

A lot of people have been asking me lately, what is the cost of testing, so I decided, that I will try to measure it, to dispel  the myth that testing takes twice as long.

For the last two weeks I have been keeping track of the amount of time I spent writing tests versus the time writing production code. The number surprised even me, but after I thought about it, it makes a lot of sense. The magic number is about 10% of time spent on writing tests. Now before, you think I am nuts, let me back it up with some real numbers from a personal project I have been working on.


Total Production Test Ratio
Commits 1,347 1,347 1,347  
LOC 14,709 8,711 5,988 40.78%
JavaScript LOC 10,077 6,819 3,258 32.33%
Ruby LOC 4,632 1,892 2,740 59.15%
Lines/Commit 10.92 6.47 4.45 40.78%
Hours(estimate) 1,200 1,080 120 10.00%
Hours/Commit 0.89 0.80 0.09  
Mins/Commit 53 48 5  

Commits refers to the number of commits I have made to the repository. LOC is lines of code which is broken down by language. The ratio shows the typical breakdown between the production and test code when you test drive and it is about half, give or take a language. It is interesting to note that on average I commit about 11 lines out of which 6.5 are production and 4.5 are test. Now, keep in mind this is average, a lot of commits are large where you add a lot of code, but then there are a lot of commits where you are tweaking stuff, so the average is quite low.

The number of hours spent on the project is my best estimate, as I have not kept track of these numbers. Also, the 10% breakdown comes from keeping track of my coding habits for the last two weeks of coding. But, these are my best guesses.

Now when I test drive, I start with writing a test which usually takes me few minutes (about 5 minutes) to write. The test represents my scenario. I then start implementing the code to make the scenario pass, and the implementation usually takes me a lot longer (about 50 minutes). The ratio is highly asymmetrical! Why does it take me so much less time to write the scenario than it does to write the implementation given that they are about the same length? Well look at a typical test and implementation:

Here is a typical test for a feature:

ArrayTest.prototype.testFilter = function() {
  var items = ["MIsKO", {name:"john"}, ["mary"], 1234];
  assertEquals(4, items.filter("").length);
  assertEquals(4, items.filter(undefined).length);

  assertEquals(1, items.filter('iSk').length);
  assertEquals("MIsKO", items.filter('isk')[0]);

  assertEquals(1, items.filter('ohn').length);
  assertEquals(items[1], items.filter('ohn')[0]);

  assertEquals(1, items.filter('ar').length);
  assertEquals(items[2], items.filter('ar')[0]);

  assertEquals(1, items.filter('34').length);
  assertEquals(1234, items.filter('34')[0]);

  assertEquals(0, items.filter("I don't exist").length);
};

ArrayTest.prototype.testShouldNotFilterOnSystemData = function() {
  assertEquals("", "".charAt(0)); // assumption
  var items = [{$name:"misko"}];
  assertEquals(0, items.filter("misko").length);
};

ArrayTest.prototype.testFilterOnSpecificProperty = function() {
  var items = [{ignore:"a", name:"a"}, {ignore:"a", name:"abc"}];
  assertEquals(2, items.filter({}).length);

  assertEquals(2, items.filter({name:'a'}).length);

  assertEquals(1, items.filter({name:'b'}).length);
  assertEquals("abc", items.filter({name:'b'})[0].name);
};

ArrayTest.prototype.testFilterOnFunction = function() {
  var items = [{name:"a"}, {name:"abc", done:true}];
  assertEquals(1, items.filter(function(i){return i.done;}).length);
};

ArrayTest.prototype.testFilterIsAndFunction = function() {
  var items = [{first:"misko", last:"hevery"},
               {first:"mike", last:"smith"}];

  assertEquals(2, items.filter({first:'', last:''}).length);
  assertEquals(1, items.filter({first:'', last:'hevery'}).length);
  assertEquals(0, items.filter({first:'mike', last:'hevery'}).length);
  assertEquals(1, items.filter({first:'misko', last:'hevery'}).length);
  assertEquals(items[0], items.filter({first:'misko', last:'hevery'})[0]);
};

ArrayTest.prototype.testFilterNot = function() {
  var items = ["misko", "mike"];

  assertEquals(1, items.filter('!isk').length);
  assertEquals(items[1], items.filter('!isk')[0]);
};

Now here is code which implements this scenario tests above:

Array.prototype.filter = function(expression) {
  var predicates = [];
  predicates.check = function(value) {
    for (var j = 0; j < predicates.length; j++) {
       if(!predicates[j](value)) {
         return false;
       }
     }
     return true;
   };
   var getter = Scope.getter;
   var search = function(obj, text){
     if (text.charAt(0) === '!') {
       return !search(obj, text.substr(1));
     }
     switch (typeof obj) {
     case "bolean":
     case "number":
     case "string":
       return ('' + obj).toLowerCase().indexOf(text) > -1;
    case "object":
      for ( var objKey in obj) {
        if (objKey.charAt(0) !== '$' && search(obj[objKey], text)) {
          return true;
        }
      }
      return false;
    case "array":
      for ( var i = 0; i < obj.length; i++) {
        if (search(obj[i], text)) {
          return true;
        }
      }
      return false;
    default:
      return false;
    }
  };
  switch (typeof expression) {
    case "bolean":
    case "number":
    case "string":
      expression = {$:expression};
    case "object":
      for (var key in expression) {
        if (key == '$') {
          (function(){
            var text = (''+expression[key]).toLowerCase();
            if (!text) return;
            predicates.push(function(value) {
              return search(value, text);
            });
          })();
        } else {
          (function(){
            var path = key;
            var text = (''+expression[key]).toLowerCase();
            if (!text) return;
            predicates.push(function(value) {
              return search(getter(value, path), text);
            });
          })();
        }
      }
      break;
    case "function":
      predicates.push(expression);
      break;
    default:
      return this;
  }
  var filtered = [];
  for ( var j = 0; j < this.length; j++) {
    var value = this[j];
    if (predicates.check(value)) {
      filtered.push(value);
    }
  }
  return filtered;
};

Now, I think that if you look at these two chunks of code, it is easy to see that even though they are about the same length, one is much harder t write. The reason, why tests take so little time to write is that they are linear in nature. No loops, ifs or interdependencies with other tests. Production code is a different story, I have to create complex ifs, loops and have to make sure that the implementation works not just for one test, but all test. This is why it takes you so much longer to write production than test code. In this particular case, I remember rewriting this function three times, before I got it to work as expected. :-)

So a naive answer is that writing test carries a 10% tax. But, we pay taxes in order to get something in return. Here is what I get for 10% which pays me back:

  • When I implement a feature I don’t have to start up the whole application and click several pages until I get to page to verify that a feature works. In this case it means that I don’t have to refreshing the browser, waiting for it to load a dataset and then typing some test data and manually asserting that I got what I expected. This is immediate payback in time saved!
  • Regression is almost nil.  Whenever you are adding new feature you are running the risk of breaking something other then what you are working on immediately (since you are not working on it you are not actively testing it). At least once a day I have a what the @#$% moment when a change suddenly breaks a test at the opposite end of the codebase which I did not expect, and I count my lucky stars. This is worth a lot of time spent when you discover that a feature you thought was working no longer is, and by this time you have forgotten how the feature is implemented.
  • Cognitive load is greatly reduced since I don’t have to keep all of the assumptions about the software in my head, this makes it really easy to switch tasks or to come back to a task after a meeting, good night sleep or a weekend.
  • I can refactor the code at will, keeping it from becoming stagnant, and hard to understand. This is a huge problem on large projects, where the code works, but it is really ugly and everyone is afraid to touch it. This is worth money tomorrow to keep you going.

These benefits translate to real value today as well as tomorrow. I write tests, because the additional benefits I get more than offset the additional cost of 10%.  Even if I don’t include the long term benefits, the value I get from test today are well worth it. I am faster in developing code with test. How much, well that depends on the complexity of the code. The more complex the thing you are trying to build is (more ifs/loops/dependencies) the greater the benefit of tests are.

So now you understand my puzzled look when people ask me how much slower/costlier the development with tests is.

From http://misko.hevery.com

Published at DZone with permission of Misko Hevery, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Tags:

Comments

Developer Dude replied on Fri, 2009/10/02 - 2:31pm

Tests are easier to write than the CUT, as you note, in part because they are linear, in part because you don't spend as much time designing them and obsessing over the architecture/etc.

That said, it is a sad comment on the state of our profession that in my experience (I have about 7 years s/w QA experience before I went over to the 'dark side') most devs don't write very good tests (if they write or run tests at all) or even decent tests. Most tests are barely 'smoke tests' if that, little testing of failure conditions, edge cases and so on. Still, such tests are better than nothing and starts devs down the road of writing tests (and retesting it when changing it). Hopefully after those tests save their bacon a few times they will become converts, but don't hold your breath. :(

What is the cost of *NOT* testing? Almost always higher than the cost of testing.

I have found that I almost always find at least one flaw with any given test scenario, whether it is a bug in the logic, or a flaw in the design, the ROI is almost always there. Then there is the refactoring and fixing of bugs, I often find bugs in my bug fixes via testing.

Plus, don't restrict yourself to just automated tests, try some new ad-hoc manual testing, if it seems useful and it is practical, add those test cases to the automated tests.

Jose Noheda replied on Sat, 2009/10/03 - 3:54am

You'll understand why developers don't code tests the moment you understand the bottom line of those maintenance projects that usually follow the main one.

Sumit Sengar replied on Sat, 2009/10/03 - 4:26am

I personally feel that, though writing junit test cases eat up fair amount of time during development, but It comes up as a big time saver for most of the cases. For eg, I was involved in creating a DAO layer for one of my web based application. Building the application and redeploying it and testing the functionality from the UI, takes a chunk of time and further repeated fixes in the DAO layer and redoing same cycle of build-deployments kinda irritates. creating a test class for the DAO and testing methods on it to check if the queries formed are correct and giving expected data, saves u from the pain of build/deployment. Any fixes in the dao do not require full build and once u r sure your DAO is behaving expectedly, do the full integration testing

Andrew McVeigh replied on Sat, 2009/10/03 - 2:52pm

are you testing all the paths through the function?  how are you measuring coverage in this case?

Jiri Kiml replied on Mon, 2009/10/05 - 7:45am

You wrote:

The magic number is about 10% of time spent on writing tests.

My experience is that there is also additional cost in maintenance of tests.

 

 

Arek Stryjski replied on Mon, 2009/10/05 - 10:34am

Very interesting post. If we (human) are able to express the same logic 5 times quicker in test code, then probably there is something wrong with the way we write implementation code. I don't mean with our code, but with languages we are using. Languages should reflect the way we are thinking not computers. They should also make expressing things easier, not more difficult... Definitely unit testing is a proof what something is wrong with the way we write software.

David Parks replied on Mon, 2009/10/05 - 8:54pm

This notion that people don't test because they are stupid and don't get that it is faster in the long run is so old that I'm starting to wonder if there is any truth to it.  I know that when I write code and don't write a test for it (which I do very, very often in the workplace but amusingly somewhat rarely in home projects) its because writing the tests will take a day, minimum.  As you pointed out, the code for tests is trivial to write and can be a savior in a way you never expected.  Most TDD people probably have experienced this at some point.  But, once you have those tests, you have to write a build script for them and integrate them into the build process.  You have to make sure that the tests run everywhere (for all platforms you support, you need to wire up the tests in your CI tool or whatever) and that they *succeed* everywhere.  You need to put them into source control in a way that isn't knotted with other code and is legible for the next person (give your functions useful names, clean up invalidated comments, etc).  You need to create a dummy set of test data which can be easy, as in your example, or can take days, such as in a complex database processor that only supports a specific (and difficult) schema. 

Needless to say, I can go on.  But I don't.  Instead, I say "the hell with it".  The problem is that our tools (which have improved 10x in the last 10 years IMHO)... suck.  Starting with the OS, all of our tools complicate document (source) management.  The filesystem is the most aggregious of these (files don't belong in a tree... they go in a database).  The 'create unit test' button in your favorite IDE is probably the first thing your customizations to the build system broke.  Of course, if you work professionally in C++ like me then there is no such button.   Your source control checkin inevitably is not a 'create project and submit to git' one-button operation.  It probably involves carefully choosing files (ok, this is rare with git... think Perforce instead) that aren't build products and manually inserting them.  Oh yeah, and it uses the file system (!) so it needs to be told where to look.  And god forbid you were trying an experiment and it failed you will enjoy backing out of your intermediate changes (ok, git helps a little here too).

So to sum up, tests are easy to write.  Tests are *not* easy to use.  Because *nothing* about development is easy (read : fast and not aggrivating).  ...and when the task that you thought should take 10 minutes took you 6 hours then you will stop doing it.

Daniel Tuchtenhagen replied on Wed, 2009/10/07 - 9:50am

Good agreeable article, just a little note:

Maybe you did not need as much time for tests, since your test coverage is not 100% ... test to search for a boolean while having this in:

 case "bolean":

... ;)

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.