As an Agile Coach, Miško is responsible for teaching his co-workers to maintain the highest level of automated testing culture, allowing frequent releases of applications with high quality. He is very involved in Open Source community and an author of several open source projects. Recently his interest in Test Driven Developement turned into http://TestabilityExplorer.org with which he hopes will change the testing culture of the open source community. Misko is a DZone MVB and is not an employee of DZone and has posted 38 posts at DZone. You can read more from them at their website. View Full User Profile

My Unified Theory Of Bugs

11.18.2008
| 43990 views |
  • submit to reddit

I think of bugs as being classified into three fundamental kinds of bugs.

  • Logical: Logical bug is the most common and classical “bug.” This is your “if”s, “loop”s, and other logic in your code. It is by far the most common kind of bug in an application. (Think: it does the wrong thing)
  • Wiring: Wiring bug is when two different objects are miswired. For example wiring the first-name to the last-name field. It could also mean that the output of one object is not what the input of the next object expects. (Think: Data gets clobbered in process to where it is needed.)
  • Rendering: Rendering bug is when the output (typical some UI or a report) does not look right. The key here is that it takes a human to determine what “right” is. (Think: it “looks” wrong)

NOTE: A word of caution. Some developers think that since they are building UI everything is a rendering bug! A rendering bug would be that the button text overlaps with the button border. If you click the button and the wrong thing happens than it is either because you wired it wrong (wiring problem) or your logic is wrong (a logical bug). Rendering bugs are rare.

Typical Application Distribution (without Testability in Mind)

The first thing to notice about these three bug types is that the probability is not evenly distributed. Not only is the probability not even, but the cost of finding and fixing them is different. (I am sure you know this from experience). My experience from building web-apps tells me that the Logical bugs are by far the most common, followed by wiring and finally rendering bugs.

Cost of Finding the Bug

Logical bugs are notoriously hard to find. This is because they only show up when the right set of input conditions are present and finding that magical set of inputs or reproducing it tends to be hard. On the other hand wiring bugs are much easier to spot since the wiring of the application is mostly fixed. So if you made a wiring error, it will show up every time you execute that code, for the most part independent of input conditions. Finally, the rendering bugs are the easiest. You simply look at the page and quickly spot that something “looks” off.

Cost of Fixing the Bug

Our experience also tells us how hard it is to fix things. A logical bug is hard to fix, since you need to understand all of the code paths before you know what is wrong and can create a solution. Once the solution is created, it is really hard to be sure that we did not break the existing functionality. Wiring problems are much simpler, since they either manifest themselves with an exception or data in wrong location. Finally rendering bugs are easy since you “look” at the page and immediately know what went wrong and how to fix it. The reason it is easy to fix is that we design our application knowing that rendering will be something which will be constantly changing.

LogicalWiringRendering
Probability of OccurrenceHighMediumLow
Difficulty of DiscoveringDifficultEasyTrivial
Cost of FixingHigh CostMediumLow

How does testability change the distribution?

It turns out that testable code has effect on the distribution of the bugs. Testable code needs:

The result of all of this is that the number of wiring bugs are significantly reduced. (So as a percentage we gain Logical Bugs. However total number of bugs is decreased.)

The interesting thing to notice is that you can get benefit from testable code without writing any tests. Testable code is better code! (When I hear people say that they sacrificed “good” code for testability, I know that they don’t really understand testable-code.)

We Like writing Unit-Tests

Unit-tests give you greatest bang for the buck. A unit test focuses on the most common bugs, hardest to track down and hardest to fix. And a unit-test forces you to write testable code which indirectly helps with wiring bugs. As a result when writing automated tests for your application we want to overwhelmingly focus on unit test. Unit-tests are tests which focus on the logic and focus on one class/method at a time.

  • Unit-tests focus on the logical bugs. Unit tests focus on your “if”s and “loop”s, a Focused unit-test does not directly check the wiring. (and certainly not rendering)
  • Unit-test are focused on a single CUT (class-under-test). This is important, since you want to make sure that unit-tests will not get in the way of future refactoring. Unit-tests should HELP refactoring not PREVENT refactorings. (Again, when I hear people say that tests prevent refactorings, I know that they have not understood what unit-tests are)
  • Unit-tests do not directly prove that wiring is OK. They do so only indirectly by forcing you to write more testable code.
  • Functional tests verify wiring, however there is a trade-off. You “may” have hard time refactoring if you have too many functional test OR, if you mix functional and logical tests.

Managing Your Bugs

I like to think of tests as bug management. (with the goal of bug free) Not all types of errors are equally likley, therefore I pick my battles of which tests I focus on. I find that I love unit-tests. But they need to be focused! Once a test starts testing a lot of classes in a single pass I may enjoy high coverage, but it is really hard to figure out what is going on when the test is red. It also may hinder refactorings. I tend to go very easy on Functional tests. A single test to prove that things are wired together is good enough to me.

I find that a lot of people claim that they write unit-tests, but upon closer inspection it is a mix of functional (wiring) and unit (logic) test.  This happens becuase people wirte tests after code, and therefore the code is not testable. Hard to test code tends to create mockeries. (A mockery is a test which has lots of mocks, and mocks returning other mocks in order to execute the desired code) The result of a mockery is that you prove little. Your test is too high level to assert anything of interest on method level. These tests are too intimate with implementation ( the intimace comes from too many mocked interactions) making any refactorings very painful.

From http://misko.hevery.com

Published at DZone with permission of Misko Hevery, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Serge Bureau replied on Tue, 2008/11/18 - 10:28am

I do not agree with your high esteem of Unit tests.

Most difficult bug are timing bug, inter-thread bugs, database related bugs, UI interactions not only the visual.

All of those are not helped by Unit tests, so to me they have some value, but not that high.

Brian Shannon replied on Tue, 2008/11/18 - 12:21pm

The vast majority of applications do not have to consider 'timing' or 'inter-thread' bugs.  With multi-core processors, these bugs are becoming more of an issue as developers seek to improve performance, but the vast majority of applications do not need to utilize multiple cores.  

That being said, I think one could argue that thread safety is a logical bug, because your code 'does the wrong thing'.  It does not account for a particular state that a section of code may or may not be in.  It does not account for the fact that another thread could potentially enter the same code and change the state of the object(s) that you are dealing with.  However, you are certainly right that they are probably the most difficult to track down and that running Unit tests against them is difficult (but not impossible). 

Database related bugs and UI interactions fall into the category of functional testing that is addressed in the post.  Indeed, they are a class of tests that cannot be fully automated or covered by Unit tests (this is why a QA team is still needed).  So, they are not helped completely by Unit tests because, by definition there are not Unit testable. As stated in the post, though, you can implement minimal Functional testing as a sort of 'smoke test' to ensure that the basic functionality still works in your system.

However, I must disagree that Unit tests are not of high value.  They are showing their value on many teams that utilize them.  They tend to create a better designed product, they allow you to change and refactor the code frequently with less worry about whether or not you will break something, and they reveal problems in the code early and often.  Teams sucessfully utilizing Unit tests (among other best practices) are releasing higher-quality, faster-to-market, better-designed products.

John Ferguson Smart replied on Tue, 2008/11/18 - 5:19pm

Well said.

Serge Bureau replied on Wed, 2008/11/19 - 9:49am in response to: Brian Shannon

[quote=bshannon]

The vast majority of applications do not have to consider 'timing' or 'inter-thread' bugs.  With multi-core processors, these bugs are becoming more of an issue as developers seek to improve performance, but the vast majority of applications do not need to utilize multiple cores.  

[/quote]

I agree with a lot of what you said, but not this part.

Almost all Java code has to work on network, and with database connections and similar issues. Most applications are also multi-tiers. All those involve lots of threads and timing issues.

So you are right, it is functional testing, Unit tests are of very limited value.

So to me there is too much time allocated to it, logical errors should also mainly come up in functional testing.

So does Unit test protects against more than 5% of the errors ? I seriously doubt it.

Brian Shannon replied on Wed, 2008/11/19 - 11:41am in response to: Serge Bureau

Thanks for your reply, Serge.  :)

You are, of course, correct that there is multi threading going on, but most of these details are hidden from us.  We are rarely actually programming a multi-threaded application ourselves.

Logical errors are the result of a failure of a certain portion of the program given state 'x'.  If we are not testing for state 'x' in our unit tests, then you are correct, that functional testing will reveal these.  But this shows a lack of implementation in our unit tests not necessarily a failure of the effectiveness of unit tests. 

Recent case studies have shown that effective unit testing can reduce bug counts by as much as half or more. More and more teams are reporting on the effectiveness of unit tests and TDD.  I know that personally they are an integral part of my own development now and I can't see going back to *not* having them at this point.  They have proven their worth time and time again during my own development.  Many of my close colleagues also are reporting on the effectiveness of unit testing and TDD.

To restate, unit testing is not only for catching bugs, but it also tends to create a better designed, more maintainable, more loosely coupled, and more extendible system.  A highly testable system, is a well designed system.  It also allows the programmer to be confident about changing and refactoring code because they have a suite of tests they can run instantly that will tell them whether or not they broke something.

Serge Bureau replied on Wed, 2008/11/19 - 12:06pm in response to: Brian Shannon

Hello bshannon,

I do agree that code should be written to be easily tested, which was far from true in the past (still often on the present ;-) )

But the 50% figure has got to be a joke, big pressure from TDD market and false numbers.

Multi-threading is hidden from you ??? We are obviously in very different environments.

By the writing to be testable does not require TDD.

I do advocate testing, but very little Unit tests. Sorry but I do not adhere to this "religion"

Brian Shannon replied on Wed, 2008/11/19 - 1:09pm

Here's one such case study:  http://www.ddj.com/cpp/193501924  A little Googling goes a long way.  Also take a look at the work that QSMA is doing.

I know it's difficult to believe, but I've also experienced similar numbers on my own projects.

Yes, by and large, multi-threading in the context that you stated above is hidden from us.  With connection pooling, distributed computing, etc., etc. all of this code is already written for us and we use it.  Yes, from time to time, an error comes up, but I've found them to be quite rare.  Unit testing isn't trying to solve for these rare, edge cases that may occur in a system, but allow for the constant change that is needed to keep the code clean.

You are correct that writing the code to be testable does not require TDD.  I've used both methods both in writing tests before coding and writing tests after.  Both have resulted in improvements in the design of the code after the tests have been written.   However, I am not married to TDD and only use it when appropriate.

The effectiveness of thorough unit testing when properly applied is being proven out.  It is unfortunate that you see my comments as 'religous', but I am speaking from experience in many different environments from TDD to non-TDD, to absolutely no unit tests, to adequate unit tests, to close to 100% coverage in unit tests.  Each environment has it's own unique challenges, circumstances and differences, but I've found that those with more thorough unit tests tend to be better designed, easier to change, higher quality, etc.

I think an important point here is that it's not solely because of the unit tests that the code is better.  There are several best practices that lead to better code of which one is good unit tests.

Serge Bureau replied on Wed, 2008/11/19 - 3:42pm in response to: Brian Shannon

Hello again bshannon,

my experience is different , the quality of the code depends on the architect and the team leaders and code reviews.

We have a large current product that was in bad shape and applying good design and Unit Testing did improve it. How ever we also have a product that is no longer developped and only supported. It is the best code I have ever seen, and there is no test at all.

We wish to include some functional testing to it.

So the team that work on this project before TDD and Unit Testing came along made a wonderful job.

Methodology do not warrant good code, people do.

 

Brian Shannon replied on Thu, 2008/11/20 - 12:30pm in response to: Serge Bureau

I definitely agree that a mechanical methodology doesn't necessarily produce good code.  Bad code can be written under any circumstance.  However, usually good coders have some best practices under their belt that they use.  It's almost a chicken and egg problem.  Do good coders use unit tests because they are good coders?

Good code can certainly come about without unit tests as you state (and I have seen this as well), but I find it ironic that you are applauding this code and yet are trying to find ways to introduce testing. ;)

Another issue is the fact that you're going to have a wide variety of coders work on a piece of code over its life.  From mediocre to good to downright bad.  Having a unit testing framework around the code will ensure that as various programmers modify the code, they are not breaking things left and right and also are working within a well-designed system that atrophies less even as a bad coder may introduce bad code into the system.

Serge Bureau replied on Thu, 2008/11/20 - 1:09pm in response to: Brian Shannon

[quote=bshannon]

Good code can certainly come about without unit tests as you state (and I have seen this as well), but I find it ironic that you are applauding this code and yet are trying to find ways to introduce testing. ;)

[/quote]

Note that it was functional testing, not unit testing.

I am a firm beliver in functional testing. Unit testing to me is not that useful.

But having a program that is designed to be easily tested funtionally is a big plus.

Jakub Exner replied on Fri, 2008/11/21 - 5:45am in response to: Serge Bureau

Hi Serge,

I am currently thinking exactly about what you wrote - functional testing. Could you please share a few words about what is it about technically in your project(s)? For example, which frameworks and tools you use? Do the tests run in normal Java processes, i.e. not in Servlet or EJB container etc.? Do you have a data preparation phase (in a database for example) before the test suite and possibly each test?

Serge Bureau replied on Fri, 2008/11/21 - 12:36pm in response to: Jakub Exner

[quote=jexner]

Hi Serge,

I am currently thinking exactly about what you wrote - functional testing. Could you please share a few words about what is it about technically in your project(s)? For example, which frameworks and tools you use? Do the tests run in normal Java processes, i.e. not in Servlet or EJB container etc.? Do you have a data preparation phase (in a database for example) before the test suite and possibly each test?

[/quote]

Hello jexner,

I am not at liberty to discuss the specifics.

But on the main product, we have Unit tests written by the developpers. The QA team has an impressive amount of functional test with a complex assortment of tools (Silk, Fit, ...)

We also have an XML interface to all internal functionality so we can easily script complex behavior.

There is also a performance team.

On the older project I am working on, XML script is not available so the functional testing is almost non existent, I am trying to change that.

So our current QA team is much better equiped than me to answer you , I have seen their tools and tests and it is impressive. Unit tests is only a small part.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.