Michael Schnell is living in Hamburg (Germany) and works as a Project Manager, Architect and Senior Java Developer. Michael is a DZone MVB and is not an employee of DZone and has posted 13 posts at DZone. You can read more from them at their website. View Full User Profile

@Ignore unit tests immediately if they fail!

07.25.2011
| 5600 views |
  • submit to reddit

In development teams, there is often discussion about using the @Ignore tag for failing tests.

In particular, developers who are very enthusiastic about writing unit tests often tend to be very dogmatic about it. They argue, "A test should never be ignored! It’s better to have the build server on Yellow if tests fail. That way we always have an overview of our problems and are forced to fix them!"

What might sound good at first is not a good idea, at least when it comes to large development teams.

If you have several commits from multiple developers and maybe even additional deliveries from different branches to the main development line, the build will definitely break from time to time. How long does it take to fix such an error? Sometimes it may be easy, but in some cases, it may take days to fix a broken test. Now let’s assume that we have implemented the rule that you should never ignore a test. This means that the build may stay Yellow for a long time.

If another developer now updates his workspace with the latest version from the trunk and runs the unit tests locally, he will see failing tests even if he didn’t make any mistakes! That’s pretty bad itself. But he then checks the build server and, after some time looking around, he realizes it wasn’t his fault, as the server is already in the Yellow state. Should he commit now? If he does, he will receive endless emails from the build server about the broken build state that he didn’t cause. If not, his work and maybe the work of others will be blocked until the test is fixed. In a typical, pretty hectic IT project, with it’s almost impossible to meet deadlines, this is not really an option.

What is the solution?

  1. If you cause a build to break, @Ignore the failing test immediately and commit the change.
  2. Now that the build server is Green again (or Blue in the case of Hudson), start fixing the test.
  3. After you have fixed the test, commit the change and check to see if the build stays Green.
  4. Have some kind of statistic page on the build server that lists all ignored tests – this allows you to easily track the tests that are currently disabled.

Don’t get me wrong: all failing unit tests should be fixed as fast as possible! But my failure when committing a change should never constrain other team members from doing their work.

Published at DZone with permission of Michael Schnell, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Timo Meinen replied on Mon, 2011/07/25 - 7:13am

In my opinion, ignoring is the worst of all possible solutions, especially if your CI server doesn't show the number of ignored tests, like Jenkins/Hudson.

After ignoring tests, some other developers might commit their changesets into a broken build. It is hard to find the problem, if the new commits leads to more broken tests. The best metaphor for this is the broken window theory. Your suggestion ist counterproductive to the simple rule "Don't commit into a broken trunk".

What are better solutions to ignoring failed tests:
1. Never Ignore tests!
2. Run tests locally before commiting
2a. If they last too long, change your testinfrastructur to run fast tests
2b. If not possible, use modern CI-Server with the possibility to have Pre-Tested-Commits. TeamCity does an excellent job.
3. Never commit into a "red" trunk
3a. Use modern VCS, like Mercurial or Git, which allows to revert Commits
3b. Give the developer 10 minutes time to fix his test, before auto-reverting the commit by the VCS


Ignoring tests and commiting manuel-code-reverts will make the troubleshooting much more complicated. Developers tends to procastination in fixing their tests, until so many commmits are made, that noone knows, why the test failed and finally delete the broken test. If it would have been fixed right after discoverinng, the test would still be there and make his contribution to the test coverage.

I think the best approach is to revert the whole commit by the VCS. The developer is forced to fix his test to "get his feature into the trunk".

Vadim Pesochinskiy replied on Mon, 2011/07/25 - 11:53am

We use TeamCity, which now supports muting failing tests. When tests fail builmaster assigns them to developer (soon TeamCity will do that automatically if only one change caused failure) and mute test accross all projects and releases. When developer submits fixes the failures are unmuted. After tests muted no build failures are sent unless other tests fail. If a lot of tests failed UI allows to mute all in 2 clicks. Have used TeamCity for 2 years now, it rocks! BTW it is free with unlimited number of users and 3 buid agents. If you outgrow the free edition, which is hard to do, paid version is about ~$3K with $300 per additional build agent.

Mladen Girazovski replied on Mon, 2011/07/25 - 4:00pm in response to: Timo Meinen

I agree with Timo, disabling tests because newly committed changes breaks them is ignoring the problem instead of fixing it, i prefer a revert anytime.

Other developers need to informed immediatly of the broken build, they must not update from the Source Repo or they be very likely wasting trying to find a problem that was created by someone else, CI Servers can and should send emails if a commit broke the build, including the name of the developer that committed the change, helps with the attitude ;)

Aaron Digulla replied on Mon, 2011/08/15 - 10:49am

I agree with the other commenters that this is probably a workaround for a missing feature in your tool chain. The main concern is that @Ignore is easy to forget (some CI servers don't tell you about them, so they can lurk in your code for a long time.

They are also (somewhat) hard to find (you can search for all places where @Ignore is used and there shouldn't be too many ... *cough* ...)

So I like Vadim's suggestion best: The tools should allow me to say "Yeah, I know, stop bothering me about it, OK?" because I don't want to ignore the test, I just don't want it to hide other/more pressing issues.

I've opened an issue against JUnit to allow to specify an "until" date in @Ignore. Along with the comment, this should help to clear the view: https://github.com/KentBeck/junit/issues/288

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.