How to Choose a Java Testing Framework
Is automated testing a standard part of your Java project plan? Which of the testing frameworks is your favorite? By means of which criteria?
In this article, 4 developers share their opinions and experiences in the area of automated testing. Hopefully, these opinions will be helpful to you when thinking about the testing phase of your project. If you have experiences to share, please add them in the comments. Hopefully, together, we can provide newbies in this area some solid advice on the approach to take and the criteria to consider when thinking about the testing phase of, in particular, Java projects.
Notes on the authors and links to referenced tools and frameworks are found at the end of this article.
On Automated Testing
by Tom Wheeler
When I teach classes to experienced developers, I find that only about 40% regularly write tests. Perhaps 40% have also never heard of JUnit and half of them have no idea what a unit test is. Developers are usually under pressure from a project manager to conform to a tight schedule -- and those managers are likely themselves under pressure from customers who want the software delivered quickly. Unfortunately, testing is one major part of the project plan that some feel can be easily cut. That's shortsighted, because it can leave you with a buggy application that's even further behind schedule.
Why is that? Because writing automated tests actually saves time in the long run. Every developer makes mistakes and the testing process should help identify those mistakes. It might be faster to test some aspect of a program's operation manually than by writing an automated test case, but a manual test requires human interaction. The results of the manual test can be inconsistent because software testers are just as likely to make mistakes as developers. Conversely, an automated test case should produce the same results regardless of who runs them.
Perhaps more importantly, there's a chance of introducing more bugs when old bugs are fixed or new features are added. You must therefore run all the tests again after making any change to the system. This is where automated tests truly pay for themselves, because the cost of running those automated test cases is insignificant compared to the cost of performing tests manually. When developers test frequently, they can find and fix mistakes easily. This helps to ensure that quality stays high and that the team will be ready to ship on time.
by Meera Subbarao
Martin Fowler wrote: "Never in the field of software engineering has so much been owed by so many to so few lines of code." JUnit has been, and even today still is, the standard for writing unit tests. And it is the most popular open source tool out there. Having said that, there are many other open source tools which developers have been using other than JUnit. In my case, along with JUnit I am also using TestNG. So, here is what I have to say about these two frameworks:
- With annotations being used in both JUnit 4 and TestNG, they both make testing simple and fun. If you have two simple test classes, one written in JUnit and one in TestNG, unless you look at the import statements, you will see no difference between them.
- If you are a strong proponent of test driven development (TDD), and have several tests running as part of your Continuous Integration process, TestNG would be a better framework to use. The facility to rerun failed tests is very helpful when you are running your nightly builds; and this feature is available only in TestNG, not in JUnit.
- The next major plus point of TestNG is parameters support. In JUnit, if you want to test for different parameters, you will have to write different test cases covering each of these parameters. On the other hand, in TestNG you provide these data XML configuration files. I have seen developers complain about XML files and they were correct when they said: "Now, instead of one Test Class which I need to maintain for all these various data, I also need to maintain an XML file".
- The HTML reports generated by JUnit are really good. I am using TestNG with Java 6, and the report it generates isn’t as pretty as JUnit in spite of using the junitreport task in my build files.
Finally, I would like to conclude that both these frameworks have their strengths and weakness, there is nothing stopping us developers to use both. Have fun with these two amazing frameworks, and enjoy writing testing.
Why I Switched from JUnit to TestNG
by Andres Almiray
I chose JUnit 3.x when I started unit testing programming because basically it was the only open source option at the time that had decent documentation and a pair of books you could refer to. It also came with a fair set of extensions like dbUnit and xmlUnit, that would let you test a larger set of components. But when requirements dictated that we should also have to have more complex tests, usually referred to as integration/functional testing, it was clear JUnit could not handle the pressure. That is when I switched to TestNG. Cedric and Alexandru, from TestNG, made it clear form the beginning that TestNG could be used for broader testing scenarios, not just unit testing. The fact that TestNG is able to run unmodified JUnit tests made the transition even smoother.
To be fair, JUnit 4.x was later released with features similar to those found on TestNG, closing the gap between the frameworks. Still, TestNG remains my favorite as it is actively updated. There is also a newcomer into the open source testing frameworks for Java, easyb, a Groovy based behavior-driven development (BDD) testing framework for Java and Groovy, which makes writing acceptance tests or scenarios a breeze, it reads as a spec even though it is executable code, which is mostly what you get with RSpec in the Ruby world.
Why JUnit Is Still A Compelling Choice
by Aslam Khan
Like most people starting out with test driven development and unit testing, I kicked off with JUnit 3.x. I found JUnit to have the most widespread tool support with JUnit test runners appearing in various places (ANT, Maven, Eclipse, IntelliJ IDEA, etc.). Also, it was the easiest to introduce to new software development teams. I also use TestNG and am impressed with its diversity. However, the various "add-ons" (dbUnit, xmlUnit, etc) for JUnit still make it a compelling choice. If you spend a lot of time with the Spring Framework, then the Spring ApplicationContext aware test cases based on JUnit also gives JUnit the edge. For testing of web-front ends, I use Selenium almost exclusively. I have dabbled with Canoo and others in the past but found the development approach very anti-TDD. With Selenium, I can hand over Selenium test scripting and recording to just about anybody and then tweak later.
If we're talking pure TDD, which is about writing good code (not just good tests) then you have to add in a mock testing suite as well. For mocking, I still use JMock which plays nicely with JUnit and I find that following a mock-based approach and developing from the boundaries of the application inwards, I get really good classes with decent messages being passed between objects. This goes a long way to towards increasing readability and maintainability. EasyMock is just as appropriate, but JMock is just a personal preference for me.
Shifting from the Java world to that of Ruby, RSpec is great and has a pretty decent DSL for describing stories and scenarios. Now that rbehave's story runner has been merged with RSpec, the combination makes for quite a compelling choice. Interestingly, rbehave was spawned from JBehave which is a behavior driven development testing framework. So, if you like the BDD style of gathering and defining requirements, you will most likely enjoy JBehave and RSpec.
About the Authors
- Tom Wheeler is a Principal Software Engineer at Object Computing, Inc. (OCI), currently doing consulting work at Boeing. He lives in the St. Louis, Missouri area.
- Andres Almiray is a Zone Leader of Groovy Zone. He has been hooked on Java since 1996 and has been involved in extensive web and desktop application development. His current interests include developer testing, software architecture, Groovy, Swing, and Spring.
- Meera Subbarao is Zone Leader of IT Book Zone. She is a Senior Software Consultant at Stelligent Incorporated. Meera has been working on J2EE and .NET technologies exclusively for the last seven years.
- Aslam Khan is the new Zone Leader for Architects Zone and SOA Zone. He is technical director of PBT Group, a software consultancy based in South Africa.