I've been fascinated with software development since I got my first C64 thirty years ago. And I still like the I emerging possibilities the technology offers us every day. I work as developer and software architect in internal and customer projects focusing on Java and Oracle. Thomas has posted 9 posts at DZone. You can read more from them at their website. View Full User Profile

Effectively Handling Exceptions in Testing Using MagicTest

03.18.2010
| 10757 views |
  • submit to reddit

Unfortunately handling exceptions thrown by methods under test has always been cumbersome. There has been some relief in the last few months, as both JUnit and TestNG made an effort to improve the handling of error conditions:

  • JUnit 4.7 added rules with ExpectedException as one implementation

  • TestNG 5.10 added the attribute expectedExceptionsMessageRegExp to the @Test annotation

However despite these efforts, the overhead for handling exceptions cases is still quite high.

The visual approach introduced by MagicTest on the other hand makes handling exceptions a breeze.

This article will present the features implemented by JUnit and TestNG and compare them to the new visual approach featured by MagicTest.

Example

To compare the different approaches, we will look at an example method which returns a configuration value. The method under test must implement the following requirements:

  • If namespace and key are valid, the configuration value must be returned

  • If the namespace is valid, but the key is not known, an error must be thrown containing the invalid key

  • If the namespace is not valid, an error must be thrown containing the invalid namespace

So a simple implementation could look like this:

public class Config {
public static Object getValue(String namespace, String key) {
if (!namespace.equals("database")) {
throw new IllegalArgumentException("Unknown namespace: " + namespace);
}
if (!key.equals("name")) {
throw new IllegalArgumentException("Unknown key: " + key);
}
return "db1";
}
}

Our test method should simply test the three requirements. So basically we would just like to have to code the following:

// Successful tests 
Config.getValue("database", "name"); // result: "db1"

// Tests with errors
Config.getValue("database", "UNKNOWN"); // error: "Unknown key: UNKNOWN"
Config.getValue("UNKNOWN", "UNKNOWN"); // error: "Unknown namespace: UNKNOWN"

As we will see, MagicTest allows us to use exactly these three lines of code to cover all testing needs.

But how would we implement testing these requirements using JUnit and TestNG?

Traditional approach

Before TestNG entered the stage, you probably used JUnit 3 for testing – which offered no support at all for testing error cases. So you had to catch the exceptions manually and ended up with code like this:

public class ConfigTest_Traditional {
@Test
public void testGetValue() {
assertEquals(Config.getValue("database", "name"), "db1");

try {
Config.getValue("database", "UNKNOWN");
fail("Exception must be thrown");
} catch (IllegalArgumentException e) {
// expected exception
}

try {
Config.getValue("UNKNOWN", "UNKNOWN");
fail("Exception must be thrown");
} catch (IllegalArgumentException e) {
assertEquals(e.getMessage(), "Unknown namespace: UNKNOWN");
}
}
}


While this approach forced you to type a lot, you had the possibilty to check the exception messages for correctness and also check several exceptions in one test method.

TestNG

TestNG added the annotation expectedExceptions which allowed you to easily check whether an exception of the given type was thrown. This could save you quite some lines of code - if you had only to check a single expection case in your test method and did not have to check the exception message.

TestNG 5.10 then added the attribute expectedExceptionsMessageRegExp, which allows checking the content of the exception message.

public class ConfigTest_TestNG510 {
@Test
public void testConcat() {
assertEquals(Config.getValue("database", "name"), "db1");
}

@Test(expectedExceptions = IllegalArgumentException.class)
public void testConcatErr1() {
Config.getValue("database", "UNKNOWN");
}

@Test(expectedExceptions = IllegalArgumentException.class, expectedExceptionsMessageRegExp = ".*null.*")
public void testConcatErr2() {
Config.getValue("UNKNOWN", "UNKNOWN");
}
}


JUnit

With version 4, JUnit adapted the annotation based configuration approach of TestNG and the ability to easily check for an expected exception. As the approach is the same, so are the drawbacks: no checking of exception message and only one exceptionper test method can be handled.

JUnit 4.7 added a concept called rules. One implementation of a rule is called ExpectedException and allows you to specify the expected exception with its message programmatically.

public class ConfigTest_Junit47 {
@Test
public void testGetValue() {
assertEquals(Config.getValue("database", "name"), "db1");
}

@Test(expected = IllegalArgumentException.class)
public void testGetValueErrJUnit40() {
Config.getValue("database", "UNKNOWN");
}

@Rule
public ExpectedException errJUnit47 = ExpectedException.none();

@Test
public void testGetValueErrJUnit47() {
errJUnit47.expect(IllegalArgumentException.class);
errJUnit47.expectMessage("Unknown namespace: UNKNOWN");
Config.getValue("UNKNOWN", "UNKNOWN");
}
}


Using ExpectedException it is possible to check the message of the exception, but if we compare the lines of code used for this approach with the traditional try-catch approach, we can conclude that both need five lines, so the benefit of this feature seems to remain quite low.

MagicTest

The visual approach introduced by MagicTest makes testing error cases as easy as sucessful ones. So we really just need to code three lines to test our three requirements:

public class ConfigTest {
@Trace
public void testGetValue() {
// Successful tests
Config.getValue("database", "name"); // result: "db1"

// Tests with errors
Config.getValue("database", "UNKNOWN"); // error: "Unknown key: UNKNOWN"
Config.getValue("UNKNOWN", "UNKNOWN"); // error: "Unknown namespace: UNKNOWN"
}
}


As MagicTest automatically catches and reports exceptions but then normally continues processing, we can have several exception cases in a row – something which is not possible using JUnit or TestNG without having to explicitly use try-catch clauses.

If we run this test, the test report shows us the result of the three method calls. As the test is run the first time, there is no reference result yet and the steps are therefore considered as failed.

 

 

If we then confirm the actual result to be correct by clicking on the Save link, it is saved as new reference result and therefore the test is successful.

 

Behind the scenes

How does this work? The magic lies in the @Trace annotation. It tells MagicTest to trace all calls to the method getValue and reports parameters and result.

To generate the needed output without having to manually code these statements, the byte code is instrumented before the test method is executed: So for each call to the getValue() method parameters, return value, and thrown exception are automatically traced.

So a call to the method under test will roughly look like this:

try {
printParameters("database", "name");
String result = Config.getValue("database", "name");
printResult(result);
} catch (Throwable t) {
printError(t);
}


The data traced out with the print-methods is then collected and visualized as HTML report.

Conclusion

We can try to collect the bare facts of the different approaches in a table:

Approach

Allows checking of exception message

Lines of code for checking exception message

Allows to handle more than one exception per test method

Traditional

yes

5

yes

JUnit <4.7

no

-

no

JUnit 4.7

yes

5

no

TestNG <5.10

no

-

no

TestNG 5.10

yes

3

no

MagicTest

yes

1

yes



Obviously the approach featured by MagicTest looks very promising.

Additionally the possibility to check successful and error cases in a single test method can greatly reduce the number of test methods needed – which may be easier to handle even if theory says that one test should just test a single thing.

Have a look at http://www.magicwerk.org/magictest to find out more about MagicTest.



Published at DZone with permission of its author, Thomas Mauch.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Nicolas Frankel replied on Thu, 2010/03/18 - 4:40am

Hi Thomas,

I don't know about MagicTest but I fail to see why handling more than one exception per method is desirable. Since the subject is unit testing, I think separating the cases make for a better failed build diagnostics, so I would advise you to use JUnit/TestNG approach.

Thomas Mauch replied on Thu, 2010/03/18 - 5:23am in response to: Nicolas Frankel

Hi Nicolas

I think the possibility to handle more than one exception can greatly reduce the number of test methods and lines of test code you have to write and maintain - which lets you more time to work on the real productive code.

In my screencast, I have an example with a method validating user input. So we have about 10 test cases with invalid user input. Do we really need 10 test methods which all raise an exception saying invalid user input? What is the benefit over putting all tests in a single method?

Of course you are free to use the traditional approach with one exception per method, even with MagicTest.

 

Nicolas Frankel replied on Fri, 2010/03/19 - 5:52am in response to: Thomas Mauch

Like I said before, the benefit is instantly to know the test which failed and thus make the diagnostics easier.

If your tests are dependent on a context that is updated throughout the test run, I would advise you to still separate your big test method into many neat little method and make them dependent: TestNG has this feature (don't know about JUnit though). This clearly gets away from pure unit testing.

Liam Knox replied on Sat, 2010/03/20 - 7:55pm

Personally in the above example I would of used 2 defined exception extending IllegalArgumentException. The code would be more readable and concise and you can construct the appropriate message in the constructor of the exception.

You can also then test with JUnit < 4.7. I agree with Nicolas, I dont really see a big use case where you would want to check multiple exception scenarios per test.

Allen Geer replied on Tue, 2010/03/23 - 10:08pm

I really don't know whats cumbersome about writing

"try {"

and

" } catch (Exception e) { ... }"

Usually just gotta right click and hit "surround with try/catch" or "add catch to method signature".

It seems neurotic to complain about having to type 25 more characters.

That being said, I find your survey of the unit testing libraries fantastic. Well written.

Thomas Mauch replied on Sun, 2010/03/28 - 6:25pm in response to: Allen Geer

I don't think it is just to about saving 25 characters

I you have a look at the concept of MagicTest, you will realize the visual approach saves you keystrokes for each test case as do not have you write the assert statement itself and - more important - the expected value. This allows you to faster write and maintain test cases. Have a look at the screencast for an introduction.

With the article I just wanted to point out that the visual approach is better suited for error handling than the features lately added to TestNG and JUnit. On the other hand, these efforts show that also other people are interested in saving a few characters of boiler plate code...

 

Tomek Kaczanowski replied on Mon, 2010/10/04 - 2:18pm

just my 3 cents (off-topic):

The following is evil and at some point will die with NullPointerException

if (!namespace.equals("database")) {

always do

NOT_NULL_CONSTANT.equals(maybeNullVariable)

 

Howgh!

--

Cheers,

Tomek Kaczanowski

Tommy Green replied on Wed, 2011/01/12 - 11:43am

MagicTest is supposed reduce the overhead need for writing and maintaining tests. I am still learning about the codes on MagicTest and I find them complicated as a novice. However, it will not hurt me to try it out though. - Jordan

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.