I've been fascinated with software development since I got my first C64 thirty years ago. And I still like the I emerging possibilities the technology offers us every day. I work as developer and software architect in internal and customer projects focusing on Java and Oracle. Thomas has posted 9 posts at DZone. You can read more from them at their website. View Full User Profile

Characterization Tests with MagicTest

12.21.2012
| 3049 views |
  • submit to reddit

MagicTest introduced a new visual approach to testing. In its first release, MagicTest was targeted at supporting unit tests. However the visual approach to compare the actual result with a stored reference output promises also to be very effective for a broader range of tests. Therefore new functionality has been added to MagicTest which explicitly targets characterization tests.

The Visual Approach of MagicTest

Let's shortly review how the visual approach works and what advantages it offers.

The idea for MagicTest was born out of frustration about the assertion based approach of traditional Java test frameworks like TestNG or JUnit. These frameworks make your tests runnable without user intervention at the cost of quite a few inconveniences.

The visual approach introduced by MagicTest addresses these issues. The idea goes back to the times before test frameworks where you used debugger or trace statements to visually verify the correctness of your program.

MagicTest automatically collects the relevant information of your test and compares this actual result at the end of the test with a stored reference output. If the output is equal, the test is successful, otherwise it is considered as failed.

MagicTest offers improvements in the following areas:

  • Positive Tests: No assert statements are necessary to integrate the expected result in the test code. This does not only reduce the amount of typing, but also makes maintaining the tests easier.
  • Negative Tests: Negative tests are as simple as positive tests, i.e. there is no necessity for workarounds like try-catch or separate test methods.
  • Complex Tests: Complex objects and huge datasets can be checked by simply dumping the relevant information.
  • Maintaining Tests: The effects of all changes are shown clearly arranged in the report and can be visually checked and confirmed.
  • Reporting: The HTML report contains the details for every method call. So it is clear for anybody what exactly has been tested.
To make use of this visual approach, you add the @Trace annotation to your test method. The @Trace annotation makes MagicTest automatically trace parameter and result values and catch exceptions thrown by the method under test.

The collected information is then presented to the developer as HTML report:

This functionality is clearly targeted at unit tests as all details of your calls to the method under test become visible without further effort.

Characterization Tests

Before presenting the new features of MagicTest, we will first look at characterization tests in general and how they differ from unit tests.

Unit tests are used to test the functionality of a single software module to its full extent. So to test a single method, our tests should possibly include all possible parameter values for positive and negative tests. Unit tests are typically written together with the software module (or even before) by the developer itself.

Characterization tests on the other hand document the behavior of an existing program without adequate unit tests, also known as legacy code. These tests are written on maintaining an application to provide a safety net, for example if you have to add new features and must guarantee that existing functionality remains unchanged.

You can implement characterization tests like unit tests using assertions. So if you observe that the legacy code returns a certain output based on an input, you can include this as characterization test. The following example is taken from the shown test report:

assert("Hello MagicTest", join("Hello", "MagicTest"));

Note that this is not a unit test even if this looks like one. The difference is that the characterization test makes no statement about the correctness of this behavior, it just states that this has been observed in the existing software and documents it this way.

But we have already learned that writing any sort of tests with assertions will not be satisfactory - that's why we finally have invented MagicTest! And now we would basically have to write all this assertion code ourself or use a tool which generates the tests out of the existing sources.

Fortunately the visual approach of MagicTest offers us a more comfortable solution: We just dump all needed information during the test runs and let MagicTest collect and compare it.

Let’s have a look at an example: You have a program which queries some data from a database and writes them properly formatted out to a report file. Unfortunately all business logic is mangled into a single completely untestable method which may look like this:

void printReport(String query, String outFile) {
   LOG.debug("Query: {}", query);
   Statemnt stmt = con.createStatement();
   ResultSet rs = stmt.executeQuery(query);
   StringBuilder content = new StringBuilder();
   while (rs.next()) {
      // 1000 cryptic lines of Java code extracting data out
      // of the result set and building up the file content
   }
   LOG.debug("Content: {}", content.toString());
   writeTextFile(outFile, content.toString()); 

}The logging statements in the code above will allow us to collect all relevant information during a program run. If we are lucky, such logging statements exist already in the application and we can just use them – otherwise we have to add such logging calls to the productive code. Instead of coding the logging statements manually, we also could benefit from an AOP framework to let us add these statements dynamically based on some configuration.

It can also be necessary to write a kind of test driver which will call the existing methods and enables us to run the test automatically. In our case this could like this:

void test() {
   String[] queries = { 
      // Add the relevant queries here
   };
   String tmpFile = "/tmp/report.tmp";
   for (String query: queries) {
      printReport(query, tmpFile);
   } 

}For our characterization tests, we would now compose a set of the most important queries and let the application process them. During the program run, we collect the output created by the application and store in a safe place.

We can then extend or refactor the existing application as needed. Any time we like, we can run our characterization tests again to prove that existing functionality has not changed by comparing the actual new output with the stored reference output.

We can do this quite well manually by firing up a diff program after each run to compare the output. After a few runs, we would probably write a script which does this automatically and just alerts us in case of a difference. But nevertheless it remains tedious work. To our luck, MagicTest offers exactly the described functionality out of the box with the @Capture annotation.

Characterizations Tests with MagicTest

To support characterization tests, the new annotation @Capture has been added to MagicTest.

If a test is annotated this way, all generated output will be captured and automatically be stored as actual output. The test will then considered successful if this output matches the stored reference output.

The following example shows the use of the @Capture annotation for a test method writing out some silly text:

@Capture(outputType=OutputType.PRE, 
      title="Capture with output type PRE")
public static void testCapture() {
   System.out.println("Line 1");
   System.out.println("Line 2");
   System.out.println("Line 3");
   System.out.println("Line 4");
   System.out.println("Line 5"); 

}Note that unlike using the @Trace annotation, here exceptions are not caught and traced automatically: if your test code could throw an exception, you must catch and handle it manually as you would do in a normal code (a test aborted by an uncaught exception will be considered as failed).

What kind of output should be captured is specified by the source attribute of the @Capture annotation. The following values are supported by the enumeration:

Type

Description

OUT

All output written to System.out is collected. This is the default value.

ERR

All output written to System.err is collected.

OUT_ERR

All output written to both System.out and System.err is collected.

LOGBACK

All output written to Logback loggers is collected. For collection, the output must be arrive at the root logger which may not be the case if you use loggers with additivity set to false.

To make the output as useful and appealing as possible, you can control the display of the collected output by the outputType argument:

Type

Description

TEXT

With the default output type text, line endings are preserved, i.e. the text is wrapped at newline characers. If texts are compared, the difference is determined line by line as known from traditional diff tools.

PRE

Output type pre is similar to text. The only difference is that a monospaced font is used for display which allows to see the exact spacing.

HTML

The collected output is interpreted as HTML code which is then included verbatim in the generated HTML report. The output is not validated for HTML conformance, but it must be well-formed XML. Diffing of HTML output is done on the HTML code level and thus showing all markup tags.

The use of HTML as output type allows adequate formatting of all data, so a SQL query result could easily be displayed as HTML table. The use of CSS allows you also to apply formatting as needed.


The output type influences also how differences between actual and reference output are displayed. The output types TEXT and PRE display differences as unified diff as you know it from well-known diff applications:


Summary

With the support of characterization tests, MagicTest opens an exciting new field of application. The @Capture annotation allows you very easily to record any existing behavior and to prove later that this functionality has not changed.

MagicTests supports you in doing everything automatically, you would otherwise have to do manually: collecting output, managing reference files, diffing results and presenting the relevant information to the developer.

The fact how easy support for characterization tests could be added shows the power and versatililty of the visual approach featured by MagicTest. So make you sure you don’t miss the @Trace annotation supporting unit tests if you like the MagicTest way of testing.

MagicTest can be downloaded from magicwerk.org. On the website, you will also find additional documentation and articles.

Published at DZone with permission of its author, Thomas Mauch.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Tags: