I am a software engineer at Google on the Android project and the creator of the Java testing framework TestNG. When I'm not updating this weblog with various software-related posts or speaking at conferences, I am busy snowboarding, playing squash, tennis, golf or volleyball or scuba diving. Cedric is a DZone MVB and is not an employee of DZone and has posted 89 posts at DZone. You can read more from them at their website. View Full User Profile

Better mock testing with TestNG

03.24.2010
| 8860 views |
  • submit to reddit

When you are using mocks in your tests, you usually follow the same pattern:

  1. Create your mock.
  2. Set it up.
  3. Play it.
  4. Verify it.

Of these four steps, only steps 2 and 3 are specific to your tests: creating and verifying your mock is usually the exact same code, which you should therefore try to factor out. It’s easy to create your mock in a @BeforeMethod but the verification aspect has never been well integrated in testing frameworks.

Until today.

Mock verification, or more generally, systematic invocation of certain test methods, is a feature that’s been requested several times by TestNG users and while I’ve often been tempted to go ahead and implement it, something has always felt “not right” about it.

As it turns out, there are apparently three ways that you can automate mock verification in TestNG: one that is incorrect, one that doesn’t work and one that actually works.

Let’s start with the incorrect one:

@BeforeMethod
public void init() {
this.mock = // create mock
}

@Test
public void t1() {
// set up and play mock
}

@Test
public void t2() {
// set up and play mock
}

@AfterMethod
public void verify() {
this.mock.verify();
}

This approach will run the verification after each test method, but the problem is that this code is being run in a configuration method (@AfterMethod) instead of a test method (@Test). Configuration methods are handled differently by TestNG in how they fail and how they get reported (in a nutshell, a configuration method that fails typically aborts the entire run since your test environment is no longer stable).

My next thought was to use TestNG’s groups and dependencies:

@BeforeMethod
public void init() {
this.mock = // create mock
}

@Test(groups = "mock")
public void t1() {
// set up and play mock
}

@Test(groups = "mock")
public void t2() {
// set up and play mock
}

@Test(dependsOnGroups = "mock")
public void verify() {
this.mock.verify();
}

At least, this solution runs the verification in a @Test annotation, but because of the way dependencies are run, the order of invocation for the code above will be t1(), t2() and then verify(). This is obviously wrong since we will only be verifyin whatever mock was played last. You could consider having each test method create and store a different mock and then have verify() go through all these mocks and calling verify() on them, but it’s clearly a subpar solution.

So let’s turn our attention to the correct solution, which involves the use of a method interceptor.

I already feature method interceptors in a previous entry that showed you how to create a new annotation @Priority that allows you to order your methods. The idea is similar here, except that instead of reordering the methods that we want TestNG to run, we are going to change their number as well.

For this to work, we introduce two new annotations: @Verify which indicates that this method needs to be verified, and @Verifier, which annotates the method that performs the verification. Here is an example of how you use them:

@Verify
@Test
public void t1() {
// set up and play mock
}

@Verify
@Test
public void t2() {
// set up and play mock
}

@Verifier
@Test
public void verify() {
this.mock.verify();
}

Now we need to write a method interceptor that will go through all the @Test methods, locate the verifier and then return a list of methods where each method annotated with @Verify is followed by the invocation of the @Verify method. The code is straightforward:

IMethodInterceptor mi = new IMethodInterceptor() {

public List<IMethodInstance> intercept(List<IMethodInstance> methods,
ITestContext context) {
List<IMethodInstance> result = Lists.newArrayList();
IMethodInstance verifier = null;

// Doing a naive approach here: we run through the list of methods
// twice, once to find the verifier and once more to actually create
// the result. Obviously, this can be done with just one loop
for (IMethodInstance m : methods) {
if (m.getMethod().getMethod().getAnnotation(Verifier.class) != null) {
verifier = m;
break;
}
}

// Create the result with each @Verify method followed by a call
// to the @Verifier method
for (IMethodInstance m : methods) {
if (m != verifier) {
result.add(m);
}

if (m.getMethod().getMethod().getAnnotation(Verify.class) != null) {
result.add(verifier);
}
}

return result;
}

};

Running it produces the following output:

t1
Verifying
t2
Verifying

 

With this simple interceptor (full source), you can now completely factor out the boiler plate logic of your mocks and focus on their business logic.

Happy mocking!

From http://beust.com/

Published at DZone with permission of Cedric Beust, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Tags:

Comments

Jonas Olsson replied on Thu, 2010/03/25 - 8:45am

This assumes that you do all your assertions before verifying that the required/expected calls were made. Minor point, but we tend to verify our mocks before asserting the result.

David Whitmore replied on Fri, 2010/03/26 - 4:30am

The style of mocking that you describe sounds like the EasyMock style. 

However, there are other styles of mocking such as that used by Mockito which differ in the final 'verify' step.

In EasyMock, verification is *general*: you verify all of your *mocks*.  (Which verifies all their expectations).

In Mockito, verification is *specific*: you verify the *methods* of your mocks directly.

I find that the Mockito style simplifies my tests:

With EasyMock, I've previously (and as you point out wrongly!) had test classes with tearDown methods to verify the mocks. 

With Mockito, however, the verification step fits nicely into individual test methods - no need for tearDown methods.

(I'm not affiliated with Mockito in any way, btw - I just like it!  And, for the record, EasyMock is awsome too!)

Cedric Beust replied on Fri, 2010/03/26 - 2:25pm

David: "wrongly" is a bit harsh. I'm describing how TestNG works, and I can guarantee you I'm right about that :-)

The fact that you used tearDown with JUnit and it works is a fluke: it's still undefined behavior to run tests in a setUp or tearDown method, you should really only be doing this in test methods. Try it: if the verification fails in your tearDown, do you see the failure as a test failure in the report?

Back to your original point: either mock approaches will work with the code I posted since all this code is being run in @Test methods.

David Whitmore replied on Sat, 2010/03/27 - 10:33am in response to: Cedric Beust

Sorry, my mistake. 

To clarify: what I meant, is that your article highlights the fact that *I* have been doing things wrongly - misusing JUnit tearDown methods to verify mocks.

I didn't mean to suggest that you pointed out something wrongly.  Ah - the hazards of bad grammar!  :)

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.