An early believer in the ability of Java to deliver "enterprise-grade" software, Andrew Phillips quickly focused on the development of high-throughput, resilient and scalable J2EE applications. Specializing in concurrency and high performance development, Andrew gained substantial experience of the intricacies, complexity and challenges of enterprise application environments while working for a succession of multinationals. Continuously focused on effectively integrating promising new developments in the Java space into corporate software development, Andrew joined XebiaLabs in March 2009, where he is a member of the development team of their deployment automation product Deployit. Amongst others, he also contributes to Multiverse, an open-source Java STM implementation, and jclouds, a leading Java cloud library. Andrew is a DZone MVB and is not an employee of DZone and has posted 22 posts at DZone. You can read more from them at their website. View Full User Profile

Testing Frameworks for Java with Code Samples

11.26.2009
| 22082 views |
  • submit to reddit

Some while back I was preparing a presentation on mocking and testing frameworks for Java. As part of the aim was to demonstrate some real, running code, I ended up spending quite some time copying, pasting, extending and correcting various examples gleaned from readmes, Javadoc, Wiki pages and blog posts. Since then, this codebase has been extended with various new features I've come across, and I've often referred to it for experiments, as a helpful reference, and suchlike.

I imagine this kind of "live" reference could also be useful to others, so I thought I'd share it.

Mock Braindump

By way of introduction, here first a braindump of various points of interest related to the frameworks in the sample1. Don't expect a detailed overview or comparison - there are plenty out there - and for specific questions about the working of the frameworks please refer to their documentation2.

Mockito or EasyMock?

EasyMock and Mockito are the de facto standard mocking frameworks out there at present. Their feature sets are more-or-less identical, and if one of them comes up with something new you can be pretty sure it'll be in the next version of the other.

Personally, I have to say that I find Mockito's syntax just that little bit nicer and more intuitive. The fact that you don't explicitly have to switch to replay mode is handy, and for things like spy (for partial mocks) or InOrder (for ordered expectations) the syntax is simply more elegant.
More fundamentally, the emphasis on stubbing rather then laboriously verifying calls to mocks was, to me, a very useful and justified distinction.

Having said all that, it might well stike you as paradoxical that I still mainly use EasyMock on a day-to-day basis: force of habit and the fact that the projects I'm working on started out with EasyMock sees to that. The advantages I believe Mockito has are not sufficiently large to justify the switch.

Statics, locals and finals: TDD's final frontier?

"To boldly go...where no mocking framework has gone before" seems to be the mission of both Jmockit and JEasyTest.

public final class ServiceA {

public void doBusinessOperationXyz(EntityX data)
throws InvalidItemStatus {
List<?> items = Database.find("select item from EntityY item where item.someProperty=?",
data.getSomeProperty());
BigDecimal total = new ServiceB().computeTotal(items);
data.setTotal(total);
Database.save(data);
}

}

public static final class ServiceB { ...

The challenge: to be able to unit test this class, specifically the find and save calls on Database and the computeTotal call on the new ServiceB instance. Using convential mocking this is nigh-on impossible:

  • find and save are static
  • ServiceB is a final class and thus can't be mocked
  • even if it could be, the ServiceB instance called is created in the code under test

My immediate reaction? If this is the kind of code you're supposed to test, you have other problems! Yes, of course it's an artificial example specifically chosen to highlight cases normal mocking can't deal with. But even if the real code you're having trouble with contains only one of these cases, I'd consider trying to refactor the code before looking for a different test framework.

Dependency Injection may have become a bit of a religion, but it is not so widespread for nothing. For code that's doing DI, the standard mocking frameworks are almost always sufficient.

But even if one perhaps shouldn't try to unit test the example, could one? Well, experience shows that there are few problems that cannot be solved with a sufficiently large dollop of bytecode manipulation.
JEasyTest works its brand of magic using a pre-test weaving step, which can be done by an
Eclipse plugin or by adding a plugin to your Maven build3. Jmockit uses the slightly more modern instrumentation approach and comes with a Java agent, which means that it only takes an additional VM argument to run in your IDE.

Usability issues aside, I found that I don't feel happy with the code to prepare a test fixture and register expectations in either framework; it's downright clumsy in some cases. Here's the JEasyTest test:

@JEasyTest
public void testBusinessOperation() throws InvalidItemStatus {
on(Database.class).expectStaticNonVoidMethod("find").with(
arg("select item from EntityY item where item.someProperty=?"),
arg("abc")).andReturn(Collections.EMPTY_LIST);
on(ServiceB.class).expectEmptyConstructor().andReturn(serviceB);
on(Database.class).expectStaticVoidMethod("save").with(arg(entity));
expect(serviceB.computeTotal(Collections.EMPTY_LIST)).andReturn(total);
replay(serviceB);
serviceA.doBusinessOperationXyz(entity);
verify(serviceB);
assertEquals(total, entity.getTotal());
}

 

expectStaticNonVoidMethod? ARGH! Feels more like ASM than unit testing.

Jmockit's "expectations" mode comes closest to what Mockito/EasyMock users are likely to be familiar with4:

@MockField
private final Database unused = null;
@MockField
private ServiceB serviceB;

@Test
public void doBusinessOperationXyz() throws Exception {
EntityX data = new EntityX();
BigDecimal total = new BigDecimal("125.40");

List<?> items = new ArrayList<Object>();
Database.find(withSubstring("select"), withAny(""));
returns(items);
new ServiceB().computeTotal(items);
returns(total);
Database.save(data);
endRecording();

new ServiceA().doBusinessOperationXyz(data);
assertEquals(total, data.getTotal());
}

 

Conceptually, this is reminiscent of an EasyMock test, with record and replay phases. The @MockField fields stand in for the creation of actual mock objects: the field declarations only indicate to Jmockit that mock of the given types are required when the test is run, cluttering the test class with unused properties.
In addition, the "mock management" methods (withAny, returns etc.) are not static, meaning they are not visually identified by e.g. being displayed in italics. I was surprised how much this seemingly minor discrepancy alienated me - it just doesn't look quite like a unit test.

JMock

From what I can see not much is happening around jMock anymore: the last release was in Aug 2008 and the last news posting over half a year ago. The syntax, which tries to mimic a pseudo-"natural language" DSL, is just a bit too cumbersome. jMock's support for multithreading prompted me to take a closer look, but it's actually simply a mechanism for ensuring that assertion errors thrown in other threads are actually registered by the test thread; there is no support for testing concurrent behaviour.

Testing concurrent code

I quite like MultithreadedTC, a small framework5 which aims to make it easy to start and coordinate multiple test threads. It does this by means of a global "clock" that moves forward whenever all threads are blocked - either "naturally" (e.g. during a call such as blockingQueue.take()
) or deliberately using a waitForTick(n) command.
As such, MultithreadedTC doesn't offer much more than can be achieved by "manual" latches as described in Iwein Fuld's recent blog post, but the clock metaphor does seem to make the test flow easier to understand, especially for longer tests.

Like latching, though, the main problem with MultithreadedTC is that you can't easily control the execution of code in the classes under test.

public void thread1() throws InterruptedException {
...
waitForTick(1);
service.someMethod();
waitForTick(2);
...
}

public void thread2() throws InterruptedException {
...
waitForTick(1);
service.otherMethod();
waitForTick(2);
...
}

 

This code will go some way to ensuring that service.someMethod() and service.otherMethod() start at almost the same time, and will guarantee that neither thread will continue until both methods have completed. But what if you want to ensure that half of someMethod completes before otherMethod is called?

For that, you'll have to be able to get access to the implementations of someMethod and otherMethod, for instance by subclassing the service implementations, or using something like Byteman.

Ultimately, though, I think unit tests are just not the right way of going about testing concurrent code. "Choreographing" the carefully-chosen actions of a small number of test threads is a poor substitute for real concurrent usage, and the bugs you'll find, if any, aren't the kind of concurrency issues that end up causing nightmares.

For proper concurrency testing, there doesn't so far seem to be a good substitute for starting a whole bunch of threads - on as many cores as possible - and running them for a good while (see, for instance, the integration tests of Multiverse, the Java STM). If it's possible to inject a certain amount of randomness into the timing (using e.g. Byteman), all the better!

JUnit Rocks Rules!

Version 4.7 of JUnit introduced rules, sort-of around aspects that are called before and after the execution of a test. Some of the standard examples demonstrate "housekeeping" functionality such as opening and closing a resource or creating and cleaning up a temporary folder. Rules can also affect the result of a test, though, e.g. causing it to fail even if all the test's assertions were successful.

Whilst one can see how the concept of rules can be useful, they still have a bit of "v1" roughness about them. The "housekeeping" rules are essentially convenient replacements for @Before/@After logic, and the syntax of the "test-influencing" rules feels messy:

public class UsesErrorCollectorTwice {
@Rule
public ErrorCollector collector = new ErrorCollector();

@Test
public void example() {
collector.addError(new Throwable("first thing went wrong"));
collector.addError(new Throwable("second thing went wrong"));
collector.checkThat("ERROR", not("ERROR"));
collector.checkThat("OK", not("ERROR"));
System.out.println("Got here!");
}
}

Wouldn't that be nicer if the assertions were a little more, um, assert-like? Or take:

public class HasExpectedException {
@Rule
public ExpectedException thrown = ExpectedException.none();

@Test
public void throwsNullPointerExceptionWithMessage() {
thrown.expect(NullPointerException.class);
thrown.expectMessage("happened?");
thrown.expectMessage(startsWith("What"));
throw new NullPointerException("What happened?");
}
}

I mean, it's certainly useful to be able to make more detailed assertions about exceptions, but could this not be integrated into either of the current exception-checking patterns6?

The potential power of rules also raises an question: is it wise to get into the habit of doing full-scale resource management (e.g. starting servers or DB connections) in a unit test?

Footnotes
  1. Sample source code here. Check out the project using svn checkout http://aphillips.googlecode.com/svn/mock-poc/trunk target-folder.
  2. See the sample code's POM.
  3. Which is not to say it's the best approach, of course!
  4. The original code is not longer being actively developed, but there has been some recent work aimed at better JUnit 4 integration.
  5. @Test
    public void tryCatchTestForException() {
    try {
    throw new NullPointerException();
    fail();
    } catch (NullPointerException exception) {
    // expected
    }
    }

    @Test(expected = NullPointerException.class)
    public void annotationTestForException() {
    throw new NullPointerException();
    }
  • Share/Bookmark
References
Published at DZone with permission of Andrew Phillips, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Tags:

Comments

Rogerio Liesenfeld replied on Thu, 2009/11/26 - 6:06pm

"@MockField"?!? This annotation has been deprecated in favor of "@Mocked" back in june 7 (in JMockit 0.98), and removed in release 0.993 (which is the current one since november 15).

Also, mock instance fields (and mock parameters) annotated as @Mocked are used in most tests, for instance method invocations. Only when all invocations to record or verify are made on static methods and/or constructors the mock field will (typically) go unused.

JMockit has a unique feature in which argument matching fields (which are static, so they will be displayed in italics) can be used instead of withAny(...) methods: simply write anyString to match any String argument value.

There aren't just two phases in the JMockit behavior-based mocking API. There are three: record-replay-verify. And there is no need to use the endRecording() method. BTW, all mocking APIs follow the record-replay-verify model of test execution in one way or another (well, with the exception of JMockit Annotations 8^). Yes, even Mockito.

For up-to-date and more complete and extensive examples of JMockit mocking APIs, see this page. (There is much more than this - check the project home page.)

For a detailed comparison between JMockit and Mockito, see the "Alternative mocking tools" section here.

Finally, to use JMockit you should be able to simply add jmockit.jar to the classpath. No need to use the "-javaagent" JVM parameter, nor add tools.jar (from the JDK) to the classpath anymore, nor even annotate test classes with @RunWith(JMockit.class) (which was required until recently).

Anuradha Gunasekara replied on Thu, 2009/11/26 - 11:18pm

What about Thread Weaver http://code.google.com/p/thread-weaver/

Steve Freeman replied on Fri, 2009/11/27 - 6:47am

jMock is by no means dead and there are plenty of people using it. What we've been busy with is our book (http://www.growing-object-oriented-software.com) which, amongst other things, explains the motivation for jMock and its syntax. The book also includes some chapters on testing multi-threaded code which might be relevant.

Once you start to think in terms of protocols between objects, rather than just methods calls, the concepts in jMock make more sense. Personally, I think record/replay is a really bad idea, but that's a longer discussion. I'm on the fence about Mockito's validation after the fact.

(Next time, I recommend at least posting a note to the list to see what we're up to.)

Andrew Phillips replied on Fri, 2009/11/27 - 8:12am

For further comments and responses see the original post.

Rogerio Liesenfeld replied on Fri, 2009/11/27 - 10:21am in response to: Steve Freeman

So, you say that jMock does not follow the record-replay model, like EasyMock does?
Well, I would like to understand how exactly you define "record/replay" then.

Aren't "record" and "replay" (and also "verify") the names of separate phases of test execution, for tests which use mocking? My OXFORD Dictionary defines a "phase" as an "stage in a process of change or development". The "process" here would be the execution of a test method.

BTW, the jMock API requires a call to the Mockery#checking(ExpectationBuilder) method in order to transition the test from record to replay. And unless the test class is annotated with @RunWith(JMock.class), a call to Mockery#assertIsSatisfied() is also required to end the replay phase.
So, in the end it is very much like EasyMock.

Steve Freeman replied on Fri, 2009/11/27 - 12:11pm in response to: Rogerio Liesenfeld

Of course the underlying mechanisms are similar, but that doesn't make them the same. I have worked with a lot of code written with EasyMock and seen some common problems.

I find that Record/Replay doesn't make a clear enough distinction between test setup and expectations--it's all within the record phase--so I see a lot of tests out there that are not clear about their intentions. I also find that a record phase makes it harder to factor out patterns of setup and expectation setting that can be used across tests, since they must be called while the test is in the right state.

The reason jMock is structured that way (apart from trying to work around broken Java syntax), is that we're trying to highlight the protocols between objects, to emphasise the relationships between them. It's not just about methods.

jMock checking is not directly analogous to a record stage. It's a call that sets up some or all of the expectations/stubs for a test. There can be any number of checking() calls in a test. They can even be interlaced with calls to the target code, although that's generally a bad idea.

Rogerio Liesenfeld replied on Fri, 2009/11/27 - 7:00pm in response to: Steve Freeman

Yes, I see what you mean.

In reality, though, EasyMock tests aren't necessarily divided in a single record phase followed by a single replay phase. The "replay(mock)" method switches a single mock object from record to replay, alowing other mocks in the same test to "remain" in the record phase.

In practice, all of these different mocking tools (EasyMock, jMock, JMockit, Mockito, Unitils Mock) "know" in which phase of the test any invocation to a mocked method occurs. They all have to keep track of this internally: when method "doSomething(...)" is invoked during the record phase the invocation is recorded (possibly with an specified return value, an exception to throw, an invocation count constraint, argument matching constraints, ...); later, when a matching invocation occurs to "doSomething(...)" during the replay phase, the previously recorded expectation will be used to determine ("replay") the appropriate result.

There is nothing that prevents a single test method from having multiple consecutive record and replay phases, or even to have invocations recorded in a setUp/@Before method.

So, I stand by my opinion that all these behavior-oriented mocking APIs follow the record-replay model (or rather the extended record-replay-verify model in the cases of JMockit, Mockito, and Unitils). The things you mentioned are API-style or syntax concerns (for example, if invocations are recorded inside an Expectations block or not).

Also, Java syntax isn't "broken". The JMockit Expectations & Verifications API proves it.

Ben Chatelain replied on Mon, 2009/11/30 - 2:26pm

A very important mocking framework that is missing from this and other comparisons is PowerMock. While not a full-blown mocking framework on its own, PowerMock works with EasyMock or Mockito to add incredibly powerful functionality. Like Jmockit and JEasyTest, PowerMock makes possible what is typically thought to be untestable (partial, static, private, final, new construction mocking and even static mocking in system classes). The main difference between PowerMock and the other "on steroids" mocking frameworks is that it implements these features using a custom class loader which makes it trivial to integrate into any build process with only the inclusion of a few JARs (no Eclipse/Maven plugin, no javaagent support necessary).

While this type of unit testing may not be appropriate for every situation, it is a great tool to have available especially for characterization tests for legacy code - written to ensure the quality of changes to a less-well understood body of code.

Rogerio Liesenfeld replied on Mon, 2009/11/30 - 3:13pm in response to: Ben Chatelain

As I stated previously, there is no need to use a "-javaagent" parameter with JMockit, or to make any change at all to the "build process".

Normally, you just add jmockit.jar to the classpath. (Also no need to annotate test classes with @RunWith, or to make them extend a base class.)

Andrew Phillips replied on Sun, 2009/12/06 - 8:40am

A follow-up to the interesting discussion here and in comments to the original post appears here.

Jane Good replied on Fri, 2011/07/15 - 8:44am

I am really not too familiar with this subject but I do like to visit blogs for layout ideas. You really expanded upon a subject that I usually don’t care much about and made it very exciting. This is a unique blog that I will take note of. I already bookmarked it for future reference. Thank you..

we use javascript in our site jump on it Sydney

Chaz Baxter replied on Sat, 2011/07/16 - 10:18pm

I'm in the same boat as Jane, I'm not a developer myself but I like to checkout blog designs and figured that the "heart of the Java developer community" would have something pretty cool. Anyways, I do like the look of your layout and in particular your logo, it's very clean and attractive! vin dicarlo pandoras box

 

Backlinks Sha replied on Mon, 2011/08/29 - 4:44am

Its nice to read some post about this nowadays. Its meaningful and somehow sharing this to readers like me, makes me want to surf the web to be able to get more wonderful ideas. backlinks

Sarah Wilson replied on Mon, 2011/08/29 - 9:54pm

I stand by my opinion that all these behavior-oriented mocking APIs follow the record-replay model (or rather the extended record-replay-verify model in the cases of JMockit, Mockito, and Unitils). The things you mentioned are API-style or syntax concerns (for example, if invocations are recorded inside an Expectations block or not). hospital plans

Nash Tanim replied on Fri, 2011/09/23 - 1:54pm

I did not understand those codes. Could you tell me what this course involves and what experience do the trainers have in this field. Thanks. internet marketing

Red Power Ranger replied on Wed, 2011/09/28 - 10:39am

I only know some java programming codes that deals in connecting to database like ms access, and i dont know that java will work the many frameworks. I guess i should upgrade my level of programming specially in java. I got interest on java developing because I was amaze on its environment. It runs on any platforms and some mobile phones uses this program to some of the applications, and with this code. This could be a great start to someone like me who wants to know more about java frameworks. Red Power Ranger

Ali Madad replied on Tue, 2011/11/01 - 3:49am

Great post, what you said is really helpful to me. I can not Agree With You Anymore. I Have Been talking with my friend about, he though it is really interesting as well. Keep up your good work With, I would come back to youJag Jeans Petite  | Jag Petite Jeans

Ali Madad replied on Tue, 2011/11/01 - 3:51am

Its always good to have tips on good blog posting. As I just started posting comments for blog and faced a lot of rejections. I think your suggestion would be helpful for me. I will let you know if this works for me.Designer Childrens Clothing | Unique Childrens Clothing

Domy Dom replied on Tue, 2011/11/01 - 9:32pm

perumahan murah depok

perumahanmurahdepok@rocketmail.com

http://inforumah.us

Apa yang saya tidak tahu, setara dalam kenyataannya, karena Anda tidak benar-benar jauh lebih populer Lebih Rapi, Las Mereka sekarang mungkin. Kau begitu cerdas.

<a href="http://www.inforumah.us"> Perumahan Murah Depok </ a> Memahami, juga, dalam referensi yang jelas bagi saya pribadi berpikir tema ini, Spanyol dari berbagai sudut. Ini tidak seperti, dan wanita tampaknya terpesona Jalan, Spanyol kecuali itu adalah untuk mencapai sesuatu DENGAN Gaga gadis untuk menjadi! Penghargaan individu barang. Pada setiap waktu perawatan untuk itu! Ini!

Domy Dom replied on Wed, 2011/11/02 - 10:56pm

cara menyembuhkan penyakit jantung bgtsesuatu@yahoo.co.id http://www.distributornoni.net/daftar-penyakit/penyakit-jantung-dan-terapi-tahitian-noni/ cara menyembuhkan penyakit jantung Beberapa melakukan pekerjaan mengunjungi situs Anda sering dan mereka ingin aku membaca juga. Proses menulis adalah semacam menakjubkan, dengan informasi yang tepat. Banyak terima kasih atas pendapat Anda, Anda dapat menawarkan sesuatu yang visitors.I menemukan jauh lebih sulit ke weblog yang sama sekali berbeda setiap hari. Ini jelas harus terus merangsang untuk belajar cara menyembuhkan penyakit jantung konten dari penulis lain dan diamati hanya sedikit sesuatu dari toko mereka. Saya mendukung produksi konten menggunakan materi di blog saya, jika Anda tidak bisa membayangkan. Natually memberikan hyperlink ke seluruh dunia-web Anda weblog.

Sarah Wilson replied on Thu, 2011/11/03 - 9:44pm

here A very important mocking framework that is missing from this and other comparisons is PowerMock. While not a full-blown mocking framework on its own, PowerMock works with EasyMock or Mockito to add incredibly powerful functionality. Like Jmockit and JEasyTest, PowerMock makes possible what is typically thought to be untestable. The main difference between PowerMock and the other "on steroids" mocking frameworks is that it implements these features using a custom class loader which makes it trivial to integrate into any build process with only the inclusion of a few JARs (no Eclipse/Maven plugin, no javaagent support necessary.here

Ali Madad replied on Sat, 2011/11/05 - 6:40am

This is a very intriguing post, I was looking for this knowledge. Just so you know I found your web site when I was searching for blogs like mine, so please check out my site sometime and leave me a comment to let me know what you thing.Adidas Skateboarding | Adidas Skate Shoes Review | Cheap Adidas Skate Shoes

Ali Madad replied on Sat, 2011/11/05 - 6:42am

Great blog article about the Great blog article about this topic, I have been lately in your blog once or twice now. I just wanted to say hi and show my thanks for the information provided.Commercial Playgrounds | Swing Set | Wood Swing Sets

Ali Madad replied on Mon, 2011/11/21 - 6:11am

I am hoping the same best work from you in the future as well. In fact your creative writing abilities has inspired me.Milano Dresses  | Tribal Dress Pants

Ali Madad replied on Mon, 2011/11/21 - 6:18am

This is very extra ordinary web blog.The postings in this blog are quite interesting and unique. Please add some more new posts in this blog. I am the frequent visitor to this blog.Access Tonneau CoversLorado Tonneau Covers

Mark Volansky replied on Sun, 2011/12/04 - 10:55am

Ce travail que votre message a donnée est très bon, je suis très impressionné. Xtreme NO

Tom Sowyer replied on Thu, 2012/06/07 - 6:03am

and I have got to say this, the page rocks :) and you cool! by Tom Sowyer the Cheat Codes for Sims 3 Geek

Carla Brian replied on Sun, 2012/07/15 - 5:24am

Good job on this. I really have no idea how to this with the framework. Good thing I saw this post. This might be really helpful. - James Stukcey

Indranil Majumder replied on Sat, 2013/10/26 - 11:36am

@Andrew - From 2009 to 2013 did any of your thoughts on the testing frameworks you have discussed above has changed. Do let us know?

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.