DevOps Zone is brought to you in partnership with:

Nicolas Frankel is an IT consultant with 10 years experience in Java / JEE environments. He likes his job so much he writes technical articles on his blog and reviews technical books in his spare time. He also tries to find other geeks like him in universities, as a part-time lecturer. Nicolas is a DZone MVB and is not an employee of DZone and has posted 232 posts at DZone. You can read more from them at their website. View Full User Profile

Introduction to Mutation Testing

  • submit to reddit

Last week, I took some days off to attend Devoxx France 2014 3rd edition. As for oysters, the largest talks do not necessarily contain the prettiest pearls. During this year’s edition, my revelation came from a 15 minutes talk by my friend Alexandre Victoor, who introduced me to the wonders of Mutation Testing. Since I’m currently writing about Integration Testing, I’m very much interested in Testing flavors I don’t know about.

Experience software developers know not to put too much faith in code coverage metrics. Reasons include:

  • Some asserts may have been forgotten (purposely or not)
  • Code with no value, such as getters and setters, may have been tested
  • And so on…

Mutation Testing tries to go beyond code coverage metrics to increase one’s faith in tests. Here’s is how this is achieved: random code changes called mutations are introduced in the tested code. If a test still succeed despite a code change, something is definitely fishy as the test is worth nothing. As an example is worth a thousand words, here is a snippet that needs to be tested:

public class DiscountEngine {
    public Double apply(Double discount, Double price) {
        return (1 - discount.doubleValue()) * price.doubleValue();

The testing code would be akin to:

public class DiscountEngineTest {
    private DiscountEngine discounter;
    protected void setUp() {
        discounter = new DiscountEngine();
    public void should_apply_discount() {
        Double price = discounter.apply(new Double(0.5), new Double(10));
        assertEquals(price, new Double(5));

Now, imagine line 16 was forgotten: results from DiscountEngineTest will still pass. In this case, however, wrong code updates in the DiscountEngine would not be detected. That’s were mutation testing enters the arena. By changing DiscountEngine, DiscountEngineTest will still pass and that would mean nothing is tested.

PIT is a Java tool offering mutation testing. In order to achieve this, PIT creates a number of alternate classes called mutants, where the un-mutated class is the initial source class. Those mutants will be tested against existing tests targeting the original class. If the test still pass, well, there’s a problem and the mutant is considered to have survived; if not, everything is fine as the mutant has been killed. For a single un-mutated class, this goes until the mutant gets killed or all tests targeting the class have been executed and it is still surviving.

Mutation testing in general and PIT in particular has a big disadvantage: the higher the number of mutants for a class, the higher the confidence in the results, but the higher the time required to execute tests. Therefore, it is advised to run Mutating Testing only on nightly builds. However, this cost is nothing in comparison to having trust in your tests again…

Out-of-the-box, PIT offers:

  • Maven integration
  • Ant integration
  • Command-line

Also, Alexandre has written a dedicated plugin for Sonar.

Source code for this article can be found in IntelliJ/Maven format there.

Published at DZone with permission of Nicolas Frankel, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)


Amar Mattey replied on Mon, 2014/04/21 - 12:51pm


Good article and interesting concept. It might be helpfull for others if you publish the results with and without the assert statement.


Amar M

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.