Olivier has posted 16 posts at DZone. View Full User Profile

Measure Code Coverage by Integration Tests with Sonar

11.09.2010
| 9889 views |
  • submit to reddit
You probably know already that JaCoCo is the most performant code coverage engine. But you might not know that you can now combine it with Sonar to assess the code coverage by integration tests. This was the most voted Sonar issue (SONAR-613) and the latest version of the Sonar JaCoCo Plugin solves it. I am now going to explain how.

Let's first come back to the needs :
  • There should be an easy way to assess code coverage by integration / functional / acceptance / UI tests (let's say integration tests), whatever system is used to execute those tests : Maven plugin (maven-surefire-plugin, maven-osgi-test-plugin, maven-failsafe-plugin...), Ant script, GreenPepper, Selenium... and also whatever the package format : jar, ear, war... This can really be tricky and even a nightmare with coverage engines that use source code instrumentation (Clover) or offline bytecode instrumentation (Cobertura or Emma). Indeed an application is usually made of multiples java libraries and doing static instrumentation of each java libraries is often too complex to automate. The only suitable approach for this usage is to do on-the-fly byte code instrumentation like JaCoCo does.
  • Code coverage by unit tests and code coverage by integration tests must not be mixed as the information provided by each one is not the same. For instance, having a very good code coverage by integration tests but a poor code coverage by unit tests is a good start but not really a good news for the productivity of the development team. On the other side, having a very good code coverage by unit tests but a poor code coverage by integration tests will not prevent regressions to appear in production environment. This is why there should be a new set of dedicated metrics for integration tests.
  • For multi-modules Maven projects, there should be a way to compute overall code coverage by unit tests across modules. This case clearly lies between unit tests and integration tests but this is widely used and a lot of users have requested the feature.

The Sonar JaCoCo plugin covers the 3 needs in two steps :
  • Launch your integration tests after having configured the JaCoCo agent to be launched with the integration tests and to dump a Jacoco result file at the end of the execution
  • Configure and launch Sonar to reuse the JaCoCo result file. The Sonar plugin will extract only the required code coverage information

Let's take an example with a Maven project containing three modules A, B and C. The C module is only used to execute integration tests with the Maven Failsafe Plugin and we would like to get the code coverage on module A and B by integration tests contained in module C. The following line must be added to the configuration of the Maven Failsafe Plugin in the pom.xml file of the C module :
<argLine>-javaagent:${jacoco.agent.path}=destfile=${jacoco.file.path}</argLine>

The integration tests on module C must then be launched with the following command line :
mvn -Djacoco.agent.path="PATH_TO_AGENT" -Djacoco.file.path="PATH_TO_DUMP" -Prun-its clean install

After having run integration tests you can execute Sonar analyzes on the overall project. Note that you still can use Clover, Cobertura or Emma to assess code coverage by unit tests together with JaCoCo for integration tests. mvn -Dsonar.jacoco.itReportPath="PATH_TO_DUMP" sonar:sonar And here is what you will get in Sonar in addition to code coverage measures by unit tests :

Of course, it's possible to drilldown to the source code in order to see which lines are covered or uncovered by integration tests :


To get the code coverage by unit tests across all Maven modules, the configuration steps are pretty similar :
  • Add the argline property to your Maven Surefire Plugin configuration
  • execute mvn clean install -Djacoco.agent.path="PATH_TO_AGENT" -Djacoco.file.path="PATH_TO_DUMP"
  • execute mvn -Dsonar.jacoco.itReportPath="PATH_TO_DUMP" sonar:sonar"

It is now time for you to give a try to new version of Sonar JaCoCo Plugin. We are waiting for your feedback !
References
Published at DZone with permission of its author, Olivier Gaudin. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Esko Luontola replied on Tue, 2010/11/09 - 3:21pm

Will this work when the integration tests launch the system-under-test (SUT) in a new process? If I give the new process the same -javaagent:${jacoco.agent.path}=destfile=${jacoco.file.path} arguments, will it work when there are two processes (integration tests and SUT) which write to the same destfile?

In my current project there are multiple application modules, a couple distribution modules (producing a launcher JAR, a Java agent for AOP, and a distributable ZIP which packages together all those and the application modules) and a module for end-to-end tests. The end-to-end tests are run using JUnit and they unpack the distributable ZIP file and launch it as the SUT using a command like this: java -javaagent:my-aop-agent.jar -jar my-launcher.jar --other-args

Also, does the JaCoCo agent write any coverage data to the destfile immediately or asynchronously? I'm writing my project as crash-only software and all tests stop the SUT by forcibly killing it at the end of each test (or in the middle, if we are testing recovery). So writing the file should not be buffered or asynchronous, or else some coverage data might be lost.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.