Hacking on GraphHopper - a Java road routing engine. Peter has posted 62 posts at DZone. You can read more from them at their website. View Full User Profile

Microbenchmarking Java - Compare Algorithms

  • submit to reddit

There are a lot of microbenchmark tips out pn the internet. The intent of them differs from author to author.Today I want to show how you could compare e.g. different algorithms. You could simply do:

int COUNT = 1000000;
long firstMillis = System.currentTimeMillis();
for(int i = 0; i < COUNT; i++) {
System.out.println("Mean runtime of algorithm in seconds:"+(System.currentTimeMillis()-firstMillis) / 1000.0 / COUNT);

There are several problems with this approach, which results in very unpredictable results:

  1. Make sure the variable COUNT is high enough (if runAlgorithm() is unexpensive). Then make sure you run this code several times. Additionally you should measure the RMS either for each runAlgorithm call or at the end of this code snippet for a better comparison with other algorithms.
  2. You should turn off the JIT-compiler with specifying -Xint as a JVM option, otherwise your outcome could depend on the (unpredictable) JIT and not on your algorithm.
  3. You should start with enough memory, so specify e.g.-Xms64m -Xmx64m. Because JVM memory allocation could get time consuming.
  4. Quit all other applications or at least you should use
    long firstNanos = mx.getCurrentThreadCpuTime();
    ThreadMXBean mx = ManagementFactory.getThreadMXBean();

    instead of

    long firstMillis = System.currentTimeMillis();
  5. Avoid that the garbage collector runs while your measurement: -Xnoclassg
    But make sure you have enough memory!

After this I got relative stable results: The difference of maximal and minimal value for the result is less then 4% after 10 runs.

From http://karussell.wordpress.com

Published at DZone with permission of its author, Peter Karussell.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)


Thomas Mueller replied on Fri, 2009/05/22 - 6:43am

> You should turn off the JIT-compiler

That's a strange idea. Compiled code has a complely different performance characteristic than interpreted code. Instead of disabling the JIT-compiler, run the benchmark multiple times and ignore the first run. The JIT is not _that_ unpredictable.

> Quit all other applications

No need to quit, just make sure they don't use the CPU. And make sure you have enough RAM.

> getCurrentThreadCpuTime

Did you know this doesn't include Thread.sleep(..), and doesn't include the time used by threads that were created by this algorithm?

> Avoid that the garbage collector runs while your measurement: -Xnoclassg

Disabling GC skews the result: it help those algorithms that generate a lot of garbage. -Xnoclassg only disables class garbage collection by the way.

Michael Astreiko replied on Fri, 2009/05/22 - 6:52am

Problem with performance tests was well enough discussed below: http://java.dzone.com/articles/why-many-java-performance-test

Peter Karussell replied on Fri, 2009/05/22 - 9:26am

Please see the comments on the original article at http://karussell.wordpress.com

  This article was not intended to be posted at dzone :-)

Dist Iller replied on Mon, 2009/05/25 - 5:24pm

Wow, the qualityof javalobby posts has dropped to new lows

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.