Comparison of the Grid/Cloud Computing Frameworks (Hadoop, GridGain, Hazelcast, DAC) - Part II

What would happen, if 60% of your cloud suddenly goes down? Can you rely on the 'fail-over' capabilities of the framework of your choice? What about consistency of your data? How big would be the performance impact of node failures? Continuing our experiments from the previous article, we have decided to give a try the following frameworks:

• Hadoop
• GridGain
• Hazelcast
• DAC

As always, we describe all the methods and results and give you access to all the sources. You should be able to repeat all the tests and receive the similar results. If not, please contact us, so we can revise our report. Moreover, taking into consideration comments from the previous article, we have added detailed cpu/memory/network usages from all the machines in our test environment.

You can find all sources used during tests in our code repository. Moreover, full report with detailed results is available on our website.

This is part II of our comparison, where we concentrate on the fail-over capabilities. There were serious node failures (up to 60% nodes went down) and multiple transactions rollbacks.

Test environment

Our test environment consisted with 5 machines (named intel1 - intel5), each one with dual Quad-Core Xeon E5410 2.33GHz, 4GB RAM on board, which gave us 40 processing units. The only difference between current test environment and the one used in part I of our comparison is the JVM version, which has been updated to 1.6.0_18.

Methodology

We based our benchmark on the same mathematical problem as in part I of our comparison. Because of that, we can easily compare results from both tests, which gives us more wider view on the given frameworks.

In this 'fail-over' comparison we used only one test scenario:

• compute problem divided into 2705 tasks (CMBF with arguments: n = 4, level = 10000)

We have simulated node failures during our tests in the following order:

• intel4 went down after 60 seconds from the beginning of computations
• intel3 went down after 180 seconds from the beginning of computations
• intel2 went down after 300 seconds from the beginning of computations

All tests were repeated ten times in order to avoid measuring error.

Results - overview

We have compared the following aspects:

• average time of computations

   

  

   

• cumulative cost (time of computation multiplied by the amount of available processing units)

   

  

   

• cumulative cost – difference with CMBF (difference with the optimal solution: single-threaded version of CMBF)

   

  

   

• total CPU usage

   

  

   

  

   

• maximum memory usage

   

  

   

• total network usage

   

  

   

You will find the detailed methodology (sources, test environment description) and results (all performed test cases with std deviation and average values) on our website.

Average CPU usage (%user) gathered on all machines:

Average CPU usage (%system) gathered on all machines:

Average memory usage gathered on all machines:

Average network usage (received bytes/s) gathered on all machines:

Average network usage (transmitted bytes/s) gathered on all machines:

Summary

The above part II concentrates on the fail-over capabilities. All frameworks properly handle node failures, but we had to slightly modify our code for Hazelcast to catch new exceptions (other frameworks resubmit invalid tasks by default). Taking the above results into consideration, we can infer the following conclusions:

• Hazelcast and GridGain are the best choice for an easily-parallelized, low-data, CPU-intensive tasks. Moreover, they are even better choice, when some unexpected node failures can happen.
• Hazelcast consumes the smallest amount of CPU and network bandwidth
• GridGain consumes the smallest amount of memory
• Hadoop was designed to manipulate large data sets, so the above not the best results are totally understandable
• DAC with its default settings do not handle node failures efficiently