Comparison of the Grid/Cloud Computing Frameworks (Hadoop, GridGain, Hazelcast, DAC) - Part I

Hello Krystian

Thanks for this article, waiting soon the second part

and increible the high value for GridGain in your statistic graphic

 

Regards

 

-Manuel

DAC test was based on the usage_of_the_CMBF sample from the DAC distribution. You can grab it from the download page.

 

However, in order to be consistent, I have also added it to the http://dacframe.org/lab, so now all java sources used in the tests can be downloaded from the lab repository. Sorry for misleading you.

Geez...

Just spent 15 mins and rewrote your test in simplified form using Executor service (instead polluting 5 nodes with excessive split). Got results totally in line with the rest of the frameworks. Case's closed.

Guys - next time you do something like that please contact the vendor. We even provide free support for situations like yours :) Amount of features and configurations in GridGain is fairly substantial comparing to other frameworks and despite our best attempts they are not always obvious or easily understood. In your case - you are hitting a known edge-case problem with "siblings" explosion for which there's a well-known solution and which we'll optimize even further in the next release.

Best,
Nikita Ivanov.
GridGain - Cloud Development Platform

Hi Nikita, Is this not the issue I reported in Oct 2009 at a customer site that was performing a similar benchmark? If so then you might forgive them a little here though I think they had a duty to contact you prior to publication due to the severity of the issue. Kind regards, William

I have updated the article and the content on the website with the results for GridGain using Executor service.

 

@William: yes, the problem is the same. It's an edge-case but nonetheless we'll be addressing it in 3.0. @Kilder: thanks for re-running the test. I would also suggest to show % delta between results even though they are negligible the chart is not very informative (or put on log scale). Thanks, Nikita.

For GridGain team

 Hello Guys, just some doubts

 1) some plan to write a book?,  It would be great really

 2) Dmitri, I have read in your blog GridGain Vs. Hadoop (Continued)

     the date is old "May 7, 2008", perhaps GridGain would do a new comparison?

 

Best Regards

-Manuel

 

 

 

Hello Dmitriy

Thanks for the reply

I hope see the book with a good support integration for Spring and Hibernate
I will waiting carefully such comparison, let know us via Dzone to be stay tuned!

I want ask for a favor, I have seen the documentation section in GridGain,
would be possible give such information in offline approach? (pdf), because read from
the monitor kill the eyes and we have not option to write our own anotations/observation
in some place, like a notebook

Best Regards

-Manuel

I could easily be missing something here. If not...

Here is my problem with the test: This happens to be a weak implementation of the algorithm in question.

That isn't always a big deal, but it is definitely one when examining optimized performance. It just does not necessarily show the important and relative distinctions between the systems when not using code similar to what would be used in production or experimentation.

To highlight this, let me give a system that dominates every one here: A single light, old machine with a tweaked version of the code.

This is a machine that is years old and far weaker than any of the nodes in the experiment here. 2 CPUs, 2 cores each; 1GB of memory.

n = 4 level = 1000: 195.6 seconds
n = 4 level = 10000: 168.1 seconds
n = 4 level = 100000: 165.4 seconds


On a somewhat more modern machine - 4 CPUs (quad-core) - although still a year old, and only using 100MB of memory, I get:

n = 4 level = 1000: 73.0 seconds
n = 4 level = 10000: 64.0 seconds
n = 4 level = 100000: 62.8 seconds


I'm a little afraid you may have described the test inaccurately, or I may have misunderstood it. But, this is the packaged code run with the specified parameters. On a single machine, it is almost 5x faster than the whole cluster.

Am I just confused on something?

I'm not saying the distributed systems couldn't also be made faster. However, you really want to be benchmarking on something realistic. Otherwise, your comparisons may be skewed or disproportionate compared to real workloads.

Hi Robert

Your results look a little bit strange :)
As you can see at Grid Comparison - cumulative cost
 counting CMBF should take around 11 888 seconds on a single CPU.

Could you please send me your version of the code?

 Hmm, I am a little confused, still. What is the 20 seconds for? Is that running on a single node? If so, it would say *your* machine is 3 times faster. Is it modified to be distributed by one of the systems?

  I doubt that either of the machines I tested with is actually 3 times as fast. They aren't that great. However, various aspects of the context can make a big difference - the JRE version, an optimized OS, the options used when starting the JVM. What is your precise setup, in terms of these and the processor model?  

  In any case, I think I do get your intent, but I would still push my assertions. It remains true that an optimized version of the code provides much faster results using half a node than the original code does with the best distributed system across the cluster of five nodes. So, I think it is important to note that jumping to any distributed system early on is often not an efficient way to speed up high performance computing - you really want to optimize your processes *first*. You may even change your mind about how and what to distribute. 

(It also is relevant to ask, "How does this compare to spending resources to optimize the process?" That isn't free, and it is often the tradeoff under consideration. There definitely are tasks where optimizing is of little use - such as web crawling - but these are not compute-intensive.)

But, more core to your task, using a realistic test case is important. It can change the relative overhead and the comparisons of the systems. There is a reason that you didn't just spin in a sleep loop as your test; you want something that reflects real workloads.

Now, the N=9 is interesting... That would likely take quite a while. Might be worth even optimizing further...

Ah. So, assuming DAC is perfectly efficient, you measured 100 machine-seconds in total. That is squarely between the two machines I reported, which makes some sense. (Especially since my slow one was 4 core, and the fast one was 16-core.) What do you see on a single one of those nodes? Also, are you running 64-bit?

I wouldn't use the default settings for Java. They won't be ideal for high-performance tasks. I would also make sure you are using 1.6.0_18. The options do matter - setting the memory can be very important, and compressed-oops can be substantial. And, of course, using the 64-bit JVM will help here. Using the server mode is critical, too, although you may be defaulting to that.

I wasn't clear when I said "using a realistic test case is important". I was asserting that that is what you are doing here (as opposed to something very synthetic). The only thing I was adding was that you want to test not just a meaningful algorithm but an optimized version of it, because that is what you would use in practice.

I do wish there were more real test cases around, but people don't seem to share them much... And Hadoop at least has never had much focus on benchmarking...

I like the straightforward nature of this experiment. I think it makes a clear statement that in the case of an easily-parallelized, low-data, CPU-intensive task, the distribution framework should not get in the way - performance should be close to the ideal. I think it is a good balance for the common tasks like terasort, which in reality are all about shufflinf data around.

I was understating it a little when I said you might want to optimize more for N=8. I am confused about your numbers, though. For 9, I would use 8? And 7 takes 0.5 seconds?

Thanks for adding the interesting detail to the writeup...

The E5410 certainly supports x64. You should definitely be using a 64-bit OS on these machines for any performance testing. With the compressed-oops support, I would also propose that the 64-bit JVM is best for most performance testing, although you could try both for a task. Even if you used a 32-bit JVM, it should still be on a 64-bit OS - I wouldn't support a 32-bit OS on a server machine for many years.

The current JRE is important because of improvements, although it is more critical with a Nehalem machine.

I thought your 'level' parameter controlled the division into subtasks. You are saying that is actually the 'n' parameter? That doesn't seem consistent with the output. Then how do you configure the program for M(7), M(8), M(9)?

The part II is now available.