Dmitriy Setrakyan manages daily operations of GridGain Systems and brings over 12 years of experience to GridGain Systems which spans all areas of application software development from design and architecture to team management and quality assurance. His experience includes architecture and leadership in development of distributed middleware platforms, financial trading systems, CRM applications, and more. Dmitriy is a DZone MVB and is not an employee of DZone and has posted 57 posts at DZone. You can read more from them at their website. View Full User Profile

GridGain: To Split Or Not To Split

05.22.2008
| 4604 views |
  • submit to reddit

GridGain - Grid Computing Made SimpleWhen designing your tasks for execution on the grid, you need to decide whether to split your task into smaller jobs for parallel execution, and what the size of your split should be.

When Not To Split

People often think of a grid as of infrastructure for execution of long running tasks. That is not always the case. Compute grids can add a lot of value even for quick, short running jobs. Imagine, for example, that your application constantly needs to calculate in real time a bunch of statistical metrics and averages on a financial portfolio, let's say for displaying them on UI. Although every single calculation may not take too long, having all calculations performed concurrently on the same server or a thick client can bring it to its knees fairly quickly. A good solution would be to take every calculation and execute it on a separate grid node.

The example above is a good case for when not to split your tasks. When deciding whether to split or not, you should take into consideration the time of local execution. If your task can execute locally fast enough, then don't split it and run it on the grid as a whole. By doing that you get the following benefits:

  • You remove a single point of failure. If a grid node crashes in a middle of calculation, then GridGain will automatically fail it over to another node.
  • You balance the load across your grid nodes. GridGain will automatically load balance your jobs to the least loaded nodes. You can also turn on job stealing and have less loaded nodes steal jobs from more loaded nodes.
  • You improve overall scalability of your application. Now you can add or remove grid nodes on demand whenever your application load peaks or slows down and, hence, keep the response times constant regardless of the load. For example, you can configure your grid to include more nodes into topology as application load grows.
  • You get GridGain's simplicity. Here is how simple it can be to execute some portfolio calculation on the grid. Note that all you have to do is attach @Gridify annotation to your Java method and that's it!
@Gridify 
public double calculatePortfolioPosition() {
...
}

 

How To Split

Now, let's say you really need to split your task into smaller jobs in order to speed up execution. A good formula to decide on what the size of your split should be is to take the time your task takes to execute locally and divide it by the time you would like to achieve. So if your task executes in 2 seconds and you would like to achieve 100 milliseconds, then the number of jobs your task should split into should be (2000 ms / 100 ms = 20). In reality, the execution time will be slightly more than 100ms as most likely your jobs will not be absolutely equal, and there is also a slight network communication overhead.

That is not to say that for this example you would only need to have 20 nodes in the grid. Ideally you should have as many nodes as your application needs in order to handle the load - let GridGain pick the most available 20 nodes for execution of individual jobs within your task.

For more information visit GridGain Wiki or watch Grid Application In 15 Minutes screencast.

References
Published at DZone with permission of Dmitriy Setrakyan, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)