Alex has posted 1 posts at DZone. View Full User Profile

Remote Access to Your Services: EJB vs Spring/Hessian

10.26.2009
| 10397 views |
  • submit to reddit
The other day, I got to come up with a design solution for a small website. Small enough to fit on a single physical server. Nothing fancy, I had a pretty good idea on the frameworks to use and it would be a classic front-end plus back-end server deployment. With a twist, the back-end server would most likely be JBoss. And this is where things got interesting.

I was planning on using Spring on the back-end. But then I thought, how much sense does it make to fire-up a full-blown JBoss, only to use Spring on top of it? Maybe EJB would be a better pick in this instance? Specifically, I was thinking about the communication between the two servers: has RMI-IIOP evolved over time enough to best Hessian? A good software engineer would know the specs for both and have the answer already. So I had to come up with something else.

I quickly wrote two services, one for Spring, one for EJB. And a client for both. It would do a number of calls in a loop and time them - for both services. The services would return a char[] of various sizes, so that I could see the effect of various payloads (0, 1024, 2048, up to 16384). True, it's not a perfect real-world scenario if I don't hit the DB and perform some business logic in the process, but I'm trying to benchmark the protocol implementations here.

Still nothing fancy. So what's this article about, then? Well... the results!

This is what I got for the first runs:

                      Run 1 - Time [s]        Run 2 - Time [s]
Calls Payload EJB Spring EJB Spring
===============================================================
1,000 0 3.652 .836 3.628 .754
1k 2.473 .694 2.679 .633
2k 2.143 .653 2.291 .634
4k 40.547 .611 40.242 .766
8k 40.311 1.102 40.335 1.251
16k 25.658 1.725 27.965 1.672
---------------------------------------------------------------
10,000 0 17.218 3.218 16.751 3.413
1k 11.919 2.236 12.017 2.279
2k 11.205 2.454 11.280 2.717
4k 401.615 4.618 401.225 4.655
8k 401.252 7.999 401.645 8.248
16k 91.225 12.545 78.676 12.762
---------------------------------------------------------------
100,000 0 112.856 13.395 111.962 14.113
1k 111.384 19.956 111.236 22.019
2k 109.806 26.476 107.961 27.201
4k 4019.744 51.204 4019.709 38.711
8k 4025.306 70.093 4013.178 68.229
16k 1094.349 123.929 1044.610 124.129

Run 1 and 2: Kubuntu 9.10 beta, kernel 2.6.31 x86_64, SunJDK 1.6.0

So there's my answer: Spring/Hessian is faster than remote EJB. Even CPU usage favours the Spring/Hessian combo. For Spring/Hessian, typical client:server load ratio is about 3:1, while for EJB it was between 1:1 and 1:2 most of the time. More specific, Spring puts about 30% CPU load on the client and 10% on the server; at the same time EJB loads about 25% - 25% (client load can dip below, server load can peak over 30%).

But while Spring exhibits lower response time and a nice, almost linear scaling, what about those whacky EJB numbers? I specifically ran this tests twice to confirm them.
For payloads up to 2k, the graphs are almost flat for both Spring and EJB, but there some nasty spikes for EJB at 4k and 8k payloads. Also strange, during 4k and 8k EJB calls, the CPU load was virtually 0 for both client and server.

Looking for bugs in JDK, I switched to OpenJDK next.

                      Run 3 - Time [s]
Calls Payload EJB Spring
=======================================
1,000 0 3.599 .870
1k 2.522 .664
2k 2.469 .590
4k 40.191 .680
8k 40.220 1.108
16k 26.832 1.671
---------------------------------------
10,000 0 16.076 3.029
1k 11.712 2.229
2k 10.673 2.876
4k 401.004 4.669
8k 401.069 7.815
16k 93.677 12.297
---------------------------------------
100,000 0 111.863 13.325
1k 107.690 19.045
2k 106.736 25.726
4k 4016.907 40.947
8k 4023.713 72.376
16k 902.372 125.865

Run 3: Kubuntu 9.10 beta, kernel 2.6.31 x86_64, OpenJDK 1.6.0

OpenJDK results are virtually identical. Must be either JBoss or the OS causing this.

Next, some Windows numbers, let's see if it's the OS.

                      Run 4 - Time [s]        Run 5 - Time [s]
Calls Payload EJB Spring EJB Spring
===============================================================
1,000 0 6.246 1.350 3.531 .781
1k 5.623 1.174 2.703 .704
2k 4.594 1.433 2.516 .718
4k 6.354 1.710 2.375 .891
8k 5.852 3.237 2.562 1.281
16k 6.729 3.996 2.563 1.953
---------------------------------------------------------------
10,000 0 41.604 6.824 23.797 4.266
1k 23.960 6.534 22.250 3.906
2k 32.041 9.198 22.406 4.781
4k 37.564 9.312 22.391 6.657
8k 23.752 10.246 22.421 10.547
16k 27.492 24.511 N/A 18.109
---------------------------------------------------------------
100,000 0 371.700 42.819 N/A 30.266
1k 267.341 59.343 N/A 39.640
2k 353.062 83.713 N/A 48.610
4k 335.661 90.985 N/A 67.265
8k 294.875 141.625 N/A 105.719
16k 306.000 270.254 N/A 182.906

Run 4: Win7RC/Virtualbox x86_64, SunJDK 1.6.0
Run 5: WinXP x86, SunJDK 1.6.0

Ok, no more spikes in EJB response time, though the times still look suspiciously flat. Plus, on WinXP, past 8k/1,000 calls, I just get an exception (Exception in thread "main" org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for InvokerLocator [socket://127.0.0.1:3873/]). Still, I am using the same JBoss revision (5.1.0GA-jdk6, btw), so it seems JBoss is not causing those spikes. This leaves me with the OS.

Back to Linux.

                      Run 6 - Time [s]        Run 7 - Time [s]
Calls   Payload         EJB      Spring         EJB      Spring
===============================================================
  1,000      0        2.996        .779       3.314       1.012
             1k       2.553        .617       2.462        .658
             2k       2.568        .843       2.301        .900
             4k      42.797       1.015      43.080       1.002
             8k      42.122       1.639      43.046       1.369
            16k       5.507       2.489      22.103       2.401
---------------------------------------------------------------
  10,000     0       21.182       3.502      22.448       4.074
             1k      19.669       4.604      22.458       6.292
             2k      19.714       5.975      22.794       8.794
             4k     423.331       8.382    2590.024      10.251
             8k     421.513      14.162    2597.836      14.667
            16k      29.179      26.055     132.132      21.854
---------------------------------------------------------------
100,000      0      188.130      26.938     221.967      53.123
             1k     189.032      42.154     224.095      67.212
             2k     189.972      55.225     226.334      86.580
             4k    4184.557      78.404   21679.748     103.160
             8k    4169.555     136.656         N/A         N/A
            16k     306.762     236.289         N/A         N/A

Run 5: Kubuntu 9.10 beta/Virtualbox, kernel 2.6.31 x86, SunJDK 1.6.0
Run 6: CentOS 5.3/Virtualbox, kernel 2.6.18 x86, SunJDK 1.6.0

Switching to 32bit, the spikes are live and kicking - this is not an architecture problem either. Is it a kernel bug? This project is supposed to be hosted on CentOS 5.3, so I tested that next (and last). Spikes are back with a vengeance (~5x higher now). I stopped testing after reaching ridiculous values (~6h for 100,000 calls).

At this point, I can only conclude that this is either a long-standing bug in the Linux kernel or a misuse of kernel API by the JVM implementations - I know I used two different JVMs, but they are very similar nonetheless. Testing with JDK 1.5.0 would be probably better, but I did not have the time to do that, too.

What can we learn from all these? A lesson we already know: never assume, always test. At least in this case, I believe the findings were pretty unexpected. Should I have just gone ahead with EJB and done some profiling on the final application, I would have only seen an application that does not scale well past a certain point (or unexpected connection failured, had I been targeting WinXP). And I would have been lucky to be able to fix that after a lot of working hours, if at all.


PS: Attached, you will find an archive of the test code, should you wish to confirm my findings or test additional combos.
PPS: I know you can always write another article about how sloppy the code is, please don't bring that to my attention. It's good enough to serve its purpose ;-)
Legacy
Article Resources: 
Published at DZone with permission of its author, Alex Mateescu.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Michal Hlavac replied on Mon, 2009/10/26 - 6:59am

You cannot compare EJB with Spring. EJB is specification. This comparison is about JBoss EJB implementation vs. Spring. Did you try run this test scenario on other open source ejb implementations such as Geronimo or Glassfish?

Daniel Cullender replied on Mon, 2009/10/26 - 7:52am

Could you also please provide your configuration for the EJB as well (ejb session pool size etc). The configuration on the server can have a dramatic effect on the performance.

Bruno Borges replied on Mon, 2009/10/26 - 9:29am

Absolutely, Running the tests with different JVMs or OS does not mean the benchmark was run with all possible forks. For EJBs what it has to be considered is the implementation and the configuration. EJB defaults and one server (JBoss) can't be taken as a "de facto" environment.

Alex Mateescu replied on Mon, 2009/10/26 - 3:43pm in response to: Michal Hlavac

You are correct, I didn't choose the title very well. However, the article does not try to compare the technologies/implementations. It's just an analysis of a real-life situation and is meant to emphasize the importance of testing and profiling.

 I did not have time to test other servers, because it takes a while to do a complete run, but I have provided the sources, should anyone want to look any deeper.

Alex Mateescu replied on Mon, 2009/10/26 - 3:46pm in response to: Daniel Cullender

Standard JBoss configuration (30 instances pool size?) was used. I don't believe this to be the issue, since the same exact configuration performed very differently on Windows than it did on Linux. Still, without confirmation, it may be.

Nathan Dolan replied on Tue, 2009/10/27 - 9:31am

You are comparing apples and pears. If you want distributed transactions, container managed security, clustering/pooling/failover, integration with CORBA objects, etc etc, then Hessian might not be the best option. If you don't need enterprise functionality like that then it makes more sense to go for a lightweight option like Hessian.

adam bien replied on Wed, 2009/10/28 - 7:42am

Hi Alex, it is also my observation, that hessian performs better, than a usual IIOP protocol. Nothing prevents you to use EJBs with Hessian. Then the performance should be similar to Spring. Comparing it to real, unmanaged POJOs, EJBs 3 are only about 3% slower (measured with Glassfish v2). But you could easily and directly expose an EJB 3 over REST or SOAP as well. The performance is dependent on the protocol, not the component model, regards, adam

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.