Performance Zone is brought to you in partnership with:

Stephen has posted 1 posts at DZone. View Full User Profile

ICEfaces Performance Report - Corrected

07.07.2009
| 16002 views |
  • submit to reddit
The purpose of this article is to present accurate performance results for the ICEfaces RIA framework, correcting gross errors originally published in the article Performance Report of Server Side RIA Frameworks.  The author of the original article was negligent in his application of the test methodology to ICEfaces, and published completely erroneous results for the ICEfaces framework.

Problems With the Tests

All three tests follow a similar methodology that involves loading the system by repeatedly refreshing a page in the application under a variable number of simulated users. Strictly speaking, this test does not exercise the Ajax functionality of the various Ajax frameworks, since the majority of Ajax interaction does not take place via full-page refresh. For this reason, the latency measurement in this test should be interpreted as the latency of the initial page view under load, not as typical interaction latency. The heap measurement is likely similar to steady-state heap use under the number of simulated users, but it does not take into account different memory use patterns during Ajax interaction or input processing. Furthermore, because the simulated clients interact with the server so frequently, more working memory is required in the heap than may be necessary to handle a real-world load. This allows the load test to complete more rapidly, but if you are load testing a production application this would not be a recommended approach, as it does not simulate realistic delays between user requests.

Despite the limitations and lack of scope existing in these tests, they do exercise the initial page load characteristics of the framework, provided that they properly simulate real-world client interaction with the application.  If a simulated browser simply sends a series of GET requests, while neglecting to send messages generated for window onunload(), it appears to the Ajax framework as if the user has opened, and expects to interact with, a great many separate browser windows.  ICEfaces includes logic to clean up resources related to window onunload() events that should occur with each page refresh, but the test methodology did not account for this.  The problem is compounded by misconfiguration of ICEfaces for this type of test.  ICEfaces includes a feature called Concurrent DOM Views that supports opening multiple windows or tabs onto the same ICEfaces application, maintaining independent state for each view, and sharing a single Ajax Push connection between all views.  The original test configuration for ICEfaces was concurrentDOMviews="true", but under the test conditions every GET request created a new view that would consume resources for the duration of the test, as the necessary onunload() events never occurred.  This is the root cause of the erroneous results originally reported.

Test Configuration - Corrected

In order to conduct the tests properly we worked with the ICEfaces 1.8.1 release, and set concurrentDOMviews="false" to compensate for the missing onunload() events.  This ensures that the ICEfaces framework only maintains one unique view for each simulated test client.  We also took the liberty of testing two different configurations of ICEfaces related to maintaining the state of the server-side DOM.  ICEfaces supports DOM compression using Fast Infoset XML compression to reduce server-side memory use between requests, so you can see the comparative trade off between memory and computation.  The rest of the test configuration is as follows.

Hardware

•    MacBook Pro
•    CPU: Intel Core 2 Duo 2.2 GHz
•    Memory: 2GB

Software

•    ICEfaces 1.8.1
•    JDK 1.6.0_07
•    Apache Tomcat 6.0.18
•    Jmeter 2.3.2

Tomcat configuration

•    Session timeout: 3 minutes
•    CATALINA_OPTS: -Xms256m -Xmx1600m
•    maxThreads: 1000
•    acceptCount: 100

Source code from the original article can be downloaded from the following url: https://zkforge.svn.sourceforge.net/svnroot/zkforge/trunk/zkTest

Measurements

Latency measurements represent an average response from measurements gathered using JMeter. The HTTP Client sampler is configured with a Response Timeout of 30 seconds, as response times in excess of that are far from usable. Memory measurements are obtained by fetching a simple JSP page after each test phase that returns Runtime.totalMemory() and Runtime.freeMemory() after garbage collection.

Test Results

We simply present our ICEfaces test results and compare with the originally published results for ICEfaces to illustrated the gross inaccuracies in the original publication.  In the case of memory usage, it was necessary to use a logrithmic scale to even illustrate the accurate and errored results on the same graph.

Test 1: Simple Form




Test 2: Grid 15



Test 3: Grid 150


In our testing this example became unstable for all frameworks as you approach 1000 threads, so results are truncated at 700 threads.





Concluding Comments

The results presented here clearly illustrate that gross errors occured in the orginal testing of the ICEfaces framework, and that the conclusions drawn by the author of the original article are completely compromised.  We are certain that the Java EE community is better servered when accurate results are presented, and trust that the data provided here will allow Java EE developers to make more informed decisions when selecting technologies for RIA development.

And with regard to the original articles proposition, "This could be a battle between comfort, and standards.", it is clear that standards-based approaches are comparably performant to one particular propriatary approach, but offer much greater community adoption and support.  You will have to decide for yourself which community offers the greatest "comfort" for your development efforts.
AttachmentSize
grid15-memory.png17.28 KB
grid15-latency.png19.58 KB
form-memory.png16.79 KB
form-latency.png18.44 KB
grid150-latency.png17.54 KB
grid150-memory.png14.37 KB
Published at DZone with permission of its author, Stephen Maryka.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Cody A_ replied on Tue, 2009/07/07 - 10:47am

Very nicely put. This kind of correction makes me appreciate those developers out there who care about the accuracy of their posts. Some of us use performance reports for major decision making...

Rainer Eschen replied on Tue, 2009/07/07 - 3:19pm

Stephen thanks for the numbers. Although, the original article already showed that is was part of a FUD campaign. Every serious tester would have asked the framework vendor why the test showed these strange results.

Finally, the original article will become bad reputation for the ZK team. Well done guys ;-).

Otengi Miloskov replied on Wed, 2009/07/08 - 12:36am

God damn is getting old this ZK comparations. I liked before ZK but they are a bunch of fools trying to put in bad shape other projects as GWT and IceFaces.

I will say that ZK blows as a framework and company.

 

PS to dzone and javalobby this should be a place for articles about tech and opinions and not self promotion as ZK team does or Jose Maria Arranz with his framework.

Jose Maria Arranz replied on Wed, 2009/07/08 - 2:25am in response to: Otengi Miloskov

OtengiM, isn't this article SELF PROMOTION? Are you kidding??!! Do you live in Moon?

By the way, I appreciate very much the work of IceFaces guys.

Maybe I do self promotion, MOST OF PEOPLE IN THIS WEB DO SELF PROMOTION. What is the problem?

In spite of this, most of people in javalobby are polite and is a pleasure to discuss with any of them, but not with everybody...

Review your comments, review your agressive style of writing and sometimes your insults. For instance the last one:

http://java.dzone.com/articles/what-appeal-ajax-and-gwt

"Greg Brown We know all in this planet nothing is perfect, all software have bugs and there is a right tool for the job we all know that that that, we dont need to repeat all this bs all over again and again.

But its that GWT rocks for building Ajax applications using the Java language thats all the point of this article. you dont need to start a war or fight saying your tool is better than mine and so on is getting so stupid."

There is an important difference between Greg Brown (Pivot), me and you... Greg and I (and Robbie Cheng with ZK and Stephen Maryka with ICEFaces) create technology for the world and CONTENT FOR JAVALOBBY... and you? What is your work for the public beyond agressive and insulting comments?

I don't buy the aggresive style of writing of Robby Cheng about ZK, that being said, competition is good, and Robbie's original article was fair, the source code was public and anyone could repeate the test and tune the technology used. 

The result: two interesting articles for the community.

I'm waiting, OtengiM, for your amazing, interesting and altruistic articles here at javalobby.

 

Stephen Maryka replied on Wed, 2009/07/08 - 10:34am in response to: Jose Maria Arranz

... and Robbie's original article was fair,,,

That is not my view of his article.Closer to libelous than fair, I would say.  Any self-respecting engineer would have questioned the performance results prior to publishing.  Could one of the most widely adopted Java RIA frameworks like ICEfaces really perform that badly and still be one of the most widely adopted Java RIA frameworks?  Unlikely, so why did he not question his results?  Certainly not because he was attempting to be fair.  

The end result for us is we had to do damage control with several existing and potential customers after the publication of that article.  Who knows how many other potential community members just went away after reading that crap?  And this is not the first time we have been undermined by ZK propaganda. So no, fair is not a fair way to describe it at all.

The only fortunate thing to come from all this is that ZK has yet again been exposed for what they are.  If their framework was on a level playing field with the likes of ICEfaces, RichFaces or GWT they would be in a position to compete fairly, but it isn't and they are not. Now that they have been repeatedly exposed as FUD-muckers, hopefully they will just go away as their community support withers.  I doubt DZone will publish anymore of their drivel, and I doubt any knowledgeable Java EE developer will believe much of what Robbie Cheng writes.

Steve

Otengi Miloskov replied on Wed, 2009/07/08 - 1:04pm in response to: Jose Maria Arranz

Jose Maria, It was nothing against you just that I remember the self promotion sometimes you post in your comments it is kind of similar way as the Zk team, They(ZK) never post something for discuss or as opinions,  Always they are promoting their framework.

If I hurt your feelings ,Im sorry It was not my intetion. I was only saying DZone should be just for articles to learn tech or opinions. I know you dont spend all your time self promoting here, also you write comments or articles about tech. But I bring up that because there are still some people just comment in here dzone for promoting their framework or application.

About me well Im not a framework developer, Im a application developer but that does not make you more elite isnt?. Why I cant talk about my projects? because Im in NDA with the company I work for but I think I can express my opinions about technologies the way Im, Im somtimes agreesive thats my mood as you put for example.

About this article in particular I dont have nothing against they are just reporting that Icefaces is not bad at all as ZK people put it and I understand that. And that is interesting because for my next project I will use Icefaces.

The problem of ZK is that they promote their self over and over, to repetitive and it is getting old that and also they claim they are superior to other frameworks and that is not true that was my last comment about it.

 

So again sorry if I hurt your feelings but please also don't judge the application developers we also contribute to the table.

Jose Maria Arranz replied on Wed, 2009/07/08 - 3:07pm in response to: Otengi Miloskov

First of all thanks for your disclaimer.

OtengiM: They(ZK) never post something for discuss or as opinions,  Always they are promoting their framework.

Because some people waste many months and money and take risks to deliver good software (if possible) for masses and most of the time developers get benefited, sometimes the benefit for the authors is much lower than the benefit got by the users.

Most of them are not below a mega-big-corporation which spends tons of money on direct marketing and sometimes with tons of happy followers around, happy doing product marketing, product promotion... for free!

Oh wait, you use Spring because someone stolen it, because Rod Johson kept Spring in secret in the closet.

OtengiM: I was only saying DZone should be just for articles to learn tech or opinions

And who is the best one teaching tech? An end user or the author?

OtengiM: I know you dont spend all your time self promoting here, also you write comments or articles about tech

I don't have very much free time to talk about the wheather, or how much NetBeans makes me happy (really it makes me happy), I already spend VERY MUCH time for the community doing open source software.

By the way I have already spent many many hours of my "altruistic-free-of-interest" time with comments and articles about tech, sorry is in Spanish.

OtengiM: Im somtimes agreesive thats my mood as you put for example.

Fine, but be ready to receive aggressive answers too. 

OtengiM: And that is interesting because for my next project I will use Icefaces.

Ah, ok, because you are interested in, the rest of the world is bad and self promotes.

I hardly can imagine Rod Jhonson talking about something different to Spring ecosystem.

I hardly can imagine Ted Goddard and Stephen Maryka talking about something different to IceFaces. 

I hardly can imagine Jean-Francois Arcand talking about something different to GlassFish. 

I hardly can imagine Howard Lewis Ship talking about something different to Tapestry.

I hardly can imagine Greg Wilkins talking about something different to Jetty. 

I hardly can imagine Greg Brown talking about something different to Pivot. 

I hardly can imagine Max Katz talking about something different to JBoss ecosystem.  

I hardly can imagine Fabrizio Giudici talking about something different to NetBeans ecosystem. 

I hardly can imagine Jonathan Locke and Eelco Hillenius talking about something different to Wicket.

Do you want more examples?

Sorry Rod, Ted, Jean, Howard, Gregs, Max, Fabrizio, Jonathan and Eelco for this rant.

Finally sorry OtengiM for my "aggressive tone" I'm very angry now. Welcome to the real world.

 

 

Tranquiliser Gt replied on Wed, 2009/07/08 - 7:16pm

How many people in the above examples have ever promoted their works by repeatly bashing others?

Jose Maria Arranz replied on Thu, 2009/07/09 - 2:12am in response to: Tranquiliser Gt

You hardly can found articles from me bashing others.

Anyway justified criticism, with arguments, is fine, and if you are exposed to the public, criticism is an expected and obvious consequence. I mean polite, fair and justified criticism, of course not this kind of absolutely blind and sectarian criticism that I've received here.

 

Robbie Cheng replied on Thu, 2009/07/09 - 3:56am in response to: Stephen Maryka

ZK is no doubt one of the most widely adopted Java RIA framworks, it accumulates almost 1 million downloads on sourceforge.net. It's the first RIA framework to support native Java phone, and Google Android. Recently, it released a spreadsheet component which be embeddable into Web applications. Innovation is the goal of ZK framework.

I publish what I observe, I have no intension to bash anyone. It's true that I didn't do the optimization because I didn't find any clue on your documentation. I didn't find any chapter about performance tip. The result of Richfaces is bad at the first time, then, I figured out it was caused by the bean-scope, so I changed the scope, and re-tested it. Anyway, thanks for your correction, next time, if I have further tests, I will use this trick.

Robbie

Ken Wong replied on Thu, 2009/07/09 - 4:25pm

If the result is reproducible under the given configuration, then nobody should feel mistreated, since the configuration is fair to all other testees. The real data does not lie. It DID reveal one fact. It did not need to reveal all fact. What is more, the tester has no responsibility to find out the best or optimized parameter for each testee. The producer shall advise the user how to take advantage the product. The so called "corrected version" use different configuration, and get different result. So what? Why didn't they do that earlier and benefit the users? Their pictures do not scale at all, some memory consumption reached 100GB while the -Xmx was 1600m.

Ken Wong replied on Thu, 2009/07/09 - 4:45pm

The difference between 0 and 1 can not be discussed. Bankers may try to add 0.1, 0.2, ... between and start to discuss about the endless 0.x. As long as the result is reproducible, then those criticism make non sense. Revealing the badness of one product under some specific situations may not be polite, but forbid anybody to do so is not acceptable. It is far beyond some technology field. Nobody is Saint, so why pretend to be something not of this world?

Ken Wong replied on Fri, 2009/07/10 - 10:45am

if "BitSet in Java was faster than C++ bitset template" is true, then we can safely say Java was faster than C++ when considering BitSet. Never forget the conditions and the context. The context here is very clear, don't ignore it and make a lot of noise. We will never know this world well enough, but I DO believe we can explore it one step by one step, and we can record down the result of every single step and based on it make the next step. Don't say since we can not know everything well enough so we should stay in the elementary school. Everything we learn we are sure it is right with the original context. That is how human and science develops. By the way, the "corrected version" uses JDK1.6.0_07 while the original one used 1.6.0._10 which had very big fundamental changes. The ICEFaces 1.8.1 and 1.7.2 have big difference too I believe. Are they the same thing? Should every single user do lots of tests to find out the best configuration or they should get some feedback/advise from others with repeatable result numbers under specific situations?

Stephen Maryka replied on Mon, 2009/07/13 - 3:45pm in response to: Ken Wong

If the result is reproducible under the given configuration, then nobody should feel mistreated, since the configuration is fair to all other testees. The real data does not lie. It DID reveal one fact. It did not need to reveal all fact. What is more, the tester has no responsibility to find out the best or optimized parameter for each testee.

 You have to remember that the root of the problem is the test itself, not the ICEfaces configuration.  Had the test simulated real-world conditions, the framework would have behaved correctly.  The modified ICEfaces configuration was necessary to counteract the deficiency in the test.

Load testing of Ajax applications is tricky.  If you do it incorrectly, the results are meaningless.  The author may not be responsible for finding the most optimized configuration for ICEfaces, but he is responsible for ensuring his test is exercising the framework correctly.  He did not, thus the meaningless results. The differences in JDK or ICEfaces releases is way down in the noise in comparison to the flawed test.

Steve

Hamish Anderson replied on Mon, 2010/01/11 - 5:22pm

Can I ask why you used a exponential scale on the memory tests?

vijay replied on Thu, 2011/06/23 - 5:58am

Hello, I was in the process of verifying these results for ICEFaces, so that i can go ahead and propose icefaces for a project. Although i am facing some problems regrading no of concurrent users, I am not able to go beyond 200 users for the given hardware configuration, so i would request Stephen or somebody else to please share with me the tomcat configuration used and any other related configurations, for the server to achieve 1000 concurrent users. Also i would like to know if a standalone jmeter setup was used or many jmeter runtimes were used? Thanks for your help, Vijay

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.