He works with Java since 1999. At the moment he is as IT-Consultant at Logica Deutschland GmbH & Co. KG in Frankfurt. Andriy has posted 3 posts at DZone. View Full User Profile

Some Aspects of JVM Tuning in Virtual Environments

08.27.2009
| 11277 views |
  • submit to reddit

I have a dream... No, I will not speak here about jobs and freedom. I have a dream that I have been no more called from operation team. I want to hear nothing about performance problems with highly loaded java application in production. The operation people have the most "fat" computer with a lot of CPUs and a huge amount of RAM. That hardware supports many virtual environments and it supports them really very good. So, what is the problem?

Each environment has enough RAM and gets some CPUs (or cores in case of CPU from Intel). It works fine as long as the garbage collector decides to clear the heap from waste. The heap is big, sometimes - very big. The garbage collector works and works and works... At some point it stops the JVM complete for short period of time, despite of concurrent execution. The rand systems are running in timeout. The more RAM for the heap leads to nothing. More memory, more work for garbage collector, more time spent to clear the heap, longer pauses in JVM. The problem will take place rarely but brings more pain. The people in operation are not happy. The chief of them is very unhappy, because his systems work with a lot of errors. The developers are not happy too. Their chief-developer is very unhappy, because their software shows very bad performance. Nobody is lucky.

You can say: the Java is not the decision for heavy loaded applications.
My answer is: It is wrong. The Java is perfect. The usage of Java is not perfect. Try to review this problem in a more detailed way.

At first, we have a virtual environment with some (mostly two or three) CPUs and the big enough RAM.
At second, we have highly loaded application. Probably, it is loaded not every moment of a day, there are times with heavily loading (for example: daytraidings) and with long pauses (nightly).
At third, our virtual environment works with other environments. Not every environment needs 100% CPU usage whole time. Some environments work hard only nights, for example, ETL processes.

This is on the one hand. On the other hand we have the server JVM with concurrent garbage collector. It looks like we cannot do anything more. "Yes, we can!" You must simply look into the fine settings of the garbage collector. There are two ways how you can do it.
The first one is to activate the JMX and use jconsole tool from jdk installation. You need start the java application with parameters:

-Dcom.sun.management.jmxremote -Dcom.sun.management.config.file=your_file

where your_file - is a JMX configuration file. More about it you find here. After the restart of the application you should use the jconsole tool to connect to the application. Select the MBeans page and go throw the tree of MBeans starting of “java.lang” to “GarbageCollector”, then to PS MarkSweep's attribute named "LastGcInfo". The double click opens the details of the attribute. So, you can find here the value of “GcThreadCount” member.

 

In my case it is equal to 8. I have the 8 CPU cores, so the JVM starts the eight threads in background to perform the garbage collection.

 

The second one: use jmap tool from jdk installation with such parameters:

jmap -heap PID,

where PID - process id of your java process. After execution you get all required information with the number of thread for GC.

You see, how many threads is used by the garbage collector. This number is equal to the number of processing units (cores of CPU). It means the huge amount of RAM will be analyzed and cleared by only those threads. If the virtual environment has only small number of processing units (two, three or four), it is not enough. In that case you have the long times for garbage collection and unlucky guys in operation. My advice is to increase the number of GC threads. It helps the GC to analyze the heap more concurrently and decreases the GC times. It utilizes more CPU time. If all CPUs have been used at 100% level, there is a time to think about new processing units for that virtual environment. As I said above, there are always some virtual environments that need the CPU in other time as your java application. The guys in operation can switch those resources dynamically between the virtual environments. So, your application gets required resources as soon as it needs them.

How can I configure the number of GC threads in JVM? You use the command-line parameter
-XX:ParallelGCThreads=X, where X - the new number of the threads for GC, for example, double of processing units. You should pay some attention to the other parameters of JVM, when you change the default parameters. The JVM likes to change something unexpected if the default parameters were changed. In my case the JVM von HP Vendor on HP-UX OS changes the default concurrent MarkSweep GC to the serial collector. I have used then the additional parameter to recover the status quo: -XX:+UseParallelGC.

With more threads for GC we achieve the significant shorter times of garbage collection. The rand systems do not fall in timeout periodically. The guys in operation are lucky and monitor the CPU usage. They will add the new processing units from other virtual environments if it is needed. As a result: Everybody is lucky and I get no calls more.

Published at DZone with permission of its author, Andriy Maksymov.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Robert Enyedi replied on Thu, 2009/08/27 - 4:53am

Great post, Andriy. Do you connect with JConsole to the JVM using SSH?

Andriy Maksymov replied on Thu, 2009/08/27 - 6:11am in response to: Robert Enyedi

Thank you. The jconsole was used without SSL. The second image is a SSH window. I have used the PuTTY program.

Rigel Kentaurus replied on Thu, 2009/08/27 - 11:42am

Great post, thanks for sharing. Never been on this scenario myself, It's enlightening to check it.

I'd be interested in knowing how did you realize that the problem wasn't the program itself, but the garbage collector instead. Most people would jump and immediately blame the actual java program for any performance issues. I know my operations area would do that, then the developers would spend days in trying to make the programs faster or less memory hungry.

Did you do profiling, or some other tool? Or was it just a correct guess, and an educated assumption derived from experience?

Robert Enyedi replied on Thu, 2009/08/27 - 12:20pm in response to: Andriy Maksymov

The jconsole was used without SSL.

I guess you meant SSH instead of SSL. The reason I'm asking is that we have these great monitoring tools like JConsole and more recently VisualVM, yet I'm yet to see someone being able to use them without a LAN or VPN.

And since all dedicated hosting providers offer SSH, it is quite a shame. Am I missing something? Does anyone have a different experience?

Andriy Maksymov replied on Fri, 2009/08/28 - 4:14am in response to: Rigel Kentaurus

Hi, we have several instances of the same java application on different hardware and operation systems (linux, HP-UX and IBM AIX). The operations guys told me about problems only on HP-UX machine. The application processes realy a lot of requests, so the Young-generation memory grow and fall with a rate at 2 GB per minute. The rest goes to old generation area. At first look, everything was right. For such amount of memory there were a lack of GC threads, I found. We have profiling and we saw that the problem lyes not on database or something others. We saw also, that the timeout problems took place if the GC was running.

Andriy Maksymov replied on Fri, 2009/08/28 - 4:19am in response to: Robert Enyedi

Hi, the jconsole uses SSL. The SSH is secure shell: the remote shell over SSL. The SSL protocol simply encrypts the data sent over TCP/IP. The activation of SSL for JMX should be not difficult. Probably I write the next article about that. Please wait some days. I must prepare a workshop for our customer and takes me some effort.

Lily Marlene replied on Thu, 2013/01/17 - 8:33am

 Your dream is big, hope you will manage to make it possible.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.