Some Aspects of JVM Tuning in Virtual Environments

I have a dream... No, I will not speak here about jobs and freedom. I have a dream that I have been no more called from operation team. I want to hear nothing about performance problems with highly loaded java application in production. The operation people have the most "fat" computer with a lot of CPUs and a huge amount of RAM. That hardware supports many virtual environments and it supports them really very good. So, what is the problem?

Each environment has enough RAM and gets some CPUs (or cores in case of CPU from Intel). It works fine as long as the garbage collector decides to clear the heap from waste. The heap is big, sometimes - very big. The garbage collector works and works and works... At some point it stops the JVM complete for short period of time, despite of concurrent execution. The rand systems are running in timeout. The more RAM for the heap leads to nothing. More memory, more work for garbage collector, more time spent to clear the heap, longer pauses in JVM. The problem will take place rarely but brings more pain. The people in operation are not happy. The chief of them is very unhappy, because his systems work with a lot of errors. The developers are not happy too. Their chief-developer is very unhappy, because their software shows very bad performance. Nobody is lucky.

You can say: the Java is not the decision for heavy loaded applications.
My answer is: It is wrong. The Java is perfect. The usage of Java is not perfect. Try to review this problem in a more detailed way.

At first, we have a virtual environment with some (mostly two or three) CPUs and the big enough RAM.
At second, we have highly loaded application. Probably, it is loaded not every moment of a day, there are times with heavily loading (for example: daytraidings) and with long pauses (nightly).
At third, our virtual environment works with other environments. Not every environment needs 100% CPU usage whole time. Some environments work hard only nights, for example, ETL processes.

This is on the one hand. On the other hand we have the server JVM with concurrent garbage collector. It looks like we cannot do anything more. "Yes, we can!" You must simply look into the fine settings of the garbage collector. There are two ways how you can do it.
The first one is to activate the JMX and use jconsole tool from jdk installation. You need start the java application with parameters:

-Dcom.sun.management.jmxremote -Dcom.sun.management.config.file=your_file

where your_file - is a JMX configuration file. More about it you find here. After the restart of the application you should use the jconsole tool to connect to the application. Select the MBeans page and go throw the tree of MBeans starting of “java.lang” to “GarbageCollector”, then to PS MarkSweep's attribute named "LastGcInfo". The double click opens the details of the attribute. So, you can find here the value of “GcThreadCount” member.

The second one: use jmap tool from jdk installation with such parameters:

jmap -heap PID,

where PID - process id of your java process. After execution you get all required information with the number of thread for GC.

You see, how many threads is used by the garbage collector. This number is equal to the number of processing units (cores of CPU). It means the huge amount of RAM will be analyzed and cleared by only those threads. If the virtual environment has only small number of processing units (two, three or four), it is not enough. In that case you have the long times for garbage collection and unlucky guys in operation. My advice is to increase the number of GC threads. It helps the GC to analyze the heap more concurrently and decreases the GC times. It utilizes more CPU time. If all CPUs have been used at 100% level, there is a time to think about new processing units for that virtual environment. As I said above, there are always some virtual environments that need the CPU in other time as your java application. The guys in operation can switch those resources dynamically between the virtual environments. So, your application gets required resources as soon as it needs them.

How can I configure the number of GC threads in JVM? You use the command-line parameter
-XX:ParallelGCThreads=X, where X - the new number of the threads for GC, for example, double of processing units. You should pay some attention to the other parameters of JVM, when you change the default parameters. The JVM likes to change something unexpected if the default parameters were changed. In my case the JVM von HP Vendor on HP-UX OS changes the default concurrent MarkSweep GC to the serial collector. I have used then the additional parameter to recover the status quo: -XX:+UseParallelGC.

With more threads for GC we achieve the significant shorter times of garbage collection. The rand systems do not fall in timeout periodically. The guys in operation are lucky and monitor the CPU usage. They will add the new processing units from other virtual environments if it is needed. As a result: Everybody is lucky and I get no calls more.