Pierre-Hugues Charbonneau (nickname P-H) is working for CGI Inc. Canada for the last 10 years as a senior IT consultant and system architect. His primary area of expertise is Java EE, middleware & JVM technologies. He is a specialist in production system troubleshooting, middleware & JVM tuning and scalability / capacity improvement; including processes improvement for Java EE support teams. Pierre - Hugues is a DZone MVB and is not an employee of DZone and has posted 43 posts at DZone. You can read more from them at their website. View Full User Profile

Java 8: From PermGen to Metaspace

02.11.2013
| 81763 views |
  • submit to reddit
As you may be aware, the JDK 8 Early Access is now available for download. This allows Java developers to experiment with some of the new language and runtime features of Java 8.
One of these features is the complete removal of the Permanent Generation (PermGen) space which has been announced by Oracle since the release of JDK 7. Interned strings, for example, have already been removed from the PermGen space since JDK 7. The JDK 8 release finalizes its decommissioning.
This article will share the information that we found so far on the PermGen successor: Metaspace. We will also compare the runtime behavior of the HotSpot 1.7 vs. HotSpot 1.8 (b75) when executing a Java program “leaking” class metadata objects.
The final specifications, tuning flags and documentation around Metaspace should be available once Java 8 is officially released.
Metaspace: 
A new memory space is born
The JDK 8 HotSpot JVM is now using native memory for the representation of class metadata and is called Metaspace; similar to the Oracle JRockit and IBM JVM's.
The good news is that it means no more java.lang.OutOfMemoryError: PermGen space problems and no need for you to tune and monitor this memory space anymore…not so fast. While this change is invisible by default, we will show you next that you will still need to worry about the class metadata memory footprint. Please also keep in mind that this new feature does not magically eliminate class and classloader memory leaks. You will need to track down these problems using a different approach and by learning the new naming convention.
I recommend that you read the PermGen removal summary and comments from Jon on this subject. 

In summary:
PermGen space situation
  • This memory space is completely removed.
  • The PermSize and MaxPermSize JVM arguments are ignored and a warning is issued if present at start-up.
Metaspace memory allocation model
  • Most allocations for the class metadata are now allocated out of native memory.
  • The klasses that were used to describe class metadata have been removed.
Metaspace capacity
  • By default class metadata allocation is limited by the amount of available native memory (capacity will of course depend if you use a 32-bit JVM vs. 64-bit along with OS virtual memory availability).
  • A new flag is available (MaxMetaspaceSize), allowing you to limit the amount of native memory used for class metadata. If you don’t specify this flag, the Metaspace will dynamically re-size depending of the application demand at runtime.
Metaspace garbage collection
  • Garbage collection of the dead classes and classloaders is triggered once the class metadata usage reaches the “MaxMetaspaceSize”.
  • Proper monitoring & tuning of the Metaspace will obviously be required in order to limit the frequency or delay of such garbage collections. Excessive Metaspace garbage collections may be a symptom of classes, classloaders memory leak or inadequate sizing for your application.
Java heap space impact
  • Some miscellaneous data has been moved to the Java heap space. This means you may observe an increase of the Java heap space following a future JDK 8 upgrade.
Metaspace monitoring
  • Metaspace usage is available from the HotSpot 1.8 verbose GC log output.
  • Jstat & JVisualVM have not been updated at this point based on our testing with b75 and the old PermGen space references are still present.
Enough theory now, let’s see this new memory space in action via our leaking Java program…

PermGen vs. Metaspace runtime comparison

In order to better understand the runtime behavior of the new Metaspace memory space, we created a class metadata leaking Java program. You can download the source here.

The following scenarios will be tested:
  • Run the Java program using JDK 1.7 in order to monitor & deplete the PermGen memory space set at 128 MB.
  • Run the Java program using JDK 1.8 (b75) in order to monitor the dynamic increase and garbage collection of the new Metaspace memory space.
  • Run the Java program using JDK 1.8 (b75) in order to simulate the depletion of the Metaspace by setting the MaxMetaspaceSize value at 128 MB.
JDK 1.7 @64-bit – PermGen depletion
  • Java program with 50K configured iterations
  • Java heap space of 1024 MB
  • Java PermGen space of 128 MB (-XX:MaxPermSize=128m)
As you can see form JVisualVM, the PermGen depletion was reached after loading about 30K+ classes. We can also see this depletion from the program and GC output. Class metadata leak simulator Author: Pierre-Hugues Charbonneau http://javaeesupportpatterns.blogspot.com ERROR: java.lang.OutOfMemoryError: PermGen space Now let’s execute the program using the HotSpot JDK 1.8 JRE. JDK 1.8 @64-bit – Metaspace dynamic re-size
  • Java program with 50K configured iterations
  • Java heap space of 1024 MB
  • Java Metaspace space: unbounded (default)


As you can see from the verbose GC output, the JVM Metaspace did expand dynamically from 20 MB up to 328 MB of reserved native memory in order to honor the increased class metadata memory footprint from our Java program. We could also observe garbage collection events in the attempt by the JVM to destroy any dead class or classloader object. Since our Java program is leaking, the JVM had no choice but to dynamically expand the Metaspace memory space.

The program was able to run its 50K of iterations with no OOM event and loaded 50K+ Classes.


Let's move to our last testing scenario.
JDK 1.8 @64-bit – Metaspace depletion
  • Java program with 50K configured iterations
  • Java heap space of 1024 MB
  • Java Metaspace space: 128 MB (-XX:MaxMetaspaceSize=128m)

As you can see form JVisualVM, the Metaspace depletion was reached after loading about 30K+ classes; very similar to the run with the JDK 1.7. We can also see this from the program and GC output. Another interesting observation is that the native memory footprint reserved was twice as much as the maximum size specified. This may indicate some opportunities to fine tune the Metaspace re-size policy, if possible, in order to avoid native memory waste. Now find below the Exception we got from the Java program output.
 Class metadata leak simulator
Author: Pierre-Hugues Charbonneau
http://javaeesupportpatterns.blogspot.com
ERROR: java.lang.OutOfMemoryError: Metadata space
Done!
As expected, capping the Metaspace at 128 MB like we did for the baseline run with JDK 1.7 did not allow us to complete the 50K iterations of our program. A new OOM error was thrown by the JVM. The above OOM event was thrown by the JVM from the Metaspace following a memory allocation failure.

#metaspace.cpp Final words

I hope you appreciated this early analysis and experiment with the new Java 8 Metaspace. The current observations definitely indicate that proper monitoring & tuning will be required in order to stay away from problems such as excessive Metaspace GC or OOM conditions triggered from our last testing scenario. Future articles may include performance comparisons in order to identify potential performance improvements associated with this new feature. Please feel free to provide any comment.
Published at DZone with permission of Pierre - Hugues Charbonneau, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Tetsu Soh replied on Mon, 2013/02/11 - 3:51am

Nice analysis. Thank you for the info.

Looking forward to performance comparison. 

André Pankraz replied on Wed, 2013/02/13 - 4:09am

Hi,

currently I don't understand this.

I always thougt the class metadata will be in the heap together with the user stuff, so I don't have to reserve / split off a defined separate memory area with separate GC rules.

Now we replace the permgen with a differently named area but with the same problems.

On production systems it's a bad idea to not restrict the used memory, so instead of finding good PermGen settings and getting OOMs with leaking classloaders we now have to find good settings for Max Meta settings and getting OOMs.

Also this Meta is now C-heap based...many interesting tools like MAT wount work anymore for this...now we have low level tools like jmap etc...

I don't get it...may be I missed the core of the story.


Cheers,

André

Pierre - Hugues... replied on Wed, 2013/02/13 - 7:44am

 Hi Andre and thanks for your comments,

You did not miss the core of the story. You raised a valid point and concern. I recommend that you review the Open JDK post about this change and motivation.

http://openjdk.java.net/jeps/122

One of the motivation is the convergence between Oracle JRockit & Oracle HotSpot (JRockit never had a PermGen space). Another benefit (when using default or unbounded) is that it eliminates the need for PermGen sizing. Now,  we have to keep in mind that native memory will be used instead so this means that you still need to perform your due diligence as per below:

- A future upgrade project From Java 5-6-7 to Java 8 should include proper capacity planning along with performance testing (vs. established baseline) so you can assess the change of behaviour and memory native footprint requirement for your application.

- Monitor closely the Java process size and native heap space of your production JVM processes for possible memory leaks and native memory footprint.

- If using 32-bit JVM, ensure that you have enough native heap for the JVM to dynamically grow the meta space (which is in a race with Java Heap space).

- If using 64-bit JVM, ensure that your OS has enough physical/virtual memory to allow the JVM to dynamically grow the meta space.

As you mentioned, for some production environments with limited virtual memory availability, using the cap mode may be preferable. Regardless of your decision of using default unbounded or cap mode, proper capacity planning and monitoring will be very important.

My experience with IBM JM and JRockit from production environments did teach me to always monitor closely the native heap space.

Finally, you will still be able to use MAT and perform Heap Dump analysis for Class & Class loader leaks (Java representation objects are still present). You can see this by running the sample program and replicating scenario #3. Add the -XX:+HeapDumpOnOutOfMemoryError flag and analyze the generated heap dump following the java.lang.OutOfMemoryError: Metadata space. You will be able to pinpoint the class loader memory leak.


Regards,

P-H

André Pankraz replied on Thu, 2013/02/14 - 1:43am

Hi,

thx for response - but instead of PermGen sizing we now need Meta sizing...thats my problem, we don't win anything if you woun't like to run into unexpected OOMs, e.g. because the 32 bit process exceeded the memory barrier and the heap GC doesn't even notice. So I wondered about the intention.

Instead of seeing PermGen directly in VisualVM or many available Web Dashboards like JavaMelody or WLS console you see it now in native OS-dependant process monitoring.

Especially for 32 bit...many people where already a little bit confused why they sometimes get OOMs even though heap + permgen are smaller then 2 GB at windows or 4 GB at linux/solaris. There where always additional native memory areas (stacks, threads, mapped mem etc.)...but not very large. Now you suddenly have 100th of MB additional used native memory (medium sized web projects) and you have to be aware of this at the same level like with PermGen.

One good thing I see is, that many interned Strings / statics go into the heap...that could help a lot to restrict this effect. I will see..


Cheers André

Glyn Normington replied on Thu, 2013/06/06 - 3:49am

Nice summary. Thanks Pierre-Hugues!

Is class GC first triggered when metaspace usage reaches MaxMetaspaceSize or -MetaspaceSize? Jon Masamitsu's post says it's MetaspaceSize.

Pierre - Hugues... replied on Tue, 2013/06/18 - 8:14am

Thanks Glyn for your comments,


The Java 8 Metaspace JVM arguments are similar to Java 7 HotSpot PermSize and MaxPermSize parameters (initial & maximum size).


For Java 8, Metaspace GC will be indeed be triggered once it reaches the current MetaspaceSize. MaxMetaspaceSize is simply the upper limit you can setup in order to avoid, for example, a 64-bit JVM process to use too much native memory (existing leak, physical resource constraints etc.).


As you saw from the above GC log snapshot, after each Metaspace GC, the JVM did trigger an expansion of the Metaspace, up to its upper limit when set. Once the Metaspace can no longer be expanded (upper limit reached, 32-bit memory address space depletion, non-availablity of OS virtual memory...) OOM is thrown by the JVM.


I will soon release a part 2 of this article focusing on garbage collection and debugging strategies for future Java 8 Metaspace / native memory leak.


Regards,

P-H

Anjinappa Ansi replied on Tue, 2014/04/15 - 10:39pm

Nice article to learn the new features !!

Amar Sharma replied on Mon, 2014/05/05 - 6:31am

Thank you for the article.Very informative and helpful.

Edo Salpr replied on Mon, 2014/08/04 - 12:12pm

Hello,

I don't under stand.

Native memory refers to the memory allocated by the OS to the JVM, right?!

So, where is the metaspace? How can it be outside?


Thanks.

Ravi Kolagani replied on Mon, 2014/09/15 - 3:18pm

It was nice presentation. I wanted to play the runtime behavior of metaspace memory, can you please send me the link to download the source code for testing the class metadata leaking..

Thanks Ravi!

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.