Enterprise Integration Zone is brought to you in partnership with:

Alan is the Technical Team Lead and Senior Mule Expert at Ricston. Here's his full bio Alan is a DZone MVB and is not an employee of DZone and has posted 11 posts at DZone. You can read more from them at their website. View Full User Profile

Mule on Waratek Cloud VM

01.20.2014
| 7237 views |
  • submit to reddit

I’m sure that whoever follows our blog is fairly familiar with Mule, but what is Waratek? Let’s cut to the chase here, and announce that for me, Waratek offers one of the coolest products I have ever seen. Waratek brought to the JVM, what VMWare brought to the hardware. Waratek offers a JVM that can be virtualised! It is like applying LPARS to your JVM. Suddenly your JVM becomes multi-tenant. How cool is that?

So how hard is it to start Mule on Waratek JVM? Well its very easy. Waratek employed the idea that the developer needs to do ZERO code changes in order to be able to run on their JVM. This is a huge relief, because it now means that to virtualise your JVM, you need zero development effort.

To test Mule on the Waratek JVM, we used the latest community edition of Mule 3.4.0 from here. As for Waratek, we downloaded the OVA image available here. This is an open format virtualisation image than can be run on several virtualisation packages such as Virtual Box and VMWare. It comes with CentOS 6 pre-configured with the Waratek Cloud VM and packaged with some demo applications .

Once we had everything installed, we just started Mule as we would normally do. Mule informs us on which JVM it is running in the boilerplate. This is what we got:

$ ./bin/mule 
MULE_HOME is set to /home/student/mule/mule-standalone-3.4.0-1
Running in console (foreground) mode by default, use Ctrl-C to exit...
MULE_HOME is set to /home/student/mule/mule-standalone-3.4.0-1
Running Mule...
--> Wrapper Started as Console
Launching a JVM...
Starting the Mule Container...
Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
  Copyright 1999-2006 Tanuki Software, Inc.  All Rights Reserved.

INFO  2013-12-17 04:30:51,029 [WrapperListener_start_runner] org.mule.module.launcher.MuleContainer: 
**********************************************************************
* Mule ESB and Integration Platform                                  *
* Version: 3.4.0 Build: c8afb471                                     *
* MuleSoft, Inc.                                                     *
* For more information go to http://www.mulesoft.org                 *
*                                                                    *
* Server started: 17/12/13 04:30                                     *
* JDK: 1.6.0-waratek-release (release mode)                          *
* OS: Linux (2.6.32-358.23.2.el6.x86_64, amd64)                      *
* Host: trainingvm.local (127.0.0.1)                                 *
**********************************************************************

Shutting down Mule can be done just as simply; hit the Ctrl+C key combination. The Waratek JVM tell’s us that the interrupt is unsupported, but Mule will still shut down cleanly. The commands to start and stop Mule as a daemon, i.e mule start, and mule stop, also work out of the box on the Waratek JVM.

^CINT trapped.  Shutting down.
Interrupt unsupported. Run 'jvckill --jvc=jvc-1',then 'jvcundefine --jvc=jvc-1' to ensure cleanup
JVM exited unexpectedly while stopping the application.
<-- Wrapper Stopped

So what is happening underneath? When we start the Waratek JVM, it usually starts as a daemon process (although it can also be started as a foreground process), called javad. This is literally a hypervisor for Java, very similar in concept to the VMWare hypervisor. Waratek also provides a script called java, to replace the plain old java command. This script, instead of launching a normal JVM, will create a JVC (Java Virtual Container) inside the Waratek JVM, and kicks it off. In a few words, a JVC is a virtualised instance of a JVM, completely isolated from any other JVC. This compares to a VMWare image.

Waratek allows us to manage the hypervisor using an SSH connection. Once you SSH into the hypervisor, you can use jirsh. If we list the current JVCs while Mule is running, we get something like this:

jirsh # list
JVCID	GROUP	STATUS      	NAME                	COMMAND
0    	0    	Shut Off    	dom-0               	<none>
2    	0    	Running     	jvc-1               	java -Dmule.home=/home/student/mule/mule-standalone-3.4.0-1 -Dmule.base=/home/student/mule/mule-standalone-3.4.0-1 -Djava.net.preferIPv4Stack=TRUE -XX:MaxPermSize=128m -Djava.endorsed.dirs=/home/student/mule/mule-standalone-3.4.0-1/lib/endorsed -Xmx512m -Djava.library.path=%LD_LIBRARY_PATH%:/home/student/mule/mule-standalone-3.4.0-1/lib/boot -classpath %MULE_LIB%:/home/student/mule/mule-standalone-3.4.0-1/conf:/home/student/mule/mule-standalone-3.4.0-1/lib/boot/mule-module-boot-3.4.0.jar:/home/student/mule/mule-standalone-3.4.0-1/lib/boot/mule-module-reboot-3.4.0.jar:/home/student/mule/mule-standalone-3.4.0-1/lib/boot/log4j-1.2.16.jar:/home/student/mule/mule-standalone-3.4.0-1/lib/boot/wrapper-3.2.3.jar:/home/student/mule/mule-standalone-3.4.0-1/lib/boot/mule-module-logging-3.4.0.jar:/home/student/mule/mule-standalone-3.4.0-1/lib/boot/commons-cli-1.2.jar -Dwrapper.key=Sjcf2v03oFkp32Oh -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=5339 -Dwrapper.version=3.2.3 -Dwrapper.native_library=wrapper -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 org.mule.module.reboot.MuleContainerBootstrap console0

One thing you have to be aware of, is that before Mule starts, it launches a Groovy script to prepare the classpath and perform some other preparations. Groovy scripts run on a JVM just like a normal Java application. So what happens here is that once you launch Mule, first a JVC is created to launch the Groovy script, and upon completion, another JVC is created to launch Mule. If you SSH into the hypervisor, and execute a list command while Mule is starting, you will get something like this:

jirsh # list
JVCID	GROUP	STATUS      	NAME                	COMMAND
0    	0    	Shut Off    	dom-0               	<none>
1    	0    	Running     	jvc-1               	java -Dmule.home=/home/student/mule/mule-standalone-3.4.0-1 -cp .:/home/student/mule/mule-standalone-3.4.0-1/conf:/home/student/mule/mule-standalone-3.4.0-1/lib/opt/groovy-all-1.8.6.jar:/home/student/mule/mule-standalone-3.4.0-1/lib/boot/commons-cli-1.2.jar org.codehaus.groovy.tools.GroovyStarter --main groovy.ui.GroovyMain --conf /home/student/mule/mule-standalone-3.4.0-1/bin/launcher.conf /home/student/mule/mule-standalone-3.4.0-1/bin/additional.groovy /home/student/mule/mule-standalone-3.4.0-1/conf/wrapper.conf -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005 -M-Djava.endorsed.dirs=/home/student/mule/mule-standalone-3.4.0-1/lib/endorsed console

Once the Groovy process is complete, the JVC is destroyed and you will have one JVC running Mule as shown above. This does come with one small caveat; Waratek limits the number of JVCs you can use depending on the licence you buy. If you have a 4 JVC licence, you can only start 3 Mule instances since the last one will not manage to start both the Groovy script and Mule at the same time. I am sure that this can be easily fixed. (Waratek have been consulted and have so far been unable to reproduce the behaviour so it may not affect all users. In other words; you should be able to run the number of JVCs you are licensed for.)

One suggestion would be to run the Groovy script on a normal JVM, like the Oracle JVM or OpenJDK, and run Mule on the Waratek JVM.

Once Mule is stopped, either by running mule stop if Mule is running as a daemon, or by pressing Ctrl+C if Mule is running in the foreground, when we inspect the hypervisor we should see the following. One thing to notice is that the JVC is not undefined from the hypervisor, it will just be in Shut Off state. This makes it very easy to restart it from the hypervisor itself by just invoking start jvc-id.

jirsh # list
JVCID	GROUP	STATUS      	NAME                	COMMAND
0    	0    	Shut Off    	dom-0               	<none>
2    	0    	Shut Off    	jvc-1               	java -Dmule.home=/home/student/mule/mule-standalone-3.4.0-1 -Dmule.base=/home/student/mule/mule-standalone-3.4.0-1 -Djava.net.preferIPv4Stack=TRUE -XX:MaxPermSize=128m -Djava.endorsed.dirs=/home/student/mule/mule-standalone-3.4.0-1/lib/endorsed -Xmx512m -Djava.library.path=%LD_LIBRARY_PATH%:/home/student/mule/mule-standalone-3.4.0-1/lib/boot -classpath %MULE_LIB%:/home/student/mule/mule-standalone-3.4.0-1/conf:/home/student/mule/mule-standalone-3.4.0-1/lib/boot/mule-module-boot-3.4.0.jar:/home/student/mule/mule-standalone-3.4.0-1/lib/boot/mule-module-reboot-3.4.0.jar:/home/student/mule/mule-standalone-3.4.0-1/lib/boot/log4j-1.2.16.jar:/home/student/mule/mule-standalone-3.4.0-1/lib/boot/wrapper-3.2.3.jar:/home/student/mule/mule-standalone-3.4.0-1/lib/boot/mule-module-logging-3.4.0.jar:/home/student/mule/mule-standalone-3.4.0-1/lib/boot/commons-cli-1.2.jar -Dwrapper.key=Sjcf2v03oFkp32Oh -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=5339 -Dwrapper.version=3.2.3 -Dwrapper.native_library=wrapper -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 org.mule.module.reboot.MuleContainerBootstrap console0

There is one thing you have to take care with when running Mule on the Waratek JVM. Mule uses the Tanuki Java Service Wrapper. This makes it very easy to run Mule as daemon process on different operating systems. It also offers other features like restarting your JVM if it hangs. The Tanuki wrapper will ping your JVM and if it does not receive a reply after a certain time, it will kill your JVM and start a new instance. This works perfectly well with the normal JVM, but it does not compliment the Waratek JVM.

If the JVM hangs, the Tanuki wrapper tries to kill your JVM by executing a shutdown process in your JVM itself (soft shutdown). However, if this fails, it performs a kill -9 on the process id.

JVM appears hung: Timed out waiting for signal from JVM.
JVM did not exit on request, terminated
JVM exited on its own while waiting to kill the application.
JVM exited in response to signal SIGKILL (9).
Launching a JVM...
Starting the Mule Container...

The biggest issue here is that the Tanuki wrapper would not have the correct process id. Mule does not have a process id, its running in a JVC inside another JVM, the Waratek JVM. Luckily the wrapper does not try to kill the Waratek JVM, but instead it will try to kill the Waratek java command that launched off the JVC. This is not too bad, because it has no side effects.

However, this does not kill Mule, and the wrapper will start a new instance of Mule. This results in having 2 instances of Mule running in different JVCs, one hung and one running fine. As you can imagine, this can cause issues.

Our suggestion for this situation is to switch off this particular feature by adding the following line in the $MULE_HOME/conf/wrapper.conf:

wrapper.ping.timeout=0

This sets the timeout to infinity, hence the wrapper will never try to kill and restart a new instance of Mule. Of course this means that you have to put some monitoring in place, but that is normal with all enterprise applications like Mule.

We still haven’t explored one of the amazing features of the Waratek JVM: elastic memory. This feature allows us to assign memory to a JVC, plus share a pool of memory between different JVCs. A JVC will only use memory from the shared pool, when it would have exhausted its own. This is a very handy feature to avoid out of memory issues that usually appear only during peak times. On top of that, Waratek allows us to change the configuration of the JVC, like memory allocation, dynamically during runtime. There is no need to reboot your application if an expansion in heap is required.

What does this lead to? Well this means that now you can have Mule applications completely isolated from each other, running in different JVCs on separate Mule instances. A shared pool can be configured to allow each JVC access to extra memory during peak times. Moreover you can use the Waratek JVM features to limit each JVC (Mule app) in memory, network, CPU pinning, and more. No application will affect the other if it runs out of memory, or acts badly.

We tried running multiple Mules on the Waratek JVM, and when we explored the hypervisor, this is what we could see:

jirsh # list
JVCID	GROUP	STATUS      	NAME                	COMMAND
0    	0    	Shut Off    	dom-0               	<none>
1    	0    	Running     	jvc-2               	java -Dmule.home=/home/student/mule/mule-standalone-3.4.0-2 -Dmule.base=/home/student/mule/mule-standalone-3.4.0-2 -Djava.net.preferIPv4Stack=TRUE -XX:MaxPermSize=128m -Djava.endorsed.dirs=/home/student/mule/mule-standalone-3.4.0-2/lib/endorsed -Xmx512m -Djava.library.path=%LD_LIBRARY_PATH%:/home/student/mule/mule-standalone-3.4.0-2/lib/boot -classpath %MULE_LIB%:/home/student/mule/mule-standalone-3.4.0-2/conf:/home/student/mule/mule-standalone-3.4.0-2/lib/boot/mule-module-boot-3.4.0.jar:/home/student/mule/mule-standalone-3.4.0-2/lib/boot/mule-module-reboot-3.4.0.jar:/home/student/mule/mule-standalone-3.4.0-2/lib/boot/log4j-1.2.16.jar:/home/student/mule/mule-standalone-3.4.0-2/lib/boot/wrapper-3.2.3.jar:/home/student/mule/mule-standalone-3.4.0-2/lib/boot/mule-module-logging-3.4.0.jar:/home/student/mule/mule-standalone-3.4.0-2/lib/boot/commons-cli-1.2.jar -Dwrapper.key=KIyPMmzQxuSyT21D -Dwrapper.port=32001 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=6572 -Dwrapper.version=3.2.3 -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 org.mule.module.reboot.MuleContainerBootstrap start0
3    	0    	Running     	jvc-1               	java -Dmule.home=/home/student/mule/mule-standalone-3.4.0-1 -Dmule.base=/home/student/mule/mule-standalone-3.4.0-1 -Djava.net.preferIPv4Stack=TRUE -XX:MaxPermSize=128m -Djava.endorsed.dirs=/home/student/mule/mule-standalone-3.4.0-1/lib/endorsed -Xmx512m -Djava.library.path=%LD_LIBRARY_PATH%:/home/student/mule/mule-standalone-3.4.0-1/lib/boot -classpath %MULE_LIB%:/home/student/mule/mule-standalone-3.4.0-1/conf:/home/student/mule/mule-standalone-3.4.0-1/lib/boot/mule-module-boot-3.4.0.jar:/home/student/mule/mule-standalone-3.4.0-1/lib/boot/mule-module-reboot-3.4.0.jar:/home/student/mule/mule-standalone-3.4.0-1/lib/boot/log4j-1.2.16.jar:/home/student/mule/mule-standalone-3.4.0-1/lib/boot/wrapper-3.2.3.jar:/home/student/mule/mule-standalone-3.4.0-1/lib/boot/mule-module-logging-3.4.0.jar:/home/student/mule/mule-standalone-3.4.0-1/lib/boot/commons-cli-1.2.jar -Dwrapper.key=TgxdQprvyLE0Hs1h -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=6458 -Dwrapper.version=3.2.3 -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 org.mule.module.reboot.MuleContainerBootstrap start0
4    	0    	Running     	jvc-3               	java -Dmule.home=/home/student/mule/mule-standalone-3.4.0-3 -Dmule.base=/home/student/mule/mule-standalone-3.4.0-3 -Djava.net.preferIPv4Stack=TRUE -XX:MaxPermSize=128m -Djava.endorsed.dirs=/home/student/mule/mule-standalone-3.4.0-3/lib/endorsed -Xmx512m -Djava.library.path=%LD_LIBRARY_PATH%:/home/student/mule/mule-standalone-3.4.0-3/lib/boot -classpath %MULE_LIB%:/home/student/mule/mule-standalone-3.4.0-3/conf:/home/student/mule/mule-standalone-3.4.0-3/lib/boot/mule-module-boot-3.4.0.jar:/home/student/mule/mule-standalone-3.4.0-3/lib/boot/mule-module-reboot-3.4.0.jar:/home/student/mule/mule-standalone-3.4.0-3/lib/boot/log4j-1.2.16.jar:/home/student/mule/mule-standalone-3.4.0-3/lib/boot/wrapper-3.2.3.jar:/home/student/mule/mule-standalone-3.4.0-3/lib/boot/mule-module-logging-3.4.0.jar:/home/student/mule/mule-standalone-3.4.0-3/lib/boot/commons-cli-1.2.jar -Dwrapper.key=hUfAN7p1KcONbWEX -Dwrapper.port=32002 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=6690 -Dwrapper.version=3.2.3 -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 org.mule.module.reboot.MuleContainerBootstrap start0

To conclude this blog, I re-iterate what I said in the beginning. The Waratek JVM is one of the coolest products I have seen. Its a very powerful and useful tool. Running Mule on it makes a lot of sense as you are able to apply a lot of control to each instance while having all instances completely isolated from each other.


Published at DZone with permission of Alan Cassar, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)