New Adventures In Real-Time : An Interview With Greg Bollella
Sugrue: What are the main challenges in writing a real-time application in Java?
Bollella: We have to first talk about what real time means and talk about the predictability you get in Real Time Java. So we have our implementation of Java RTS and we run these predictability tests on it. What these tests measure is primarily if you have a periodic process defined in JSR1, and you do everything you can to the machine to make this process wake up as soon as it possibly can at the start of each of it’s periods. Measure How much interference is there from garage collection, file I/O, disk I/O and network operations – all those things might cause the logic of the periodic process to be delayed. That's the maximum interference.
There’s three environments in which you write code in JSR1. There’s the regular Java environment which is based on thread types from java.lang and behaves very like Java. There are two others, the Real Time Thread Context and the No Heap Thread Context.
Real Time Thread behaves sematically just like a java lang thread – same programming model. You can actually take java.lang.Thread and change it’s type to a real time thread and it will still run, so they are perfectly semantically equal. The real time thread gets it’s predictability because of the real time garbage collection system we have in the VM, and a number of other features like the real-time priorities (59 of them), access to Immortal Memory and Scoped Memory, Initialization Time Compilation, Async Events, etc.
So with real time threads, in the RTT Context, in this periodic process testing we can make assertions that no matter what else is going on in the VM, that thread will get control within about 150 microseconds after the beginning of the period. So that’s our maximum worst case latency that we experience. The best case is 1 or 2 microseconds. But the worst case when you have a lot of GC running and I/O occurring, it gets pushed up to about 150 microseconds. With that we’re able to do some pretty interesting things.
In the No Heap area do the same tests. With No Heap Threads, they’re not able to touch any objects on the heap, and so you have special memory areas for them to allocate their objects in and it’s a more restrictive environment – it’s harder to program, but we wouldn’t expect people to do complex things there, just very low level loops doing some sort of high frequency activity. And there we can get down that maximum interference to about 15 microseconds, so you can write some pretty high frequency Java code in the no heap thread, but the programming model is different and a bit tougher to get right.
One of the challenges in Real Time Java would be if you are writing in the real time thread context, which a lot of our customers do is balancing the behaviour of the garbage collector and the threads that are doing the time critical work. The collection algorithm we have allows you to set the priority of threads higher than that of the real time GC, so threads can actually interrupt the GC. The critical section between the threads and the GC is actually very small. It’s within that 150 microseconds window that I talked about. But you have to ensure that these time critical threads, if they’re going to continually interrupt the GC there has to be some periodicity to them in some way, shape or form that allows the GC to get enough cycles to clean up the heap behind them. In typical real time programming you don’t use dynamic memory at all – usually the statically allocate and reuse memory locations. We encourage people to use dynamic allocation and let the GC do the work.
Sugrue: Apart from BlueWonder, what other Java Real Time demo systems have you built?
Bollella: Last year at JavaONE we did a controller for an ABB IRB 340 FlexPicker robot.
We did the whole feedback controller in real time java on top of Solaris for this robot. What we did was take photographs of attendees, took the resulting jpeg, did some edge detection on it, turned it into vector list – manipulated it into a list that was understandable by the controller, produced a trajectory for the robot which picked up a pen and drew a picture of the person on a piece of paper. That was all done with Real Time Threads. We needed a 1,000 HZ process, a 1 millisecond period, to read the position of the motors, find out where they are now, figure out where we told them to go in the last ms, figure out what the delta is to that – we know where we want them to go in the next ms, and we use all that to figure what voltage and amps to send off to the motors. We did that reliably at 1000HZ and drove the robot just fine.
Although there are challenges you can do fairly demanding real time control systems in RT Java. The whole controller was a couple of graduate students and myself – it took less a month and I didn’t contribute much to that.
The other advantage through this kind of robotics is they also build a Java3D simulation of this robot and the control code is exactly the same code and class file for the simulator as well as the code driving the actual robot. So there’s no loss of fidelity – in simulators a lot of the time you need to write the control code and compile through the simulation host – to run on the target you then need to recompile for the target. So we avoid the porting effort between simulation and target – the same set of bytecode can be used, and the timing is well controlled for both.
Using Blue Wonder systems for control means you can do simulation and not have a loss in fidelity. There’s always the problem between the simulator itself and the actual robot – how well do you match simulator and the robot – but that problem exists no matter what. At least we take out half of the problem.
Sugrue: How did Sun get involved in Industrial Automation?
Bollella: There was this large industrial automation company that build power generation plants. You think PLCs for the mid-tier control – so they have a HMI layer and an administration, accounting and training layer up top. The middle layer is a layer of PLCs doing the real time control and then there’s the field devices. They wanted to replace those PLCs with something that was less expensive, more open and more modern. They actually took a competitive product that runs on a version of Linux that they got and hacked up themselves. They speced up some hardware and got it built up by a hardware vendor. They built their control system for power plants on top of that – and it worked really well and they were happy with it. What they weren’t happy with though was the VM was at it’s limit and they couldn’t do any non-realtime stuff on it and they had to do all the integration work. They were specifying the hardware, doing operating systems maintenance and put all the pieces together, which they didn’t want to do. So they came to Sun and we said, “OK we’ll do this box for you and deliver it, install Solaris & JavaRTS and even install your program on it when you start shipping to the plants themselves.” So we’re moving forward on this – that’s how the Blue Wonder project was born.
It’s gotten a lot of interest from other people in the automation space, as well as in the automotive space. So we’re pretty happy about the reception that we’ve got.
Sugrue: Do you think it means that at some stage Sun will become a PLC vendor?
Bollella: I don’t know if you’d say we’re a PLC vendor, but what we’d like to do is produce systems that could be used in the same way that PLC’s are being used.
Think about this – you have a factory or power plant that has 200 to 1000 of these PLCs, Currently they are isolated separate units that don’t talk to one another. To fiddle with one of them you have to walk over to it. But if you imagine replacing those PLCs with Blue Wonder systems every one of those is now just a Solaris box on the network.
And so the opportunity for simple things like NFS – all of these PLCs can be writing log files to an NFS mounted directory on the higher level server. So management and everything else is exactly the same as 1000 machines in an IT shop, because they’re all Solaris. It really makes a big difference to some of our customers – it’s opening up different worlds for them. A PLC is no longer just an isolated blob that has to be managed individually – it’s just a Solaris node on the network. All the advantages that come from that, now accrue to that PLC.
Sugrue: You’re providing the hardware to this company. So it’s a kind of a one off product rather than working for a major automation vendor?
Bollella: Well I wouldn’t call it a one off project – we have a number of people interested in it. It has a part number and we intend to sell them to whoever wants to buy them.
Sugrue: Are they available for the general public or automation companies or is that in the future?
Bollella: Right now we have certifications for CE and FCC certification so we can sell them to anybody who deploys them in the EU and a few other countries. We’re going to start working on UL certification for the US. We’re not worldwide yet, but we can take orders for anybody who wants to deploy in Europe.
Sugrue: How much would one of these cost?
Bollella : It depends on the size of the order – if you wanted to buy one I think it would be around $3000-$4000 dollars for one unit with volume discounts available. That price is without any Profibus cards included. Those cards are not inexpensive; they start at a bit under $1000 for one card from the company we’re buying them from (again, volume discounts apply).
If there’s customers that have a particular project in mind and they’re interested in Blue Wonder and would like to experiment with it, we can get some loaner machines for a few months.
In larger quantities we can drive the price of the system down.
Sugrue: Did you write you own Java drivers for the Profibus cards?
Bollella: We wrote a Solaris device driver that talks to the profibus card. There’s a PC104+ expansion bus in there and that’s electrically the same as PCI, just a different form factor. So we wrote a PCI device driver and then wrote Java classes on top of that so you get write to the driver from Real Time Java code - that comes bundles with Blue Wonder.