Enterprise Integration Zone is brought to you in partnership with:

Nitin has posted 391 posts at DZone. View Full User Profile

Q&A - Greg Hinkle on JOPR and application managment

  • submit to reddit
Artist:Greg Hinkle Plays:166
Length:23:42 minutes (8 MB) Downloads:11849
Format:MP3 Stereo 8kHz 128Kbps (CBR) Download:Click to download

About this podcast

In this interview, Jopr product lead Greg Hinkle discusses some of the challenges associated with managing and monitoring increasingly heterogeneous enterprise applications.  He looks at some of the basic features of the Jopr management platform such as auto discovery, the monitoring subsystem, configuration and auditing and also highlights the differences between Jopr and the JBoss Operations Network (JON). Greg also looks at Jopr-JMX integration, optimizing performance metric aggregation and previews some upcoming features of the platform.

The complete transcript of the interview has been provided below:

DZone: We're sitting today with Greg Hinkle, product lead on the JBoss operations network and Jopr. Hi, Greg. Can you tell us a little bit about yourself and what you do at Red Hat?

Greg Hinkle: Sure. I've been at Red Hat and JBoss for about four years. Actually, I started in the open source community before joining JBoss, working on some JMX management technologies. I actually founded the MC4J management project. And just looking at the tools out there, I liked to work on management technologies for the other work I did in consulting, and did a lot of work on sort of developer oriented tools. And then, when I came to JBoss, it was a great opportunity to begin working on these same sorts of tools for enterprise use.

DZone: Greg, in your experience, what would you say are some of the risks associated with not having a sound application management strategy? In other words, why do I need to manage my applications?

Greg: You really said it with the first word, "risks." That is the entire deal with management. It's about the uncertainty - not being able to tell when something breaks. I mean, the truth is you don't need monitoring until something does break, but you won't know if you didn't have it. So it's an insurance policy first. But it also helps to run your business, run your applications better, because it's going to give you more information. It's giving you information about how your applications are being used, how well they're performing. Are your customers sitting there waiting for responses to get back to them? Are they getting the experience that you need them to experience with your technologies?

So hopefully, it helps you better serve them with whatever application you're presenting to them. It's also going to allow you to avoid the situation where you don't really know how they're experiencing the application. If you have an application that could be, occasionally sending them exceptions, you need to be aware of that.

And so being able to have log file monitoring - for example, looking for exceptions - so that you can bubble up that information and report on it, so you really know how your customers are experiencing the technology, and how well that information is getting across.

DZone: Now, is application management primarily just a run-time concern? I mean, is it just a matter of I built my application, I throw it over the wall, and now we need to monitor it? Is there something that needs to happen across the life cycle to ensure that applications can be manageable?

Greg: Absolutely, yeah. You've said it right there. Again, it's not so much about just being able to see what happens in production, because if you haven't built the application to be monitorable, the information that's exposed isn't going to be generally all that useful. So you want to really begin this up front, think about what is the useful information. What exceptions do you need to make sure get into the log files, so that you can report on them? How to avoid the loss of information? The other thing is that it's not just system administrators that need to know how an application is performing.

As your developers are working on the next version of the application, you'll want to be able to expose to them information about how the current version is working - how it's performing, and other things that would help them build the next version better.

So if you can expose to them certain levels of administration management, without maybe letting them change the production environments, but just let them get in there and see some of the information, hopefully they can build new technologies into the next versions to improve the overall reliability of the application.

DZone: Who would you say are some of the other stakeholders that really need information from a management dashboard?

Greg: Well, it certainly goes all the way up the chain to anybody that's really responsible for the application. And of course, the amount of information you want to expose and how you expose it is going to be different at each level. But anybody that's a stakeholder in the application may want to know, is the application up and running? Is it happy? So at the very top level, you want to know, what is the cost if it's down? Is it up right now? And what are my options if it does go down? As you go down the chain, you're going to want to know in more detail, OK, what problems are we currently seeing? How often are we seeing them? How is it performing? How is it reacting to different levels of load over time? Those sorts of things.

DZone: JBoss developers have had the ability to implement management systems using the operations network, which is a product currently offered from Red Hat. What is Jopr, and how does it differ from the operations network product?

Greg: Sure. Jopr is our work with the open source community, to release some of the joint technology. And we're hoping that it helps to broaden the use of monitoring and management, across all areas, both developers getting access to it. We know that developers, obviously use a lot of open-source technologies. This is making it easier for them to get in and see the technologies, and perhaps also build their applications to work with these technologies, so that they do become more manageable and they get to production. So Jopr is our release to open-source of all this technology. We also have a project called RHQ, which Jopr is based on.

It's the actual upstream of Jopr, and that project is about the infrastructure for management. So what we're trying to do is build these cohesive sets of projects that work together to provide an overall solution, but also provide different levels and different places for people to interact and build new technologies.

DZone: When would I need to use Jopr versus the operations network?

Greg: Well, the Jopr project again is open-source, and so it does not come with our typical support contracts and the ability to perform patch installation, et cetera, on enterprise applications or enterprise applications platform.

DZone: If I needed additional functionality and management features, I would then move into the more 'stable' operations network product?

Sure, yes. So the operations network product is going to be guaranteed to - it's going to have a [inaudible 06:24] contract, so we can provide support for it and deliver patches. It's also going to be sort of a completed, end-to-end integrated product, so it's not a collection of open source projects. It's that we've taken it together, and productionized it, and tested it together as a cohesive solution.

The technologies are available in Jopr, and they're certainly useful to open source developers. But as you work with our enterprise application platforms and some of our other products, you'll want to move into using the operations network.

DZone: Could you describe some of the basics of the management platform?

Greg: Sure. The management platform is, really about an abstraction for core infrastructure management. It has technologies to help you build management solutions, but nothing specific around managing any particular technology. So built into the system, we have features like auto-discovery, a monitoring subsystem, alerting, as well as configuration management and auditing, operational control, and also content inventory and deployment.

All of these features are provided and exposed to plug-in developers, so that they can build direct solutions for a particular technology. So for example, we can auto-discover running instances of JBoss application server. We do this by looking for those particular JVM processes. The auto-discovery looking for those particular JVM processes.

The auto-discovery, the tracking of the auto-discovery, and the management within the user interface is all part of the base platform. But the actual technology that goes out and finds the JBoss instance is part of the JBoss plugin.

DZone: What are some of the more advanced types of plugins that you'd see developers implementing or building on top of that base platform?

Greg: You can certainly go deep into any one of the specific abstraction. With monitoring, you can do some base monitoring. You can monitor basically at one level a few different metrics. Or you can provide a deep inventory of resource structure, where you can drill in and see low-level details even down to how a certain EJB is performing. You can also do things like response time monitoring, as well as exposing, alerting, and other event operations. For example, we can expose with the JBoss application server all the log file events that are exposed into a log file. We can track those back into the application server and search across them.

Some of the other things that a plugin writer might want to get into is doing specific configuration management. We support the ability to go in and see how an application is configured, no matter what type of application it is. It's all dynamic.

So, if they have a business application that has the necessary configuration, they can actually expose them into the application. We can audit them and track them over time, as well as give them an interface to change the configuration.

DZone: Can I use this platform for managing and monitoring OSGI-based applications as well?

Greg: We don't have any specific technologies yet for OSGI, although that's certainly something we're looking at. But the technologies we use as far as our plugin development are quite similar in how you lay things out. The plugins are run in isolated class loaders, so that each individual plugin writer can use the right libraries that it needs to connect. Some of the other advanced features we do support, and help support plugin writers with, is for example being able to use separate class loaders for each connection.

This is actually an interesting one, but when you have multiple different versions of a product that you're connecting to, you might need to use different versions of a library or different Java classes for zero evasion compatibility. We can help support that as well.

DZone: Can I leverage JMX and other common APIs using Jopr?

Greg: Yes. We do support out of the box a JMX plugin which provides based JMX connectivity. It provides access to a JBoss application server and other application servers, or a standard Java five JMX remoting interface. That basic JMX plugin can provide access to the platform endbeams that are built into standard Java five JVMs. But it also can be extended so that you can monitor your own custom endbeams. If you have endbeams that are customized for your application, you can create an extension plugin to the JMX plugin.

And you can use all the infrastructure that the JMX plugin provides for things like auto-discovery, inventory management, and connectivity to the server. You can just focus on writing the management of that particular endbeam running in your application server.

Some of the other technologies that we have support for is SNMT monitoring. We can track request log files in the standard Apache format and things like that. All these sorts of technologies can be used by other plugins to extend support to new products.

DZone: It sounds like Jopr and Operations Network were built from the ground up with these micro-container development models in mind.

Greg: That's right. We really wanted to build something that could manage anything that came along. As we roll out to other projects, both within JBoss as well as the other applications that are being used with JBoss, we need that sort of cross-technology support. So, anything that you can do with Java, you can do with a plugin.

DZone: What would you say are some of the biggest technical challenges associated with monitoring increaginly larger and heterogeneous environments such as we're seeing today?

Yes, that is definitely a trend we're seeing. The real situation is that you'll end up with different tools for a lot of the different technologies you have. You'll have a monitoring tool for this application over here, and a different monitoring tool for this application. But the two applications may be dependent on each other. How do you really know whether the end solution is available to your customers?

I think the real danger in these different technologies is forcing you into swivel chair administration, where you're trying to monitor multiple screens and track information in different systems that may not expose it properly.

That's another big issue with proprietary technologies, you don't know how much they are going to expose and how easy it will be to integrate it into your solution. I think this is a real key behind going towards the open standards of management and some of the open agent architectures we're seeing discussed in the community.

As well there's providing APIs and publishing data out of the application. That's something that we're definitely taking on. We're trying to; really just liberate the information and the technology.

One of the other big technical challenges, though, is going to be dealing with these different technologies. There are different infrastructure technologies. You have different protocols to collect information from different applications. That was a real key behind going to our abstraction plugin base model.

One system allows us to use SNMT for collecting information from Apache, while it uses native libraries to collect OS-level information. But it can use JMX connectivity to collect information from a JBoss application service.

DZone: Where would you say most application bottlenecks are found? Is it the database tier, the OS tier, or the application server tier? It's a general question, but is there really one place where we're seeing most of the performance bottlenecks?

Greg: I think that as far as management solutions go, the key is not to impact the application, or to reduce the impact as much as possible to the application that you're monitoring. There are going to be elements where to manage and model at the levels you want, you have to get in and instrument the application, and alter it. It's a real balance to find the right level of instrumentation to be able to turn it on when needed, but turn it off in normal cases. That's going to allow you to monitor your applications and really know what's going on, but not impact them negatively.

Some of the things that we do, built into our infrastructure, is aggregating and optimizing the requests of information from the managed application. Rather than remotely polling it for individual metrics, we can group the metric requests together depending on the technologies that collect the information.

And, then we can bundle them up and aggregating them in sending them back to our enterprise server, so that we reduce the number of network round trips and other types of load on the system.

DZone: All the metadata that's generated from the monitoring solution, does that create extra overhead? In other words extra requirements for storage, analysis, and data integration. Does the metadata itself create its own issues around data management?

Greg: Depending on what level of monitoring you're performing, there can be hundreds of thousands of metrics that are falling through the systems on a regular basis. And it can be a lot of other types of information as well. We can monitor for configuration changes, for example. We're monitoring down to the detailed level of configuration files. We can monitor down to the response time level of web applications.

There's so much information flowing that we have to be very careful about being able to store and present it in a usable way. If we have so much information that you can't see anything useful, it's not much good.

Some of the things we do to help that situation, for example, is the possibility to baselines metrics, and then show metrics in your system that currently seem to be out of their baselines. We'll track running averages and those sorts of things, just to give you key data points without much setup about what's going on and what's changing in your environment.

Another thing we do, for example, is compression. It's very useful to be able to see a lot of useful information about how your apps are performing right now. But it's also very useful to be able to compare how your application is performing, let's say, this holiday season to last holiday season.

It's something that only happens yearly. You really want to be able to store information for a long enough periods that you can track the performance over a long period of time. We do things like compression to help that.

DZone: Does it make sense to tie in another solution like JBoss rules with a monitoring solution like Jopr, to enforce policy and overall application governance?

Greg: Yes, certainly that would be a very neat idea. We have been discussing in the community about different ways to provide our information to other applications, an application like a rules engine that could process it and use it to execute. We do have complex alerting built into the system, but it isn't rule-based yet. That would be a very interesting idea, incorporating that into things like LSA management. There are a lot of good ideas in there.

DZone: We're hearing a lot about event-driven architectures (EDA). How do Jopr and the Operations Network facilitate EDA?

Greg:  A lot of the tools that are out there will provide separate solutions for some of the systems that we've incorporated into this base platform. There's a system for monitoring and a completely separate system for configuration management. What we see out of building this into a cohesive platform is the ability to say, OK, I see monitoring has showed me that the application has begun to perform badly yesterday. I can go back in time, and on the same screen with the same information look at a timeline that shows me, OK, this configuration was changed yesterday at the same time that we started to see a problem.

We can use that information, process it together, hook it into our log files so we can see what events are flowing through those, and even perform operations automatically. We can create an alert and have it execute an operation to help to mitigate a certain problem.

DZone: Has the increasing prevalence of virtualization technologies changed the way in which we think about how applications are managed and monitored?

Greg: I think it is starting to change the way that system administrators see Java enterprise applications. For a long time, JEE applications have always been deployed in a sort of virtualized model. We often deploy many JVM instances running separate copies of an application server on a box, so that we can get the usefulness of shared resource usage and higher availability by spreading the risk across hardware.

So, in a sense we've been doing virtualization for a long time. But as we move it down and it becomes more prevalent at the operating system level, you have to deal with new technologies. We're seeing more interest in something that can both understand the application server, but also understand how the hardware is doing, or how the virtualized OS is doing.

For example, we do have a technology demonstration plugin to manage Zen KVM virtualized instances, so we can see how they're performing, as well as how the hardware is performing, and also how the application is performing. It's to combine that information together.

DZone: We're currenty at revision 2.1 for Jopr and the Operations Network. What can we expect to see in future release of the Jopr product?

We're, definitely looking at improving our ability to see a lot of information combined together. So, it's providing more cluster-oriented views that show your applications as they really are logically laid out, rather than how they're physically deployed. Hopefully this will allow us to monitor and manage larger environments with less effort. It's also going to provide new models for seeing a lot of together that simplify this. There are so many other things.

We're, definitely going to be releasing more APIs for integration and extension. We'll improve our plugin API, but also improve our remote APIs, so that all the information we expose can be used in other applications. It can be integrated into cohesive solutions, so that we can take the data that is useful information and process it into a useful server.

We'll also be adding more support for other projects, obviously, such as new plugins and support for other JBoss products. For example, we just released support for the JBoss cellular platform, and we did release a technical preview of monitoring for our enterprise data services products.

But I think you'll also see support for other open source products that our customers have to use, and which are used in the community. It's things like the Hudson plugin that released recently, or our Postgres plugin.

DZone: Do you have any final words of advice on how developers can make their applications more manageable?

Sure. I think that the key is imagining how the application is going to perform, and how it's going to work in production, really preparing. It is something that has to be allocated into the development, thinking about how you're going to maintain the application once it gets into production, and what you're going to do with something goes wrong.

It's thinking of things ahead of time like instrumenting useful information, exposing it, and maybe creating custom endbeams so you can get into real critical data and quickly get at log files. Those sorts of things are critical to making an application be reliable once it gets into production.

So, it's really about planning and seeing ahead.

DZone: Greg, on behalf of DZone I'd like to thank you for taking the time to chat with us today.

Greg: Sure, I'm happy to do it.

Published at DZone with permission of its author, Nitin Bharti.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)