I've been a zone leader with DZone since 2008, and I'm crazy about community. Every day I get to work with the best that JavaScript, HTML5, Android and iOS has to offer, creating apps that truly make at difference, as principal front-end architect at Avego. James is a DZone Zone Leader and has posted 639 posts at DZone. You can read more from them at their website. View Full User Profile

Galileo Podcast Series: ECF and Distributed OSGi

  • submit to reddit
Artist:DZone Plays:53
Length:41:09 minutes (37.68 MB) Downloads:18111
Format:MP3 Mono 44kHz 128Kbps (CBR) Download:Click to download

About this podcast

The next project in my series interviews around the Eclipse Galileo release is ECF. I spoke with Scott Lewis, project lead, to discuss the ECF project, and find out the importance of Distributed OSGi. 


Find out more about ECF


James Sugrue: Could you introduce yourself to the listeners please?

Scott Lewis: Sure. As James said, I am the Project Lead for the Eclipse Communications Framework and I have been the Project Lead for about five years now. ECF as a project was started as a technology project and has recently moved into being a runtime project. I have been doing communications and distributed systems work for too long now, a good 15 years I guess, professionally. I am still kind of interested in building such systems and working on them. So, I am doing that with ECF.

Sugrue: You have been the ECF project lead for quite some time now. Are you involved with lot of other things in Eclipse?

Lewis: Yeah, I have been involved - I was an Elected Committer Board representative for two years, a couple of years ago and I have also been working some with the P2 project. ECF has been contributing the file transfer part of P2 for doing the install and update mechanism within Eclipse. And I have also worked at several companies whose products have been based upon Eclipse or parts of Eclipse including BEA and now EclipseSource, who is my current employer.

Sugrue: And what does EclipseSource do?

Lewis: Well, EclipseSource provides a combination of consulting, support and products, all based around Eclipse and Equinox and so I work on some of those products as well as provide support and consulting services when needed for the parts of Eclipse that I am most familiar with.

Sugrue: It sounds very interesting. Earlier on there you mentioned that ECF has moved onto being top level runtime project, what does that actually mean?

Lewis: Sure. Well, the Eclipse Foundation has a number of top-level projects and those top-level projects are organized into sub projects and ECF moved over from being in the technology project into being in the runtime project during this past year. And what that means to us for the project is that we are working sort of more directly with some of the other projects in the runtime top-level project. That means working closely with the Equinox folks, working more closely with some of the other projects like TopLink and trying to support kind of different use cases for runtimes within Eclipse and use cases around server applications and RCP applications as well as Eclipse, the IDE.

Sugrue: So, could you describe the ECF project in a nutshell?

Lewis: Sure. ECF is basically a framework for building applications that require communications. So, those applications can be applications that are for human to human communications, so things like instant messaging, chat, Voice over IP and real-time shared editing. We are doing APIs for all of those things. Also, ECF can be used to support machine to machine communications - so distributed services, SOA architectures, service oriented architectures and other kind of server to server or peer to peer communications are also supported by ECF.

Sugrue: OK. I mean, really to me, ECF always seems like a really really important framework for the future. I think you probably agree with that?

Lewis: Well, yeah, I definitely do and I think it is important in both of the respects there that I was talking about for tool collaboration and real-time collaboration in not just Eclipse and IDEs. I think it is important there because distributed teams are sort of the way that software is done these days. But also more and more people are building distributed systems for like SOA-based approaches where there are multiple kind of machines all coordinating to provide the functionality needed by a single application. And that is also getting much more common. So, ECF's role in those two forms of communication, human to human and machine to machine, are also I think very important.

Sugrue: And I think the real-time shared editing function, it really is one of the showcases for ECF.

Lewis: Yeah, I think it is very exciting and hopefully everyone has had a chance to kind of see the video that's online. It is an application that I think is a great one for both Eclipse and ECF because it shows off  Eclipse's strength as an editor basically, providing editing capabilities that are way beyond normal text editing. And then kind of adding the ECF kind of real-time shared editing allows you to show kind of new usage models, like pair programming support right within Eclipse can be delivered without sort of fundamentally re-architecting the entire platform.

Sugrue: Yeah. I mean I saw the video and I thought "that's really cool".  I had ECF installed in my own Eclipse installation, tried it out and I just couldn't believe that it was there and that it was working without me having to do any configuration. I was able to do real-time Java editing, which is cool because you are working at home and you can work with someone that's in the office and that is just brilliant.

Lewis: Well, thank you. We are very proud of it and actually our intention is to put a lot more work into that piece of ECF precisely because we think we can do not just real-time shared text editing but also other forms of shared editing to support used cases of like creating kind of graphical models with shared editing and other kinds of used cases there with multiple persons - not just two, but supporting more than two - as well as used cases around things like presentation and review as well as kind of immediate multi-person construction.

Sugrue: What's coming out in the Galileo release train in ECF 3.0?

Lewis: Well, there are a couple of things that are new. One is we did a lot of work in this past year on supporting Distributed OSGi. Now, I can talk about that a little bit more, but I want to list some of the other things that we did too, in addition to the Distributed OSGi work. There's also a lot of work on network service discovery. Then there was a new user search API added to support instant messaging and present systems that have user search, like Twitter. We also implemented an Eclipse Team Provider. The team API and Eclipse is the API that is used to support source control systems, like CVS and Subversion and other source control systems. We added a team provider that does real-time synchronization of resources. So, as you change the resources, they're updated on other team members' local workspaces.

We also generalized the real-time shared editing that we were talking about and created what we call the sync API for doing distributed synchronization and conflict detection and resolution. Then we also added a number of providers to ECF. These are the protocol implementations that are underneath, have the ECF APIs. One was R-OSGi which stands for remote OSGi.

We also added a service locator protocol, which is a standard network service location protocol. Zeroconf support, we added, which is the protocol used by Apple iTunes and other Apple devices like the iPhone, and the iPod Touch. Then we added ActiveMQ, which is an open source JMS provided. Then we also added a provider based on JavaGroups, which is a reliable multicast based protocol.

Then we also have a number of other small API additions and simplifications as well as the usual many bug fixes over the course of the past year.

Sugrue: It sounds like you've been very busy. Just before we go into  Distributed-OSGi, the team provider sounds interesting. So, basically if you've got real-time synchronization, does that mean you don't use any version control? You can just base one person's set of files as being the master set, or how does that work?

Lewis: Well, it can be used that way, but it doesn't actually necessarily imply that a version control system isn't useful. Having real-time synchronization of resources is, sometimes, terrific. But, it's not necessarily what you want in a synchronization kind of approach. So, there are lots of use cases where you really do want to have a kind of commit happen before others are made aware of the changes, because the changes are maybe temporary. So, having real-time synchronization of those kinds of changes may not necessarily make sense. Real-time synchronization in a team provider is really cool and appropriate for some use cases for distributed source code control, but not all of them.

Sugrue: OK, so use with care.

Lewis: Yeah. Well, use with care, certainly, but it's not meant to displace the regular version control systems, but rather support new use cases for immediate multi-person, you know working on the same source at the same time and being able to be aware of what the others' changes are.

Sugrue: So Distributed-OSGi is probably the main thing in ECF 3.0, seeing it was at the top of your list.

Lewis: Well, yeah, with the move to the runtime project and the work that's going on within the OSGi Enterprise Experts Group, that was a big emphasis for us over the past year.

Sugrue: What exactly is it as opposed to normal OSGi?

Lewis: It's basically taking the OSGi normal service model and making it so that you can access services via remote frameworks. So, OSGi already has a really strong and well established service model. Over the past few years, there have been a number of folks who have become interested in extending what has already been shown to be clean and simple and a really nice general service model and extending it to allow remote services to be accessed. So, what we've done is create an implementation of a draft specification that's coming out of the OSGi Enterprise Experts Group. That draft specification has the name RFC119. It's a draft at this point and it hasn't been completed. It's expected to be completed by the end of 2009 along with the OSGi-4.2 release.

But, we've taken the existing specification in draft form and implemented it using the ECF APIs so that people could begin building Distributed-OSGi applications using Equinox and ECF and benefit from the effort that's gone into creating the standard so that it's easy to create those applications.

Sugrue: OK. Is it much easier than writing a standard OSGi application, to use the ECF Distributed-OSGi?

Lewis: Yeah. It's basically as easy as using the OSGi service interfaces. So, if you create a new service, all you do to make that service available is register that service with the OSGi service registry. What Distributed-OSGi does is, basically, you do exactly the same thing to make that remote service available, but you just specify a property on the service saying that this service should be made available to remote to other frameworks. That triggers all of the machinery underneath to publish that service via a discovery mechanism so that other clients can discover that new service, and then allow the discovered service to be accessed with Proxy to actually call the methods exposed by that service.

Sugrue: And, why is it that if you have to implement Dihestributed OSGi rather than Equinox itself being the implementation?

Lewis: Organizationally, Equinox and ECF are both part of the runtime project, so we sort of work together. Essentially in this case, what we did was just coordinate between us and agreed that ECF would be a more reasonable place to implement this part of the OSGi specification, because ECF already had a number of the components that went into implementing the distributed OSGi specification. So, not only did we have a number of those pieces already available within ECF, we were able to kind of take the load off the Equinox team for implementing that piece of things. They were happy to have another runtime project take on the responsibility for implementing that piece of the OSGi 4.2 spec.

Sugrue:  I've messed around a little bit with Riena and remote services. How does your ECF implementation differ to how Riena approached distributed OSGi?

Lewis: Well, because Riena was developed before the RFC 119 specification was actually created, they have their own distinct API for registering it and providing access to remote services. Applications that use the Riena API are therefore bound to Riena specifically and can't be used with other implementations.

Why is this important? One reason is that frequently folks who want to create distributed applications want to also be able to switch between the use of different protocols and serialization implementations in order to deal with changing interoperability requirements or changing security requirements or changing deployment requirements.

So, what ECF has done is we have an API that is used to implement the distributed OSGi spec called the Remote Services API. This is a protocol independent API, which has a number of different providers that have already implemented this remote services API.

We have one based on XMPP, which is the Jabber protocol. We have one based on Active MQ. We have a provider based on the ECF generic protocol which is just a simple TCP messaging protocol. And we have one based on JavaGroups and Skype.

We are always working on other providers as well so what we are hopeful of is that the Riena team will take the Riena work and essentially create a provider out of Riena and that will then kind of make it so that those who are using the RFC 119 standard API will then be able to just use Riena underneath to implement that API and, therefore, be able to move among different providers based upon their requirements for their particular app.

Sugrue: That makes a lot of sense when you put it like that. So, you've got a lot of providers already at the moment. Are there plans for lots more in the future?

Lewis: There are, as many as we can take on. We are getting offers for contributions from our community for other providers and we are very pleased to see that because we would like to make it so that rather than having just a small set of providers, there are many more out there so that people can, and we would hope would, implement the providers completely separate from us. If they wanted to do a commercial provider based upon their own existing system or their own existing protocol, then they are completely welcome to do that. So, yes, we would like to see lots of different providers and hopefully that will happen more and more over time.

Sugrue: And, the Skype provider actually, while we are talking about providers, is unique as it requires the user to start on Skype in order to get it to work. Are there plans for a more seamless integration?

Lewis: You're right. The Skype provider is based upon the Skype client rather than kind of talking over the wire directly to a Skype server. We would love to see sort of what you were referring to, I think, as a native implementation of a Skype provider, and that is one that didn't depend directly on a Skype client. The thing that is holding us back there is that the protocol for Skype at the wire level is not open. So although we would love to see a native provider like that, until we can have some full end knowledge of the wire protocol, that's a difficult thing. So, we can't guarantee that we can do that unless we sort of reverse engineer the protocol and sometimes organizations like Skype may not be welcoming of that kind of thing.

Sugrue: Right, so you kind of need someone from Skype to come to you in this case?

Lewis: Or, if they open the protocol, we'd be happy to take that protocol spec and implement it ourselves. But, until some kind of change occurs, we can't address this as fully as we'd like.

Sugrue: Is there any other way of getting a video task, like your instant messaging, integrated into your environments at the moment?

Lewis: We have been and are working on Voice over IP editions, that is, using protocols like Jingle, which is the Google Talk Voice over IP protocol. We also have a Google Summer of Code student who is working on a SIP, Session Initiation Protocol implementation. But, we don't currently have any work going on for video within ECF, although we would welcome that work if there is someone who would like to contribute to the project, and or join the project as a committer and who is interested in working on video, we'd love to see it. But, we don't have anything going on with that right now.

Sugur: Do you have a large set of committers?

Lewis: We have about seven active committers right now, which is, I don't know, kind of medium/small I would say. But, all the committers with ECF are, right now, part-time and that's including me also. And we're getting more and more contributors to ECF and so that's actually growing, I think, a lot faster than the number of committers is the number of contributors, which is, I think is a really great thing for the project because, it makes it possible for us to kind of grow and add on to ECF without necessarily kind of adding committers and growing a relatively large core team. And, as the project lead, kind of keeping things kind of small and with as little sort of management or organizational overhead as possible, is something that I'm trying to do.

Sugrue: As you know yourself, I have been working on ECF stuff and, I mean, working with any of the committers, it's very easy. Working as a contributor, it's easy for the person who is working, I guess, there's less need for commitment. Plus, the team behind you at ECF is very friendly and very open to any ideas and, I mean, I found them, generally, very, very helpful.

Lewis: Well, that's great to hear. I mean, as the project lead, I try to kind of have a really sort of open project in the sense that, not only are the people who have been committers on the project, kind of open about what they're doing, but we're also sort of as much community-driven as we can. And so, if people who sort of want to see something more or something other in ECF are willing to work with us, we try our best to kind of make that happen. We are somewhat limited in terms of resources, so we're not going to be able to do something as large as Google Wave or something immediately by ourselves, but we do have what I think of as a good, open community-driven ethic about the project.

Sugrue: Apart from the providers, what are your plans for the future of ECF or where would you like to see ECF going?

Lewis: Well, a number of directions, as many as we can manage. We are trying to move into kind of more and more audio and video. We are doing work with Jingle for Voice over IP. We already have a Skype-based provider that works with Skype for doing Voice over IP within Eclipse. We also are quite aware that there are a lot of folks who are interested in building applications of REST-based protocols and so we're working a lot now with implementing REST-based providers so that people who have REST protocols, REST standing for representational state transfer, those folks can easily use ECF APIs and source code for building their applications.

We also are trying to support more and more server applications. So, since we've moved into the runtime project, we are kind of broadening our base of applications that we are supporting and so supporting sort of folks who are doing RCP development as well as Equinox server-based development as well as Eclipse and even sort of small device, network device development based upon Equinox. Those are all kind of areas where we think ECF could add value to what people are doing at the application level.

Sugrue: As you know again, working on TweetHub and our Twitter RCP client, it's working out very well. Marcelo is there working on the provider level, adding things to the Twitter provider that we think we need for our clients and then just working away on the RCP layer and trying to make a nice rich application to really showcase what ECF can do. So, I'm looking forward to having that ready.

Lewis: Yeah, we are too. I mean, I think TweetHub is a really a very exciting application. There's a lot of interest in Twitter these days and having kind of an open source and open client based upon Eclipse and Eclipse RCP user interface technology, I think is a really great application.

Sugrue: Yeah and, hopefully, we can move it on to the web with RAP, although I guess we have had Twitter web clients already with Twitter.com. But, I think there're a lot of great ideas that have come from people just in the Eclipse community, thus, we put up what people like to see and that's Twitter clients and there's some great ideas and they're great requirements.

Lewis:  Yeah, and  I think, there's also a lot of opportunity for integration between sort of like tooling and the functionality provided by Twitter to support sort of distributed team awareness and those kinds of use cases.

Sugrue: It's really good. The Twitter provider remains outside of the core of ECF at the moment, doesn't it?

Lewis: It is right now, although there's every opportunity to kind of take it into the core of ECF along with any other work that people want to contribute to the project under EPL, it could be distributed as part of ECF although that's not necessary. If someone wants to kind of do something interesting and just make it available separately from ECF, that's also completely fine with us.

Sugrue: OK. So, are you working with more Eclipse projects with ECF now?

Lewis: Yes, absolutely. I mean we are supporting the P2 work within the platform in Equinox and Buckminster another project that is using ECF and there are others that have expressed interest in, they're in a project as well a couple of others, including the E4 project, which is something that we may end up supporting through some of the work we've already done.

Sugrue: And what are you most proud of within the ECF project?

Lewis: Well, I'm proud of a number of things since I've been involved with the project for quite a while. One of the things I'm most proud of, I guess the thing I'm most proud of is that the project really has kind of taken on a life of its own. And a lot of people are doing a number of different things with the project that I hadn't thought of and other committers hadn't thought of. And that's really exciting because the notion of community kind of projects that are really sort of driven by the community needs and have a really high sort of level of communication with their own community, I think, is a really exciting way to do software development. I think we're beginning to see that really flower with ECF. So, I think, kind of getting to that point in the project is the thing I'm most proud of. I think there's been a tremendous amount of terrific software development for the project to this point and I'm expecting there'll be lots more, because it's an area where people are interested and I think it tends to attract folks who are really topnotch folks and interested in doing really good and effective work.

Sugrue: And I've heard there's an ECF book on the horizon? Do you have any details on that?

Lewis: Sure, we have plans for an ECF book and, although we're just sort of starting off in that direction, we have yet to kind of even complete a full outline. We'd like to try an approach for creating the book that is maybe a little bit new. We'd like to try to have community contributions actually make it into the book and also commit to have all the proceeds received from the book sales go back into the project. So, both sort of community created content, where people who are contributors and committers can contribute to the creation of the book and drive the creation of that. But also, kind of receive the benefits of whatever the book brings in and use it to feed back into kind of making the project grow and be able to support it in ways that wouldn't be possible otherwise.

Sugrue: That's a really good idea, using the proceeds to keep the project going. That would be a really good model for a lot of open source projects.

Lewis: Yeah, I hope so. We'll see how it works out. I mean, obviously, we're just starting out on that road and we've got a number of things to kind of figure out there, but we'll see what happens.

Sugrue: And what are you looking forward to most in Galileo?

Lewis: In Galileo, well, I guess getting it done. I mean, it's maybe a little bit boring to say, but over the past few months, my life has been doing builds and getting the software kind of into distributable form. And I'm looking forward to getting back to development on the next release and getting input from people about sort of what they want to see out of ECF and what they're also hearing about what people are using it for and what they're doing with it and what kinds of things they want to see next.

Sugrue: OK, so back to creating new fresh classes and packages.

Lewis: Yup, back to development.

Sugrue: Have you noticed any changes in the Eclipse ecosystem over the past year?

Lewis: Well, one thing I've noticed is that in the ecosystem the importance of innovation really seems to be rising. That is, there are now a number of projects within the Eclipse Foundation umbrella, which are now getting a lot of usage and some more exposure. And so, this is, I think, a really good development for Eclipse and that is sort of the broadening out from being just an IDE. I mean, having Eclipse become more of a platform for RCP apps or Equinox servers or network devices, as well as a tooling platform for doing great software development tooling, I think is a really terrific kind of change over the past couple of years. And, I hope it continues.

And we see sort of more and more different usage of Eclipse outside of just being a great tooling platform.

Sugrue: And what are the most interesting uses of Eclipse that you've seen?

Lewis: Well, of course, any that have used ECF are interesting. I mean, there is a project called the Coffee Project, which used Eclipse and ECF to create a collaborative environment for education, software development education. And that was really exciting for me to see. I'm also extremely excited that by the appearance of Google Wave, in fact, we have plans to implement the Google Wave server-to-server protocol as a new ECF provider. And we think this will be really straightforward for us to do given that Google has made the Wave protocol open, although the protocol development hasn't yet completed, they've made sort of a commitment to be open and kind of work with the community to complete the protocol development.

So, I think that's very exciting because with the combination of the work that we've already done with real-time shared editing. And Google Wave, that would allow any Eclipse-based editor to potentially kind of use the Google Wave, well, interoperate with Google Wave, so you could have both a single kind of editor which people could use from a, say, JavaScript based browser client, as well as from an Eclipse-based kind of IDE editor. And they could do real-time shared editing with one another in those two environments and interoperate that way. I think that would be and that will be very exciting I think for us, in the coming year, to work on that.

Sugrue: Google Wave does sound intriguing. I suppose when I did watched the presentation, it did seem closely aligned to the types of stuff you're doing. So, it's really good that you're able to kind of hop straight on to it.

Lewis: Yeah, I mean, the compatibility is really already there because we have a provider architecture we can substitute in the Google Wave protocol and they've exposed the protocol in an open way. We can simply implement their protocol and then interoperate with Google servers immediately.

Sugrue: OK. So, I think that's all my questions, Scott, and thanks a million for all of your time. And best of luck with the upcoming versions of ECF.

Lewis: OK, James, well, thank you for being a contributor on ECF and thanks very much for giving me an opportunity to talk about it.