Galileo Podcast Series: Creating DSLs With Xtext

About this podcast

The next project that we're focussing on in our Galileo podcast series is Xtext . I met Sven Efftinge who works with itemis and leads the Xtext project. We discuss what a DSL is and how Xtext can help you add your DSL into Eclipse. 

James Sugrue: Could you introduce yourself and your company to us?

Sven Efftinge: My name is Sven Efftinge, and I'm based in northern Germany near Hamburg. I'm basically a passionate programmer with big interests in programming languages and their development, as well as domain-specific languages, obviously.

I've been working with itemis for the last two years now, and it's a company with a strong focus on model-driven software development and language design. It has its headquarters in the middle of Germany near Dortmund, and we also have some branches in Switzerland and France and Canada. I lead a small lab with nine very smart people near Hamburg, and the even better thing is that we can work on open source technology there. So Itemis pays us for doing both technology and collaboration on that.

The business model is quite simple there. We try to come up with the best open source software possible, so that the technologies can be consumed liked products. This of course, means we have to care about many things, so that everybody can use it, and it can become mainstream. There is a need for training and stuff, so we do earn our money with consulting in the end.

Sugrue: It sounds really good. So how are you involved with Eclipse?

Efftinge: I've been using Eclipse as a Java IDE since its initial release. And at some point, I was interested in all the small stuff, and I became a collaborator to several Eclipse Modeling Projects. And then, two years ago, I proposed the Xtext Project and became the project leader of that.

I already knew itemis very well, and they just saw it was a good idea to fund me and give me a team to make the project happen. In addition, I'm a member of several internal parts at Eclipse, like the Eclipse Modeling PMC. And I also have become a member of the Architecture Council just a couple of months ago.

Sugrue: So with Xtext, we're talking about DSLs. Maybe you could explain what a DSL is to people?

Efftinge: Yeah, of course. A DSL is a small programming language, which focuses on a particular domain. So such a domain can be, more or less, anything. The idea is that its concepts and notation are as close as possible to what you have in mind when you think about solving that problem in that domain.

So of course, we're talking about problems, which can be solved or processed by computers in the end.

The opposite of a DSL is the so-called a GPL (General Purpose Language), such as Java or any other common programming language. And with such GPLs, we can, of course, solve, more or less, every computer problem because they are complete. But it might not always be the best way to solve it, because it may not express enough, or you write things down, not in the way that you think of them, but in the way you have to do it because of the language.

I'm currently at the Code Generation Conference here in Cambridge, and I joined a keynote by Steven Kelly and Markus Vaulter yesterday. And they came up with a nice example, which is: Imagine you want to remove the core from an apple. You could, of course, use a Swiss army knife to cut it out; and that's, of course, reasonable if you have to do it just once or twice. But as soon as you have to do that on a regular basis, it might be more efficient to use an apple corer than a Swiss army knife for that. So that's the main idea behind that.

Sugrue: Nice analogy. Are there any examples of popular DSLs that we may have heard of?

Efftinge: Sure. There are a couple of well-known examples of DSLs. For instance, SQL is an example that's always stated. It's actually a DSL, which focuses on querying relational databases. Other DSLs are regular expressions or even languages provided by tools like MatLab.

Also most XML languages are actually domain-specific languages. So the whole purpose of XML is to allow for easy creation of new languages. Unfortunately, the syntax is not that nice, so you better use Xtext.

Sugrue: That brings us onto, what the XText Project all about?

Efftinge: Xtext is the framework to build and maintain specific languages. With Xtext, you can create the infrastructure for those languages. Actually we do not only cover domain specific languages, but we can, of course, create programming languages in general with that.

And by infrastructure, I mean parsers, serializers, grammar, compilers, and of course, IDEs. So the main idea is to make the creation and maintenance of such languages as easy as possible.

Sugrue: OK. So any language that I can think of once it textual, right? So before XText, what DSL support was available in Eclipse?

Efftinge: It was, of course, possible to create IDEs for any language using the Text Editor framework that JFace put out, but you have to write a lot of code to do so. And I did that a couple of times, and at one point, I had enough and I started creating the first prototype of Xtext.

Sugrue: I've never actually used Xtext up until now. It's sounding really good. You mentioned there it would be good for any textual language.  Is there any possibility for something going a bit more graphical? I supposed we'd call it "XGraphic".

Efftinge: No. Actually it's called GMF in Eclipse Modeling.

The main project, which XText is a component of, is called TMF, like textual modeling framework. Then we also have the graphical modeling framework.

So that's one possibility to come up with diagram-based languages. We also have, for instance, the test framework. We have GEF. We can use GEF. And, yeah, which not part of Eclipse is tools like graphers we use to come up with views on models, so you can just generate diagrams. So there are a couple of possibilities to use.

Sugrue: OK. There are options there. That's good. So Xtext, it seems very based on model-driven software. So why has this got so popular, recently, and especially why has it got popular in the Eclipse community?

Efftinge: Yeah. OK. I think it's because using these little specialized languages to solve recurring problems in a much more concise way improves the productivity. An improved productivity is, of course, a good thing for everybody involved in the project, not only programmers.

It can not only be a huge performance gain while solving the problems or writing that software, but it also makes it much easier to come back later and understand and maintain the software system. Right? Because you wrote all these things down in a way that you would think of it.

Another question might be, why it took so long to become popular? Because I'm convinced that the idea is just great. And for that, I see mainly two reasons.

First is that the tools only likely become mature and usable by mainstream. Right? So in the last year only people like me, people who are interested in all this stuff, know how to use the technology. That's one reason, so now the things are better and it becomes more mainstream.

The other reason is that there are modeling and modeling. One modeling is more this MDA-like modeling, right? This big up front. Having big models. Coming up with step words. Everything needs to be graphical. Using UML to do the modeling stuff.

This is, at times, very dogmatic and complicated, over complicated. That's not the kind of modeling I like, so my view of modeling can be very different from that. That's also why we, at Itemis, who run the XText project, sometimes emphasize the DSL term more than the modeling term, because we like more this agile... We want to solve real problems.

We are, is some ways, also code-centric. We care about programming. In that sense, I'm glad Ed Merks, for instance, who leads the Eclipse Modeling Project, more or less shares my views on that. He's also that kind of modeling guy I like. Right? 

Sugrue: Yeah. Ed is fantastic. The work he's done on EMF and modelling is brilliant. So what would be the process for me to go and create my own DSL using XText? What would I need to do?

Efftinge: OK. That's quite simple. So if you have downloaded the distribution with everything in it, you can just use a project builder which creates the typical project structure for you.

This project structure more or less consists of two empty OSGI bundles projects which have the needed dependencies set and then there's also a proposed grammar file. In that grammar file, that's the place where you describe the language. So this grammar file, it's a domain-specific language to describe domain-specific languages, more or less.

The grammar language is, of course, is built with Xtext itself. Actually, this self-describing grammar was one of the first artifacts we checked into CVS when we started work on Xtext. And just after that it took us two days the initial bootstrapping. Just, we first had the grammar. Afterwards, we wrote a parser by hand by using ANTLR, in that case, and generated that grammar again so we had the bootstrapping in it. Then we could throw that initial stuff away.

Sugrue: Is there a tooling available for all of this?

Efftinge: Sure. There's, of course, tooling and there's a special editor for that grammar language which supports a number of nice editing features like code completion, syntax highlighting, outline use, syntactic analysis, formatting and, yeah, other things you might know from programming language IDE's such as the JDT.

We have that for the grammar language. You get the same when you create your own DSL. Right? You get all these things, out of the box, with XText. You just write your grammar down and press a button. Generate your infrastructure and you're done. Install the plug-in and you have a full featured text editor, everything cross-linked, navigation and all these things.

Sugrue: It really is that easy?

Efftinge: It's really that easy, yeah. And then, so it's because we come up with very reasonable default implementations for all aspects of languages. All right. We have these aspects of concrete syntax, of course, which has to be described. From that we can derive the models, the Ecore model which is used.

By the way, if you use Xtext, you get EMF resource implementation, a full implementation. So you can use the Xtext models with using Java and you have an object tree and you can store it back to the text again. We have default implementations and you get all this stuff for free. If you want to change something, you can do so, as well.

Sugrue: That sounds brilliant. So, then, if my DSL changes and I add new parts, is it a short process in applying these changes?

Efftinge: Yeah, we've put a strong focus on short turnarounds and iterative development. So, with Xtext, you typically have very little redundant code and, at the same, it comes with a rich default implementation, as I just said. So, yeah, in most cases, it's just a matter of describing your change in the grammar, so changing the grammar and regenerate, and you're done, right.

Of course you might want to do something with that language, so you might have a co-generator or an interpreter, so some code which works against that derived Ecore model, and you have of course adapt that as well. But all the tools we use are statically typed. We use Java a lot, but we also use other templating languages and you get feedback from the IDE that some things have changed.

Sugrue: OK It really, really does sound good. I'm going to have to try this very, very soon. The Galileo release train is the first time for Xtext being part of one of the major Eclipse releases, is that correct?

Efftinge: Yes, it's the first time. Actually, as I said, the framework we ship with Galileo has just been developed a year ago, so we've started to rewrite. Although there is a previous version of Xtext, but we didn't use that code.

So we wrote it from scratch because we had some scalability issues, some architectural issues with that old code, so we just moved the ideas and the concepts and came up with this new code.

Sugrue: OK. And what can we expect for Xtext, now that it has been part of the release train, and what's going to be coming up in the next year of development for Xtext?

Efftinge: Yeah, that's a good question. In the current iteration we focused a lot on developing a solid foundation. We wanted the code and the concepts to be as clean as possible. During the last iteration, we often threw lots of code away and did the same thing again because we found that we didn't get it right the first time.

And on the other hand, we didn't add too much features. So there are a lot of features of course, but as we stated on our website, we value the quality of features over the quantity of them. But however, now that there are some solid bases made up of these clean orthogonal concepts we are very happy with, it's time to harvest some low-hanging fruits.

So if for instance, we would like to add rename refactoring or quick fixes, which should be - especially rename refactoring should be a very easy-to-develop feature. And also we want to improve the integration with all this visitor stuff you asked me about earlier. So we want to improve the visualization, integration with Zest and GMF. So those are the main additional features.

And then there is one very important thing that is we want to come up with language as libraries. So Xtext supports extending existing languages and change them later on, reusing parse and add additional concepts.

For instance, we plan to write a very nice expression language, which is syntactically, maybe close to what Groovy or Scala has, and that means it should be concise and should support things like closure type inference and operator overloading and these things.

But the difference to those languages would be that with Xtext, you're able to change the language on the path of that, right? You can add additional things like new little words for your domain concepts, and you can add that in your domain-specific languages, this expression, which makes this a very powerful idea, I think.

Sugrue: So there is the lot coming up really in the next year, and it will be great if it has those extra little parts for editor integration. I've noticed that itemis is also quite involved in openArchitectureWare. So could you explain what openArchitectureWare is?

Efftinge: Yes, our openArchitectureWare is a modular generator framework. At its heart it has this language called Xpand, which is a specialized co-generator language.

It's a statically typed template language with a pluggable type system, so that you can write co-generators not only against Java types, but also against other type systems like Ecore from EMF, or XSD Schema or whatever, because we have this mapping in there and you don't need to translate them or something like that.

So there are also people who used to use, for instance, XSLT to do XML transformation, and they just switch to Xpand and extend them. And yeah, we have model transformation in it and constraints and it's very open and flexible.

 Actually, I developed this initial version of this openArchitecture framework during my thesis a couple of years ago, and that was also the time when I first came in contact with itemis, the managers of itemis.

 And they had this openArchitectureWare, but it's a well-known framework at least in Europe or especially Germany. And all these things are now going to Eclipse. Have you heard of that?

So we had this first version of Xtext and OAW, and it's now at Eclipse, and also the Xpand template language is now part of Eclipse. It's in the M2T Xpand project. And then there is this workflow engine which is used to configure and define generator sequences.

So a generator typically consists of parsing models, validating them, transforming them, generating code. So we have different steps and each step needs to be configured, and this is done by the workflow engine, which is part of EMFT. And all these things are as well in the Galileo release.

Sugrue: And actually, one thing you mentioned there as well that's interesting, I think, is how popular modeling is in Europe and especially in Germany.  Again, I wonder, why is it that it's more popular in Europe than it is in the United States?

Efftinge: Yeah. I think it is because of these reasons I told you earlier about. There are a lot of dogmatic ideas around this modeling stuff. And I think that especially American and English people, they just don't like that.

And I think I don't like that as well, but in Europe, there are many more people who like this engineering idea. They go along the way with these ideas, so there is much value in these ideas, but at some point, they're just over engineered and over complicated. All this over complication makes some people maybe become frightened of that idea.

We try to solve this by coming up with pragmatic, easy-to-use ideas, and they are, I think, very easy to use, but maybe people have to differentiate between things like MDA and case tools, and UML's tools for modelling, and what we in Eclipse Modeling and openArchitecture are building on. Two different things.

Sugrue: I think one thing that does help to clear everything up is Ed Merks's presentation that he gave at Eclipse Summit Europe, and I think it was Peter Friese that gave the presentation in EclipseCon all about modeling and the misconceptions behind it. I think that probably clears up a lot of things for some people.

Efftinge: Yeah, there are all these misconceptions from people who look from outside on modeling, and there are also a lot of misconceptions within the so-called modeling community.

I thought about coming up with this Stupidity of Modeling talk, version two, in some sense. But this time, I wanted to talk to modeling people and tell them how stupid it is to use UML as a DSL, because it's not a DSL, and can't be, because it has so much stuff in it. There are a lot of misconceptions, also, within the modeling community, so that would also be interesting.

Sugrue: That would make a great talk. Do you think you'll have a chance to do that for Eclipse Summit Europe?

Efftinge: Maybe. Maybe I could prepare for that.

Sugrue: Very good. I'd be looking forward to it. I'll be in the front. So what else are you looking forward to in the Galileo release?

Efftinge: Besides that all this OAW stuff that is now in Eclipse... by the way, I did mention that OAW itself, the idea of the community, is also brought to Eclipse now? Because we founded our working group at Eclipse, and in this OAW working group, we can just take care that the different components which are made up for a great experience work tightly together.

That's the point behind this OAW working group, and of course we also want to make sure that the whole community moves to Eclipse now. We have a lot of users and a lot of active people there, and we want them to go to Eclipse and join the bigger Eclipse community.

But yeah, what else is important in Galileo? We use, for instance, the WikiText component from Mylyn to write our documentation in Xtext. That's very nice, because with WikiText you can just write in the wiki some text, your document, and then use cogeneration to come up with .doc and .pdf in Eclipse Help from that.

We have this version, and we have edits for nice syntax, so we can refer to fully qualified Java names on the CVS left view. We have a little syntax where we write our Java name, and then the link is ultimately created in the documentation, and because we have all that in our workspace, as soon as we rename a class, it is renamed in the documentation as well. That's pretty nice.

Sugrue: That sounds cool, all right. Have you noticed any changes in the Eclipse ecosystem in the whole community over the past year, since the last release, the Ganymede release?

Efftinge: Of course, I recognized the discussion about the future of Eclipse earlier this year in the blogs. I also noted that, given the financial crisis - at least, I think that's the reason - some strategic members quit their membership, but others joined at the same time, so it seems that... yeah.

While Eclipse was previously driven by a couple of big companies, we now have more and more small companies and individuals in it. That might, of course, change some things, and we need to make sure that the whole organization is, in some sense, agile enough in order to support these new, maybe positive, forces and directions which come up now.

Sugrue: And what are your thoughts on the future of Eclipse, and of E4, and things like this?

Efftinge: I think there are some challenges we have to address, and maybe, in some sense, we have to become a bit more agile. We have lots of discussions with API contracts and life cycles, but all in all, Eclipse is doing very well. I like the organization, and it's really fun to work at Eclipse together with all these nice people. And especially the modeling project has grown. It provides incredibly many nice tools and frameworks these days. Even at E4, they use modeling now, right? They model out the workbench with EMF.

Sugrue: It really shows how far model-driven development has come when you see that, how E4 is being built. What are the most interesting uses of Eclipse itself that you've seen? Or Eclipse technologies, I should say?

Efftinge: Using Eclipse as an RCP framework. I've seen many applications of that. It's very useful, of course, and I've also been involved with such projects. I've also used the Equinox as an OSGi runtime in projects where I don't use Eclipse at all. Those are nice applications, I think.

Still, the most interesting use I've seen, and also used, are these cases where people have developed project-specific IDEs. These IDEs have optimized turnarounds, and have integrated DSLs and generators floating with the Java development tools. In the end, the software development is just a lot of fun, because you're very productive.

Sugrue: I agree. Well, it was great talking to you, Sven, and I'll definitely be looking forward to your version two of The Unbearable Stupidity of Modeling, or at least reading about it in your blog post or something. Thank you very much for your time, and best of luck with the release and your work coming up in the next year.

Efftinge: Thank you, James. It was fun.

You can find out more about developing DSLs in Sven's upcoming webinar on July 15