With the Eclipse Helios release just around the corner, Xtext will be providing their 1.0 release of their highly successful framework for creating DSLs. I spoke with Sven Efftinge to find out more about this release.
DZone: First, just in case anyone doesn't know about Xtext, could you explain it?
Sven Efftinge: Of course. Xtext is what we call a language development framework. That is an open set of libraries and domain-specific languages (DSLs) used to implement a complete infrastructure for any kind of language, whether it's regular programming languages like Java or DSLs. It covers all aspects of a compiler or interpreter, such as parsing, validation and linking, and it provides you with a full-blown Eclipse-based IDE for your language.
Xtext is an open-source project developed at Eclipse.org. It is driven by itemis, which is a strategic member at Eclipse. I have the pleasure to lead a fantastic team of highly motivated and skilled developers, and itemis is funding us to work almost full time on Xtext.
DZone: So what are the highlight features of Xtext 1.0?
Sven: Since the last release, we've implemented over 80 new features and fixed several hundreds of bugs. Besides greatly improved performance especially within the linker, we have introduced the possibility to bind and integrate with any JVM language. That means it is a matter of minutes to add references to Java elements such as types, fields, methods and even annotations. In the IDE this results in a tight integration with Eclipse's Java Development Tools (JDT) while at runtime Xtext uses Java reflection to look up referenced Java elements. That's pretty cool and something a lot of users now make use of.
The most important addition in Xtext 1.0 is the indexing infrastructure. Based on a description of your project structure Xtext indexes all contained files and remembers the names and types of your elements as well as any cross references. As a result Xtext now lets you do namespace based linking of any element in any file in your project. In the IDE this information is used to provide a lot of new cool features, like a global "Open Model Element" dialog, where you can search for elements similar to the "Open Type" dialog in JDT. In fact, by default Xtext will leverage the Java classpath mechanism to describe your project structure. That is you can have your Xtext files in jars, class folders or even OSGi bundles and Xtext will reuse the Java classpath configuration to determine the boundaries of your project. Anyway, leveraging JDT is just a convenient default, you can use the index without JDT.
DZone: What are the limitations of Xtext?
Sven: You can implement almost any kind of programming language or DSL with Xtext. There is one exception, that is if you need to use so called 'Semantic Predicates' which is a rather complicated thing I don't think is worth being explained here. Very few languages really need this concept. However the prominent example is C/C++. We want to look into that topic for the next release.
DZone: I see that Itemis are doing iPhone apps now. Is Xtext a basis for this?
Sven: Yes, partly. We have developed a small domain-specific language to develop a very common type of data-driven mobile application. If you search the different app stores it is surprising how often you find this kind of application. With 'Applause', which is the name of that DSL, you can very easily describe such applications and generate an iPhone as well as an Android app from it. It's pretty impressive.
In addition to that we do a lot of different things on modern mobile platforms. Mostly we use this new platform as a great new client technology for enterprise software systems. We already did a couple of cool projects and are currently working on a big one.
DZone: We did a poll recently and it seems that about 66% of developers who answered, don't use model driven development. Can you convince them otherwise?
Sven: I think the term model-driven is used in very different ways. I myself don't like riding that old horse anymore, because there is too much dogmatic stuff going on with it. Especially the ideas behind MDA seem very strange and impractical to me and the technologies and languages coming from the OMG are complicated and also not very well designed. I never use UML for programming nor do I use OCL or QVT. I also try to to avoid the typical modeling terminology but instead use simple and existing terminology from general programming and the language community. I try to focus on writing good software because that is what all this should be used for in the end.
What I can say is, that being able to write languages and code generators or interpreters is a very valuable addition to every developer's toolbox.
With Xtext we try to make these tasks as easy and as much fun as possible. And I think we made some progress here :-) Obviously, as part of Eclipse Modeling (EMP) we cannot and don't want to completely omit the term "Modeling", but I think that not only Xtext but also a lot of other technologies at EMP are very helpful and pragmatically designed technologies. They actually don't have that much to do with the general perception of "Modeling". That is why I call what we do "Modeling 2.0" from time to time. I hope this will give people a hint, that it doesn't fit into the old idea of "Modeling".
DZone: So for anyone who is convinced, can you give us an easy three step plan to getting started with modelling?
Sven: You mean Modeling 2.0? :-)
There are two different very cool things you can do with EMF, which is the core framework at EMP. You can use EMF in any software system at runtime as your domain model technology, similar to how people use JavaBeans. EMF is just much nicer, has much richer semantics and comes with a lot of great additional technology. For instance CDO can really be seen as a superior replacement for any object-relational database mapper. It is highly scalable, flexible and gives you the deep semantics of EMF. If you are interested in that I suggest to watch the webinar on CDO.
Xtext on the other hand uses EMF as the in-memory representation of languages. These models are usually called Abstract Syntax Trees (AST) but in Xtext they are implemented using EMF. These in- memory EMF models repesent the main data the different aspects of a language infrastructure work with and it is and was a very good decision to leverage EMF here. It turned out that the semantics and features EMF provides are a fantastic match to what we needed in Xtext. Also by
implementing EMF API any EMF client can transparently use Xtext under the hood. Which for instance allows to combine Xtext with other existing EMF-based
tools such as the Graphical Modeling Framework (GMF).
If you want to learn how implementing programming languages feels like in 2010, I recommend to have a look at our website, watch the webinar or read the introductory examples in the documentation. You'll be surprised how easy language development can be.
DZone: It seems that a lot of the focus for this release has been to improve the editor support, such as content assist and other useful editor features. Can you give us a rundown of these features? Which was the most difficult to implement?
Sven: Yes, we have done a lot of UI features. An Xtext language IDE should really feel like Eclipse's JDT and the framework should come with great default semantics as well as easy to use API for all the different aspects. During the last year we vastly improved content assist, added support for quick fixes, rewrote the whole validation framework as well as the scoping framework. The editor now supports folding, bracket matching, and auto edit. We have support for semantic highlighting and quick outline. I'm sure I missed a couple of UI features, but we have a very nice new & noteworthy page which not only contains descriptions and screenshots but also small screencasts whenever appropriate.
DZone: The index infrastructure seems to have some additions too. Could you explain these changes and their benefits?
Sven: The index infrastructure was a tough one. Actually we implemented it three times before we got it right. For the last implementation Sebastian and I closed any communication tools and locked up the doors for three weeks, to get it done. We are very happy with the result, we like the core abstractions a lot, and the framework has become an important basis for other cool features we have implemented later.
A very hard thing to solve was the computation of the transitive hull of all affected resources. Imagine you have a broken reference to something called 'MyBookingService' because in the declaration you misspelled the name ('MyBokingService'). In that case you of course get an error marker that the reference can not be resolved. If you now open the declaration and fix the typo, we need to know that the previously broken reference can be fixed but of course we cannot rebuild the whole workspace to find any possible occurrences. So we remember which elements have asked for which names and inform them as soon as such a name has changed. As a language designer you don't have to deal with this complex problem, the index infrastructure solves this for you.
Another very cool thing is the "Dirty Editor State" support. That means, that the builder state is shadowed by the state of all editors containing unsaved changes. JDT tries to do the same but it doesn't work as reliable as in Xtext. :-)
DZone: Also, the performance figures look impressive - linking is 40 times faster, and less builders invoked! How did you get those improvements?
Sven: Actually we hadn't found the time to do comprehensive profiling for the previous 0.7 release. So there were a lot of unused opportunities to improve performance. The most compelling performance gain was done in the linker phase, which previously was also the most expensive phase. Now it's on par with parsing which is very fast. We accomplished this by introducing a generic caching utility for EMF models, which is based on EMF's adapters. The builder framework was introduced in M4 / M5. We optimized it in M6. I can proudly say, that scalability of Xtext 1.0 is in a pretty good shape.