I am a software engineer at Google on the Android project and the creator of the Java testing framework TestNG. When I'm not updating this weblog with various software-related posts or speaking at conferences, I am busy snowboarding, playing squash, tennis, golf or volleyball or scuba diving. Cedric is a DZone MVB and is not an employee of DZone and has posted 90 posts at DZone. You can read more from them at their website. View Full User Profile

YAML, the Forgotten Victim of the Format Wars

  • submit to reddit

I have been thinking about providing an additional way of specifying test suites for TestNG for a while. The current format is XML, which has quite a few benefits, but no matter how hard you try to specify an XML file to make it as concise as possible, the end result will always end up being more verbose than you’d really like, especially if that format grows organically throughout the years and ends up having to cover cases that you didn’t envision initially. I designed the TestNG DTD in 2004, and while I think it’s evolved reasonably well, I’m looking for a format that might be easier to work with.

YAML attracted my attention when I started playing with Ruby on Rails a few years ago. It’s fairly expressive and concise, but for some reason, it never really took off in the Java world. These past few days, I started experimenting with YAML and TestNG, and I have to say that the initial results are pretty promising.

Here is a simple testng.xml:

<suite name="SingleSuite" verbose="2" thread-count="4">

<parameter name="n" value="42" />

<test name="Regression2">
<exclude name="broken" />

<class name="test.listeners.ResultEndMillisTest" />

 and here is its YAML version:

name: SingleSuite
threadCount: 4
parameters: { n: 42 }

- name: Regression2
parameters: { count: 10 }
excludedGroups: [ broken ]
- test.listeners.ResultEndMillisTest

The result is even more telling on bigger files:

-rw-r--r--  1 cbeust  502  17638 Aug  6 11:07 src/test/resources/testng.xml
-rw-r--r-- 1 cbeust 502 10732 Aug 12 16:03 src/test/resources/testng.yaml

That’s almost a 40% reduction in size. And the gain in readability is also pretty obvious: here is testng.xml and testng.yaml. By the way, this is the main file that describes all the TestNG tests and the YAML file is completely equivalent to its XML version.

Adding a YAML front-end to TestNG turned out to be pretty easy. I found several libraries and I ended up picking SnakeYAML because it still seemed relatively active (the last activity dates from January) and also because it’s available in Maven.

TestNG’s XML front-end is entirely captured in the org.testng.xml package, which only contains six classes. Each of these classes maps exactly to a TestNG tag (<suite> to XmlSuite, <test> to XmlTest, etc…). As long as you can hand an XmlSuite object that defines the root of your suite to TestNG, the engine doesn’t care what file was used to produce it.

The only task was therefore to parse the YAML file and create all the XML objects that TestNG expects. YAML makes this pretty easy since it lets you map keywords to classes.

The entire effort is contained in these few lines:

Constructor constructor = new Constructor(XmlSuite.class);

TypeDescription suiteDescription = new TypeDescription(XmlSuite.class);
suiteDescription.putListPropertyType("packages", XmlPackage.class);
suiteDescription.putListPropertyType("listeners", String.class);
suiteDescription.putListPropertyType("tests", XmlTest.class);

TypeDescription testDescription = new TypeDescription(XmlTest.class);
testDescription.putListPropertyType("classes", XmlClass.class);
testDescription.putMapPropertyType("metaGroups", String.class, List.class);

Loader loader = new Loader(constructor);
org.yaml.snakeyaml.Yaml y = new org.yaml.snakeyaml.Yaml(loader);
FileInputStream is = new FileInputStream(new File(filePath));
XmlSuite result = (XmlSuite) y.load(is);

This part was really easy. I had to make a few additional adjustments that I’ll gloss over, but overall, the process was very smooth.

Additionally, since XML objects can dump themselves in XML, adding the similar functionality for YAML gave me an XML <-> YAML converter for free, which will come in handy for users who want to convert their files or just see for themselves how their XML files will look once converted to YAML.

As for YAML itself, the specification makes it look harder than it really is, so here are a few simple rules:

  • You can define key-value pairs, lists and maps.
  • Lists and maps can be specified either on one line or on several lines.
  • Single line list: [ a, b, c ]
  • Single line map: { a:1, b:2, c:3 }
  • Multi line list:
  • Multi line map:

There are a couple of downsides to YAML: there are not a lot of tools available for it and it’s also not very easy to validate (I am not aware of anything similar to a DTD or a schema). Despite these limitations, I’m still very tempted to officially add support for YAML in TestNG because of the convenience it brings.

If you have any experience to share about YAML, please feel free to comment.

From http://beust.com/weblog/2010/08/15/yaml-the-forgotten-victim-of-the-format-wars/

Published at DZone with permission of Cedric Beust, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)



Martin Wildam replied on Mon, 2010/08/16 - 3:04am

I also like YAML a lot.

Only thing: I noticed some unclear specifications (at least in the documentation from about 2 years ago), which did not talk about character sets allowed and how to specify.

Apart from that I still use inifile style a lot. In some cases where some sort of nesting is needed I refer to other inifiles or other sections within the same inifile.

Your particular case could be handled also with a starting section like [Summary] and then for each test having a section, e.g. [Test:Bla1], [Test:Bla2], ....

For the classes you could just have one variable "classes" that holds the names separated by comma or if you want to have a shortname for each you could use variable name prefixes as identifier, like


If you want to avoid nexting, you mostly can do at low cost (in the meaning of reduced flexibility as cost).

Liran Mendelovich replied on Mon, 2010/08/16 - 4:21am

Looks similar to JSON.
I don't see that reasons are enough to move from XML to this. Maybe XML is bigger in size, but it's nicer to read IMO,
and about the size:
I haven't check, but in big files, the difference in size can be less significant if the XML is compressed by ZIP or something like that.

Josh Berry replied on Mon, 2010/08/16 - 8:04am

The problem I have with either XML or YAML is that there is not really anything "discoverable" about them.  At least if I am writing my own code, I get some semblance of code completion.  Have you looked into writing your own embedded DSL such that you get the niceties of a readable section, with the added bonus of the code completion most IDEs already support?

Arek Stryjski replied on Mon, 2010/08/16 - 8:56am

I have no idea why YAML loose the standard war with JSON, but if we have a winner for XML successor why start it again?

Jacek Furmankiewicz replied on Mon, 2010/08/16 - 11:57am

Technically, YAML is a *superset* of JSON. Any JSON content is automatically valid JSON. But YAML offers more features, in particular great handling of whitespace, multi-line free-form text (something that would require ugly CDATA tags in XML).

Eamonn Smyth replied on Tue, 2010/11/02 - 9:27pm

Don't forget the latest technology to ruffle the feathers of the pack.

 OpenQL (Quick Language) beats XML, json and yaml on average 50% or more in size and is simpler. Resulting in lower costs for datahouses and increased energy savings for power conscious devices. As the inventor I am obviously biased but the figures speak for themselves. So watch out for OpenQL over the coming years. http://openql.com


King Sam replied on Fri, 2012/02/24 - 10:23am

Interesting side note regarding YAML. Maven 3 in the form of polyglot maven supports pom files in yaml theformat and it does indeed make the files pleasant to work with staying expressive yet tight. A nice ddition to TestNG.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.