Daniel have developed support for proprietary technology which is processing huge volumes of XML data. Now is working on Register of traffic accidents(J2EE application build on Websphere) for Ministry of Interior of the Slovak Republic. Daniel has posted 9 posts at DZone. You can read more from them at their website. View Full User Profile

XML Pipelines: the Process Description

  • submit to reddit

The world of XML is moving fast, in its twelve year old history has raised many technologies. Many ways to validate, transform or query the data carried by the XML. Actually the validations and the transformations are commonly used together in the processes today so commonly called XML Pipelines.

XML Pipeline

By definition “XML Pipeline is formed when XML processes, especially XML transformations and XML validations, are connected together”, that means if we send the output of the validation as input to the transformation (and that is a common use case), we have just created XML Pipeline, more accurately the linear XML Pipeline. Well non-linear pipeline would be the one, where we decide between two process branches based on a condition, where we are looping, running parallel branches or handling errors on the fly.

XProc, W3C Recommended XML pipeline language

XProc is a XML language describing processes, using XSLT for transformations and the XSD, RelaxNG or the Schematron for validation. The greatest thing about XProc is that you describe processing of the XML document by XML language, using the W3C standards on the way.

There is a catch

Well with the XProc, we describe the process by one language, validate by another language and transform by yet another language. By using the three different languages for describing even the simplest use case of the XML Pipelining, a huge space for flaws opens. If we want to make something as trivial as changing the name of one element, we have to make changes in three different languages.

An alternative exist, it’s called XDefinition

If you remember my previous article called “XML Processing and Validation Merging Together” where I wrote briefly about an interesting XML technology called XDefinitions, you will find one language able to transform and validate XML. And because its ability to validate the multiple inputs, transform them, and validate output again, all written in only one highly readable language, the space for the errors is limited to unavoidable minimum.

Simple XML Pipeline


XML Pipeline by XDefinition
Every XDefinition process can initiate another XDefinition process. This kind of scalability allows us to model linear or nonlinear XML pipelines by one language. If we want to change the structure, we will do it on one place only.

<xd:def xmlns:xd = "http://www.syntea.cz/xdef/2.0" xd:root = "Weather" >

<!-- declaration of variables -->
String $source; /*URL to source data */
float $sum = 0; /*total sum of temperatures */
int $n = 0; /*number of measurements*/

<!-- model of input data -->
<Weather date = "optional datetime('yyyy-M-d')">
<Measurement xd:script = "occurs 1..;"
wind = "required float()"
temperature = "required float(-99, +99);
onTrue {/* process value of temperature. */
$n++; /* increase number of measurements. */
$sum += parseFloat(getText()); /* add value to total sum. */
time = "required datetime('HH:mm')" />

<!-- model of result -->
<html xmlns = "http://www.w3.org/1999/xhtml"
xd:script="create parseXml($source, '*');
/* Parse and validate input data with the Weather xdefinition. */">
<h1> create "Date: " + from("/Weather/@date") </h1>
<li xd:script="occurs +; create from('//Weather/Measurement')">
create "Time: " + from("@time") +
", wind: " + from("@wind") +
", temperature: " + from("@temperature");
create $n == 0 ? "No data" /* $n is equal to 0 */ :
"Average temperature : " + ($sum/$n); /*print average*/


 Input document

<?xml version="1.0" encoding="UTF-8"?>
<Weather date = "2005-05-11" >
<Measurement wind = "5.3" temperature ="13.0" time = "05:00" />
<Measurement wind = "7.2" temperature ="15.2" time = "11:00" />
<Measurement wind = "8.7" temperature ="18.1" time = "15:00" />
<Measurement wind = "3.9" temperature ="16.5" time = "20:00" />

 Output document

<html xmlns="http://www.w3.org/1999/xhtml">
Date: 2005-05-11
Time: 05:00, wind: 5.3, temperature: 13.0
Time: 11:00, wind: 7.2, temperature: 15.2
Time: 15:00, wind: 8.7, temperature: 18.1
Time: 20:00, wind: 3.9, temperature: 16.5
Average temperature : 15.7

 Try this example here online. As you can see XDefinition contains a model of the input document against which it’s being validated. You can even use variables from the validation in the transformation/composition as is $n and $sum (used in the example) that are global for every model of XDefinition. That is very important, because XDefinition can complete validation with non fatal errors. So your process doesn’t have to end with invalid but well formed input. Invalid document still contains the information value and you can work with it.

Why “composition”? Because the word composition describes the process better which is being done by the XDefinition. Nowadays the overwhelming majority of the data generated by every transformation process has XML output. If you take that for granted the  whole XML Pipeline has to be well formed and the transformations are then compositions of smaller nodes. Any composition with the XDefinition is XML data modeling and your pipeline has a complete right to be called a XML Pipeline.

-    http://www.syntea.cz/syntea_web/?page=xdef_guidepost&hl=en_US
-    http://en.wikipedia.org/wiki/XML_pipeline
-    http://www.w3.org/TR/xml-pipeline

clanekxmlpipeswellEN.pdf69.36 KB
Published at DZone with permission of its author, Daniel Kec.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)


Christian Schli... replied on Thu, 2010/11/11 - 6:51am

I prefer the Apache Cocoon project for this kind of task. It's a mature ecosystem build around the concept of SAX pipelines. If you need to build RESTful web services or similar, Cocoon is the weapon of choice. Cocoon's sitemap is so powerful, you hardly ever need to write any Java.

Daniel Kec replied on Thu, 2010/11/11 - 7:54am in response to: Christian Schlichtherle

Cocoon is great I do agree, the problem is the validation that it is still not fully implemented. And the issue with redundancy description of the structure remains ... same disadvantages as the XProc have

Andrew McVeigh replied on Thu, 2010/11/11 - 12:26pm

i had a look at xproc about a year ago when we had a requirement for an XSLT2 pipeline for fpml trade transformations.  i just couldn't seem to find any implementations or any tools around it. it looked to be a moving target, in working draft state, and had been there for quite a while in various guises.

we were using Saxonica under .Net and in the end just made our own simple pipeline language with 10 different step types.  worked a treat, but i always thought it was a shame the xproc thing hadn't resulted in anything concrete.

was i missing a mature implementation of xproc which was available?  if not, what are the actual options -- xproc itself is large and making my own implementation is out of the question. is xdefinition a possible alternative and what are the step types?  i.e. does it have things like choice, enrich etc.  also, does an implementation exist?

Liezel Jane Jandayan replied on Thu, 2011/08/25 - 6:22am

For instance, given two transformations T1 and T2, the two can be connected together so that an input XML document is transformed by T1 and then the output of T1 is fed as input document to T2. Simple pipelines like the one described above are called linear; a single input document always goes through the same sequence of transformations to produce a single output document.-Jonathan Berkowitz

Dinni Davidson replied on Fri, 2011/10/14 - 9:44am

Isn't this kind of software usually being used to build alignment sheet plans for installing real pipelines? I'd appreciate any support in this direction, I am trying to understand the technology since one of our local companies is about to start this kind of project right next to where we live.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.