ItsNat Java Web Framework: DHTML on the Server

Talk of there being "too many Java web frameworks" is deceptive, argues Jose Maria Arranz, because their development styles hardly differ. His ItsNat web framework provides a completely new approach, which he describes as "DHTML programming on the server". What's that all about? Here he explains, in an in depth interview.

In this interview, we spend some time investigating ItsNat, which was announced towards the end of last year. ItsNat has been discussed on-line in various places already, in particular in an article called Has Web Development Lost Its Way?, though several more references are found near the end of this interview.

Jose is a Spanish industrial engineer, working as a university professor. He teaches mainly programming languages at the UPM in Madrid, Spain. That's not his fulltime job, because he also runs a small start-up, Innowhere Software Services.

Here we ask him about the background of ItsNat and what it wants to do in an already busy web framework space.

Jose, you created the web framework ItsNat. Why is it called ItsNat?

ItsNat means “It’s Natural”. Why natural? Because “natural” is the framework's spirit: a natural approach to web development. ItsNat promotes a return to roots, a return to simplicity, and a return to the old and venerable W3C web standards as the main technologies for developing new Web 2.0 applications. As ItsNat is heavily based on AJAX, the framework 's “surname” is “Natural AJAX”. And here ends the "fluff" marketing... :-)

Why was it necessary to create another web framework? How does ItsNat compare to other frameworks, like Struts or Wicket or Tapestry or any of the others?

Some time ago I was working on JNIEasy, which is a Java tool to access native libraries without JNI, without having very much contact with the web world. At the time, I frequently read web sites about software development, most of the time related to Java, and so I frequently read news about web development and specifically web frameworks. When I read about these frameworks, I usually thought: “Oh my God, what a ton of artifacts! Java web development is becoming a nightmare, I think another web framework should be possible”. When I say “artifacts”, what I mean is the custom tags and tag libs that are mixed with tons of XML-based programming, language expressions, XML based navigation, XML descriptors for bindings, and so on. The king of the artifacts was, and still is, JSF, although Struts is not very different.

The JSF rationale is fine: markup is generated and adapted to the runtime environment and there is less freedom for the developer and more power to the framework. When JSF was born, the mobile internet was starting with the “multi-channel” plethora, the JSF architecture was designed to this kind of environment, while hiding the HTML as far as possible and replacing it with new JSF-defined markup, because WAP/WML technology was different to HTTP/HTML. Another focus was always on tooling: tools love XML, and JSF was conceived to deal well with tools, which is the reason why it is so declarative. But WAP days are gone; now the mobile Internet is HTML and HTTP, and now most of that special markup is overengineered because nearly 100% of web clients are HTML devices...

Then Wicket came with all the good stuff it carries: pure HTML used as patterns attached to Java components. Wicket is fine and ItsNat was motivated partially by Wicket. But the pure HTML promise is broken in Wicket too, with custom tags like <wicket:extend> <wicket:link> <wicket:panel> and < wicket:head>. And, when you see some components, for instance the tree component, the HTML content of the component is missing in the template and is generated as a typical black boxed component. This is not very different to custom tags generating markup. I am not railing against Wicket, this is the correct way to improve web frameworks (I would do the same) and is a direct consequence of the traditional techniques of templating.

Next, AJAX came to change everything. Most of the current frameworks were designed before the AJAX era. Before AJAX event/component based development, simulating a desktop application was achieved using complex techniques to keep track of user navigation, including back and forward buttons. All of these artifacts oriented towards classic (non-AJAX) navigation are now examples of overengineering. An action-based framework, such as Struts, does not have very much to offer in a world of applications with a single main page, similar to the typical desktop application.

JSF saw in AJAX a new opportunity to shine: AJAX development is very hard if the developer has control of the HTML layout, the black-boxed approach would win without frameworks like ItsNat. Then came GWT. GWT is heavily based on AJAX but again the same mantra: "I’m sorry, the markup is mine". Furthermore, JSF offers a static layout of the page, I can (mentally) "visualize" the final HTML layout, while with GWT the markup is absolutely lost.

Why is it so important to keep control of the HTML layout?

Because that is the web culture: the web is an amazing place of visual innovation, including web applications. When anyone develops a Swing application, there is not very much space for visual design, unless you are Romain Guy or Chet Haase... in the web space, anyone can change the visual appearance of a web application. Furthermore, on the web it is not difficult to create a list or a table containing complex forms, while in Swing the JList or JTable is not so easy to customize.

The key is the templating technology: most (all?) web frameworks manage the markup as "dead code", that is, as plain text, while only the custom tags, expression languages, and so on, constitute the live code. This introduces severe limitations and, in the long run, the blackbox approach wins because there is no compromise of the framework with the web designer, so that the web designer loses. Another problem of the current templating technologies is the view-pull approach: usually the view is “executed”, which recollects data much like an imperative language.

Returning to the beginning, with the thought "another web framework is possible", AJAX made me think up the opportunity to build a new radical alternative to the “dead code templating” approach: build the template as a W3C DOM tree on the server, such as the browser does on the client. The server repeats "the nature of the client": the client DOM is a clone of the server DOM. This approach fits perfectly with AJAX, simulating a W3C browser on the server receiving W3C DOM Events, and any change to the server DOM can be detected automatically using mutation events. In this way, the framework can detect what is changed in the server DOM tree and synchronize the client DOM , generating the appropriate JavaScript adapted to the browser. The initial template can be a pure HTML file with no special markup and view logic, because the view logic is performed in Java using W3C DOM APIs and all of the goodies of Object Oriented Programming.

What does the user gain with this approach?

With this approach the developer has absolute control of the layout. In fact, the developer has never had so much control before. I think this is revolutionary in a time of blackboxed components. Entry level is very low. Web development with ItsNat is amazingly easy. Only a very few ItsNat interfaces are needed, which mostly are wrappers of servlet interfaces and Java W3C DOM APIs, which are mainly the Core part and Events. In short, the server is coded as DHTML in a browser, but with Java and W3C DOM standards.

The DOM on the server approach is not new. In fact, it is a very old technique used previously for classical navigation, such as in the old Cocoon version and in Barracuda. It is unused today because a DOM tree consumes a lot of memory and takes a lot of time building and serializing. Many years ago I developed a classical Java web application using DOM and “primitive” custom tags when JSP was not a mainstream technology. Today is not the 90s, 64 bit processors have broken the 2Gb barrier, web applications save the state on the server to improve the user experience, and we are in the AJAX era, transporting minor changes from the server…

Anyway, the DOM tree consumes memory, and building and serializing a DOM tree takes time. These problems are solved with ItsNat, using shared DOM trees, where new DOM trees are cloned, and using caching techniques. Static parts of web pages are saved and shared as plain text that is already serialized in memory, and not as DOM. Only the dynamic parts, that is, DOM subtrees that are going to be changed on the server, are in memory as DOM.

Can you summarize ItsNat's main features for us?

ItsNat is focused on AJAX. However, it can also be used for classical navigation, including JavaScript disabled pages. It can also generate XML, though that is not its focus. Pure HTML templates, view logic in Java with absolute control of the layout, AJAX timers, with the same API as java.util.Timer, an amazingly easy COMET support, SVG inside XHTML support, with the SVG elements being managed by the server too, including components, Swing-like components, such as buttons, labels, lists, tables, and trees, with attachable views reusing Swing, such as its data and selection models.

To explain how ItsNat components work, I usually show the following joke, which is something like the ItsNat-based Rorschach’s test:

Q) What do you see in this markup?

<table id="table1">
	<thead>
		<tr><th>Customer Id</th><th>Customer Name</th></tr>
	</thead>
	<tbody>
		<tr><td>Customer Id</td><td>Customer Name</td></tr>
	</tbody>
</table>

A) Of course, the pattern of a customer HTML table. If you attach it to an ItsNat Table component, you see a Swing TableModel, with row and column selection models automatically using this table pattern.

Q) Next, what do you see in this markup?

<div id="table1">
	<div style="width:50%">
		<div style="border:1px solid;">Customer Id</div>
		<div style="border:1px solid;">Customer Name</div>
	</div>
	<div style="width:50%; margin-top:10px;">
		<div style="border:1px solid;">Customer Id</div>
		<div style="border:1px solid;">Customer Name</div>
	</div>
</div>

A) Easy, the same table, but with a different layout, in this case vertical. If this table is attached to an ItsNat Table component, you see a Swing TableModel, with row and column selection models automatically making use of this table pattern.

Q) Finally... now what do you see?

<h:dataTable id="table1" value="{pc_Customers.customer}" var="varcustomer" styleClass="dataTable">
	<h:column id="column1">
		<f:facet name="header">
			<h:outputText styleClass="outputText" value="Customerid" id="text1"></h:outputText>
		</f:facet>
		<h:outputText id="text2" value="{varcustomer.CUSTOMERID}" styleClass="outputText">
			<f:convertNumber />
		</h:outputText>
	</h:column>
	<h:column id="column2">
		<f:facet name="header">
			<h:outputText styleClass="outputText" value="Custfirstname" id="text3"></h:outputText>
		</f:facet>
		<h:outputText id="text4" value="{varcustomer.CUSTNAME}" styleClass="outputText"></h:outputText>
	</h:column>
</h:dataTable>

A) Err… I have a headache.

But for me, the most interesting features are server-sent events and remote control.

• Server-sent events. These constitute "the web upside down". ItsNat can fire, from the server, a Java W3C DOM Event and sent to the client as a native client event, this event is dispatched as a normal event, if a server listener is listening this event type again is sent to the server and dispatched to the target server DOM element and listener. Server-sent events can be used for automated functional testing, where the testing code is in the server, DOM tree and server data can be used to test the view and the business logic of course using Java. Server-sent events can be used too to simulate user actions to drive the application to a desired state etc. Furthermore ItsNat allows server-sent events directly dispatched to the server DOM tree and listeners, very useful to functional testing without browser (faster), for page bookmarking in an AJAX application and building Google compatible AJAX applications.
• Remote view/control. This is my favorite feature! As the DOM tree of the user page is on the server, it is not difficult to copy the server DOM into a new browser client working as a “spy”, which ItsNat provides “out of the box”. Using timers or COMET, the spy can see the exact state of the user web page and any change of the observed page, which is usually as a result of user actions, is updated on the “spy” page. This could also be multiple pages because many remote views are possible. The current implementation only supports remote read-only views, that is, some remote control is possible on the server. The upcoming version will support complete remote control. The first time I saw another page and kept track of its actions, I said something like: “Oh my God I can’t believe it actually works!"

Can you walk us through a "Hello world" example?

OK, here we go:

1. Create a new Java web project with your preferred IDE, the name is not important we will use "itsnat" .
2. Copy the following jars to the WEB-INF/lib (these jars are located in fw_dist/lib in the ItsNat distribution): ItsNat.jar, nekohtml.jar, serializer.jar, xercesImpl.jar, xml-apis.jar
3. Copy the following JavaScript files (they all located in fw_dist/js) to a new public web folder "js" (for instance <project>/web/js folder in NetBeans IDE): itsnat.js, itsnat_ajax.js, itsnat_msie.js, itsnat_w3c.js
4. Create a new servlet using the wizard of your IDE. In this example it is named "servlet", but this name is not mandatory. The default web.xml is valid as is. Remove any code and add the following:
   

   import javax.servlet.*;
   import org.itsnat.core.DocumentTemplate;
   import org.itsnat.core.http.HttpServletWrapper;
   import org.itsnat.core.http.ItsNatHttpServlet;
   
   public class servlet extends HttpServletWrapper
   {
       public void init(ServletConfig config) throws ServletException
       {
           super.init(config);
   
           ItsNatHttpServlet itsNatServlet = getItsNatHttpServlet();
   
           String pathPrefix = getServletContext().getRealPath("/");
           pathPrefix += "WEB-INF/pages/manual/";
          
           DocumentTemplate docTemplate;
           docTemplate = itsNatServlet.registerDocumentTemplate("manual.core.example","text/html",pathPrefix + "core_example.xhtml");
           docTemplate.addItsNatServletRequestListener(new CoreExampleLoadListener());
   
       }
   
   }

   
   The HttpServletWrapper class is an ItsNat utility class, it only forwards a normal request to ItsNat.
   

   The standard servlet method "init" is used to configure ItsNat and to add an ItsNat template (core_example.xhtml) registered with the name "manual.core.example" and the class which process this template, CoreExampleLoadListener. A template is used to construct the final served page (ItsNat supports templates as page fragments too).
   

5. Create a new XHTML file with name "core_example.xhtml" in a new folder "WEB-INF/pages/manual/", this folder name and location is not mandatory. This file is a ItsNat template, as you can see is pure XHTML (HTML is supported too):

   <!-- <?xml version="1.0" encoding="UTF-8"?> -->
   <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
   "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
   
   <html xmlns="http://www.w3.org/1999/xhtml">
   
       <head>
           <title>ItsNat Core Example</title>
       </head>
   
       <body>
   
           <h3>ItsNat Core Example</h3>
          
           <div itsnat:nocache="true" xmlns:itsnat="http://itsnat.org/itsnat">
               <div id="clickableId1">Clickable Elem 1</div>
               <div id="clickableId2">Clickable Elem 2</div>
           </div>
   
       </body>
   
   </html>
   
   
   

6. Add a new Java class with name CoreExampleLoadListener (the package/location is not important and the class name is only an example) and the following code:
   

   package org.itsnat.manual.core;
   
   import org.itsnat.core.ItsNatServletRequest;
   import org.itsnat.core.ItsNatServletResponse;
   import org.itsnat.core.event.ItsNatServletRequestListener;
   import org.itsnat.core.html.ItsNatHTMLDocument;
   
   public class CoreExampleLoadListener implements ItsNatServletRequestListener
   {
       public CoreExampleLoadListener() {}
   
       public void processRequest(ItsNatServletRequest request, ItsNatServletResponse response)
       {
           ItsNatHTMLDocument itsNatDoc = (ItsNatHTMLDocument)request.getItsNatDocument();
           new CoreExampleProcessor(itsNatDoc);
       }
   
   }

   The method processRequest (implementing the interface ItsNatServletRequestListener) is called when a user loads a new page with a URL:

   http://<host>:<port>/<yourapp>/servlet?itsnat_doc_name=manual.core.example
   

   
   ItsNat creates automatically an ItsNatHTMLDocument object when a new page is loaded, this object wraps the org.w3c.dom.Document object build using the specified page template (manual.core.example), this object is got calling ItsNatHTMLDocument.getDocument(). This DOM document represents the client page, any change performed to the DOM tree will be propagated to the client.
7. The CoreExampleLoadListener delegates to a new CoreExampleProcessor instance:

   package org.itsnat.manual.coreimport org.itsnat.core.html.ItsNatHTMLDocument;
   
   import org.w3c.dom.Element;
   import org.w3c.dom.Text;
   import org.w3c.dom.events.Event;
   import org.w3c.dom.events.EventListener;
   import org.w3c.dom.events.EventTarget;
   import org.w3c.dom.html.HTMLDocument;
   
   public class CoreExampleProcessor implements EventListener
   {
       protected ItsNatHTMLDocument itsNatDoc;
       protected Element clickElem1;
       protected Element clickElem2;
   
       public CoreExampleProcessor(ItsNatHTMLDocument itsNatDoc)
       {
           this.itsNatDoc = itsNatDoc;
           load();
       }
   
       public void load()
       {
           HTMLDocument doc = itsNatDoc.getHTMLDocument();
           this.clickElem1 = doc.getElementById("clickableId1");
           this.clickElem2 = doc.getElementById("clickableId2");
           clickElem1.setAttribute("style","color:red;");
           Text text1 = (Text)clickElem1.getFirstChild();
           text1.setData("Click Me!");
           Text text2 = (Text)clickElem2.getFirstChild();
           text2.setData("Cannot be clicked");
           Element noteElem = doc.createElement("p");
           noteElem.appendChild(doc.createTextNode("Ready to receive clicks..."));
           doc.getBody().appendChild(noteElem);
   		itsNatDoc.addEventListener((EventTarget)clickElem1,"click",this,false);
       }
   
       public void handleEvent(Event evt)
       {
           EventTarget currTarget = evt.getCurrentTarget();
           if (currTarget == clickElem1)
           {
               removeClickable(clickElem1);
               setAsClickable(clickElem2);
           }
           else
           {
               setAsClickable(clickElem1);
               removeClickable(clickElem2);
           }
           HTMLDocument doc = (HTMLDocument)itsNatDoc.getDocument();
           Element noteElem = doc.createElement("p");
           noteElem.appendChild(doc.createTextNode("Clicked " + ((Element)currTarget).getAttribute("id")));
           doc.getBody().appendChild(noteElem);
       }
   
       public void setAsClickable(Element elem)
       {
           elem.setAttribute("style","color:red;");
           Text text = (Text)elem.getFirstChild();
           text.setData("Click Me!"); 
   		itsNatDoc.addEventListener((EventTarget)elem,"click",this,false);
       }
   
       public void removeClickable(Element elem)
       {
           elem.removeAttribute("style");
           Text text = (Text)elem.getFirstChild();
           text.setData("Cannot be clicked"); 
   		itsNatDoc.removeEventListener((EventTarget)elem,"click",this,false);
       }
   
   }

   As you can see, the page view is changed using standard W3C DOM methods at load time, when an event is received. To listen to events, the method ItsNatHTMLDocument.addEventListener is called, this method is very similar to org.w3c.dom.events.EventTarget.addEventListener:
   

   itsNatDoc.addEventListener((EventTarget)clickElem1,"click",this,false)
   

   
   This method adds the CoreExampleProcessor instance as a listener, because this class implements org.w3c.dom.events.EventListener, ready to receive mouse click events. When the user clicks the specified element in the client an event is sent to the server using AJAX techniques and converted to a W3C DOM MouseEvent, and the method handleEvent(Event) is called, any change is transparently propagated to the client as the event result.

   The method ItsNatHTMLDocument.removeEventListener is used to unregister a listener, again mimics org.w3c.dom.events.EventTarget.removeEventListener.
   

8. Now run the application. The following image shows the client page state after the user clicks the first element:

   

Where is ItsNat currently being used?

ItsNat is a baby with a university degree... :-) it is only three months old. I consider ItsNat to be production ready. But then why is it 0.x? Because I want to stabilize the API and collect feedback from users and, of course, fix notable bugs, if any are reported.

How long did it take you to create it?

One year, but it never was full time work. Anyway, ItsNat is not a “pet project” or “Sunday project” and was not done in my spare time.

And do you have tips and tricks for others who are creating web frameworks?

There are two scenarios:

1. If you want to compete in a mature market, for instance by creating a new JSF implementation, you need to be the best or the cheapest, or both.
2. Otherwise, be disruptive, create something different: a new and innovative technology. Avoid DRY (Don’t Repeat Yourself) and NIH (Not Invented Here) temptations. Of course, the big guys will be suspicious about your work, but the smart (“agile”) people may value your work if it improves something, such as if it provides a new style of development, simplifies something, increases productivity or quality, or something similar.

What are the plans going forward?

From the technical point of view, the upcoming 0.3 version, which isn't public yet, already supports Safari 3.0 and Opera 9 partially, though full Opera support is not guaranteed. The limitation of the “same browser family” in remote views is removed so that, for instance, now you can “spy” with FireFox or Safari another web page opened with MSIE, and any other combination.

Other pending features: scalable COMET, current implementation is servlet container independent but modern containers introduce proprietary technologies to avoid the 1 request-1 thread limitation. Full remote control, this feature will open a new world of real time collaboration tools, such as a help desk. More components, attachment to server of already loaded web pages, some (optional) integration with popular JavaScript frameworks, and so on.

From the professional point of view: ItsNat wants to be professional and open source, but ItsNat is not completely free. The AGPL license mandates the release of the source of your ItsNat-based web application, if your web application is non-profit, or you do not have problems with releasing the source code, or both, then ItsNat is 100% free, otherwise a commercial license would remove this limitation. Commercial licenses are licensed per processor/core for closed source applications that are in production. I think production licenses are the best model because the license cost is tied to the benefit the technology provides. If you are a software development company, ItsNat is free for you, and you can deliver commercial applications with no cost, while only the beneficiary, that is, the one exploiting ItsNat, assumes the cost.

I believe this is, or should be, the future, because both worlds, open source and commercial software, fit well together. I don't like the tricks of some business-driven open sourced projects, such as selling the documentation as books, getting you to pay for configuration, binaries that are not distributed for free, and so on. I do'nt like projects that are fully developed in spare time, without a business model behind it, in terms of its“development model” not the software.

Can you give us some references for further exploration of this framework?

Here are some to get you started:

• The main site.
• The online demo, the thing everybody wants to see.
• Javalobby article: "Server-sent events: the web upside down"
  
• An audio interview talking about ItsNat (Spanish)

Do you have any final words for readers of this interview?

ItsNat doesn't want to be all things to all people, and technically it can't be that anyway. For instance, if you need highly interactive components, you may need some cool JavaScript framework on the client, such as Dojo, prototype, YUI, and so on. Many people talk about there being "too many Java web frameworks" and it is true that there are many web frameworks out there. However, when you analyze their "development styles", there aren't very many. This speaks to the greatness of Java: you can select the style you like the most, because every style has pros and cons. I think ItsNat introduces a new web development style, that of DHTML programming on the server. If you like this new style and it fits with your needs, then that's fine and I would like to hear your comments and questions.

Finally, thanks to everybody for reading this very long interview. See you on the Javalobby forums!