Big Data/Analytics Zone is brought to you in partnership with:

Arun is a curious programmer who has spent the last 9 years trying to create better and faster programs. When the sun goes down, he experiments with various technologies and programming languages. He blogs at http://rerun.me. Arun is a DZone MVB and is not an employee of DZone and has posted 22 posts at DZone. You can read more from them at their website. View Full User Profile

Log4j JSONLayout and JLogIndexer

09.01.2012
| 5839 views |
  • submit to reddit
Last year I was working on building a Solr based log indexing and search.  My idea was simple.  Write a custom appender, use XML Layout, parse the XML into a Document and use SolrJ to push to Solr. Very soon reality struck me :

1)  log4j XML Layout is not a good idea for our requirement.  I really dont want to spend time on parsing about 100 XMLs every second.  That is a lot of time even for a StAX based Woodstox.  So, I just extended log4j layout and over-rid a couple of methods to create a JSON Layout.

2)  Using Jackson Streaming API to parse the JSON string from the Layout opened my eyes.  There were too many external dependency libraries (jackson, solrj and their dependent jars) that I was adding on to the application which ultimately will all go to the host application.  That is totally insane.

So, I figured out that way to go is to use

1) log4j JMS appender to put JSON strings into

2) Active MQ running on a app server running Solr.

I am hoping that this should give us the following benefits :

1) No complications on the host application. (copying libraries, classloader issues with library version incompatibility etc)

2) Non-log4j based java applications and non-java applications could use one of the various connectivity options available with ActiveMQ to put messages into its queue.

http://activemq.apache.org/connectivity.html

3)  Extension and replacing the layout and message parsing.

4)  Solr Document construction and the interface (initial version will still be on SolrJ) can be customized.

Published at DZone with permission of Arun Manivannan, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)