Enterprise Integration Zone is brought to you in partnership with:

"Corporate Duct Tape" engineer Richard has posted 3 posts at DZone. You can read more from them at their website. View Full User Profile

Real-Time Search and Anaytics in Minutes with ElasticSearch and Mule ESB

11.12.2013
| 5097 views |
  • submit to reddit

Search, (near) real-time analytics and data visualization are powerful tools driving the Business 2.0 economy. These often come with a slew of new complex tools,  complicated deploy and steep learning curve .However with Mule ESB and ElasticSearch you can be up and running processing large amounts of data in literally minutes not weeks.

ElasticSearch is mostly known for enterprise search, powering many well-known websites like SoundCloud and GitHub. It does this by being a very powerful, scalable JSON document store with sophisticated storage and retrieval options. ElasticSearch has a web based visualization tool called Kibana.

Mule ESB is one of the leading tools to secure, move, control and manage enterprise data from on premise or the cloud.

To integrate these two powerful tools:

  • Download ElasticSearch server from http://www.elasticsearch.com/ , extract and run ElasticSearch, and decide whether you want to change its default node name to prevent it forming a cluster with other default installs.

  • Download MuleStudio, the Eclipse development environment, from http://www.mulesoft.org/


Step 1: Migrate CSV Data to Elastic Search


The 5 simple steps in the flow are:

  • Create a file folder watcher to let you drop in CSV files

  • The enterprise component, DataMapper, can map from a variety of sources: CSV, XML, Excel etc. to JSON, the ElasticSearch native format. In my example, I’m mapping an application log mapfile to ElasticSearch for analysis.






  • The JsonObject is passed through the Mule flow and can be converted to a String for logging and posting to the ElasticSearch server.

  • The final endpoint performs a simple post of the payload to ElasticSearch 



Step2: Drop some test CSV data into the file folder being watched by Mule

... and hey presto each line of the data will be converted and inserted into ElasticSearch.

Confirm the JSON data has been inserted into ElasticSearch with

curl -XGET 'http://localhost:9200/test2/_search?pretty=true' -d '{
 "query": {
     "bool": {
         "must": [ {"match_all": {}} ]

    }
}}'

You can view the CSV data being converted to JSON with the logger in the flow




Visualizing and Searching the Data with Mule and Kibana3

ElasticSearch also provide a data search and visualization tool called Kibana. This powerful webapp can be hosted in Mule’s own Jetty Global component:

Simply create a new jetty component and point it to an extracted Kibana

Kibana, out of the box, provides simple dashboards that will vew all your ElasticSearch indexs, a few clicks and you can have graphs and tables and a search box for your ElasticSearch data.





With a little more tweaking, you can have access to a  powerful analytics and insight dashboards including geo-location searches:

As you can see on the Kibana demo page http://demo.kibana.org/



Just the Tip of the (data) Iceberg 

This simple integration is just the tip of a very promising iceberg, with Mule's vast arrange of  connectors (http://www.mulesoft.org/connectors)  from legacy to the latest cloud offerings the enterprise is able to query and mashup data to all new forms of analysis and products. 

Published at DZone with permission of its author, Richard Donovan.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)