Big Data/Analytics Zone is brought to you in partnership with:

Enthusiastic Java, Scala and Haskell programmer with a long history of large and successful systems. Known author, speaker, motivator and coach. Jan is a DZone MVB and is not an employee of DZone and has posted 26 posts at DZone. You can read more from them at their website. View Full User Profile

Haskell, WebSockets and D3.js for Data Analysis and Visualization

01.08.2014
| 6483 views |
  • submit to reddit

This is the first in the series of posts where I will explore a simple application that allows you to analyze and present data in interesting ways. I will be writing a nice AngularJS client, with the WebSocket connection to the Haskell back-end.

If you are impatient, grab the source code from https://github.com/janm399/hwsexp; and run cabal run to start the WebSocket server. To see the (at the moment primitive) user interface open web/numbers.html in a modern browser. You will see (funky & moving) HTML view.

$ cabal run
Preprocessing library hwssexp-0.1.0.0...
In-place registering hwssexp-0.1.0.0...
Preprocessing executable 'ws' for hwssexp-0.1.0.0...
Server is running

result

The Haskell code

Our Haskell server code listens on all local addresses on port 9160 for WebSocket connections. We would also like to maintain a state that is a list of the connected sessions. During the server’s lifetime, we will be modifying this state, which is shared between the threads.

-- |The main entry point for the WS application
main :: IO ()
main = do
  putStrLn "Server is running"
  state < - newMVar newServerState
  WS.runServer "0.0.0.0" 9160 $ application state

Leaving the obvious putStrLn aside, we create the MVar ServerState, which is the state at the server startup. We then use the ServerState value when we then (recall the desugaring Haskell does!) start the WebSocket server. The state our server keeps is a list of the query that the user sent together with the WebSocket connection the server will push the results to. This gives us a good place to define these types, together with the newServerState function.

-- |Client is a combination of the statement that we're running and the
--  WS connection that we can send results to
type Client = (Text, WS.Connection)

-- |Server state is simply an array of active @Client@s
type ServerState = [Client]

-- |Named function that retuns an empty @ServerState@
newServerState :: ServerState
newServerState = []

Great; to complete the picture, let’s add functions that allow us to add and remove clients.

-- |Adds new client to the server state
addClient :: Client      -- ^ The client to be added
          -> ServerState -- ^ The current state
          -> ServerState -- ^ The state with the client added
addClient client clients = client : clients

-- |Removes an existing client from the server state
removeClient :: Client      -- ^ The client being removed
             -> ServerState -- ^ The current state 
             -> ServerState -- ^ The state with the client removed
removeClient client = filter ((/= fst client) . fst)

We now have all the auxiliary code ready; all that we need to do is to provide the implementation of the application function; this function represents what we’d call controller in the old world. It receives requests, and–as a side effect–may modify the server state and produce responses.

-- |The handler for the application's work
application :: MVar ServerState -- ^ The server state
            -> WS.ServerApp     -- ^ The server app that will handle the work
application state pending = do
  conn < - WS.acceptRequest pending
  query <- WS.receiveData conn
  clients <- liftIO $ readMVar state
  let client = (query, conn)
  modifyMVar_ state $ return . addClient client
  perform state client

We first accept the request (we accept any WS requests), giving us a Connection value; we then receive the data that the client has sent, giving us String. Finally, we pull out the ServerState value from the shared MVar ServerState.

We construct the Client value: a tuple containing the query and the WebSocket connection for that query. The rather complex line is modifyMVar_ state $ return . addClient client. We are modifying the shared ServerState. If we expand the expression, eliminating the point-free style, we will have

modifyMVar_ state (\s' -> return (addClient client s'))

We can eliminate s' in the (\s' -> return ... s') equation: return . addClient client is the same thing: a function that takes ServerState and returns IO ServerState. Great. Finally, we can eliminate the final brackets using the ($) function. This gives us the final

modifyMVar_ state $ return . addClient client

That leaves us with just the last expression; one that actually does the work. (We will keep this portion quite light at this stage, but improve it over the next few posts; and believe me, there is a lot to improve!)

-- |Performs the query on behalf of the client, 
--  cleaning up after itself when the client disconnects
perform :: MVar ServerState -- ^ The server state
        -> Client           -- ^ The query to perform and the conn for results
        -> IO ()            -- ^ The output
perform state client@(query, conn) = handle catchDisconnect $
  forever $ do
    numbers < - replicateM 100 ((`mod` 100) <$> randomIO :: IO Int)
    WS.sendTextData conn (T.pack $ show numbers)
    threadDelay 1000000
  where
    catchDisconnect :: SomeException -> IO ()
    catchDisconnect _ = liftIO $ modifyMVar_ state $ return . removeClient client

Dissecting the code, we wrap the forever-repeating computation in an exception handler. This gives us the basic shape of the code. In the forever block, we generate 100 random numbers, each in the range 0 – 100, which we send to the listening WebSocket. Then (Oh the humanity! More on that in the future.) we sleep for 1 second and then repeat. The catch block handles any exception by removing the client from the server’s shared ServerState.

The web application

Moving on, let’s hack together a nice AngularJS web application. We want to connect to our Haskell server, and then display the numbers we receive in a text field, and also–using D3.js–in a pretty chart.

<!doctype html>
<html>
<head>
    <title>Number cruncher</title>
    ...
</head>
<body>
<div ng-app="numbers.app" ng-controller="NumbersCtrl">
    <tabs>
        <pane title="Raw">
            <h3>Raw data</h3>
            <pre>{{numbers}}</pre>
        </pane>
        <pane title="Canvas">
            <h3>Visual representation</h3>
            <barchart2d data="{{numbers}}"/>
        </pane>
    </tabs>
</div>
</body>
</html>

And that’s all there is to it–well, if you decide to ignore the AngularJS magic, specifically the NumbersCtrl controller and the barchart2d component. It is worth exploring those in slightly more detail, starting with the NumbersCtrl.

angular.module('numbers.app', ['d3.directives', 'numbers.directives'])
  .controller('NumbersCtrl', ['$scope', function($scope) {
    function createWebSocket(path) {
      var host = window.location.hostname;
      if (host == '') host = 'localhost';
      var uri = 'ws://' + host + ':9160' + path;

      var Socket = "MozWebSocket" in window ? MozWebSocket : WebSocket;
      return new Socket(uri);
    }

    $scope.numbers = {};

    var socket = createWebSocket('/'); 
    socket.onopen = function() {
       // we'll have that in the next session
       socket.send("even 0-100 every 1s"); 
    };
    socket.onmessage = function(e) {
       $scope.$apply(function() {
         $scope.numbers = e.data;
       });
    };
  }]);

This is the code of our AngularJS application. It depends on d3.directives and numbers.directives modules; these modules contain directives (think components) for the D3 charts and our tab control. The tab directives are the raw AngularJS example, so let’s explore the D3 components. We’ve split it into two modules: one that provides the d3 service (by pulling in the D3 JavaScript), and then the module that exposes the components.

// creates the d3.core module, which contains the d3Service
angular.module('d3.core', [])
  // creates d3Service by injecting the D3JS JavaScript to the document
  .factory('d3Service', ['$document', '$q', '$rootScope',
    function($document, $q, $rootScope) {
      ...
      return {
        d3: ... // the d3 namespace
      };
  }]);

Grab the code from https://github.com/janm399/hwsexp for the full gory details. The d3.directives module provides the barchart2d for us to use:

// creates the d3.core module, which contains the various D3 charts
angular.module('d3.directives', ['d3.core'])
  .directive('barchart2d', ['d3Service', function(d3Service) {
    return {
      restrict: 'E',
      transclude: true,
      scope: { data: '@' },
      template: '<div class="barchart2d" ng-transclude=""></div>',
      replace: true,
      link: function(scope, element, attrs) {
              d3Service.d3().then(function(d3) {
                function fmt(element, x) {
                  element.style("width", function(d) { return x(d) + "px"; })
                         .text(function(d) { return d; });
                }

                attrs.$observe('data', function(rawValue) {
                  var data = JSON.parse(rawValue);

                  var x = d3.scale.linear()
                      .domain([0, d3.max(data)])
                      .range([0, 420]);

                  var p = d3.select(element[0]).selectAll("div").data(data);
                  fmt(p.enter().append("div"), x);
                  fmt(p.transition(), x);
                  p.exit().remove();
                });
              });
            }
    };
  }]);

Again, this is the basic D3 chart, the slight tickery involves using the attrs.$observe to connect to the changes of the given model.

Summary

And there you have it. You can run a Haskell-based WebSocket server and then have a modern web application that displays the output that the Haskell server sends. Now that we have the basic building blocks, we’ll be adding more features–especially to parse the query that the users type in and then execute it.


Published at DZone with permission of Jan Machacek, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)