Mobile Zone is brought to you in partnership with:

Jorge is the author of three software development books: "Building a Sencha Touch Application", "How to Build a jQuery Mobile Application", and the "Ext JS 3.0 Cookbook". He runs a software development and developer education shop that focuses on mobile, web and desktop technologies. Jorge is a DZone MVB and is not an employee of DZone and has posted 52 posts at DZone. You can read more from them at their website. View Full User Profile

Mobile Web Development Tips: Using Compression to Reduce Database Size

05.19.2014
| 2825 views |
  • submit to reddit

Mobile Web Development Tips: Using Compression to Reduce Database Size

In this mobile web development tips article, we will review a practice that can greatly improve your mobile applications. One of my current projects is a large web-based tablet app that allows building inspectors to access their list of assigned inspections and capture different types of information as they perform a walk-through of a property under inspection. The app has two sides: a client-side that uses jQuery Mobile and JavaScript modules, and a server-side that uses .NET and SQL Server. The two sides communicate over HTTP on the cellular network in order to keep data in sync.

This app can work in offline mode. We added the feature because building inspectors frequently find themselves in physical locations where radio coverage on their tablet devices is spotty or non-existent at all. To accomplish this, the app’s client-side uses the mobile browser’s localStorage or IndexedDB, depending on which storage API the tablet supports.

The client-side data storage challenge

One of the challenges that we faced with working offline is the performance degradation caused by the storage of large amounts of data through the mobile browser on the tablet device. The root of the challenge is the nature of the application. Building inspectors can perform over a dozen types of inspections on a property, and each type of inspection has a large number of unique data properties. This requires a data architecture with multiple data structures, each with more than 20 attributes (or columns, if you prefer to think about it in terms of tables and columns) that map to each inspection type, which results in each inspection having a data model with more than 240 data attributes (columns) that need to be maintained in the app.

Combine the above with the fact that if the user running the app happens to be an administrator, they have rights to edit inspections for all users of the system, not just for themselves. An administrator can potentially download all these data to their tablet. And the app then needs to cache them using localStorage or IndexedDB.

building-inspection-app

We know that storing large amounts of data on the client-side still isn’t a good practice due to the current limitations of browsers, memory, processor, bandwidth and other resources of today’s mobile platforms. In order to not fall victim to premature optimization, we tested these limitations with the building inspectors app and found that the app would either crash the mobile browser, or the UI would become unresponsive, when handling large amounts of data. As expected, the issues were more frequent when the user of the app had administrator rights.

Approaches to mitigating client-side data storage issues

Server-side paging

One of the approaches that we took to overcome this challenge is to implement data paging on the server side. I wrote about this subject in this Preventing Unbounded Resultsets article. Essentially, the server only sends to the mobile web app a small subset of the data that exists for the user. That subset of data is then cached on the client and is available to the user even when working offline. System administrators can control how many records are sent to the client app via configuration settings.

Data compression

The second approach, which is the primary focus of this article, involves using compression to reduce the size of the data saved on the device by the client app. This is as simple as adding two steps to the data access routines:

  • Compress the data before saving it to the local database
  • De-compress the data after reading it from the local database

An example of client-side data storage with compression

Let’s take a look at a short example so you can see how this approach works. Consider a simplified version of the building inspections app where we have a Class called BuildingInspectionRequest that represents a request to perform a building inspection:

var app = app || {};
app.model = app.model || {};
app.model.BuildingInspectionRequest = function () {
    this.Id = null;
    this.DateCreated = null;
    this.DateLastModified = null;
    this.DateScheduled = null;
    this.WeatherAndTemperature = null;
    this.CustomerName = null;
    this.CustomerEmailAddress = null;
    this.CustomerHomePhone = null;
    this.CustomerWorkPhone = null;
    this.CustomerCellPhone = null;
    this.PropertyAddress = null;
    this.PropertyCity = null;
    this.PropertyState = null;
    this.PropertyZipCode = null;
    this.PropertyIsOccupied = null;
    this.StatusId = null;
    this.InspectionType = null;
    this.BuildingType = null;
    this.CustomerSignature = null;
    this.DateSignedByCustomer = null;
    this.OperationalNotes = null;
};

The app running on the user’s mobile browser will download these requests from the server and then save them to localStorage. When working in offline mode, the app will retrieve the local requests, allowing the user to perform edits and save them back to local cache.

Let’s create a simple data access layer that satisfies this requirement. We will encapsulate this layer in the following Controller Class:

var app = app || {};
app.Controller = function () {

    var save = function (key, value) {

        var serialized = JSON.stringify(value);
        window.localStorage.setItem(key, serialized);
        return serialized.length;
    };

    var load = function (key) {

        var value = window.localStorage.getItem(key);
        if (value) {
            return JSON.parse(value);
        } else {
            return null;
        }
    }
      
    return {
        load: load,
        save: save
    };

};

Controller is a Class with two public methods, load and save, that give us access the localStorage API. What we are going to do next is add two more public methods so we can take advantage of compression to reduce the size of the data saved in localStorage. In this example we will use the lzstring library to take care of the compression, but you can use whichever library works for you.

The modified Controller will look like this:

var app = app || {};
app.Controller = function () {

    var save = function (key, value) {

        var serialized = JSON.stringify(value);
        window.localStorage.setItem(key, serialized);
        return serialized.length;
    };

    var load = function (key) {

        var value = window.localStorage.getItem(key);
        if (value) {
            return JSON.parse(value);
        } else {
            return null;
        }
    }

    var saveCompressed = function (key, value) {

        var compressed = LZString.compress(JSON.stringify(value));
        window.localStorage.setItem(key, compressed);
        return compressed.length;
    };

    var loadCompressed = function (key) {

        var value = window.localStorage.getItem(key);
        if (value) {
            return JSON.parse(LZString.decompress(value));
        } else {
            return null;
        }
    }

    return {
        load: load,
        save: save,        
        loadCompressed: loadCompressed,
        saveCompressed: saveCompressed
    };

};

The saveCompressed method is a modified version of the save method where, prior to saving to localStorage, we call LZString’s compress method. In the loadCompressed method, we invoke LZString’s decompress method on the value retrieved from localStorage and return the decompressed data to the caller.

Let’s test these methods so we can see what difference compression makes:

$(document).ready(function () {

    var inspectionRequests = [],
        controller = new app.Controller(),
        loadedFromPlain,
        loadedFromCompressed;

    for (var i = 0; i < 100; i++) {

        var request = new app.model.BuildingInspectionRequest();

        inspectionRequests.push(request);
    }

    var sizeOfPlain = controller.save("plain", inspectionRequests);
    var sizeOfCompressed = controller.saveCompressed("comppressed", inspectionRequests);

    loadedFromPlain = controller.load("plain");
    loadedFromCompressed = controller.loadCompressed("comppressed");
});

In the code above we are writing 100 BuildingInspectionRequest instances in localStorage, with and without compression. When we run the code, the console will show something similar to this:

client-side-compression-1

Notice the significant difference between the compressed and uncompressed data. And notice that the size reduction exists even when the BuildingInspectionRequest instances we created are all “empty”. This difference can be larger when the BuildingInspectionRequest instances are initialized with real-world data.

Client-Side data compression affects application performance

One adverse effect of using compression is that execution time suffers. Let’s see how much longer it takes to execute the data access in this example. We will make a few changes to the app so we can time the Controller’s functions:

$(document).ready(function () {

    var inspectionRequests = [],
        controller = new app.Controller(),
        loadedFromPlain,
        loadedFromCompressed;

    for (var i = 0; i < 100; i++) {

        var request = new app.model.BuildingInspectionRequest();

        inspectionRequests.push(request);
    }

    console.time("save duration");
    var sizeOfPlain = controller.save("plain", inspectionRequests);
    console.timeEnd("save duration");
    console.log("sizeOfPlain: " + sizeOfPlain);

    console.time("saveCompressed duration");
    var sizeOfCompressed = controller.saveCompressed("comppressed", inspectionRequests);
    console.timeEnd("saveCompressed duration");
    console.log("sizeOfCompressed: " + sizeOfCompressed);

    console.time("load duration");
    loadedFromPlain = controller.load("plain");
    console.timeEnd("load duration");
    console.log("loadedFromPlain: " + loadedFromPlain);

    console.time("loadCompressed duration");
    loadedFromCompressed = controller.loadCompressed("comppressed");
    console.timeEnd("loadCompressed duration");
    console.log("loadedFromCompressed: " + loadedFromCompressed);
});

Running the app should produce a console output similar to the one below:

client-side-compression-2

Both the save and the load operations take longer to execute when the compression and decompression are active. This might or might not be an issue in your applications. In a line-of-business app like our building inspections app, sacrificing load and save speed so we can reduce storage size on the client is generally a good compromise when you take measures to maintain a good user experience.

Wrapping up

This article illustrates an approach you can take to make mobile web apps more usable by using compression to reduce the amount of data saved on the client side. While we used a simple example with measurements that aren’t extremely rigorous, it is clear that the approach can potentially make the difference between a usable and an unusable app. However, be mindful that carefully measuring its favorable and unfavorable impacts on the app is what ultimately will help you decide whether or not to put the compression approach in place.

Source Code

Download from GitHub: Reduce a web app’s database size using compression


Published at DZone with permission of Jorge Ramon, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)