NoSQL Zone is brought to you in partnership with:

Developer Advocate at MongoLab. Previously interned at VMware and NetApp. UCSD CSE 2013 Alum. Chris is a DZone MVB and is not an employee of DZone and has posted 21 posts at DZone. You can read more from them at their website. View Full User Profile

Tutorial: Scaling Meteor with MongoDB Oplog Tailing

07.24.2014
| 3049 views |
  • submit to reddit

Ever since Meteor 0.7.0 first introduced oplog tailing, we’ve had a lot of users asking us about using the MongoDB oplog with their Meteor applications. As a result, we thought a step-by-step tutorial would help folks get started.

Meteor Oplog Tailing Overview

If you’re still feeling your way around the Meteor framework, you may not know about oplog tailing just yet. The Meteor team released an improvement to observeChanges, which monitors MongoDB’s oplog to avoid extra operations on the database. This improvement significantly reduces the number of queries needed to obtain the freshest changes to your MongoDB.

The “local” database holds the MongoDB oplog

MongoDB’s operations log keeps a rolling record of all inserts, updates and deletes. This log is stored in a special database called “local” which exists on each member of a replica set and does not replicate.

From MongoDB’s documentation on the oplog: “MongoDB applies database operations on the primary and then records the operations on the primary’s oplog. The secondary members then copy and apply these operations in an asynchronous process. All replica set members contain a copy of the oplog.”

The Meteor framework smartly uses the oplog to keep track of changes to your data. This minimizes the number of queries necessary when searching for changes to your data.

Meteor Oplog Tailing Tutorial

We’ve hacked up our own example using Meteor’s oplog tailing with a database hosted on MongoLab (don’t worry, you can still use this tutorial even if you’re not using MongoLab). Here’s a link to this example project’s repo on GitHub.

To give you an idea of what we’re trying to accomplish, we’ll be setting up a simple Meteor app that displays a list of players and their scores. The app will also have an input form for inserting new players and scores to show real-time updates in the client view.

When our project is complete, we’ll observe real-time metrics from two apps, one with oplog tailing configured and one without, which are otherwise identical.  The output should look like the following:

Meteor serverFacts output. Oplog tailing enabled output on left, no tailing enabled on right.

Pre-requisites

In this tutorial we’ll assume that you have:

  • a MongoDB database (with access to the “local” database)
    • your own MongoDB deployment OR
    • any for-pay subscription with MongoLab (starts at $15/mo)
  • Node.js installed
  • Meteor installed

Set up the project

First, create a Meteor project.

> meteor create app

Then navigate into your app to start modifying the project files.

> cd app

Now we’ll remove the “autopublish” package which is not recommended for production use.

> meteor remove autopublish

We’ll also want to run the “facts” package, which contains real-time information about our Meteor server. This will help us determine if oplog-tailing is working.

> meteor add facts

Configure the view

First we’ll define a view to display relevant content to the client. We’ll edit our app.html file to look like the following:

<head>
  <title>app</title>
</head>

<body>
  <div>
    <!-- Players template to display all players in database -->
    {{> players}}
  </div>
  <div>
    <!-- User input form to add players -->
    {{> form}}
  </div>
  <div>
    <!-- Real-time metrics to confirm oplog tailing -->
    {{> serverFacts}}
  </div>
</body>

<template name="players">
  <h1>List of players</h1>
    <ul>
      {{#each scorers}}
        <li>
          <div>
            Name: {{ this.name }}
          <div>
          <div>
            Score: {{ this.score }}
          </div>
        </li>
      {{/each}}
    </ul>
</template>

<template name ="form">
  <form id="myform">                       
    Name: <input type="text" name="name"/>
    Score: <input type="number" name="score"/>
    <input type="submit" id="submit"/>
  </form>
</template>

You’ll notice that the file contains conventional HTML code and Meteor’s templating language, Spacebars (inspired by Handlebars). In the body, we’ll lay out what we want the user to see: a list of players with their scores, a form to add new players, and “serverFacts” metrics.

The “serverFacts” template is a report auto-generated by Meteor that contains real-time information for the current server. This is how we will later verify that our application has oplog-tailing enabled.

Now that we have the templates in place, we need to configure them. We’ll configure the “players” template to iterate over all the players and list their names and scores. The “form” template is straightforward as well – configure it as you would a HTML form.

Create the model

Now we’ll create a new file, models.js, in our project directory to specify a collection to store our documents.

Players = new Meteor.Collection("players");

Our Players model maps to a “players” collection in your MongoDB. You don’t need to create this collection ahead of time as MongoDB will lazily create it for you (it doesn’t already exist) once you insert a document.

Published at DZone with permission of Chris Chang, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)