Tutorial: Scaling Meteor with MongoDB oplog tailing

Updated October 18, 2016: Made version 2.6+ compatibility section default

Updated September 12, 2014: Added section on compatibility with 2.6

Ever since Meteor 0.7.0 first introduced oplog tailing, we've had a lot of users asking us about using the MongoDB oplog with their Meteor applications. As a result, we thought a step-by-step tutorial would help folks get started.

Meteor Oplog Tailing Overview

If you're still feeling your way around the Meteor framework, you may not know about oplog tailing just yet. The Meteor team released an improvement to observeChanges, which monitors MongoDB's oplog to avoid extra operations on the database. This improvement significantly reduces the number of queries needed to obtain the freshest changes to your MongoDB.

The "local" database holds the MongoDB oplog

MongoDB's operations log keeps a rolling record of all inserts, updates and deletes. This log is stored in a special database called "local" which exists on each member of a replica set and does not replicate.

From MongoDB's documentation on the oplog: "MongoDB applies database operations on the primary and then records the operations on the primary's oplog. The secondary members then copy and apply these operations in an asynchronous process. All replica set members contain a copy of the oplog."

The Meteor framework smartly uses the oplog to keep track of changes to your data. This minimizes the number of queries necessary when searching for changes to your data.

Meteor Oplog Tailing Tutorial

We've hacked up our own example using Meteor's oplog tailing with a database hosted on MongoLab (don't worry, you can still use this tutorial even if you're not using MongoLab). Here's a link to this example project's repo on GitHub.

To give you an idea of what we're trying to accomplish, we'll be setting up a simple Meteor app that displays a list of players and their scores. The app will also have an input form for inserting new players and scores to show real-time updates in the client view.

When our project is complete, we'll observe real-time metrics from two apps, one with oplog tailing configured and one without, which are otherwise identical.  The output should look like the following:

Meteor serverFacts output. Oplog tailing enabled output on left, no tailing enabled on right.

Pre-requisites

In this tutorial we'll assume that you have:

  • a MongoDB database (with access to the "local" database)
    • your own MongoDB deployment OR
    • any for-pay subscription with MongoLab (starts at $15/mo)
  • Node.js installed
  • Meteor installed

Set up the project

First, create a Meteor project.

> meteor create app

Then navigate into your app to start modifying the project files.

> cd app

Now we'll remove the "autopublish" package which is not recommended for production use.

> meteor remove autopublish

We'll also want to run the "facts" package, which contains real-time information about our Meteor server. This will help us determine if oplog-tailing is working.

> meteor add facts

Configure the view

First we'll define a view to display relevant content to the client. We'll edit our app.html file to look like the following:

You'll notice that the file contains conventional HTML code and Meteor's templating language, Spacebars (inspired by Handlebars). In the body, we'll lay out what we want the user to see: a list of players with their scores, a form to add new players, and "serverFacts" metrics.

The "serverFacts" template is a report auto-generated by Meteor that contains real-time information for the current server. This is how we will later verify that our application has oplog-tailing enabled.

Now that we have the templates in place, we need to configure them. We'll configure the "players" template to iterate over all the players and list their names and scores. The "form" template is straightforward as well - configure it as you would a HTML form.

Create the model

Now we'll create a new file, models.js, in our project directory to specify a collection to store our documents.

Our Players model maps to a "players" collection in your MongoDB. You don't need to create this collection ahead of time as MongoDB will lazily create it for you (it doesn't already exist) once you insert a document.

Link together the client, server, and MongoDB

In order for the client (browser) view to display all the players, we need to query for them on our database and pass the cursor to the client. We'll replace the default app.js file to look like the following:

Starting from the top, we have our server side code. This uses Meteor's publish method to link together the client and server; we'll cover the intricacies of the publish method in the next section. We also set up the code needed to publish server metrics to the "serverFacts" template that we created back in our app.html file.

We then have our client side code. Similar to the publish method in the server side code, here we use Meteor's subscribe method, which we'll also cover in the next section.

Next we create a template helper function that queries on the Players model and passes a cursor to the client (browser) view. This allows the "players" template from our view (app.html file) to iterate through all the Players that are returned and display them in the view.

Finally we create a handler for the "form" template that activates when the form is submitted. Typically you want to put your data validation code here as well, but our example inserts the data directly into the database.

Explore Meteor's publish and subscribe methods

In order to fully understand what Meteor's publish and subscribe methods do, it's important to note that unlike other applications (Rails, Django, etc.) Meteor applications live on both the server and the client. This architecture allows Meteor to send raw data to the client (data on the wire) and access that data instantaneously without having to wait for a round-trip to the server. Meteor ensures that the client and server data (that you specify) are in sync.

In the server code we copied above, we use Meteor's publish method to help implement oplog tailing. From the Meteor wiki: "Whenever a cursor is returned from the publish function, Meteor calls observeChanges on the cursor and provides it with callbacks which publish the query's changes to clients." This means that Meteor continues to watch the published query and calls a  callback function when results change.

Previously, Meteor's only strategy for implementing observeChanges was to re-run the query frequently and calculate the difference between each set of results. With the introduction of the OplogObserveDriver class, Meteor can now read changes from the oplog.

Back to our example, we publish a record set with the name "playerData". Once you've published the query to the "playerData" record set, you need to listen, or subscribe, to that record set on the client side. The subscribe method in our client code tells the server to send this particular set of records to the client, which is stored in the client-side database called MiniMongo.

Now when the data in our MongoDB changes for our players query, our view should reflect those changes in real-time and with minimal cost on the database.

Set up the app

We're almost there! Next, you'll need to bundle then extract your project to set up the application.

> meteor bundle app.tgz

> tar -zxvf app.tgz

Create a "local" database user

You'll need to create a user for your local database so that your Meteor app can access the oplog. You will need to give the Meteor app your user credentials when you run the application. We recommend creating a user with read-only access.

You will be unable to create a user directly on the local database. Instead, we recommend creating a user on the "admin" database and using the following URI for oplog access:

> mongodb://adminUser:adminPass@host:port/local?authSource=admin

This URI will use your "admin" database credentials to authenticate to the "admin", then switch over to the "local" database.

If you're using MongoLab, you can visit our docs to find instructions on creating a "local" database user.

Once you're done, be sure to copy the MongoDB URI for your "local" database for the next step.

Run the app

Once you're set with credentials, you can run your application with the following command:

> PORT=3000 MONGO_URL=<your_uri> MONGO_OPLOG_URL=<your_local_uri> node bundle/main.js

The MONGO_URL points to the MongoDB database that your application reads and writes to, whereas the MONGO_OPLOG_URL should point to your "local" database (which contains the oplog).

See the difference in real-time!

Once your application is running, we can verify if oplog tailing is working. Again, we highly recommend reading the Meteor wiki on the Oplog Observe Driver so you understand the underlying details.

To check if oplog tailing is enabled, you'll want to verify that the observe-drivers-oplog metric is rendered and the observe-drivers-polling metric is at 0 or not rendered at all. This difference is subtle, but very important!

For extra fun, clone your app and run multiple copies at once to see real-time updates between concurrent clients. I recommend running one copy of your app with oplog tailing and one without. To run the app without oplog tailing, simply leave out the MONGO_OPLOG_URL option in the command.

> PORT=3000 MONGO_URL=mongolab_uri node bundle/main.js

Get the most out of your MongoDB on Meteor

In addition to this tutorial, we highly recommend watching David Glasser's Devshop 10 talk on oplog tailing. He clearly articulates the background story, problem and solution and provides excellent visual examples.

We hope this tutorial helps you leverage all the tools at your disposal so that you can get the most out of your MongoDB on Meteor. We're excited to see what you hack up!

29 Responses to Tutorial: Scaling Meteor with MongoDB oplog tailing

  1. Ives 2014/07/24 at 1:58 am #

    There’s an error in app.html. The closing div:

    Name: {{ this.name }}

  2. Chris Chang 2014/07/24 at 7:34 am #

    Good catch, thanks!

  3. Ruben Sainiuc 2014/08/28 at 3:08 am #

    Great tutorial! Works perfectly! Thanks!

  4. Franco Coronel 2014/08/29 at 1:26 pm #

    Muchas gracias!

  5. Protik Khan 2014/08/30 at 3:11 am #

    Wow ! How nice the site is ! It is looking awesome, Its category systems are also good relevant as writing service, Are you searching best writing services ? But Can’t take decision which site is prefect for you ? Nothing to worry about- I have seen a site that you are looking for ! To visit just click here paying someone to write a paper

  6. jungwon jin 2014/10/09 at 11:13 pm #

    Quite a nice article-
    I translated this into Korean and posted my google+.

    (https://plus.google.com/109529760167802368324/posts/S9cyLiZLA4Z)
    Thanks-

  7. Valory 2014/11/26 at 5:33 am #

    Great information!
    http://courseworkwritingservice.org/

  8. jschnip 2015/02/01 at 2:20 am #

    While troubleshooting the error:

    not authorized for query on local.system.replset

    It appears that there is a privilege missing from mongolab’s defined readOplog role in order to find the system.replset

    Since shared accounts can not edit roles on admin users, any idea if it be possible to have this privilege added to Mongolabs readOplog role

    { resource: { db: ‘local’, collection: ‘system.replset’}, actions: [‘find’]}

    I think this should clear up the issue for us (and anyone else using the meteor Oplog-Observe-Driver) – advice is welcome if I am missing something,

  9. Juana Dalton 2015/03/17 at 8:03 am #

    https://en.wikipedia.org/wiki/Essay

  10. KenzoMan 2015/06/04 at 7:32 am #

    Amazing work! Love your tutorial, many thanks from essaymaxi

  11. James Woodard 2015/06/19 at 2:12 am #

    Like this post! Thanks from college hero

  12. Giuche.Hostingsiteforfree.Com 2017/03/11 at 1:57 am #

    I used to be able to find good info from your content.

  13. Booldgogefish41 2017/03/11 at 10:40 am #

    You could test personal guitar lessons, however are really pricey and can involve a big ti.Hunt is often a stone-cold fox and I ACTUALLY thank God he sets my religion over all the others? s beliefs.There’re asked to write the answers towards following questions.Food photographers are more likely to use natural light in lieu of artificial studio lights and less props to stay the shot looking clean, simple and giving more focus on the food.The reality is, these stations could carry out these new songs today one of the many other classics on currently on the playlist.
    For those who own a particular types of salamander, you needs to be accurate when giving them the right nutrition.But in case you follow the four simple steps I gave you, you can do the work and grow the guitarist you would like to be.And so long as we don? t really have to fight about that, truth be told there? s no problem.Privided you can help someone with a difficulty they are having you definitely will do alot more effective at getting subscribers for ones publications.It is a happy time, but if the baring continues for 30 minutes it can be a lot.
    A lot of women? s habits and tendencies seem foreign in your direction best oakley sunglasses for baseball
    just like exotic indoor plants.Your individual condition is the ultimate thing which will decide how much flowing hair replacement will cost.This incredible website could stay like its forever, unless there was an issue that changed with the unit.It is a much better proposition as opposed to aged idea of ' just keeping them over streets'? Ten Point Summary Read the entire eBook for free today at Concerning Author Steve Baba features a Ph.
    When doing all of your color poster printing as an illustration, be sure to make new friends and infuse your readers with information which could make you a credible source of solutions to their wants.Efficient they may lose involvement in the baby, and may have thoughts of hurting them too.This passion have invariably been translated to your consumer — you only have to trust me on this.He’s got an ideal one-liner about retirement: "All I do will be play golf and play music — which one are you wanting me to give " up ".Tell her reality of life in these kinds of areas as drugs, sexual and alcohol.
    Related Articles

  14. sex dating 2017/03/12 at 2:15 pm #

    Amazing! Its actually awesome paragraph, I have got
    much clear idea about from this piece of writing.

  15. Margaret 2017/03/18 at 8:48 am #

    You really make it seem so easy with your presentation but I find this topic to be actually something
    which I think I would never understand. It seems too complex and very broad for me.
    I’m looking forward for your next post, I’ll try to
    get the hang of it!

  16. must cartier faux 2017/03/18 at 3:16 pm #

    cartierlovejesduas Where can I get some oil, mother really ill with colon cancer. I live in Cape Town
    must cartier faux http://www.vendre-montres.ru/fr/cartier-must-21-chronograph-replica-watch-black-for-men-p576/

  17. florida gators apparel 2017/03/18 at 10:23 pm #

    Hi there! Would you mind if I share your blog with my facebook group?
    There’s a lot of people that I think would really enjoy
    your content. Please let me know. Thanks

  18. jacksonville jaguars 2017/03/18 at 10:52 pm #

    Everything is very open with a precise explanation of the issues.
    It was really informative. Your website is very useful.
    Many thanks for sharing!

  19. Shad 2017/03/19 at 12:57 am #

    Hello, this weekend is fastidious in favor of
    me, because this point in time i am reading this wonderful educational article here at my residence.

  20. Dura Matters Massages 2017/03/20 at 11:06 pm #

    Your way of describing everything in this article is really nice, all
    be able to simply understand it, Thanks a lot.

  21. georgia Bulldog Apparel 2017/03/20 at 11:28 pm #

    You actually make it seem so easy with your presentation but I find this topic to be actually something
    which I think I would never understand. It seems too complicated and very broad for
    me. I am looking forward for your next post, I will try to get
    the hang of it!

  22. faux cartier ballon bleu quartz 2017/03/26 at 9:12 pm #

    cartierbraceletlove It’s my observation that people who parachute into higher levels, with very few exceptions, disadvantage themselves as they don’t have the benefit of the acclimatisation to the paradigm that the lower levels, particularly Level 1, provide; rather like Nigel’s marathon training analogy.
    faux cartier ballon bleu quartz http://www.perfectcawatch.ru/fr/large-ballon-bleu-c88_101/

  23. cartierbraceletlove It’s not about latex expression but about the script that renders them. Just unblock it from the blocked scripts in the site preferences.
    alhambra bracelet van cleef réplique http://www.holidayvcagift.cn/fr/

  24. copie montre tank solo cartier 2017/03/26 at 9:12 pm #

    cartierbraceletlove Joy, I agree. Andrea Bocelli can’t “read” music either.
    copie montre tank solo cartier http://www.perfectcawatch.ru/fr/replica-cartier-tank-americaine-diamond-small-18k-white-gold-watch-for-women-wb7073l1-p448/

  25. seo 2017/03/27 at 12:45 pm #

    Hello Web Admin, I noticed that your On-Page SEO is is missing a few factors, for one you do not use all three H tags in your post, also I notice that you are not using bold or italics properly in your SEO optimization. On-Page SEO means more now than ever since the new Google update: Panda. No longer are backlinks and simply pinging or sending out a RSS feed the key to getting Google PageRank or Alexa Rankings, You now NEED On-Page SEO. So what is good On-Page SEO?First your keyword must appear in the title.Then it must appear in the URL.You have to optimize your keyword and make sure that it has a nice keyword density of 3-5% in your article with relevant LSI (Latent Semantic Indexing). Then you should spread all H1,H2,H3 tags in your article.Your Keyword should appear in your first paragraph and in the last sentence of the page. You should have relevant usage of Bold and italics of your keyword.There should be one internal link to a page on your blog and you should have one image with an alt tag that has your keyword….wait there’s even more Now what if i told you there was a simple WordPress plugin that does all the On-Page SEO, and automatically for you? That’s right AUTOMATICALLY, just watch this 4minute video for more information at. Seo Plugin

Trackbacks/Pingbacks

  1. Sysadmin Sunday 189 - Server Density Blog - 2014/08/10

    […] Scaling Meteor with MongoDB oplog tailing […]

  2. Respondly explains why devs love Meteor and MongoDB - 2014/11/04

    […] value. In particular, Meteor includes real-time monitoring of MongoDB for changes to data (via oplog tailing). This monitoring allows Meteor […]

  3. Brute Force D3.js Visualization | BigSnarf blog - 2015/02/16

    […] http://blog.mongolab.com/2014/07/tutorial-scaling-meteor-with-mongodb-oplog-tailing/ […]

  4. 10 reasons why should use Meteorjs for next Web development | - 2016/01/13

    […] Meteor works with your existing back-end via REST APIs if needed, but you’re better off using DDP, a simple and powerful publish/subscribe protocol. The server side runs on top of Node.js but eliminates callback hell by using Fibers. The underlying database is MongoDB and Meteor includes real-time and highly efficient monitoring for changes to data via oplog tailing. […]

Leave a Reply