Replication Lag & The Facts of Life

So you're checking in on your latest awesome application one day -- it's really getting traction! You're proud of its uptime record, thanks in part to the MongoDB replica set underneath it. But now ... something's wrong. Users are complaining that some of their data has gone missing. Others are noticing stuff they deleted has suddenly reappeared. What's going on?!?

Don't worry... we'll get to the bottom of this! In doing so, we'll examine a source of risk that's easy to overlook in a MongoDB application: replication lag -- what it means, why it happens, and what you can do about it.

Here's what we're going to cover:

Continuing this cautionary tale... Seriously, wtf?! You were doing everything right!

Using MongoDB with a well-designed schema and lovingly-tuned indexes, your application back-end has been handling thousands of transactions per second without breaking a sweat. You've got multiple nodes arranged in a replica set with no single point of failure. Your application tier's Mongo driver connections are aware of the replica set and can follow changes in the PRIMARY node during failover. All critical writes are "safe" writes. Your app has been up without interruption for almost six months now! How could this have happened?

This unsettling situation has the hallmarks of an insidious foe in realm of high-availability data stewardship: unchecked replication lag.

Closely monitoring a MongoDB replica set for replication lag is critical.

What is replication lag?

As you probably know, like many data stores MongoDB relies on replication -- making redundant copies of data -- to meet design goals around availability.

The Facts of LifeIn a perfect world, data replication would be instantaneous; but in reality, thanks to pesky laws of physics, some delay is inevitable -- it's a fact of life. We need to be able to reason about how it affects us so as to manage around the phenomenon appropriately. Let's start with definitions...

For a given secondary node, replication lag is the delay between the time an operation occurs on the primary and the time that same operation gets applied on the secondary.

For the replica set as a whole, replication lag is (for most purposes) the smallest replication lag found among all its secondary nodes.

In a smoothly running replica set, all secondaries closely follow changes on the primary, fetching each group of operations from its oplog and replaying them approximately as fast as they occur. That is, replication lag remains as close to zero as possible. Reads from any node are then reasonably consistent; and, should the current primary become unavailable, the secondary that assumes the PRIMARY role will be able to serve to clients a dataset that is almost identical to the original.

For a variety of reasons, however, secondaries may fall behind. Sometimes elevated replication lag is transient and will remedy itself without intervention. Other times, replication lag remains high or continues to rise, indicating a systemic problem that needs to be addressed. In either case, the larger the replication lag grows and the longer it remains that way, the more exposure your database has to the associated risks.

Why is lag problematic?

Significant replication lag creates failure modes that can be problematic for a MongoDB database deployment that is meant to be highly available.  Here's why:

  • If your replica set fails over to a secondary that is significantly behind the primary, a lot of un-replicated data may be on the original primary that will need to be manually reconciled. This will be painful or impossible if the original primary is unrecoverable.
  • If the failed primary cannot be recovered quickly, you may be forced to run on a node whose data is not up-to-date, or forced to take down your database altogether until the primary can be recovered.
  • If you have only one secondary, and it falls farther behind than the earliest history retained in the primary's oplog, your secondary will require a full resynchronization from the primary.
    • During the resync, your cluster will lack the redundancy of a valid secondary; the cluster will not return to high availability until the entire data set is copied.
    • If you only take backups from your secondary (which we highly recommend), backups must be suspended for the duration of the resync.
  • Replication lag makes it more likely that results of any read operations distributed across secondaries will be inconsistent.
  • A "safe" write with 'w' > 1 -- i.e., requiring multiple nodes acknowledge the write before it returns -- will incur latency proportional to the current replication lag, and/or may time out.

Strictly speaking, the problem of replication lag is distinct from the problem of data durability. But as the last point above regarding multi-node write concern illustrates, the two concepts are most certainly linked. Data that has not yet been replicated is not completely protected from single-node failure; and client writes specified to be safe from single-node failure must block until replication catches up to them.

What causes a secondary to fall behind?

In general, a secondary falls behind on replication any time it cannot keep up with the rate at which the primary is writing data. Some common causes:

Secondary is weak

To have the best chance of keeping up, a secondary host should match the primary host's specs for CPU, disk IOPS, and network I/O. If it's outmatched by the primary on any of these specs, a secondary may fall behind during periods of sustained write activity. Depending on load this will, at best, create brief excursions in replication lag and, at worst, cause the secondary to fall irretrievably behind.

Bursty writes

In the wake of a burst of write activity on the primary, a secondary may not be able to fetch and apply the ops quickly enough. If the secondary is underpowered, this effect can be quite dramatic. But even when the nodes have evenly matched specs, such a situation is possible. For example, a command like:

 db.coll.update({x: 7}, {$set: {y: 42}}, {multi: true}}

can place an untold number of separate "update" ops in the primary's oplog. To keep up, a secondary must fetch those ops (max 4MB at a time for each getMore command!), read into RAM any index and data pages necessary to satisfy each _id lookup (remember: each oplog entry references a single target document by _id; the original query about "x" is never directly reflected the oplog), and finally perform the update op, altering the document and placing the corresponding entry into its oplog; and it must do all this in the same amount of time that the primary does merely the last step. Multiplied by a large enough number of ops, that disparity can amount to a noticeable lag.

Map/reduce output

A specific type of the extreme write burst scenario might be a command like:

 db.coll.mapReduce( ... { out: other_coll ... })

From the point of view of the oplog, the entire output collection basically materializes at once, from which point the replication to the secondary plays out as above.

Index build

It may surprise you to learn that, even if you build an index in the background on the primary, it will be built in the foreground on each secondary. There is currently no way to build indexes in the background on secondary nodes (cf. SERVER-2771). Therefore, whenever a secondary builds an index, it will block all other operations, including replication, for the duration. If the index builds quickly, this may not be a problem; but long-running index builds can swiftly manifest as significant replication lag.

Secondary is locked for backup

One of the suggested methods for backing up data in a replica set involves explicitly locking a secondary against changes while the backup is taken. Assuming the primary is still conducting business as usual, of course replication lag will climb until the backup is complete and the lock is released.

Secondary is offline

Similarly, if the secondary is not running or cannot reach the primary for whatever reason, it cannot make progress against the replication backlog. When it rejoins the replica set, the replication lag will naturally reflect the time spent away.

How do I measure lag?

Run the db.printSlaveReplicationInfo() command

To determine the current replication lag of your replica set, you can use the mongo shell and run the db.printSlaveReplicationInfo() command.

rs-ds046297:PRIMARY db.printSlaveReplicationInfo()

source: ds046297-a1.mongolab.com:46297
syncedTo: Tue Mar 05 2013 07:48:19 GMT-0800 (PST)
      = 7475 secs ago (2.08hrs)
source: ds046297-a2.mongolab.com:46297
syncedTo: Tue Mar 05 2013 07:48:19 GMT-0800 (PST)
      = 7475 secs ago (2.08hrs)

More than 2 hours -- whoa, isn't that a lot? Maybe!

See, those "syncedTo" times don't have much to do with the clock on the wall; they're just the timestamp on the last operation that the replica has copied over from the PRIMARY. If the last write operation on the PRIMARY happened 5 minutes ago, then yes: 2 hours is a lot. On the other hand, if the last op was 2.08 hours ago, then this is golden!

To fill in that missing piece of the story, we can use the db.printReplicationInfo() command.

rs-ds046297:PRIMARY db.printReplicationInfo()

configured oplog size:   1024MB
log length start to end: 5589secs (1.55hrs)
oplog first event time:  Tue Mar 05 2013 06:15:19 GMT-0800 (PST)
oplog last event time:   Tue Mar 05 2013 07:48:19 GMT-0800 (PST)
now:                     Tue Mar 05 2013 09:53:07 GMT-0800 (PST)

Let's see ... PRIMARY's "oplog last event time" - SECONDARY's "syncedTo" = 0.0. Yay.

As fun as that subtraction may be, it's seldom called for. If there is a steady flow of write operations, the last op on the PRIMARY will usually have been quite recent. Thus, a figure like "2.08 hours" should probably raise eyebrows; you would expect to see a nice low number there instead -- perhaps as high as a few seconds. And, having seen a low number, there would be no need to qualify its context with the second command.

Examine the "repl lag" graph in Telemetry

You can also view recent and historical replication lag using MongoLab Telemetry. On the Status tab of each SECONDARY node, you'll find the repl lag graph:

How do I monitor for lag?

It is critical that the replication lag of your replica set(s) be monitored continuously.   Since you have to sleep occasionally, this is a job best done by robots.  It is essential that these robots be reliable, and that they notify you promptly whenever a replica set is lagging too far behind.

Here are a couple ways you can make sure this is taken care of:

  • If MongoLab is hosting your replica set, relax! For any multi-node, highly-available replica set we host for you, you can monitor replication lag in our UI and by default you will receive automated alerts whenever the replication lag exceeds 10 minutes.

What can I do to minimize lag?

Out of courtesy (for them or for ourselves), we would like to make those lag-monitoring automata's lives as boring as possible. Here are some tips:

Tip #1: Make sure your secondary has enough horsepower

It's not uncommon for people to run under-powered secondaries to save money -- this can be fine if the write load is light. But in scenarios where the write load is heavy, the secondary might not be able to keep up with the primary. To avoid this, you should beef up your secondary so that it's as powerful as your primary.

Specifically, a SECONDARY node should have enough network bandwidth that it can retrieve ops from the PRIMARY's oplog at roughly the rate they're created and also enough storage throughput that it can apply the ops -- i.e., read any affected documents and their index entries into RAM, and commit the altered documents back to disk -- at that same rate. CPU rarely becomes a bottleneck, but it may need to be considered if there are many index keys to compute and insert for the documents that are being added or changed.

Tip #2: Consider adjusting your write concern

Your secondary may be lagging simply because your primary's oplog is filling up faster than it can be replicated. Even with an equally-brawny SECONDARY node, the PRIMARY will always be capable of depositing 4MB in its memory-mapped oplog in a fraction of the time those same 4MB will need to make it across a TCP/IP connection.

One viable way to apply some back-pressure to the primary might be to adjust your write concern.

If you are currently using a write concern that does not acknowledge writes (aka "fire-and-forget" mode), you can change your write concern to require an acknowledgement from the primary (w:1) and/or a write to the primary's journal (j:true). Doing so will slow down the rate at which the concerned connection can generate new ops needing replication.

Other times it may be appropriate to use a 'w' > 1 or a 'w' set to "majority" to ensure that each write to the cluster is replicated to more than one node before the command returns. Requiring confirmation that a write has replicated to secondaries will effectively guarantee that those secondaries have caught up (at least up to the timestamp of this write) before the next command on the same connection can produce more ops in the backlog.

As previously alluded to, choosing the most appropriate write concern for the data durability requirements of your application -- or for particular critical write operations within the application -- is something you must give thought to irrespective of the replication lag issue we're focusing on here. But you should be aware of the interrelationship: just as the durability guarantee of w>1 can be used as a means of forcing a periodic "checkpoint" on replication, excessive replication lag can show up as a surprisingly high latency (or timeout) for that very occasional critical write operation where you've used "w: majority" to make sure it's truly committed.

Adjust to taste

Having servers acknowledge every write can be a big hit to system throughput. If it makes sense for your application, you can amortize that penalty by doing inserts in batches, requiring acknowledgement only at the end of each batch. The smaller the batch, the greater the back-pressure on PRIMARY data creation rate, and correspondingly greater potential adverse impact to overall throughput.

Don't overdo it

Using a large value for 'w' can itself be problematic. It represents a demand that w nodes finish working through their existing backlog before the command returns. So, if replication lag is high (in the sense of there being a large volume of data waiting to copy over) when the write command is issued, the command execution time will suffer a proportionally high latency. Also, if enough nodes go offline such that 'w' cannot be satisfied, you have effectively locked up your database. This is basically the opposite of "high availability."

Tip #3: Plan for index builds

As mentioned earlier, an index build on a secondary is a foreground, blocking operation. If you're going to create an index that is sizeable, perhaps you can arrange to do it during a period of low write activity on the primary. Alternately, if you have more than one secondary, you can follow the steps here to minimize the impact of building large indexes.

Tip #4: Take backups without blocking

Earlier we discussed the technique of locking the secondary to do a backup. There are other alternatives to consider here, including filesystem snapshots and "point-in-time" backups using the "--oplog" option of mongodump without locking. These are preferable to locking the secondary during a period of active writes if there's any chance you'll use the secondary for anything other than backups.

Tip #5: Be sure capped collections have an _id field & a unique index

Reliable replication is not possible unless there is a unique index on the _id field. Before MongoDB version 2.2, capped collections did not have an _id field or index by default. If you have a collection like this, you should create an index on the _id field, specifying unique: true. Failing to do this can, in certain situations, cause replication to halt entirely. So ... this should not be regarded as optional.

Tip #6: Check for replication errors

If you see that replication lag is only increasing (and never falling), your replica set could be experiencing replication errors. To check for errors, run rs.status() and look at the errmsg field in the result. Additionally, check the log file of your secondary and look for error messages there.

One specific example: if you see "RS102 too stale to catch up" in the secondary's mongodb.log or in the errmsg field when running rs.status(), it means that secondary has fallen so far behind that there is not enough history retained by the primary (its "oplog size") to bring it up to date. In this case, your secondary will require a full resynchronization from the primary.

In general, though, what you do in response to an error depends on the error. Sometimes you can simply restart the mongod process for your secondary; but the majority of the time you will need to understand the root cause of the error before you can fix the problem.

Don't let replication lag take you by surprise.

At the end of the day, replication lag is just one more source of risk in any high-availability system that we need to understand and design around. Striking the right balance between performance and "safety" of write operations is an exercise in risk management -- the "right" balance will be different in different situations. For an application on a tight budget with occasional spikes in write volume, for example, you might decide that a large replication lag in the wake of those spikes is acceptable given the goals of the application, and so an underpowered secondary makes sense. At the opposite extreme, for an application where every write is precious and sacred, the required "majority" write concern will mean you have essentially no tolerance for replication lag above the very minimum possible.  The good news is that MongoDB makes this all very configurable, even on an operation by operation basis.

We hope this article has given you some insight into the phenomenon of replication lag that will enable you to reason about the risk it poses for a high-availability MongoDB application, and armed you with some tools for managing it. As always, let us know if we can help!

137 Responses to Replication Lag & The Facts of Life

  1. jean 2013/04/11 at 3:09 pm #

    Hello! this is a very useful post. I have a replica set of 3 members.

    currently I have problems with 1 member, It has a laggin and felt out behind the master.

    The other secondary is ok and always uptime. The cluster doesn’t have traffic, only I did a mongorestore of 26 GB and this replica can’t replicate the data quickly

    My master is ssd and secondaries is sata.

    The oplog is by default. And I saw a lot of read operation when I did mongotop over the slow replica

  2. Tim Hawkins 2013/06/20 at 4:25 pm #

    One thing that folks often fail to take into account is replication network bandwidth, once you get beyond a certain size of cluster (about 6-7 machines), your replication traffic on a system that is handling a lot of writes, is going to be an appreciable percentage of the overall available bandwidth of your internal network. Latency can build up soon, and your primary is going to be sending out a lot of oplog copies to each secondary, which can result in saturation of its nic. If you lump application traffic in on top of that, then the bandwidth requirements can become extreem.

  3. Manan Shah 2013/11/04 at 5:17 pm #

    Is there a way to check the lag in milliseconds of time? Currently MMS show the lag in seconds.

  4. Max Hodges 2015/02/20 at 2:33 am #

    is milliseconds necessary?

  5. Max Hodges 2015/02/20 at 2:35 am #

    >My master is ssd and secondaries is sata.

    To have the best chance of keeping up, a secondary host should match the primary host’s specs for CPU, disk IOPS, and network I/O. If it’s outmatched by the primary on any of these specs, a secondary may fall behind during periods of sustained write activity. Depending on load this will, at best, create brief excursions in replication lag and, at worst, cause the secondary to fall irretrievably behind.

  6. replika cartie armband 2017/03/15 at 4:38 pm #

    cartierbraceletlove Me gusta el look! La verdad es que nunca me había planteado llevar ponchos con un look un poco más formal. Si te apetece pasarte por mi blog, me encantaría leer tu opinión sobre mis últimos posts! 😉
    replika cartie armband http://www.bestlovegift.nl/de/

  7. cartierbraceletlove HAHA! Love the 4th ‘stache – Charlie Chaplin reincarnated.
    van cleef arpels braccialetto oro rosa copia http://www.braccialegioielli.cn/

  8. replique bague 3 diamants cartier 2017/03/15 at 4:39 pm #

    cartierbraceletlove My husband and I don’t eat seafood – could I substitute chicken? Or something else?
    replique bague 3 diamants cartier http://www.toplevejewelry.com.ru/fr/

  9. seo 2017/03/16 at 8:46 pm #

    Hello Web Admin, I noticed that your On-Page SEO is is missing a few factors, for one you do not use all three H tags in your post, also I notice that you are not using bold or italics properly in your SEO optimization. On-Page SEO means more now than ever since the new Google update: Panda. No longer are backlinks and simply pinging or sending out a RSS feed the key to getting Google PageRank or Alexa Rankings, You now NEED On-Page SEO. So what is good On-Page SEO?First your keyword must appear in the title.Then it must appear in the URL.You have to optimize your keyword and make sure that it has a nice keyword density of 3-5% in your article with relevant LSI (Latent Semantic Indexing). Then you should spread all H1,H2,H3 tags in your article.Your Keyword should appear in your first paragraph and in the last sentence of the page. You should have relevant usage of Bold and italics of your keyword.There should be one internal link to a page on your blog and you should have one image with an alt tag that has your keyword….wait there’s even more Now what if i told you there was a simple WordPress plugin that does all the On-Page SEO, and automatically for you? That’s right AUTOMATICALLY, just watch this 4minute video for more information at. Seo Plugin

  10. air max 90 online india 2017/03/21 at 7:12 am #

    San Francisco’s coaching staff adjusted to Carolina’s rash of read-option looks and stymied Newton with a blitz package that churned out five sacks and nine quarterback hits on the day.

  11. basket neo adidas 2017/03/23 at 4:08 am #

    Sanchez is superior to the noodle-armed Matt Flynn.

  12. Sheeree 2017/03/26 at 4:38 pm #

    Hello I am so grateful I found your blog page, I really found you by
    error, while I was looking on Yahoo for something else, Regardless I am here now and would
    just like to say many thanks for a remarkable post and a all round thrilling
    blog (I also love the theme/design), I don’t have time to read through
    it all at the minute but I have bookmarked it and also added in your RSS feeds, so when I have
    time I will be back to read more, Please do keep up the fantastic work.

  13. nike air max sat?n al 2017/03/27 at 12:09 am #

    saying Detroit is always looking for a player who can ;make a 5-yard play into a 90-yard play.

  14. nike air max textile 2017/03/28 at 1:15 am #

    Sanchez cited Kelly’s ;cutting edgeapproach to nutrition.

  15. Frank 2017/03/29 at 12:27 am #

    I couldd nnot refrain from commenting. Very wel written! Whoa!

    This blokg looks eexactly like mmy oold one!

    It’s onn a entirely differenjt tipic bbut iit haas pretgty mhch the same layout and design. Wonderrful choice oof colors!
    Hello, I ead your neww stff regularly. Yourr wrtiting sfyle iis awesome, krep
    doing whqt you’re doing! http://foxnews.org

  16. nike air max thea light blue 2017/03/29 at 7:51 am #

    He always gives me his left hand but today I said to him: 鈥楪ive me the right hand.

  17. adidas superstar ii homme 2017/03/29 at 8:09 pm #

    saying Monday: ;I’ll tell you one thing: If your quarterback isn’t right.

  18. lace frontal 2017/03/31 at 11:05 am #

    My new lace frontal https://www.youtube.com/watch?v=ny8rUpI_98I arrived today and that i just like them! They suit fantastic……they can be pretty awesome .

  19. abcdefgsl 2017/03/31 at 12:27 pm #

    I dont waste my free time in watching video clips however I like to read posts on net and get updated from newest technologies.
    abcdefgsl

  20. colimas oceanside 2017/03/31 at 9:24 pm #

    You’ll find final costs, fix up property profits which have to be paid and costs, thus reducing the net profits
    to be divided between you.

  21. Gene 2017/04/01 at 4:47 am #

    Today, I went too the beach with my children. I found a sea shell and gave it to
    my 4 year oold daughter aand said “You can hear the ocean if you put this to your ear.” She placed the shell to her ear and
    screamed. There was a hermit crab inside and it pinched
    her ear. She never wants to go back! LoL I know tis is entirely
    off topic but I had to trll someone!

  22. Https://goo.gl 2017/04/01 at 8:14 am #

    I need to to thank you for this good read!! I definitely
    loved every bit of it. I have got you bookmarked to check out new
    stuff you post…

  23. Gabriella 2017/04/02 at 3:50 pm #

    Hello, after reading this remarkable paraggraph i am too happy to share
    my familiarity here with mates.

  24. como conquistar uma amiga bonita 2017/04/03 at 3:31 am #

    Sim! Finalmente algo sobre como conquistar uma amiga homem alpha.

  25. Este projeto é ímpios ! Você certamente sabe como manter um
    leitor divertia. Entre sua sagacidade e seus vídeos, eu estava quase mudou-se para começar o
    meu próprio blog (bem, quase… HaHa!) Trabalho excelente.
    Eu realmente desfrutado que você tinha a dizer, e mais do que isso, como você apresentou.
    Legal demais!

  26. Ah, isso foi um muito bom post. Passar algum tempo e esforço real para criar um
    soberba artigo… mas o que posso dizer… Eu adiar coisas muito
    e não parecem obter nada feito.

  27. curso 2017/04/03 at 6:22 am #

    Queria te agradecer por isto fantástico ler!!!!
    Eu absolutamente amei cada um pouco de isso.
    Eu tenho você livro marcado olhar novo coisas que você postar…

  28. Pedro João 2017/04/03 at 7:15 am #

    É o melhor tempo tornar algumas planos para o futuro e
    é tempo para ser feliz. Eu tenho ler isto acondicionados e se eu
    poderia eu quer recomendar você alguns interessante questões ou conselhos .
    Talvez você poderia escrever próxima artigos sobre este artigo.
    Eu desejo ler ainda mais questões sobre isso!

  29. academia do Importador login gratis 2017/04/03 at 8:03 am #

    Inspiradoras busca lá. O que ocorreu depois? Obrigado
    !

  30. O Guia da reconquista perfeita 2017/04/03 at 9:44 am #

    Excelente , o que um página Web é! Isto site
    dá útil informações para nós, continue assim.

  31. O Guia da reconquista perfeita 2017/04/03 at 10:11 am #

    Muito bom postar. Eu simplesmente tropeçou em
    seu weblog e queria mencionar que tenho verdadeiramente desfrutado navegandoing ao redor seu weblog postos.
    Afinal de contas Vou estar assinando em seu alimentar e espero você escrever mais uma
    vez muito em breve!

  32. marketing digital para medicos 2017/04/03 at 11:27 am #

    Eu acho que o admin deste web site é verdadeiramente trabalhando duro em favor da
    dele página web, como aqui todas as material é qualidade com base
    material.

  33. marketing digital nos estados unidos 2017/04/03 at 11:50 am #

    Quando alguém escreve um post He/she retém o pensei
    de um usuário em sua cérebro que como um usuário pode estar ciente de.
    Então é por isso que este post é grande . Obrigado!

  34. marketing digital ferramentas 2017/04/03 at 12:05 pm #

    Para mais up-to-date notícias você tem que
    visite web e diante web encontrei isto web site como um mais excelente site para
    mais novo atualizações.

  35. O Guia da reconquista perfeita 2017/04/03 at 12:42 pm #

    Acho que é entre o mais vital informação para mim.
    E sou que bom ler seu artigo. Mas quero observação na alguns
    coisas em geral, o web site estilo é ideal , os artigos é realmente legal :
    D. bom trabalho, felicidades

  36. viva-social.com 2017/04/03 at 1:31 pm #

    Grande batida! Eu deseja aprendiz enquanto você alterar seu web site ,
    como pode inscrever-se para um weblog local ?
    A conta ajudou me um aplicável lidar. Eu tinha sido minúscula bit familiar
    isto sua transmissão ofereceu vibrante limpar
    ideia

  37. formula negócio online login 2017/04/03 at 3:13 pm #

    Isto post é genuinamente um agradável
    que ajuda novo internet usuários , que estão desejando
    em favor da blogging.

  38. Tuyet 2017/04/03 at 10:26 pm #

    Mi consejo acostumbra a ser que si la cocina tiene más de quince
    años se debe cambiar todas y cada una de las cañerías viejas, a una instalación nueva, y acomodar los desagües a los requerimientos del cliente.

  39. garcinia cambogia 2017/04/04 at 12:16 am #

    Really when someone doesn’t understand then its up to other visitors
    that they will assist, so here it takes place.

  40. An intriguing discussion is worth comment. There’s no doubt that that you ought to
    publish more on this subject, it may not be a taboo subject but usually people
    don’t discuss such subjects. To the next! All the best!!

  41. Reuben 2017/04/04 at 10:07 am #

    Foi assim que tive a ideia de disponibilizar meu e também-book grátis nas redes sociais,
    da mesma maneira que faço com os mapas mentais, mas uma amiga minha me alertou sobre pirataria, uma vez que existem pessoas aproveitadoras que copiam seu material e também vendem na internet dizendo que
    foram eles que fizeram. http://wiki.hiddenparloratx.com/User:JillAplin0899

  42. formula negócio online download 2017/04/04 at 1:36 pm #

    Eu acreditar o que publicado foi muito razoável .
    No entanto, isto? se você digitado uma catchier título ?
    Quer dizer, eu não quero para te dizer como executar seu site, no entanto suponha você adicionou algo para possivelmente pegar do
    povo atenção? É % BLOG_TITLE % um pouco chato.
    Você poderia ápice no Yahoo frente página e
    ver como eles criar artigo títulos para agarrar pessoas interessados .
    Você pode adicionar um vídeo relacionado ou um imagens relacionadas ou
    dois para obter leitores interessados sobre
    o que já tem a dizer. Só minha opinião -pode fazer seu blog um pouco
    mais interessante.

  43. Maria Isis 2017/04/04 at 8:30 pm #

    Isto post é verdadeiramente um agradável
    que auxilia novo web usuários , que estão desejando
    em favor da blogging.

  44. Taylah 2017/04/05 at 8:05 pm #

    Why users still use to read news papers when in this technological globe
    all is available on web?

  45. l celle skin care 2017/04/05 at 11:29 pm #

    If some one wishes to be updated with most recent technologies afterward hhe must be go to seee this web page and be up to date every day.

  46. marketing digital descrição 2017/04/06 at 12:21 am #

    Eu Divirta-se com , chumbo para encontrei exatamente o costumava ser
    tendo um olhar para. Você tem terminou meu quatro
    dia longa caçar! Deus abençoe que você cara.
    Tem um grande dia. Adeus

  47. nuvega Lash Eye 2017/04/06 at 12:46 am #

    Hi there great website! Does running a bllg like this take a lot oof
    work? I have absolutely nno understanding of computer programming but I had been hoping
    to start my own blog soon. Anyway, if you have any ideas or tips for new blog owners
    please share. I understand this is off subnect howdver I simply had
    to ask. Thanks!

  48. I think the admin of this site is in fact working hard in support of his site, as here every material is
    quality based stuff.

  49. Advanced lash Review 2017/04/06 at 5:14 am #

    This is my first time ppay a visit aat here andd i am
    actually happy to read all at alone place.

  50. advanced Lash 2017/04/06 at 7:27 am #

    Thannk you for some other magnificent article. The
    place else may juxt anybody get that kind of info in suuch an ideal
    approah of writing? I have a presentation subsequent week, and I am
    on the look ffor such info.

  51. healthygarciniacambogiapro.Net 2017/04/06 at 1:54 pm #

    If you would like to take much from this articpe then you have to
    apply these methods to your won webpage.

  52. Garcinia Slim Diet Supplement 2017/04/06 at 2:07 pm #

    I got this web page frm my ppal who shared with me
    on the toplic of this webb page and at the moment this time I am visiting tnis site and reading very
    informative content at this place.

  53. 1K daily Profit 2017/04/06 at 5:42 pm #

    It’s amazing to pay a visit this web age and reading the views of
    all colleagues regarding this post, while I am aloso zealous of getting knowledge.

  54. Jessica 2017/04/06 at 7:25 pm #

    What’s Taking place i am new to this, I stumbled upoon this I
    have found It positively useful and it has helped me out loads.
    I hope to giv a contribution & aid other customers like its helped me.
    Great job.

  55. اغانى شعبى 2017/04/09 at 1:09 pm #

    Hi there,I read your blogs named “Replication Lag in MongoDB Replica Sets” like every week.Your writing style is witty, keep up the good work! And you can look our website about اغانى شعبى.

  56. cambogia draw 2017/04/09 at 1:45 pm #

    Hello, I enjoy reading through your article post. I wanted to write a little comment to support
    you.

  57. sports chiropractic 2017/04/09 at 2:19 pm #

    I’m curious to find out what blog platform you have been utilizing?
    I’m having some minor security issues with my
    latest blog and I would like to find something more secure.
    Do you have any suggestions?

  58. plasticsurgerydetroit.xyz 2017/04/09 at 3:08 pm #

    You are a very clever person!

  59. Http://Maxno2.Net 2017/04/09 at 5:39 pm #

    Howdy! This article couldn’t be written much better!
    Looking at this article reminds me of my prvious roommate!

    He always kept talking about this. I am going to send this articlee to him.

    Pretty sure he’s going to have a very good read.

    Thankks for sharing!

  60. revitasence cream Anti-Aging 2017/04/09 at 5:42 pm #

    Hi there! I just wanted to ask if you ever have any trouble with hackers?
    My last blog (wordpress) was hacked and I ended up losing several weeks of hard wkrk due tto no datra
    backup. Do yoou have any methods to prevent hackers?

  61. Alpha Muscle Complex Supplement 2017/04/09 at 7:32 pm #

    This paragraph will help the internet viewers for setting up neew
    blog or evfen a blog from start to end.

  62. Lacell Serum 2017/04/09 at 7:49 pm #

    Hi, i think that i saw you visited mmy website so
    i came tto “return the favor”.I’m attempting to find things to improve my site!I suppose its ok to use some
    of your ideas!!

  63. Fitcrew USA TST 1700 Supplement 2017/04/09 at 8:00 pm #

    Great web site. Plenty of useful informaton here.

    I’m sending it to a few friends ans additrionally sharing in delicious.

    And naturally, thank youu on your sweat!

  64. Charlene 2017/04/09 at 8:17 pm #

    Hi there friends, pleasant piece of wriiting and fastidious urging commented at this place, I am iin fact enjoying by these.

  65. no right click 2017/04/09 at 9:30 pm #

    Thank you for the good writeup. It in fact used to be a
    amusement account it. Look advanced to more added agreeable from you!
    By the way, how could we keep in touch?

  66. Winstoniceks 2017/04/09 at 9:43 pm #

    thanks towards this countless illuminating website, obstruct up the momentous jobless check out this [url=http://onlinecasinos-x.com]casino[/url] offers

  67. Flawless forskolin diet 2017/04/09 at 9:45 pm #

    Hi, Neat post. There’s a problem along with your website in internet explorer, may check
    this? IE still is the market chief and a large part of folks will
    miss your excellent writing due to this problem.

  68. NO2 X 720 Review 2017/04/09 at 9:52 pm #

    This is really fascinating, You’re an overly prokfessional blogger.

    I’ve joined your rss feed and stay up for searching for extgra of your fantastic post.
    Additionally, I’ve shared your wwebsite in my social
    networks

  69. Sherryl 2017/04/09 at 10:14 pm #

    These places where you could locate various sorts of ATVs for sale will certainly be able to help you to obtain the
    very best feasible ATV for your life.

  70. Ultimate Alpha Extreme Review 2017/04/09 at 11:07 pm #

    Attractive section of content. I just stumbled upon your site and inn accession capital to assert thatt I acquire in fact enjoyed account your blog posts.
    Anyway I’ll be subscribing tto our feeds and even I achievement you access
    consistently fast.

  71. Fitcrew USA Pump 2400 2017/04/10 at 12:04 am #

    Great delivery. Solid arguments. Keeep up the amazing work.

  72. Aurore Clear Serum Skincare 2017/04/10 at 6:20 am #

    If some one desires to be updated with newest
    technklogies therefore he must be visit thi
    web site and be upp to date everyday.

  73. crevtest 2017/04/10 at 6:27 am #

    Very nice post. I simply stumbgled upon your weblog and wanted to mention tnat I’vetruly enjoyed surfing around
    your blog posts. In any case I will be subscribing to your rsss feed and I am hoping you write agaijn very
    soon!

  74. Aurore Clear Serum Skincare 2017/04/10 at 7:34 am #

    I’ve learn a few exfellent stuff here. Certainly value bookmarking for revisiting.
    I wonder how a lot effort you put to make any such fantastic informative website.

  75. fitcrew usa tst 1700 2017/04/10 at 8:30 am #

    This is very interesting, You are a very skilled blogger.
    I hve joined your rss feed and look forward to seeking more
    of your excellent post. Also, I have shared your site in my social networks!

  76. Revitasence Cream Anti-Aging 2017/04/10 at 9:50 am #

    Since the admin off this web page iis working, no doubt very quickly it
    wull bbe renowned, due too its feature contents.

  77. utv yxz 1000r 2017/04/10 at 10:00 am #

    Emergency services companies should place just as much
    time, initiative, believed and due diligence into the acquisition of their
    UTV as they would for their following ambulance or fire truck.

  78. Revi Spa Cream 2017/04/10 at 10:41 am #

    Heya superb blog! Does running a blog similar to this require a lot of work?
    I have absolutely nno understanding off programming but
    I had been hoping to start my own blog soon. Anyway, should you have
    any recommendations or techniques forr new blog owners please share.
    I understand this is off topic but I sumply wanted to ask.

    Appreciate it!

  79. Elliott 2017/04/10 at 11:26 am #

    When you go shopping online, you’ll have accessibility to a broad selection of sizes and kinds of ATV tires as well as
    house delivery, saving you money and time.

  80. Louann 2017/04/10 at 11:58 am #

    Howdy! I’m at work browsing your blog from my new iphone 4!
    Just wanted to say I love reading your blog and look forward to all your posts!
    Carry on the great work!

  81. Margo 2017/04/10 at 12:07 pm #

    You simply choose the amount of ATV insurance protection you want,
    and you’re safeguarded as much as those limitations.

  82. Latanya 2017/04/10 at 12:29 pm #

    When you choose a service center from our In your corner ® Specialty Repair NetworkSM,
    all repair works come with a composed assurance for as long as you rent the atv or own.

  83. Http://Sihuskincares.Com 2017/04/10 at 1:58 pm #

    Greetings! Very helpful advice within this article!
    It’s the little changes which will make the greatest changes.
    Many thanks for sharing!

  84. patent pending 2017/04/11 at 9:53 am #

    Way cool! Some extremely valid points! I appreciate you writing this article and also the rest of the site is really good.

  85. 4rx 2017/04/11 at 3:22 pm #

    Incredible….such a handy online site. http://bit.ly/2p7LTKb

  86. juliannzmt257blog.blogdigy.com 2017/04/12 at 4:29 am #

    Ahaa, its nice dialogue concerning this paragraph at this place at this blog, I have read
    all that, so at this time me also commenting at this place.

  87. enriquecxqi443blog.mybjjblog.com 2017/04/12 at 4:35 am #

    I simply wanted to jot down a small remark to say thanks to you for the
    precious secrets you are giving here. My particularly long internet look up has at
    the end of the day been paid with extremely good content to share with
    my company. I would assert that most of us website visitors are really fortunate
    to exist in a magnificent network with many special professionals with beneficial suggestions.
    I feel quite blessed to have seen your entire webpage and look forward to
    plenty of more pleasurable moments reading here.
    Thanks a lot again for a lot of things.

  88. Velma 2017/04/12 at 9:34 am #

    These are the most popular names around who specialize in UTV manufacturing together with
    other sporting activity craft like bikes, ATV’s, jet skis, snow sleds, etc

  89. dawsondxnf838blog.blogkoo.com 2017/04/13 at 4:42 pm #

    Hi there! This blog post couldn�t be written any better!
    Looking at this post reminds me of my previous roommate!
    He constantly kept preaching about this. I will send this information to
    him. Fairly certain he will have a good read.
    Thank you for sharing!

  90. https://Www.Ogs.Ny.gov 2017/04/13 at 5:43 pm #

    Excellent post.Never knew this, thank you for letting me know.

  91. Www.Oneprayer.Net 2017/04/13 at 6:08 pm #

    Wow, this post is nice, my younger sister is analyzing these
    things, thus I am going to let know her.

  92. www.vvwra.Com 2017/04/13 at 6:45 pm #

    I like this site very much, Its a real nice berth to read and find info.

  93. finance 2017/04/13 at 7:16 pm #

    Generally Ido not learn article on blogs, however I would like tto say that this write-up very compelle me
    too chek out andd do it! Your writing taste has been amaze me.
    Thank you, quite nice post.

  94. phonepsychicreadings.world 2017/04/13 at 9:55 pm #

    Keep up the wonderful piece of work, I read several posts on this
    website and I think that your web blog is really attention-grabbing and also includes many outstanding information.

  95. Phone Psychic Readings 2017/04/13 at 10:24 pm #

    Thank you for sharing your impressive and amazing recommendations.
    I will not be hesitant to share your website to everybody who should receive tips and hints such as these.

  96. http://phonepsychicreadings.club 2017/04/13 at 10:34 pm #

    So great to see amazing articles within this blog.
    Appreciation for posting and sharing them.

  97. Miranda 2017/04/14 at 6:06 am #

    As a Newbie, I am continuously searching online for articles that can help me.
    Thank you

  98. 薬剤師求人 沖縄 2017/04/14 at 7:28 am #

    Pretty component of content. I simply stumbled upon your web site and in accession capital to claim that I get in fact enjoyed account your weblog posts.
    Any way I’ll be subscribing in your feeds and even I success you get right of entry to persistently rapidly.

  99. Www.Aiellocalabro.Org 2017/04/14 at 8:05 am #

    Wow, this post is fastidious, my sister is analyzing these kinds of things,
    so I am going to convey her.

  100. thumbnail 2017/04/16 at 8:36 am #

    Appreciate this post. Let me try it out.

  101. review temulawak cream 2017/04/16 at 10:41 pm #

    I every time spent my half an hour to read this webpage’s content every day along with a cup of coffee.

  102. Freemyapps Credits Adder 2017/04/17 at 7:06 am #

    Hello! I could have sworn I’ve been to this website before but after checking through some
    of the post I realized it’s new to me. Anyhow, I’m definitely happy I
    found it and I’ll be bookmarking and checking back frequently!

  103. Cathryn 2017/04/17 at 7:42 am #

    Wow, that’s what I was searching for, what a stuff! present here at this website, thanks admin of this website.

  104. nyautostyle.com 2017/04/17 at 7:50 am #

    I definitely wanted to type a simple word to be able to thank you for some of the marvelous suggestions you are giving
    here. My considerable internet research has at the end
    been recognized with beneficial facts to exchange with my companions.

    I would claim that we website visitors actually
    are unquestionably lucky to exist in a really good network with many special people with good advice.

    I feel pretty privileged to have encountered your site and look
    forward to so many more cool minutes reading here. Thanks a lot again for everything.

  105. adult toy review 2017/04/17 at 3:25 pm #

    I’m really impressed with your writing talents as smartly as with the structure to your blog.
    Is that this a paid subject or did you modify
    it yourself? Either way keep up the excellent quality writing,
    it’s uncommon to look a great weblog like this one today..

  106. cultbytes.com 2017/04/17 at 9:00 pm #

    It’s enormous that you are getting thoughts from this paragraph as well as from
    our argument made at this time.

  107. myfeixin.cn 2017/04/17 at 11:42 pm #

    It’s an amazing article in support of all the web viewers; they will obtain benefit
    from it I am sure.

  108. moneysmartonline.com 2017/04/18 at 5:51 am #

    I think the admin of this site is really working
    hard in support of his web site, since here every information is quality based stuff.

  109. Steve 2017/04/18 at 7:11 am #

    I’ve been exploring for a little for any high-quality articles or blog posts on this sort of house .

    Exploring in Yahoo I at last stumbled upon this website.
    Reading this info So i am glad to express that I’ve a very just right uncanny
    feeling I found out exactly what I needed. I such a lot indisputably will make certain to don?t omit this web
    site and provides it a look regularly.

  110. http://www.semqx.com/ 2017/04/18 at 8:35 am #

    It’s fantastic that you are getting ideas from this article as
    well as from our argument made at this time.

  111. Wilford 2017/04/18 at 10:47 am #

    I likewise conceive thus, perfectly indited post!

  112. Greetings from Los angeles! I’m bored at work so I decided to check out your site on my iphone during lunch break.
    I really like the information you present here and can’t wait to take a look when I get home.
    I’m amazed at how fast your blog loaded on my phone .. I’m not even using WIFI, just 3G ..
    Anyhow, good site!

  113. Finley 2017/04/18 at 3:05 pm #

    Hello there! This article couldn�t be written any better!
    Reading through this post reminds me of my
    previous roommate! He continually kept talking about this.
    I most certainly will forward this article to him.
    Pretty sure he will have a great read. Thank you for sharing!

  114. Good article and straight to the point. I don’t know if this is truly the best place to
    ask but do you guys have any thoughts on where to hire some professional writers?
    Thank you :)

  115. adana.cloud9escorts.com 2017/04/18 at 8:52 pm #

    Greetings! I’ve been reading your blog for some time now and finally
    got the bravery to go ahead and give you a shout
    out from Kingwood Tx! Just wanted to say keep
    up the fantastic job!

  116. find more information 2017/04/19 at 1:11 pm #

    Sweet blog! I found it while searching on Yahoo News. Do you have any tips on how to get listed in Yahoo News?
    I’ve been trying for a while but I never seem to get there!
    Appreciate it

  117. Christian 2017/04/19 at 5:59 pm #

    They need to move their own campaigns to take note of
    regional SEO as an alternative.

  118. Natalia 2017/04/20 at 1:14 pm #

    I love what you guys are usually up too. This kind
    of clever work and coverage! Keep up the good works guys I’ve incorporated you guys to my personal
    blogroll. http://katleriokortos.lt/?option=com_k2&view=itemlist&task=user&id=1875621

  119. r-l-design.de 2017/04/21 at 9:20 am #

    Hola! I’ve been following your web site for a long time now and finally got
    the courage to go ahead and give you a shout out from Porter Texas!

    Just wanted to mention keep up the excellent work!

  120. nbdb8.com 2017/04/21 at 9:25 am #

    Hello there! This article couldn�t be written much better!
    Reading through this post reminds me of my previous
    roommate! He constantly kept preaching about this. I am going to forward this
    article to him. Pretty sure he’ll have a great read. Many thanks for sharing!

  121. uanloginepfo.hatenablog.com 2017/04/21 at 12:41 pm #

    Thanks for your really good information. They’re very useful.

  122. uanloginepfo.Com 2017/04/21 at 12:49 pm #

    Interesting post. I’ll be sticking around to find out much more from you guys.
    Thanks!

  123. Johnette 2017/04/21 at 1:03 pm #

    This is very interesting, You’re a really skilled
    blogger. I have joined your feed and look forward to reading more of your excellent post.
    Also, I have shared your website in my social networks!

  124. Armando 2017/04/21 at 3:22 pm #

    Heya i’m for the first time here. I came across this board and I find It really useful
    & it helped me out much. I hope to give something back and help others
    like you helped me.

  125. rustenburginsider.co.za 2017/04/22 at 1:47 pm #

    I believe other website proprietors should take this
    site as an model, very clean and excellent user pleasant layout.

  126. side-effects free 2017/04/22 at 6:17 pm #

    I loved as much as you will receive carried out right here.
    The sketch is tasteful, your authored subject matter stylish.

    nonetheless, you command get got an shakiness
    over that you wish be delivering the following. unwell unquestionably come further formerly again since exactly the same nearly very often inside case you
    shield this hike.

  127. myfeixin.cn 2017/04/22 at 6:44 pm #

    Greetings! Very helpful advice in this particular post!
    It’s the little changes which will make the biggest changes.
    Thanks a lot for sharing!

  128. Alejandra 2017/04/22 at 9:27 pm #

    Greetings! Very useful advice within this post! It’s the little changes
    which will make the most important changes. Thanks for sharing!

  129. Salvatore 2017/04/23 at 12:49 pm #

    Hi! I’ve been following your site for a long time now and finally got the bravery to go ahead and give you a shout out from Austin Tx!
    Just wanted to say keep up the good work!

  130. Danilo 2017/04/23 at 1:28 pm #

    I’ve been exploring for a little for any high quality
    articles or blog posts on this kind of space . Exploring in Yahoo I at
    last stumbled upon this web site. Reading this information So i am satisfied to show that I’ve a
    very excellent uncanny feeling I found out exactly what I needed.
    I such a lot for sure will make sure to don?t fail to remember this website
    and give it a glance on a continuing basis.

  131. Jaclyn 2017/04/23 at 1:51 pm #

    I have been exploring for a bit for any high quality articles or weblog posts
    on this kind of house . Exploring in Yahoo I ultimately stumbled upon this web site.
    Studying this info So i’m glad to show that I have a very excellent uncanny
    feeling I came upon exactly what I needed. I so much unquestionably will make sure to don?t fail to
    remember this site and provides it a look on a continuing basis.

  132. Judson 2017/04/23 at 4:06 pm #

    Hi! I’ve been reading your website for a while
    now and finally got the courage to go ahead and give you a shout out from Humble Tx!
    Just wanted to tell you keep up the fantastic work!

  133. Awsome info and right to the point. I am not sure if this is truly the best place to ask but do you
    guys have any ideea where to hire some professional writers?

    Thank you :)

Trackbacks/Pingbacks

  1. Sysadmin Sunday 119 - Server Density Blog - 2013/03/24

    […] Replication Lag in MongoDB Replica Sets […]

  2. Octopus vs Makara : Read-write adapters for ActiveRecord | The Wisdom Tree - 2015/11/15

    […] from a MongoLab blog, replication lag is the delay between the time an operation occurs on the primary and the time […]

Leave a Reply