Big Data and Replication: challenge accepted.

Having PB of data and analyzing them is a problem Bugsense solved by creating our own stream database called LDB (written in Erlang and C), but replicating data is another thing, and way trickier for that matter.

So, let’s talk about replication.

Let’s assume that there is one Master node and 8 nodes that contain the replicated data.The most interesting problem is not what happens when the Master node goes down but what happens when it comes back. Is the Master node going to take over? Is it going to be a replicated node meaning that someone else will be elected as the Master? As with all problems, there is no one-size-fits-all solution.

If you choose to have the Master node take over when it comes back (because it has more processing power than the other nodes) then you are facing two problems:

Either Master’s data will be out of date or it will take some time until it updates with the new data from the replication nodes. But which node has the latest data? Again, there is no single solution. You can either wait for a unanimous answer (read: transcations) or have a vote.

If let’s say, more than 25% of the nodes return the same result, you’re good to go! The higher the precentage, the slower the restore but more consistent. That’s how PAXOS algorithm (very roughly!) works. 

Another solution, which I am more fond of is treating every node equally (talk about democracy!).

When the Master node goes down, a replication node is elected as Master. The election is either random (if you don’t care about consistency at all) or based on the last node that had data. When the Master comes back, it gets pushed to the replication pool and becomes “one of them”.

And that’s roughly how LDB’s replication system works. I am planning to publish drafts of most of my experiments with replication strategies here, so find me on twitter if you want to start some interesting conversations! 

- Jon