Once you start storing data you now have more problems. What if there was some disaster and the only data that you had was inaccessible?
Let’s look how Replication of data can help give us some resiliency in case of a disaster.
- Partitioning
- Data Protection Laws
Let’s see if we can break it all down in a post. If it takes longer, then we will split this article.
What is replication?
Replication in this context is about maintaining multiple copies of the data stored in the database. This is done for various reasons. In relational databases, data is replicated or mirrored to another instance, in order to be prepared for a failover. This helps in case the database in one region is impacted by a disaster.
In case of a NoSQL database, as they tend to be designed as distributed clusters, replication is done between nodes of the cluster to ensure high availability, scalability and performance.
You might be wondering why talk specifically about this if the database system already takes care of this. Not all databases support or handle every use case of replication. Thus it is important to understand replication as a concept and what happens behind the scenes and the different complexities that arise with a replication setup.
Taking a step back
We talked about the benefits of using a database in the first place instead of writing everything into just another text file. We want our databases to have the following:
- High availability despite faults
- High Scalability as load increases - rise in writes/reads
- High performance - low latency and high throughput
Complexities that come with replication
Some important questions that one has to consider when replicating data across multiple nodes:
- How are multiple copies of the data kept consistent with one another?
- What happens when replication fails?
- Is replication done asynchronously or synchronously? What are the pros and cons of either approach?
Synchronous and asynchronous replication
In a database system where data is spread across multiple nodes, data is generally replicated to a certain point to ensure high availability. So let’s look at the different ways to replicate data.
Let us walk through a scenario where a client - maybe a user, does a change in the UI that results in data to be updated in the database.
In this example let us imagine that the database has one primary node and has 2 replicas.
Synchronous replication
Occurring or existing at the same time - Wordnik definition of Synchronous
- Update request gets to the database server
- It sends the update to the primary node
- The primary node updates the data locally and triggers a replicate data notification/request to the replicas and waits
- The replicas do their job of replicating that update and sends back an acknowledgment to the primary saying “Job done”
- The primary then responds to the database server with the successful update status
So if the replication operation takes a while, the response to the update request also takes a while.
Advantage of Sync
Primary and secondary nodes are always in sync - data consistency is pretty high here!
Disadvantage of Sync
Imagine if one of the replicas failed to replicate data, it wouldn’t acknowledge the update and the primary would be sat there waiting till it gets a response from all its known replicas. Resulting in the client waiting forever. Poor user experience.
Asynchronous replication
Using the same example as earlier, the steps would look slightly different.
- Update request gets to the database server
- The database server sends the update to the primary node
- The primary node updates the data locally and responds immediately to the database server saying success
- It then triggers a notification to replicate the data
- The replicas do their job as soon as possible
Notice how the primary doesn’t really wait for any of the replicas to get back to it. It is like an empowered product company. The primary just trusts its replicas to do their jobs. Just because I said empowered doesn’t mean, this is ideal in every case.
Advantage of Async
No time wasted in waiting for anyone. Clients experience instant responses as soon as the primary finishes writing data! In fact the primary can continue with its next data update/read/write request and move on with life or work in this case.
Disadvantage of Async
As the name suggests, the updates happen asynchronously, not necessarily at the same time.
So imagine the primary having completed the update and responded to the server saying, “All great! Update Successful!”, and suddenly lost power in that region before it could trigger replicate.
The client would request the updated data but now that a disaster has happened, the database system switched its primary off from the pool and is now serving data from one of the secondary nodes who do not have the update just yet!
This lag of secondary nodes getting the update can get worse during peak update times. High volumes of updates happening at the primary and primary responding as fast as it can to the database server while also firing replicate notifications to the secondary nodes without waiting for a success response from them. The secondary nodes gradually catchup with all the updates, but there is always a chance that the primary would be way ahead of the secondary nodes.
Replication Models
There are many configurations in which data can be replicated synchronously or asynchronously.
Single Leader replication
This model is also called the primary-secondary replication. In this model, a node is chosen as a leader or primary. All write requests go to the primary. All read requests are spread across the cluster. Thus in case of writes, the primary acts like a controller that processes all writes to the cluster. It sends all the information downstream - to all the secondary nodes or followers and ensures that the data is in sync.
| Advantages | Disadvantages |
|---|---|
| In case of a read-heavy workload, one could add more followers to scale. | At the same time, too many followers can cause the primary to be overloaded. Not suitable for write-heavy workloads. |
| Resiliency in reads - in case of a primary failure, secondary or followers can continue serving data. | Asynchronous replication in this case could result in inconsistent reads - different followers may have different versions of the data in case the primary has failed. |
When a leader or primary fails, your cluster will not become useless. Generally in this approach, there is always an automated approach to selecting the next leader - generally called leader election. You can find more about the various Leader election algorithms on Distributed Systems Blog.
This would probably be a whole topic of its own. I found this video pretty nice.
Popular variations of Single Leader replication
Write-ahead log shipping
If you’ve used relational database, you are probably familiar with this. All information about a transaction in a database is actually recorded in a transaction log file. This is a persistent record of the series of events/transactions that happened. Then a process that’s part of the Database management system, runs and parses the transaction log and runs them on the primary and sends them to the secondary.
The transaction log also acts like a backup in case of a disaster/crash.
Statement-based replication
A method used in MySQL. You might as well read all the details from the official documentation.
In summary, all data modification statements - the inserts and updates and deletes, get executed on the primary node first and then gets written into a log file. The log file is then sent to the secondary for execution. This log file is not the transaction log, but actually a full list of statements.
What do you think might be a problem here?
Logical (row-based) replication
Many relational databases offer this method. Changes made to the database are captured at row level of a table. Unlike putting a change in a log file, the change to the row is captured on the primary with all the values and then later executed on the secondary nodes. I think some databases implement Change Data Capture using a combination of this method and Write Ahead Log.
Multi-leader replication
I think the name gives away the key difference from what was discussed earlier. This is a good alternative to the single leader approach and is indeed designed to tackle its limitations. However, there are always trade-offs to consider.
There are multiple leader nodes that processes writes and actually sends the same updates to other primaries and also secondaries! This is how they all maintain some level of consistency. Thus if your workload is write-heavy, increase the number of primaries. You’d be thinking that solves all my problems. But the more primaries you have, the higher the chances for conflicts. Many primaries get similar write requests and they have to update the same record then you are going to need to figure out how to deal with such conflicts.
Some popular approaches to dealing with conflicts are:
- Avoid conflicts: updates happen through a single leader or some locking mechanism has to be introduced. There are limitations to how far you can avoid conflicts like this. There is always an element of concurrency somewhere to be dealt with.
- Last write wins: the simplest way - accept that conflicts are inevitable. Follow a first-come, first-serve approach and let the state reflect what update happened last.
Peer-to-peer or leaderless replication
Leaderless - says it all. Eliminate bottlenecks caused by having to go through leaders. A bit of anarchy really, everyone takes an active leader like role.
However, this approach also has the same problem of eventual consistency. All writes have to be replicated and hence not all nodes in the cluster would have the same state of the data. But as with all problems in computer science, there is a solution to this one too - quorums! If you have used systems like Kafka, you might already be familiar with one such algorithm - Raft - consensus algorithm that implements quorum.