1. Active->Slave replication with manual failover
This is where you can configure one machine to output all changes to a local daemon and another machine to implement the changes that are read from a local daemon.
Pro:
simplist implementation, since it makes no assumptions about how you are going to use it, it also sets no limits on how it is used.
This is the basic functionality that all other variations will need so it's not wasted work no matter what is done later
allows for multiple slaves from a single master
allows for the propogation traffic pattern to be defined by the sysadmin (either master directly to all slaves or a tree-like propogation to save on WAN bandwidth when multiple slaves are co-located
by involving a local daemon at each server there is a lot of flexibility in exactly how the replication takes place.
for example you could
use netcat as your daemon for instant transmission of the messages
have a daemon that caches the messages so that if the link drops the messages are saved
have a daemon that gets an acknowlegement from the far side that the message got through
have a daemon that batches the messages up and compresses them for more efficiant transport
have a daemon that delays all messages by a given time period to give you a way to recover from logical corruption without having to go to a backup
have a daemon that filters the messages (say one that updates everything except it won't delete any messages so you have a known safe archive of all messages)
etc
Con:
since it makes no assumptions about how you are going to use it, it also gives you no help in useing it in any particular way
2. Active->Slave replication with automatic failover
This takes #1, limits it to a pair of boxes and through changes to murder or other parts of cyrus will swap the active/slave status of the two boxes
Pro: makes setting up of a HA pair of boxes easier
increases availability by decreasing downtime
Con:
this functionality can be duplicated without changes to cyrus by the use of an external HA/cluster software package.
Since this now assumes a particular mode of operation it starts to limit other uses (for example, if this is implemented as part of murder then it won't help much if you are trying to replicate to a DR datacenter several thousand miles away).
Split-brain conditions are the responsibility of cyrus to prevent or solve. These are fundamentaly hard problems to get right in all cases
3. Active->Slave replication with Slave able to accept client connections
This takes #1 and then further modifies the slave so that requests that would change the contents of things get relayed to the active box and then the results of the change get propogated back down before they are visable to the client.
Pro:
simulates active/active operation although it does cause longer delays when clients issue some commands.
use of slaves for local access can reduce the load on the master resulting in higher performance.
can be cascaded to multiple slaves and multiple tiers of slaves as needed
in case of problems on the master the slaves can continue to operate as read-only servers providing degraded service while the master is fixed. depending on the problem with the master this may be very preferable to having to re-sync the master or recover from a split-brain situation
Con:
more extensive modifications needed to trap all changes and propogate them up to the master
how does the slave know when the master has implemented the change (so that it can give the result to the client)
raises questions about the requirement to get confirmation og all updates before the slave can respond to the client (for example, if a slave decides to read a message that is flagged as new should the slave wait until the master confirms that it knows the message has been read before it gives it to the client, or should it give the message to the client and not worry if the update fails on the master)
since the slave needs to send updates to the master the latency of the link between them can become a limiting factor in the performance that clients see when connecting to the slave
4. #3 with automatic failover
Since #3 supports multiple slaves the number of failover senerios grow significantly. you have multiple machines that could be the new master and you have the split-brain senerio to watch out for.
Pro: increased availability by decreasing failover time
potentially easier to setup then with external clustering software
Con: increased complexity
runs the risk of breaking some deployment senerios in an attempt to simplify others
5. Active/Active
designate one of the boxes as primary and identify all items in the datastore that absolutly must not be subject to race conditions between the two boxes (message UUID for example). In addition to implementing the replication needed for #1 modify all functions that need to update these critical pieces of data to update them on the master and let the master update the other box.
Pro:
best use of available hardware as the load is split almost evenly between the boxes.
best availability becouse if there is a failure half of the clients won't see it at all
Con: significantly more complex then the other options.
behavior during a failure is less obvious
split-brain recovery is not straightforward and if automatic failover is active the sysadmin will have no option to have things degraded slightly while a problem is fixed
depending on the implementation this may be very sensitive to network latency between the machines and could be very suitable for working with machines in the same datacenter, but worthless for machines thousands of miles apart.
6. active/active/active/...
Take #5 and extend the idea to more then a pair of boxes. this makes the updates more complex to propogate (they now need to be sent to every other machine in the cluster)
Pro: better load balancing then #5
allows for the ability to have a HA pair in a primary location and a backup in a remote location (i.e. your main HQ has two boxes, but your disaster recovery center has one as well)
Con:
the complexity goes up significantly when you shift from 2 to n boxes in a cluster.
the bandwidth required for updates increases by a factor of roughly n!
significantly more split-brain senerios become possible and need to be accounted for.
-------------------------------------------------------------------------
while #6 is the ideal option to have it can get very complex
personally I would like to see #1 (with a sample daemon or two to provide basic functionality and leave the doors open for more creative uses) followed by #3 while people try and figure out all the problems with #5 and #6
there are a lot of senerios that are possible with #1 or #3 that are not possible with #5 and very little of the work needed to release #1 and #3 as supported options is not work that needs to be done towards #5/6 anyway (the pieces need to be identified in the code and hooks put in place in the code at those locations. the details of the hooks will differ slightly
David Lang
-- There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
-- C.A.R. Hoare
---
Cyrus Home Page: http://asg.web.cmu.edu/cyrus
Cyrus Wiki/FAQ: http://cyruswiki.andrew.cmu.edu
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
