> Speaking of split brains, I haven't really been able to discern how to > recover from a split brain. What are the general techniques to recovering > from a split brain?
Recovering from a split-brain may be impossible depending on what happens during the split and what kind of logging you have in place or it may be as simple as just restarting the backup. Consider that once the brokers split they will have the same data, and as different clients interact with one broker or the other then each broker's copy of the data will begin to diverge from the other. One broker may receive messages that the other doesn't and clients may consume messages from one that aren't consumed from the other. If *no* client ever connects to the split slave then you won't have any of these problems. The solution in this case is just to restart the slave. Obviously the faster you can detect the split the better as that reduces the chances that any client will connect to the split slave. However, if one or more clients do connect to the split slave then you'll need to identify what that client did. If all the client does is produce messages then you should be able to see that simply by comparing the journal data from each broker [1]. Likewise if the client simply consumes messages. If both consumption and production occur then it can get a bit more tricky. If this happens on different queues (e.g. produce on queue "foo", consume from queue "bar") then a data comparison again should work. However, if production and consumption happen on the same queue then you could run into a situation where the same message is produced and consumed from one broker and that wouldn't show up in a data comparison since the message would have been consumed and wouldn't be in the journal. The LoggingActiveMQServerPlugin [2] should help with that, but of course it will need to be active/configured *before* the split occurs. In any event, this is a tedious, manual, and error prone process. You'll want to avoid it if at all possible. Justin [1] For example, using the journal XML export tool (using this CLI command: ./artemis data exp ...) [2] http://activemq.apache.org/components/artemis/documentation/latest/broker-plugins.html#using-the-loggingactivemqserverplugin On Thu, Jun 6, 2019 at 2:48 PM ahuhatwork <ahuhatw...@protonmail.com> wrote: > Thanks for the quick response Justin. > > I've configured Artemis to use replication as the infrastructure for > shared-storage isn't... great. > > So for my situation at work, the hypervisors tend to randomly die on us > (and > thus taking the VMs with them). We have 3 zones/hypervisors. > > I wanted a single master because we do not think the workload is high > enough > to require more than that. Due to the random hypervisor deaths, I wanted a > slave running in each zone. > > So: > zone 1: master > zone 2: slave > zone 3: slave > > The zones are connected by a nice giant heavy duty router which handles all > traffic, so I find it difficult to imagine that a split brain can occur for > that particular reason (perhaps other reasons which I am ignorant to). > > Speaking of split brains, I haven't really been able to discern how to > recover from a split brain. What are the general techniques to recovering > from a split brain? > > Albert > > > > > > -- > Sent from: > http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html >
