Hello!
I'm currently experimenting how a good DRBD Dual Primary Setup can be
achieved with Pacemaker. I know all of the "you have to have good
fencing in place things" ... that is just what I'am currently trying to
test in my setup beside other things.
But even without a node crashing or the link dropping I already have the
problem that I always run into a split brain situation when a node comes
up that was e.g. in Standby before.
For example: I have both Nodes running connected both primary,
everything is fine. I put one node into standby and DRBD is stopped on
this node.
I do some work, reboot the server and so on finally I try to re join the
node in the cluster. Pacemaker is starting all resources and finally
DRBD drops the connection informing me about a split brain.
In the log this looks like:
Oct 1 19:44:42 storage-test-d kernel: [ 111.138512] block drbd10:
disk( Diskless -> Attaching )
Oct 1 19:44:42 storage-test-d kernel: [ 111.139283] drbd testdata1:
Method to ensure write ordering: drain
Oct 1 19:44:42 storage-test-d kernel: [ 111.139288] block drbd10: max
BIO size = 1048576
Oct 1 19:44:42 storage-test-d kernel: [ 111.139296] block drbd10:
drbd_bm_resize called with capacity == 838835128
Oct 1 19:44:42 storage-test-d kernel: [ 111.144488] block drbd10:
resync bitmap: bits=104854391 words=1638350 pages=3200
Oct 1 19:44:42 storage-test-d kernel: [ 111.144494] block drbd10: size
= 400 GB (419417564 KB)
Oct 1 19:44:42 storage-test-d kernel: [ 111.289327] block drbd10:
recounting of set bits took additional 3 jiffies
Oct 1 19:44:42 storage-test-d kernel: [ 111.289334] block drbd10: 0 KB
(0 bits) marked out-of-sync by on disk bit-map.
Oct 1 19:44:42 storage-test-d kernel: [ 111.289346] block drbd10:
disk( Attaching -> UpToDate )
Oct 1 19:44:42 storage-test-d kernel: [ 111.289352] block drbd10:
attached to UUIDs
A41D74E79299A144:0000000000000000:86B0140AA1A527C0:86AF140AA1A527C1
Oct 1 19:44:42 storage-test-d kernel: [ 111.321564] drbd testdata2:
conn( StandAlone -> Unconnected )
Oct 1 19:44:42 storage-test-d kernel: [ 111.321628] drbd testdata2:
Starting receiver thread (from drbd_w_testdata [3211])
Oct 1 19:44:42 storage-test-d kernel: [ 111.321794] drbd testdata2:
receiver (re)started
Oct 1 19:44:42 storage-test-d kernel: [ 111.321822] drbd testdata2:
conn( Unconnected -> WFConnection )
Oct 1 19:44:42 storage-test-d kernel: [ 111.337708] drbd testdata1:
conn( StandAlone -> Unconnected )
Oct 1 19:44:42 storage-test-d kernel: [ 111.337764] drbd testdata1:
Starting receiver thread (from drbd_w_testdata [3215])
Oct 1 19:44:42 storage-test-d kernel: [ 111.337904] drbd testdata1:
receiver (re)started
Oct 1 19:44:42 storage-test-d kernel: [ 111.337927] drbd testdata1:
conn( Unconnected -> WFConnection )
Oct 1 19:44:43 storage-test-d kernel: [ 111.808897] block drbd10:
role( Secondary -> Primary )
Oct 1 19:44:43 storage-test-d kernel: [ 111.810883] block drbd11:
role( Secondary -> Primary )
Oct 1 19:44:43 storage-test-d kernel: [ 111.820040] drbd testdata2:
Handshake successful: Agreed network protocol version 101
Oct 1 19:44:43 storage-test-d kernel: [ 111.820046] drbd testdata2:
Agreed to support TRIM on protocol level
Oct 1 19:44:43 storage-test-d kernel: [ 111.823292] block drbd10: new
current UUID
8369EB6F395C0D29:A41D74E79299A144:86B0140AA1A527C0:86AF140AA1A527C1
Oct 1 19:44:43 storage-test-d kernel: [ 111.836096] drbd testdata1:
Handshake successful: Agreed network protocol version 101
Oct 1 19:44:43 storage-test-d kernel: [ 111.836108] drbd testdata1:
Agreed to support TRIM on protocol level
Oct 1 19:44:43 storage-test-d kernel: [ 111.848917] block drbd11: new
current UUID
69A056C665A38F35:C8B4320C2FE11A0C:D13C0AA6DC58CC8C:D13B0AA6DC58CC8D
Oct 1 19:44:43 storage-test-d kernel: [ 111.871100] drbd testdata2:
conn( WFConnection -> WFReportParams )
Oct 1 19:44:43 storage-test-d kernel: [ 111.871108] drbd testdata2:
Starting asender thread (from drbd_r_testdata [3249])
Oct 1 19:44:43 storage-test-d kernel: [ 111.909687] drbd testdata1:
conn( WFConnection -> WFReportParams )
Oct 1 19:44:43 storage-test-d kernel: [ 111.909695] drbd testdata1:
Starting asender thread (from drbd_r_testdata [3270])
Oct 1 19:44:43 storage-test-d kernel: [ 111.943986] drbd testdata2:
meta connection shut down by peer.
Oct 1 19:44:43 storage-test-d kernel: [ 111.944063] drbd testdata2:
conn( WFReportParams -> NetworkFailure )
Oct 1 19:44:43 storage-test-d kernel: [ 111.944067] drbd testdata2:
asender terminated
Oct 1 19:44:43 storage-test-d kernel: [ 111.944070] drbd testdata2:
Terminating drbd_a_testdata
Oct 1 19:44:43 storage-test-d kernel: [ 111.988005] drbd testdata1:
meta connection shut down by peer.
Oct 1 19:44:43 storage-test-d kernel: [ 111.988089] drbd testdata1:
conn( WFReportParams -> NetworkFailure )
Oct 1 19:44:43 storage-test-d kernel: [ 111.988094] drbd testdata1:
asender terminated
Oct 1 19:44:43 storage-test-d kernel: [ 111.988098] drbd testdata1:
Terminating drbd_a_testdata
Oct 1 19:44:43 storage-test-d kernel: [ 112.031948] drbd testdata2:
Connection closed
Oct 1 19:44:43 storage-test-d kernel: [ 112.032116] drbd testdata2:
conn( NetworkFailure -> Unconnected )
Oct 1 19:44:43 storage-test-d kernel: [ 112.032121] drbd testdata2:
receiver terminated
Oct 1 19:44:43 storage-test-d kernel: [ 112.032124] drbd testdata2:
Restarting receiver thread
Oct 1 19:44:43 storage-test-d kernel: [ 112.032127] drbd testdata2:
receiver (re)started
Oct 1 19:44:43 storage-test-d kernel: [ 112.032136] drbd testdata2:
conn( Unconnected -> WFConnection )
Oct 1 19:44:43 storage-test-d kernel: [ 112.096002] drbd testdata1:
Connection closed
Oct 1 19:44:43 storage-test-d kernel: [ 112.096194] drbd testdata1:
conn( NetworkFailure -> Unconnected )
To resolve this problem I simply put the cluster into maintenance mode,
stop drbd on the one node which I just brought back on and reconnect on
the other side then start the other node's DRBD again and it finally
connects without a problem into Secondary/Primary state. Without
enforcing data being dropped. Afterwards I can go back into
Primary/Primary. In a real world setup at this point the fencing would
have kicked in and with bit of bad luck even fenced the healthy node (as
I already saw the split brain detected messages on either side of the
cluster), bringing the possibly outdated side up with all of it's
ancient data.
As I can see from the logs the resource is being promoted already even
it is still in WFConnection state. I assume this might be be problem
here, that both sides are primary already when they come to the point
where the connection is established and then one node drops the
connection. I don't think that this can be the desired behaviour. How
can pacemaker be made aware of that it is promoting drbd only if it is
already in a connected state and with (assumed) good data? Or to say
1. bring up drbd into secondary
2. let drbd determine if data has to be resynced and so on
3. when drbd is finally in "Secondary/UpToDate" state promote it. and
afterwards start services that rely on the drbd device. If something
goes wrong the promote should fail and the cluster could finally fence
the outdated node. I know there might be a RARE situation where it might
be necesary to start a Secondary/Unknown node up to Primary (e.g.
Cluster was degraded and for some reason the remainung good node had
restarted (or had to be restarted) - but this might be a thing that
could be handled manually.
This is a portion from the cluster config (should generally speaking be
everything that is related to drbd directly):
primitive drbd_testdata1 ocf:linbit:drbd \
params drbd_resource="testdata1" \
op monitor interval="29s" role="Master" \
op monitor interval="31s" role="Slave"
ms ms_drbd_testdata1 drbd_testdata1 \
meta master-max="2" master-node-max="1" clone-max="2"
clone-node-max="1" notify="true" target-role="Master"
location l-drbd1 ms_drbd_testdata1 \
rule $id="l-drbd1-rule" 0: #uname eq storage-test-d or #uname
eq storage-test-c
Thanks for any hints in advance,
Felix
_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
