Status: New
Owner: ----
New issue 270 by [email protected]: drbd sync fails on readded node
http://code.google.com/p/ganeti/issues/detail?id=270
What software version are you running? Please provide the output of "gnt-
cluster --version" and "gnt-cluster version".
gnt-cluster (ganeti v2.4.5) 2.4.5
Software version: 2.4.5
Internode protocol: 2040000
Configuration format: 2040000
OS api version: 20
Export interface: 0
What distribution are you using?
Gentoo and Ubuntu 12.04
both ganeti and drbd is the same version:
version: 8.3.11 (api:88/proto:86-96)
What steps will reproduce the problem?
1. Reinstalled node2
2. readded node2
3. gnt-instance replace-disks -s test1
What is the expected output? What do you see instead?
Please provide any additional information below.
Sync fails after some time syncing telling the disk are degraded.
gnt-instance replace-disks -s test1
Tue Sep 25 21:36:25 2012 Replacing disk(s) 0 for test1.lan
Tue Sep 25 21:36:36 2012 STEP 1/6 Check device existence
Tue Sep 25 21:36:36 2012 - INFO: Checking disk/0 on node1.lan
Tue Sep 25 21:36:40 2012 - INFO: Checking disk/0 on bestefar.ss.uib.no
Tue Sep 25 21:36:40 2012 - INFO: Checking volume groups
Tue Sep 25 21:36:42 2012 STEP 2/6 Check peer consistency
Tue Sep 25 21:36:42 2012 - INFO: Checking disk/0 consistency on node
node1.lan
Failure: command execution error:
Node node1.lan has degraded storage, unsafe to replace disks for instance
test1.lan
primary:
Sep 25 21:35:16 node1 ganeti-rapi[17004]: INFO 127.0.0.1:43227 GET /version
HTTP/1.1 200
Sep 25 21:35:16 node1 ganeti-masterd[676]: (ClientReq1) INFO Received
config values query request for ['master_node']
Sep 25 21:35:16 node1 ganeti-masterd[676]: (ClientReq5) INFO Received job
autoarchive request for age 21600, timeout 29
Sep 25 21:35:16 node1 ganeti-masterd[676]: (ClientReq5) INFO Archiving jobs
with age more than 21600 seconds
Sep 25 21:35:16 node1 ganeti-masterd[676]: (ClientReq15) INFO Received new
job
Sep 25 21:35:17 node1 ganeti-masterd[676]: (ClientReq11) INFO Received job
poll request for 81612
Sep 25 21:35:17 node1 ganeti-masterd[676]: (ClientReq7) INFO Received job
poll request for 81612
Sep 25 21:35:17 node1 ganeti-masterd[676]: (JobQueue3/Job81612) INFO Op
1/2: opcode INSTANCE_QUERY waiting for locks
Sep 25 21:35:17 node1 ganeti-masterd[676]: (ClientReq10) INFO Received job
poll request for 81612
Sep 25 21:35:17 node1 ganeti-masterd[676]: (ClientReq13) INFO Received job
poll request for 81612
Sep 25 21:35:17 node1 ganeti-noded[17015]: INFO 192.168.1.35:51808 PUT
/all_instances_info HTTP/1.1 200
Sep 25 21:35:17 node1 ganeti-masterd[676]: (ClientReq3) INFO Received job
poll request for 81612
Sep 25 21:35:17 node1 ganeti-masterd[676]: (JobQueue22/Job81612) INFO Op
2/2: opcode NODE_QUERY waiting for locks
Sep 25 21:35:17 node1 ganeti-masterd[676]: (ClientReq4) INFO Received job
poll request for 81612
Sep 25 21:35:18 node1 ganeti-masterd[676]: (ClientReq8) INFO Received job
poll request for 81612
Sep 25 21:35:19 node1 ganeti-noded[17038]: INFO 192.168.1.35:51813 PUT
/node_info HTTP/1.1 200
Sep 25 21:35:19 node1 ganeti-masterd[676]: (JobQueue22/Job81612) INFO
Finished job 81612, status = success
Sep 25 21:35:19 node1 ganeti-masterd[676]: (ClientReq2) INFO Received job
query request for 81612
Sep 25 21:35:19 node1 ganeti-masterd[676]: (ClientReq14) INFO Received job
archive request for 81612
Sep 25 21:35:19 node1 ganeti-masterd[676]: (ClientReq14) INFO Archiving job
81612
Sep 25 21:35:19 node1 ganeti-masterd[676]: (ClientReq12) INFO Received new
job
Sep 25 21:35:20 node1 ganeti-masterd[676]: (ClientReq16) INFO Received job
poll request for 81613
Sep 25 21:35:20 node1 ganeti-masterd[676]: (ClientReq9) INFO Received job
poll request for 81613
Sep 25 21:35:20 node1 ganeti-masterd[676]: (JobQueue13/Job81613) INFO Op
1/1: opcode CLUSTER_VERIFY_DISKS waiting for locks
Sep 25 21:35:20 node1 ganeti-masterd[676]: (ClientReq6) INFO Received job
poll request for 81613
Sep 25 21:35:20 node1 ganeti-masterd[676]: (ClientReq1) INFO Received job
poll request for 81613
Sep 25 21:35:21 node1 ganeti-noded[17055]: INFO 192.168.1.35:51821 PUT
/lv_list HTTP/1.1 200
Sep 25 21:35:21 node1 ganeti-masterd[676]: (JobQueue13/Job81613) INFO
Finished job 81613, status = success
Sep 25 21:35:22 node1 ganeti-masterd[676]: (ClientReq5) INFO Received job
query request for 81613
Sep 25 21:35:22 node1 ganeti-masterd[676]: (ClientReq15) INFO Received job
archive request for 81613
Sep 25 21:35:22 node1 ganeti-masterd[676]: (ClientReq15) INFO Archiving job
81613
Sep 25 21:36:24 node1 gnt-instance replace-disks[17172]: INFO run with
arguments '-s test1'
Sep 25 21:36:24 node1 ganeti-masterd[676]: (MainThread) INFO Accepted
connection from pid=17172, uid=0, gid=0
Sep 25 21:36:24 node1 ganeti-masterd[676]: (ClientReq11) INFO Received new
job
Sep 25 21:36:24 node1 ganeti-masterd[676]: (ClientReq7) INFO Received job
poll request for 81614
Sep 25 21:36:24 node1 ganeti-masterd[676]: (ClientReq10) INFO Received job
poll request for 81614
Sep 25 21:36:25 node1 ganeti-masterd[676]: (JobQueue25/Job81614) INFO Op
1/1: opcode INSTANCE_REPLACE_DISKS(test1) waiting for locks
Sep 25 21:36:25 node1 ganeti-masterd[676]: (ClientReq13) INFO Received job
poll request for 81614
Sep 25 21:36:25 node1 ganeti-masterd[676]: (ClientReq3) INFO Received job
poll request for 81614
Sep 25 21:36:25 node1 ganeti-noded[17183]: INFO 192.168.1.35:51829 PUT
/hooks_runner HTTP/1.1 200
Sep 25 21:36:25 node1 ganeti-masterd[676]: (ClientReq4) INFO Received job
poll request for 81614
Sep 25 21:36:29 node1 kernel: [10648783.529804] block drbd1: Starting
worker thread (from drbdsetup [17196])
Sep 25 21:36:29 node1 kernel: [10648783.529911] block drbd1: disk( Diskless
-> Attaching )
Sep 25 21:36:29 node1 kernel: [10648783.531221] block drbd1: No usable
activity log found.
Sep 25 21:36:29 node1 kernel: [10648783.531225] block drbd1: Method to
ensure write ordering: flush
Sep 25 21:36:29 node1 kernel: [10648783.531232] block drbd1: drbd_bm_resize
called with capacity == 41943040
Sep 25 21:36:29 node1 kernel: [10648783.531443] block drbd1: resync bitmap:
bits=5242880 words=81920 pages=160
Sep 25 21:36:29 node1 kernel: [10648783.531447] block drbd1: size = 20 GB
(20971520 KB)
Sep 25 21:36:29 node1 kernel: [10648783.536060] block drbd1: bitmap READ of
160 pages took 5 jiffies
Sep 25 21:36:29 node1 kernel: [10648783.536797] block drbd1: recounting of
set bits took additional 0 jiffies
Sep 25 21:36:29 node1 kernel: [10648783.536800] block drbd1: 17 GB (4388992
bits) marked out-of-sync by on disk bit-map.
Sep 25 21:36:29 node1 kernel: [10648783.536807] block drbd1: disk(
Attaching -> UpToDate ) pdsk( DUnknown -> Outdated )
Sep 25 21:36:29 node1 kernel: [10648783.536812] block drbd1: attached to
UUIDs 3534D005D75C18E0:0005000000000004:0004000000000004:0003000000000004
Sep 25 21:36:29 node1 kernel: [10648783.575210] block drbd1: conn(
StandAlone -> Unconnected )
Sep 25 21:36:29 node1 kernel: [10648783.575239] block drbd1: Starting
receiver thread (from drbd1_worker [17197])
Sep 25 21:36:29 node1 kernel: [10648783.575288] block drbd1: receiver
(re)started
Sep 25 21:36:29 node1 kernel: [10648783.575297] block drbd1: conn(
Unconnected -> WFConnection )
Sep 25 21:36:29 node1 ganeti-noded[17187]: INFO 192.168.1.35:51830 PUT
/blockdev_assemble HTTP/1.1 200
Sep 25 21:36:30 node1 kernel: [10648784.383073] block drbd1: Handshake
successful: Agreed network protocol version 96
Sep 25 21:36:30 node1 kernel: [10648784.383365] block drbd1: Peer
authenticated using 16 bytes of 'md5' HMAC
Sep 25 21:36:30 node1 kernel: [10648784.383375] block drbd1: conn(
WFConnection -> WFReportParams )
Sep 25 21:36:30 node1 kernel: [10648784.383396] block drbd1: Starting
asender thread (from drbd1_receiver [17212])
Sep 25 21:36:30 node1 kernel: [10648784.383869] block drbd1:
data-integrity-alg: <not-used>
Sep 25 21:36:30 node1 kernel: [10648784.383896] block drbd1:
drbd_sync_handshake:
Sep 25 21:36:30 node1 kernel: [10648784.383900] block drbd1: self
3534D005D75C18E0:0005000000000004:0004000000000004:0003000000000004
bits:4388992 flags:0
Sep 25 21:36:30 node1 kernel: [10648784.383905] block drbd1: peer
0005000000000004:0000000000000000:0000000000000000:0000000000000000
bits:4388992 flags:0
Sep 25 21:36:30 node1 kernel: [10648784.383909] block drbd1:
uuid_compare()=1 by rule 70
Sep 25 21:36:30 node1 kernel: [10648784.383912] block drbd1: Becoming sync
source due to disk states.
Sep 25 21:36:30 node1 kernel: [10648784.383922] block drbd1: peer( Unknown
-> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( Outdated ->
Inconsistent )
Sep 25 21:36:30 node1 kernel: [10648784.431059] block drbd1: helper
command: /bin/true before-resync-source minor-1
Sep 25 21:36:30 node1 kernel: [10648784.431734] block drbd1: helper
command: /bin/true before-resync-source minor-1 exit code 0 (0x0)
Sep 25 21:36:30 node1 kernel: [10648784.431741] block drbd1: conn(
WFBitMapS -> SyncSource )
Sep 25 21:36:30 node1 kernel: [10648784.431751] block drbd1: Began resync
as SyncSource (will sync 17555968 KB [4388992 bits set]).
Sep 25 21:36:30 node1 kernel: [10648784.431756] block drbd1: updated sync
UUID 3534D005D75C18E0:0006000000000004:0005000000000004:0004000000000004
Sep 25 21:36:36 node1 kernel: [10648790.672360] block drbd1: role(
Secondary -> Primary )
Sep 25 21:36:36 node1 ganeti-noded[17222]: INFO 192.168.1.35:51832 PUT
/blockdev_assemble HTTP/1.1 200
Sep 25 21:36:36 node1 ganeti-masterd[676]: (JobQueue25/Job81614) INFO
Checking disk/0 on node1.lan
Sep 25 21:36:37 node1 ganeti-masterd[676]: (ClientReq8) INFO Received job
poll request for 81614
Sep 25 21:36:37 node1 ganeti-masterd[676]: (ClientReq2) INFO Received job
poll request for 81614
Sep 25 21:36:40 node1 ganeti-noded[17277]: INFO 192.168.1.35:51833 PUT
/blockdev_find HTTP/1.1 200
Sep 25 21:36:40 node1 ganeti-masterd[676]: (JobQueue25/Job81614) INFO
Checking disk/0 on bestefar.ss.uib.no
Sep 25 21:36:40 node1 ganeti-masterd[676]: (ClientReq14) INFO Received job
poll request for 81614
Sep 25 21:36:40 node1 ganeti-masterd[676]: (JobQueue25/Job81614) INFO
Checking volume groups
Sep 25 21:36:40 node1 ganeti-masterd[676]: (ClientReq12) INFO Received job
poll request for 81614
Sep 25 21:36:42 node1 ganeti-noded[17288]: INFO 192.168.1.35:51835 PUT
/vg_list HTTP/1.1 200
Sep 25 21:36:42 node1 ganeti-masterd[676]: (JobQueue25/Job81614) INFO
Checking disk/0 consistency on node node1.lan
Sep 25 21:36:42 node1 ganeti-masterd[676]: (ClientReq16) INFO Received job
poll request for 81614
Sep 25 21:36:42 node1 ganeti-masterd[676]: (ClientReq9) INFO Received job
poll request for 81614
Sep 25 21:36:45 node1 ganeti-noded[17295]: INFO 192.168.1.35:51837 PUT
/blockdev_find HTTP/1.1 200
Sep 25 21:36:45 node1 ganeti-noded[17303]: INFO 192.168.1.35:51838 PUT
/instance_list HTTP/1.1 200
Sep 25 21:36:48 node1 kernel: [10648802.423076] block drbd1: role( Primary
-> Secondary )
Sep 25 21:36:48 node1 kernel: [10648802.482407] block drbd1: peer(
Secondary -> Unknown ) conn( SyncSource -> Disconnecting )
Sep 25 21:36:48 node1 kernel: [10648802.482514] block drbd1: asender
terminated
Sep 25 21:36:48 node1 kernel: [10648802.482520] block drbd1: Terminating
drbd1_asender
Sep 25 21:36:48 node1 kernel: [10648802.493169] block drbd1: bitmap WRITE
of 152 pages took 3 jiffies
Sep 25 21:36:48 node1 kernel: [10648802.516910] block drbd1: 16 GB (4112512
bits) marked out-of-sync by on disk bit-map.
Sep 25 21:36:48 node1 kernel: [10648802.516922] block drbd1: Connection
closed
Sep 25 21:36:48 node1 kernel: [10648802.516932] block drbd1: conn(
Disconnecting -> StandAlone )
Sep 25 21:36:48 node1 kernel: [10648802.516954] block drbd1: receiver
terminated
Sep 25 21:36:48 node1 kernel: [10648802.516957] block drbd1: Terminating
drbd1_receiver
Sep 25 21:36:48 node1 kernel: [10648802.516997] block drbd1: disk( UpToDate
-> Failed )
Sep 25 21:36:48 node1 kernel: [10648802.517023] block drbd1: Sending state
for detaching disk failed
Sep 25 21:36:48 node1 kernel: [10648802.517037] block drbd1: disk( Failed
-> Diskless )
Sep 25 21:36:48 node1 kernel: [10648802.517221] block drbd1: drbd_bm_resize
called with capacity == 0
Sep 25 21:36:48 node1 kernel: [10648802.517270] block drbd1: worker
terminated
Sep 25 21:36:48 node1 kernel: [10648802.517273] block drbd1: Terminating
drbd1_worker
Sep 25 21:36:50 node1 ganeti-noded[17305]: INFO 192.168.1.35:51839 PUT
/blockdev_shutdown HTTP/1.1 200
Sep 25 21:36:50 node1 ganeti-masterd[676]: (JobQueue25/Job81614) ERROR Op
1/1: Caught exception in INSTANCE_REPLACE_DISKS(test1)
Sep 25 21:36:50 node1 Traceback (most recent call last):
Sep 25 21:36:50 node1
File "/usr/lib64/python2.7/site-packages/ganeti/jqueue.py", line 936, in
_ExecOpCodeUnlocked
Sep 25 21:36:50 node1 timeout=timeout, priority=op.priority)
Sep 25 21:36:50 node1
File "/usr/lib64/python2.7/site-packages/ganeti/mcpu.py", line 352, in
ExecOpCode
Sep 25 21:36:50 node1 priority)
Sep 25 21:36:50 node1
File "/usr/lib64/python2.7/site-packages/ganeti/mcpu.py", line 296, in
_LockAndExecLU
Sep 25 21:36:50 node1 result = self._LockAndExecLU(lu, level + 1,
calc_timeout, priority)
Sep 25 21:36:50 node1
File "/usr/lib64/python2.7/site-packages/ganeti/mcpu.py", line 305, in
_LockAndExecLU
Sep 25 21:36:50 node1 result = self._LockAndExecLU(lu, level + 1,
calc_timeout, priority)
Sep 25 21:36:50 node1
File "/usr/lib64/python2.7/site-packages/ganeti/mcpu.py", line 296, in
_LockAndExecLU
Sep 25 21:36:50 node1 result = self._LockAndExecLU(lu, level + 1,
calc_timeout, priority)
Sep 25 21:36:50 node1
File "/usr/lib64/python2.7/site-packages/ganeti/mcpu.py", line 256, in
_LockAndExecLU
Sep 25 21:36:50 node1 result = self._ExecLU(lu)
Sep 25 21:36:50 node1
File "/usr/lib64/python2.7/site-packages/ganeti/mcpu.py", line 231, in
_ExecLU
Sep 25 21:36:50 node1 result = lu.Exec(self.Log)
Sep 25 21:36:50 node1
File "/usr/lib64/python2.7/site-packages/ganeti/cmdlib.py", line 270, in
Exec
Sep 25 21:36:50 node1 tl.Exec(feedback_fn)
Sep 25 21:36:50 node1
File "/usr/lib64/python2.7/site-packages/ganeti/cmdlib.py", line 8365, in
Exec
Sep 25 21:36:50 node1 result = fn(feedback_fn)
Sep 25 21:36:50 node1
File "/usr/lib64/python2.7/site-packages/ganeti/cmdlib.py", line 8536, in
_ExecDrbd8DiskOnly
Sep 25 21:36:50 node1 False)
Sep 25 21:36:50 node1
File "/usr/lib64/python2.7/site-packages/ganeti/cmdlib.py", line 8431, in
_CheckDisksConsistency
Sep 25 21:36:50 node1 (node_name, self.instance.name))
Sep 25 21:36:50 node1 OpExecError: Node node1.lan has degraded storage,
unsafe to replace disks for instance test1.lan
Sep 25 21:36:50 node1 ganeti-masterd[676]: (JobQueue25/Job81614) INFO
Finished job 81614, status = error
Sep 25 21:36:50 node1 ganeti-masterd[676]: (ClientReq6) INFO Received job
query request for 81614
Sep 25 21:36:50 node1 gnt-instance replace-disks[17172]: ERROR Error during
command processing
Sep 25 21:36:50 node1 Traceback (most recent call last):
Sep 25 21:36:50 node1
File "/usr/lib64/python2.7/site-packages/ganeti/cli.py", line 1939, in
GenericMain
Sep 25 21:36:50 node1 result = func(options, args)
Sep 25 21:36:50 node1
File "/usr/lib64/python2.7/site-packages/ganeti/client/gnt_instance.py",
line 781, in ReplaceDisks
Sep 25 21:36:50 node1 SubmitOrSend(op, opts)
Sep 25 21:36:50 node1
File "/usr/lib64/python2.7/site-packages/ganeti/cli.py", line 1757, in
SubmitOrSend
Sep 25 21:36:50 node1 return SubmitOpCode(op, cl=cl,
feedback_fn=feedback_fn, opts=opts)
Sep 25 21:36:50 node1
File "/usr/lib64/python2.7/site-packages/ganeti/cli.py", line 1734, in
SubmitOpCode
Sep 25 21:36:50 node1 reporter=reporter)
Sep 25 21:36:50 node1
File "/usr/lib64/python2.7/site-packages/ganeti/cli.py", line 1715, in
PollJob
Sep 25 21:36:50 node1 return GenericPollJob(job_id, _LuxiJobPollCb(cl),
reporter)
Sep 25 21:36:50 node1
File "/usr/lib64/python2.7/site-packages/ganeti/cli.py", line 1537, in
GenericPollJob
Sep 25 21:36:50 node1 errors.MaybeRaise(msg)
Sep 25 21:36:50 node1
File "/usr/lib64/python2.7/site-packages/ganeti/errors.py", line 448, in
MaybeRaise
Sep 25 21:36:50 node1 raise errcls, args
Sep 25 21:36:50 node1 OpExecError: Node node1.lan has degraded storage,
unsafe to replace disks for instance test1.lan
secondary:
Sep 25 21:36:29 node2 kernel: [ 597.351382] block drbd1: disk( Diskless ->
Attaching )
Sep 25 21:36:29 node2 kernel: [ 597.352712] block drbd1: No usable
activity log found.
Sep 25 21:36:29 node2 kernel: [ 597.352717] block drbd1: Method to ensure
write ordering: flush
Sep 25 21:36:29 node2 kernel: [ 597.352725] block drbd1: drbd_bm_resize
called with capacity == 41943040
Sep 25 21:36:29 node2 kernel: [ 597.352878] block drbd1: resync bitmap:
bits=5242880 words=81920 pages=160
Sep 25 21:36:29 node2 kernel: [ 597.352882] block drbd1: size = 20 GB
(20971520 KB)
Sep 25 21:36:29 node2 kernel: [ 597.355536] block drbd1: bitmap READ of
160 pages took 1 jiffies
Sep 25 21:36:29 node2 kernel: [ 597.355673] block drbd1: recounting of set
bits took additional 0 jiffies
Sep 25 21:36:29 node2 kernel: [ 597.355676] block drbd1: 17 GB (4388992
bits) marked out-of-sync by on disk bit-map.
Sep 25 21:36:29 node2 kernel: [ 597.355682] block drbd1: disk( Attaching
-> Inconsistent )
Sep 25 21:36:29 node2 kernel: [ 597.355685] block drbd1: attached to UUIDs
0005000000000004:0000000000000000:0000000000000000:0000000000000000
Sep 25 21:36:29 node2 kernel: [ 597.395008] block drbd1: conn( StandAlone
-> Unconnected )
Sep 25 21:36:29 node2 kernel: [ 597.395058] block drbd1: Starting receiver
thread (from drbd1_worker [3587])
Sep 25 21:36:29 node2 kernel: [ 597.395300] block drbd1: receiver
(re)started
Sep 25 21:36:29 node2 kernel: [ 597.395308] block drbd1: conn( Unconnected
-> WFConnection )
Sep 25 21:36:30 node2 kernel: [ 597.895093] block drbd1: Handshake
successful: Agreed network protocol version 96
Sep 25 21:36:30 node2 kernel: [ 597.895335] block drbd1: Peer
authenticated using 16 bytes of 'md5' HMAC
Sep 25 21:36:30 node2 kernel: [ 597.895343] block drbd1: conn(
WFConnection -> WFReportParams )
Sep 25 21:36:30 node2 kernel: [ 597.895451] block drbd1: Starting asender
thread (from drbd1_receiver [3601])
Sep 25 21:36:30 node2 kernel: [ 597.895703] block drbd1:
data-integrity-alg: <not-used>
Sep 25 21:36:30 node2 kernel: [ 597.895720] block drbd1:
drbd_sync_handshake:
Sep 25 21:36:30 node2 kernel: [ 597.895724] block drbd1: self
0005000000000004:0000000000000000:0000000000000000:0000000000000000
bits:4388992 flags:0
Sep 25 21:36:30 node2 kernel: [ 597.895729] block drbd1: peer
3534D005D75C18E0:0005000000000004:0004000000000004:0003000000000004
bits:4388992 flags:0
Sep 25 21:36:30 node2 kernel: [ 597.895732] block drbd1: uuid_compare()=-1
by rule 50
Sep 25 21:36:30 node2 kernel: [ 597.895735] block drbd1: Becoming sync
target due to disk states.
Sep 25 21:36:30 node2 kernel: [ 597.895742] block drbd1: peer( Unknown ->
Secondary ) conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate )
Sep 25 21:36:30 node2 kernel: [ 597.941628] block drbd1: conn( WFBitMapT
-> WFSyncUUID )
Sep 25 21:36:30 node2 kernel: [ 597.985957] block drbd1: updated sync uuid
0006000000000004:0000000000000000:0000000000000000:0000000000000000
Sep 25 21:36:30 node2 kernel: [ 597.994754] block drbd1: helper command:
/bin/true before-resync-target minor-1
Sep 25 21:36:30 node2 kernel: [ 597.995590] block drbd1: helper command:
/bin/true before-resync-target minor-1 exit code 0 (0x0)
Sep 25 21:36:30 node2 kernel: [ 597.995597] block drbd1: conn( WFSyncUUID
-> SyncTarget )
Sep 25 21:36:30 node2 kernel: [ 597.995606] block drbd1: Began resync as
SyncTarget (will sync 17555968 KB [4388992 bits set]).
Sep 25 21:36:36 node2 kernel: [ 604.174503] block drbd1: peer( Secondary
-> Primary )
Sep 25 21:36:48 node2 kernel: [ 615.966592] block drbd1: peer( Primary ->
Secondary )
Sep 25 21:36:48 node2 kernel: [ 615.966815] block drbd1: peer( Secondary
-> Unknown ) conn( SyncTarget -> TearDown ) pdsk( UpToDate -> DUnknown )
Sep 25 21:36:48 node2 kernel: [ 615.967136] block drbd1: meta connection
shut down by peer.
Sep 25 21:36:48 node2 kernel: [ 615.967627] block drbd1: asender terminated
Sep 25 21:36:48 node2 kernel: [ 615.967651] block drbd1: Terminating
drbd1_asender
Sep 25 21:36:48 node2 kernel: [ 615.977086] block drbd1: bitmap WRITE of
152 pages took 2 jiffies
Sep 25 21:36:48 node2 kernel: [ 615.997371] block drbd1: 16 GB (4112512
bits) marked out-of-sync by on disk bit-map.
Sep 25 21:36:48 node2 kernel: [ 615.997392] block drbd1: Connection closed
Sep 25 21:36:48 node2 kernel: [ 615.997398] block drbd1: conn( TearDown ->
Unconnected )
Sep 25 21:36:48 node2 kernel: [ 615.997404] block drbd1: receiver
terminated
Sep 25 21:36:48 node2 kernel: [ 615.997406] block drbd1: Restarting
drbd1_receiver
Sep 25 21:36:48 node2 kernel: [ 615.997409] block drbd1: receiver
(re)started
Sep 25 21:36:48 node2 kernel: [ 615.997414] block drbd1: conn( Unconnected
-> WFConnection )
Sep 25 21:36:50 node2 kernel: [ 618.163815] block drbd1: conn(
WFConnection -> Disconnecting )
Sep 25 21:36:50 node2 kernel: [ 618.163905] block drbd1: Discarding
network configuration.
Sep 25 21:36:50 node2 kernel: [ 618.164037] block drbd1: Connection closed
Sep 25 21:36:50 node2 kernel: [ 618.164051] block drbd1: conn(
Disconnecting -> StandAlone )
Sep 25 21:36:50 node2 kernel: [ 618.164238] block drbd1: receiver
terminated
Sep 25 21:36:50 node2 kernel: [ 618.164243] block drbd1: Terminating
drbd1_receiver
Sep 25 21:36:50 node2 kernel: [ 618.164301] block drbd1: disk(
Inconsistent -> Failed )
Sep 25 21:36:50 node2 kernel: [ 618.164369] block drbd1: Sending state for
detaching disk failed
Sep 25 21:36:50 node2 kernel: [ 618.164848] block drbd1: disk( Failed ->
Diskless )
Sep 25 21:36:50 node2 kernel: [ 618.166200] block drbd1: drbd_bm_resize
called with capacity == 0
Sep 25 21:36:50 node2 kernel: [ 618.166243] block drbd1: worker terminated
Sep 25 21:36:50 node2 kernel: [ 618.166246] block drbd1: Terminating
drbd1_worker
