On Tue, May 8, 2012 at 4:27 AM, Robinson, Eric <eric.robin...@psmnv.com> wrote: > Hi guys, we rebooted a standby node of a healthy cluster and suddenly all the > resources on the primary cluster restarted. What's up with that? Before > rebooting the standby node, we did the normal stuff to verify that all was > well. > > crm_mon showed all nodes online, in their expected roles, with correct quorum > votes > cat /proc/drbd showed correct dbbd status > corosync-cfgtool -s showed all rings active without faults > > When we rebooted the standby node (ha08c), crm_mon on the primary node > (ha08a) showed that all the resources stopped and then restarted, resulting > in brief loss of availability to customers.
Did they actually restart though? When a new DC gathers the current state of the cluster, the resources may appear to restart in crm_mon but its often just a display issue. Granted its quite annoying. > > Following is what crm_mon showed before server ha08c was rebooted and after > it came back up. Following that is our crm configuration. > > ============ > Last updated: Mon May 7 11:13:32 2012 > Stack: openais > Current DC: ha08a.mycharts.md - partition with quorum > Version: 1.1.2-f059ec7ced7a86f18e5490b67ebf4a0b963bccfe > 3 Nodes configured, 3 expected votes > 4 Resources configured. > ============ > > Online: [ ha08a.mycharts.md ha08b.mycharts.md ha08c.mycharts.md ] > > Master/Slave Set: ms_drbd0 > Masters: [ ha08a.mycharts.md ] > Slaves: [ ha08c.mycharts.md ] > Master/Slave Set: ms_drbd1 > Masters: [ ha08b.mycharts.md ] > Slaves: [ ha08c.mycharts.md ] > Resource Group: g_clust06 > p_fs_clust06 (ocf::heartbeat:Filesystem): Started > ha08a.mycharts.md > p_vip_clust06 (ocf::heartbeat:IPaddr2): Started > ha08a.mycharts.md > p_mysql_371 (lsb:mysql_371): Started ha08a.mycharts.md > p_mysql_372 (lsb:mysql_372): Started ha08a.mycharts.md > p_mysql_373 (lsb:mysql_373): Started ha08a.mycharts.md > p_mysql_374 (lsb:mysql_374): Started ha08a.mycharts.md > p_mysql_375 (lsb:mysql_375): Started ha08a.mycharts.md > p_mysql_376 (lsb:mysql_376): Started ha08a.mycharts.md > p_mysql_047 (lsb:mysql_047): Started ha08a.mycharts.md > p_mysql_100 (lsb:mysql_100): Started ha08a.mycharts.md > p_mysql_379 (lsb:mysql_379): Started ha08a.mycharts.md > p_mysql_377 (lsb:mysql_377): Started ha08a.mycharts.md > p_mysql_378 (lsb:mysql_378): Started ha08a.mycharts.md > p_mysql_380 (lsb:mysql_380): Started ha08a.mycharts.md > p_mysql_381 (lsb:mysql_381): Started ha08a.mycharts.md > p_mysql_382 (lsb:mysql_382): Started ha08a.mycharts.md > p_mysql_383 (lsb:mysql_383): Started ha08a.mycharts.md > p_mysql_384 (lsb:mysql_384): Started ha08a.mycharts.md > p_mysql_385 (lsb:mysql_385): Started ha08a.mycharts.md > p_mysql_386 (lsb:mysql_386): Started ha08a.mycharts.md > p_mysql_387 (lsb:mysql_387): Started ha08a.mycharts.md > p_mysql_002 (lsb:mysql_002): Started ha08a.mycharts.md > p_mysql_035 (lsb:mysql_035): Started ha08a.mycharts.md > p_mysql_049 (lsb:mysql_049): Started ha08a.mycharts.md > p_mysql_097 (lsb:mysql_097): Started ha08a.mycharts.md > p_mysql_024 (lsb:mysql_024): Started ha08a.mycharts.md > p_mysql_077 (lsb:mysql_077): Started ha08a.mycharts.md > p_mysql_084 (lsb:mysql_084): Started ha08a.mycharts.md > p_mysql_113 (lsb:mysql_113): Started ha08a.mycharts.md > p_mysql_116 (lsb:mysql_116): Started ha08a.mycharts.md > p_mysql_388 (lsb:mysql_388): Started ha08a.mycharts.md > p_mysql_389 (lsb:mysql_389): Started ha08a.mycharts.md > p_mysql_390 (lsb:mysql_390): Started ha08a.mycharts.md > p_mysql_391 (lsb:mysql_391): Started ha08a.mycharts.md > p_mysql_392 (lsb:mysql_392): Started ha08a.mycharts.md > p_mysql_393 (lsb:mysql_393): Started ha08a.mycharts.md > p_mysql_394 (lsb:mysql_394): Started ha08a.mycharts.md > p_mysql_395 (lsb:mysql_395): Started ha08a.mycharts.md > p_mysql_396 (lsb:mysql_396): Started ha08a.mycharts.md > p_mysql_397 (lsb:mysql_397): Started ha08a.mycharts.md > p_mysql_398 (lsb:mysql_398): Started ha08a.mycharts.md > p_mysql_399 (lsb:mysql_399): Started ha08a.mycharts.md > p_mysql_400 (lsb:mysql_400): Started ha08a.mycharts.md > p_mysql_401 (lsb:mysql_401): Started ha08a.mycharts.md > p_mysql_402 (lsb:mysql_402): Started ha08a.mycharts.md > p_mysql_403 (lsb:mysql_403): Started ha08a.mycharts.md > p_mysql_404 (lsb:mysql_404): Started ha08a.mycharts.md > p_mysql_405 (lsb:mysql_405): Started ha08a.mycharts.md > p_mysql_104 (lsb:mysql_104): Started ha08a.mycharts.md > p_mysql_406 (lsb:mysql_406): Started ha08a.mycharts.md > p_mysql_407 (lsb:mysql_407): Started ha08a.mycharts.md > p_mysql_408 (lsb:mysql_408): Started ha08a.mycharts.md > p_mysql_409 (lsb:mysql_409): Started ha08a.mycharts.md > p_mysql_410 (lsb:mysql_410): Started ha08a.mycharts.md > p_mysql_017 (lsb:mysql_017): Started ha08a.mycharts.md > p_mysql_089 (lsb:mysql_089): Started ha08a.mycharts.md > p_mysql_090 (lsb:mysql_090): Started ha08a.mycharts.md > p_mysql_109 (lsb:mysql_109): Started ha08a.mycharts.md > p_mysql_411 (lsb:mysql_411): Started ha08a.mycharts.md > p_mysql_412 (lsb:mysql_412): Started ha08a.mycharts.md > p_mysql_413 (lsb:mysql_413): Started ha08a.mycharts.md > p_mysql_414 (lsb:mysql_414): Started ha08a.mycharts.md > p_mysql_415 (lsb:mysql_415): Started ha08a.mycharts.md > p_mysql_416 (lsb:mysql_416): Started ha08a.mycharts.md > p_mysql_417 (lsb:mysql_417): Started ha08a.mycharts.md > p_mysql_020 (lsb:mysql_020): Started ha08a.mycharts.md > p_mysql_076 (lsb:mysql_076): Started ha08a.mycharts.md > p_mysql_101 (lsb:mysql_101): Started ha08a.mycharts.md > p_mysql_051 (lsb:mysql_051): Started ha08a.mycharts.md > Resource Group: g_clust07 > p_fs_clust07 (ocf::heartbeat:Filesystem): Started > ha08b.mycharts.md > p_vip_clust07 (ocf::heartbeat:IPaddr2): Started > ha08b.mycharts.md > p_mysql_418 (lsb:mysql_418): Started ha08b.mycharts.md > p_mysql_419 (lsb:mysql_419): Started ha08b.mycharts.md > p_mysql_420 (lsb:mysql_420): Started ha08b.mycharts.md > p_mysql_421 (lsb:mysql_421): Started ha08b.mycharts.md > p_mysql_422 (lsb:mysql_422): Started ha08b.mycharts.md > p_mysql_423 (lsb:mysql_423): Started ha08b.mycharts.md > p_mysql_424 (lsb:mysql_424): Started ha08b.mycharts.md > p_mysql_425 (lsb:mysql_425): Started ha08b.mycharts.md > p_mysql_426 (lsb:mysql_426): Started ha08b.mycharts.md > p_mysql_427 (lsb:mysql_427): Started ha08b.mycharts.md > p_mysql_056 (lsb:mysql_056): Started ha08b.mycharts.md > p_mysql_133 (lsb:mysql_133): Started ha08b.mycharts.md > p_mysql_428 (lsb:mysql_428): Started ha08b.mycharts.md > p_mysql_429 (lsb:mysql_429): Started ha08b.mycharts.md > p_mysql_430 (lsb:mysql_430): Started ha08b.mycharts.md > p_mysql_431 (lsb:mysql_431): Started ha08b.mycharts.md > p_mysql_432 (lsb:mysql_432): Started ha08b.mycharts.md > p_mysql_433 (lsb:mysql_433): Started ha08b.mycharts.md > p_mysql_434 (lsb:mysql_434): Started ha08b.mycharts.md > p_mysql_435 (lsb:mysql_435): Started ha08b.mycharts.md > p_mysql_436 (lsb:mysql_436): Started ha08b.mycharts.md > p_mysql_437 (lsb:mysql_437): Started ha08b.mycharts.md > p_mysql_438 (lsb:mysql_438): Started ha08b.mycharts.md > p_mysql_439 (lsb:mysql_439): Started ha08b.mycharts.md > p_mysql_440 (lsb:mysql_440): Started ha08b.mycharts.md > p_mysql_441 (lsb:mysql_441): Started ha08b.mycharts.md > p_mysql_442 (lsb:mysql_442): Started ha08b.mycharts.md > p_mysql_443 (lsb:mysql_443): Started ha08b.mycharts.md > p_mysql_444 (lsb:mysql_444): Started ha08b.mycharts.md > p_mysql_445 (lsb:mysql_445): Started ha08b.mycharts.md > p_mysql_446 (lsb:mysql_446): Started ha08b.mycharts.md > p_mysql_447 (lsb:mysql_447): Started ha08b.mycharts.md > p_mysql_448 (lsb:mysql_448): Started ha08b.mycharts.md > p_mysql_449 (lsb:mysql_449): Started ha08b.mycharts.md > p_mysql_450 (lsb:mysql_450): Started ha08b.mycharts.md > p_mysql_019 (lsb:mysql_019): Started ha08b.mycharts.md > p_mysql_120 (lsb:mysql_120): Started ha08b.mycharts.md > p_mysql_451 (lsb:mysql_451): Started ha08b.mycharts.md > p_mysql_452 (lsb:mysql_452): Started ha08b.mycharts.md > p_mysql_453 (lsb:mysql_453): Started ha08b.mycharts.md > p_mysql_454 (lsb:mysql_454): Started ha08b.mycharts.md > p_mysql_455 (lsb:mysql_455): Started ha08b.mycharts.md > p_mysql_456 (lsb:mysql_456): Started ha08b.mycharts.md > p_mysql_457 (lsb:mysql_457): Started ha08b.mycharts.md > p_mysql_458 (lsb:mysql_458): Started ha08b.mycharts.md > p_mysql_459 (lsb:mysql_459): Started ha08b.mycharts.md > p_mysql_460 (lsb:mysql_460): Started ha08b.mycharts.md > > Here is our crm config... > > ^[[?1034hnode ha08a.mycharts.md \ > attributes standby="off" > node ha08b.mycharts.md \ > attributes standby="off" > node ha08c.mycharts.md \ > attributes standby="off" > primitive p_drbd0 ocf:linbit:drbd \ > params drbd_resource="ha01_mysql" \ > op monitor interval="30s" role="Slave" \ > op monitor interval="15s" role="Master" > primitive p_drbd1 ocf:linbit:drbd \ > params drbd_resource="ha02_mysql" \ > op monitor interval="30s" role="Slave" \ > op monitor interval="15s" role="Master" > primitive p_fs_clust06 ocf:heartbeat:Filesystem \ > params device="/dev/drbd0" directory="/ha01_mysql" fstype="ext3" > options="noatime" > primitive p_fs_clust07 ocf:heartbeat:Filesystem \ > params device="/dev/drbd1" directory="/ha02_mysql" fstype="ext3" > options="noatime" > primitive p_mysql_002 lsb:mysql_002 > primitive p_mysql_017 lsb:mysql_017 > primitive p_mysql_019 lsb:mysql_019 > primitive p_mysql_020 lsb:mysql_020 > primitive p_mysql_024 lsb:mysql_024 > primitive p_mysql_035 lsb:mysql_035 > primitive p_mysql_047 lsb:mysql_047 > primitive p_mysql_049 lsb:mysql_049 > primitive p_mysql_051 lsb:mysql_051 > primitive p_mysql_056 lsb:mysql_056 > primitive p_mysql_076 lsb:mysql_076 > primitive p_mysql_077 lsb:mysql_077 > primitive p_mysql_084 lsb:mysql_084 > primitive p_mysql_089 lsb:mysql_089 > primitive p_mysql_090 lsb:mysql_090 > primitive p_mysql_097 lsb:mysql_097 > primitive p_mysql_100 lsb:mysql_100 > primitive p_mysql_101 lsb:mysql_101 > primitive p_mysql_104 lsb:mysql_104 > primitive p_mysql_109 lsb:mysql_109 > primitive p_mysql_113 lsb:mysql_113 > primitive p_mysql_116 lsb:mysql_116 > primitive p_mysql_120 lsb:mysql_120 > primitive p_mysql_133 lsb:mysql_133 > primitive p_mysql_371 lsb:mysql_371 > primitive p_mysql_372 lsb:mysql_372 > primitive p_mysql_373 lsb:mysql_373 > primitive p_mysql_374 lsb:mysql_374 > primitive p_mysql_375 lsb:mysql_375 > primitive p_mysql_376 lsb:mysql_376 > primitive p_mysql_377 lsb:mysql_377 > primitive p_mysql_378 lsb:mysql_378 > primitive p_mysql_379 lsb:mysql_379 > primitive p_mysql_380 lsb:mysql_380 > primitive p_mysql_381 lsb:mysql_381 > primitive p_mysql_382 lsb:mysql_382 > primitive p_mysql_383 lsb:mysql_383 > primitive p_mysql_384 lsb:mysql_384 > primitive p_mysql_385 lsb:mysql_385 > primitive p_mysql_386 lsb:mysql_386 > primitive p_mysql_387 lsb:mysql_387 > primitive p_mysql_388 lsb:mysql_388 > primitive p_mysql_389 lsb:mysql_389 > primitive p_mysql_390 lsb:mysql_390 > primitive p_mysql_390 lsb:mysql_390 > primitive p_mysql_391 lsb:mysql_391 > primitive p_mysql_392 lsb:mysql_392 > primitive p_mysql_393 lsb:mysql_393 > primitive p_mysql_394 lsb:mysql_394 > primitive p_mysql_395 lsb:mysql_395 > primitive p_mysql_396 lsb:mysql_396 > primitive p_mysql_397 lsb:mysql_397 > primitive p_mysql_398 lsb:mysql_398 > primitive p_mysql_399 lsb:mysql_399 > primitive p_mysql_400 lsb:mysql_400 > primitive p_mysql_401 lsb:mysql_401 > primitive p_mysql_402 lsb:mysql_402 > primitive p_mysql_403 lsb:mysql_403 > primitive p_mysql_404 lsb:mysql_404 > primitive p_mysql_405 lsb:mysql_405 > primitive p_mysql_406 lsb:mysql_406 > primitive p_mysql_407 lsb:mysql_407 > primitive p_mysql_408 lsb:mysql_408 > primitive p_mysql_409 lsb:mysql_409 > primitive p_mysql_410 lsb:mysql_410 > primitive p_mysql_411 lsb:mysql_411 > primitive p_mysql_412 lsb:mysql_412 > primitive p_mysql_413 lsb:mysql_413 > primitive p_mysql_414 lsb:mysql_414 > primitive p_mysql_415 lsb:mysql_415 > primitive p_mysql_416 lsb:mysql_416 > primitive p_mysql_417 lsb:mysql_417 > primitive p_mysql_418 lsb:mysql_418 > primitive p_mysql_419 lsb:mysql_419 > primitive p_mysql_420 lsb:mysql_420 > primitive p_mysql_421 lsb:mysql_421 > primitive p_mysql_422 lsb:mysql_422 > primitive p_mysql_423 lsb:mysql_423 > primitive p_mysql_424 lsb:mysql_424 > primitive p_mysql_425 lsb:mysql_425 > primitive p_mysql_426 lsb:mysql_426 > primitive p_mysql_427 lsb:mysql_427 > primitive p_mysql_428 lsb:mysql_428 > primitive p_mysql_429 lsb:mysql_429 > primitive p_mysql_430 lsb:mysql_430 > primitive p_mysql_431 lsb:mysql_431 > primitive p_mysql_432 lsb:mysql_432 > primitive p_mysql_433 lsb:mysql_433 > primitive p_mysql_434 lsb:mysql_434 > primitive p_mysql_435 lsb:mysql_435 > primitive p_mysql_436 lsb:mysql_436 > primitive p_mysql_437 lsb:mysql_437 > primitive p_mysql_438 lsb:mysql_438 > primitive p_mysql_439 lsb:mysql_439 > primitive p_mysql_440 lsb:mysql_440 > primitive p_mysql_441 lsb:mysql_441 > primitive p_mysql_442 lsb:mysql_442 > primitive p_mysql_443 lsb:mysql_443 > primitive p_mysql_444 lsb:mysql_444 > primitive p_mysql_445 lsb:mysql_445 > primitive p_mysql_446 lsb:mysql_446 > primitive p_mysql_447 lsb:mysql_447 > primitive p_mysql_448 lsb:mysql_448 > primitive p_mysql_449 lsb:mysql_449 > primitive p_mysql_450 lsb:mysql_450 > primitive p_mysql_451 lsb:mysql_451 > primitive p_mysql_452 lsb:mysql_452 > primitive p_mysql_453 lsb:mysql_453 > primitive p_mysql_454 lsb:mysql_454 > primitive p_mysql_455 lsb:mysql_455 > primitive p_mysql_456 lsb:mysql_456 > primitive p_mysql_457 lsb:mysql_457 > primitive p_mysql_458 lsb:mysql_458 > primitive p_mysql_459 lsb:mysql_459 > primitive p_mysql_460 lsb:mysql_460 > primitive p_vip_clust06 ocf:heartbeat:IPaddr2 \ > params ip="192.168.10.208" cidr_netmask="32" \ > op monitor interval="15s" > primitive p_vip_clust07 ocf:heartbeat:IPaddr2 \ > params ip="192.168.10.209" cidr_netmask="32" \ > op monitor interval="15s" > group g_clust06 p_fs_clust06 p_vip_clust06 p_mysql_371 p_mysql_372 > p_mysql_373 p_mysql_374 p_mysql_375 p_mysql_376 p_mysql_047 p_mysql_100 > p_mysql_379 p_mysql_377 p_mysql_378 p_mysql_380 p_mysql_381 p_mysql_382 > p_mysql_383 p_mysql_384 p_mysql_385 p_mysql_386 p_mysql_387 p_mysql_002 > p_mysql_035 p_mysql_049 p_mysql_097 p_mysql_024 p_mysql_077 p_mysql_084 > p_mysql_113 p_mysql_116 p_mysql_388 p_mysql_389 p_mysql_390 p_mysql_391 > p_mysql_392 p_mysql_393 p_mysql_394 p_mysql_395 p_mysql_396 p_mysql_397 > p_mysql_398 p_mysql_399 p_mysql_400 p_mysql_401 p_mysql_402 p_mysql_403 > p_mysql_404 p_mysql_405 p_mysql_104 p_mysql_406 p_mysql_407 p_mysql_408 > p_mysql_409 p_mysql_410 p_mysql_017 p_mysql_089 p_mysql_090 p_mysql_109 > p_mysql_411 p_mysql_412 p_mysql_413 p_mysql_414 p_mysql_415 p_mysql_416 > p_mysql_417 p_mysql_020 p_mysql_076 p_mysql_101 p_mysql_051 \ > meta target-role="Started" > group g_clust07 p_fs_clust07 p_vip_clust07 p_mysql_418 p_mysql_419 > p_mysql_420 p_mysql_421 p_mysql_422 p_mysql_423 p_mysql_424 p_mysql_425 > p_mysql_426 p_mysql_427 p_mysql_056 p_mysql_133 p_mysql_428 p_mysql_429 > p_mysql_430 p_mysql_431 p_mysql_432 p_mysql_433 p_mysql_434 p_mysql_435 > p_mysql_436 p_mysql_437 p_mysql_438 p_mysql_439 p_mysql_440 p_mysql_441 > p_mysql_442 p_mysql_443 p_mysql_444 p_mysql_445 p_mysql_446 p_mysql_447 > p_mysql_448 p_mysql_449 p_mysql_450 p_mysql_019 p_mysql_120 p_mysql_451 > p_mysql_452 p_mysql_453 p_mysql_454 p_mysql_455 p_mysql_456 p_mysql_457 > p_mysql_458 p_mysql_459 p_mysql_460 > ms ms_drbd0 p_drbd0 \ > meta master-max="1" master-node-max="1" clone-max="2" > clone-node-max="1" notify="true" is-managed="true" > ms ms_drbd1 p_drbd1 \ > meta master-max="1" master-node-max="1" clone-max="2" > clone-node-max="1" notify="true" is-managed="true" > location cli-prefer-g_clust06 g_clust06 \ > rule $id="cli-prefer-rule-g_clust06" inf: #uname eq ha08a.mycharts.md > location loc1 g_clust06 50: ha08a.mycharts.md > location loc3 g_clust07 50: ha08b.mycharts.md > location loc_ms_drbd0 ms_drbd0 \ > rule $id="loc_ms_drbd0-rule" -inf: #uname ne ha08a.mycharts.md and > #uname ne ha08c.mycharts.md > location loc_ms_drbd1 ms_drbd1 \ > rule $id="loc_ms_drbd1-rule" -inf: #uname ne ha08b.mycharts.md and > #uname ne ha08c.mycharts.md > colocation c_clust06 inf: g_clust06 ms_drbd0:Master > colocation c_clust07 inf: g_clust07 ms_drbd1:Master > order o1 inf: ms_drbd0:promote g_clust06:start > order o2 inf: ms_drbd1:promote g_clust07:start > property $id="cib-bootstrap-options" \ > dc-version="1.1.2-f059ec7ced7a86f18e5490b67ebf4a0b963bccfe" \ > cluster-infrastructure="openais" \ > expected-quorum-votes="3" \ > stonith-enabled="false" \ > no-quorum-policy="ignore" \ > symmetric-cluster="true" \ > last-lrm-refresh="1320604660" \ > maintenance-mode="false" > rsc_defaults $id="rsc-options" \ > resource-stickiness="400" > > > -- > Eric Robinson > > > > > > > > Disclaimer - May 7, 2012 > This email and any files transmitted with it are confidential and intended > solely for linux-ha@lists.linux-ha.org. If you are not the named addressee > you should not disseminate, distribute, copy or alter this email. Any views > or opinions presented in this email are solely those of the author and might > not represent those of Physicians' Managed Care or Physician Select > Management. Warning: Although Physicians' Managed Care or Physician Select > Management has taken reasonable precautions to ensure no viruses are present > in this email, the company cannot accept responsibility for any loss or > damage arising from the use of this email or attachments. > This disclaimer was added by Policy Patrol: http://www.policypatrol.com/ > _______________________________________________ > Linux-HA mailing list > Linux-HA@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems _______________________________________________ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
