Re: [ClusterLabs] Live migrate a VM in a cluster group

2018-05-28 Thread Jason Gauthier 
On Mon, May 28, 2018 at 12:03 AM, Andrei Borzenkov  wrote:
> 28.05.2018 05:50, Jason Gauthier пишет:
>> Greetings,
>>
>>  I've set up a cluster intended for VMs.  I created a VM, and have
>> been pretty pleased with migrating it back and forth between the two
>> nodes.  Now, I am also using the VNC console, which requires listening
>> on port 59xx.  Of course, once the machine is migrated the IP to
>> access the VNC console is different.
>> So, I thought I would be clever and create a cluster IP address.  I
>> did, and as a stand alone resource it migrates between nodes
>> perfectly.  I then put these two primitives into a group.
>>
>> When I try to migrate the group nothing happens.
>> There aren't any cluster errors, and the logs do not give me any kind
>> of indication of error either.
>>
>
> "migrate" is ambiguous here - it may mean moving resource or
> live-migrating VM. Show exact command(s) you use and logs related to
> theses commands.
>
Sure, okay.  The command I am using is this:
# crm resource migrate g_Calibre alpha

The logs from the current owner looks like this:

May 28 07:42:32 [4333] betacib: info: cib_process_request:
 Forwarding cib_delete operation for section constraints to all
(origin=local/crm_resource/3)
May 28 07:42:32 [4333] betacib: info: cib_perform_op:
 Diff: --- 1.189.27 2
May 28 07:42:32 [4333] betacib: info: cib_perform_op:
 Diff: +++ 1.190.0 f3b87f88cebf5470aed9f061e5f564c4
May 28 07:42:32 [4333] betacib: info: cib_perform_op:
 -- /cib/configuration/constraints/rsc_location[@id='cli-prefer-g_Calibre']
May 28 07:42:32 [4333] betacib: info: cib_perform_op:
 +  /cib:  @epoch=190, @num_updates=0
May 28 07:42:32 [4333] betacib: info: cib_process_request:
 Completed cib_delete operation for section constraints: OK (rc=0,
origin=beta/crm_resource/3, version=1.190.0)
May 28 07:42:32 [4334] beta stonith-ng: info:
update_cib_stonith_devices_v2:Updating device list from the
cib: delete rsc_location[@id='cli-prefer-g_Calibre']
May 28 07:42:32 [4334] beta stonith-ng: info: cib_devices_update:
 Updating devices to version 1.190.0
May 28 07:42:32 [4333] betacib: info: cib_process_request:
 Forwarding cib_modify operation for section constraints to all
(origin=local/crm_resource/4)
May 28 07:42:32 [4334] beta stonith-ng:   notice: unpack_config:
 Watchdog will be used via SBD if fencing is required
May 28 07:42:32 [4333] betacib: info: cib_perform_op:
 Diff: --- 1.190.0 2
May 28 07:42:32 [4333] betacib: info: cib_perform_op:
 Diff: +++ 1.191.0 (null)
May 28 07:42:32 [4333] betacib: info: cib_perform_op:
 +  /cib:  @epoch=191
May 28 07:42:32 [4333] betacib: info: cib_perform_op:
 ++ /cib/configuration/constraints:  
May 28 07:42:32 [4333] betacib: info: cib_process_request:
 Completed cib_modify operation for section constraints: OK (rc=0,
origin=beta/crm_resource/4, version=1.191.0)
May 28 07:42:32 [4333] betacib: info: cib_file_backup:
 Archived previous version as /var/lib/pacemaker/cib/cib-28.raw
May 28 07:42:32 [4333] betacib: info:
cib_file_write_with_digest:   Wrote version 1.190.0 of the CIB to disk
(digest: 3526dbb181e0e21078f2ff0f421c7031)
May 28 07:42:32 [4333] betacib: info:
cib_file_write_with_digest:   Reading cluster configuration file
/var/lib/pacemaker/cib/cib.WVOQBs (digest:
/var/lib/pacemaker/cib/cib.i4m2ZP)
May 28 07:42:32 [4333] betacib: info: cib_file_backup:
 Archived previous version as /var/lib/pacemaker/cib/cib-29.raw
May 28 07:42:32 [4333] betacib: info:
cib_file_write_with_digest:   Wrote version 1.191.0 of the CIB to disk
(digest: 4c39c8efbd77c9e3ee831a20890f611d)
May 28 07:42:32 [4333] betacib: info:
cib_file_write_with_digest:   Reading cluster configuration file
/var/lib/pacemaker/cib/cib.rxLoCv (digest:
/var/lib/pacemaker/cib/cib.6mH61S)
May 28 07:42:37 [4333] betacib: info: cib_process_ping:
 Reporting our current digest to alpha:
820a81c6a738a338a6dee5f02685d464 for 1.191.0 (0x56502e4a4130 0)


The logs from the other node look like this:
May 28 07:41:23 [5082] alphacib: info: cib_process_ping:
 Reporting our current digest to alpha:
193a1c4a09c2515bb160b2430c40079f for 1.189.27 (0x55de0024d9e0 0)
May 28 07:42:32 [5082] alphacib: info: cib_perform_op:
 Diff: --- 1.189.27 2
May 28 07:42:32 [5082] alphacib: info: cib_perform_op:
 Diff: +++ 1.190.0 f3b87f88cebf5470aed9f061e5f564c4
May 28 07:42:32 [5082] alphacib: info: cib_perform_op:
 -- /cib/configuration/constraints/rsc_location[@id='cli-prefer-g_Calibre']
May 28 07:42:32 [5082] alphacib: info: cib_perform_op:
 +  /cib:  @epoch=190, @num_updates=0
May 28 07:42:32 [5082] alphacib: info:
cib_process_request: Completed cib_delete operation for section
constraints: OK (rc=0,

Re: [ClusterLabs] Questions about SBD behavior

2018-05-28 Thread Andrei Borzenkov 
On Mon, May 28, 2018 at 10:47 AM, Klaus Wenninger  wrote:
> On 05/28/2018 09:43 AM, Klaus Wenninger wrote:
>> On 05/26/2018 07:23 AM, Andrei Borzenkov wrote:
>>> 25.05.2018 14:44, Klaus Wenninger пишет:
 On 05/25/2018 12:44 PM, Andrei Borzenkov wrote:
> On Fri, May 25, 2018 at 10:08 AM, Klaus Wenninger  
> wrote:
>> On 05/25/2018 07:31 AM, 井上 和徳 wrote:
>>> Hi,
>>>
>>> I am checking the watchdog function of SBD (without shared 
>>> block-device).
>>> In a two-node cluster, if one cluster is stopped, watchdog is triggered 
>>> on the remaining node.
>>> Is this the designed behavior?
>> SBD without a shared block-device doesn't really make sense on
>> a two-node cluster.
>> The basic idea is - e.g. in a case of a networking problem -
>> that a cluster splits up in a quorate and a non-quorate partition.
>> The quorate partition stays over while SBD guarantees a
>> reliable watchdog-based self-fencing of the non-quorate partition
>> within a defined timeout.
> Does it require no-quorum-policy=suicide or it decides completely
> independently? I.e. would it fire also with no-quorum-policy=ignore?
 Finally it will in any case. But no-quorum-policy decides how
 long this will take. In case of suicide the inquisitor will immediately
 stop tickling the watchdog. In all other cases the pacemaker-servant
 will stop pinging the inquisitor which will makes the servant
 timeout after a default of 4 seconds and then the inquisitor will
 stop tickling the watchdog.
 But that is just relevant if Corosync doesn't have 2-node enabled.
 See the comment below for that case.

>> This idea of course doesn't work with just 2 nodes.
>> Taking quorum info from the 2-node feature of corosync (automatically
>> switching on wait-for-all) doesn't help in this case but instead
>> would lead to split-brain.
> So what you are saying is that SBD ignores quorum information from
> corosync and takes its own decisions based on pure count of nodes. Do
> I understand it correctly?
 Yes, but that is just true for this case where Corosync has 2-node
 enabled.
> In all other cases (might it be clusters with more than 2 nodes
 or clusters with just 2 nodes but without 2-node enabled in
 Corosync) pacemaker-servant takes quorum-info from
 pacemaker, which will probably come directly from Corosync
 nowadays.
 But as said if 2-node is configured with Corosync everything
 is different: The node-counting is then actually done
 by the cluster-servant and this one will stop pinging the
 inquisitor (instead of the pacemaker-servant) if it doesn't
 count more than 1 node.

>>> Is it conditional on having no shared device or it just checks two_node
>>> value? If it always behaves this way, even with real shared device
>>> present, it means sbd is fundamentally incompatible with two_node and it
>>> better be mentioned in documentation.
>> If you are referring to counting the nodes instead of taking
>> quorum-info from pacemaker in case of 2-node configured
>> with corosync, that is universal.
>>
>> And actually the reason why it is there is to be able to use
>> sbd with a single disk on 2-nodes having 2-node enabled.
>>
>> Imagine quorum-info from corosync/pacemaker being used
>> in that case:
>> Image a cluster (node-a & node-b). node-a looses connection
>> to the network and to the shared storage. node-a will still
>> receive positive quorum from corosync as it has seen the other
>> node already (since it is up). This will make it ignore the
>> loss of the disk (survive on pacemaker).
>> node-b is quoruate as well, sees the disk, uses the disk to
>> fence node-a and will after a timeout assume node-a to be
>> down -> split-brain.
> Seeing the disk will prevent the reboot if that is what
> was missing for you.

Yes, this was not exactly clear. Thank you!

>>
 That all said I've just realized that setting 2-node in Corosync
 shouldn't really be dangerous anymore although it doesn't make
 the cluster especially useful either in case of SBD without disk(s).

 Regards,
 Klaus
>> ___
>> Users mailing list: Users@clusterlabs.org
>> https://lists.clusterlabs.org/mailman/listinfo/users
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Questions about SBD behavior

2018-05-28 Thread Klaus Wenninger 
On 05/28/2018 09:43 AM, Klaus Wenninger wrote:
> On 05/26/2018 07:23 AM, Andrei Borzenkov wrote:
>> 25.05.2018 14:44, Klaus Wenninger пишет:
>>> On 05/25/2018 12:44 PM, Andrei Borzenkov wrote:
 On Fri, May 25, 2018 at 10:08 AM, Klaus Wenninger  
 wrote:
> On 05/25/2018 07:31 AM, 井上 和徳 wrote:
>> Hi,
>>
>> I am checking the watchdog function of SBD (without shared block-device).
>> In a two-node cluster, if one cluster is stopped, watchdog is triggered 
>> on the remaining node.
>> Is this the designed behavior?
> SBD without a shared block-device doesn't really make sense on
> a two-node cluster.
> The basic idea is - e.g. in a case of a networking problem -
> that a cluster splits up in a quorate and a non-quorate partition.
> The quorate partition stays over while SBD guarantees a
> reliable watchdog-based self-fencing of the non-quorate partition
> within a defined timeout.
 Does it require no-quorum-policy=suicide or it decides completely
 independently? I.e. would it fire also with no-quorum-policy=ignore?
>>> Finally it will in any case. But no-quorum-policy decides how
>>> long this will take. In case of suicide the inquisitor will immediately
>>> stop tickling the watchdog. In all other cases the pacemaker-servant
>>> will stop pinging the inquisitor which will makes the servant
>>> timeout after a default of 4 seconds and then the inquisitor will
>>> stop tickling the watchdog.
>>> But that is just relevant if Corosync doesn't have 2-node enabled.
>>> See the comment below for that case.
>>>
> This idea of course doesn't work with just 2 nodes.
> Taking quorum info from the 2-node feature of corosync (automatically
> switching on wait-for-all) doesn't help in this case but instead
> would lead to split-brain.
 So what you are saying is that SBD ignores quorum information from
 corosync and takes its own decisions based on pure count of nodes. Do
 I understand it correctly?
>>> Yes, but that is just true for this case where Corosync has 2-node
>>> enabled.
 In all other cases (might it be clusters with more than 2 nodes
>>> or clusters with just 2 nodes but without 2-node enabled in
>>> Corosync) pacemaker-servant takes quorum-info from
>>> pacemaker, which will probably come directly from Corosync
>>> nowadays.
>>> But as said if 2-node is configured with Corosync everything
>>> is different: The node-counting is then actually done
>>> by the cluster-servant and this one will stop pinging the
>>> inquisitor (instead of the pacemaker-servant) if it doesn't
>>> count more than 1 node.
>>>
>> Is it conditional on having no shared device or it just checks two_node
>> value? If it always behaves this way, even with real shared device
>> present, it means sbd is fundamentally incompatible with two_node and it
>> better be mentioned in documentation.
> If you are referring to counting the nodes instead of taking
> quorum-info from pacemaker in case of 2-node configured
> with corosync, that is universal.
>
> And actually the reason why it is there is to be able to use
> sbd with a single disk on 2-nodes having 2-node enabled.
>
> Imagine quorum-info from corosync/pacemaker being used
> in that case:
> Image a cluster (node-a & node-b). node-a looses connection
> to the network and to the shared storage. node-a will still
> receive positive quorum from corosync as it has seen the other
> node already (since it is up). This will make it ignore the
> loss of the disk (survive on pacemaker).
> node-b is quoruate as well, sees the disk, uses the disk to
> fence node-a and will after a timeout assume node-a to be
> down -> split-brain.
Seeing the disk will prevent the reboot if that is what
was missing for you.
>
>>> That all said I've just realized that setting 2-node in Corosync
>>> shouldn't really be dangerous anymore although it doesn't make
>>> the cluster especially useful either in case of SBD without disk(s).
>>>
>>> Regards,
>>> Klaus
> ___
> Users mailing list: Users@clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Questions about SBD behavior

2018-05-28 Thread Klaus Wenninger 
On 05/26/2018 07:23 AM, Andrei Borzenkov wrote:
> 25.05.2018 14:44, Klaus Wenninger пишет:
>> On 05/25/2018 12:44 PM, Andrei Borzenkov wrote:
>>> On Fri, May 25, 2018 at 10:08 AM, Klaus Wenninger  
>>> wrote:
 On 05/25/2018 07:31 AM, 井上 和徳 wrote:
> Hi,
>
> I am checking the watchdog function of SBD (without shared block-device).
> In a two-node cluster, if one cluster is stopped, watchdog is triggered 
> on the remaining node.
> Is this the designed behavior?
 SBD without a shared block-device doesn't really make sense on
 a two-node cluster.
 The basic idea is - e.g. in a case of a networking problem -
 that a cluster splits up in a quorate and a non-quorate partition.
 The quorate partition stays over while SBD guarantees a
 reliable watchdog-based self-fencing of the non-quorate partition
 within a defined timeout.
>>> Does it require no-quorum-policy=suicide or it decides completely
>>> independently? I.e. would it fire also with no-quorum-policy=ignore?
>> Finally it will in any case. But no-quorum-policy decides how
>> long this will take. In case of suicide the inquisitor will immediately
>> stop tickling the watchdog. In all other cases the pacemaker-servant
>> will stop pinging the inquisitor which will makes the servant
>> timeout after a default of 4 seconds and then the inquisitor will
>> stop tickling the watchdog.
>> But that is just relevant if Corosync doesn't have 2-node enabled.
>> See the comment below for that case.
>>
 This idea of course doesn't work with just 2 nodes.
 Taking quorum info from the 2-node feature of corosync (automatically
 switching on wait-for-all) doesn't help in this case but instead
 would lead to split-brain.
>>> So what you are saying is that SBD ignores quorum information from
>>> corosync and takes its own decisions based on pure count of nodes. Do
>>> I understand it correctly?
>> Yes, but that is just true for this case where Corosync has 2-node
>> enabled.
>>> In all other cases (might it be clusters with more than 2 nodes
>> or clusters with just 2 nodes but without 2-node enabled in
>> Corosync) pacemaker-servant takes quorum-info from
>> pacemaker, which will probably come directly from Corosync
>> nowadays.
>> But as said if 2-node is configured with Corosync everything
>> is different: The node-counting is then actually done
>> by the cluster-servant and this one will stop pinging the
>> inquisitor (instead of the pacemaker-servant) if it doesn't
>> count more than 1 node.
>>
> Is it conditional on having no shared device or it just checks two_node
> value? If it always behaves this way, even with real shared device
> present, it means sbd is fundamentally incompatible with two_node and it
> better be mentioned in documentation.

If you are referring to counting the nodes instead of taking
quorum-info from pacemaker in case of 2-node configured
with corosync, that is universal.

And actually the reason why it is there is to be able to use
sbd with a single disk on 2-nodes having 2-node enabled.

Imagine quorum-info from corosync/pacemaker being used
in that case:
Image a cluster (node-a & node-b). node-a looses connection
to the network and to the shared storage. node-a will still
receive positive quorum from corosync as it has seen the other
node already (since it is up). This will make it ignore the
loss of the disk (survive on pacemaker).
node-b is quoruate as well, sees the disk, uses the disk to
fence node-a and will after a timeout assume node-a to be
down -> split-brain.

>
>> That all said I've just realized that setting 2-node in Corosync
>> shouldn't really be dangerous anymore although it doesn't make
>> the cluster especially useful either in case of SBD without disk(s).
>>
>> Regards,
>> Klaus

___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org