Re: [ClusterLabs] Live migrate a VM in a cluster group
On Tue, May 29, 2018 at 11:46 AM, Ken Gaillot wrote: > On Tue, 2018-05-29 at 10:14 -0500, Ken Gaillot wrote: >> On Sun, 2018-05-27 at 22:50 -0400, Jason Gauthier wrote: >> > Greetings, >> > >> > I've set up a cluster intended for VMs. I created a VM, and have >> > been pretty pleased with migrating it back and forth between the >> > two >> > nodes. Now, I am also using the VNC console, which requires >> > listening >> > on port 59xx. Of course, once the machine is migrated the IP to >> > access the VNC console is different. >> > So, I thought I would be clever and create a cluster IP address. I >> > did, and as a stand alone resource it migrates between nodes >> > perfectly. I then put these two primitives into a group. >> > >> > When I try to migrate the group nothing happens. >> > There aren't any cluster errors, and the logs do not give me any >> > kind >> > of indication of error either. >> > >> > I'm wondering if this is even something I should expect to work. I >> > would certainly like it to. >> > Here's the relevant config: >> > >> > primitive p_Calibre VirtualDomain \ >> > params config="/vms/Calibre/Calibre.xml" >> > hypervisor="qemu:///system" migration_transport=ssh \ >> > meta allow-migrate=true \ >> > op start timeout=120s interval=0 \ >> > op stop timeout=120s interval=0 \ >> > op monitor timeout=30 interval=10 depth=0 \ >> > utilization cpu=2 hv_memory=8192 >> > primitive p_CalibreVNC IPaddr2 \ >> > params ip=192.168.10.10 cidr_netmask=24 nic=br10 \ >> > op monitor interval=10s >> > >> > group g_Calibre p_CalibreVNC p_Calibre \ >> > meta target-role=Started >> > >> > location cli-prefer-g_Calibre g_Calibre role=Started inf: alpha >> > location cli-prefer-p_Calibre p_Calibre role=Started inf: beta >> >> The group prefers alpha, but its p_Calibre member prefers beta. >> You're >> confusing the cluster :) >> >> Any constraints starting with "cli-" were added by command-line tools >> doing move/ban/etc. They stay in effect until they are manually >> removed >> (the same tool will generally have a "clear" option). > > Oh, and IPaddr2 can't live-migrate, so putting it in a group (or > colocation+ordering constraint) with p_Calibre will make the entire > group unable to live-migrate. > > What exact relationships do you really need? > > Does the IP need to be up before the VM starts in order for VNC to > work? If so, can it even work with a live migration (the IP can't be up > on both nodes at once)? > > If the IP doesn't need to be up first, you could simply colocate the IP > with the VM, without any ordering constraints. Whenever the VM moves, > the IP will follow; the VM would live-migrate, and the IP would > stop+start. > > If the IP does need to be up first, I'm thinking you could create a > group with p_CalibreVNC and an ocf:pacemaker:attribute resource (which > sets a node attribute when it is running), and then use a rule-based > constraint to colocate+order p_Calibre relative to the node attribute. > I'd make the colocation non-infinite and the order optional, so the VM > can stay up even if the IP fails or begins to move. > > You would trigger a migration by moving the group. It would proceed > like this: > - The IP, attribute resource, node attribute, and VM start out on node > 1 > - The attribute resource stops on node 1, removing the node attribute > from node 1 (this is where making the constraints optional comes in > handy) > - The IP resource stops on node 1 > - The IP resource starts on node 2 > - The attribute resource starts on node 2, adding the node attribute to > node 2 > - The VM live-migrates to node 2 > > That does leave a window where the IP and VM are on different nodes, > but that sounds acceptable in your setup in order to preserve live > migration. > You and Andrei were correct. The contraints were causing an issue. I am able to move them as a unit, but as you said.. The results are unpleasant. To achieve what I really want, I will need to execute the plan you mentioned above. I am not sure how to do that, but your steps are perfect. I do not care about the IP going up and down since it's just console management. But I do want the machine to stay up and be live migrated. So, I will look into the process you mentioned. It does seem that the IP needs to be up first, or the listener can't bind, and qemu fails. Thanks for the advice. I have some more learning to do. >> > location cli-prefer-p_CalibreVNC p_CalibreVNC role=Started inf: >> > beta >> > >> > Any help is appreciated. > -- > Ken Gaillot > ___ > Users mailing list: Users@clusterlabs.org > https://lists.clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org ___ Users mailing list: Users@clusterlabs.org
Re: [ClusterLabs] Live migrate a VM in a cluster group
28.05.2018 14:44, Jason Gauthier пишет: > On Mon, May 28, 2018 at 12:03 AM, Andrei Borzenkov > wrote: >> 28.05.2018 05:50, Jason Gauthier пишет: >>> Greetings, >>> >>> I've set up a cluster intended for VMs. I created a VM, and have >>> been pretty pleased with migrating it back and forth between the two >>> nodes. Now, I am also using the VNC console, which requires listening >>> on port 59xx. Of course, once the machine is migrated the IP to >>> access the VNC console is different. >>> So, I thought I would be clever and create a cluster IP address. I >>> did, and as a stand alone resource it migrates between nodes >>> perfectly. I then put these two primitives into a group. >>> >>> When I try to migrate the group nothing happens. >>> There aren't any cluster errors, and the logs do not give me any kind >>> of indication of error either. >>> >> >> "migrate" is ambiguous here - it may mean moving resource or >> live-migrating VM. Show exact command(s) you use and logs related to >> theses commands. >> > Sure, okay. The command I am using is this: > # crm resource migrate g_Calibre alpha > OK, that is "resource move" which simply creates constraint and relies on policy engine to compute new resource placement. It does *not* remove any constraint created previously by "crm resource move" with another node as argument. ... > May 28 07:42:32 [5086] alphapengine: info: > determine_op_status: Operation monitor found resource p_CalibreVNC > active on beta > May 28 07:42:32 [5086] alphapengine: info: > determine_op_status: Operation monitor found resource p_Calibre active > on beta Currently both resources are active on beta. ... > May 28 07:42:32 [5086] alphapengine: info: LogActions: Leave > p_CalibreVNC(Started beta) > May 28 07:42:32 [5086] alphapengine: info: LogActions: Leave > p_Calibre (Started beta) And policy engine decides to leave it there. ... >>> >>> group g_Calibre p_CalibreVNC p_Calibre \ >>> meta target-role=Started >>> >>> location cli-prefer-g_Calibre g_Calibre role=Started inf: alpha >>> location cli-prefer-p_Calibre p_Calibre role=Started inf: beta >>> location cli-prefer-p_CalibreVNC p_CalibreVNC role=Started inf: beta >>> You have mutually contradictory constraints. Apparently the one for "beta" is result of previous "crm resource move p_Calibre(VNC) beta" invocation. At this point pacemaker is actually free to select any node. I suppose there are some implementation defined behavior but fundamentally it is garbage in, garbage out. It decided to leave resources where they are. After any "crm resource move|migrate|ban" you *MUST* remove constraints using e.g. "crm resource clear". You can set constraint lifetime so it is done automatically. ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Live migrate a VM in a cluster group
On Tue, 2018-05-29 at 10:14 -0500, Ken Gaillot wrote: > On Sun, 2018-05-27 at 22:50 -0400, Jason Gauthier wrote: > > Greetings, > > > > I've set up a cluster intended for VMs. I created a VM, and have > > been pretty pleased with migrating it back and forth between the > > two > > nodes. Now, I am also using the VNC console, which requires > > listening > > on port 59xx. Of course, once the machine is migrated the IP to > > access the VNC console is different. > > So, I thought I would be clever and create a cluster IP address. I > > did, and as a stand alone resource it migrates between nodes > > perfectly. I then put these two primitives into a group. > > > > When I try to migrate the group nothing happens. > > There aren't any cluster errors, and the logs do not give me any > > kind > > of indication of error either. > > > > I'm wondering if this is even something I should expect to work. I > > would certainly like it to. > > Here's the relevant config: > > > > primitive p_Calibre VirtualDomain \ > > params config="/vms/Calibre/Calibre.xml" > > hypervisor="qemu:///system" migration_transport=ssh \ > > meta allow-migrate=true \ > > op start timeout=120s interval=0 \ > > op stop timeout=120s interval=0 \ > > op monitor timeout=30 interval=10 depth=0 \ > > utilization cpu=2 hv_memory=8192 > > primitive p_CalibreVNC IPaddr2 \ > > params ip=192.168.10.10 cidr_netmask=24 nic=br10 \ > > op monitor interval=10s > > > > group g_Calibre p_CalibreVNC p_Calibre \ > > meta target-role=Started > > > > location cli-prefer-g_Calibre g_Calibre role=Started inf: alpha > > location cli-prefer-p_Calibre p_Calibre role=Started inf: beta > > The group prefers alpha, but its p_Calibre member prefers beta. > You're > confusing the cluster :) > > Any constraints starting with "cli-" were added by command-line tools > doing move/ban/etc. They stay in effect until they are manually > removed > (the same tool will generally have a "clear" option). Oh, and IPaddr2 can't live-migrate, so putting it in a group (or colocation+ordering constraint) with p_Calibre will make the entire group unable to live-migrate. What exact relationships do you really need? Does the IP need to be up before the VM starts in order for VNC to work? If so, can it even work with a live migration (the IP can't be up on both nodes at once)? If the IP doesn't need to be up first, you could simply colocate the IP with the VM, without any ordering constraints. Whenever the VM moves, the IP will follow; the VM would live-migrate, and the IP would stop+start. If the IP does need to be up first, I'm thinking you could create a group with p_CalibreVNC and an ocf:pacemaker:attribute resource (which sets a node attribute when it is running), and then use a rule-based constraint to colocate+order p_Calibre relative to the node attribute. I'd make the colocation non-infinite and the order optional, so the VM can stay up even if the IP fails or begins to move. You would trigger a migration by moving the group. It would proceed like this: - The IP, attribute resource, node attribute, and VM start out on node 1 - The attribute resource stops on node 1, removing the node attribute from node 1 (this is where making the constraints optional comes in handy) - The IP resource stops on node 1 - The IP resource starts on node 2 - The attribute resource starts on node 2, adding the node attribute to node 2 - The VM live-migrates to node 2 That does leave a window where the IP and VM are on different nodes, but that sounds acceptable in your setup in order to preserve live migration. > > location cli-prefer-p_CalibreVNC p_CalibreVNC role=Started inf: > > beta > > > > Any help is appreciated. -- Ken Gaillot ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Live migrate a VM in a cluster group
On Sun, 2018-05-27 at 22:50 -0400, Jason Gauthier wrote: > Greetings, > > I've set up a cluster intended for VMs. I created a VM, and have > been pretty pleased with migrating it back and forth between the two > nodes. Now, I am also using the VNC console, which requires > listening > on port 59xx. Of course, once the machine is migrated the IP to > access the VNC console is different. > So, I thought I would be clever and create a cluster IP address. I > did, and as a stand alone resource it migrates between nodes > perfectly. I then put these two primitives into a group. > > When I try to migrate the group nothing happens. > There aren't any cluster errors, and the logs do not give me any kind > of indication of error either. > > I'm wondering if this is even something I should expect to work. I > would certainly like it to. > Here's the relevant config: > > primitive p_Calibre VirtualDomain \ > params config="/vms/Calibre/Calibre.xml" > hypervisor="qemu:///system" migration_transport=ssh \ > meta allow-migrate=true \ > op start timeout=120s interval=0 \ > op stop timeout=120s interval=0 \ > op monitor timeout=30 interval=10 depth=0 \ > utilization cpu=2 hv_memory=8192 > primitive p_CalibreVNC IPaddr2 \ > params ip=192.168.10.10 cidr_netmask=24 nic=br10 \ > op monitor interval=10s > > group g_Calibre p_CalibreVNC p_Calibre \ > meta target-role=Started > > location cli-prefer-g_Calibre g_Calibre role=Started inf: alpha > location cli-prefer-p_Calibre p_Calibre role=Started inf: beta The group prefers alpha, but its p_Calibre member prefers beta. You're confusing the cluster :) Any constraints starting with "cli-" were added by command-line tools doing move/ban/etc. They stay in effect until they are manually removed (the same tool will generally have a "clear" option). > location cli-prefer-p_CalibreVNC p_CalibreVNC role=Started inf: beta > > Any help is appreciated. -- Ken Gaillot ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Live migrate a VM in a cluster group
On Mon, May 28, 2018 at 12:03 AM, Andrei Borzenkov wrote: > 28.05.2018 05:50, Jason Gauthier пишет: >> Greetings, >> >> I've set up a cluster intended for VMs. I created a VM, and have >> been pretty pleased with migrating it back and forth between the two >> nodes. Now, I am also using the VNC console, which requires listening >> on port 59xx. Of course, once the machine is migrated the IP to >> access the VNC console is different. >> So, I thought I would be clever and create a cluster IP address. I >> did, and as a stand alone resource it migrates between nodes >> perfectly. I then put these two primitives into a group. >> >> When I try to migrate the group nothing happens. >> There aren't any cluster errors, and the logs do not give me any kind >> of indication of error either. >> > > "migrate" is ambiguous here - it may mean moving resource or > live-migrating VM. Show exact command(s) you use and logs related to > theses commands. > Sure, okay. The command I am using is this: # crm resource migrate g_Calibre alpha The logs from the current owner looks like this: May 28 07:42:32 [4333] betacib: info: cib_process_request: Forwarding cib_delete operation for section constraints to all (origin=local/crm_resource/3) May 28 07:42:32 [4333] betacib: info: cib_perform_op: Diff: --- 1.189.27 2 May 28 07:42:32 [4333] betacib: info: cib_perform_op: Diff: +++ 1.190.0 f3b87f88cebf5470aed9f061e5f564c4 May 28 07:42:32 [4333] betacib: info: cib_perform_op: -- /cib/configuration/constraints/rsc_location[@id='cli-prefer-g_Calibre'] May 28 07:42:32 [4333] betacib: info: cib_perform_op: + /cib: @epoch=190, @num_updates=0 May 28 07:42:32 [4333] betacib: info: cib_process_request: Completed cib_delete operation for section constraints: OK (rc=0, origin=beta/crm_resource/3, version=1.190.0) May 28 07:42:32 [4334] beta stonith-ng: info: update_cib_stonith_devices_v2:Updating device list from the cib: delete rsc_location[@id='cli-prefer-g_Calibre'] May 28 07:42:32 [4334] beta stonith-ng: info: cib_devices_update: Updating devices to version 1.190.0 May 28 07:42:32 [4333] betacib: info: cib_process_request: Forwarding cib_modify operation for section constraints to all (origin=local/crm_resource/4) May 28 07:42:32 [4334] beta stonith-ng: notice: unpack_config: Watchdog will be used via SBD if fencing is required May 28 07:42:32 [4333] betacib: info: cib_perform_op: Diff: --- 1.190.0 2 May 28 07:42:32 [4333] betacib: info: cib_perform_op: Diff: +++ 1.191.0 (null) May 28 07:42:32 [4333] betacib: info: cib_perform_op: + /cib: @epoch=191 May 28 07:42:32 [4333] betacib: info: cib_perform_op: ++ /cib/configuration/constraints: May 28 07:42:32 [4333] betacib: info: cib_process_request: Completed cib_modify operation for section constraints: OK (rc=0, origin=beta/crm_resource/4, version=1.191.0) May 28 07:42:32 [4333] betacib: info: cib_file_backup: Archived previous version as /var/lib/pacemaker/cib/cib-28.raw May 28 07:42:32 [4333] betacib: info: cib_file_write_with_digest: Wrote version 1.190.0 of the CIB to disk (digest: 3526dbb181e0e21078f2ff0f421c7031) May 28 07:42:32 [4333] betacib: info: cib_file_write_with_digest: Reading cluster configuration file /var/lib/pacemaker/cib/cib.WVOQBs (digest: /var/lib/pacemaker/cib/cib.i4m2ZP) May 28 07:42:32 [4333] betacib: info: cib_file_backup: Archived previous version as /var/lib/pacemaker/cib/cib-29.raw May 28 07:42:32 [4333] betacib: info: cib_file_write_with_digest: Wrote version 1.191.0 of the CIB to disk (digest: 4c39c8efbd77c9e3ee831a20890f611d) May 28 07:42:32 [4333] betacib: info: cib_file_write_with_digest: Reading cluster configuration file /var/lib/pacemaker/cib/cib.rxLoCv (digest: /var/lib/pacemaker/cib/cib.6mH61S) May 28 07:42:37 [4333] betacib: info: cib_process_ping: Reporting our current digest to alpha: 820a81c6a738a338a6dee5f02685d464 for 1.191.0 (0x56502e4a4130 0) The logs from the other node look like this: May 28 07:41:23 [5082] alphacib: info: cib_process_ping: Reporting our current digest to alpha: 193a1c4a09c2515bb160b2430c40079f for 1.189.27 (0x55de0024d9e0 0) May 28 07:42:32 [5082] alphacib: info: cib_perform_op: Diff: --- 1.189.27 2 May 28 07:42:32 [5082] alphacib: info: cib_perform_op: Diff: +++ 1.190.0 f3b87f88cebf5470aed9f061e5f564c4 May 28 07:42:32 [5082] alphacib: info: cib_perform_op: -- /cib/configuration/constraints/rsc_location[@id='cli-prefer-g_Calibre'] May 28 07:42:32 [5082] alphacib: info: cib_perform_op: + /cib: @epoch=190, @num_updates=0 May 28 07:42:32 [5082] alphacib: info: cib_process_request: Completed cib_delete operation for section constraints: OK (rc=0, origin=beta/crm_resource/
Re: [ClusterLabs] Live migrate a VM in a cluster group
28.05.2018 05:50, Jason Gauthier пишет: > Greetings, > > I've set up a cluster intended for VMs. I created a VM, and have > been pretty pleased with migrating it back and forth between the two > nodes. Now, I am also using the VNC console, which requires listening > on port 59xx. Of course, once the machine is migrated the IP to > access the VNC console is different. > So, I thought I would be clever and create a cluster IP address. I > did, and as a stand alone resource it migrates between nodes > perfectly. I then put these two primitives into a group. > > When I try to migrate the group nothing happens. > There aren't any cluster errors, and the logs do not give me any kind > of indication of error either. > "migrate" is ambiguous here - it may mean moving resource or live-migrating VM. Show exact command(s) you use and logs related to theses commands. > I'm wondering if this is even something I should expect to work. I > would certainly like it to. > Here's the relevant config: > > primitive p_Calibre VirtualDomain \ > params config="/vms/Calibre/Calibre.xml" > hypervisor="qemu:///system" migration_transport=ssh \ > meta allow-migrate=true \ > op start timeout=120s interval=0 \ > op stop timeout=120s interval=0 \ > op monitor timeout=30 interval=10 depth=0 \ > utilization cpu=2 hv_memory=8192 > primitive p_CalibreVNC IPaddr2 \ > params ip=192.168.10.10 cidr_netmask=24 nic=br10 \ > op monitor interval=10s > > group g_Calibre p_CalibreVNC p_Calibre \ > meta target-role=Started > > location cli-prefer-g_Calibre g_Calibre role=Started inf: alpha > location cli-prefer-p_Calibre p_Calibre role=Started inf: beta > location cli-prefer-p_CalibreVNC p_CalibreVNC role=Started inf: beta > > Any help is appreciated. > > Jason > ___ > Users mailing list: Users@clusterlabs.org > https://lists.clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] Live migrate a VM in a cluster group
Greetings, I've set up a cluster intended for VMs. I created a VM, and have been pretty pleased with migrating it back and forth between the two nodes. Now, I am also using the VNC console, which requires listening on port 59xx. Of course, once the machine is migrated the IP to access the VNC console is different. So, I thought I would be clever and create a cluster IP address. I did, and as a stand alone resource it migrates between nodes perfectly. I then put these two primitives into a group. When I try to migrate the group nothing happens. There aren't any cluster errors, and the logs do not give me any kind of indication of error either. I'm wondering if this is even something I should expect to work. I would certainly like it to. Here's the relevant config: primitive p_Calibre VirtualDomain \ params config="/vms/Calibre/Calibre.xml" hypervisor="qemu:///system" migration_transport=ssh \ meta allow-migrate=true \ op start timeout=120s interval=0 \ op stop timeout=120s interval=0 \ op monitor timeout=30 interval=10 depth=0 \ utilization cpu=2 hv_memory=8192 primitive p_CalibreVNC IPaddr2 \ params ip=192.168.10.10 cidr_netmask=24 nic=br10 \ op monitor interval=10s group g_Calibre p_CalibreVNC p_Calibre \ meta target-role=Started location cli-prefer-g_Calibre g_Calibre role=Started inf: alpha location cli-prefer-p_Calibre p_Calibre role=Started inf: beta location cli-prefer-p_CalibreVNC p_CalibreVNC role=Started inf: beta Any help is appreciated. Jason ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
