Re: [ClusterLabs] Live migration not working on shutdown

2016-11-08 Thread Ken Gaillot 
On 11/04/2016 05:51 AM, IT Nerb GmbH wrote:
> Zitat von Klaus Wenninger :
> 
>> On 11/02/2016 06:32 PM, Ken Gaillot wrote:
>>> On 10/26/2016 06:12 AM, Rainer Nerb wrote:
 Hello all,

 we're currently testing a 2-node-cluster with 2 vms and live migration
 on CentOS 7.2 and Pacemaker 1.1.13-10 with disks on iSCSI-targets and
 migration via ssh-method.

 Live migration works, if we issue "pcs resource move ...", "pcs cluster
 standby", "pcs cluster stop" and even "systemctl rescue".
 The latter only worked, after adding the following additional
 dependencies to pacemaker.service and leaving the management of those
 services to systemd:

   * After/Requires=systemd-machined.service
   * After/Requires=systemd-machine-id-commit.service
   * After/Requires=remote-fs.target
   * After/Requires=libvirtd.service
   * After/Requires=iscsi.service
   * After/Requires=iscsid.service
   * After/Requires=sshd.service
>>> This makes sense when clustered resources depend on services that aren't
>>> themselves managed by the cluster. It's dependent on your situation, so
>>> it's not something that pacemaker can solve generically.
> First approach was to use systemd-resources as there are no ocf:
> resource-agents for iSCSI-Initiators or libvirtd in our distribution.
> But then migration failed even on "systemctl rescue".
>>>
>>> You may already be aware, but the easiest way to add such requirements
>>> is to put them in a systemd unit override, e.g.
>>> /etc/systemd/system/pacemaker.service.d/dependencies.conf.
> Yes, that's how we implemented the additional dependencies.
>>>
 When shutting down or rebooting migration fails and not even the
 regular
 shutdown of the vms succeeds. Systemd seems to tear down the vms by
 terminating something they depend on.

 Is this a known issue? Did we miss any further dependencies?
>>> There was a shutdown issue when using systemd-class cluster resources
>>> (systemd: instead of ocf:), but I believe that was fixed in the package
>>> you're using, and it's probably not relevant here anyway.
>> Speaking of
>> https://github.com/ClusterLabs/pacemaker/pull/887/commits/6aae8542abedc755b90c8c49aa5c429718fd12f1?
>>
>>
>> It shouldn't be in Centos 7.2 but I agree unless there are no
>> systemd-resources involved it wouldn't matter.
>>
>>>
>>> It does sound like there's another dependency, but I don't know what.
>>>
>>> What log messages do you see on the failure?
> See attached log files.

The line that stands out to me is:

Nov  4 11:11:39 kvm02 systemd: Stopping Virtual Machine qemu-2-samba2.

Systemd is stopping the VM before pacemaker is able to migrate it. I'm
guessing that is due to this line in the libvirt unit:

Before=libvirt-guests.service

It appears systemd feels free to do that part in parallel, even though
libvirt itself has to wait until pacemaker finishes stopping. Try adding
libvirt-guests to your pacemaker override.

>>>
 Tia
 Rainer


 

 IT Nerb GmbH
 Lessingstraße 8
 85098 Großmehring

 Telefon : +49 700 ITNERBGMBH
 Telefax : +49 8407 939 284
 email : i...@it-nerb.de
 Internet : www.it-nerb.de 
 Geschäftsführer:Rainer Nerb
 Handelsregister:HRB 2592
 HR-Gericht:Ingolstadt

 

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Live migration not working on shutdown

2016-11-05 Thread IT Nerb GmbH 

Zitat von Klaus Wenninger :


On 11/02/2016 06:32 PM, Ken Gaillot wrote:

On 10/26/2016 06:12 AM, Rainer Nerb wrote:

Hello all,

we're currently testing a 2-node-cluster with 2 vms and live migration
on CentOS 7.2 and Pacemaker 1.1.13-10 with disks on iSCSI-targets and
migration via ssh-method.

Live migration works, if we issue "pcs resource move ...", "pcs cluster
standby", "pcs cluster stop" and even "systemctl rescue".
The latter only worked, after adding the following additional
dependencies to pacemaker.service and leaving the management of those
services to systemd:

  * After/Requires=systemd-machined.service
  * After/Requires=systemd-machine-id-commit.service
  * After/Requires=remote-fs.target
  * After/Requires=libvirtd.service
  * After/Requires=iscsi.service
  * After/Requires=iscsid.service
  * After/Requires=sshd.service

This makes sense when clustered resources depend on services that aren't
themselves managed by the cluster. It's dependent on your situation, so
it's not something that pacemaker can solve generically.
First approach was to use systemd-resources as there are no ocf:  
resource-agents for iSCSI-Initiators or libvirtd in our distribution.  
But then migration failed even on "systemctl rescue".


You may already be aware, but the easiest way to add such requirements
is to put them in a systemd unit override, e.g.
/etc/systemd/system/pacemaker.service.d/dependencies.conf.

Yes, that's how we implemented the additional dependencies.



When shutting down or rebooting migration fails and not even the regular
shutdown of the vms succeeds. Systemd seems to tear down the vms by
terminating something they depend on.

Is this a known issue? Did we miss any further dependencies?

There was a shutdown issue when using systemd-class cluster resources
(systemd: instead of ocf:), but I believe that was fixed in the package
you're using, and it's probably not relevant here anyway.

Speaking of
https://github.com/ClusterLabs/pacemaker/pull/887/commits/6aae8542abedc755b90c8c49aa5c429718fd12f1?

It shouldn't be in Centos 7.2 but I agree unless there are no
systemd-resources involved it wouldn't matter.



It does sound like there's another dependency, but I don't know what.

What log messages do you see on the failure?

See attached log files.



Tia
Rainer



IT Nerb GmbH
Lessingstraße 8
85098 Großmehring

Telefon :   +49 700 ITNERBGMBH
Telefax :   +49 8407 939 284
email   :   i...@it-nerb.de
Internet:   www.it-nerb.de 
Geschäftsführer :   Rainer Nerb
Handelsregister :   HRB 2592
HR-Gericht  :   Ingolstadt



___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org





---
IT Nerb GmbH
Lessingstraße 8
85098 Großmehring

Telefon: +49 700 ITNERBGMBH
Telefax: +49 8407 939 284
email  : i...@it-nerb.de
Internet   : www.it-nerb.de

Geschäftsführer: Rainer Nerb
Handelsregister: HRB 2592
HR-Gericht : Ingolstadt

Nov 04 11:11:39 [1937] kvm01.test-cluster1   crmd: info: 
crm_update_peer_expected:  handle_request: Node kvm02.test-cluster1[2] - 
expected state is now down (was member)
Nov 04 11:11:39 [1937] kvm01.test-cluster1   crmd: info: 
handle_shutdown_request:   Creating shutdown request for 
kvm02.test-cluster1 (state=S_IDLE)
Nov 04 11:11:39 [1935] kvm01.test-cluster1  attrd: info: 
attrd_peer_update: Setting shutdown[kvm02.test-cluster1]: 0 -> 1478254299 
from kvm01.test-cluster1
Nov 04 11:11:39 [1935] kvm01.test-cluster1  attrd: info: 
write_attribute:   Sent update 14 with 2 changes for shutdown, id=, 
set=(null)
Nov 04 11:11:39 [1929] kvm01.test-cluster1cib: info: 
cib_process_request:   Forwarding cib_modify operation for section status to 
master (origin=local/attrd/14)
Nov 04 11:11:39 [1929] kvm01.test-cluster1cib: info: 
cib_perform_op:Diff: --- 0.23.48 2
Nov 04 11:11:39 [1929] kvm01.test-cluster1cib: info: 
cib_perform_op:Diff: +++ 0.23.49 (null)
Nov 04 11:11:39 [1929] kvm01.test-cluster1cib: info: 
cib_perform_op:+  /cib:  @num_updates=49
Nov 0

Re: [ClusterLabs] Live migration not working on shutdown

2016-11-03 Thread Klaus Wenninger 
On 11/02/2016 06:32 PM, Ken Gaillot wrote:
> On 10/26/2016 06:12 AM, Rainer Nerb wrote:
>> Hello all,
>>
>> we're currently testing a 2-node-cluster with 2 vms and live migration
>> on CentOS 7.2 and Pacemaker 1.1.13-10 with disks on iSCSI-targets and
>> migration via ssh-method.
>>
>> Live migration works, if we issue "pcs resource move ...", "pcs cluster
>> standby", "pcs cluster stop" and even "systemctl rescue".
>> The latter only worked, after adding the following additional
>> dependencies to pacemaker.service and leaving the management of those
>> services to systemd:
>>
>>   * After/Requires=systemd-machined.service
>>   * After/Requires=systemd-machine-id-commit.service
>>   * After/Requires=remote-fs.target
>>   * After/Requires=libvirtd.service
>>   * After/Requires=iscsi.service
>>   * After/Requires=iscsid.service
>>   * After/Requires=sshd.service
> This makes sense when clustered resources depend on services that aren't
> themselves managed by the cluster. It's dependent on your situation, so
> it's not something that pacemaker can solve generically.
>
> You may already be aware, but the easiest way to add such requirements
> is to put them in a systemd unit override, e.g.
> /etc/systemd/system/pacemaker.service.d/dependencies.conf.
>
>> When shutting down or rebooting migration fails and not even the regular
>> shutdown of the vms succeeds. Systemd seems to tear down the vms by
>> terminating something they depend on.
>>
>> Is this a known issue? Did we miss any further dependencies?
> There was a shutdown issue when using systemd-class cluster resources
> (systemd: instead of ocf:), but I believe that was fixed in the package
> you're using, and it's probably not relevant here anyway.
Speaking of
https://github.com/ClusterLabs/pacemaker/pull/887/commits/6aae8542abedc755b90c8c49aa5c429718fd12f1?

It shouldn't be in Centos 7.2 but I agree unless there are no
systemd-resources involved it wouldn't matter.

>
> It does sound like there's another dependency, but I don't know what.
>
> What log messages do you see on the failure?
>
>> Tia
>> Rainer
>>
>>
>> 
>> IT Nerb GmbH
>> Lessingstraße 8
>> 85098 Großmehring
>>
>> Telefon  :   +49 700 ITNERBGMBH
>> Telefax  :   +49 8407 939 284
>> email:   i...@it-nerb.de
>> Internet :   www.it-nerb.de 
>> Geschäftsführer  :   Rainer Nerb
>> Handelsregister  :   HRB 2592
>> HR-Gericht   :   Ingolstadt
>>
>> 
> ___
> Users mailing list: Users@clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org



___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Live migration not working on shutdown

2016-11-02 Thread Ken Gaillot 
On 10/26/2016 06:12 AM, Rainer Nerb wrote:
> Hello all,
> 
> we're currently testing a 2-node-cluster with 2 vms and live migration
> on CentOS 7.2 and Pacemaker 1.1.13-10 with disks on iSCSI-targets and
> migration via ssh-method.
> 
> Live migration works, if we issue "pcs resource move ...", "pcs cluster
> standby", "pcs cluster stop" and even "systemctl rescue".
> The latter only worked, after adding the following additional
> dependencies to pacemaker.service and leaving the management of those
> services to systemd:
> 
>   * After/Requires=systemd-machined.service
>   * After/Requires=systemd-machine-id-commit.service
>   * After/Requires=remote-fs.target
>   * After/Requires=libvirtd.service
>   * After/Requires=iscsi.service
>   * After/Requires=iscsid.service
>   * After/Requires=sshd.service

This makes sense when clustered resources depend on services that aren't
themselves managed by the cluster. It's dependent on your situation, so
it's not something that pacemaker can solve generically.

You may already be aware, but the easiest way to add such requirements
is to put them in a systemd unit override, e.g.
/etc/systemd/system/pacemaker.service.d/dependencies.conf.

> When shutting down or rebooting migration fails and not even the regular
> shutdown of the vms succeeds. Systemd seems to tear down the vms by
> terminating something they depend on.
> 
> Is this a known issue? Did we miss any further dependencies?

There was a shutdown issue when using systemd-class cluster resources
(systemd: instead of ocf:), but I believe that was fixed in the package
you're using, and it's probably not relevant here anyway.

It does sound like there's another dependency, but I don't know what.

What log messages do you see on the failure?

> 
> Tia
> Rainer
> 
> 
> 
> IT Nerb GmbH
> Lessingstraße 8
> 85098 Großmehring
> 
> Telefon   :   +49 700 ITNERBGMBH
> Telefax   :   +49 8407 939 284
> email :   i...@it-nerb.de
> Internet  :   www.it-nerb.de 
> Geschäftsführer   :   Rainer Nerb
> Handelsregister   :   HRB 2592
> HR-Gericht:   Ingolstadt
> 
> 

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org