Thanks for the update Ken!


From: Ken Gaillot
Sent: Saturday, 21 October 2017 7:06 AM
To: Cluster Labs - All topics related to open-source clustering welcomed
Subject: Re: [ClusterLabs] crm_resource --wait

I've narrowed down the cause.

When the "standby" transition completes, vm2 has more remaining
utilization capacity than vm1, so the cluster wants to run sv-fencer
there. That should be taken into account in the same transition, but it
isn't, so a second transition is needed to make it happen.

Still investigating a fix. A workaround is to assign some stickiness or
utilization to sv-fencer.

On Wed, 2017-10-11 at 14:01 +1000, Leon Steffens wrote:
> I've attached two files:
> 314 = after standby step
> 315 = after resource update
> 
> On Wed, Oct 11, 2017 at 12:22 AM, Ken Gaillot <kgail...@redhat.com>
> wrote:
> > On Tue, 2017-10-10 at 15:19 +1000, Leon Steffens wrote:
> > > Hi Ken,
> > >
> > > I managed to reproduce this on a simplified version of the
> > cluster,
> > > and on Pacemaker 1.1.15, 1.1.16, as well as 1.1.18-rc1
> > 
> > > The steps to create the cluster are:
> > >
> > > pcs property set stonith-enabled=false
> > > pcs property set placement-strategy=balanced
> > >
> > > pcs node utilization vm1 cpu=100
> > > pcs node utilization vm2 cpu=100
> > > pcs node utilization vm3 cpu=100
> > >
> > > pcs property set maintenance-mode=true
> > >
> > > pcs resource create sv-fencer ocf:pacemaker:Dummy
> > >
> > > pcs resource create sv ocf:pacemaker:Dummy clone notify=false
> > > pcs resource create std ocf:pacemaker:Dummy meta resource-
> > > stickiness=100
> > >
> > > pcs resource create partition1 ocf:pacemaker:Dummy meta resource-
> > > stickiness=100
> > > pcs resource create partition2 ocf:pacemaker:Dummy meta resource-
> > > stickiness=100
> > > pcs resource create partition3 ocf:pacemaker:Dummy meta resource-
> > > stickiness=100
> > >
> > > pcs resource utilization partition1 cpu=5
> > > pcs resource utilization partition2 cpu=5
> > > pcs resource utilization partition3 cpu=5
> > >
> > > pcs constraint colocation add std with sv-clone INFINITY
> > > pcs constraint colocation add partition1 with sv-clone INFINITY
> > > pcs constraint colocation add partition2 with sv-clone INFINITY
> > > pcs constraint colocation add partition3 with sv-clone INFINITY
> > >
> > > pcs property set maintenance-mode=false
> > >  
> > >
> > > I can then reproduce the issues in the following way:
> > >
> > > $ pcs resource
> > >  sv-fencer      (ocf::pacemaker:Dummy): Started vm1
> > >  Clone Set: sv-clone [sv]
> > >      Started: [ vm1 vm2 vm3 ]
> > >  std    (ocf::pacemaker:Dummy): Started vm2
> > >  partition1     (ocf::pacemaker:Dummy): Started vm3
> > >  partition2     (ocf::pacemaker:Dummy): Started vm1
> > >  partition3     (ocf::pacemaker:Dummy): Started vm2
> > >
> > > $ pcs cluster standby vm3
> > >
> > > # Check that all resources have moved off vm3
> > > $ pcs resource
> > >  sv-fencer      (ocf::pacemaker:Dummy): Started vm1
> > >  Clone Set: sv-clone [sv]
> > >      Started: [ vm1 vm2 ]
> > >      Stopped: [ vm3 ]
> > >  std    (ocf::pacemaker:Dummy): Started vm2
> > >  partition1     (ocf::pacemaker:Dummy): Started vm1
> > >  partition2     (ocf::pacemaker:Dummy): Started vm1
> > >  partition3     (ocf::pacemaker:Dummy): Started vm2
> > 
> > Thanks for the detailed information, this should help me get to the
> > bottom of it. From this description, it sounds like a new
> > transition
> > isn't being triggered when it should.
> > 
> > Could you please attach the DC's pe-input file that is listed in
> > the
> > logs after the standby step above? That would simplify analysis.
> > 
> > > # Wait for any outstanding actions to complete.
> > > $ crm_resource --wait --timeout 300
> > > Pending actions:
> > >         Action 22: sv-fencer_monitor_10000      on vm2
> > >         Action 21: sv-fencer_start_0    on vm2
> > >         Action 20: sv-fencer_stop_0     on vm1
> > > Error performing operation: Timer expired
> > >
> > > # Check the resources again - sv-fencer is still on vm1
> > > $ pcs resource
> > >  sv-fencer      (ocf::pacemaker:Dummy): Started vm1
> > >  Clone Set: sv-clone [sv]
> > >      Started: [ vm1 vm2 ]
> > >      Stopped: [ vm3 ]
> > >  std    (ocf::pacemaker:Dummy): Started vm2
> > >  partition1     (ocf::pacemaker:Dummy): Started vm1
> > >  partition2     (ocf::pacemaker:Dummy): Started vm1
> > >  partition3     (ocf::pacemaker:Dummy): Started vm2
> > >
> > > # Perform a random update to the CIB.
> > > $ pcs resource update std op monitor interval=20 timeout=20
> > >
> > > # Check resource status again - sv_fencer has now moved to vm2
> > (the
> > > action crm_resource was waiting for)
> > > $ pcs resource
> > >  sv-fencer      (ocf::pacemaker:Dummy): Started vm2 
> > <<<============
> > >  Clone Set: sv-clone [sv]
> > >      Started: [ vm1 vm2 ]
> > >      Stopped: [ vm3 ]
> > >  std    (ocf::pacemaker:Dummy): Started vm2
> > >  partition1     (ocf::pacemaker:Dummy): Started vm1
> > >  partition2     (ocf::pacemaker:Dummy): Started vm1
> > >  partition3     (ocf::pacemaker:Dummy): Started vm2
> > >
> > > I do not get the problem if I:
> > > 1) remove the "std" resource; or
> > > 2) remove the co-location constraints; or
> > > 3) remove the utilization attributes for the partition resources.
> > >
> > > In these cases the sv-fencer resource is happy to stay on vm1,
> > and
> > > crm_resource --wait returns immediately.
> > >
> > > It looks like the pcs cluster standby call is
> > creating/registering
> > > the actions to move the sv-fencer resource to vm2, but it doesn't
> > > include it in the cluster transition.  When the CIB is later
> > updated
> > > by something else, the action is included in that transition.
> > >
> > >
> > > Regards,
> > > Leon
> > 
> > _______________________________________________
> > Users mailing list: Users@clusterlabs.org
> > http://lists.clusterlabs.org/mailman/listinfo/users
> > 
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratc
> > h.pdf
> > Bugs: http://bugs.clusterlabs.org
> > 
> 
> _______________________________________________
> Users mailing list: Users@clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
-- 
Ken Gaillot <kgail...@redhat.com>

_______________________________________________
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

_______________________________________________
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Reply via email to