[ClusterLabs] Master-Slaver resource Restarted after configuration change

2016-06-09 Thread Ilia Sokolinski
Hi,

We have a custom Master-Slave resource running on a 3-node pcs cluster on 
CentOS 7.1

As part of what is supposed to be an NDU we do update some properties of the 
resource.
For some reason this causes both Master and Slave instances of the  resource to 
be restarted.

Since restart takes a fairly long time for us, the update becomes very much 
disruptive.

Is this expected? 
We have not seen this behavior with the previous release of pacemaker.


Jun 10 02:06:11 dev-ceph02 crmd[30570]: notice: State transition S_IDLE -> 
S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL 
origin=abort_transition_graph ]
Jun 10 02:06:11 dev-ceph02 attrd[30568]: notice: Updating all attributes after 
cib_refresh_notify event
Jun 10 02:06:11 dev-ceph02 crmd[30570]: notice: State transition S_ELECTION -> 
S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED 
origin=election_timeout_popped ]
Jun 10 02:06:11 dev-ceph02 crmd[30570]: warning: FSA: Input I_ELECTION_DC from 
do_election_check() received in state S_INTEGRATION

Jun 10 02:06:12 dev-ceph02 pengine[30569]: notice: Restart L3:0 (Master 
d-l303-a.dev-bos.csdops.net)
Jun 10 02:06:12 dev-ceph02 pengine[30569]: notice: Restart L3:1 (Slave 
d-l303-b.dev-bos.csdops.net)
Jun 10 02:06:12 dev-ceph02 pengine[30569]: notice: Calculated Transition 4845: 
/var/lib/pacemaker/pengine/pe-input-2934.bz2
Jun 10 02:06:12 dev-ceph02 crmd[30570]: notice: Initiating action 63: demote 
L3_demote_0 on d-l303-a.dev-bos.csdops.net
Jun 10 02:06:14 dev-ceph02 crmd[30570]: notice: Initiating action 64: stop 
L3_stop_0 on d-l303-a.dev-bos.csdops.net
Jun 10 02:06:14 dev-ceph02 crmd[30570]: notice: Initiating action 66: stop 
L3_stop_0 on d-l303-b.dev-bos.csdops.net
Jun 10 02:06:15 dev-ceph02 crmd[30570]: notice: Initiating action 17: start 
L3_start_0 on d-l303-a.dev-bos.csdops.net
Jun 10 02:06:15 dev-ceph02 crmd[30570]: notice: Initiating action 18: start 
L3_start_0 on d-l303-b.dev-bos.csdops.net 


Here is the cluster configuration:

pcs status
Cluster name: L3_cluster
Last updated: Fri Jun 10 03:17:31 2016  Last change: Fri Jun 10 
02:06:11 2016 by root via cibadmin on d-l303-a.dev-bos.csdops.net
Stack: corosync
Current DC: dev-ceph02.dev-bos.csdops.net (version 1.1.13-a14efad) - partition 
with quorum
3 nodes and 12 resources configured

Online: [ d-l303-a.dev-bos.csdops.net d-l303-b.dev-bos.csdops.net 
dev-ceph02.dev-bos.csdops.net ]

Full list of resources:

 idrac-d-l303-b.dev-bos.csdops.net  (stonith:fence_idrac):  Started 
dev-ceph02.dev-bos.csdops.net
 idrac-d-l303-a.dev-bos.csdops.net  (stonith:fence_idrac):  Started 
d-l303-b.dev-bos.csdops.net
 noop-dev-ceph02.dev-bos.csdops.net (stonith:fence_noop):   Started 
d-l303-a.dev-bos.csdops.net
 L3-5bb92-0-ip  (ocf::heartbeat:IPaddr2):   Started 
d-l303-a.dev-bos.csdops.net
 Master/Slave Set: L3-5bb92-0-master [L3-5bb92-0]
 Masters: [ d-l303-a.dev-bos.csdops.net ]
 Slaves: [ d-l303-b.dev-bos.csdops.net ]
 L3-86a2c-1-ip  (ocf::heartbeat:IPaddr2):   Started 
d-l303-b.dev-bos.csdops.net
 Master/Slave Set: L3-86a2c-1-master [L3-86a2c-1]
 Masters: [ d-l303-b.dev-bos.csdops.net ]
 Slaves: [ d-l303-a.dev-bos.csdops.net ]
 L3-ip  (ocf::heartbeat:IPaddr2):   Started d-l303-a.dev-bos.csdops.net
 Master/Slave Set: L3-master [L3]
 Masters: [ d-l303-a.dev-bos.csdops.net ]
 Slaves: [ d-l303-b.dev-bos.csdops.net ]

PCSD Status:
  d-l303-b.dev-bos.csdops.net: Online
  d-l303-a.dev-bos.csdops.net: Online
  dev-ceph02.dev-bos.csdops.net: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

Rpms:

pcs-0.9.137-13.el7_1.4.x86_64
pacemaker-cluster-libs-1.1.12-22.el7_1.4.x86_64
pacemaker-cli-1.1.12-22.el7_1.4.x86_64
pacemaker-libs-1.1.12-22.el7_1.4.x86_64
pacemaker-1.1.12-22.el7_1.4.x86_64


Thanks a lot

Ilia Sokolinski___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org


Re: [ClusterLabs] Informing RAs about recovery: failed resource recovery, or any start-stop cycle?

2016-06-09 Thread Andrew Beekhof
On Wed, Jun 8, 2016 at 6:23 PM, Adam Spiers  wrote:
> Andrew Beekhof  wrote:
>> On Wed, Jun 8, 2016 at 12:11 AM, Adam Spiers  wrote:
>> > Ken Gaillot  wrote:
>> >> On 06/06/2016 05:45 PM, Adam Spiers wrote:
>> >> > Adam Spiers  wrote:
>> >> >> Andrew Beekhof  wrote:
>> >> >>> On Tue, Jun 7, 2016 at 8:29 AM, Adam Spiers  wrote:
>> >>  Ken Gaillot  wrote:
>> >> > My main question is how useful would it actually be in the proposed 
>> >> > use
>> >> > cases. Considering the possibility that the expected start might 
>> >> > never
>> >> > happen (or fail), can an RA really do anything different if
>> >> > start_expected=true?
>> >> 
>> >>  That's the wrong question :-)
>> >> 
>> >> > If the use case is there, I have no problem with
>> >> > adding it, but I want to make sure it's worthwhile.
>> >> 
>> >>  The use case which started this whole thread is for
>> >>  start_expected=false, not start_expected=true.
>> >> >>>
>> >> >>> Isn't this just two sides of the same coin?
>> >> >>> If you're not doing the same thing for both cases, then you're just
>> >> >>> reversing the order of the clauses.
>> >> >>
>> >> >> No, because the stated concern about unreliable expectations
>> >> >> ("Considering the possibility that the expected start might never
>> >> >> happen (or fail)") was regarding start_expected=true, and that's the
>> >> >> side of the coin we don't care about, so it doesn't matter if it's
>> >> >> unreliable.
>> >> >
>> >> > BTW, if the expected start happens but fails, then Pacemaker will just
>> >> > keep repeating until migration-threshold is hit, at which point it
>> >> > will call the RA 'stop' action finally with start_expected=false.
>> >> > So that's of no concern.
>> >>
>> >> To clarify, that's configurable, via start-failure-is-fatal and on-fail
>> >
>> > Sure.
>> >
>> >> > Maybe your point was that if the expected start never happens (so
>> >> > never even gets a chance to fail), we still want to do a nova
>> >> > service-disable?
>> >>
>> >> That is a good question, which might mean it should be done on every
>> >> stop -- or could that cause problems (besides delays)?
>> >
>> > No, the whole point of adding this feature is to avoid a
>> > service-disable on every stop, and instead only do it on the final
>> > stop.  If there are corner cases where we never reach the final stop,
>> > that's not a disaster because nova will eventually figure it out and
>> > do the right thing when the server-agent connection times out.
>> >
>> >> Another aspect of this is that the proposed feature could only look at a
>> >> single transition. What if stop is called with start_expected=false, but
>> >> then Pacemaker is able to start the service on the same node in the next
>> >> transition immediately afterward? Would having called service-disable
>> >> cause problems for that start?
>> >
>> > We would also need to ensure that service-enable is called on start
>> > when necessary.  Perhaps we could track the enable/disable state in a
>> > local temporary file, and if the file indicates that we've previously
>> > done service-disable, we know to run service-enable on start.  This
>> > would avoid calling service-enable on every single start.
>>
>> feels like an over-optimization
>> in fact, the whole thing feels like that if i'm honest.
>
> Huh ... You didn't seem to think that when we discussed automating
> service-disable at length in Austin.

I didn't feel the need to push back because RH uses the systemd agent
instead so you're only hanging yourself, but more importantly because
the proposed implementation to facilitate it wasn't leading RA writers
down a hazardous path :-)

>  What changed?  Can you suggest a
> better approach?

Either always or never disable the service would be my advice.
"Always" specifically getting my vote.

>
>> why are we trying to optimise the projected performance impact
>
> It's not really "projected"; we know exactly what the impact is.  And
> it's not really a performance impact either.  If nova-compute (or a
> dependency) is malfunctioning on a compute node, there will be a
> window (bounded by nova.conf's rpc_response_timeout value, IIUC) in
> which nova-scheduler could still schedule VMs onto that compute node,
> and then of course they'll fail to boot.

Right, but that window exists regardless of whether the node is or is
not ever coming back.
And as we already discussed, the proposed feature still leaves you
open to this window because we can't know if the expected restart will
ever happen.

In this context, trying to avoid the disable call under certain
circumstances, to avoid repeated and frequent flip-flopping of the
state, seems ill-advised.  At the point nova compute is bouncing up
and down like that, you have a more fundamental issue somewhere in
your 

Re: [ClusterLabs] Minimum configuration for dynamically adding a node to a cluster

2016-06-09 Thread Kristoffer Grönlund
Jehan-Guillaume de Rorthais  writes:

> Le 8 juin 2016 13:36:03 GMT+02:00, Nikhil Utane  
> a écrit :
>>Hi,
>>
>>Would like to know the best and easiest way to add a new node to an
>>already
>>running cluster.
>>
>>Our limitation:
>>1) pcsd cannot be used since (as per my understanding) it communicates
>>over
>>ssh which is prevented.
>
> As far as i remember,  pcsd deamons use their own tcp port (not the ssh one) 
> and communicate with each others using http queries (over ssl i suppose).
>
> As far as i understand, crmsh uses ssh, not pcsd.

crmsh does use ssh but can be configured to communicate over an
alternative port in case the standard ssh port is blocked.

Cheers,
Kristoffer

>
> ___
> Users mailing list: Users@clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

-- 
// Kristoffer Grönlund
// kgronl...@suse.com

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org