Re: [ClusterLabs] Antw: Re: Q: ordering for a monitoring op only?

[Ken Gaillot]

On Tue, 2018-08-21 at 07:49 +0200, Ulrich Windl wrote:
> > > > Ken Gaillot  schrieb am 20.08.2018 um
> > > > 16:49 in
> 
> Nachricht
> <1534776566.6465.5.ca...@redhat.com>:
> > On Mon, 2018‑08‑20 at 10:51 +0200, Ulrich Windl wrote:
> > > Hi!
> > > 
> > > I wonder whether it's possible to run a monitoring op only if
> > > some
> > > specific resource is up.
> > > Background: We have some resource that runs fine without NFS, but
> > > the
> > > start, stop and monitor operations will just hang if NFS is down.
> > > In
> > > effect the monitor operation will time out, the cluster will try
> > > to
> > > recover, calling the stop operation, which in turn will time out,
> > > making things worse (i.e.: causing a node fence).
> > > 
> > > So my idea was to pause the monitoing operation while NFS is down
> > > (NFS itself is controlled by the cluster and should recover
> > > "rather
> > > soon" TM).
> > > 
> > > Is that possible?
> > 
> > A possible mitigation would be to set on‑fail=block on the
> > dependent
> > resource monitor, so if NFS is down, the monitor will still time
> > out,
> > but the cluster will not try to stop it. Of course then you lose
> > the
> > ability to automatically recover from an actual resource failure.
> > 
> > The only other thing I can think of probably wouldn't be reliable:
> > you
> > could put the NFS resource in a group with an
> > ocf:pacemaker:attribute
> > resource. That way, whenever NFS is started, a node attribute will
> > be
> > set, and whenever NFS is stopped, the attribute will be unset.
> > Then,
> > you can set a rule using that attribute. For example you could make
> > the
> > dependent resource's is‑managed property depend on the node
> > attribute
> > value. The reason I think it wouldn't be reliable is that if NFS
> > failed, there would be some time before the cluster stopped the NFS
> > resource and updated the node attribute, and the dependent resource
> > monitor could run during that time. But it would at least diminish
> > the
> > problem space.
> 
> Hi!
> 
> That sounds interesting, even though it's still a work-around and not
> the
> solution for the original problem. Could you show a sketch of the
> mechanism:
> How to set the attribute with the resource, and how to make the
> monitor
> operation depend on it?

ocf:pacemaker:attribute by default creates an attribute named "opa-
" with a value of "1" when the resource is started and "0"
when it is stopped (though note that the attribute will not exist
before the first time it is started).

Therefore by creating a group of the NFS resource and an attribute
resource, the attribute will be set to 1 immediately after the NFS
resource is started, and to 0 immediately before the NFS resource is
stopped.

Conceptually:

  create resource rsc-nfs
  create resource rsc-attr
  create group rsc-group = rsc-nfs rsc-attr
  create resource rsc-dependent
 meta-attributes
rule: opa-rsc-attr=0 -> is-managed=false
default rule: is-managed=true
  asymmetric order start rsc-group then start rsc-dependent

With that, at start-up, rsc-dependent will wait for rsc-group to start
due to the order constraint (that takes care of the initial case where
the attribute does not yet exist). Because it's asymmetric, rsc-group
can fail or stop without affecting it.

If rsc-group fails (or is disabled), it will be stopped, at which point
the attribute will go to 0, and rsc-dependent will become unmanaged.
The monitor will still run but will be ignored. If the monitor happens
to run (and complete) between when NFS actually fails and when the
cluster stops rsc-group, there will still be trouble. But otherwise,
the cluster will ignore rsc-dependent's monitor failures.

The advantage of that over on-fail=block is that as long as rsc-group
is running, the cluster will still recover rsc-dependent if it fails.
Only when rsc-group is down will the cluster ignore rsc-dependent.

> > Probably any dynamic solution would have a similar race condition
> > ‑‑
> > the NFS will be failed in reality for some amount of time before
> > the
> > cluster detects the failure, so the cluster could never prevent the
> > monitor from running during that window.
> 
> I agree completely.
> 
> Regards,
> Ulrich
> 
> > 
> > > And before you ask: No, I have not written that RA that has the
> > > problem; a multi‑million‑dollar company wrote it (Years before I
> > > had
> > > written a monito

Re: [ClusterLabs] Q: Forcing a role change of master/slave resource

[Ken Gaillot]

On Mon, 2018-08-20 at 10:41 +0200, Ulrich Windl wrote:
> Hi!
> 
> Browsing the documentation, I'm still woindering whether this is
> possible:
> Make a master/slave resource change roles, i.e.: master becomes
> slave, slave becomes master, preferrably without stopping the master.
> The documentation for "crm_resource --ban --master ..." is somewhat
> vague: Will it stop the master resource, or will it be demoted?

It will be demoted (assuming nothing else prevents that).

> The command I'm asking for is useful for tsting whether a
> master/slave resource works correctly on eac node.
> 
> An extension of my request would be the command to run a master or
> slave role on any given node in the cluster...
> 
> Regards,
> Ulrich

The other possibility is to manipulate the master scores directly with
"crm_master -r  -N " using "-G" to see the current scores or
"-v " to change them. The node with the highest score will be
promoted.
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Q: ordering for a monitoring op only?

[Ken Gaillot]

On Mon, 2018-08-20 at 10:51 +0200, Ulrich Windl wrote:
> Hi!
> 
> I wonder whether it's possible to run a monitoring op only if some
> specific resource is up.
> Background: We have some resource that runs fine without NFS, but the
> start, stop and monitor operations will just hang if NFS is down. In
> effect the monitor operation will time out, the cluster will try to
> recover, calling the stop operation, which in turn will time out,
> making things worse (i.e.: causing a node fence).
> 
> So my idea was to pause the monitoing operation while NFS is down
> (NFS itself is controlled by the cluster and should recover "rather
> soon" TM).
> 
> Is that possible?

A possible mitigation would be to set on-fail=block on the dependent
resource monitor, so if NFS is down, the monitor will still time out,
but the cluster will not try to stop it. Of course then you lose the
ability to automatically recover from an actual resource failure.

The only other thing I can think of probably wouldn't be reliable: you
could put the NFS resource in a group with an ocf:pacemaker:attribute
resource. That way, whenever NFS is started, a node attribute will be
set, and whenever NFS is stopped, the attribute will be unset. Then,
you can set a rule using that attribute. For example you could make the
dependent resource's is-managed property depend on the node attribute
value. The reason I think it wouldn't be reliable is that if NFS
failed, there would be some time before the cluster stopped the NFS
resource and updated the node attribute, and the dependent resource
monitor could run during that time. But it would at least diminish the
problem space.

Probably any dynamic solution would have a similar race condition --
the NFS will be failed in reality for some amount of time before the
cluster detects the failure, so the cluster could never prevent the
monitor from running during that window.

> And before you ask: No, I have not written that RA that has the
> problem; a multi-million-dollar company wrote it (Years before I had
> written a monitor for HP-UX' cluster that did not have this problem,
> even though the configuration files were read from NFS (It's not
> magic: Just periodically copy them to shared memory, and read the
> config from shared memory).
> 
> Regards,
> Ulrich
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Q: automaticlly remove expired location constraints

[Ken Gaillot]

On Thu, 2018-08-23 at 12:27 +0200, Ulrich Windl wrote:
> Hi!
> 
> I have a non-trivial question: How can I remove expired manual
> migration requests, like the following?:
> location cli-standby-rsc rsc rule -inf: #uname eq host and date lt
> "2013-06-12 13:47:26Z"
> 
> One problem is that the date value is not a constant, and it had to
> be compared against the current date
> 
> Regards,
> Ulrich

crm_resource --clear -r RSC will clear all cli-* constraints
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Q: (SLES11 SP4) lrm_rsc_op without last-run?

[Ken Gaillot]

On Thu, 2018-08-23 at 08:08 +0200, Ulrich Windl wrote:
> Hi!
> 
> Many years ago I wrote a parser that could format the CIB XML in a
> flexible way. Today I used it again to print some statistics for
> "exec-time". Thereby I discovered one operation that has a valid
> "exec-time", a valid "last-rc-change", but no "last-run".
> All other operations had "last-run". Can someone explain how this can
> happen? The operation in question is "monitor", so it should be run
> frequently (specified as "op monitor interval=600 timeout=30").

Recurring actions never get last-run, because the crmd doesn't initiate
each run of a recurring action.

> I see no failed actions regarding the resource.
> 
> The original XML part for the operation looks like this:
>   operation_key="prm_cron-cleanup_monitor_60" operation="monitor"
> crm-debug-origin="build_active_RAs" crm_feature_set="3.0.10"
> transition-key="92:3:0:6c6eff09-0d57-4844-9c3c-bc300c095bb6"
> transition-magic="0:0;92:3:0:6c6eff09-0d57-4844-9c3c-bc300c095bb6"
> on_node="h06" call-id="151" rc-code="0" op-status="0"
> interval="60" last-rc-change="1513367408" exec-time="13" queue-
> time="0" op-digest="2351b51e5316689a0eb89e8061445728"/>
> 
> The node is not completely up-to-date, and it's using pacemaker-
> 1.1.12-18.1...
> 
> Regards,
> Ulrich
> 
> 
> _______
> Users mailing list: Users@clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Redundant ring not recovering after node is back

[Ken Gaillot]

t; Here you have some of my configuration settings on node 1
> > > > > > > (I probed
> > > > > > > already
> > > > > > > to change rrp_mode):
> > > > > > > 
> > > > > > > *- corosync.conf*
> > > > > > > 
> > > > > > > 
> > > > > > > totem {
> > > > > > >    version: 2
> > > > > > >    cluster_name: node
> > > > > > >    token: 5000
> > > > > > >    token_retransmits_before_loss_const: 10
> > > > > > >    secauth: off
> > > > > > >    threads: 0
> > > > > > >    rrp_mode: passive
> > > > > > >    nodeid: 1
> > > > > > >    interface {
> > > > > > >    ringnumber: 0
> > > > > > >    bindnetaddr: 192.168.0.0
> > > > > > >    #mcastaddr: 226.94.1.1
> > > > > > >    mcastport: 5405
> > > > > > >    broadcast: yes
> > > > > > >    }
> > > > > > >    interface {
> > > > > > >    ringnumber: 1
> > > > > > >    bindnetaddr: 192.168.1.0
> > > > > > >    #mcastaddr: 226.94.1.2
> > > > > > >    mcastport: 5407
> > > > > > >    broadcast: yes
> > > > > > >    }
> > > > > > > }
> > > > > > > 
> > > > > > > logging {
> > > > > > >    fileline: off
> > > > > > >    to_stderr: yes
> > > > > > >    to_syslog: yes
> > > > > > >    to_logfile: yes
> > > > > > >    logfile: /var/log/corosync/corosync.log
> > > > > > >    debug: off
> > > > > > >    timestamp: on
> > > > > > >    logger_subsys {
> > > > > > >    subsys: AMF
> > > > > > >    debug: off
> > > > > > >    }
> > > > > > > }
> > > > > > > 
> > > > > > > amf {
> > > > > > >    mode: disabled
> > > > > > > }
> > > > > > > 
> > > > > > > quorum {
> > > > > > >    provider: corosync_votequorum
> > > > > > >    expected_votes: 2
> > > > > > > }
> > > > > > > 
> > > > > > > nodelist {
> > > > > > >    node {
> > > > > > >    nodeid: 1
> > > > > > >    ring0_addr: 192.168.0.1
> > > > > > >    ring1_addr: 192.168.1.1
> > > > > > >    }
> > > > > > > 
> > > > > > >    node {
> > > > > > >    nodeid: 2
> > > > > > >    ring0_addr: 192.168.0.2
> > > > > > >    ring1_addr: 192.168.1.2
> > > > > > >    }
> > > > > > > }
> > > > > > > 
> > > > > > > aisexec {
> > > > > > >    user: root
> > > > > > >    group: root
> > > > > > > }
> > > > > > > 
> > > > > > > service {
> > > > > > >    name: pacemaker
> > > > > > >    ver: 1
> > > > > > > }
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > *- /etc/hosts*
> > > > > > > 
> > > > > > > 
> > > > > > > 127.0.0.1   localhost
> > > > > > > 10.4.172.5  node1.upc.edu node1
> > > > > > > 10.4.172.6  node2.upc.edu node2
> > > > > > > 
> > > > > > > 
> > > > > > > So machines have 3 NICs? 2 for corosync/cluster traffic
> > > > > > > and one for
> > > > > > 
> > > > > > regular traffic/services/outside world?
> > > > > > 
> > > > > > 
> > > > > > Thank you for you help in advance!
> > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > > To conclude:
> > > > > > - If you are using NetworkManager, try to install
> > > > > > NetworkManager-config-server, it will probably help
> > > > > > - If you are brave enough, try corosync 3.x (current Alpha4
> > > > > > is pretty
> > > > > > stable - actually some other projects gain this stability
> > > > > > with SP1 :) )
> > > > > > that has no RRP but uses knet for support redundant links
> > > > > > (up-to 8 links
> > > > > > can be configured) and doesn't have problems with ifdown.
> > > > > > 
> > > > > > Honza
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > > 
> > > > > > > ___
> > > > > > > Users mailing list: Users@clusterlabs.org
> > > > > > > https://lists.clusterlabs.org/mailman/listinfo/users
> > > > > > > 
> > > > > > > Project Home: http://www.clusterlabs.org
> > > > > > > Getting started: http://www.clusterlabs.org/doc
> > > > > > > /Cluster_from_Scratch.pdf
> > > > > > > Bugs: http://bugs.clusterlabs.org
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > 
> > > > > --
> > > > > *David Tolosa Martínez*
> > > > > Customer Support & Infrastructure
> > > > > UPCnet - Edifici Vèrtex
> > > > > Plaça d'Eusebi Güell, 6, 08034 Barcelona
> > > > > Tel: 934054555
> > > > > 
> > > > > <https://www.upcnet.es>
> > > > > 
> > > > > 
> > > > 
> > > > 
> > > > 
> > 
> > 
> 
> ___
> Users mailing list: Users@clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Spurious node loss in corosync cluster

[Ken Gaillot]

mits_before_loss_const;
changing token is generally enough to deal with transient network
unreliability. However 18 seconds is a really long time to raise the
token to, and it's uncertain from the information here whether the root
cause was networking or something on the host.

I notice your configuration is corosync 1 with the pacemaker plugin;
that is a long-deprecated setup, and corosync 3 is about to come out,
so you may want to consider upgrading to at least corosync 2 and a
reasonably recent pacemaker. That would give you some reliability
improvements, including real-time priority scheduling of corosync,
which could have been the issue here if CPU load rather than networking
was the root cause.

> 
> Currently my corosync.conf looks like this :
> 
> compatibility: whitetank
> totem {
>     version: 2
>     secauth: on
>     threads: 0
>     interface {
>     member {
>             memberaddr: 172.20.0.4
>         }
> member {
>             memberaddr: 172.20.0.9
>         }
> member {
>             memberaddr: 172.20.0.12
>         }
> 
>     bindnetaddr: 172.20.0.12
> 
>     ringnumber: 0
>     mcastport: 5405
>     ttl: 1
>     }
>     transport: udpu
>     token: 1
>     token_retransmits_before_loss_const: 10
> }
> 
> logging {
>     fileline: off
>     to_stderr: yes
>     to_logfile: yes
>     to_syslog: no
>     logfile: /var/log/cluster/corosync.log
>     timestamp: on
>     logger_subsys {
>     subsys: AMF
>     debug: off
>     }
> }
> service {
>     name: pacemaker
>     ver: 1
> }
> amf {
>     mode: disabled
> }
> 
> Thanks in advance for the help.
> Prasad
> 
> ___
> Users mailing list: Users@clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Different Times in the Corosync Log?

[Ken Gaillot]

Whoa, I think you win some sort of fubar prize. :-)

AFAIK, any OS-level time or timezone change affects all processes
equally. (I occasionally deal with cluster logs where the OS time
jumped backward or forward, and all logs system-wide are equally
affected.)

Some applications have their own timezone setting that can override the
system default, but pacemaker isn't one of them. It's even more bizarre
when you consider that the daemons here are the children of the same
process (pacemakerd), and thus have an identical set of environment
variables and so forth. (And as Jan pointed out, they appear to have
been started within a fraction of a second of each other.)

Apparently there is a dateshift kernel module that can put particular
processes in different apparent times, but I assume you'd know if you
did that on purpose. :-) It does occur to me that the module would be a
great prank to play on someone (especially combined with a cron job
that randomly altered the configuration).

If you figure this out, I'd love to hear what it was. Gremlins ...

On Tue, 2018-08-21 at 11:45 +0200, Jan Pokorný wrote:
> On 21/08/18 08:43 +, Eric Robinson wrote:
> > > I could guess that the processes run with different timezone
> > > settings (for whatever reason).
> > 
> > That would be my guess, too, but I cannot imagine how they ended up
> > in that condition. 
> 
> Hard to guess, the PIDs indicate the expected state of covering a
> very
> short interval sequentially (i.e. no intermittent failure recovered
> with
> a restart of lrmd, AFAICT).  In case it can have any bearing, how do
> you start pacemaker -- systemd, initscript, as a corosync plugin,
> something else?
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: Spurious node loss in corosync cluster

[Ken Gaillot]

On Tue, 2018-08-21 at 15:29 +0200, Ulrich Windl wrote:
> > > > Prasad Nagaraj  schrieb am
> > > > 21.08.2018 um 11:42 in
> 
> Nachricht
> :
> > Hi Ken - Thanks for you response.
> > 
> > We do have seen messages in other cases like
> > corosync [MAIN  ] Corosync main process was not scheduled for
> > 17314.4746 ms
> > (threshold is 8000. ms). Consider token timeout increase.
> > corosync [TOTEM ] A processor failed, forming new configuration.
> > 
> > Is this the indication of a failure due to CPU load issues and will
> > this
> > get resolved if I upgrade to Corosync 2.x series ?

Yes, most definitely this is a CPU issue. It means corosync isn't
getting enough CPU cycles to handle the cluster token before the
timeout is reached.

Upgrading may indeed help, as recent versions ensure that corosync runs
with real-time priority in the kernel, and thus are more likely to get
CPU time when something of lower priority is consuming all the CPU.

But of course, there is some underlying problem that should be
identified and addressed. Figure out what's maxing out the CPU or I/O.
Ulrich's monitoring suggestion is a good start.

> Hi!
> 
> I'd strongly recommend starting monitoring on your nodes, at least
> until you know what's going on. The good old UNIX sa (sysstat
> package) could be a starting point. I'd monitor CPU idle
> specifically. Then go for 100% device utilization, then look for
> network bottlenecks...
> 
> A new corosync release cannot fix those, most likely.
> 
> Regards,
> Ulrich
> 
> > 
> > In any case, for the current scenario, we did not see any
> > scheduling
> > related messages.
> > 
> > Thanks for your help.
> > Prasad
> > 
> > On Mon, Aug 20, 2018 at 7:57 PM, Ken Gaillot 
> > wrote:
> > 
> > > On Sun, 2018-08-19 at 17:35 +0530, Prasad Nagaraj wrote:
> > > > Hi:
> > > > 
> > > > One of these days, I saw a spurious node loss on my 3-node
> > > > corosync
> > > > cluster with following logged in the corosync.log of one of the
> > > > nodes.
> > > > 
> > > > Aug 18 12:40:25 corosync [pcmk  ] notice: pcmk_peer_update:
> > > > Transitional membership event on ring 32: memb=2, new=0, lost=1
> > > > Aug 18 12:40:25 corosync [pcmk  ] info: pcmk_peer_update: memb:
> > > > vm02d780875f 67114156
> > > > Aug 18 12:40:25 corosync [pcmk  ] info: pcmk_peer_update: memb:
> > > > vmfa2757171f 151000236
> > > > Aug 18 12:40:25 corosync [pcmk  ] info: pcmk_peer_update: lost:
> > > > vm728316982d 201331884
> > > > Aug 18 12:40:25 corosync [pcmk  ] notice: pcmk_peer_update:
> > > > Stable
> > > > membership event on ring 32: memb=2, new=0, lost=0
> > > > Aug 18 12:40:25 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
> > > > vm02d780875f 67114156
> > > > Aug 18 12:40:25 corosync [pcmk  ] info: pcmk_peer_update: MEMB:
> > > > vmfa2757171f 151000236
> > > > Aug 18 12:40:25 corosync [pcmk  ] info:
> > > > ais_mark_unseen_peer_dead:
> > > > Node vm728316982d was not seen in the previous transition
> > > > Aug 18 12:40:25 corosync [pcmk  ] info: update_member: Node
> > > > 201331884/vm728316982d is now: lost
> > > > Aug 18 12:40:25 corosync [pcmk  ] info:
> > > > send_member_notification:
> > > > Sending membership update 32 to 3 children
> > > > Aug 18 12:40:25 corosync [TOTEM ] A processor joined or left
> > > > the
> > > > membership and a new membership was formed.
> > > > Aug 18 12:40:25 [4544] vmfa2757171f stonith-ng: info:
> > > > plugin_handle_membership: Membership 32: quorum retained
> > > > Aug 18 12:40:25 [4544] vmfa2757171f stonith-ng:   notice:
> > > > crm_update_peer_state_iter:   plugin_handle_membership: Node
> > > > vm728316982d[201331884] - state is now lost (was member)
> > > > Aug 18 12:40:25 [4548] vmfa2757171f   crmd: info:
> > > > plugin_handle_membership: Membership 32: quorum retained
> > > > Aug 18 12:40:25 [4548] vmfa2757171f   crmd:   notice:
> > > > crm_update_peer_state_iter:   plugin_handle_membership: Node
> > > > vm728316982d[201331884] - state is now lost (was member)
> > > > Aug 18 12:40:25 [4548] vmfa2757171f   crmd: info:
> > > > peer_update_callback: vm728316982d is now lost (was member)
> > > > Aug 18 12:40:25 [4548] vmfa2757171f   crmd:  warning:
> > > > match

Re: [ClusterLabs] FYI: regression using 2.0.0 / 1.1.19 Pacemaker Remote node with older cluster nodes

[Ken Gaillot]

Upon further investigation, there is no problem when resource agents
are called by the cluster, which thankfully makes this issue less
significant.

The problem occurs when "crm_node -n" is called on the command line or
by a script, on a Pacemaker Remote node running 1.1.19 or 2.0.0 or
later, with cluster nodes running 1.1.18 or earlier. Upgrading cluster
nodes before Pacemaker Remote nodes avoids the issue.

If you have any custom resource agents, a good practice is to make sure
that they do not call any unnecessary commands (including "crm_node -n"
or "ocf_local_nodename") for meta-data actions. This will not only be
more efficient, but also make command-line meta-data calls immune to
issues like this.

A complete solution would make every command-line "crm_node -n" call
take longer and have more chances to fail, so I'm inclined to leave
this as a known issue, and rely on the workarounds.

On Mon, 2018-07-16 at 09:21 -0500, Ken Gaillot wrote:
> Hi all,
> 
> The just-released Pacemaker 2.0.0 and 1.1.19 releases have an issue
> when a Pacemaker Remote node is upgraded before the cluster nodes.
> 
> Pacemaker 2.0.0 contains a fix (also backported to 1.1.19) for the
> longstanding issue of "crm_node -n" getting the wrong name when run
> on
> the command line of a Pacemaker Remote node whose node name is
> different from its local hostname.
> 
> However, the fix can cause resource agents running on a Pacemaker
> Remote node to hang when used with a cluster node older than 2.0.0 /
> 1.1.19.
> 
> The only workaround is to upgrade all cluster nodes before upgrading
> any Pacemaker Remote nodes (which is the recommended practice
> anyway).
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] ping Resource Agent doesnt work

[Ken Gaillot]

On Fri, 2018-07-20 at 08:41 +0800, Confidential Company wrote:
> I configured ping-RA via ocf:pacemaker. During testing the resource-
> agent works fine, it executes ping action to my gateway every 10
> seconds and 3 attempts. But I noticed that even I disconnect the
> physical link of node where the resource resides, it doesnt failover
> to other node.

The ping agent by itself only sets a node attribute. Failover requires
additional constraints:

http://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html-single/Pa
cemaker_Explained/index.html#_moving_resources_due_to_connectivity_chan
ges


> 
> 
> IP CONFIG:
> 
> 172.16.10.0/24 = interface to gateway
> 172.16.11.0/24= heartbeat link (interface to avoid split brain as my
> goal is to only failover ping-resource after 3-attempts failure to
> ping my gateway)
> 
> CONFIG:
> 
> [root@node1 ~]# pcs config
> Cluster Name: clusterPa
> Corosync Nodes:
>  node1 node2
> Pacemaker Nodes:
>  node1 node2
> 
> Resources:
>  Resource: ping-gateway (class=ocf provider=pacemaker type=ping)
>   Attributes: host_list=172.16.10.1
>   Operations: monitor interval=10 timeout=60 (ping-gateway-monitor-
> interval-10)
>               start interval=0s timeout=60 (ping-gateway-start-
> interval-0s)
>               stop interval=0s timeout=20 (ping-gateway-stop-
> interval-0s)
> 
> Stonith Devices:
> Fencing Levels:
> 
> Location Constraints:
> Ordering Constraints:
> Colocation Constraints:
> Ticket Constraints:
> 
> Alerts:
>  No alerts defined
> 
> Resources Defaults:
>  No defaults set
> Operations Defaults:
>  No defaults set
> 
> Cluster Properties:
>  cluster-infrastructure: corosync
>  cluster-name: clusterPa
>  dc-version: 1.1.16-12.el7-94ff4df
>  have-watchdog: false
>  last-lrm-refresh: 1531899781
>  stonith-enabled: false
> 
> Quorum:
>   Options:
> [root@node1 ~]#  
> ___
> Users mailing list: Users@clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] FYI: regression using 2.0.0 / 1.1.19 Pacemaker Remote node with older cluster nodes

[Ken Gaillot]

Hi all,

The just-released Pacemaker 2.0.0 and 1.1.19 releases have an issue
when a Pacemaker Remote node is upgraded before the cluster nodes.

Pacemaker 2.0.0 contains a fix (also backported to 1.1.19) for the
longstanding issue of "crm_node -n" getting the wrong name when run on
the command line of a Pacemaker Remote node whose node name is
different from its local hostname.

However, the fix can cause resource agents running on a Pacemaker
Remote node to hang when used with a cluster node older than 2.0.0 /
1.1.19.

The only workaround is to upgrade all cluster nodes before upgrading
any Pacemaker Remote nodes (which is the recommended practice anyway).
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] ping Resource Agent doesnt work

[Ken Gaillot]

On Fri, 2018-07-20 at 08:41 +0800, Confidential Company wrote:
> I configured ping-RA via ocf:pacemaker. During testing the resource-
> agent works fine, it executes ping action to my gateway every 10
> seconds and 3 attempts. But I noticed that even I disconnect the
> physical link of node where the resource resides, it doesnt failover
> to other node.

The ping agent by itself only sets a node attribute. Failover requires
additional constraints:

http://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html-single/Pacemaker_Explained/index.html#_moving_resources_due_to_connectivity_changes


> 
> 
> IP CONFIG:
> 
> 172.16.10.0/24 = interface to gateway
> 172.16.11.0/24= heartbeat link (interface to avoid split brain as my
> goal is to only failover ping-resource after 3-attempts failure to
> ping my gateway)
> 
> CONFIG:
> 
> [root@node1 ~]# pcs config
> Cluster Name: clusterPa
> Corosync Nodes:
>  node1 node2
> Pacemaker Nodes:
>  node1 node2
> 
> Resources:
>  Resource: ping-gateway (class=ocf provider=pacemaker type=ping)
>   Attributes: host_list=172.16.10.1
>   Operations: monitor interval=10 timeout=60 (ping-gateway-monitor-
> interval-10)
>               start interval=0s timeout=60 (ping-gateway-start-
> interval-0s)
>               stop interval=0s timeout=20 (ping-gateway-stop-
> interval-0s)
> 
> Stonith Devices:
> Fencing Levels:
> 
> Location Constraints:
> Ordering Constraints:
> Colocation Constraints:
> Ticket Constraints:
> 
> Alerts:
>  No alerts defined
> 
> Resources Defaults:
>  No defaults set
> Operations Defaults:
>  No defaults set
> 
> Cluster Properties:
>  cluster-infrastructure: corosync
>  cluster-name: clusterPa
>  dc-version: 1.1.16-12.el7-94ff4df
>  have-watchdog: false
>  last-lrm-refresh: 1531899781
>  stonith-enabled: false
> 
> Quorum:
>   Options:
> [root@node1 ~]#  
> ___
> Users mailing list: Users@clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Weird Fencing Behavior

[Ken Gaillot]

On Tue, 2018-07-17 at 21:29 +0800, Confidential Company wrote:
> 
> > Hi,
> >
> > On my two-node active/passive setup, I configured fencing via
> > fence_vmware_soap. I configured pcmk_delay=0 on both nodes so I
> expected
> > that both nodes will be stonithed simultaenously.
> >
> > On my test scenario, Node1 has ClusterIP resource. When I
> disconnect
> > service/corosync link physically, Node1 was fenced and Node2 keeps
> alive
> > given pcmk_delay=0 on both nodes.
> >
> > Can you explain the behavior above?
> >
> 
> #node1 could not connect to ESX because links were disconnected. As
> the
> #most obvious explanation.
> 
> #You have logs, you are the only one who can answer this question
> with
> #some certainty. Others can only guess.
> 
> 
> Oops, my bad. I forgot to tell. I have two interfaces on each virtual
> machine (nodes). second interface was used for ESX links, so fence
> can be executed even though corosync links were disconnected. Looking
> forward to your response. Thanks

Having no fence delay means a death match (each node killing the other)
is possible, but it doesn't guarantee that it will happen. Some of the
time, one node will detect the outage and fence the other one before
the other one can react.

It's basically an Old West shoot-out -- they may reach for their guns
at the same time, but one may be quicker.

As Andrei suggested, the logs from both nodes could give you a timeline
of what happened when.

> > See my config below:
> >
> > [root@ArcosRhel2 cluster]# pcs config
> > Cluster Name: ARCOSCLUSTER
> > Corosync Nodes:
> >  ArcosRhel1 ArcosRhel2
> > Pacemaker Nodes:
> >  ArcosRhel1 ArcosRhel2
> >
> > Resources:
> >  Resource: ClusterIP (class=ocf provider=heartbeat type=IPaddr2)
> >   Attributes: cidr_netmask=32 ip=172.16.10.243
> >   Operations: monitor interval=30s (ClusterIP-monitor-interval-30s)
> >               start interval=0s timeout=20s (ClusterIP-start-
> interval-0s)
> >               stop interval=0s timeout=20s (ClusterIP-stop-
> interval-0s)
> >
> > Stonith Devices:
> >  Resource: Fence1 (class=stonith type=fence_vmware_soap)
> >   Attributes: action=off ipaddr=172.16.10.151 login=admin
> passwd=123pass
> > pcmk_host_list=ArcosRhel1 pcmk_monitor_timeout=60s
> port=ArcosRhel1(Joniel)
> > ssl_insecure=1 pcmk_delay_max=0s
> >   Operations: monitor interval=60s (Fence1-monitor-interval-60s)
> >  Resource: fence2 (class=stonith type=fence_vmware_soap)
> >   Attributes: action=off ipaddr=172.16.10.152 login=admin
> passwd=123pass
> > pcmk_delay_max=0s pcmk_host_list=ArcosRhel2
> pcmk_monitor_timeout=60s
> > port=ArcosRhel2(Ben) ssl_insecure=1
> >   Operations: monitor interval=60s (fence2-monitor-interval-60s)
> > Fencing Levels:
> >
> > Location Constraints:
> >   Resource: Fence1
> >     Enabled on: ArcosRhel2 (score:INFINITY)
> > (id:location-Fence1-ArcosRhel2-INFINITY)
> >   Resource: fence2
> >     Enabled on: ArcosRhel1 (score:INFINITY)
> > (id:location-fence2-ArcosRhel1-INFINITY)
> > Ordering Constraints:
> > Colocation Constraints:
> > Ticket Constraints:
> >
> > Alerts:
> >  No alerts defined
> >
> > Resources Defaults:
> >  No defaults set
> > Operations Defaults:
> >  No defaults set
> >
> > Cluster Properties:
> >  cluster-infrastructure: corosync
> >  cluster-name: ARCOSCLUSTER
> >  dc-version: 1.1.16-12.el7-94ff4df
> >  have-watchdog: false
> >  last-lrm-refresh: 1531810841
> >  stonith-enabled: true
> >
> > Quorum:
> >   Options:
> >
> >
> >
> > ___
> > Users mailing list: Users@clusterlabs.org
> > https://lists.clusterlabs.org/mailman/listinfo/users
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratc
> h.pdf
> > Bugs: http://bugs.clusterlabs.org
> >
> ___
> Users mailing list: Users@clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: Q: ordering clones with interleave=false

[Ken Gaillot]

On Thu, 2018-08-30 at 08:28 +0200, Ulrich Windl wrote:
> > > > Ken Gaillot  schrieb am 29.08.2018 um
> > > > 20:30 in
> 
> Nachricht
> <1535567455.5594.5.ca...@redhat.com>:
> > On Wed, 2018‑08‑29 at 13:30 +0200, Ulrich Windl wrote:
> > > Hi!
> > > 
> > > Reading the docs I have a question: WHen I run a clone with
> > > interleave=false in a three‑node cluster, and the clne cannot be
> > > started on one node, will ordering for such a clone be possible?
> > > Does
> > > it make a difference, whether the resource cannot run on an
> > > online
> > > node, or is unable due to a standby or offline node?
> > > 
> > > Regards,
> > > Ulrich
> > 
> > Interleave=false only applies to instances that will be started in
> > the
> > current transition, so offline nodes don't prevent dependent
> > resources
> > from starting on online nodes.
> 
> Ken,
> 
> thanks for responding and explaining. However as I read about "the
> transition
> thing" more than once in an answer: Where is that concept explained
> in greater
> detail?

Good question, it's not. There are additions such as a Troubleshooting
chapter planned for the new "Pacemaker Administration" document, when
that mythical available time appears, and that would be a good place
for it.

A transition is simply the set of actions Pacemaker plans in response
to a new situation. The situation is defined by the CIB (including
status section). Whenever something interesting happens (configuration
change, unexpected action result, or cluster re-check interval), the
scheduler uses the current live CIB to calculate what (if anything)
needs to be done, and that is a transition.

> 
> Regards,
> Ulrich
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Q: native_color scores for clones

[Ken Gaillot]

On Thu, 2018-08-30 at 12:23 +0200, Ulrich Windl wrote:
> Hi!
> 
> After having found showscores.sh, I thought I can improve the
> perfomance by porting it to Perl, but it seems the slow part actually
> is calling pacemakers helper scripts like crm_attribute,
> crm_failcount, etc...
> 
> But anyway: Being quite confident what my program produces (;-)), I
> found some odd score values for clones that run in a two node
> cluster. For example:
> 
> Resource   Node Score Stickin. Fail Count Migr. Thr.
> --  -  -- --
> prm_DLM:1  h02  10  0  0
> prm_DLM:1  h06  00  0  0
> prm_DLM:0  h02  -INFINITY0  0  0
> prm_DLM:0  h06  10  0  0
> prm_O2CB:1 h02  10  0  0
> prm_O2CB:1 h06  -INFINITY0  0  0
> prm_O2CB:0 h02  -INFINITY0  0  0
> prm_O2CB:0 h06  10  0  0
> prm_cfs_locks:0h02  -INFINITY0  0  0
> prm_cfs_locks:0h06  10  0  0
> prm_cfs_locks:1h02  10  0  0
> prm_cfs_locks:1h06  -INFINITY0  0  0
> prm_s02_ctdb:0 h02  -INFINITY0  0  0
> prm_s02_ctdb:0 h06  10  0  0
> prm_s02_ctdb:1 h02  10  0  0
> prm_s02_ctdb:1 h06  -INFINITY0  0  0
> 
> For prm_DLM:1 for example one node has score 0, the other node has
> score 1, but for prm:DLM:0 the host that has 1 for prm_DLM:1 has
> -INFINITY (not 0), while the other host has the usual 1. So I guess
> that even without -INFINITY the configuration would be stable. For
> prm_O2CB two nodes have -INFINITY as score. For prm_cfs_locks the
> pattern is as usual, and for rpm_s02_ctdb to nodes have -INFINITY
> again.
> 
> I don't understand where those -INFINITY scores come from. Pacemaker
> is SLES11 SP4 (1.1.12-f47ea56).

Scores are something of a mystery without tracing through code line by
line. It's similar to (though much simpler than) AI in respect of the
cause of an output being obscured by the combination of so many inputs
and processing steps.

Resources (including clone instances) are placed one by one. My guess
is that prm_DLM:1 was placed first, on h02, which made it no longer
possible for any other instance to be placed on h02 due to clone-node-
max being 1 (which I'm assuming is the case here, the default). All
instances processed after that (which is only prm_DLM:0 in this case)
get -INFINITY on h02 for that reason.

The clones with mirrored -INFINITY scores likely had additional
processing due to constraints or whatnot.

> 
> It might also be a bug, because when I look at a three-node cluster,
> I see that a ":0" resource had score 1 once, and 0 twice, but the
> corrsponding ":2" resource has scores 0, 1, and -INFINITY, and the
> ":1" resource has score 1 once and -INFINITY twice.
> 
> When I look at the "clone_solor" scores, the prm_DLM:* primitives
> look as expected (no -INFINITY). However the cln_DLM clones have
> score like 1, 8200 and 2200 (depending on the node).
> 
> Can someone explain, please?
> 
> Regards,
> Ulrich
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Pacemaker startup retries

[Ken Gaillot]

On Thu, 2018-08-30 at 17:24 +0200, Cesar Hernandez wrote:
> Hi
> 
> I have a two-node corosync+pacemaker which, starting only one node,
> it fences the other node. It's ok as the default behaviour as the
> default "startup-fencing" is set to true.
> But, the other node is rebooted 3 times, and then, the remaining node
> starts resources and doesn't fence the node anymore.
> 
> How can I change these 3 times, to, for example, 1 reboot , or more,
> 5? I use a custom fencing script so I'm sure these retries are not
> done by the script but pacemaker, and I also see the reboot
> operations on the logs:
> 
> Aug 30 17:22:08 [12978] 1       crmd:   notice: te_fence_node:
> Executing reboot fencing operation (81) on 2 (timeout=18)
> Aug 30 17:22:31 [12978] 1       crmd:   notice: te_fence_node:
> Executing reboot fencing operation (87) on 2 (timeout=18)
> Aug 30 17:22:48 [12978] 1       crmd:   notice: te_fence_node:
> Executing reboot fencing operation (89) on 2 (timeout=18)

Do you mean you have a custom fencing agent configured? If so, check
the return value of each attempt. Pacemaker should request fencing only
once as long as it succeeds (returns 0), but if the agent fails
(returns nonzero or times out), it will retry, even if the reboot
worked in reality.

If instead you mean you have a script that can request fencing (e.g.
via stonith_admin), then check the logs before each attempt to see if
the request was initiated by the cluster (which should show a policy
engine transition for it) or your script.

FYI, corosync 2 has a "two_node" setting that includes "wait_for_all"
-- with that, you don't need to ignore quorum in pacemaker, and the
cluster won't start until both nodes have seen each other at least
once.

> Software versions:
> 
> corosync-1.4.8
> crmsh-2.1.5
> libqb-0.17.2
> Pacemaker-1.1.14
> resource-agents-3.9.6
> Reusable-Cluster-Components-glue--glue-1.0.12
> 
> Some parameters:
> 
> property cib-bootstrap-options: \
>   have-watchdog=false \
>   dc-version=1.1.14-70404b0e5e \
>   cluster-infrastructure="classic openais (with plugin)" \
>   expected-quorum-votes=2 \
>   stonith-enabled=true \
>   no-quorum-policy=ignore \
>   default-resource-stickiness=200 \
>   stonith-timeout=180s \
>   last-lrm-refresh=1534489943
> 
> 
> Thanks
> 
> César Hernández Bañó
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Pacemaker startup retries

[Ken Gaillot]

On Fri, 2018-08-31 at 08:37 +0200, Cesar Hernandez wrote:
> Hi
> 
> > 
> > 
> > Do you mean you have a custom fencing agent configured? If so,
> > check
> > the return value of each attempt. Pacemaker should request fencing
> > only
> > once as long as it succeeds (returns 0), but if the agent fails
> > (returns nonzero or times out), it will retry, even if the reboot
> > worked in reality.
> > 
> 
> Yes, custom fencing agent, and it always returns 0 
> > 
> > 
> > FYI, corosync 2 has a "two_node" setting that includes
> > "wait_for_all"
> > -- with that, you don't need to ignore quorum in pacemaker, and the
> > cluster won't start until both nodes have seen each other at least
> > once.
> 
> Well I'm ok with the quorum behaviour but I want to know why it
> reboots 3 times on startup.
> When both nodes are up and running, and if one node stops responding,
> the other node fences it only 1 time, not 3
> > 
> > 
> 
> Do you know why it happens?
> 
> Thanks
> Cesar

Check the pacemaker logs on both bodes around the time it happens.

One of the nodes will be the DC, and will have "pengine:" logs with
"saving inputs".

The first thing I'd look for is who requested fencing. The DC will have
stonith logs with "Client ... wants to fence ...". The client will
either be crmd (i.e. the cluster itself) or some external program.

If it's the cluster, I'd look at the "pengine:" logs on the DC before
that, to see if there are any hints (node unclean, etc.). Then keep
going backward until the ultimate cause is found.
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Pacemaker startup retries

[Ken Gaillot]

> Aug 31 11:00:05 [30616] node1   crmd:   notice:
> te_rsc_command:Initiating action 66: start p_fs_database_start_0
> on node1 (local)
> Aug 31 11:00:05 [30616] node1   crmd: info:
> do_lrm_rsc_op: Performing key=66:2:0:c64efa2b-9366-4c07-b5f1-
> 6a2dbee79fe7 op=p_fs_database_start_0
> Aug 31 11:00:05 [30613] node1   lrmd: info:
> log_execute:   executing - rsc:p_fs_database action:start
> call_id:70
> Aug 31 11:00:05 [30616] node1   crmd:   notice:
> te_rsc_command:Initiating action 77: start p_fs_datosweb_start_0
> on node1 (local)
> Aug 31 11:00:05 [30616] node1   crmd: info:
> do_lrm_rsc_op: Performing key=77:2:0:c64efa2b-9366-4c07-b5f1-
> 6a2dbee79fe7 op=p_fs_datosweb_start_0
> Aug 31 11:00:05 [30613] node1   lrmd: info:
> log_execute:   executing - rsc:p_fs_datosweb action:start
> call_id:71
> Aug 31 11:00:05 [30611] node1cib: info:
> cib_process_request:   Completed cib_delete operation for section
> //node_state[@uname='node2']/lrm: OK (rc=0, origin=local/crmd/69,
> version=0.60.30)
> Aug 31 11:00:05 [30611] node1cib: info:
> cib_process_request:   Completed cib_delete operation for section
> //node_state[@uname='node2']/transient_attributes: OK (rc=0,
> origin=local/crmd/70, version=0.60.30)
> Aug 31 11:00:05 [30611] node1cib: info:
> cib_process_request:   Completed cib_modify operation for section
> status: OK (rc=0, origin=local/crmd/71, version=0.60.30)
> Aug 31 11:00:05 [30616] node1   crmd: info:
> cib_fencing_updated:   Fencing update 71 for node2: complete
> Aug 31 11:00:05 [30611] node1cib: info:
> cib_process_request:   Completed cib_delete operation for section
> //node_state[@uname='node2']/lrm: OK (rc=0, origin=local/crmd/72,
> version=0.60.30)
> Aug 31 11:00:05 [30611] node1cib: info:
> cib_process_request:   Completed cib_delete operation for section
> //node_state[@uname='node2']/transient_attributes: OK (rc=0,
> origin=local/crmd/73, version=0.60.30)
> Filesystem(p_fs_datosweb)[962]: 2018/08/31_11:00:05 INFO: Running
> start for /dev/drbd/by-res/datoswebstorage on /mnt/datosweb
> Filesystem(p_fs_database)[961]: 2018/08/31_11:00:05 INFO: Running
> start for /dev/drbd/by-res/databasestorage on /mnt/database
> 
> 
> ..
> 
> 
> Can you give me more clues?
> 
> Thanks a lot
> 
> Cesar
> 
> ___
> Users mailing list: Users@clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Rebooting a standby node triggers lots of transitions

[Ken Gaillot]

On Wed, 2018-09-05 at 17:43 +0200, Ulrich Windl wrote:
> > > > Kadlecsik József  schrieb am
> > > > 05.09.2018 um
> 
> 15:33 in Nachricht  i.hu>:
> > Hi,
> > 
> > For testing purposes one of our nodes was put in standby node and
> > then 
> > rebooted several times. When the standby node started up, it joined
> > the 
> > cluster as a new member and it resulted in transitions between the
> > online 
> > nodes. However, when the standby node was rebooted in
> > mid‑transitions, it 
> > triggered another transitions again. As a result, live migrations
> > was 
> > aborted and guests stopped/started.
> > 
> > How can one make sure that join/leave operations of standby nodes
> > do not 
> > affect the location of the running resources?
> > 
> > It's pacemaker 1.1.16‑1 with corosync 2.4.2‑3+deb9u1 on debian
> > stretch 
> > nodes.

Node joins/leaves do and should trigger new transitions, but that
should not result in any actions if the node is in standby.

The cluster will wait for any actions in progress (such as a live
migration) to complete before beginning a new transition, so there is
likely something else going on that is affecting the migration.

> Logs and more details, please!

Particularly the detail log on the DC should be helpful. It will have
"pengine:" messages with "saving inputs" at each transition.

> > 
> > Best regards,
> > Jozsef
> > ‑‑
> > E‑mail : kadlecsik.joz...@wigner.mta.hu 
> > PGP key: http://www.kfki.hu/~kadlec/pgp_public_key.txt 
> > Address: Wigner Research Centre for Physics, Hungarian Academy of
> > Sciences
> >  H‑1525 Budapest 114, POB. 49, Hungary
> > ___
> > Users mailing list: Users@clusterlabs.org 
> > https://lists.clusterlabs.org/mailman/listinfo/users 
> > 
> > Project Home: http://www.clusterlabs.org 
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratc
> > h.pdf 
> > Bugs: http://bugs.clusterlabs.org 
> 
> 
> 
> ___________
> Users mailing list: Users@clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: Antw: Q: native_color scores for clones

[Ken Gaillot]

On Wed, 2018-09-05 at 09:32 +0200, Ulrich Windl wrote:
> > > > Ken Gaillot  schrieb am 04.09.2018 um
> > > > 19:21 in Nachricht
> 
> <1536081690.4387.6.ca...@redhat.com>:
> > On Tue, 2018-09-04 at 11:22 +0200, Ulrich Windl wrote:
> > > > > > In Reply to my message am 30.08.2018 um 12:23 in Nachricht
> > > > > > <5B87C5A0.A46 : 161 :
> > > 
> > > 60728>:
> > > > Hi!
> > > > 
> > > > After having found showscores.sh, I thought I can improve the
> > > > perfomance by 
> > > > porting it to Perl, but it seems the slow part actually is
> > > > calling
> > > > pacemakers 
> > > > helper scripts like crm_attribute, crm_failcount, etc...
> > > 
> > > Actually the performance gain was less than expected, until I
> > > added a
> > > cache for calling external programs reading stickiness, fail
> > > count
> > > and migration threshold. Here are the numbers:
> > > 
> > > showscores.sh (original):
> > > real0m46.181s
> > > user0m15.573s
> > > sys 0m21.761s
> > > 
> > > showscores.pl (without cache):
> > > real0m46.053s
> > > user0m15.861s
> > > sys 0m20.997s
> > > 
> > > showscores.pl (with cache):
> > > real0m25.998s
> > > user0m7.940s
> > > sys 0m12.609s
> > > 
> > > This made me think whether it's possible to retrieve such
> > > attributes
> > > in a more efficient way, arising the question how the
> > > corresponding
> > > tools actually do work (those attributes are obviously not part
> > > of
> > > the CIB).
> > 
> > Actually they are ... the policy engine (aka scheduler in 2.0) has
> > only
> > the CIB to make decisions. The other daemons have some additional
> > state
> > that can affect their behavior, but the scheduling of actions
> > relies
> > solely on the CIB.
> > 
> > Stickiness and migration-threshold are in the resource
> > configuration
> > (or defaults); fail counts are in the transient node attributes in
> > the
> > status section (which can only be retrieved from the live CIB or
> > attribute daemons, not the CIB on disk, which may be why you didn't
> > see
> > them).
> 
> Looking for the fail count, the closest match I got looked like this:
>  operation_key="prm_LVM_VMD_monitor_0" operation="monitor" crm-debug-
> origin="build_active_RAs" crm_feature_set="3.0.10" transition-
> key="58:107:7:8a33be7f-1b68-45bf-9143-54414ff3b662" transition-
> magic="0:0;58:107:7:8a33be7f-1b68-45bf-9143-54414ff3b662"
> on_node="h05" call-id="143" rc-code="0" op-status="0" interval="0"
> last-run="1533809981" last-rc-change="1533809981" exec-time="89"
> queue-time="0" op-digest="22698b9dba36e2926819f13c77569222"/>

That's the most recently failed instance of the operation (for that
node and resource). It's used to show the "failed actions" section of
the crm_mon display.

> Do I have to count and filter such elements, or is there a more
> direct way to get the fail count?

The simplest way to get the fail count is with the crm_failcount tool.
That's especially true since per-operation fail counts were added in
1.1.17.

In the CIB XML, fail counts are stored within  
. The attribute name will start with "fail-count-
".

They can also be queried via attrd_updater, if you know the attribute
name.

> > > I can get the CIB via cib_admin and I can parse the XML if
> > > needed,
> > > but how can I get these other attributes? Tracing crm_attribute,
> > > it
> > > seems it reads some partially binary file from locations like
> > > /dev/shm/qb-attrd-response-*.
> > 
> > The /dev/shm files are simply where IPC communication is buffered,
> > so
> > that's just the response from a cib query (the equivalent of
> > cibadmin
> > -Q).
> > 
> > > I would think that _all_ relevant attributes should be part of
> > > the
> > > CIB...
> > 
> > Yep, they are :)
> 
> I'm still having problems, sorry.
> 
> > 
> > Often a final value is calculated from the CIB configuration,
> > rather
> > than directly in it. For example, for stickiness, the actual value
> > could be in the resource configuration, a resource template, or
> > resource defaults, or (pre-2.0) the legac

Re: [ClusterLabs] Pacemaker startup retries

[Ken Gaillot]

On Wed, 2018-09-05 at 16:38 +0200, Cesar Hernandez wrote:
> Hi
> 
> > 
> > Ah, this rings a bell. Despite having fenced the node, the cluster
> > still considers the node unseen. That was a regression in 1.1.14
> > that
> > was fixed in 1.1.15. :-(
> > 
> 
>  Oh :( I'm using Pacemaker-1.1.14.
> Do you know if this reboot retries are just run 3 times? All the
> tests I've done the rebooting is finished after 3 times.
> 
> Thanks
> Cesar

No, if I remember correctly, it would just keep going until it saw the
node. Not sure why it stops after 3.
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Pacemaker startup retries

[Ken Gaillot]

On Wed, 2018-09-05 at 09:51 -0500, Ken Gaillot wrote:
> On Wed, 2018-09-05 at 16:38 +0200, Cesar Hernandez wrote:
> > Hi
> > 
> > > 
> > > Ah, this rings a bell. Despite having fenced the node, the
> > > cluster
> > > still considers the node unseen. That was a regression in 1.1.14
> > > that
> > > was fixed in 1.1.15. :-(
> > > 
> > 
> >  Oh :( I'm using Pacemaker-1.1.14.
> > Do you know if this reboot retries are just run 3 times? All the
> > tests I've done the rebooting is finished after 3 times.
> > 
> > Thanks
> > Cesar
> 
> No, if I remember correctly, it would just keep going until it saw
> the
> node. Not sure why it stops after 3.

P.S. If the issue is just a matter of timing when you're starting both
nodes, you can start corosync on both nodes first, then start pacemaker
on both nodes. That way pacemaker on each node will immediately see the
other node's presence.
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Pacemaker startup retries

[Ken Gaillot]

On Wed, 2018-09-05 at 17:21 +0200, Cesar Hernandez wrote:
> > 
> > P.S. If the issue is just a matter of timing when you're starting
> > both
> > nodes, you can start corosync on both nodes first, then start
> > pacemaker
> > on both nodes. That way pacemaker on each node will immediately see
> > the
> > other node's presence.
> > -- 
> 
> Well rebooting a server lasts 2 minutes approximately. 
> I think I'm going to keep the same workaround I have on other
> servers:
> 
> -set crm stonith-timeout=300s
> -have a "sleep 180" in the fencing script, so the fencing will always
> last 3 minutes
> 
> So when crm fences a node on startup, the fencing script will return
> after 3 minutes. And at that time, the other node should be up and it
> won't be retried fencing
> 
> What you think about this workaround?
> 
> 
> The other solution would be updating pacemaker, but this 1.1.14 I
> have tested on many servers, and I don't want to take the risk to
> update to 1.1.15 and (maybe) have some other new issues...
> 
> Thanks a lot!
> Cesar

If you build from source, you can apply the patch that fixes the issue
to the 1.1.14 code base:

https://github.com/ClusterLabs/pacemaker/commit/98457d1635db1222f93599b6021e662e766ce62d
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] About fencing stonith

[Ken Gaillot]

On Thu, 2018-09-06 at 17:33 -0300, Marcos Renato da Silva Junior wrote:
> Hi,
> I created a testing environment based on Raspberry Pi (attached
> diagram), in my tests it has worked well, but I did not implement
> anything about fencing and stonith, what is the need and how to
> implement?
> Thanks,
> Marcos.

The need for fencing is to be able to recover from problems that
disrupt normal cluster operation.

For example, if one node has the IP but becomes overloaded and can't
respond to requests, the cluster can't stop the IP on the unresponsive
node. If the cluster brings up the IP on another node anyway (which it
will do if there's no fencing), packets will randomly go to one or the
other node, causing the service to fail. With fencing, the cluster
powers down the unresponsive node, and can safely bring up the IP
elsewhere.

> corosync.conf :
> totem {
>     version: 2
>     cluster_name: debian
>     token: 3000
>     token_retransmits_before_loss_const: 10
>     clear_node_high_bit: yes
>     crypto_cipher: aes256
>     crypto_hash: sha1
>     interface {
>         bindnetaddr: 192.168.0.0
>         mcastaddr: 239.255.1.1
>         mcastport: 5405
>         ttl: 1
>     }
> }
> 
> logging {
>     fileline: off
>     to_stderr: no
>     to_logfile: no
>     to_syslog: yes
>     syslog_facility: daemon
>     debug: off
>     timestamp: on
>     logger_subsys {
>         subsys: QUORUM
>         debug: off
>     }
> }
> 
> quorum {
>     provider: corosync_votequorum
>     expected_votes: 2
>     two_node: 1
> }
> 
> crm configure property stonith-enabled=false
> 
> crm configure property no-quorum-policy=ignore
> 
> crm configure primitive LDAP-IP ocf:heartbeat:IPaddr2 \
> params ip="192.168.0.10" nic="eth0" cidr_netmask="24" \
> op monitor interval=10s timeout=20s
> crm configure primitive LDAP ocf:heartbeat:slapd params \
>  slapd="/usr/sbin/slapd" \
>  config="/etc/ldap/slapd.d/" \
>  user="openldap" group="openldap" \
>  services="ldap:///; \
>  watch_suffix="dc=acme,dc=lab" \
>  bind_dn="cn=admin,dc=acme,dc=lab" \
>  password="password" \
>  op monitor interval=10s timeout=20s
> crm configure clone LDAP-CLONE LDAP
> crm configure colocation LDAP-IP_WITH_LDAP inf: LDAP-IP LDAP-CLONE
> 
> crm configure order LDAP-IP_BEFORE_LDAP inf: LDAP-IP LDAP-CLONE
> 
> crm configure location PREFER_RASP4_LDAP LDAP-IP 50: rasp4
> 
> crm configure primitive RADIUS-IP ocf:heartbeat:IPaddr2 \
> params ip="192.168.0.9" nic="eth0" cidr_netmask="24" \
> op monitor interval=10s timeout=20s
> crm configure primitive RADIUS lsb:freeradius op monitor interval=10s
> timeout=20s
> crm configure clone RADIUS-CLONE RADIUS
> crm configure colocation RADIUS-IP_WITH_RADIUS inf: RADIUS-IP RADIUS-
> CLONE
> 
> crm configure order RADIUS-IP_BEFORE_RADIUS inf: RADIUS-IP RADIUS-
> CLONE
> 
> crm configure location PREFER_RASP4_RADIUS RADIUS-IP 50: rasp4
> 
> 
> diagram.jpg
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Q: ordering clones with interleave=false

[Ken Gaillot]

On Wed, 2018-08-29 at 13:30 +0200, Ulrich Windl wrote:
> Hi!
> 
> Reading the docs I have a question: WHen I run a clone with
> interleave=false in a three-node cluster, and the clne cannot be
> started on one node, will ordering for such a clone be possible? Does
> it make a difference, whether the resource cannot run on an online
> node, or is unable due to a standby or offline node?
> 
> Regards,
> Ulrich

Interleave=false only applies to instances that will be started in the
current transition, so offline nodes don't prevent dependent resources
from starting on online nodes.
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Q: Resource Groups vs Resources for stickiness and colocation?

[Ken Gaillot]

On Wed, 2018-08-29 at 18:40 +0100, Ian Underhill wrote:
> im guessing this is just a "feature", but something that will
> probably stop me using groups
> 
> Scenario1 (working):
> 1) Two nodes (1,2) within a cluster (default-stickiness = INFINITY)
> 2) Two resources (A,B) in a cluster running on different nodes 
> 3) colocation constraint between resources of A->B score=-1
> 
> a) pcs standby node2, the resource B moves to node 1
> b) pcs unstandby node2, the resource B stays on node 1 - this is good
> and expected
> 
> Secanrio 2 (working):
> 1) exactly the same as above but the resource exist within their own
> group (G1,G2)
> 2) the colocation constraint is between the groups
> 
> Secanrio 3 (not working):
> 1) Same as above however each group has two resources in them
> 
>  Resource Group: A_grp
>      A(ocf::test:fallover):   Started mac-devl03
>      A_2  (ocf::test:fallover):   Started mac-devl03
>  Resource Group: B_grp
>      B(ocf::test:fallover):   Started mac-devl11
>      B_2  (ocf::test:fallover):   Started mac-devl11
> 
> a) pcs standby node2, the group moves to node 1
> b) pcs unstandby node2, the group moves to node 2, but I have
> INFINITY stickiness (maybe I need INFINITY+1 ;) )
> 
> crm_simulate -sL doesnt really explain why there is a difference.
> 
> any ideas?  (environment pacemaker-cluster-libs-1.1.16-12.el7.x86_64)
> 
> /Ian

This sounds like a bug. Feel free to submit a report at
bugs.clusterlabs.org and attach the policy engine input file with the
unexpected behavior.

FYI a group's stickiness is the sum of the stickiness of each active
member, though no score can be bigger than INFINITY.
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Problem with pacemaker resources when NTP sync is done

[Ken Gaillot]

On Wed, 2018-07-04 at 10:10 +0530, Pankaj wrote:
> Hi,
> 
> We have facing problem pacemaker when ntp time sync is done. When
> time jumps forwards, we have seen that fail-count timer would not
> expire as expected. There are other issues like pacemaker discarding
> old events.
> 
> It seems pacemaker is not using monotonic clock.
> If this is the case, is it possible to use monotonic clock or if any
> patch is available or if there is any build option?
> If there is any workaround to handle this, please let me know.

Currently, no, pacemaker has no way to handle large time jumps. Jumps
forward aren't too bad, but jumps backward can cause significant
trouble.

> #  pacemakerd --version
> Pacemaker 1.1.16
> Written by Andrew Beekhof
> # corosync -v
> Corosync Cluster Engine, version '2.4.2'
> Copyright (c) 2006-2009 Red Hat, Inc.
> 
> Regards,
> Pankaj
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Cluster from scratch - 7.6. Configure the Cluster for the DRBD device

[Ken Gaillot]

On Tue, 2018-07-03 at 16:06 +0200, cont...@eplex.eu wrote:
> Hi everyone
> I followed step by step this tutorial http://clusterlabs.org/pacemake
> r/doc/en-US/Pacemaker/1.1/pdf/Clusters_from_Scratch/Pacemaker-1.1-
> Clusters_from_Scratch-en-US.pdf and 
> I have stuck on this point and I'm looking for a help with this
>  
> after I run this command:
> pcs -f drbd_cfg resource create WebData2 ocf:linbit:drbd
> drbd_resource=wwwdata op monitor interval=60s
> I get error like below
> Error: Agent 'ocf:linbit:drbd' is not installed or does not provide
> valid metadata: Metadata query for ocf:linbit:drbd failed:
> Input/output error, use --force to override
> To this point everything works fine
> Maciek

After doing "7.1. Install the DRBD Packages", you should be able to
verify that /usr/lib/ocf/resource.d/linbit/drbd exists. I believe the
note about the version shipped with CentOS 7.1 is no longer an issue
with recent versions.
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: OCF Return codes OCF_NOT_RUNNING

[Ken Gaillot]

On Wed, 2018-07-11 at 13:44 +0200, Ulrich Windl wrote:
> > > > Ian Underhill  schrieb am 11.07.2018
> > > > um 13:27 in
> 
> Nachricht
> :
> > im trying to understand the behaviour of pacemaker when a resource
> > monitor
> > returns OCF_NOT_RUNNING instead of OCF_ERR_GENERIC, and does
> > pacemaker
> > really care.
> > 
> > The documentation states that a return code OCF_NOT_RUNNING from a
> > monitor
> > will not result in a stop being called on that resource, as it
> > believes the
> > node is still clean.
> > 
> > https://www.clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html/
> > Pacemaker 
> > _Explained/s-ocf-return-codes.html
> > 
> > This makes sense, however in practice is not what happens (unless
> > im doing
> > something wrong :) )
> > 
> > When my resource returns OCF_NOT_RUNNING for a monitor call (after
> > a start
> > has been performed) a stop is called.
> 
> Well: it depends: If your start was successful, pacemaker believes
> the resource is running. If the monitor says it's stopped, pacemaker
> seems to try a "clean stop" by calling the stop method (possibly
> before trying to start it again). Am I right?

Yes, I think the documentation is wrong. It depends on what state the
cluster thinks the resource is supposed to be in. If the cluster
expects the resource is already stopped (for example, when doing a
probe), then "not running" will not result in a stop. If the cluster
expects the resource to be running (for example, when doing a normal
recurring monitor), then the documentation is incorrect, recovery
includes a stop then start.

> > if I have a resource threshold set >1,  i get start->monitor->stop
> > cycle
> > until the threshold is consumed
> 
> Then either your start is broken, or your monitor is broken. Try to
> validate your RA using ocf-tester before using it.
> 
> Regards,
> Ulrich
> 
> > 
> > /Ian.
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] What triggers fencing?

[Ken Gaillot]

orking reliably.
> > > > You just have to know your fencing-devices and
> > > > which delays they involve.
> > > > 
> > > > If we are talking about SBD (with disk as otherwise
> > > > it doesn't work in a sensible way in 2-node-clusters)
> > > > for instance I would strongly advise using a delay.
> > > > 
> > > > So I guess it is important to understand the basic
> > > > idea behind this different delay-based fence-race
> > > > avoidance.
> > > > Afterwards you can still decide why it is no issue
> > > > in your own setup.
> > > > 
> > > > > > > If the delay is set on both nodes, and they are
> > > > > > > different, it will work
> > > > > > > fine. The reason not to do this is that if you use 0,
> > > > > > > then don't use
> > > > > > > anything at all (0 is default), and any other value
> > > > > > > causes avoidable
> > > > > > > fence delays.
> > > > > > > 
> > > > > > > > > Don't use a delay on the other node, just the node
> > > > > > > > > you want to live
> > > > 
> > > > in
> > > > > > > > > such a case.
> > > > > > > > > 
> > > > > > > > > > > *Given Node1 is active and Node2 goes down, does
> > > > > > > > > > > it mean fence1
> > > > 
> > > > will
> > > > > > > > > > > first execute and shutdowns Node1 even though
> > > > > > > > > > > Node2 goes down?
> > > > > > > > > > 
> > > > > > > > > > If Node2 managed to sign off properly it will not.
> > > > > > > > > > If network-connection is down so that Node2 can't
> > > > > > > > > > inform Node1 that
> > > > 
> > > > it
> > > > > > > > > > is going
> > > > > > > > > > down and finally has stopped all resources it will
> > > > > > > > > > be fenced by
> > > > 
> > > > Node1.
> > > > > > > > > > Regards,
> > > > > > > > > > Klaus
> > > > > > > > > 
> > > > > > > > > Fencing occurs in two cases;
> > > > > > > > > 
> > > > > > > > > 1. The node stops responding (meaning it's in an
> > > > > > > > > unknown state, so
> > > > 
> > > > it is
> > > > > > > > > fenced to force it into a known state).
> > > > > > > > > 2. A resource / service fails to stop stop. In this
> > > > > > > > > case, the
> > > > 
> > > > service is
> > > > > > > > > in an unknown state, so the node is fenced to force
> > > > > > > > > the service into
> > > > 
> > > > a
> > > > > > > > > known state so that it can be safely recovered on the
> > > > > > > > > peer.
> > > > > > > > > 
> > > > > > > > > Graceful withdrawal of the node from the cluster, and
> > > > > > > > > graceful
> > > > 
> > > > stopping
> > > > > > > > > of services will not lead to a fence (because in both
> > > > > > > > > cases, the
> > > > 
> > > > node /
> > > > > > > > > service are in a known state - off).
> > > > > > > > > 
> > > 
> > > 
> > > ___
> > > Users mailing list: Users@clusterlabs.org
> > > https://lists.clusterlabs.org/mailman/listinfo/users
> > > 
> > > Project Home: http://www.clusterlabs.org
> > > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scra
> > > tch.pdf
> > > Bugs: http://bugs.clusterlabs.org
> > > 
> > 
> > ___
> > Users mailing list: Users@clusterlabs.org
> > https://lists.clusterlabs.org/mailman/listinfo/users
> > 
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratc
> > h.pdf
> > Bugs: http://bugs.clusterlabs.org
> 
> ___
> Users mailing list: Users@clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Problem with pacemaker init.d script

[Ken Gaillot]

On Wed, 2018-07-11 at 18:43 +0200, Salvatore D'angelo wrote:
> Hi,
> 
> Yes that was clear to me, but question is pacemaker install
> /etc/init.d/pacemaker script but its header is not compatible with
> newer system that uses LSB.
> So if pacemaker creates scripts in /etc/init.d it should create them
> so that they are compatible with OS supported (not sure if Ubuntu is
> one).
> when I run “make install” anything is created for systemd env.

With Ubuntu 16, you should use "systemctl enable pacemaker" instead of
update-rc.d.

The pacemaker configure script should have detected that the OS uses
systemd and installed the appropriate unit file.

> I am not a SysV vs System expert, hoping I haven’t said anything
> wrong.
> 
> > On 11 Jul 2018, at 18:40, Andrei Borzenkov 
> > wrote:
> > 
> > 11.07.2018 18:08, Salvatore D'angelo пишет:
> > > Hi All,
> > > 
> > > After I successfully upgraded Pacemaker from 1.1.14 to 1.1.18 and
> > > corosync from 2.3.35 to 2.4.4 on Ubuntu 14.04 I am trying to
> > > repeat the same scenario on Ubuntu 16.04.
> > 
> > 16.04 is using systemd, you need to create systemd unit. I do not
> > know
> > if there is any compatibility layer to interpret upstart
> > configuration
> > like the one for sysvinit.
> > 
> > > As my previous scenario I am using Docker for test purpose before
> > > move to Bare metal.
> > > The scenario worked properly after I downloaded the correct
> > > dependencies versions.
> > > 
> > > The only problem I experienced is that in my procedure install I
> > > set corosync and pacemaker to run at startup updating the init.d
> > > scripts with this commands:
> > > 
> > > update-rc.d corosync defaults
> > > update-rc.d pacemaker defaults 80 80
> > > 
> > > I noticed that links in /etc/rc are not created.
> > > 
> > > I have also the following errors on second update-rc.d command:
> > > insserv: Service corosync has to be enabled to start service
> > > pacemaker
> > > insserv: exiting now!
> > > 
> > > I was able to solve the issue manually replacing these lines in
> > > /etc/init.d/corosync and /etc/init.d/pacemaker:
> > > # Default-Start:
> > > # Default-Stop:
> > > 
> > > with this:
> > > # Default-Start:    2 3 4 5
> > > # Default-Stop: 0 1 6
> > > 
> > > I didn’t understand if this is a bug of corosync or pacemaker or
> > > simply there is a dependency missing on Ubuntu 16.04 that was
> > > installed by default on 14.04. I found other discussion on this
> > > forum about this problem but it’s not clear the solution.
> > > Thanks in advance for support.
> > > 
> > > 
> > > 
> > > 
> > > ___
> > > Users mailing list: Users@clusterlabs.org
> > > https://lists.clusterlabs.org/mailman/listinfo/users
> > > 
> > > Project Home: http://www.clusterlabs.org
> > > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scra
> > > tch.pdf
> > > Bugs: http://bugs.clusterlabs.org
> > > 
> > 
> > ___
> > Users mailing list: Users@clusterlabs.org
> > https://lists.clusterlabs.org/mailman/listinfo/users
> > 
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratc
> > h.pdf
> > Bugs: http://bugs.clusterlabs.org
> 
> ___
> Users mailing list: Users@clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] Pacemaker 1.1.19 released

[Ken Gaillot]

Source code for the final release of Pacemaker version 1.1.19 is
available at:

https://github.com/ClusterLabs/pacemaker/releases/tag/Pacemaker-1.1.19

This is a maintenance release that backports selected fixes and
features from the 2.0.0 version. The 1.1 series is no longer actively
maintained, but is still supported for users (and distributions) who
want to keep support for features dropped by the 2.0 series (such as
CMAN or heartbeat as the cluster layer).

The most significant changes in this release are:

* stonith_admin has a new --validate option to validate option
configurations

* .mount, .path, and .timer unit files are now supported as "systemd:"-
class resources

* 5 regressions in 1.1.17 and 1.1.18 have been fixed

For a more detailed list of bug fixes and other changes, see the change
log:

https://github.com/ClusterLabs/pacemaker/blob/1.1/ChangeLog

Many thanks to all contributors of source code to this release,
including Andrew Beekhof, Gao,Yan, Hideo Yamauchi, Jan Pokorný, Ken
Gaillot, and Klaus Wenninger.
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Problem with pacemaker init.d script

[Ken Gaillot]

On Wed, 2018-07-11 at 22:07 +0300, Andrei Borzenkov wrote:
> 11.07.2018 21:01, Salvatore D'angelo пишет:
> > Yes, but doing what you suggested the system find that sysV is
> > installed and try to leverage on update-rc.d scripts and the
> > failure occurs:
> 
> Then you built corosync without systemd integration. systemd will
> prefer
> native units.
> 
> > 
> > root@pg1:~# systemctl enable corosync
> > corosync.service is not a native service, redirecting to systemd-
> > sysv-install
> > Executing /lib/systemd/systemd-sysv-install enable corosync
> > update-rc.d: error: corosync Default-Start contains no runlevels,
> > aborting.
> > 
> > the only fix I found was to manipulate manually the header of
> > /etc/init.d/corosync adding the rows mentioned below.
> > But this is not a clean approach to solve the issue.
> > 
> > What pacemaker suggest for newer distributions?

Red Hat (Fedora/RHEL/CentOS) and SuSE provide enterprise support for
pacemaker, and regularly contribute code upstream, so those and their
derivatives tend to be the most stable. Debian has a good team of
volunteers that greatly improved support in the current (stretch)
release, so I suppose Ubuntu will eventually pick that up. I know
people have compiled on Arch Linux, FreeBSD, OpenBSD, and likely
others, but that usually takes some extra work.

> > 
> > If you look at corosync code the init/corosync file does not
> > container run levels in header.
> > So I suspect it is a code problem. Am I wrong?
> > 
> 
> Probably not. Description of special comments in LSB standard imply
> that
> they must contain at least one value. Also how should service manager
> know for which run level to enable service without it? It is amusing
> that this problem was first found on a distribution that does not
> even
> use SysV for years ...

I'm not sure. Pacemaker packages are intended to be installed without
enabling at boot, since so much configuration must be done first. So
maybe the idea was to always require someone to specify run levels. But
it does make more sense that they would be listed in the LSB header.
One reason it wouldn't have been an issue before is some older distros
use the init script's chkconfig header instead of the LSB header.


> > > On 11 Jul 2018, at 19:29, Ken Gaillot 
> > > wrote:
> > > 
> > > On Wed, 2018-07-11 at 18:43 +0200, Salvatore D'angelo wrote:
> > > > Hi,
> > > > 
> > > > Yes that was clear to me, but question is pacemaker install
> > > > /etc/init.d/pacemaker script but its header is not compatible
> > > > with
> > > > newer system that uses LSB.
> > > > So if pacemaker creates scripts in /etc/init.d it should create
> > > > them
> > > > so that they are compatible with OS supported (not sure if
> > > > Ubuntu is
> > > > one).
> > > > when I run “make install” anything is created for systemd env.
> > > 
> > > With Ubuntu 16, you should use "systemctl enable pacemaker"
> > > instead of
> > > update-rc.d.
> > > 
> > > The pacemaker configure script should have detected that the OS
> > > uses
> > > systemd and installed the appropriate unit file.
> > > 
> > > > I am not a SysV vs System expert, hoping I haven’t said
> > > > anything
> > > > wrong.
> > > > 
> > > > > On 11 Jul 2018, at 18:40, Andrei Borzenkov  > > > > om>
> > > > > wrote:
> > > > > 
> > > > > 11.07.2018 18:08, Salvatore D'angelo пишет:
> > > > > > Hi All,
> > > > > > 
> > > > > > After I successfully upgraded Pacemaker from 1.1.14 to
> > > > > > 1.1.18 and
> > > > > > corosync from 2.3.35 to 2.4.4 on Ubuntu 14.04 I am trying
> > > > > > to
> > > > > > repeat the same scenario on Ubuntu 16.04.
> > > > > 
> > > > > 16.04 is using systemd, you need to create systemd unit. I do
> > > > > not
> > > > > know
> > > > > if there is any compatibility layer to interpret upstart
> > > > > configuration
> > > > > like the one for sysvinit.
> > > > > 
> > > > > > As my previous scenario I am using Docker for test purpose
> > > > > > before
> > > > > > move to Bare metal.
> > > > > > The scenario worked properly after I downloaded the correct
> > > > &g

Re: [ClusterLabs] Pacemaker alert framework

[Ken Gaillot]

On Fri, 2018-07-06 at 15:58 +0200, Klaus Wenninger wrote:
> On 07/06/2018 03:41 PM, Ian Underhill wrote:
> > requirement:
> > when a resource fails perform an actionm, run a script on all nodes
> > within the cluster, before the resource is relocated. i.e.
> > information gathering why the resource failed.
>  
> Have in mind that trying to run a script on all nodes in the cluster
> before
> proceeding is a delicate issue because not being able to run it on
> one
> node might prevent relocation and thus availability.
> Of course this largely depends on how it is implemented - just wanted
> to raise attention.
>   
> > what I have looked into:
> > 1) Use the monitor call within the resource to SSH to all nodes,
> > again SSH config needed.
> > 2) Alert framework : this only seems to be triggered for nodes
> > involved in the relocation of the resource. i.e. if resource moves
> > from node1 to node 2 node 3 doesnt know. so back to the SSH
> > solution :(
>  
> Alerts are designed not to block anything (no other alerts as well)
> so the alert-agents are 
> just called on nodes that anyway already have to do with that very
> event.
> 
> > 3) sending a custom alert to all nodes in the cluster? is this
> > possible? not found a way?
> > 
> > only solution I have:
> > 1) use SSH within an alert monitor (stop) to SSH onto all nodes to
> > perform the action, the nodes could be configured using the alert
> > monitors recipients, but I would still need to config SSH users and
> > certs etc.
> >      1.a) this doesnt seem to be usable if the resource is
> > relocated back to the same node, as the alerts start\stop are run
> > at the "same time". i.e I need to delay the start till the SSH has
> > completed.
> > 
> > what I would like:
> > 1) delay the start\relocation of the resource until the information
> > from all nodes is complete, using only pacemaker behaviour\config
> > 
> > any ideas?
>  
> Not 100% sure of what what your exact intention is ...
> You could experiment with a clone that depends on the running
> instance and
> use the stop of that to trigger whatever you need.
> Not sure but I'd expect that pacemaker would tear down all clone
> instances
> before it relocates your resource.

Yes, that sounds like a good solution. See clone notifications:

http://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html-single/Pa
cemaker_Explained/index.html#_clone_resource_agent_requirements

You could even combine everything into a single custom resource agent
for use as a master/slave resource, where the master is the only
instance that actually runs the resource, and the slaves just act on
the notifications.

> 
> Regards,
> Klaus
> 
> > Thanks
> > 
> > /Ian.
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] Pacemaker 2.0.0 has been released

[Ken Gaillot]

I am very happy to announce that source code for the final release of
Pacemaker version 2.0.0 is now available at:

https://github.com/ClusterLabs/pacemaker/releases/tag/Pacemaker-2.0.0

The main goal of the change from Pacemaker 1 to 2 is to drop support
for deprecated legacy usage, in order to make the code base more
maintainable going into the future.

Rolling (live) upgrades are possible only from Pacemaker 1.1.11 or
later, on top of corosync 2 or later. Other setups can be upgraded with
the cluster stopped.

If upgrading an existing cluster, it is recommended to run "cibadmin --
upgrade" (or the equivalent in your higher-level tool of choice) both
before and after the upgrade.

Extensive details about the changes in this release are available in
the change log:

  https://github.com/ClusterLabs/pacemaker/blob/2.0/ChangeLog

and in a special wiki page for the 2.0 release:

  https://wiki.clusterlabs.org/wiki/Pacemaker_2.0_Changes

Highlights:

* Support has been dropped for heartbeat and corosync 1 (whether using
CMAN or plugin), and many legacy aliases for cluster options (including
default-resource-stickiness, which should be set as resource-
stickiness in rsc_defaults instead).

* The logs should be a little more user-friendly. The Pacemaker daemons
have been renamed for easier log searching. The default location of the
Pacemaker detail log is now /var/log/pacemaker/pacemaker.log, and
Pacemaker will no longer use Corosync's logging preferences.

* The master XML tag is deprecated (though still supported) in favor of
using the standard clone tag with a new "promotable" meta-attribute set
to true. The "master-max" and "master-node-max" master meta-attributes
are deprecated in favor of new "promoted-max" and "promoted-node-max"
clone meta-attributes. Documentation now refers to these as promotable
clones rather than master/slave, stateful or multistate clones.

* The record-pending option now defaults to true, which means pending
actions will be shown in status displays.

* The "Pacemaker Explained" document has grown large enough that topics
related to cluster administration have been moved to their own new
document, "Pacemaker Administration":

  http://clusterlabs.org/pacemaker/doc/

Many thanks to all contributors of source code to this release,
including Andrew Beekhof, Bin Liu, Bruno Travouillon, Gao,Yan, Hideo
Yamauchi, Jan Pokorný, Ken Gaillot, and Klaus Wenninger.
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Clearing failed actions

[Ken Gaillot]

On Mon, 2018-07-09 at 09:11 +0200, Jehan-Guillaume de Rorthais wrote:
> On Fri, 06 Jul 2018 10:15:08 -0600
> Casey Allen Shobe  wrote:
> 
> > Hi,
> > 
> > I found a web page which suggested to clear the Failed Actions, to
> > use
> > `crm_resource -P`.  Although this appears to work, it's not
> > documented on the
> > man page at all.  Is this deprecated and is there a more correct
> > way to be
> > doing this?
> 
> -P means "reprobe", so I guess this is a side effect or a pre-
> requisit, but not
> only to clean failcounts.

In the 1.1 series, -P is a deprecated synonym for --cleanup / -C. The
options clear fail counts and resource operation history (for a
specific resource and/or node if specified with -r and/or -N, otherwise
all).

In the 2.0 series, -P is gone. --refresh / -R now does what cleanup
used to; --cleanup / -C now cleans up only resources that have had
failures. In other words, the old --cleanup and new --refresh clean
resource history, forcing a re-probe, regardless of whether a resource
failed or not, whereas the new --cleanup will skip resources that
didn't have failures. 

> > Also, is there a way to clear one specific item from the list, or
> > is clearing
> > all the only option?
> 
> pcs failcount reset  [node]

With the low level tools, you can use -r / --resource and/or -N / --
node with crm_resource to limit the clean-up.
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Clone resource active only if all nodes are active

[Ken Gaillot]

On Mon, 2018-01-22 at 14:18 +0100, Ulrich Windl wrote:
> Did you try  meta clone-node-min=3?

clone-min does not affect the clone it's configured on, but rather
anything ordered relative to it via a constraint. It's for the case
where a resource needs a certain number of instances running before
other services can consider it effective.

It could be helpful here if the reason you want the service to stop is
to get some other resource to stop.

> "alu...@poczta.onet.pl" <alu...@poczta.onet.pl> schrieb am
> > > > 22.01.2018 um 13:29
> 
> in Nachricht <844fdf99-1680-3ed8-6afc-8b3e2ddea...@poczta.onet.pl>:
> > I need to create configuration when one resource is active on all
> > nodes
> > and is only active when all nodes are active. It should has the 

There's no built-in capability for that.

It's possible you could get something to work using clone
notifications. You could write a custom OCF agent that simply sets a
local node attribute to 0 or 1. If
OCF_RESKEY_CRM_meta_notify_inactive_resource is empty, it would set it
to 1, otherwise it would set it to 0. (You could use
ocf:pacemaker:attribute from a recent Pacemaker as a starting point.)

Then you could use a rule to locate your desired resource where the
attribute is set to 1.


http://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html-single/Pa
cemaker_Explained/index.html#_clone_resource_agent_requirements

http://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html-single/Pa
cemaker_Explained/index.html#idm140583511697312

> > same
> > priority on all nodes. The major target is to have my service
> > started
> > when all nodes are active and to have my service stopped in
> > "degraded"
> > state.
> > 
> > I tried clone resource with different requires options but when one
> > of
> > the node is suspended resource is still active on online node. 
> > 
> > Is there any options to have such type of resource?
> 
> 
> ___
> Users mailing list: Users@clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
-- 
Ken Gaillot <kgail...@redhat.com>

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Resources stopped due to unmanage

[Ken Gaillot]

On Mon, 2018-03-12 at 22:36 +0300, Pavel Levshin wrote:
> Hello.
> 
> 
> I've just expiriensed a fault in my pacemaker-based cluster.
> Seriously, 
> I'm completely disoriented after this. Hopefully someone can give me
> a 
> hint...
> 
> 
> Two-node cluster runs few VirtualDomains along with their common 
> infrastructure (libvirtd, NFS and so on). It is Pacemaker 1.1.16 
> currently. Resources have ordering and colocation constraints, which 
> must ensure proper start and stop order. Unfortunately, these 
> constraints have unwanted side effects. In particular, due to
> mandatory 
> ordering constraints, the cluster tends to restart libvirtd when I
> need 
> to stop a VM. This was on the list prevoiusly: 
> https://lists.clusterlabs.org/pipermail/users/2016-October/004288.htm
> l
> 
> 
> And today I've tried to perform maintenance task on one of VMs. I've
> typed:
> 
> pcs resource unmanage vm3
> 
> and all other VMs have been suddenly stopped. Seriously?!!!
> 
> Logs show that the cluster performed internal custom_action
> "vm3_stop 
> (unmanaged)" and "vm3_start (unmanaged)", and then this triggered 
> libvirtd_stop, which leads to every VM stopping due to colocation.

That's really perplexing. Unmanaging by itself should never lead to any
actions being done.

Feel free to report it at bugs.clusterlabs.org, and attach the output
of a crm_report.

> The question is: is there a sane way to run VMs under pacemaker's 
> control? If yes, is it described somewhere?
> 
> 
> --
> Pavel
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] copy file

[Ken Gaillot]

On Thu, 2018-03-08 at 18:49 +0100, Mevo Govo wrote:
> Hi, 
> thanks for advice and your intrest.
> We would use oracle database over DRBD. Datafiles (and control and
> redo files) will be on DRBD. FRA also (on an other DRBD device). But
> we are new in DRBD, and DRBD is also a component what can fails. We
> plan a scenario to recover the database without DRBD (without data
> loss, if possible). We would use nfs or local filesystem for this. If
> we use local FS, the control file is out of sync on the B node when
> switch over (from A to B). We would copy controlfile (and redo files)
> from DRBD to the local FS. After this, oracle can start, and it keeps
> the controlfiles syncronized. If other backup related files (archlog,
> backup) are also available on the local FS of either node, we can
> recover the DB without DRBD (without data loss)
> (I know it is a worst case scenario, because if DRBD fails, the FS on
> it should be available at least on one node)
> Thanks: lados.

Why not use native database replication instead of copying files?

Any method getting files from a DRBD cluster to a non-DRBD node will
have some inherent problems: it would have to be periodic, losing some
data since the last run; it would still fail if some DRBD issue
corrupted the on-disk data, because you would be copying the corrupted
data; and databases generally have in-memory state information that
makes files copied from a live server insufficient for data integrity.

Native replication would avoid all that.

> 2018-03-07 10:20 GMT+01:00 Klaus Wenninger <kwenn...@redhat.com>:
> > On 03/07/2018 10:03 AM, Mevo Govo wrote:
> > > Thanks for advices, I will try!
> > > lados.
> > >
> > > 2018-03-05 23:29 GMT+01:00 Ken Gaillot <kgail...@redhat.com
> > > <mailto:kgail...@redhat.com>>:
> > >
> > >     On Mon, 2018-03-05 at 15:09 +0100, Mevo Govo wrote:
> > >     > Hi,
> > >     > I am new in pacemaker. I think, I should use DRBD instead
> > of copy
> > >     > file. But in this case, I would copy a file from a DRBD to
> > an
> > >     > external device. Is there a builtin way to copy a file
> > before a
> > >     > resource is started (and after the DRBD is promoted)? For
> > example a
> > >     > "copy" resource? I did not find it. 
> > >     > Thanks: lados.
> > >     >
> > >
> > >     There's no stock way of doing that, but you could easily
> > write an
> > >     agent
> > >     that simply copies a file. You could use ocf:pacemaker:Dummy
> > as a
> > >     template, and add the copy to the start action. You can use
> > standard
> > >     ordering and colocation constraints to make sure everything
> > happens in
> > >     the right sequence.
> > >
> > >     I don't know what capabilities your external device has, but
> > another
> > >     approach would be to an NFS server to share the DRBD file
> > system, and
> > >     mount it from the device, if you want direct access to the
> > original
> > >     file rather than a copy.
> > >
> > 
> > csync2 & rsync might be considered as well although not knowing
> > your scenario in detail it is hard to tell if it would be overkill.
> > 
> > Regards,
> > Klaus
> > 
> > >     --
> > >     Ken Gaillot <kgail...@redhat.com <mailto:kgail...@redhat.com>
> > >
> > >     ___
> > >     Users mailing list: Users@clusterlabs.org
> > >     <mailto:Users@clusterlabs.org>
> > >     https://lists.clusterlabs.org/mailman/listinfo/users
> > >     <https://lists.clusterlabs.org/mailman/listinfo/users>
> > >
> > >     Project Home: http://www.clusterlabs.org
> > >     Getting started:
> > >     http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > >     <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
> > >     Bugs: http://bugs.clusterlabs.org
> > >
> > >
> > >
> > >
> > > ___
> > > Users mailing list: Users@clusterlabs.org
> > > https://lists.clusterlabs.org/mailman/listinfo/users
> > >
> > > Project Home: http://www.clusterlabs.org
> > > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scra
> > tch.pdf
> > > Bugs: http://bugs.clusterlabs.org
> > 
> > ___
> > Users mailing list: Users@clusterlabs

Re: [ClusterLabs] [Problem]The pengine core dumps when changing attributes of bundle.

[Ken Gaillot]

On Sat, 2018-03-10 at 05:47 +0900, renayama19661...@ybb.ne.jp wrote:
> Hi All, 
> 
> [Sorry..There was a defect in line breaks. to send again.]
> 
> I was checking the operation of Bundle with Pacemaker version 2.0.0-
> 9cd0f6cb86. 
> When Bundle resource is configured in Pacemaker and attribute is
> changed, pengine core dumps. 

Hi Hideo,

At first glance, it's confusing. The backtrace shows that
find_container_child() is being called with a NULL rsc, but I don't see
how it's possible to call it that way.

We'll investigate further and get back on the BZ

> 
> Step1) Start Pacemaker and pour in the settings. (The replicas and
> replicas-per-host are set to 1.) 
> 
> [root@rh74-test ~]# cibadmin --modify --allow-create --scope
> resources -X '
>   replicas-per-host="1" options="--log-driver=journald" />  range-start="192.168.20.188" host-interface="ens192" host-
> netmask="24">  
>   root="/var/local/containers" target-dir="/var/www/html"
> options="rw"/>  root="/var/log/pacemaker/bundles" target-dir="/etc/httpd/logs"
> options="rw"/>   provider="heartbeat" type="apache" >id="rabbitmq-start-interval-0s" interval="0s" name="start"
> timeout="200s"/>  name="stop" timeout="200s" on-fail="fence" /> 
> 
> ' 
> 
> Step2) Bundle is configured. 
> 
> [root@rh74-test ~]# crm_mon -1 -Af
> Stack: corosync
> Current DC: rh74-test (version 2.0.0-9cd0f6cb86) - partition WITHOUT
> quorum
> Last updated: Fri Mar  9 10:09:20 2018
> Last change: Fri Mar  9 10:06:30 2018 by root via cibadmin on rh74-
> test 2 nodes configured
> 
> 4 resources configured Online: [ rh74-test ]
> GuestOnline: [ httpd-bundle-0@rh74-test ] 
> 
> Active resources: 
> Docker container: httpd-bundle [pcmktest:http] httpd-bundle-0
> (192.168.20.188)      (ocf::heartbeat:apache):        
> 
> Started rh74-test Node Attributes:
> * Node httpd-bundle-0@rh74-test:
> * Node rh74-test: Migration Summary:
> * Node rh74-test:
> * Node httpd-bundle-0@rh74-test: 
> 
> Step3) Change attributes of bundle with cibadmin command. (The
> replicas and replicas-per-host change to 3.)
> 
> 
> [root@rh74-test ~]# cibadmin --modify -X ' image="pcmktest:http" replicas="3" replicas-per-host="3" options="
> --log-driver=journald"/>' 
> 
> Step4) The pengine will core dump. (snip)
> Mar  9 10:10:21 rh74-test pengine[17726]:  notice: On loss of quorum:
> Ignore
> Mar  9 10:10:21 rh74-test pengine[17726]:    info: Node rh74-test is
> online
> Mar  9 10:10:21 rh74-test crmd[17727]:  error: Connection to pengine
> failed
> Mar  9 10:10:21 rh74-test crmd[17727]:  error: Connection to
> pengine[0x55f2d068bfb0] closed (I/O condition=25)
> Mar  9 10:10:21 rh74-test pacemakerd[17719]:  error: Managed process
> 17726 (pengine) dumped core
> Mar  9 10:10:21 rh74-test pacemakerd[17719]:  error: pengine[17726]
> terminated with signal 11 (core=1)
> Mar  9 10:10:21 rh74-test pacemakerd[17719]:  notice: Respawning
> failed child process: pengine
> Mar  9 10:10:21 rh74-test pacemakerd[17719]:    info: Using uid=990
> and group=984 for process pengine
> Mar  9 10:10:21 rh74-test pacemakerd[17719]:    info: Forked child
> 19275 for process pengine
> (snip) 
> 
> This event reproduces 100 percent. 
> 
> Apparently the problem seems to be due to different handling of
> clone(httpd) resources in the Bundle resource. 
> 
> - I registered this content with the following Bugzilla.
> (https://bugs.clusterlabs.org/show_bug.cgi?id=5337)
> 
> Best Regards
> Hideo Yamauchi.
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] corosync 2.4 CPG config change callback

[Ken Gaillot]

On Fri, 2018-03-09 at 17:26 +0100, Jan Friesse wrote:
> Thomas,
> 
> > Hi,
> > 
> > On 3/7/18 1:41 PM, Jan Friesse wrote:
> > > Thomas,
> > > 
> > > > First thanks for your answer!
> > > > 
> > > > On 3/7/18 11:16 AM, Jan Friesse wrote:
> 
> ...
> 
> > TotemConfchgCallback: ringid (1.1436)
> > active processors 3: 1 2 3
> > EXIT
> > Finalize  result is 1 (should be 1)
> > 
> > 
> > Hope I did both test right, but as it reproduces multiple times
> > with testcpg, our cpg usage in our filesystem, this seems like
> > valid tested, not just an single occurrence.
> 
> I've tested it too and yes, you are 100% right. Bug is there and
> it's 
> pretty easy to reproduce when node with lowest nodeid is paused.
> It's 
> slightly harder when node with higher nodeid is paused.
> 
> Most of the clusters are using power fencing, so they simply never
> sees 
> this problem. That may be also the reason why it wasn't reported
> long 
> time ago (this bug exists virtually at least since OpenAIS
> Whitetank). 
> So really nice work with finding this bug.
> 
> What I'm not entirely sure is what may be best way to solve this 
> problem. What I'm sure is, that it's going to be "fun" :(
> 
> Lets start with very high level of possible solutions:
> - "Ignore the problem". CPG behaves more or less correctly.
> "Current" 
> membership really didn't changed so it doesn't make too much sense
> to 
> inform about change. It's possible to use cpg_totem_confchg_fn_t to
> find 
> out when ringid changes. I'm adding this solution just for
> completeness, 
> because I don't prefer it at all.
> - cpg_confchg_fn_t adds all left and back joined into left/join list
> - cpg will sends extra cpg_confchg_fn_t call about left and joined 
> nodes. I would prefer this solution simply because it makes cpg
> behavior 
> equal in all situations.
> 
> Which of the options you would prefer? Same question also for @Ken (-

Pacemaker should react essentially the same whichever of the last two
options is used. There could be differences due to timing (the second
solution might allow some work to be done between when the left and
join messages are received), but I think it should behave reasonably
with either approach.

Interestingly, there is some old code in Pacemaker for handling when a
node left and rejoined but "the cluster layer didn't notice", that may
have been a workaround for this case.

> > 
> what would you prefer for PCMK) and @Chrissie.
> 
> Regards,
>    Honza
> 
> 
> > 
> > cheers,
> > Thomas
> > 
> > > > 
> > > > > Now it's really cpg application problem to synchronize its
> > > > > data. Many applications (usually FS) are using quorum
> > > > > together with fencing to find out, which cluster partition is
> > > > > quorate and clean inquorate one.
> > > > > 
> > > > > Hopefully my explanation help you and feel free to ask more
> > > > > questions!
> > > > > 
> > > > 
> > > > They help, but I'm still a bit unsure about why the CB could
> > > > not happen here,
> > > > may need to dive a bit deeper into corosync :)
> > > > 
> > > > > Regards,
> > > > >    Honza
> > > > > 
> > > > > > 
> > > > > > help would be appreciated, much thanks!
> > > > > > 
> > > > > > cheers,
> > > > > > Thomas
> > > > > > 
> > > > > > [1]: https://git.proxmox.com/?p=pve-cluster.git;a=tree;f=da
> > > > > > ta/src;h=e5493468b456ba9fe3f681f387b4cd5b86e7ca08;hb=HEAD
> > > > > > [2]: https://git.proxmox.com/?p=pve-cluster.git;a=blob;f=da
> > > > > > ta/src/dfsm.c;h=cdf473e8226ab9706d693a457ae70c0809afa0fa;hb
> > > > > > =HEAD#l1096
> > > > > > 
> > > > 
> > > > 
> > > > 
> > > 
> > > 
> > 
> > 
> > 
> 
> 
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] Pacemaker 2.0.0-rc2 now available

[Ken Gaillot]

Source code for the second release candidate for Pacemaker version
2.0.0 is now available at:

https://github.com/ClusterLabs/pacemaker/releases/tag/Pacemaker-2.0.0-r
c2

The main feature of this release is an automatic transform for pre-
2.0.0 configurations to the latest syntax. Because this is a fairly
heavyweight operation, it is recommended to upgrade the CIB (via
"cibadmin --upgrade" or a higher-level tool equivalent) both before
*and* after upgrading from a pre-2.0 version to this version.

Rolling (live) upgrades are possible only from Pacemaker 1.1.11 or
later, on top of corosync 2 or 3. Other setups can be upgraded with the
cluster stopped.

There were also a number of bugs fixed. More details are available in
the change log:

  https://github.com/ClusterLabs/pacemaker/blob/2.0/ChangeLog

and in a special wiki page for the 2.0 release:

  https://wiki.clusterlabs.org/wiki/Pacemaker_2.0_Changes

I expect one more release candidate with significant changes, most
importantly the new daemon names discussed earlier on this list. The
final release will hopefully be not long after that.

Everyone is encouraged to download, compile and test the new release.
We do many regression tests and simulations, but we can't cover all
possible use cases, so your feedback is important and appreciated.

Many thanks to all contributors of source code to this release,
including Andrew Beekhof, Gao,Yan, Hideo Yamauchi, Jan Pokorný, and Ken
Gaillot. Thanks also to Fabio DiNitto and Jiří Belka for help getting a
successful build on *BSD systems.
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Possible idea for 2.0.0: renaming the Pacemaker daemons

[Ken Gaillot]

Based on the list discussion and feedback I could coax out of others, I
will change the Pacemaker daemon names, including the log tags, for
2.0.0-rc3.

I will add symlinks for the old names, to allow help/version/metadata
calls in user scripts and higher-level tools to continue working during
a transitional time. (Even if we update all known tools, we need to
keep compatibility with existing versions for a good while.)

I won't change the systemd unit file names or API library names, since
they aren't one-to-one with the daemons, and will have a bigger impact
on client apps.

Here's my current plan:

Old name  New name
  
pacemakerdpacemakerd
attrd pacemaker-attrd
cib   pacemaker-confd
crmd  pacemaker-controld
lrmd  pacemaker-execd
pengine   pacemaker-schedulerd
stonithd  pacemaker-fenced
pacemaker_remoted pacemaker-remoted

I had planned to use the "pcmk-" prefix, but I kept thinking about the
goal of making things more intuitive for novice users, and a novice
user's first instinct will be to search the logs for "pacemaker". Most
of the names stay under the convenient 15-character limit anyway.

On Wed, 2018-03-28 at 12:40 -0500, Ken Gaillot wrote:
> Hi all,
> 
> Andrew Beekhof brought up a potential change to help with reading
> Pacemaker logs.
> 
> Currently, pacemaker daemon names are not intuitive, making it
> difficult to search the system log or understand what each one does.
> 
> The idea is to rename the daemons, with a common prefix, and a name
> that better reflects the purpose.
> 
> I think it's a great idea, but we have to consider two drawbacks:
> 
> * I'm about to release 2.0.0-rc2, and it's late in the cycle for a
> major change. But if we don't do it now, it'll probably sit on the
> back
> burner for a few years, as it wouldn't make sense to introduce such a
> change shortly after a major bump.
> 
> * We can change *only* the names used in the logs, which will be
> simple, but give us inconsistencies with what shows up in "ps", etc.
> Or
> we can try to change everything -- process names, library names, API
> function/structure names -- but that will impact other projects such
> as
> sbd, crmsh, etc., potentially causing compatibility headaches.
> 
> What are your thoughts? Change or not? Now or later? Log tags, or
> everything?
> 
> And the fun part, what would we change them to ...
> 
> Beekhof suggested renaming "pengine" to "cluster-planner", as an
> example.
> 
> I think a prefix indicating pacemaker specifically would be better
> than
> "cluster-" for grepping and intuitiveness.
> 
> For intuitiveness, long names are better ("pacemaker-FUNCTION"). On
> the
> other hand, there's an argument for keeping names to 15 characters,
> which is the default "ps" column width, and a reasonable limit for
> log
> line tags. Maybe "pm-" or "pcmk-"? This prefix could also be used for
> library names.
> 
> Looking at other projects with server processes, most use the
> traditional "d" ending (for example, "rsyslogd"). A few add "-daemon"
> ("rtkit-daemon"), and others don't bother with any suffix ("gdm").
> 
> Here are the current names, with some example replacements:
> 
>  pacemakerd: PREFIX-launchd, PREFIX-launcher
> 
>  attrd: PREFIX-attrd, PREFIX-attributes
> 
>  cib: PREFIX-configd, PREFIX-state
> 
>  crmd: PREFIX-controld, PREFIX-clusterd, PREFIX-controller
> 
>  lrmd: PREFIX-locald, PREFIX-resourced, PREFIX-runner
> 
>  pengine: PREFIX-policyd, PREFIX-scheduler
> 
>  stonithd: PREFIX-fenced, PREFIX-stonithd, PREFIX-executioner
> 
>  pacemaker_remoted: PREFIX-remoted, PREFIX-remote
> 
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Possible idea for 2.0.0: renaming the Pacemaker daemons

[Ken Gaillot]

On Tue, 2018-04-10 at 08:50 +0200, Jehan-Guillaume de Rorthais wrote:
> On Tue, 10 Apr 2018 00:54:01 +0200
> Jan Pokorný <jpoko...@redhat.com> wrote:
> 
> > On 09/04/18 12:10 -0500, Ken Gaillot wrote:
> > > Based on the list discussion and feedback I could coax out of
> > > others, I
> > > will change the Pacemaker daemon names, including the log tags,
> > > for
> > > 2.0.0-rc3.
> > > 
> > > I will add symlinks for the old names, to allow
> > > help/version/metadata
> > > calls in user scripts and higher-level tools to continue working
> > > during
> > > a transitional time. (Even if we update all known tools, we need
> > > to
> > > keep compatibility with existing versions for a good while.)
> > > 
> > > I won't change the systemd unit file names or API library names,
> > > since
> > > they aren't one-to-one with the daemons, and will have a bigger
> > > impact
> > > on client apps.
> > > 
> > > Here's my current plan:
> > > 
> > > Old name  New name
> > >   
> > > pacemakerdpacemakerd
> > > attrd pacemaker-attrd
> > > cib   pacemaker-confd  
> > 
> > Let's restate it: do we indeed want to reinforce a misnomer that
> > CIB
> > is (user) configuration only?
> 
> Agree. FWIW, +1 for the "Infod" suggestion.

I'm not opposed to it, but no option suggested so far intuitively
covers what the cib does (including "cib"). All the daemons maintain
information of some sort for querying and setting -- attrd maintains
node attributes, stonithd maintains fence history, lrmd maintains a
list of registered resources, etc.

-confd, -cfgd, or -configd would emphasize the configuration aspect of
cib's workload, at the cost of hiding the dynamic status aspect.

-infod, -datad, or -stated (or cib) would emphasize the broadness of
cib's workload, at the cost of being vague and including aspects of
other daemons' responsibilities.

-iod would emphasize cib's role as a disk I/O abstraction layer, at the
 cost of downplaying the more essential configuration+status roles.

Given all that, I'm leaning to -confd because configuration management
is the cib's primary responsibility. The CIB stored on disk is entirely
configuration (status is only in-memory), so it seems most intuitive to
me. Keep in mind that the primary purpose of this renaming is to make
the log files easier to read, so picture the name in front of a typical
CIB log message (also considering that "info" is a log severity).

But if there's a consensus otherwise, I'm willing to change it.

> 
> > > crmd  pacemaker-controld
> > > lrmd  pacemaker-execd
> > > pengine   pacemaker-schedulerd
> > > stonithd  pacemaker-fenced
> > > pacemaker_remoted pacemaker-remoted
> > > 
> > > I had planned to use the "pcmk-" prefix, but I kept thinking
> > > about the
> > > goal of making things more intuitive for novice users, and a
> > > novice
> > > user's first instinct will be to search the logs for
> > > "pacemaker"  
> > 
> > journalctl -u pacemaker?
> > 
> > We could also ship an example syslog configuration that aggegrates
> > messages from enumerated programs (that we know and user may not
> > offhand)
> > into a dedicated file (well, this would be quite redundant to
> > native
> > logging into the file).
> > 
> > IOW, I wouldn't worry that much.
> 
> +1
> 
> My 2¢...
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] STONITH forever?

[Ken Gaillot]

On Tue, 2018-04-10 at 07:26 +, Stefan Schlösser wrote:
> Hi,
>  
> I have a 3 node setup on ubuntu 16.04. Corosync/Pacemaker services
> are not started automatically.
>  
> If I put all 3 nodes to offline mode, with 1 node in an „unclean“
> state I get a never ending STONITH.
>  
> What happens is that the STONITH causes a reboot of the unclean node.
>  
> 1) I would have thought with all nodes in standby no STONITH can
> occur. Why does it?

Standby prevents a node from running resources, but it still
participates in quorum voting. I suspect *starting* a node in standby
mode would prevent it from using fence devices, but *changing* a node
to standby will have no effect on whether it can fence.

> 2) Why does it keep on killing the unclean node?

Good question. The DC's logs will have the most useful information --
each pengine run should say why fencing is being scheduled.

>  
> The only way to stop it, is to temporarily disable stonith and bring
> the unclean node back online manually, and the enable it again.
>  
> Here is a log extract of node c killing node a:
> Apr 10 09:08:30 [2276] xxx-c stonith-ng:   notice: log_operation:  
> Operation 'reboot' [2428] (call 5 from crmd.2175) for host 'xxx-a'
> with device 'stonith_a' returned: 0 (OK)
> Apr 10 09:08:30 [2276] xxx-c stonith-ng:   notice: remote_op_done: 
> Operation reboot of xxx-a by xxx-c for crmd.2175@xxx-b.20531831: OK
> Apr 10 09:08:30 [2275] xxx-c    cib: info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=xxx-b/crmd/83, version=0.164.37)
> Apr 10 09:08:30 [2275] xxx-c    cib: info:
> cib_process_request: Completed cib_delete operation for section
> //node_state[@uname='xxx-a']/lrm: OK (rc=0, origin=xxx-b/crmd/84,
> version=0.164.37)
> Apr 10 09:08:30 [2275] xxx-c    cib: info:
> cib_process_request: Completed cib_delete operation for section
> //node_state[@uname='xxx-a']/transient_attributes: OK (rc=0,
> origin=xxx-b/crmd/85, version=0.164.37)
> Apr 10 09:08:30 [2275] xxx-c    cib: info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=xxx-b/crmd/86, version=0.164.37)
> Apr 10 09:08:30 [2275] xxx-c    cib: info:
> cib_process_request: Completed cib_delete operation for section
> //node_state[@uname='xxx-a']/lrm: OK (rc=0, origin=xxx-b/crmd/87,
> version=0.164.37)
> Apr 10 09:08:30 [2275] xxx-c    cib: info:
> cib_process_request: Completed cib_delete operation for section
> //node_state[@uname='xxx-a']/transient_attributes: OK (rc=0,
> origin=xxx-b/crmd/88, version=0.164.37)
>  
> This the repeats forevermore ...
>  
> Thanks for any hints,
>  
> cheers,
>  
> Stefan
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: Possible idea for 2.0.0: renaming the Pacemaker daemons

[Ken Gaillot]

On Wed, 2018-04-11 at 08:49 +0200, Ulrich Windl wrote:
> > > > Ken Gaillot <kgail...@redhat.com> schrieb am 09.04.2018 um
> > > > 19:10 in Nachricht
> 
> <1523293841.5734.7.ca...@redhat.com>:
> > Based on the list discussion and feedback I could coax out of
> > others, I
> > will change the Pacemaker daemon names, including the log tags, for
> > 2.0.0-rc3.
> > 
> > I will add symlinks for the old names, to allow
> > help/version/metadata
> > calls in user scripts and higher-level tools to continue working
> > during
> > a transitional time. (Even if we update all known tools, we need to
> > keep compatibility with existing versions for a good while.)
> > 
> > I won't change the systemd unit file names or API library names,
> > since
> > they aren't one-to-one with the daemons, and will have a bigger
> > impact
> > on client apps.
> > 
> > Here's my current plan:
> > 
> > Old name  New name
> >   
> > pacemakerdpacemakerd
> > attrd pacemaker-attrd
> > cib   pacemaker-confd
> > crmd  pacemaker-controld
> > lrmd  pacemaker-execd
> > pengine   pacemaker-schedulerd
> > stonithd  pacemaker-fenced
> > pacemaker_remoted pacemaker-remoted
> 
> I think the common "pacemaker-" prefix is too long. "pcmkr-" instead?

If we're going to shorten it, I'd go with "pcmk-", since that is
already in use in multiple places (PCMK_* environment variables, pcmk_*
fence device parameters, some API function names).

> [...]
> > 
> > I had planned to use the "pcmk-" prefix, but I kept thinking about
> > the
> > goal of making things more intuitive for novice users, and a novice
> > user's first instinct will be to search the logs for "pacemaker".
> > Most
> > of the names stay under the convenient 15-character limit anyway.
> 
> If the user searches logs before reading the docs, the user has a
> more severe problem IMHO.

Even after reading docs, when someone starts troubleshooting a problem,
they're going to look at the logs (which may be grepping a file,
running journalctl, typing a string into a log collector's search bar,
etc.). Anyone's first instinct will be to search for the name of the
program. People will learn to adapt to whatever we pick if they use it
often enough, but the goal of this rename is to make things intuitive
enough that people don't need to remember bits of pacemaker trivia (and
the person filling in for the person who normally maintains the cluster
while they're on vacation doesn't pull their hair out).

The other advantage of full "pacemaker" is that we can keep the name of
the master process (pacemakerd) the same. That seems to be a strong
preference. Otherwise pcmk-initd would be the likely choice for that.

I'm still leaning to "pacemaker-", but if there's a strong sentiment
for "pcmk-", there's still time to switch (though not much, I'm working
on this now and hope to have rc3 out next week).

> 
> [...]
> 
> Regards,
> Ulrich
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] 答复: No slave is promoted to be master

[Ken Gaillot]

On Thu, 2018-04-12 at 07:29 +, 范国腾 wrote:
> Hello,
>  
> We use the following command to create the cluster. Node2 is always
> the master when the cluster starts. Why does pacemaker not select
> node1 as the default master?
> How to configure if we want node1 to be the default master?

You can specify a location constraint giving pgsqld's master role a
positive score (not INFINITY) on node1, or a negative score (not
-INFINITY) on node2. Using a non-infinite score in a constraint tells
pacemaker that it's a preference, but not a requirement.

However, there's rarely a good reason to do that. In HA, the best
practice is that all nodes should be completely interchangeable, so
that the service can run equally on any node (since it might have to,
in a failure scenario). Such constraints can be useful temporarily,
e.g. if you need to upgrade the software on one node, or if one node is
underperforming (perhaps it's waiting on a hardware upgrade to come in,
or running some one-time job consuming a lot of CPU).

>  
> pcs cluster setup --name cluster_pgsql node1 node2
> pcs resource create pgsqld ocf:heartbeat:pgsqlms
> bindir=/usr/local/pgsql/bin pgdata=/home/postgres/data op start
> timeout=600s op stop timeout=60s op promote timeout=300s op demote
> timeout=120s op monitor interval=15s timeout=100s role="Master" op
> monitor interval=16s timeout=100s role="Slave" op notify
> timeout=60s;pcs resource master pgsql-ha pgsqld notify=true
> interleave=true;
>  
>  
> Sometimes it reports the following error, how to configure to avoid
> it?
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] General Capabilities Question

[Ken Gaillot]

On Thu, 2018-04-12 at 21:09 -0700, Cliff Burdick wrote:
> Hi, I had a general question about Pacemaker to see if it would work
> for a somewhat unique situation. I have a cluster of 10 active
> machines + 2 standby that each have 3 interfaces (2 control, 1
> management). I want each of the control interfaces to use virtual
> IPs, such that if any of those 10 nodes fail, one of the two standby
> nodes can assume the virtual IPs of the failed node. For the virtual
> IPs, I wanted the failover to happen entirely in userspace and
> essentially just receive a notification that a node failed, and allow
> my application on the standby node to do all of the virtual IP
> reconfiguration. By userspace I mean that these interfaces are
> typically not configured by Linux, so the failure of those interfaces
> and the configuration would be done by a user script.
> 
> The reason I'm looking at pacemaker is it appears to be robust on
> node failure detections, and the STONITH is something I'd like to do.
> Is this something pacemaker could be configured to do, and if so,
> which part of the documentation should I focus on?

Yes, that's what Pacemaker is designed for. I recommend starting with
the "Clusters from Scratch" document to get familiar with the basics.
Then you can look at "Pacemaker Explained" to get detailed info about
particular configuration options (especially constraints).

I'm not sure what your interface/VIP stuff does, but you probably want
to write your own OCF resource agent (see IPaddr2 as an example) to
manage the IP, and let Pacemaker call it as needed.
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] How to cancel a fencing request?

[Ken Gaillot]

On Tue, 2018-04-10 at 00:02 +0200, Jehan-Guillaume de Rorthais wrote:
> On Tue, 03 Apr 2018 17:35:43 -0500
> Ken Gaillot <kgail...@redhat.com> wrote:
> 
> > On Tue, 2018-04-03 at 21:46 +0200, Klaus Wenninger wrote:
> > > On 04/03/2018 05:43 PM, Ken Gaillot wrote:  
> > > > On Tue, 2018-04-03 at 07:36 +0200, Klaus Wenninger wrote:  
> > > > > On 04/02/2018 04:02 PM, Ken Gaillot wrote:  
> > > > > > On Mon, 2018-04-02 at 10:54 +0200, Jehan-Guillaume de
> > > > > > Rorthais
> > > > > > wrote:  
> 
> [...]
> > > > > 
> > > > > -inf constraints like that should effectively prevent
> > > > > stonith-actions from being executed on that nodes.  
> > > > 
> > > > It shouldn't ...
> > > > 
> > > > Pacemaker respects target-role=Started/Stopped for controlling
> > > > execution of fence devices, but location (or even whether the
> > > > device is
> > > > "running" at all) only affects monitors, not execution.
> > > >   
> > > > > Though there are a few issues with location constraints
> > > > > and stonith-devices.
> > > > > 
> > > > > When stonithd brings up the devices from the cib it
> > > > > runs the parts of pengine that fully evaluate these
> > > > > constraints and it would disable the stonith-device
> > > > > if the resource is unrunable on that node.  
> > > > 
> > > > That should be true only for target-role, not everything that
> > > > affects
> > > > runnability  
> > > 
> > > cib_device_update bails out via a removal of the device if
> > > - role == stopped
> > > - node not in allowed_nodes-list of stonith-resource
> > > - weight is negative
> > > 
> > > Wouldn't that include a -inf rule for a node?  
> > 
> > Well, I'll be ... I thought I understood what was going on there.
> > :-)
> > You're right.
> > 
> > I've frequently seen it recommended to ban fence devices from their
> > target when using one device per target. Perhaps it would be better
> > to
> > give a lower (but positive) score on the target compared to the
> > other
> > node(s), so it can be used when no other nodes are available. you
> > could
> > re-manage.  
> 
> Wait, you mean a fencing resource can be triggered from its own
> target? Wat
> happen then? Node suicide and all the cluster nodes are shutdown?
> 
> Thanks,

A node can fence itself, though it will be the cluster's last resort
when no other node can. It doesn't necessarily imply all other nodes
are shut down ... there may be other nodes up, but they are not allowed
execute the relevant fence device for whatever reason. But of course
there might be no other nodes up, in which case, yes, the cluster dies
(the idea being that the node is known to be malfunctioning, so stop it
from possibly corrupting data).
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Failing operations immediately when node is known to be down

[Ken Gaillot]

On Tue, 2018-04-10 at 12:56 -0500, Ryan Thomas wrote:
> I’m trying to implement a HA solution which recovers very quickly
> when a node fails.  It my configuration, when I reboot a node, I see
> in the logs that pacemaker realizes the node is down, and decides to
> move all resources to the surviving node.  To do this, it initiates a
> ‘stop’ operation on each of the resources to perform the move.  The
> ‘stop’ fails as expected after 20s (the default action timeout). 
> However, in this case, with the node known to be down,  I’d like to
> avoid this 20 second delay.  The node is known to be down, so any
> operations sent to the node will fail.  It would be nice if
> operations sent to a down node would immediately fail, thus reducing
> the time it takes the resource to be started on the surviving node.
>  I do not want to reduce the timeout for the operation, because the
> timeout is sensible for when a resource moves due to a non-node-
> failure.  Is there a way to accomplish this? 
> 
> Thanks for your help.

How are you rebooting -- cleanly (normal shutdown) or simulating a
failure (e.g. power button)?

In a normal shutdown, pacemaker will move all resources off the node
before it shuts down. These operations shouldn't fail, because the node
isn't down yet.

When a node fails, corosync should detect this and notify pacemaker.
Pacemaker will not try to execute any operations on a failed node.
Instead, it will fence it.

What log messages do you see from corosync and pacemaker indicating
that the node is down? Do you have fencing configured and tested?
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] How can I prevent multiple start of IPaddr 2 in an environment using fence_mpath?

[Ken Gaillot]

 via cibadmin on
> x3650e
> 
>   2 nodes configured
>   13 resources configured
> 
>   Node x3650f: UNCLEAN (offline)
>   Online: [ x3650e ]
> 
>   Full list of resources:
> 
>    fenceMpath-x3650e(stonith:fence_mpath):  Started x3650e
>    fenceMpath-x3650f(stonith:fence_mpath):  Started[ x3650e
> x3650f ]
>    Resource Group: grpPostgreSQLDB
>    prmFsPostgreSQLDB1   (ocf::heartbeat:Filesystem):Start
> ed x3650e
>    prmFsPostgreSQLDB2   (ocf::heartbeat:Filesystem):Start
> ed x3650e
>    prmFsPostgreSQLDB3   (ocf::heartbeat:Filesystem):Start
> ed x3650e
>    prmApPostgreSQLDB(ocf::heartbeat:pgsql): Stopped
>    Resource Group: grpPostgreSQLIP
>    prmIpPostgreSQLDB(ocf::heartbeat:IPaddr2):   Stopp
> ed
>    Clone Set: clnDiskd1 [prmDiskd1]
>    prmDiskd1(ocf::pacemaker:diskd): Started x3650f
> (UNCLEAN)
>    Started: [ x3650e ]
>    Clone Set: clnDiskd2 [prmDiskd2]
>    prmDiskd2(ocf::pacemaker:diskd): Started x3650f
> (UNCLEAN)
>    Started: [ x3650e ]
>    Clone Set: clnPing [prmPing]
>    prmPing  (ocf::pacemaker:ping):  Started x3650f (UNCLEAN)
>    Started: [ x3650e ]
> 
>   Node Attributes:
>   * Node x3650e:
>   + default_ping_set: 100
>   + diskcheck_status: normal
>   + diskcheck_status_internal   : normal
> 
>   Migration Summary:
>   * Node x3650e:
>  prmApPostgreSQLDB: migration-threshold=1 fail-count=1 last-
> failure='Fri Apr  6 13:16:39 2018'
> 
>   Failed Actions:
>   * prmApPostgreSQLDB_monitor_1 on x3650e 'not running' (7):
> call=60, status=complete, exitreason='Configuration file
> /dbfp/pgdata/data/postgresql.conf doesn't exist',
>   last-rc-change='Fri Apr  6 13:16:39 2018', queued=0ms, exec=0ms
> ==
> 
> We regard this behavior as a problem.
> Is there a way to avoid this behavior?
> 
> Regards, Yusuke

Hi Yusuke,

One possibility would be to implement network fabric fencing as well,
e.g. fence_snmp with an SNMP-capable network switch. You can make a
fencing topology level with both the storage and network devices.

The main drawback is that unfencing isn't automatic. After a fenced
node is ready to rejoin, you have to clear the block at the switch
yourself.
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] 答复: No slave is promoted to be master

[Ken Gaillot]

On Wed, 2018-04-18 at 02:33 +, 范国腾 wrote:
> Thank you very much, Rorthais,
>  
> I see now. I have two more questions.
>  
> 1. If I change the "cluster-recheck-interval" parameter from the
> default 15 minutes to 10 seconds, is there any bad impact? Could this
> be a workaround?

Personally I wouldn't lower it below 5 minutes. The policy engine will
run at least this often, so setting it very low means the PE will run
continuously, putting unnecessary CPU and I/O load on the cluster.

>  
> 2. This issue happens only in the following configuration.
> 
> But it does not happen in the following configuration. Why is the
> behaviors different?

I'm not sure. I don't know of any differences in those two versions (or
two-node vs. three-node) that would affect this issue.

> -邮件原件-
> 发件人: Jehan-Guillaume de Rorthais [mailto:j...@dalibo.com] 
> 发送时间: 2018年4月17日 17:47
> 收件人: 范国腾 <fanguot...@highgo.com>
> 抄送: Cluster Labs - All topics related to open-source clustering
> welcomed <users@clusterlabs.org>
> 主题: Re: [ClusterLabs] No slave is promoted to be master
>  
> On Tue, 17 Apr 2018 04:16:38 +
> 范国腾 <fanguot...@highgo.com> wrote:
>  
> > I check the status again. It is not not promoted but it promoted
> about
> > 15 minutes after the cluster starts.
> >
> > I try in three labs and the results are same: The promotion happens
> 15
> > minutes after the cluster starts.
> >
> > Why is there about 15 minutes delay every time?
>  
> This was a bug in Pacemaker up to 1.1.17. I did a report about this
> last August and Ken Gaillot fixed it few days later in 1.1.18. See:
>  
> https://lists.clusterlabs.org/pipermail/developers/2017-August/001110
> .html
> https://lists.clusterlabs.org/pipermail/developers/2017-September/001
> 113.html
>  
> I wonder if disabling the pgsql resource before shutting down the
> cluster might be a simpler and safer workaround. Eg.:
>  
> pcs resource disable pgsql-ha  --wait
> pcs cluster stop --all
>  
> and
>  
> pcs cluster start --all
> pcs resource enable pgsql-ha
>  
> Another fix would be to force a master score on one node **if
> needed** using:
>  
>   crm_master -N  -r  -l forever -v 1
>  
> ___
> Users mailing list: Users@clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Pacemaker resources are not scheduled

[Ken Gaillot]

ut=20 \
> >> op stop interval=0 timeout=20 \
> >> op monitor timeout=20s interval=10s depth=0 \
> >> meta migration-threshold=3 failure-timeout=60s
> >> [...]
> >> colocation inetmanager_col +inf: inetmanager_vip inetmanager
> >> order inetmanager_order Mandatory: inetmanager inetmanager_vip
> >> 
> >> [...]
> >> 
> >> $ crm status
> >> [...]
> >> Full list of resources:
> >> [...]
> >>  inetmanager_vip(ocf::heartbeat:IPaddr2):   Stopped
> >>  inetmanager(ocf::heartbeat:inetmanager):   Stopped
> >> 
> >> [...]
> >> 
> >> corosync.log of node 122.0.1.10
> >> Apr 13 23:49:56 [6137] paas-controller-122-0-1-
> 10   crmd:  warning: status_from_rc: Action 24
> (inetmanager_monitor_0) on 122.0.1.9 failed (target: 7 vs. rc: 1):
> Error
> >> Apr 13 23:49:56 [6137] paas-controller-122-0-1-
> 10   crmd: info: abort_transition_graph: Transition aborted
> by operation inetmanager_monitor_0 'modify' on 122.0.1.9: Event
> failed | magic=0:1;24:360:7:a7901eb1-462f-4259-a613-e0023ce8a6be
> cib=0.124.2400 source=match_graph_event:310 complete=false
> >> Apr 13 23:49:56 [6137] paas-controller-122-0-1-
> 10   crmd: info: match_graph_event:  Action
> inetmanager_monitor_0 (24) confirmed on 122.0.1.9 (rc=1)
> >> Apr 13 23:49:56 [6137] paas-controller-122-0-1-
> 10   crmd: info: process_graph_event:Detected action
> (360.24) inetmanager_monitor_0.2152=unknown error: failed
> >> Apr 13 23:49:56 [6137] paas-controller-122-0-1-
> 10   crmd:  warning: status_from_rc: Action 24
> (inetmanager_monitor_0) on 122.0.1.9 failed (target: 7 vs. rc: 1):
> Error
> >> Apr 13 23:49:56 [6137] paas-controller-122-0-1-
> 10   crmd: info: abort_transition_graph: Transition aborted
> by operation inetmanager_monitor_0 'modify' on 122.0.1.9: Event
> failed | magic=0:1;24:360:7:a7901eb1-462f-4259-a613-e0023ce8a6be
> cib=0.124.2400 source=match_graph_event:310 complete=false
> >> Apr 13 23:49:56 [6137] paas-controller-122-0-1-
> 10   crmd: info: match_graph_event:  Action
> inetmanager_monitor_0 (24) confirmed on 122.0.1.9 (rc=1)
> >> Apr 13 23:49:56 [6137] paas-controller-122-0-1-
> 10   crmd: info: process_graph_event:Detected action
> (360.24) inetmanager_monitor_0.2152=unknown error: failed
> fasd
> > What causes inetmanager agent to return 1 (OCF_ERR_GENERIC) when
> > 7 (OCF_NOT_RUNNING) is expected?  It may be a trivial issue in the
> > implementation of the agent, making the whole group together with
> > "inetmanager_vip" resource fail (due to the respective
> constraints).
> 
> > It may be similar with other isolated sets of resources.
> 
> > You may find ocf-tester (ocft) tool from resource-agents project
> useful
> > to check a basic sanity of the custom agents:
> > https://github.com/ClusterLabs/resource-agents/tree/master/tools/oc
> ft
> 
> 
> I did run ocf-tester and the result was passed. Here I carefully read
> the log. When this error was printed in the log, the latest operation
> that pacemaker made to the inetmanager's RA was start instead of
> stop. Why did pacemaker think that the RA should return 7 at this
> time?
> I may know the reason for this problem. Because of the implementation
> of the start and monitor methods in my RA. Because pacemaker recovers
> one resource at a time, it must wait for other resources that are
> starting. In order to eliminate this dependency, my RA's start method
> is just a startup instance and does not wait for business to work.
> The monitor method will loop to wait for business to be normal. And
> this can also filter out some occasional monitor failures. However, I
> tested and found that from the start of the start until the first
> monitor is successful, it will also block the scheduling of other
> resources. Inetmanager is the cause of the problem. Is my analysis
> correct? If so, how should I solve it?

Pacemaker does rely on the service being fully functional when start
completes -- that's what lets it know it's safe to start dependencies.
Instead of looping in the monitor, do a looping monitor at the end of
the start action, so it doesn't return until it's ready.
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Regarding patch submission for PCS

[Ken Gaillot]

On Mon, 2018-04-23 at 06:10 +, Roshni Chatterjee wrote:
>  
> Hi ,
>  
> We have fixed some issues existing in pcs code version-0.9.163
> Please let us know the detailed procedure to submit patch for the
> fixes .
>  
>  
> Regards,
> Roshni

Hi Roshni,

The best way is via a github pull request at:

   https://github.com/ClusterLabs/pcs

If you are not familiar with github, or can't go that route for
whatever reason, let us know.
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Displaying "original" resources location scores?

[Ken Gaillot]

On Tue, 2018-03-27 at 18:46 +0300, Andrei Borzenkov wrote:
> "Original" for lack of better word. To explain - "crm_simulate -s"
> will
> show final scores as determined by possible colocation contraints.
> E.g.
> if B is colocated with A, scores for A will be adjusted by scores for
> B.
> In large or dynamic configuration with multiple constraints it may
> not
> be exactly obvious where final result comes from. So is it possible
> to
> show original values used to compute native_colors shown by
> crm_simulate?

No, there is no way to automatically calculate just a portion of the
score.
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] Announcing the first ClusterLabs video karaoke contest!

[Ken Gaillot]

In honor of the recent 10th anniversary of the first public release of
Pacemaker, ClusterLabs is proud to announce its first video karaoke
contest!

To participate, simply record video of yourself singing karaoke to this
tune:

  https://www.youtube.com/watch?v=r7TADGV2fLI

using these lyrics:

  Sometimes it's hard to be a sysop
  Running all your jobs on just one host.
  You'll have bad times
  And it'll have downtimes,
  Doin' things that you don't understand.
  But if you need it, you'll cluster it,
  even though logs are hard to understand.
  And if you built it, Oh be proud of it,
  'Cause after all it's just a node.
  Standby your node,
  Because you want to upgrade
  And watch your resource migrate.
  Five nines are bold and lovely.
  Standby your node,
  And serve the world your resource.
  Keep serving all the things you can.
  Standby your node.
  Standby your node,
  And show the world your uptime.
  Keep serving all the things you can.
  Standby your node.

Users list members will vote on all submissions, and the winner will
receive a COMPLETE SET of all available ClusterLabs swag!*

-- 
Ken Gaillot <kgail...@redhat.com>

* DISCLAIMERS: ClusterLabs currently has no available swag. Some
restrictions may apply. Not responsible for damaged or lost
submissions, unless you configured fencing. Your mileage may vary.
Offer valid only for submissions received by April Fools Day 2018.
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] Possible idea for 2.0.0: renaming the Pacemaker daemons

[Ken Gaillot]

Hi all,

Andrew Beekhof brought up a potential change to help with reading
Pacemaker logs.

Currently, pacemaker daemon names are not intuitive, making it
difficult to search the system log or understand what each one does.

The idea is to rename the daemons, with a common prefix, and a name
that better reflects the purpose.

I think it's a great idea, but we have to consider two drawbacks:

* I'm about to release 2.0.0-rc2, and it's late in the cycle for a
major change. But if we don't do it now, it'll probably sit on the back
burner for a few years, as it wouldn't make sense to introduce such a
change shortly after a major bump.

* We can change *only* the names used in the logs, which will be
simple, but give us inconsistencies with what shows up in "ps", etc. Or
we can try to change everything -- process names, library names, API
function/structure names -- but that will impact other projects such as
sbd, crmsh, etc., potentially causing compatibility headaches.

What are your thoughts? Change or not? Now or later? Log tags, or
everything?

And the fun part, what would we change them to ...

Beekhof suggested renaming "pengine" to "cluster-planner", as an
example.

I think a prefix indicating pacemaker specifically would be better than
"cluster-" for grepping and intuitiveness.

For intuitiveness, long names are better ("pacemaker-FUNCTION"). On the
other hand, there's an argument for keeping names to 15 characters,
which is the default "ps" column width, and a reasonable limit for log
line tags. Maybe "pm-" or "pcmk-"? This prefix could also be used for
library names.

Looking at other projects with server processes, most use the
traditional "d" ending (for example, "rsyslogd"). A few add "-daemon"
("rtkit-daemon"), and others don't bother with any suffix ("gdm").

Here are the current names, with some example replacements:

 pacemakerd: PREFIX-launchd, PREFIX-launcher

 attrd: PREFIX-attrd, PREFIX-attributes

 cib: PREFIX-configd, PREFIX-state

 crmd: PREFIX-controld, PREFIX-clusterd, PREFIX-controller

 lrmd: PREFIX-locald, PREFIX-resourced, PREFIX-runner

 pengine: PREFIX-policyd, PREFIX-scheduler

 stonithd: PREFIX-fenced, PREFIX-stonithd, PREFIX-executioner

 pacemaker_remoted: PREFIX-remoted, PREFIX-remote

-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] symmetric-cluster=false doesn't work

[Ken Gaillot]

On Tue, 2018-03-20 at 22:03 +0300, George Melikov wrote:
> Hello,
>  
> I tried to create an asymmetric cluster via property symmetric-
> cluster=false , but my resources try to start on any node, though I
> have set locations for them.
>  
> What did I miss?
>  
> cib: https://pastebin.com/AhYqgUdw
>  
> Thank you for any help!
> 
> Sincerely,
> George Melikov

That output looks fine -- the resources are started only on nodes where
they are allowed. What are you expecting to be different?

Note that resources will be *probed* on every node (a one-time monitor
action to check whether they are already running there), but they
should only be *started* on allowed nodes.
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Colocation constraint for grouping all master-mode stateful resources with important stateless resources

[Ken Gaillot]

ai
> > > l%2fusers%2f2017-November%2f006788%2ehtml
> > > 
> > 
> > I do not see how it answers the question. It explains how to use
> > other
> > criteria than node name for colocating resources, but it does not
> > change
> > basic fact that colocating is asymmetrical. Actually this thread
> > explicitly suggests "Pick one resource as your base resource that
> > everything else should go along with".
> > 
> > If you you actually have configuration that somehow implements
> > symmetrical colocation between resources, I would appreciate if you
> > could post your configuration.
> > 
> > Regarding the original problem, the root cause is slightly
> > different though.
> > 
> > @Sam, the behavior you describe is correct for your constraints
> > that you
> > show. When colocating with resource set, all resources in the set
> > must
> > be active on the same node. It means that in your case of
> > 
> >   > id="pcs_rsc_colocation_set_drbdfs_set_drbd.master_inside-interface-
> > sameip.master_outside-interface-sameip.master"
> > score="INFINITY">
> > 
> >   
> > 
> >  > id="pcs_rsc_set_drbd.master_inside-interface-sameip.master_outside-
> > interface-sameip.master"
> > role="Master" sequential="false">
> >   
> >   
> >   
> > 
> >  
> > 
> > if one IP resource (master) is moved to another node, dependent
> > resource
> > (drbdfs) simply cannot run anywhere.
> > 
> > Before discussing low level pacemaker implementation you really
> > need to
> > have high level model of resources relationship. On one hand you
> > apparently intend to always run everything on the same node - on
> > the
> > other hand you have two rules that independently decide where to
> > place
> > two resources. That does not fit together.
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] copy file

[Ken Gaillot]

On Fri, 2018-03-09 at 10:35 +0100, Mevo Govo wrote:
> Hi, 
> Thank for advices, I'm thinking an optimal config for us. While the
> DB is working, it would do native DB replication. But oracle needs
> synchronized controlfiles when it starts normally. I can save the
> file before overwrite it. Currently I mean this (c1, c2, c3, c4, c5,
> c6 are control files):
> 
> c1: on node A, local file system
> c2: on node A, on DRBD device1
> c3: on node A, on DRBD device2 (FRA)
> c4: on node B, on DRBD device2 (FRA)
> c5: on node B, on DRBD device1
> c6: on node B, local file system
> 
> c2+c3 is a "standard" oracle config. c2 is replicated into FRA (fast
> recovery area of oracle). c1 (and c6) is just if all data in DRBD is
> lost.
> c1, c2, c3, c4, c5 (but not c6) are in sync while DB runs on node A.
> (c1,c2,c3: native DB replication, c2-c5, c3-c4 DRBD replication,
> protocol C)
> When I switch from node A to node B, c6 is out of sync (older
> version). I can (and I will) save it before overwrite by c5. But if
> c5 is corrupt, manual repair is needed, and there are other
> replications to repair it (c4, c3, c2, c1)
> If c1 and c6 would be the same file on an nfs filesystem, there would
> be a replication outside of DRBD without this "copy sync" problem.
> But in this case, the fail of only one component (the nfs) would
> cause unavailable the oracle DB on both node. (oracle DB stops if
> either of controlfile is lost or corrupted. No automatic reapair
> happens)
> As I think, the above consideration is similar to 3 node.
> If we trust DRBD, no c1 and c6 would be needed, but we are new users
> of DRBD.
> Thanks: lados.

I'm not familiar with Oracle, so I'm making some guesses here ...

If I follow you correctly, you have an Oracle configuration consisting
of two parts (c2+c3), and you're using DRBD to mirror these to another
node (c4+c5, which are really synchronized instances of c2+c3).

Somehow you are saving a copy of each file outside DRBD (c1+c6). Before
starting the db, you want a resource that checks whether the original
config needs repair, and if so, copy it from the backup outside DRBD.

It sounds like you should make a copy of the oracle agent, and modify
its start action to do what you want.

> 2018-03-08 20:12 GMT+01:00 Ken Gaillot <kgail...@redhat.com>:
> > On Thu, 2018-03-08 at 18:49 +0100, Mevo Govo wrote:
> > > Hi, 
> > > thanks for advice and your intrest.
> > > We would use oracle database over DRBD. Datafiles (and control
> > and
> > > redo files) will be on DRBD. FRA also (on an other DRBD device).
> > But
> > > we are new in DRBD, and DRBD is also a component what can fails.
> > We
> > > plan a scenario to recover the database without DRBD (without
> > data
> > > loss, if possible). We would use nfs or local filesystem for
> > this. If
> > > we use local FS, the control file is out of sync on the B node
> > when
> > > switch over (from A to B). We would copy controlfile (and redo
> > files)
> > > from DRBD to the local FS. After this, oracle can start, and it
> > keeps
> > > the controlfiles syncronized. If other backup related files
> > (archlog,
> > > backup) are also available on the local FS of either node, we can
> > > recover the DB without DRBD (without data loss)
> > > (I know it is a worst case scenario, because if DRBD fails, the
> > FS on
> > > it should be available at least on one node)
> > > Thanks: lados.
> > 
> > Why not use native database replication instead of copying files?
> > 
> > Any method getting files from a DRBD cluster to a non-DRBD node
> > will
> > have some inherent problems: it would have to be periodic, losing
> > some
> > data since the last run; it would still fail if some DRBD issue
> > corrupted the on-disk data, because you would be copying the
> > corrupted
> > data; and databases generally have in-memory state information that
> > makes files copied from a live server insufficient for data
> > integrity.
> > 
> > Native replication would avoid all that.
> > 
> > > 2018-03-07 10:20 GMT+01:00 Klaus Wenninger <kwenn...@redhat.com>:
> > > > On 03/07/2018 10:03 AM, Mevo Govo wrote:
> > > > > Thanks for advices, I will try!
> > > > > lados.
> > > > >
> > > > > 2018-03-05 23:29 GMT+01:00 Ken Gaillot <kgail...@redhat.com
> > > > > <mailto:kgail...@redhat.com>>:
> > > > >
> > > > >     On Mon, 2018-03-05 at 15:09 +0100, Mevo Govo wrote:
> > > > >     > Hi,
> > &g

Re: [ClusterLabs] Dependency loop

[Ken Gaillot]

.localdomain    pengine:     info:
> rsc_merge_weights:lvm_data-bak: Breaking dependency loop at
> cluster1_vip
> Mar 16 07:29:54 [3670] filesrv12.localdomain    pengine:     info:
> rsc_merge_weights:lvm_data-bak: Breaking dependency loop at
> cluster1_vip
> 
> Here's the output of "pcs constraint --full"
> 
> Location Constraints:
> Ordering Constraints:
> Colocation Constraints:
>   cluster1_vip with lvm_data (score:INFINITY) (id:colocation-
> cluster1_vip-lvm_data-INFINITY)
>   cluster1_vip with lun_data (score:INFINITY) (id:colocation-
> cluster1_vip-lun_data-INFINITY)
>   cluster1_vip with lvm_titanas (score:INFINITY) (id:colocation-
> cluster1_vip-lvm_titanas-INFINITY)
>   cluster1_vip with lun_titanas (score:INFINITY) (id:colocation-
> cluster1_vip-lun_titanas-INFINITY)
>   cluster1_vip with lvm_misc (score:INFINITY) (id:colocation-
> cluster1_vip-lvm_misc-INFINITY)
>   cluster1_vip with lun_misc (score:INFINITY) (id:colocation-
> cluster1_vip-lun_misc-INFINITY)
>   cluster1_vip with lvm_data-bak (score:INFINITY) (id:colocation-
> cluster1_vip-lvm_data-bak-INFINITY)
>   cluster1_vip with lvm_titanas-bak (score:INFINITY) (id:colocation-
> cluster1_vip-lvm_titanas-bak-INFINITY)
>   cluster1_vip with lvm_misc-bak (score:INFINITY) (id:colocation-
> cluster1_vip-lvm_misc-bak-INFINITY)
>   cluster1_vip with nfsserver (score:INFINITY) (id:colocation-
> cluster1_vip-nfsserver-INFINITY)
>   cluster1_vip with data-export-squash (score:INFINITY)
> (id:colocation-cluster1_vip-data-export-squash-INFINITY)
>   cluster1_vip with data-export-nosquash (score:INFINITY)
> (id:colocation-cluster1_vip-data-export-nosquash-INFINITY)
>   cluster1_vip with data-export-nosquash-linfarm (score:INFINITY)
> (id:colocation-cluster1_vip-data-export-nosquash-linfarm-INFINITY)
>   cluster1_vip with titanas-export-squash (score:INFINITY)
> (id:colocation-cluster1_vip-titanas-export-squash-INFINITY)
>   cluster1_vip with titanas-export-nosquash (score:INFINITY)
> (id:colocation-cluster1_vip-titanas-export-nosquash-INFINITY)
>   cluster1_vip with titanas-export-nosquash-linfarm (score:INFINITY)
> (id:colocation-cluster1_vip-titanas-export-nosquash-linfarm-INFINITY)
>   cluster1_vip with misc-export-squash (score:INFINITY)
> (id:colocation-cluster1_vip-misc-export-squash-INFINITY)
>   cluster1_vip with misc-export-nosquash (score:INFINITY)
> (id:colocation-cluster1_vip-misc-export-nosquash-INFINITY)
>   cluster1_vip with misc-export-nosquash-linfarm (score:INFINITY)
> (id:colocation-cluster1_vip-misc-export-nosquash-linfarm-INFINITY)
>   cluster1_vip with samba (score:INFINITY) (id:colocation-
> cluster1_vip-samba-INFINITY)
>   cluster1_vip with httpd (score:INFINITY) (id:colocation-
> cluster1_vip-httpd-INFINITY)

I'm guessing you have the sense of colocation constraints flipped. The
above says: place a whole bunch of resources first, with no
dependencies among them, then try to place cluster1_vip on a node with
all of them (which is likely impossible).

If you want to pick a node for the IP first, then place all the
resources on that node, flip the constraints, e.g. httpd with
cluster1_vip.

> 
> Here's my pcs cluster cib output:
> https://pastebin.com/CkW5wQmS
> 
> I am afraid that this could be the cause of my resources falling back
> to a node that has recovered from a fail over although I have a
> stickiness score of INFINITY.
> 
> Thanks,
> George
> ___
> Users mailing list: Users@clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] How to cancel a fencing request?

[Ken Gaillot]

On Mon, 2018-04-02 at 10:54 +0200, Jehan-Guillaume de Rorthais wrote:
> On Sun, 1 Apr 2018 09:01:15 +0300
> Andrei Borzenkov <arvidj...@gmail.com> wrote:
> 
> > 31.03.2018 23:29, Jehan-Guillaume de Rorthais пишет:
> > > Hi all,
> > > 
> > > I experienced a problem in a two node cluster. It has one FA per
> > > node and
> > > location constraints to avoid the node each of them are supposed
> > > to
> > > interrupt. 
> > 
> > If you mean stonith resource - for all I know location it does not
> > affect stonith operations and only changes where monitoring action
> > is
> > performed.
> 
> Sure.
> 
> > You can create two stonith resources and declare that each
> > can fence only single node, but that is not location constraint, it
> > is
> > resource configuration. Showing your configuration would be
> > helpflul to
> > avoid guessing.
> 
> True, I should have done that. A conf worth thousands of words :)
> 
>   crm conf< 
>   primitive fence_vm_srv1 stonith:fence_virsh   \
> params pcmk_host_check="static-list" pcmk_host_list="srv1"  \
>    ipaddr="192.168.2.1" login=""  \
>    identity_file="/root/.ssh/id_rsa"\
>    port="srv1-d8" action="off"  \
> op monitor interval=10s
> 
>   location fence_vm_srv1-avoids-srv1 fence_vm_srv1 -inf: srv1
> 
>   primitive fence_vm_srv2 stonith:fence_virsh   \
> params pcmk_host_check="static-list" pcmk_host_list="srv2"  \
>    ipaddr="192.168.2.1" login=""  \
>    identity_file="/root/.ssh/id_rsa"\
>    port="srv2-d8" action="off"  \
> op monitor interval=10s
> 
>   location fence_vm_srv2-avoids-srv2 fence_vm_srv2 -inf: srv2
>   
>   EOC
> 
> 
> > > During some tests, a ms resource raised an error during the stop
> > > action on
> > > both nodes. So both nodes were supposed to be fenced.
> > 
> > In two-node cluster you can set pcmk_delay_max so that both nodes
> > do not
> > attempt fencing simultaneously.
> 
> I'm not sure to understand the doc correctly in regard with this
> property. Does
> pcmk_delay_max delay the request itself or the execution of the
> request?
> 
> In other words, is it:
> 
>   delay -> fence query -> fencing action
> 
> or 
> 
>   fence query -> delay -> fence action
> 
> ?
> 
> The first definition would solve this issue, but not the second. As I
> understand it, as soon as the fence query has been sent, the node
> status is
> "UNCLEAN (online)".

The latter -- you're correct, the node is already unclean by that time.
Since the stop did not succeed, the node must be fenced to continue
safely.

> > > The first node did, but no FA was then able to fence the second
> > > one. So the
> > > node stayed DC and was reported as "UNCLEAN (online)".
> > > 
> > > We were able to fix the original ressource problem, but not to
> > > avoid the
> > > useless second node fencing.
> > > 
> > > My questions are:
> > > 
> > > 1. is it possible to cancel the fencing request 
> > > 2. is it possible reset the node status to "online" ? 
> > 
> > Not that I'm aware of.
> 
> Argh!
> 
> ++

You could fix the problem with the stopped service manually, then run
"stonith_admin --confirm=" (or higher-level tool equivalent).
That tells the cluster that you took care of the issue yourself, so
fencing can be considered complete.

The catch there is that the cluster will assume you stopped the node,
and all services on it are stopped. That could potentially cause some
headaches if it's not true. I'm guessing that if you unmanaged all the
resources on it first, then confirmed fencing, the cluster would detect
everything properly, then you could re-manage.
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Advisory order for cluster-managed resources

[Ken Gaillot]

On Mon, 2018-04-02 at 18:47 +, Sam Gardner wrote:
> Is there any way that I can order the startup of resources A, B, C in
> that order such that if B fails to start for some reason, A and C
> will still start, but the order of operations will always be start A,
> start B, start C?
> 
> I've tried setting it like this:
> set rscA rscB.master rscC action=start require-all=false setoptions
> kind=Optional symmetrical=false

Sets are always confusing IMO. Hopefully one of the pcs developers can
chime in to clarify some of this ...

I'm not familiar with the rscB.master syntax to specify a role.

action=start with symmetrical=false means that the order would be
respected when starting, but there would be no order required when
stopping. Since you're looking to affect only the master role, this is
probably insufficient, since you probably want to order the *promote*
of rscB before the start of rscC (start would only ensure slave role).

require-all=false has no effect when sequential=true, which it is by
default, and it would only affect the ordering between multiple sets,
not the ordering within a single set.

In short, I'd recommend separate constraints :) e.g. start rscA then
promote rscB, promote rscB then start rscC.

One thing to keep in mind: optional ordering constraints have an
effect only if all the actions are occurring in the same transition. In
other words, with an optional constraint, C *can* start before B, if B
doesn't happen to be starting at the moment -- e.g. if B also happens
to be ordered after resource X which hasn't started yet, C can go ahead
and start, since B is waiting on X. Another example is if B is stopped
because it has failed too many times, C can still start.

> 
> Where rscB.master is the Master half of a Master/Slave resource, but
> if rscB.master fails to come up then rscC also won't start.
> 
> I want this to be a stable order so that I know the order in which
> the resources start up - I can do it with separate constraints but it
> seems like an order set is a useful shortcut here.
> -- 
> Sam Gardner   
> Trustwave | SMART SECURITY ON DEMAND
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] How to cancel a fencing request?

[Ken Gaillot]

On Tue, 2018-04-03 at 07:36 +0200, Klaus Wenninger wrote:
> On 04/02/2018 04:02 PM, Ken Gaillot wrote:
> > On Mon, 2018-04-02 at 10:54 +0200, Jehan-Guillaume de Rorthais
> > wrote:
> > > On Sun, 1 Apr 2018 09:01:15 +0300
> > > Andrei Borzenkov <arvidj...@gmail.com> wrote:
> > > 
> > > > 31.03.2018 23:29, Jehan-Guillaume de Rorthais пишет:
> > > > > Hi all,
> > > > > 
> > > > > I experienced a problem in a two node cluster. It has one FA
> > > > > per
> > > > > node and
> > > > > location constraints to avoid the node each of them are
> > > > > supposed
> > > > > to
> > > > > interrupt. 
> > > > 
> > > > If you mean stonith resource - for all I know location it does
> > > > not
> > > > affect stonith operations and only changes where monitoring
> > > > action
> > > > is
> > > > performed.
> > > 
> > > Sure.
> > > 
> > > > You can create two stonith resources and declare that each
> > > > can fence only single node, but that is not location
> > > > constraint, it
> > > > is
> > > > resource configuration. Showing your configuration would be
> > > > helpflul to
> > > > avoid guessing.
> > > 
> > > True, I should have done that. A conf worth thousands of words :)
> > > 
> > >   crm conf< > > 
> > >   primitive fence_vm_srv1 stonith:fence_virsh   \
> > > params pcmk_host_check="static-list" pcmk_host_list="srv1"  \
> > >    ipaddr="192.168.2.1" login=""  \
> > >    identity_file="/root/.ssh/id_rsa"\
> > >    port="srv1-d8" action="off"  \
> > > op monitor interval=10s
> > > 
> > >   location fence_vm_srv1-avoids-srv1 fence_vm_srv1 -inf: srv1
> > > 
> > >   primitive fence_vm_srv2 stonith:fence_virsh   \
> > > params pcmk_host_check="static-list" pcmk_host_list="srv2"  \
> > >    ipaddr="192.168.2.1" login=""  \
> > >    identity_file="/root/.ssh/id_rsa"\
> > >    port="srv2-d8" action="off"  \
> > > op monitor interval=10s
> > > 
> > >   location fence_vm_srv2-avoids-srv2 fence_vm_srv2 -inf: srv2
> > >   
> > >   EOC
> > > 
> 
> -inf constraints like that should effectively prevent
> stonith-actions from being executed on that nodes.

It shouldn't ...

Pacemaker respects target-role=Started/Stopped for controlling
execution of fence devices, but location (or even whether the device is
"running" at all) only affects monitors, not execution.

> Though there are a few issues with location constraints
> and stonith-devices.
> 
> When stonithd brings up the devices from the cib it
> runs the parts of pengine that fully evaluate these
> constraints and it would disable the stonith-device
> if the resource is unrunable on that node.

That should be true only for target-role, not everything that affects
runnability

> But this part is not retriggered for location contraints
> with attributes or other content that would dynamically
> change. So one has to stick with constraints as simple
> and static as those in the example above.
> 
> Regarding adding/removing location constraints dynamically
> I remember a bug that should have got fixed round 1.1.18
> that led to improper handling and actually usage of
> stonith-devices disabled or banned from certain nodes.
> 
> Regards,
> Klaus
>  
> > > > > During some tests, a ms resource raised an error during the
> > > > > stop
> > > > > action on
> > > > > both nodes. So both nodes were supposed to be fenced.
> > > > 
> > > > In two-node cluster you can set pcmk_delay_max so that both
> > > > nodes
> > > > do not
> > > > attempt fencing simultaneously.
> > > 
> > > I'm not sure to understand the doc correctly in regard with this
> > > property. Does
> > > pcmk_delay_max delay the request itself or the execution of the
> > > request?
> > > 
> > > In other words, is it:
> > > 
> > >   delay -> fence query -> fencing action
> &g

Re: [ClusterLabs] Possible idea for 2.0.0: renaming the Pacemaker daemons

[Ken Gaillot]

On Tue, 2018-04-03 at 08:33 +0200, Kristoffer Grönlund wrote:
> Ken Gaillot <kgail...@redhat.com> writes:
> 
> > > I
> > > would vote against PREFIX-configd as compared to other cluster
> > > software,
> > > I would expect that daemon name to refer to a more generic
> > > cluster
> > > configuration key/value store, and that is something that I have
> > > some
> > > hope of adding in the future ;) So I'd like to keep "config" or
> > > "database" for such a possible future component...
> > 
> > What's the benefit of another layer over the CIB?
> > 
> 
> The idea is to provide a more generalized key-value store that other
> applications built on top of pacemaker can use. Something like a
> HTTP REST API to a key-value store with transactional semantics
> provided
> by the cluster. My understanding so far is that the CIB is too heavy
> to
> support that kind of functionality well, and besides that the
> interface
> is not convenient for non-cluster applications.

My first impression is that it sounds like a good extension to attrd,
cluster-wide attributes instead of node attributes. (I would envision a
REST API daemon sitting in front of all the daemons without providing
any actual functionality itself.)

The advantage to extending attrd is that it already has code to
synchronize attributes at start-up, DC election, partition healing,
etc., as well as features such as write dampening.

Also cib -> pcmk-configd is very popular :)

> My most immediate applications for that would be to build file
> syncing
> into the cluster and to avoid having to have an extra communication
> layer for the UI.

How would file syncing via a key-value store work?

One of the key hurdles in any cluster-based sync is
authentication/authorization. Authorization to use a cluster UI is not
necessarily equivalent to authorization to transfer arbitrary files as
root.

> 
> Cheers,
> Kristoffer
> 
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Advisory order for cluster-managed resources

[Ken Gaillot]

On Tue, 2018-04-03 at 16:24 +, Sam Gardner wrote:
> "In short, I'd recommend separate constraints :) e.g. start rscA then
> promote rscB, promote rscB then start rscC."
> 
> I've fiddled around with my code and gotten it to do this.
> 
> Just out of curiosity, is there an upper limit on the number of
> individual constraints that we can put in place?

Nope

> 10 or 15 resources colocated on the same node, starting up in a
> consistent order will require (at least) 2x the number of constraints
> in this scheme - it seemed to me that sets would simplify things but
> perhaps not.

Groups are easier to follow if you have simple colocation+order
sequences.

Sets can help with more complicated set-ups, but they are tricky to get
right and always difficult to read.
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] How to cancel a fencing request?

[Ken Gaillot]

On Tue, 2018-04-03 at 21:33 +0200, Jehan-Guillaume de Rorthais wrote:
> On Mon, 02 Apr 2018 09:02:24 -0500
> Ken Gaillot <kgail...@redhat.com> wrote:
> > On Mon, 2018-04-02 at 10:54 +0200, Jehan-Guillaume de Rorthais
> > wrote:
> > > On Sun, 1 Apr 2018 09:01:15 +0300
> > > Andrei Borzenkov <arvidj...@gmail.com> wrote:
> 
> [...]
> > > > In two-node cluster you can set pcmk_delay_max so that both
> > > > nodes
> > > > do not
> > > > attempt fencing simultaneously.  
> > > 
> > > I'm not sure to understand the doc correctly in regard with this
> > > property. Does
> > > pcmk_delay_max delay the request itself or the execution of the
> > > request?
> > > 
> > > In other words, is it:
> > > 
> > >   delay -> fence query -> fencing action
> > > 
> > > or 
> > > 
> > >   fence query -> delay -> fence action
> > > 
> > > ?
> > > 
> > > The first definition would solve this issue, but not the second.
> > > As I
> > > understand it, as soon as the fence query has been sent, the node
> > > status is
> > > "UNCLEAN (online)".  
> > 
> > The latter -- you're correct, the node is already unclean by that
> > time.
> > Since the stop did not succeed, the node must be fenced to continue
> > safely.
> 
> Thank you for this clarification.
> 
> Do you want to patch to add this clarification to the documentation ?

Sure, it never hurts :)

> 
> > > > > The first node did, but no FA was then able to fence the
> > > > > second
> > > > > one. So the
> > > > > node stayed DC and was reported as "UNCLEAN (online)".
> > > > > 
> > > > > We were able to fix the original ressource problem, but not
> > > > > to
> > > > > avoid the
> > > > > useless second node fencing.
> > > > > 
> > > > > My questions are:
> > > > > 
> > > > > 1. is it possible to cancel the fencing request 
> > > > > 2. is it possible reset the node status to "online" ?   
> > > > 
> > > > Not that I'm aware of.  
> > > 
> > > Argh!
> > > 
> > > ++  
> > 
> > You could fix the problem with the stopped service manually, then
> > run
> > "stonith_admin --confirm=" (or higher-level tool
> > equivalent).
> > That tells the cluster that you took care of the issue yourself, so
> > fencing can be considered complete.
> 
> Oh, OK. I was wondering if it could help.
> 
> For the complete story, while I was working on this cluster, we tried
> first to
> "unfence" the node using "stonith_admin --unfence "...and
> it actually
> rebooted the node (using fence_vmware_soap) without cleaning its
> status??
> 
> ...So we actually cleaned the status using "--confirm" after the
> complete
> reboot.
> 
> Thank you for this clarification again.
> 
> > The catch there is that the cluster will assume you stopped the
> > node,
> > and all services on it are stopped. That could potentially cause
> > some
> > headaches if it's not true. I'm guessing that if you unmanaged all
> > the
> > resources on it first, then confirmed fencing, the cluster would
> > detect
> > everything properly, then you could re-manage.
> 
> Good to know. Thanks again.
> 
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] How to cancel a fencing request?

[Ken Gaillot]

On Tue, 2018-04-03 at 21:46 +0200, Klaus Wenninger wrote:
> On 04/03/2018 05:43 PM, Ken Gaillot wrote:
> > On Tue, 2018-04-03 at 07:36 +0200, Klaus Wenninger wrote:
> > > On 04/02/2018 04:02 PM, Ken Gaillot wrote:
> > > > On Mon, 2018-04-02 at 10:54 +0200, Jehan-Guillaume de Rorthais
> > > > wrote:
> > > > > On Sun, 1 Apr 2018 09:01:15 +0300
> > > > > Andrei Borzenkov <arvidj...@gmail.com> wrote:
> > > > > 
> > > > > > 31.03.2018 23:29, Jehan-Guillaume de Rorthais пишет:
> > > > > > > Hi all,
> > > > > > > 
> > > > > > > I experienced a problem in a two node cluster. It has one
> > > > > > > FA
> > > > > > > per
> > > > > > > node and
> > > > > > > location constraints to avoid the node each of them are
> > > > > > > supposed
> > > > > > > to
> > > > > > > interrupt. 
> > > > > > 
> > > > > > If you mean stonith resource - for all I know location it
> > > > > > does
> > > > > > not
> > > > > > affect stonith operations and only changes where monitoring
> > > > > > action
> > > > > > is
> > > > > > performed.
> > > > > 
> > > > > Sure.
> > > > > 
> > > > > > You can create two stonith resources and declare that each
> > > > > > can fence only single node, but that is not location
> > > > > > constraint, it
> > > > > > is
> > > > > > resource configuration. Showing your configuration would be
> > > > > > helpflul to
> > > > > > avoid guessing.
> > > > > 
> > > > > True, I should have done that. A conf worth thousands of
> > > > > words :)
> > > > > 
> > > > >   crm conf< > > > > 
> > > > >   primitive fence_vm_srv1
> > > > > stonith:fence_virsh   \
> > > > > params pcmk_host_check="static-list"
> > > > > pcmk_host_list="srv1"  \
> > > > >    ipaddr="192.168.2.1"
> > > > > login=""  \
> > > > >    identity_file="/root/.ssh/id_rsa" 
> > > > >    \
> > > > >    port="srv1-d8"
> > > > > action="off"  \
> > > > > op monitor interval=10s
> > > > > 
> > > > >   location fence_vm_srv1-avoids-srv1 fence_vm_srv1 -inf: srv1
> > > > > 
> > > > >   primitive fence_vm_srv2
> > > > > stonith:fence_virsh   \
> > > > > params pcmk_host_check="static-list"
> > > > > pcmk_host_list="srv2"  \
> > > > >    ipaddr="192.168.2.1"
> > > > > login=""  \
> > > > >    identity_file="/root/.ssh/id_rsa" 
> > > > >    \
> > > > >    port="srv2-d8"
> > > > > action="off"  \
> > > > > op monitor interval=10s
> > > > > 
> > > > >   location fence_vm_srv2-avoids-srv2 fence_vm_srv2 -inf: srv2
> > > > >   
> > > > >   EOC
> > > > > 
> > > 
> > > -inf constraints like that should effectively prevent
> > > stonith-actions from being executed on that nodes.
> > 
> > It shouldn't ...
> > 
> > Pacemaker respects target-role=Started/Stopped for controlling
> > execution of fence devices, but location (or even whether the
> > device is
> > "running" at all) only affects monitors, not execution.
> > 
> > > Though there are a few issues with location constraints
> > > and stonith-devices.
> > > 
> > > When stonithd brings up the devices from the cib it
> > > runs the parts of pengine that fully evaluate these
> > > constraints and it would disable the stonith-device
> > > if the resource is unrunable on that node.
> > 
> > That should be true only for target-role, not everything that
> > affects
> > runnability
> 
> cib_device_update bails out via a removal of the device i

Re: [ClusterLabs] Possible idea for 2.0.0: renaming the Pacemaker daemons

[Ken Gaillot]

On Thu, 2018-03-29 at 10:35 +0200, Kristoffer Grönlund wrote:
> Ken Gaillot <kgail...@redhat.com> writes:
> 
> > Hi all,
> > 
> > Andrew Beekhof brought up a potential change to help with reading
> > Pacemaker logs.
> > 
> > Currently, pacemaker daemon names are not intuitive, making it
> > difficult to search the system log or understand what each one
> > does.
> > 
> > The idea is to rename the daemons, with a common prefix, and a name
> > that better reflects the purpose.
> > 
> 
> [...]
> 
> > Here are the current names, with some example replacements:
> > 
> >  pacemakerd: PREFIX-launchd, PREFIX-launcher
> > 
> >  attrd: PREFIX-attrd, PREFIX-attributes
> > 
> >  cib: PREFIX-configd, PREFIX-state
> > 
> >  crmd: PREFIX-controld, PREFIX-clusterd, PREFIX-controller
> > 
> >  lrmd: PREFIX-locald, PREFIX-resourced, PREFIX-runner
> > 
> >  pengine: PREFIX-policyd, PREFIX-scheduler
> > 
> >  stonithd: PREFIX-fenced, PREFIX-stonithd, PREFIX-executioner
> > 
> >  pacemaker_remoted: PREFIX-remoted, PREFIX-remote
> 
> Better to do it now rather than later. I vote in favor of changing
> the
> names. Yes, it'll mess up crmsh, but at least for distributions it's
> just a simple search/replace patch to apply.
> 
> I would also vote in favour of sticking to the 15 character limit,
> and
> to use "pcmk" as the prefix. That leaves 11 characters for the name,
> which should be enough for anyone ;)
> 
> My votes:
> 
> pacemakerd -> pcmk-launchd
> attrd -> pcmk-attrd
> cib -> pcmk-stated
> crmd -> pcmk-controld
> lrmd -> pcmk-resourced
> pengine -> pcmk-schedulerd
> stonithd -> pcmk-fenced
> pacemaker_remoted -> pcmk-remoted

Those are all acceptable to me. I'd also be fine with pcmk-execd for
lrmd, as suggested elsewhere.

> 
> The one I'm the most divided about is cib. pcmk-cibd would also work.

That is the most difficult one, isn't it? :-)

Looking at it from another direction, maybe pcmk-iod, since it
abstracts disk I/O for the other daemons? It doesn't encompass its
entire purpose, but it points in the right direction.

> I
> would vote against PREFIX-configd as compared to other cluster
> software,
> I would expect that daemon name to refer to a more generic cluster
> configuration key/value store, and that is something that I have some
> hope of adding in the future ;) So I'd like to keep "config" or
> "database" for such a possible future component...

What's the benefit of another layer over the CIB?

> 
> Cheers,
> Kristoffer
> 
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Resource switchover taking more time upon shutting off one of the node in a 2 node cluster

[Ken Gaillot]

On Sat, 2018-02-24 at 15:02 +0530, avinash sharma wrote:
> Hi Ken,
> 
> Thanks for the reply. 
> Here the resource in question is RoutingManager and floatingips which
> has no dependency on stateful_consul resource so i think we can
> ignore stateful_consul_promote failures.
> RoutingManager (MS) and aaaip, nataccessgwip, accessip,
> natcpcoregwip, cpcoreip from floatingips resource group, are the
> resources for which switcover action by crmd got delayed.

The delay happens around this:

Feb 21 21:42:26 [24021] IVM-1   lrmd:  warning: operation_finished:
stateful_wildfly_promote_0:869 - timed out after 30ms

So it's waiting on that, whether due to a constraint or for some other
reason. For example, if the transition that was started in got aborted,
the cluster has to wait for that result before starting a new
transition.

> Thanks,
> Avinash Sharma
> 
> On Fri, Feb 23, 2018 at 8:57 PM, Ken Gaillot <kgail...@redhat.com>
> wrote:
> > On Fri, 2018-02-23 at 16:15 +0530, avinash sharma wrote:
> > > Subject: Switchover of resource(MS) 'RoutingManager' and resource
> > > group 'floatingips', which have 'colocation' and 'after'
> > constraints
> > > on each other, are taking around 5 minutes to get promoted when
> > node
> > > running master instance goes down.
> > 
> > 
> > 
> > When Pacemaker runs the resource agent, it will log any error
> > messages
> > that the agent prints. I didn't look at the entire log, but I
> > suspect
> > this is the cause, the promote action didn't succeed during that
> > time:
> > 
> > > Feb 21 21:37:40 [24021] IVM-1       lrmd:   notice:
> > > operation_finished:   stateful_consul_promote_0:864:stderr [
> > > ssh_exchange_identification: Connection closed by remote host
> > >  ]
> > > Feb 21 21:37:40 [24021] IVM-1       lrmd:   notice:
> > > operation_finished:   stateful_consul_promote_0:864:stderr [
> > > rsync: connection unexpectedly closed (0 bytes received so far)
> > > [sender] ]
> > > Feb 21 21:37:40 [24021] IVM-1       lrmd:   notice:
> > > operation_finished:   stateful_consul_promote_0:864:stderr [
> > > rsync error: unexplained error (code 255) at io.c(226)
> > [sender=3.1.2]
> > > ]
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Error observed while starting cluster

[Ken Gaillot]

aker.log or whatever is set in corosync.conf,
often /var/log/cluster/corosync.log).

When you compiled, did you "make install", or "make rpm" and then
install the rpm? The rpm will do steps that install doesn't, like
create the hacluster user and haclient group, that you would have to do
by hand after make install if not already present.

> Mar 20 10:55:45 [26932] pcmk3 pacemakerd:   notice:
> pcmk_process_exit:  Respawning failed child process: crmd
> Mar 20 10:55:45 [26932] pcmk3 pacemakerd: info:
> start_child:    Using uid=189 and group=189 for process crmd
> Mar 20 10:55:45 [26932] pcmk3 pacemakerd: info:
> start_child:    Forked child 27036 for process crmd
> Mar 20 10:55:45 [26932] pcmk3 pacemakerd: info:
> mcp_cpg_deliver:    Ignoring process list sent by peer for local node
> Mar 20 10:55:45 [26932] pcmk3 pacemakerd: info:
> mcp_cpg_deliver:    Ignoring process list sent by peer for local node
> Mar 20 10:55:45 [26932] pcmk3 pacemakerd:    error:
> pcmk_child_exit:    The crmd process (27036) exited: Key has expired
> (127)
> Mar 20 10:55:45 [26932] pcmk3 pacemakerd:   notice:
> pcmk_process_exit:  Respawning failed child process: crmd
> Mar 20 10:55:45 [26932] pcmk3 pacemakerd: info:
> start_child:    Using uid=189 and group=189 for process crmd
> Mar 20 10:55:45 [26932] pcmk3 pacemakerd: info:
> start_child:    Forked child 27037 for process crmd
> Mar 20 10:55:45 [26932] pcmk3 pacemakerd: info:
> mcp_cpg_deliver:    Ignoring process list sent by peer for local node
> Mar 20 10:55:45 [26932] pcmk3 pacemakerd: info:
> mcp_cpg_deliver:    Ignoring process list sent by peer for local node
> Mar 20 10:55:45 [26932] pcmk3 pacemakerd:    error:
> pcmk_child_exit:    The crmd process (27037) exited: Key has expired
> (127)
> Mar 20 10:55:45 [26932] pcmk3 pacemakerd:   notice:
> pcmk_process_exit:  Respawning failed child process: crmd
>  
>  
> Regards,
> Roshni
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] state file not created for Stateful resource agent

[Ken Gaillot]

On Sat, 2018-03-17 at 15:35 +0530, ashutosh tiwari wrote:
> Hi,
> 
> 
> We have two node active/standby cluster with a dummy  Stateful
> resource (pacemaker/Stateful).
> 
> We observed that in case one node is up with master resource and
> other node is booted up, state file for dummy resource is not created
> on the node coming up.
> 
> /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instanc
> e_attributes[@id='status-2']:   name="master-unicloud" value="5"/>
> Mar 17 12:22:29 [24875] tigana       lrmd:   notice:
> operation_finished:        unicloud_start_0:25729:stderr [
> /usr/lib/ocf/resource.d/pw/uc: line 94: /var/run/uc/role: No such
> file or directory ]

The resource agent is ocf:pw:uc -- I assume this is a local
customization of the ocf:pacemaker:Stateful agent?

It looks to me like the /var/run/uc directory is not being created on
the second node. /var/run is a memory filesystem, so it's wiped at
every reboot, and any directories need to be created (as root) before
they are used, every boot.

ocf:pacemaker:Stateful puts its state file directly in /var/run to
avoid needing to create any directories. You can change that by setting
the "state" parameter, but in that case you have to make sure the
directory you specify exists beforehand.

> This issue is not observed in case secondary do not wait for  cib
> sync and starts the resource on secondary as well.
> 
> We are in process of upgrading from centos6 to centos7, We never
> observed this issue with centos6 releases.
> 
> Attributes for clone resource: master-max=1 master-node-max=1 clone-
> max=2 clone-node-max=1 
> 
> setup under observation is:
> 
> CentOS Linux release 7.4.1708 (Core)
> corosync-2.4.0-9.el7.x86_64
> pacemaker-1.1.16-12.el7.x86_64.
> 
> 
> Thanks and Regards,
> Ashutosh 
> 
> ___
> Users mailing list: Users@clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Colocation constraint for grouping all master-mode stateful resources with important stateless resources

[Ken Gaillot]

         id="pcs_rsc_set_drbd.master_inside-interface-sameip.master_outside-
> interface-sameip.master-1" role="Master">
>           
>           
>           
>         
>         
>           
>         
>       

The above constraint says: if promoting any of drbd.master and the two
interfaces and/or starting drbdfs, do each action one at a time (in any
order). Other actions (including demoting and stopping) can happen in
any order.

>        rsc="inside-interface-sameip.master">
>          score="-INFINITY">
>           
>         
>       
>        rsc="outside-interface-sameip.master">
>          score="-INFINITY">
>           
>         
>       

The above constraints keep inside-interface on a node where eth1 is
good, and outside-interface on a node where eth2 is good.

I'm guessing you want to keep these two constraints, and start over
from scratch on the others. What are your intended relationships
between the various resources?

>     
> -- 
> Sam Gardner  
> Trustwave | SMART SECURITY ON DEMAND
> ___
> Users mailing list: Users@clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] copy file

[Ken Gaillot]

On Mon, 2018-03-05 at 15:09 +0100, Mevo Govo wrote:
> Hi,
> I am new in pacemaker. I think, I should use DRBD instead of copy
> file. But in this case, I would copy a file from a DRBD to an
> external device. Is there a builtin way to copy a file before a
> resource is started (and after the DRBD is promoted)? For example a
> "copy" resource? I did not find it. 
> Thanks: lados.
> 

There's no stock way of doing that, but you could easily write an agent
that simply copies a file. You could use ocf:pacemaker:Dummy as a
template, and add the copy to the start action. You can use standard
ordering and colocation constraints to make sure everything happens in
the right sequence.

I don't know what capabilities your external device has, but another
approach would be to an NFS server to share the DRBD file system, and
mount it from the device, if you want direct access to the original
file rather than a copy.
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] why some resources blocked

[Ken Gaillot]

On Fri, 2018-03-02 at 11:53 +0100, Mevo Govo wrote:
> 
> Hi, 
> 
> I am new in pacemaker, corosync and on this list. 
> I created a cluster (based on "Clusters from Scratch" doc). The DRBD
> and my filesystem on it work fine. Then I add an oracle and oralsnr
> resource. But these oracle resources remain stopped. I can start and
> stop the ora resources well by "pcs resource debug-start". (I can
> login into database, lsnrctl status ...). Could you help me, why do
> not these resources start automatically? I do not see errors in
> "/var/log/cluster/corosync.log", only this:
> 
> Mar  1 10:44:40 xetest1 crmd[12440]: warning: Input I_ELECTION_DC
> received in state S_INTEGRATION from do_election_check
> Mar  1 10:44:40 xetest1 pengine[12439]:  notice: On loss of CCM
> Quorum: Ignore
> Mar  1 10:44:40 xetest1 pengine[12439]:  notice: Start  
> fs_drbd1#011(xetest1)
> Mar  1 10:44:40 xetest1 pengine[12439]:  notice: Start  
> ora_listener#011(xetest1 - blocked)
> Mar  1 10:44:40 xetest1 pengine[12439]:  notice: Start  
> ora_db_xe#011(xetest1 - blocked)
> 
> 
> [root@xetest1 /]# pcs status
> Cluster name: cluster_xetest
> Stack: corosync
> Current DC: xetest2 (version 1.1.16-12.el7-94ff4df) - partition with
> quorum
> Last updated: Fri Mar  2 10:03:04 2018
> Last change: Fri Mar  2 10:02:48 2018 by root via cibadmin on xetest1
> 
> 2 nodes configured
> 5 resources configured
> 
> Online: [ xetest1 xetest2 ]
> 
> Full list of resources:
> 
>  Master/Slave Set: drbd1_sync [drbd1]
>  Masters: [ xetest1 ]
>  Slaves: [ xetest2 ]
>  fs_drbd1   (ocf::heartbeat:Filesystem):    Started xetest1
>  ora_listener   (ocf::heartbeat:oralsnr):   Stopped
>  ora_db_xe  (ocf::heartbeat:oracle):    Stopped
> 
> Daemon Status:
>   corosync: active/disabled
>   pacemaker: active/disabled
>   pcsd: active/disabled
> [root@xetest1 /]#
> 
> # I created oracle resources by these commands (OCFMON user also
> created successful during debug-start)
> 
> pcs -f clust_ora_cfg_tmp resource create ora_listener
> ocf:heartbeat:oralsnr \
>   sid="XE" \
>   home="/u01/app/oracle/product/11.2.0/xe" \
>   user="oracle" \
>   listener="LISTENER" \
>   op monitor interval=30s
> 
> pcs -f clust_ora_cfg_tmp constraint colocation add ora_listener with
> fs_drbd1 INFINITY
> pcs -f clust_ora_cfg_tmp constraint order promote fs_drbd1 then start
> ora_listener

^^^ fs_drbd1 is not a master/slave resource, so it can't be promoted

I'm guessing you want to colocate fs_drbd1 with the master role of
drbd1_sync (and order it after the promote of that).

If you want ora_listener and then ora_db_exe to start in order after
that, I'd group fs_drbd1, ora_listener, and ora_db_exe, then
colocate/order the group with the master role of drbd1_sync.

> 
> pcs -f clust_ora_cfg_tmp resource create ora_db_xe
> ocf:heartbeat:oracle \
>   sid="XE" \
>   home="/u01/app/oracle/product/11.2.0/xe" \
>   user="oracle" \
>   monuser="OCFMON" \
>   monpassword="**" \
>   shutdown_method="immediate" \
>   op monitor interval=30s
> 
> pcs -f clust_ora_cfg_tmp constraint colocation add ora_db_xe with
> ora_listener INFINITY
> pcs -f clust_ora_cfg_tmp constraint order promote ora_listener then
> start ora_db_xe
> 
> pcs -f clust_ora_cfg_tmp constraint
> pcs -f clust_ora_cfg_tmp resource show
> 
> pcs cluster cib-push clust_ora_cfg_tmp
> pcs status
> 
> Thanks: lados.
> 
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] 答复: 答复: 答复: How to configure to make each slave resource has one VIP

[Ken Gaillot]

On Sat, 2018-02-24 at 03:02 +, 范国腾 wrote:
> Thank you, Ken,
> 
> So I could use the following command: pcs constraint colocation set
> pgsql-slave-ip1 pgsql-slave-ip2 pgsql-slave-ip3 setoptions score=-
> 1000

Correct

(sorry for the late reply)

> 
> -邮件原件-
> 发件人: Users [mailto:users-boun...@clusterlabs.org] 代表 Ken Gaillot
> 发送时间: 2018年2月23日 23:14
> 收件人: Cluster Labs - All topics related to open-source clustering
> welcomed <users@clusterlabs.org>
> 主题: Re: [ClusterLabs] 答复: 答复: How to configure to make each slave
> resource has one VIP
> 
> On Fri, 2018-02-23 at 12:45 +, 范国腾 wrote:
> > Thank you very much, Tomas.
> > This resolves my problem.
> > 
> > -邮件原件-
> > 发件人: Users [mailto:users-boun...@clusterlabs.org] 代表 Tomas Jelinek
> > 发送时间: 2018年2月23日 17:37
> > 收件人: users@clusterlabs.org
> > 主题: Re: [ClusterLabs] 答复: How to configure to make each slave
> > resource 
> > has one VIP
> > 
> > Dne 23.2.2018 v 10:16 范国腾 napsal(a):
> > > Tomas,
> > > 
> > > Thank you very much. I do the change according to your
> > > suggestion 
> > > and it works.
> 
> One thing to keep in mind: a score of -INFINITY means the IPs will
> *never* run on the same node, even if one or more nodes go down. If
> that's what you want, of course, that's good. If you want the IPs to
> stay on different nodes normally, but be able to run on the same node
> in case of node outage, use a finite negative score.
> 
> > > 
> > > There is a question: If there are too much nodes (e.g.  total 10 
> > > slave nodes ), I need run "pcs constraint colocation add pgsql- 
> > > slave-ipx with pgsql-slave-ipy -INFINITY" many times. Is there a 
> > > simple command to do this?
> > 
> > I think colocation set does the trick:
> > pcs constraint colocation set pgsql-slave-ip1 pgsql-slave-ip2
> > pgsql-slave-ip3 setoptions score=-INFINITY You may specify as many 
> > resources as you need in this command.
> > 
> > Tomas
> > 
> > > 
> > > Master/Slave Set: pgsql-ha [pgsqld]
> > >   Masters: [ node1 ]
> > >   Slaves: [ node2 node3 ]
> > >   pgsql-master-ip(ocf::heartbeat:IPaddr2):   Started
> > > node1
> > >   pgsql-slave-ip1(ocf::heartbeat:IPaddr2):   Started
> > > node3
> > >   pgsql-slave-ip2(ocf::heartbeat:IPaddr2):   Started
> > > node2
> > > 
> > > Thanks
> > > Steven
> > > 
> > > -邮件原件-
> > > 发件人: Users [mailto:users-boun...@clusterlabs.org] 代表 Tomas
> > > Jelinek
> > > 发送时间: 2018年2月23日 17:02
> > > 收件人: users@clusterlabs.org
> > > 主题: Re: [ClusterLabs] How to configure to make each slave
> > > resource 
> > > has one VIP
> > > 
> > > Dne 23.2.2018 v 08:17 范国腾 napsal(a):
> > > > Hi,
> > > > 
> > > > Our system manages the database (one master and multiple
> > > > slave).
> > > > We
> > > > use one VIP for multiple Slave resources firstly.
> > > > 
> > > > Now I want to change the configuration that each slave
> > > > resource 
> > > > has a separate VIP. For example, I have 3 slave nodes and my
> > > > VIP 
> > > > group has
> > > > 2 vip; The 2 vips binds to node1 and node2 now; When the node2 
> > > > fails, the vip could move to the node3.
> > > > 
> > > > 
> > > > I use the following command to add the VIP
> > > > 
> > > > /      pcs resource group add pgsql-slave-group pgsql-slave-
> > > > ip1 
> > > > pgsql-slave-ip2/
> > > > 
> > > > /      pcs constraint colocation add pgsql-slave-group with
> > > > slave 
> > > > pgsql-ha INFINITY/
> > > > 
> > > > But now the two VIPs are the same nodes:
> > > > 
> > > > /Master/Slave Set: pgsql-ha [pgsqld]/
> > > > 
> > > > / Masters: [ node1 ]/
> > > > 
> > > > / Slaves: [ node2 node3 ]/
> > > > 
> > > > /pgsql-master-ip    (ocf::heartbeat:IPaddr2):  
> > > > Started 
> > > > node1/
> > > > 
> > > > /Resource Group: pgsql-slave-group/
> > > > 
> > > > */ pgsql-slave-ip1    (ocf::heartbeat:IPaddr2):  
> > > > Started
> > > > node2/*
> > > > 
> > > > */ pgsql-slave-ip2    (o

Re: [ClusterLabs] 答复: 答复: How to configure to make each slave resource has one VIP

[Ken Gaillot]

> 
> > > ___
> > > Users mailing list: Users@clusterlabs.org 
> > > https://lists.clusterlabs.org/mailman/listinfo/users
> > > 
> > > Project Home: http://www.clusterlabs.org Getting started:
> > > http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > > Bugs: http://bugs.clusterlabs.org
> > > 
> > 
> > ___
> > Users mailing list: Users@clusterlabs.org 
> > https://lists.clusterlabs.org/mailman/listinfo/users
> > 
> > Project Home: http://www.clusterlabs.org Getting started: 
> > http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
> > ___
> > Users mailing list: Users@clusterlabs.org 
> > https://lists.clusterlabs.org/mailman/listinfo/users
> > 
> > Project Home: http://www.clusterlabs.org Getting started: 
> > http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
> > 
> 
> ___
> Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.o
> rg/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
> ___
> Users mailing list: Users@clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
-- 
Ken Gaillot <kgail...@redhat.com>
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: Antw: Re: Resources not monitored in SLES11 SP4 (1.1.12-f47ea56)

[Ken Gaillot]

On Thu, 2018-06-28 at 09:09 +0200, Ulrich Windl wrote:
> > > > Ken Gaillot  schrieb am 27.06.2018 um
> > > > 16:18 in Nachricht
> 
> <1530109097.6452.1.ca...@redhat.com>:
> > On Wed, 2018-06-27 at 07:41 +0200, Ulrich Windl wrote:
> > > > > > Ken Gaillot  schrieb am 26.06.2018 um
> > > > > > 18:22 in Nachricht
> > > 
> > > <1530030128.5202.5.ca...@redhat.com>:
> > > > On Tue, 2018-06-26 at 10:45 +0300, Vladislav Bogdanov wrote:
> > > > > 26.06.2018 09:14, Ulrich Windl wrote:
> > > > > > Hi!
> > > > > > 
> > > > > > We just observed some strange effect we cannot explain in
> > > > > > SLES
> > > > > > 11
> > > > > > SP4 (pacemaker 1.1.12-f47ea56):
> > > > > > We run about a dozen of Xen PVMs on a three-node cluster
> > > > > > (plus
> > > > > > some
> > > > > > infrastructure and monitoring stuff). It worked all well so
> > > > > > far,
> > > > > > and there was no significant change recently.
> > > > > > However when a colleague stopped on VM for maintenance via
> > > > > > cluster
> > > > > > command, the cluster did not notice when the PVM actually
> > > > > > was
> > > > > > running again (it had been started not using the cluster (a
> > > > > > bad
> > > > > > idea, I know)).
> > > > > 
> > > > > To be on a safe side in such cases you'd probably want to
> > > > > enable 
> > > > > additional monitor for a "Stopped" role. Default one covers
> > > > > only 
> > > > > "Started" role. The same thing as for multistate resources,
> > > > > where
> > > > > you 
> > > > > need several monitor ops, for "Started/Slave" and "Master"
> > > > > roles.
> > > > > But, this will increase a load.
> > > > > And, I believe cluster should reprobe a resource on all nodes
> > > > > once
> > > > > you 
> > > > > change target-role back to "Started".
> > > > 
> > > > Which raises the question, how did you stop the VM initially?
> > > 
> > > I thought "(...) stopped one VM for maintenance via cluster
> > > command"
> > > is obvious. It was something like "crm resource stop ...".
> > > 
> > > > 
> > > > If you stopped it by setting target-role to Stopped, likely the
> > > > cluster
> > > > still thinks it's stopped, and you need to set it to Started
> > > > again.
> > > > If
> > > > instead you set maintenance mode or unmanaged the resource,
> > > > then
> > > > stopped the VM manually, then most likely it's still in that
> > > > mode
> > > > and
> > > > needs to be taken out of it.
> > > 
> > > The point was when the command to start the resource was given,
> > > the
> > > cluster had completely ignored the fact that it was running
> > > already
> > > and started to start the VM on a second node (which may be
> > > desastrous). But that's leading away from the main question...
> > 
> > Ah, this is expected behavior when you start a resource manually,
> > and
> > there are no monitors with target-role=Stopped. If the node where
> > you
> > manually started the VM isn't the same node the cluster happens to
> > choose, then you can get multiple active instances.
> > 
> > By default, the cluster assumes that where a probe found a resource
> > to
> > be not running, that resource will stay not running unless started
> > by
> > the cluster. (It will re-probe if the node goes away and comes
> > back.)
> 
> But didn't this behavior change? I tohought it was different maybe a
> year ago or so.

Not that I know of. We have fixed some issues around probes, especially
around probing Pacemaker Remote connections and the resources running
on those nodes, and around ordering of various actions with probes.

> > If you wish to guard against resources being started outside
> > cluster
> > control, configure a recurring monitor with target-role=Stopped,
> > and
> > the cluster will run that on all nodes where it thinks the resource
> > is
> > not supposed to be running. Of course

Re: [ClusterLabs] Install fresh pacemaker + corosync fails

[Ken Gaillot]

On Thu, 2018-06-28 at 17:17 +0200, Salvatore D'angelo wrote:
> Hi All,
> 
> I am here again. I am still fighting against upgrade problems but now
> I am trying to change the approach.
> I want now to try to install fresh a new version Corosync and
> Postgres to have it working.
> For the moment I am not interested to a specific configuration, just
> three nodes where I can run a dummy resource as in this tutorial.
> 
> I prefer to download specific version of the packages but I am ok to
> whatever new version for now.
> I followed the following procedure:
> https://wiki.clusterlabs.org/wiki/SourceInstall
> 
> but this procedure fails the compilation. If I want to compile from
> source it’s not clear what are the dependencies. 
> I started from a scratch Ubuntu 14.04 (I only configured ssh to
> connect to the machines).
> 
> For libqb I had to install with apt-get install the following
> dependencies:
> autoconf
> libtool 
> 
> Corosync compilation failed to the step 
> ./autogen.sh && ./configure --prefix=$PREFIX 
> with the following error:
> 
> checking for knet... no
> configure: error: Package requirements (libknet) were not met:
> 
> No package 'libknet' found
> 
> Consider adjusting the PKG_CONFIG_PATH environment variable if you
> installed software in a non-standard prefix.
> 
> I tried to install with apt-get the libraries libknet1 and libknet-
> dev but they were not found. Tried to download the source code of
> this library here:
> https://github.com/kronosnet/kronosnet
> 
> but the ./autogen.sh && ./configure --prefix=$PREFIX step failed too
> with this error:
> 
> configure: error: Package requirements (liblz4) were not met:
> No package 'liblz4’ found
> 
> I installed liblz4 and liblz4-dev but problem still occurs.
> 
> I am going around in circle here. I am asking if someone tested the
> install procedure on Ubuntu 14.04 and can give me the exact steps to
> install fresh pacemaker 1.1.18 (or later) with corosync 2.4.4 (or
> later).
> Thanks in advance for help.
> 

Pacemaker's dependencies are listed at:

https://github.com/ClusterLabs/pacemaker/blob/master/INSTALL.md

Some of the package names may be different on Ubuntu. Unless you are
looking for specific features, I'd go with whatever stock packages are
available for libqb and corosync.

knet will be supported by corosync 3 and is bleeding-edge at the moment
(though probably solid). If you do want to compile libqb and/or
corosync, the guide on the wiki grabs the latest master branch of the
various projects using git; I'd recommend downloading source tarballs
of the latest official releases instead, as they will be more stable.
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Pacemaker not restarting Resource on same node

[Ken Gaillot]

On Thu, 2018-06-28 at 19:58 +0300, Andrei Borzenkov wrote:
> 28.06.2018 18:35, Dileep V Nair пишет:
> > 
> > 
> > Hi,
> > 
> > I have a cluster with DB2 running in HADR mode. I have used the
> > db2
> > resource agent. My problem is whenever DB2 fails on primary it is
> > migrating
> > to the secondary node. Ideally it should restart thrice (Migration
> > Threshold set to 3) but not happening. This is causing extra
> > downtime for
> > customer. Is there any other settings / parameters which needs to
> > be set.
> > Did anyone face similar issue ? I am on pacemaker version 1.1.15-
> > 21.1.
> > 
> 
> It is impossible to answer without good knowledge of application and
> resource agent. From quick look at resource agent, it removes master
> score from current node if database failure is detected which means
> current node will not be eligible for fail-over.
> 
> Note that pacemaker does not really have concept of "restarting
> resource
> on the same node". Every time it performs full node selection using
> current scores. It usually happens to be "same node" simply due to
> non-zero resource stickiness by default. You could attempt to adjust
> stickiness so that final score will be larger than master score on
> standby. But that also needs agent cooperation - are you sure agent
> will
> even attempt to restart failed master locally?

Also, some types of errors cannot be recovered by a restart on the same
node.

For example, by default, start failures will not be retried on the same
node (see the cluster property start-failure-is-fatal), to avoid a
repeatedly failing start preventing the cluster from doing anything
else. Certain OCF resource agent exit codes are considered "hard"
errors that prevent retrying on the same node: missing dependencies,
file permission errors, etc.
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: Antw: Re: Resources not monitored in SLES11 SP4 (1.1.12-f47ea56)

[Ken Gaillot]

On Thu, 2018-06-28 at 09:13 +0200, Ulrich Windl wrote:
> > > > Ken Gaillot  schrieb am 27.06.2018 um
> > > > 16:32 in Nachricht
> 
> <1530109926.6452.3.ca...@redhat.com>:
> > On Wed, 2018-06-27 at 09:18 -0500, Ken Gaillot wrote:
> > > On Wed, 2018-06-27 at 07:41 +0200, Ulrich Windl wrote:
> > > > > > > Ken Gaillot  schrieb am 26.06.2018
> > > > > > > um
> > > > > > > 18:22 in Nachricht
> > > > 
> > > > <1530030128.5202.5.ca...@redhat.com>:
> > > > > On Tue, 2018-06-26 at 10:45 +0300, Vladislav Bogdanov wrote:
> > > > > > 26.06.2018 09:14, Ulrich Windl wrote:
> > > > > > > Hi!
> > > > > > > 
> > > > > > > We just observed some strange effect we cannot explain in
> > > > > > > SLES
> > > > > > > 11
> > > > > > > SP4 (pacemaker 1.1.12-f47ea56):
> > > > > > > We run about a dozen of Xen PVMs on a three-node cluster
> > > > > > > (plus
> > > > > > > some
> > > > > > > infrastructure and monitoring stuff). It worked all well
> > > > > > > so
> > > > > > > far,
> > > > > > > and there was no significant change recently.
> > > > > > > However when a colleague stopped on VM for maintenance
> > > > > > > via
> > > > > > > cluster
> > > > > > > command, the cluster did not notice when the PVM actually
> > > > > > > was
> > > > > > > running again (it had been started not using the cluster
> > > > > > > (a
> > > > > > > bad
> > > > > > > idea, I know)).
> > > > > > 
> > > > > > To be on a safe side in such cases you'd probably want to
> > > > > > enable 
> > > > > > additional monitor for a "Stopped" role. Default one covers
> > > > > > only 
> > > > > > "Started" role. The same thing as for multistate resources,
> > > > > > where
> > > > > > you 
> > > > > > need several monitor ops, for "Started/Slave" and "Master"
> > > > > > roles.
> > > > > > But, this will increase a load.
> > > > > > And, I believe cluster should reprobe a resource on all
> > > > > > nodes
> > > > > > once
> > > > > > you 
> > > > > > change target-role back to "Started".
> > > > > 
> > > > > Which raises the question, how did you stop the VM initially?
> > > > 
> > > > I thought "(...) stopped one VM for maintenance via cluster
> > > > command"
> > > > is obvious. It was something like "crm resource stop ...".
> > > > 
> > > > > 
> > > > > If you stopped it by setting target-role to Stopped, likely
> > > > > the
> > > > > cluster
> > > > > still thinks it's stopped, and you need to set it to Started
> > > > > again.
> > > > > If
> > > > > instead you set maintenance mode or unmanaged the resource,
> > > > > then
> > > > > stopped the VM manually, then most likely it's still in that
> > > > > mode
> > > > > and
> > > > > needs to be taken out of it.
> > > > 
> > > > The point was when the command to start the resource was given,
> > > > the
> > > > cluster had completely ignored the fact that it was running
> > > > already
> > > > and started to start the VM on a second node (which may be
> > > > desastrous). But that's leading away from the main question...
> > > 
> > > Ah, this is expected behavior when you start a resource manually,
> > > and
> > > there are no monitors with target-role=Stopped. If the node where
> > > you
> > > manually started the VM isn't the same node the cluster happens
> > > to
> > > choose, then you can get multiple active instances.
> > > 
> > > By default, the cluster assumes that where a probe found a
> > > resource
> > > to
> > > be not running, that resource will stay not running unless
> > > started by
> > > the cluster. (It

Re: [ClusterLabs] Antw: Salvaging aborted resource migration

[Ken Gaillot]

On Thu, 2018-09-27 at 09:36 +0200, Ulrich Windl wrote:
> Hi!
> 
> Obviously you violated the most important cluster rule that is "be
> patient".
> Maybe the next important is "Don't change the configuration while the
> cluster
> is not in IDLE state" ;-)

Agreed -- although even idle, removing a ban can result in a migration
back (if something like stickiness doesn't prevent it).

There's currently no way to tell pacemaker that an operation (i.e.
migrate_from) is a no-op and can be ignored. If a migration is only
partially completed, it has to be considered a failure and reverted.

I'm not sure why the reload was scheduled; I suspect it's a bug due to
a restart being needed but no parameters having changed. There should
be special handling for a partial migration to make the stop required.

> I feel these are issues that should be fixed, but the above rules
> make your
> life easier while these issues still exist.
> 
> Regards,
> Ulrich
> 
> > > > Ferenc Wágner  schrieb am 27.09.2018
> > > > um 08:37
> 
> in
> Nachricht <87tvmb5ttw@lant.ki.iif.hu>:
> > Hi,
> > 
> > The current behavior of cancelled migration with Pacemaker 1.1.16
> > with a
> > resource implementing push migration:
> > 
> > # /usr/sbin/crm_resource ‑‑ban ‑r vm‑conv‑4
> > 
> > vhbl03 crmd[10017]:   notice: State transition S_IDLE ‑>
> > S_POLICY_ENGINE
> > vhbl03 pengine[10016]:   notice: Migrate vm‑conv‑4#011(Started
> > vhbl07 ‑>
> 
> vhbl04)
> > vhbl03 crmd[10017]:   notice: Initiating migrate_to operation 
> > vm‑conv‑4_migrate_to_0 on vhbl07
> > vhbl03 pengine[10016]:   notice: Calculated transition 4633, saving
> > inputs 
> > in /var/lib/pacemaker/pengine/pe‑input‑1069.bz2
> > [...]
> > 
> > At this point, with the migration still ongoing, I wanted to get
> > rid of
> > the constraint:
> > 
> > # /usr/sbin/crm_resource ‑‑clear ‑r vm‑conv‑4
> > 
> > vhbl03 crmd[10017]:   notice: Transition aborted by deletion of 
> > rsc_location[@id='cli‑ban‑vm‑conv‑4‑on‑vhbl07']: Configuration
> > change
> > vhbl07 crmd[10233]:   notice: Result of migrate_to operation for
> > vm‑conv‑4
> 
> on 
> > vhbl07: 0 (ok)
> > vhbl03 crmd[10017]:   notice: Transition 4633 (Complete=6,
> > Pending=0, 
> > Fired=0, Skipped=1, Incomplete=6, 
> > Source=/var/lib/pacemaker/pengine/pe‑input‑1069.bz2): Stopped
> > vhbl03 pengine[10016]:   notice: Resource vm‑conv‑4 can no longer
> > migrate to
> > vhbl04. Stopping on vhbl07 too
> > vhbl03 pengine[10016]:   notice: Reload  vm‑conv‑4#011(Started
> > vhbl07)
> > vhbl03 pengine[10016]:   notice: Calculated transition 4634, saving
> > inputs 
> > in /var/lib/pacemaker/pengine/pe‑input‑1070.bz2
> > vhbl03 crmd[10017]:   notice: Initiating stop operation
> > vm‑conv‑4_stop_0 on
> > vhbl07
> > vhbl03 crmd[10017]:   notice: Initiating stop operation
> > vm‑conv‑4_stop_0 on
> > vhbl04
> > vhbl03 crmd[10017]:   notice: Initiating reload operation
> > vm‑conv‑4_reload_0
> > on vhbl04
> > 
> > This recovery was entirely unnecessary, as the resource
> > successfully
> > migrated to vhbl04 (the migrate_from operation does
> > nothing).  Pacemaker
> > does not know this, but is there a way to educate it?  I think in
> > this
> > special case it is possible to redesign the agent making migrate_to
> > a
> > no‑op and doing everything in migrate_from, which would
> > significantly
> > reduce the window between the start points of the two "halfs", but
> > I'm
> > not sure that would help in the end: Pacemaker could still decide
> > to do
> > an unnecessary stop+start recovery.  Would it?  I failed to find
> > any
> > documentation on recovery from aborted migration transitions.  I
> > don't
> > expect on‑fail (for migrate_* ops, not me) to apply here, does it?
> > 
> > Side question: why initiate a reload in any case, like above?
> > 
> > Even more side question: could you please consider using space
> > instead
> > of TAB in syslog messages?  (Actually, I wouldn't mind getting rid
> > of
> > them altogether in any output.)
> > ‑‑ 
> > Thanks,
> > Feri
> > ___________
> > Users mailing list: Users@clusterlabs.org 
> > https://lists.clusterlabs.org/mailman/listinfo/users 
> > 
> > Project Home: http://www.clusterlabs.org 
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratc
> > h.pdf 
> > Bugs: http://bugs.clusterlabs.org 
> 
> 
> 
> ___
> Users mailing list: Users@clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Understanding the behavior of pacemaker crash

[Ken Gaillot]

On Thu, 2018-09-27 at 13:45 +0530, Prasad Nagaraj wrote:
> Hello - I was trying to understand the behavior or cluster when
> pacemaker crashes on one of the nodes. So I hard killed pacemakerd
> and its related processes.
> 
> ---
> -
> [root@SG-mysqlold-907 azureuser]# ps -ef | grep pacemaker
> root      74022      1  0 07:53 pts/0    00:00:00 pacemakerd
> 189       74028  74022  0 07:53 ?        00:00:00
> /usr/libexec/pacemaker/cib
> root      74029  74022  0 07:53 ?        00:00:00
> /usr/libexec/pacemaker/stonithd
> root      74030  74022  0 07:53 ?        00:00:00
> /usr/libexec/pacemaker/lrmd
> 189       74031  74022  0 07:53 ?        00:00:00
> /usr/libexec/pacemaker/attrd
> 189       74032  74022  0 07:53 ?        00:00:00
> /usr/libexec/pacemaker/pengine
> 189       74033  74022  0 07:53 ?        00:00:00
> /usr/libexec/pacemaker/crmd
> 
> root      75228  50092  0 07:54 pts/0    00:00:00 grep pacemaker
> [root@SG-mysqlold-907 azureuser]# kill -9 74022
> 
> [root@SG-mysqlold-907 azureuser]# ps -ef | grep pacemaker
> root      74030      1  0 07:53 ?        00:00:00
> /usr/libexec/pacemaker/lrmd
> 189       74032      1  0 07:53 ?        00:00:00
> /usr/libexec/pacemaker/pengine
> 
> root      75303  50092  0 07:55 pts/0    00:00:00 grep pacemaker
> [root@SG-mysqlold-907 azureuser]# kill -9 74030
> [root@SG-mysqlold-907 azureuser]# kill -9 74032
> [root@SG-mysqlold-907 azureuser]# ps -ef | grep pacemaker
> root      75332  50092  0 07:55 pts/0    00:00:00 grep pacemaker
> 
> [root@SG-mysqlold-907 azureuser]# crm satus
> ERROR: status: crm_mon (rc=107): Connection to cluster failed:
> Transport endpoint is not connected
> ---
> --
> 
> However, this does not seem to be having any effect on the cluster
> status from other nodes
> ---
> 
> 
> [root@SG-mysqlold-909 azureuser]# crm status
> Last updated: Thu Sep 27 07:56:17 2018          Last change: Thu Sep
> 27 07:53:43 2018 by root via crm_attribute on SG-mysqlold-909
> Stack: classic openais (with plugin)
> Current DC: SG-mysqlold-908 (version 1.1.14-8.el6_8.1-70404b0) -
> partition with quorum
> 3 nodes and 3 resources configured, 3 expected votes
> 
> Online: [ SG-mysqlold-907 SG-mysqlold-908 SG-mysqlold-909 ]

It most definitely would make the node offline, and if fencing were
configured, the rest of the cluster would fence the node to make sure
it's safely down.

I see you're using the old corosync 1 plugin. I suspect what happened
in this case is that corosync noticed the plugin died and restarted it
quickly enough that it had rejoined by the time you checked the status
elsewhere.

> 
> Full list of resources:
> 
>  Master/Slave Set: ms_mysql [p_mysql]
>      Masters: [ SG-mysqlold-909 ]
>      Slaves: [ SG-mysqlold-907 SG-mysqlold-908 ]
> 
> 
> [root@SG-mysqlold-908 azureuser]# crm status
> Last updated: Thu Sep 27 07:56:08 2018          Last change: Thu Sep
> 27 07:53:43 2018 by root via crm_attribute on SG-mysqlold-909
> Stack: classic openais (with plugin)
> Current DC: SG-mysqlold-908 (version 1.1.14-8.el6_8.1-70404b0) -
> partition with quorum
> 3 nodes and 3 resources configured, 3 expected votes
> 
> Online: [ SG-mysqlold-907 SG-mysqlold-908 SG-mysqlold-909 ]
> 
> Full list of resources:
> 
>  Master/Slave Set: ms_mysql [p_mysql]
>      Masters: [ SG-mysqlold-909 ]
>      Slaves: [ SG-mysqlold-907 SG-mysqlold-908 ]
> 
> ---
> ---
> 
> I am bit surprised that other nodes are not able to detect that
> pacemaker is down on one of the nodes - SG-mysqlold-907 
> 
> Even if I kill pacemaker on the node which is a DC - I observe the
> same behavior with rest of the nodes not detecting that DC is down. 
> 
> Could some one explain what is the expected behavior in these cases ?
>  
> I am using corosync 1.4.7 and pacemaker 1.1.14
> 
> Thanks in advance
> Prasad
> 
> _______
> Users mailing list: Users@clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: pcmk 1.1.17: Which effective user is calling OCF agents for querying meta-data?

[Ken Gaillot]

On Thu, 2018-09-27 at 10:23 +, cfpubl...@verimatrix.com wrote:
> > > With pacemaker 1.1.17, we observe the following messages during 
> > > startup of
> > > pacemaker:
> > > 2018-09-18T11:58:18.452951+03:00 p12-0001-bcsm03
> > > crmd[2871]:  warning: 
> > > Cannot execute '/usr/lib/ocf/resource.d/verimatrix/anything4': 
> > > Permission denied (13)
> > > 2018-09-18T11:58:18.453179+03:00 p12-0001-bcsm03
> > > crmd[2871]:error: 
> > > Failed to retrieve meta-data for ocf:verimatrix:anything4
> > > 2018-09-18T11:58:18.453291+03:00 p12-0001-bcsm03
> > > crmd[2871]:error: No 
> > > metadata for ocf::verimatrix:anything4
> > > 
> > 
> > Could it be as simple as
> > /usr/lib/ocf/resource.d/verimatrix/anything4 not having the execute
> > bit set (for the user)?
> 
> The OCF agent is not physically located there.
> /usr/lib/ocf/resource.d/verimatrix is a symbolic link that points to
> a directory in our software distribution. That part is not reachable
> for "other".
> 
> > > 
> > > It seems that on startup, crmd is querying the meta-data on the
> > > OCF 
> > > agents using a non-root user (hacluster?) while the regular
> > > resource 
> > > control activity seems to be done as root.
> > > The OCF resource in question intentionally resides in a directory
> > > that 
> > > is inaccessible to non-root users.
> > 
> > Why? You can selectively grat access (man setfacl)!
> 
> That's an option, thank you.
> 
> > > 
> > > Is this behavior of using different users intended? If yes, any
> > > clue 
> > > why was
> > > it working with pacemaker 1.1.7 under RHEL6?
> > 
> > Finally: Why are you asking thei list for help, when you removed
> > execute permission for your home-grown (as it seems) resource
> > agent?
> > What could WE do?
> 
> I would like to understand the concept behind it to determine the
> best solution. If there was a way to configure or tell crmd to do the
> meta-data query as root, I would prefer that. It's because even if
> the actual access to the OCF agent scripts worked, we would hit the
> next problems because some OCF scripts use the "ocf_is_root" function
> from /usr/lib/ocf/lib/heartbeat/ocf-shellfuncs. These OCFs would fail
> miserably.
> So before I revisit all our OCFs to check if the well-behave if
> called as non-root, I wanted to check if there is another way.
> 
> Thanks,
>   Carsten

The crmd does indeed run meta-data queries as the hacluster user, while
the lrmd executes all other resource actions as root. It's a
longstanding goal to correct this by making crmd go through lrmd to get
meta-data, but unfortunately it's a big project and there are many more
pressing issues to address. :-(

There's no workaround within pacemaker, but the setfacl approach sounds
useful.

As a best practice, an agent's meta-data action should not do anything
other than print meta-data. I.e. many agents have common initialization
that needs to be done before all actions, but this should be avoided
for meta-data. Even without this issue, regular users and higher-level
tools should be able to obtain meta-data without permission issues or
side effects.
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Corosync 3 release plans?

[Ken Gaillot]

On Thu, 2018-09-27 at 15:32 +0200, Ferenc Wágner wrote:
> Christine Caulfield  writes:
> 
> > TBH I would be quite happy to leave this to logrotate but the
> > message I
> > was getting here is that we need additional help from libqb. I'm
> > willing
> > to go with a consensus on this though
> 
> Yes, to do a proper job logrotate has to have a way to get the log
> files
> reopened.  And applications can't do that without support from libqb,
> if
> I understood Honza right.

There are two related issues:

* Issue #142, about automatically rotating logs once they reach a
certain size, can be done with logrotate already. If it's a one-time
thing (e.g. running a test with trace), the admin can control rotation
directly with logrotate --force /etc/logrotate.d/whatever.conf. If it's
desired permanently, a maxsize or size line can be added to the
logrotate config (which, now that I think about it, would be a good
idea for the default pacemaker config).

* Issue #239, about the possibility of losing messages with
copytruncate, has a widely used, easily implemented, and robust
solution of using a signal to indicate reopening a log. logrotate is
then configured to rotate by moving the log to a new name, sending the
signal, then compressing the old log.
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Corosync 3 release plans?

[Ken Gaillot]

On Thu, 2018-09-27 at 09:58 -0500, Ken Gaillot wrote:
> On Thu, 2018-09-27 at 15:32 +0200, Ferenc Wágner wrote:
> > Christine Caulfield  writes:
> > 
> > > TBH I would be quite happy to leave this to logrotate but the
> > > message I
> > > was getting here is that we need additional help from libqb. I'm
> > > willing
> > > to go with a consensus on this though
> > 
> > Yes, to do a proper job logrotate has to have a way to get the log
> > files
> > reopened.  And applications can't do that without support from
> > libqb,
> > if
> > I understood Honza right.
> 
> There are two related issues:
> 
> * Issue #142, about automatically rotating logs once they reach a
> certain size, can be done with logrotate already. If it's a one-time
> thing (e.g. running a test with trace), the admin can control
> rotation
> directly with logrotate --force /etc/logrotate.d/whatever.conf. If
> it's
> desired permanently, a maxsize or size line can be added to the
> logrotate config (which, now that I think about it, would be a good
> idea for the default pacemaker config).
> 
> * Issue #239, about the possibility of losing messages with
> copytruncate, has a widely used, easily implemented, and robust
> solution of using a signal to indicate reopening a log. logrotate is
> then configured to rotate by moving the log to a new name, sending
> the
> signal, then compressing the old log.

Regarding implementation in libqb's case, libqb would simply provide
the API for reopening the log, and clients such as pacemaker would
intercept the signal and call the API.

A minor complication is that pacemaker would have to supply different
logrotate configs depending on the version of libqb available.
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Corosync 3 release plans?

[Ken Gaillot]

On Thu, 2018-09-27 at 16:09 +0100, Christine Caulfield wrote:
> On 27/09/18 16:01, Ken Gaillot wrote:
> > On Thu, 2018-09-27 at 09:58 -0500, Ken Gaillot wrote:
> > > On Thu, 2018-09-27 at 15:32 +0200, Ferenc Wágner wrote:
> > > > Christine Caulfield  writes:
> > > > 
> > > > > TBH I would be quite happy to leave this to logrotate but the
> > > > > message I
> > > > > was getting here is that we need additional help from libqb.
> > > > > I'm
> > > > > willing
> > > > > to go with a consensus on this though
> > > > 
> > > > Yes, to do a proper job logrotate has to have a way to get the
> > > > log
> > > > files
> > > > reopened.  And applications can't do that without support from
> > > > libqb,
> > > > if
> > > > I understood Honza right.
> > > 
> > > There are two related issues:
> > > 
> > > * Issue #142, about automatically rotating logs once they reach a
> > > certain size, can be done with logrotate already. If it's a one-
> > > time
> > > thing (e.g. running a test with trace), the admin can control
> > > rotation
> > > directly with logrotate --force /etc/logrotate.d/whatever.conf.
> > > If
> > > it's
> > > desired permanently, a maxsize or size line can be added to the
> > > logrotate config (which, now that I think about it, would be a
> > > good
> > > idea for the default pacemaker config).
> > > 
> > > * Issue #239, about the possibility of losing messages with
> > > copytruncate, has a widely used, easily implemented, and robust
> > > solution of using a signal to indicate reopening a log. logrotate
> > > is
> > > then configured to rotate by moving the log to a new name,
> > > sending
> > > the
> > > signal, then compressing the old log.
> > 
> > Regarding implementation in libqb's case, libqb would simply
> > provide
> > the API for reopening the log, and clients such as pacemaker would
> > intercept the signal and call the API.
> > 
> 
> That sounds pretty easy to achieve. I'm also looking into high-res
> timestamps for logfiles too.
> 
> > A minor complication is that pacemaker would have to supply
> > different
> > logrotate configs depending on the version of libqb available.
> > 
> 
> Can't you just intercept the signal anyway and not do anything if an
> old
> libqb is linked in?
> 
> Chrissie

Yes, but the logrotate config will need to keep the copytruncate option
or not depending on whether the new API is available. It's not too
difficult via the configure script, just reminding myself to do it :)
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: pcmk 1.1.17: Which effective user is calling OCF agents for querying meta-data?

[Ken Gaillot]

On Thu, 2018-09-27 at 14:57 +, cfpubl...@verimatrix.com wrote:
> > On Thu, 2018-09-27 at 10:23 +, cfpubl...@verimatrix.com wrote:
> > > > > With pacemaker 1.1.17, we observe the following messages
> > > > > during 
> > > > > startup of
> > > > > pacemaker:
> > > > > 2018-09-18T11:58:18.452951+03:00 p12-0001-bcsm03
> > > > > crmd[2871]:  warning:
> > > > > Cannot execute
> > > > > '/usr/lib/ocf/resource.d/verimatrix/anything4':
> > > > > Permission denied (13)
> > > > > 2018-09-18T11:58:18.453179+03:00 p12-0001-bcsm03
> > > > > crmd[2871]:error:
> > > > > Failed to retrieve meta-data for ocf:verimatrix:anything4
> > > > > 2018-09-18T11:58:18.453291+03:00 p12-0001-bcsm03
> > > > > crmd[2871]:error: No
> > > > > metadata for ocf::verimatrix:anything4
> > > > > 
> > > > 
> > > > Could it be as simple as
> > > > /usr/lib/ocf/resource.d/verimatrix/anything4 not having the
> > > > execute 
> > > > bit set (for the user)?
> > > 
> > > The OCF agent is not physically located there.
> > > /usr/lib/ocf/resource.d/verimatrix is a symbolic link that points
> > > to a 
> > > directory in our software distribution. That part is not
> > > reachable for 
> > > "other".
> > > 
> > > > > 
> > > > > It seems that on startup, crmd is querying the meta-data on
> > > > > the 
> > > > > OCF agents using a non-root user (hacluster?) while the
> > > > > regular 
> > > > > resource control activity seems to be done as root.
> > > > > The OCF resource in question intentionally resides in a
> > > > > directory 
> > > > > that is inaccessible to non-root users.
> > > > 
> > > > Why? You can selectively grat access (man setfacl)!
> > > 
> > > That's an option, thank you.
> > > 
> > > > > 
> > > > > Is this behavior of using different users intended? If yes,
> > > > > any 
> > > > > clue why was it working with pacemaker 1.1.7 under RHEL6?
> > > > 
> > > > Finally: Why are you asking thei list for help, when you
> > > > removed 
> > > > execute permission for your home-grown (as it seems) resource
> > > > agent?
> > > > What could WE do?
> > > 
> > > I would like to understand the concept behind it to determine the
> > > best 
> > > solution. If there was a way to configure or tell crmd to do the 
> > > meta-data query as root, I would prefer that. It's because even
> > > if the 
> > > actual access to the OCF agent scripts worked, we would hit the
> > > next 
> > > problems because some OCF scripts use the "ocf_is_root" function
> > > from 
> > > /usr/lib/ocf/lib/heartbeat/ocf-shellfuncs. These OCFs would fail 
> > > miserably.
> > > So before I revisit all our OCFs to check if the well-behave if
> > > called 
> > > as non-root, I wanted to check if there is another way.
> > > 
> > > Thanks,
> > >   Carsten
> > 
> > The crmd does indeed run meta-data queries as the hacluster user,
> > while the lrmd executes all other resource actions as root. It's a
> > longstanding goal to correct this by making crmd go through lrmd to
> > get meta-data, but unfortunately it's a big project and there are
> > many
> > more pressing issues to address. :-(
> > 
> > There's no workaround within pacemaker, but the setfacl approach
> > sounds useful.
> > 
> > As a best practice, an agent's meta-data action should not do
> > anything other than print meta-data. I.e. many agents have common
> > initialization that needs to be done before all actions, but this
> > should be avoided for meta-data. Even without this issue, regular
> > users and
> > higher-level tools should be able to obtain meta-data without
> > permission issues or side effects.
> >  --
> > Ken Gaillot 
> 
> Thank you, Ken, for the confirmation that there is no workaround
> within pacemaker. I will re-visit all our OCFs then and make sure
> they are world accessible and the meta-data action requires no
> special permissions.
> 
> I still wonder why it worked with pacemaker 1.1.7, though ;-)
> 
> Thanks,
>   Carsten

I'm going to guess you were using the corosync 1 plugin, and that it
ran everything as root. I never worked with that code, so it's just a
guess :)
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Salvaging aborted resource migration

[Ken Gaillot]

On Thu, 2018-09-27 at 18:00 +0200, Ferenc Wágner wrote:
> Ken Gaillot  writes:
> 
> > On Thu, 2018-09-27 at 09:36 +0200, Ulrich Windl wrote:
> > 
> > > Obviously you violated the most important cluster rule that is
> > > "be
> > > patient".  Maybe the next important is "Don't change the
> > > configuration while the cluster is not in IDLE state" ;-)
> > 
> > Agreed -- although even idle, removing a ban can result in a
> > migration
> > back (if something like stickiness doesn't prevent it).
> 
> I've got no problem with that in general.  However, I can't gurantee
> that every configuration change happens in idle state, certain
> operations (mostly resource additions) are done by several
> administrators without synchronization, and of course asynchronous
> cluster events can also happen any time.  So I have to ask: what are
> the
> consequences of breaking this "impossible" rule?

It's not truly a rule, just a "better safe than sorry" approach. In
general the cluster is very forgiving about frequent config changes
from any node. The only non-obvious consequence is that if the cluster
is still making changes based on the previous config, then it will wait
for any action already in progress to complete, then abandon that and
recalculate based on the new config, which might reverse actions just
taken.

> > There's currently no way to tell pacemaker that an operation (i.e.
> > migrate_from) is a no-op and can be ignored. If a migration is only
> > partially completed, it has to be considered a failure and
> > reverted.
> 
> OK.  Are there other complex operations which can "partially
> complete"
> if a transition is aborted by some event?

I believe migration is the only one in that category. Perhaps a restart
could be considered similar, as it involves a separate stop and start,
but a completed stop doesn't have to be reversed in that case, so that
wouldn't cause any similar issues.

> Now let's suppose a pull migration scenario: migrate_to does nothing,
> but in this tiny window a configuration change aborts the transition.
> The resources would go through a full recovery (stop+start), right?

Yes

> Now let's suppose migrate_from gets scheduled and starts performing
> the
> migration.  Before it finishes, a configuration change aborts the
> transition.  The cluster waits for the outstanding operation to
> finish,
> doesn't it?  And if it finishes successfully, is the migration
> considered complete requiring no recovery?

Correct. If an agent has actually been executed, the cluster will wait
for that operation to complete or timeout before recalculating.

(As an aside, that can cause problems of a different sort: if an
operation in progress has a very long timeout and takes that whole
time, it can delay recovery of other resources that newly fail, even if
their recovery would not depend on the outcome of that operation.
That's a complicated problem to solve because that last clause is not
obvious to a computer program without simulating all possible results,
and even then, it can't be sure that the operation won't do something
like change a node attribute that might affect other resources.)

> > I'm not sure why the reload was scheduled; I suspect it's a bug due
> > to
> > a restart being needed but no parameters having changed. There
> > should
> > be special handling for a partial migration to make the stop
> > required.
> 
> Probably CLBZ#5309 again...  You debugged a pe-input file for me with
> a
> similar issue almost exactly a year ago (thread subject "Pacemaker
> resource parameter reload confusion").  Time to upgrade this cluster,
> I
> guess.
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Position of pacemaker in today's HA world

[Ken Gaillot]

be needed -- making the framework complete, offering the
> library/API so that the applications are built on top of this
> natively.  An example of this I've seen when doing a brief research
> some time ago is <https://github.com/NetComposer/nkcluster>
> (on that note, Erlang was designed specifically with fault-tolerant,
> resilient and distributed applications in mind, making me wonder
> if it was ever considered originally in what later became pacemaker).

Not that I'm aware of. The first problem that comes to mind is the
ecosystem; I don't think there's the breadth of library support that
pacemaker needs, nor a large enough community of users. My only
personal encounters with erlang have been via rabbit-mq, and that
hasn't been positive (high resource usage and questionable
reliability).

Personally, I'm not fond of functional programming in general. I find
it to be the veganism of programming languages. (No disrespect for
anyone else's choice, but it's not for me ...)

> Also, one of the fields where pacemaker used to be very helpful was
> a concise startup/shutdown ordering amongst multiple on-node
> services.  This is partially obviated with smart init managers, most
> blatantly systemd on Linux platform, playing whole another league
> than old, inflexible init systems of the past when the foundation
> of pacemaker was laid out.

Agreed, though those still lack cross-node dependencies

> 
> * * *
> 
> Please, don't take this as a blasphemy, I am just trying to put my

I think the only blasphemy in any tech field is not looking ahead

> head out of the tunnel (or sand, if you want), to view the value of
> corosync-pacemaker stack in the IT infrastructures of today and
> future,
> and to gather feedback on this topic, perhaps together with ideas how
> to stay indeed relevant amongst all the "private clustering",
> management and orchestration of resources proliferation we
> can observe for the past years (which makes the surface slightly
> different than it was when heartbeat [and Red Hat Cluster Suite]
> was the thing).
> 
> Please share your thoughts with me/us, even if it will not be
> the most encouraging thing to hear, since
> - staying realistic is important
> - staying relevant is what prevents becoming a fossil tomorrow
> :-)
> 
> Happy World Teacher's Day.

I think the biggest challenge today is that the ClusterLabs stack is
infrastructure, and infrastructure is increasingly concentrated in a
small number of cloud providers. Fewer organizations of all sizes have
their own data center space. The danger is a shrinking community base
and ever greater dependence on a handful of backers. Thankfully, we've
seen the opposite in the past few years -- the community has grown. I
think the consolidation of cluster stacks and improvement of the Debian
packages have helped in that regard. There's been a trickling of
interest from *BSD and Arch users, and I think encouraging that however
we can would be helpful.

The next big challenge is that high availability is becoming a subset
of the "orchestration" space in terms of how we fit into IT
departments. Systemd and Kubernetes are the clear leaders in service
orchestration today and likely will be for a long while. Other forms of
orchestration such as Ansible are also highly relevant. Tighter
integration with these would go a long way toward establishing
longevity.

That brings up another challenge, which is developer resources. It is
difficult to keep up with triaging bug reports much less handling them.
We occasionally have the opportunity to add significant new
enhancements (bundles and alerts come to mind), but the size and
frequency of such projects are greatly limited. It would be awesome to
adapt Pacemaker to be a Kubernetes scheduler, or a cross-node
dependency manager for systemd, but a whole new team of developers
would be needed to tackle something that size. And if we're asking
Santa Claus, I'd like two or three additional full-time developers to
focus on known bugs and RFEs ...
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] weird corosync - [TOTEM ] FAILED TO RECEIVE

[Ken Gaillot]

On Fri, 2018-10-12 at 15:51 +0100, lejeczek wrote:
> hi guys,
> I have a 3-node cluser(centos 7.5), 2 nodes seems fine but 
> third(or probably something else in between) is not right.
> I see this:
> 
>   $ pcs status --all
> Cluster name: CC
> Stack: corosync
> Current DC: whale.private (version 
> 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum
> Last updated: Fri Oct 12 15:40:39 2018
> Last change: Fri Oct 12 15:14:57 2018 by root via 
> crm_resource on whale.private
> 
> 3 nodes configured
> 8 resources configured (1 DISABLED)
> 
> Online: [ rental.private whale.private ]
> OFFLINE: [ rider.private ]
> 
> and that third node logs:
> 
> [TOTEM ] FAILED TO RECEIVE
>   [TOTEM ] A new membership (10.5.6.100:2504344) was formed. 
> Members left: 2 4
>   [TOTEM ] Failed to receive the leave message. failed: 2 4
>   [QUORUM] Members[1]: 1
>   [MAIN  ] Completed service synchronization, ready to 
> provide service.
>   [TOTEM ] A new membership (10.5.6.49:2504348) was formed. 
> Members joined: 2 4
>   [TOTEM ] FAILED TO RECEIVE
> 
> and it just keeps going like that.
> Sometimes reboot(or stop of services + wait + start) of that 
> third node would help.
> But, I get this situation almost every time a node gets 
> (orderly) shut down or reboot.
> Network-wise, connectivity, seem okey. Where to start?
> 
> many thanks, L

Odd. I'd recommend turning on debug logging in corosync.conf, and
posting the log here. Hopefully one of the corosync developers can
chime in at that point.
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] How to generate RPMs for Pacemaker release 2.x on Centos

[Ken Gaillot]

On Mon, 2018-10-15 at 14:39 +0200, Klaus Wenninger wrote:
> On 10/15/2018 01:52 PM, Lopez, Francisco Javier [Global IT] wrote:
> > Hello guys !
> > 
> > We are planning to use Pacemaker as a base HA Software in our
> > Company.
> > 
> > Our requirements will be:
> > 
> > - Centos-7
> > - PostgreSql-10
> > 
> > We did several tests with Pacemaker release 1.1.8 and fixed the
> > problems found with
> > the RA. We finally created new RPMs from source (4.x).
> > 
> > Now we want to test Pacemaker release 2.x but, as we will have to
> > create some clusters,
> > we want to create new RPMs for this release instead of doing manual
> > installation on
> > each new box. As per what I see the RPMs for our Centos have not
> > been created yet.
> > 
> > We could run 'autogen' + 'configure' but I do not find the way to
> > generate the RPMs.
> > Anyone could share with me the correct paths to do this please ?
>  
> The spec-file found in the pacemaker-github-repo should work straight
> forward
> using mock to build against the repos of your Centos-Version.
> Just check that you are on current corosync, libqb, knet, ...
> Pacemaker 2 seems to build well against the packages coming with
> Centos 7.5.
> Maybe others can comment on how advisable it is running that combo
> though.
> 
> Klaus

Also, there is a convenient target for building RPMs from the spec
file, you can just run "make rpm" (after autogen.sh + configure).

> 
> > Perhaps there are some steps written somewhere and I did not find
> > them out ...
> > 
> > Appreciate your help.
> > 
> > Regards
> > Javier
> > Francisco Javier    Lopez     IT System E
> > ngineer  |  Global IT     O: +34 619 728 249
> >  |  M: +34 619 728 249   | 
> > franciscojavier.lo...@solera.com |  Solera.com  
> >   Audatex Datos, S.A.    |  Avda. de Bruselas, 36, Sali
> > da 16, A‑1 (Diversia)   ,   Alcobendas  ,   
> > Madrid  ,   28108   ,   Spain   
> >  
> > 
> > "This e-mail, any associated files and the information contained in
> > them are confidential and is intended for the addressee(s) only. If
> > you have received this message in error please notify the
> > originator and delete the email immediately. The unauthorised use,
> > disclosure, copying or alteration of this message is strictly
> > forbidden. E-mails to and from the company are monitored for
> > operational reasons and in accordance with lawful business
> > practices. Any opinions expressed are those of the individual and
> > do not necessarily represent the views of the company. The company
> > does not conclude contracts by email and all negotiations are
> > subject to contract. We make every effort to maintain our network
> > free from computer viruses but accept no responsibility for any
> > viruses which might be transferred by this e-mail."
> > 
> > 
> > ___
> > Users mailing list: Users@clusterlabs.org
> > https://lists.clusterlabs.org/mailman/listinfo/users
> > 
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratc
> > h.pdf
> > Bugs: http://bugs.clusterlabs.org
>  
> ___
> Users mailing list: Users@clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Re: How to generate RPMs for Pacemaker release 2.x on Centos

[Ken Gaillot]

On Mon, 2018-10-15 at 14:37 +, Lopez, Francisco Javier [Global IT]
wrote:
> Klaus/Ken.
> 
> Thx. for you reply.
> 
> The issue is ...
> 
> - I downloaded the source from GIT.
> - Downloaded the OS required packages.
> - Unzipped the source.
> - ./autogen.sh + ./configure ---> OK
> - Then, indeed, I tried: make rpm 
>   But I got thousands of errors:

Ah, I forgot it uses information from the repository. Rather than
download the source, you'd have to git clone the repository, and run
from there. By default you'll be in the latest master branch; if you
prefer to run a released version, you can check it out like "git
checkout Pacemaker-2.0.0".

> $ make rpm
> fatal: Not a git repository (or any of the parent directories): .git
> fatal: Not a git repository (or any of the parent directories): .git
> fatal: Not a git repository (or any of the parent directories): .git
> fatal: Not a git repository (or any of the parent directories): .git
> fatal: Not a git repository (or any of the parent directories): .git
> fatal: Not a git repository (or any of the parent directories): .git
> /bin/sh: -c: line 0: syntax error near unexpected token `Pacemaker-*'
> /bin/sh: -c: line 0: `case  in Pacemaker-*) echo '' | cut -c11-;; *)
> git log --pretty=format:%h -n 1 '';; esac'
> ...
> ...
> 
> What made me think, that downloading the source to the box I'm
> testing could not be the
> best, so I decided to ask the experts.
> 
> Best Regards
> 
> Francisco Javier      Lopez     IT System Eng
> ineer  |  Global IT     O: +34 619 728 249
>  |    M: +34 619 728 249   | 
> franciscojavier.lo...@solera.com   |  Solera.com  
>   Audatex Datos, S.A.  |  Avda. de Bruselas, 36, Salida
>  16, A‑1 (Diversia)   ,   Alcobendas  ,   Madr
> id,   28108   ,   Spain     On
> 15/10/18 16:27, Ken Gaillot wrote:
> > On Mon, 2018-10-15 at 14:39 +0200, Klaus Wenninger wrote:
> > > On 10/15/2018 01:52 PM, Lopez, Francisco Javier [Global IT]
> > > wrote:
> > > > Hello guys !
> > > > 
> > > > We are planning to use Pacemaker as a base HA Software in our
> > > > Company.
> > > > 
> > > > Our requirements will be:
> > > > 
> > > > - Centos-7
> > > > - PostgreSql-10
> > > > 
> > > > We did several tests with Pacemaker release 1.1.8 and fixed the
> > > > problems found with
> > > > the RA. We finally created new RPMs from source (4.x).
> > > > 
> > > > Now we want to test Pacemaker release 2.x but, as we will have
> > > > to
> > > > create some clusters,
> > > > we want to create new RPMs for this release instead of doing
> > > > manual
> > > > installation on
> > > > each new box. As per what I see the RPMs for our Centos have
> > > > not
> > > > been created yet.
> > > > 
> > > > We could run 'autogen' + 'configure' but I do not find the way
> > > > to
> > > > generate the RPMs.
> > > > Anyone could share with me the correct paths to do this please
> > > > ?
> > > 
> > >  
> > > The spec-file found in the pacemaker-github-repo should work
> > > straight
> > > forward
> > > using mock to build against the repos of your Centos-Version.
> > > Just check that you are on current corosync, libqb, knet, ...
> > > Pacemaker 2 seems to build well against the packages coming with
> > > Centos 7.5.
> > > Maybe others can comment on how advisable it is running that
> > > combo
> > > though.
> > > 
> > > Klaus
> > 
> > Also, there is a convenient target for building RPMs from the spec
> > file, you can just run "make rpm" (after autogen.sh + configure).
> > 
> > > > Perhaps there are some steps written somewhere and I did not
> > > > find
> > > > them out ...
> > > > 
> > > > Appreciate your help.
> > > > 
> > > > Regards
> > > > Javier
> > > > Francisco Javier    Lopez     IT Syst
> > > > em E
> > > > ngineer  |  Global IT     O: +34 619 728 249
> > > > 
> > > >  |  M: +34 619 728 249   | 
> > > > franciscojavier.lo...@solera.com |  Solera.com  
> > > >   Audatex Datos, S.A.    |  Avda. de Bruselas, 36, 
> > > > Sali
> > > > da 16, A‑1 (Diversia)   ,   Alcobend

Re: [ClusterLabs] Re: How to generate RPMs for Pacemaker release 2.x on Centos

[Ken Gaillot]

On Wed, 2018-10-17 at 15:58 +, Lopez, Francisco Javier [Global IT]
wrote:
> Hello guys.
> 
> I finally created the RPMs for Pacemaker and Resource Agents. I will
> paste
> in this thread the way to do that so it can help any other like me :-
> )
> 
> I need a final update, hopefully, from you guys about this issue ...
> 
> I need to know if there is any compatibility or certification matrix
> somewhere.
> I'm asking this because as I'm creating the packages from source, I
> need to be
> sure that Pacemaker, PCS, Corosync, the Agents, ... releases match.
> This would
> guarantee that if we find an issue using the product, the problem is
> not a 
> compatibility among them.

For the most part, each component is written to be independent of other
components' versions.

The big exception is pcs, which is closely tied to both corosync and
pacemaker versions.

The pcs 0.9 series is compatible with corosync 2 and pacemaker 1.1,
while the pcs 0.10 series is compatible with the new kronosnet project,
corosync 3, and pacemaker 2.0. pcs may work "most of the time" with
other versions, but full compatibility requires that line-up.

I'd say your choices from easiest to hardest are:

* Stick with what you have. If a pcs command works, you're fine. If a
pcs command fails, figure out how to do the same thing with direct
editing of corosync.conf and the pacemaker CLI tools.

* Switch to the pacemaker 1.1 series (either distro-provided or your
own build), and keep everything else as you have it. You don't lose
much, since most bug fixes and some features from 2.0 have been
backported to 1.1. This will likely continue for at least the next year
or two, though at some point the frequency of backports will decline.
This would buy you enough time for the distros to catch up to the newer
versions of everything.

* Build kronosnet, corosync 3, and pcs 0.10 yourself. This would put
you on the "way of the future" but obviously is more maintenance for
you when it comes to updates, and you lose the early release of
backported bugfixes that distro packages include.

> 
> These are the releases I have now:
> 
> - Pacemaker: 2.0.0, created by me.
> - Resource Agents: 4.1.1, created by me.
> - Corosync: corosynclib-2.4.3-2.el7_5.1.x86_64
>     corosync-2.4.3-2.el7_5.1.x86_64
>   Installed from repos.
> - PCS: pcs-0.9.162-5.el7.centos.1.x86_64
>   Installed from repos.
> 
> As before, I appreciate all your help
> 
> Best regards
> 
> 
> Francisco Javier      Lopez     IT System Eng
> ineer  |  Global IT     O: +34 619 728 249
>  |    M: +34 619 728 249   | 
> franciscojavier.lo...@solera.com   |  Solera.com  
>   Audatex Datos, S.A.  |  Avda. de Bruselas, 36, Salida
>  16, A‑1 (Diversia)   ,   Alcobendas  ,   Madr
> id,   28108   ,   Spain     On
> 15/10/18 16:42, Ken Gaillot wrote:
> > On Mon, 2018-10-15 at 14:37 +, Lopez, Francisco Javier [Global
> > IT]
> > wrote:
> > > Klaus/Ken.
> > > 
> > > Thx. for you reply.
> > > 
> > > The issue is ...
> > > 
> > > - I downloaded the source from GIT.
> > > - Downloaded the OS required packages.
> > > - Unzipped the source.
> > > - ./autogen.sh + ./configure ---> OK
> > > - Then, indeed, I tried: make rpm 
> > >   But I got thousands of errors:
> > 
> > Ah, I forgot it uses information from the repository. Rather than
> > download the source, you'd have to git clone the repository, and
> > run
> > from there. By default you'll be in the latest master branch; if
> > you
> > prefer to run a released version, you can check it out like "git
> > checkout Pacemaker-2.0.0".
> > 
> > > $ make rpm
> > > fatal: Not a git repository (or any of the parent directories):
> > > .git
> > > fatal: Not a git repository (or any of the parent directories):
> > > .git
> > > fatal: Not a git repository (or any of the parent directories):
> > > .git
> > > fatal: Not a git repository (or any of the parent directories):
> > > .git
> > > fatal: Not a git repository (or any of the parent directories):
> > > .git
> > > fatal: Not a git repository (or any of the parent directories):
> > > .git
> > > /bin/sh: -c: line 0: syntax error near unexpected token
> > > `Pacemaker-*'
> > > /bin/sh: -c: line 0: `case  in Pacemaker-*) echo '' | cut -c11-;; 
> > > *)
> > > git log --pretty=format:%h -n 1 '';; esac'
> > > ...
> > > ...
> > > 
> > > What made me think, that downloading the source to the box I'm

Re: [ClusterLabs] New cluster.target to control cluster services

[Ken Gaillot]

On Mon, 2018-10-22 at 18:31 +0200, Michal Koutný wrote:
> Hello,
> I'd like to collect some feedback regarding $SUBJECT.
> 
> The current state of the main cluster services and their dependencies
> condenses into the following administrative commands:
> 
> Enable cluster services:
> systemctl enable \
>   pacemaker [sbd] [dlm]
> 
> Starting cluster:
> systemctl start pacemaker
> 
> Stopping cluster (all cluster services):
> systemctl stop corosync
> 
> It's not a great drawback but the asymmetric control is rather
> counter-intuitive and IMO it unnecessarily leaks the internal
> dependencies to the administrator.
> 
> I was thus thinking about the change below (I included only
> requirement
> dependencies, not orderings):
> 
>  # pacemaker.service
>  [Unit]
>  Requires=corosync.service
> +PartOf=cluster.target
> 
>  [Install]
> -WantedBy=multi-user.target
> 
> 
> # corosync.service
>  [Unit]
> +PartOf=cluster.target
>  [Install]
>  WantedBy=multi-user.target
> 
> 
> # sbd.service
>  [Unit]
> -PartOf=corosync.service
> +PartOf=cluster.target
>  RefuseManualStop=true
>  RefuseManualStart=true
>  [Install]
>  RequiredBy=corosync.service
>  RequiredBy=pacemaker.service
>  RequiredBy=dlm.service
> 
> 
> # dlm.service
>  [Unit]
>  Requires=corosync.service
>  [Install]
> -WantedBy=multi-user.target
> +WantedBy=cluster.target
> 
> 
> # cluster.target
> +[Unit]
> +Wants=pacemaker.service
> +[Install]
> +WantedBy=multi-user.target
> 
> The resulting control interface would be then more homogenous:
> 
> Enable cluster service:
> systemctl enable \
>   cluster.target [sbd] [dlm]
> 
> Starting cluster:
> systemctl start cluster.target
> 
> Stopping cluster:
> systemctl stop cluster.target
> 
> 
> I'm aware that there are crms/pcs commands providing the similar
> common
> control interface.
> 
> Do you have any use cases that would benefit from the cluster.target?
> What drawbacks do you see if the common cluster.target was used?
> 
> Thanks,
> Michal

It makes sense to me. Of course crm/pcs would need changes to work with
it, and it might be better to use a more specific name than "cluster"
(pacemaker-cluster? clusterlabs-ha? high-availability?).

The only drawback I see is that it's theoretically possible to deploy a
different membership layer than corosync (in the past others were
supported, and that may happen again in the future), and possible to
run corosync without pacemaker (typically for use by some custom app
with its own HA). It would be difficult to accommodate all of that, but
then again, it might not be important to do so.
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Fwd: Not getting Fencing monitor alerts

[Ken Gaillot]

_fence_name with
> > > master unicloud-master INFINITY
> > >         pcs constraint order start $peer_fence_name then promote
> > > unicloud-master
> > >         pcs stonith update $peer_fence_name meta failure-
> > > timeout=3s
> > >     fi
> > >                                                                  
> > >                                                                  
> > >                                                             
> > >     pcs property set stonith-enabled=true
> > > 
> > > 
> > > Thanks,
> > > Rohit
> > > 
> > > 
> > > ___
> > > Users mailing list: Users@clusterlabs.org
> > > https://lists.clusterlabs.org/mailman/listinfo/users
> > > 
> > > Project Home: http://www.clusterlabs.org
> > > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scra
> > > tch.pdf
> > > Bugs: http://bugs.clusterlabs.org
> >  
> > 
> 
> ___
> Users mailing list: Users@clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] About the Pacemaker

[Ken Gaillot]

On Tue, 2018-10-23 at 21:20 +0800, T. Ladd Omar wrote:
> For the question one, I don't thinkstart-failure-is-fatal is good
> way for me. It barely has no interval for retrying and easily leads
> to flooding log output in a short time.
> 
> T. Ladd Omar  于2018年10月23日周二 下午9:06写道：
> > Hi all, I send this message to get some answers for my questions
> > about Pacemaker.
> > 1. In order to cleanup start-failed resources automatically, I add
> > failure-timeout attribute for resources, however, the common way to
> > trigger the recovery is by cluster-recheck whose interval is 15min
> > by default. I wonder how lower value could I set for the cluster-
> > recheck-interval. I had to let the failed resources recover
> > somewhat quickly while little impact taken by the more frequent
> > cluster-recheck.
> > Or, is there another way to automatically cleanup start-failed
> > resources ?

failure-timeout with a lower cluster-recheck-interval is fine. I don't
think there's ever been solid testing on what a lower bound for the
interval is. I've seen users set it as low as 1 minute, but that seems
low to me. My gut feeling is 5 minutes is a good trade-off. The simpler
your cluster is (# nodes / # resources / features used), the lower the
number could be.

> > 2. Is Pacemaker suitable for the Master-Slave model HA ? I had some
> > productive problems when I use Pacemaker. If only one resource
> > stopped on one node, should I failover all this node for the whole
> > cluster? If not, the transactions from the ports on this node may
> > fail for this failure. If yes, it seems to be big action for just
> > one resource failure.

Definitely, master/slave operation is one of the most commonly used
Pacemaker features. You have the flexibility of failing over any
combination of resources you want. Look into clone resources,
master/slave clones, colocation constraints, and the on-fail property
of operations.
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Floating IP active in both nodes

[Ken Gaillot]

 not
> > have it.
> > 
> > > meta target-role=Started is-managed=true
> > > location cli-prefer-site_one_ip site_one_ip role=Started inf:
> > 908soffid01
> > > location site_one_ip_pref site_one_ip 100: 908soffid01
> > > property cib-bootstrap-options: \
> > > have-watchdog=false \
> > > dc-version=1.1.14-70404b0 \
> > > cluster-infrastructure=corosync \
> > > cluster-name=debian \
> > > stonith-enabled=false \
> > > no-quorum-policy=ignore \
> > > maintenance-mode=false
> > > 
> > > Apparently, it works fine, and floating IP address is active in
> > node1:
> > > root@908soffid02:~# crm_mon -1
> > > Last updated: Fri Oct 26 10:06:12 2018 Last change: Fri Oct 26
> > 10:02:53
> > > 2018 by root via cibadmin on 908soffid02
> > > Stack: corosync
> > > Current DC: 908soffid01 (version 1.1.14-70404b0) - partition with
> > quorum
> > > 2 nodes and 1 resource configured
> > > 
> > > Online: [ 908soffid01 908soffid02 ]
> > > 
> > >  site_one_ip (ocf::heartbeat:IPaddr): Started 908soffid01
> > > 
> > > But when node2 tries to connect to the floating IP address, it
> > gets
> > > connected to itself, despite the IP address is bound to the first
> > node:
> > > root@908soffid02:~# ssh root@10.6.12.118 hostname
> > > root@soffiddb's password:
> > > 908soffid02
> > > 
> > > I'd want the second node connects to the actual floating IP
> > address, but I
> > > cannot see how to set it up. Any help is welcome.
> > > 
> > > I am using pacemaker 1.1.14-2ubuntu1.4 and corosync 2.3.5-
> > 3ubuntu2.1
> > > 
> > > Kind regards.
> > > 
> > > 
> > > 
> > > Gabriel Buades
> > > 
> > > 
> > > ___
> > > Users mailing list: Users@clusterlabs.org
> > > https://lists.clusterlabs.org/mailman/listinfo/users
> > > 
> > > Project Home: http://www.clusterlabs.org
> > > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scra
> > tch.pdf
> > > Bugs: http://bugs.clusterlabs.org
> > > 
> > 
> > ___
> > Users mailing list: Users@clusterlabs.org
> > https://lists.clusterlabs.org/mailman/listinfo/users
> > 
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratc
> > h.pdf
> > Bugs: http://bugs.clusterlabs.org__
> > _
> > Users mailing list: Users@clusterlabs.org
> > https://lists.clusterlabs.org/mailman/listinfo/users
> > 
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratc
> > h.pdf
> > Bugs: http://bugs.clusterlabs.org
> > 
> > 
> > 
> > ___
> > Users mailing list: Users@clusterlabs.org
> > https://lists.clusterlabs.org/mailman/listinfo/users
> > 
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratc
> > h.pdf
> > Bugs: http://bugs.clusterlabs.org
> > 
> 
> ___
> Users mailing list: Users@clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] Coming in 2.0.1 / 1.1.20: sbd compatibility with guest nodes and bundles

[Ken Gaillot]

Believe it or not, we're not terribly far away from starting the next
release cycle -- hopefully before the end of the year. So, it's time to
start rounding up the significant changes. :)

One helpful new feature is the ability to run guest nodes and bundles
in clusters that use sbd for fencing.

Previously, the cluster would require that sbd be configured inside the
guest or container, which is not practical or desirable, since the
cluster "fences" them by restarting their resource.

Now, the cluster will skip the sbd requirements for guest nodes and
bundles, allowing them to work properly (with no loss of safety).
-- 
Ken Gaillot 
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org