Re: [ClusterLabs] Recovering after split-brain

[Ken Gaillot]

On 06/20/2016 08:30 AM, Nikhil Utane wrote:
> Hi,
> 
> For our solution we are making a conscious choice to not use
> quorum/fencing as for us service availability is more important than
> having 2 nodes take up the same active role. Split-brain is not an issue
> for us (at least i think that way) since we have a second line of
> defense. We have clients who can connect to only one of the two active
> nodes. So in that sense, even if we end up with 2 nodes becoming active,
> since the clients can connect to only 1 of the active node, we should
> not have any issue.
> 
> Now my question is what happens after recovering from split-brain since
> the resource will be active on both the nodes. From application point of
> view we want to be able to find out which node is servicing the clients
> and keep that operational and make the other one as standby.
> 
> Does Pacemaker make it easy to do this kind of thing through some means?
> Are there any issues that I am completely unaware due to letting
> split-brain occur?
> 
> -Thanks
> Nikhil

Usually, split brain is most destructive when the two nodes need to
synchronize data in some way (DRBD, shared storage, cluster file
systems, replication, etc.). If both nodes attempt to write without
being able to coordinate with each other, it usually results in
incompatible data stores that cause big recovery headaches (sometimes
throw-it-away-and-restore-from-backup headaches). For a resource such as
a floating IP, the consequences are less severe, but it can result in
the service becoming unusable (if both nodes claim the IP, packets go
every which way).

In the scenario you describe, if a split brain occurs and then is
resolved, Pacemaker will likely stop the services on both nodes, then
start them on one node.

The main questions I see are (1) does your service require any sort of
coordination/synchronization between the two nodes, especially of data;
and (2) how do clients know which node to connect to?

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Cluster reboot fro maintenance

[Ken Gaillot]

On 06/20/2016 07:45 AM, ma...@nucleus.it wrote:
> Hi,
> i have a two node cluster with some vms (pacemaker resources) running on
> the two hypervisors:
> pacemaker-1.0.10
> corosync-1.3.0
> 
> I need to do maintenance stuff , so i need to:
> - put on maintenance the cluster so the cluster doesn't
>   touch/start/stop/monitor the vms
> - update the vms
> - stop the vms
> - stop cluster stuff (corosync/pacemaker) so it do not
>   start/stop/monitor vms
> - reboot the hypervisors.
> - start cluster stuff
> - remove maintenance from the cluster stuff so it start all the vms
> 
> What is the corret way to do that ( corosync/pacemaker) side ?
> 
> 
> Best regards
> Marco

Maintenance mode provides this ability. Set the maintenance-mode cluster
proprerty to true, do whatever you want, then set it back to false when
done.

That said, I've never used pacemaker/corosync versions that old, so I'm
not 100% sure that applies to those versions, though I would guess it does.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] crm_resource --cleanup and cluster-recheck-interval

[Ken Gaillot]

On 06/15/2016 05:44 AM, Vladislav Bogdanov wrote:
> Hi,
> 
> It seems that after recent commit which introduces staggered probes
> running 'crm_resource --cleanup' (without --resource) leads to cluster
> to finish recheck too long after cleanup was done. What I see: cluster
> fires probes for the first batch of resources, receives their status,
> writes it to CIB, and then sleep until cluster-recheck-interval lapses.
> After that next batch is reprobed and so on.
> 
> This looks like a regression to me.
> 
> Should I file bug for that or that can be fixed quite fast? IMHO this is
> major enough to be fixed in 1.1.15.
> 
> Best,
> Vladislav

Hi,

Please file a bug, and attach any relevant logs. Obviously, we weren't
able to handle this in time for 1.1.15, but if it's related to staggered
probes, that was introduced in 1.1.14.


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Help required for N+1 redundancy setup

[Ken Gaillot]

On 01/08/2016 06:55 AM, Nikhil Utane wrote:
> Would like to validate my final config.
> 
> As I mentioned earlier, I will be having (upto) 5 active servers and 1
> standby server.
> The standby server should take up the role of active that went down. Each
> active has some unique configuration that needs to be preserved.
> 
> 1) So I will create total 5 groups. Each group has a "heartbeat::IPaddr2
> resource (for virtual IP) and my custom resource.
> 2) The virtual IP needs to be read inside my custom OCF agent, so I will
> make use of attribute reference and point to the value of IPaddr2 inside my
> custom resource to avoid duplication.
> 3) I will then configure location constraint to run the group resource on 5
> active nodes with higher score and lesser score on standby.
> For e.g.
> Group  NodeScore
> -
> MyGroup1node1   500
> MyGroup1node6   0
> 
> MyGroup2node2   500
> MyGroup2node6   0
> ..
> MyGroup5node5   500
> MyGroup5node6   0
> 
> 4) Now if say node1 were to go down, then stop action on node1 will first
> get called. Haven't decided if I need to do anything specific here.

To clarify, if node1 goes down intentionally (e.g. standby or stop),
then all resources on it will be stopped first. But if node1 becomes
unavailable (e.g. crash or communication outage), it will get fenced.

> 5) But when the start action of node 6 gets called, then using crm command
> line interface, I will modify the above config to swap node 1 and node 6.
> i.e.
> MyGroup1node6   500
> MyGroup1node1   0
> 
> MyGroup2node2   500
> MyGroup2node1   0
> 
> 6) To do the above, I need the newly active and newly standby node names to
> be passed to my start action. What's the best way to get this information
> inside my OCF agent?

Modifying the configuration from within an agent is dangerous -- too
much potential for feedback loops between pacemaker and the agent.

I think stickiness will do what you want here. Set a stickiness higher
than the original node's preference, and the resource will want to stay
where it is.

> 7) Apart from node name, there will be other information which I plan to
> pass by making use of node attributes. What's the best way to get this
> information inside my OCF agent? Use crm command to query?

Any of the command-line interfaces for doing so should be fine, but I'd
recommend using one of the lower-level tools (crm_attribute or
attrd_updater) so you don't have a dependency on a higher-level tool
that may not always be installed.

> Thank You.
> 
> On Tue, Dec 22, 2015 at 9:44 PM, Nikhil Utane <nikhil.subscri...@gmail.com>
> wrote:
> 
>> Thanks to you Ken for giving all the pointers.
>> Yes, I can use service start/stop which should be a lot simpler. Thanks
>> again. :)
>>
>> On Tue, Dec 22, 2015 at 9:29 PM, Ken Gaillot <kgail...@redhat.com> wrote:
>>
>>> On 12/22/2015 12:17 AM, Nikhil Utane wrote:
>>>> I have prepared a write-up explaining my requirements and current
>>> solution
>>>> that I am proposing based on my understanding so far.
>>>> Kindly let me know if what I am proposing is good or there is a better
>>> way
>>>> to achieve the same.
>>>>
>>>>
>>> https://drive.google.com/file/d/0B0zPvL-Tp-JSTEJpcUFTanhsNzQ/view?usp=sharing
>>>>
>>>> Let me know if you face any issue in accessing the above link. Thanks.
>>>
>>> This looks great. Very well thought-out.
>>>
>>> One comment:
>>>
>>> "8. In the event of any failover, the standby node will get notified
>>> through an event and it will execute a script that will read the
>>> configuration specific to the node that went down (again using
>>> crm_attribute) and become active."
>>>
>>> It may not be necessary to use the notifications for this. Pacemaker
>>> will call your resource agent with the "start" action on the standby
>>> node, after ensuring it is stopped on the previous node. Hopefully the
>>> resource agent's start action has (or can have, with configuration
>>> options) all the information you need.
>>>
>>> If you do end up needing notifications, be aware that the feature will
>>> be disabled by default in the 1.1.14 release, because changes in syntax
>>> are expected in further development. You can define a compile-time
>>> constant to enable them.
>>>
>>>
> 


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Parallel adding of resources

[Ken Gaillot]

On 01/08/2016 12:34 AM, Arjun Pandey wrote:
> Hi
> 
> I am running a 2 node cluster with this config on centos 6.6
> 
> Master/Slave Set: foo-master [foo]
> Masters: [ messi ]
> Stopped: [ronaldo ]
>  eth1-CP(ocf::pw:IPaddr):   Started messi
>  eth2-UP(ocf::pw:IPaddr):   Started messi
>  eth3-UPCP  (ocf::pw:IPaddr):   Started messi
> 
> where i have a multi-state resource foo being run in master/slave mode
> and  IPaddr RA is just modified IPAddr2 RA. Additionally i have a
> collocation constraint for the IP addr to be collocated with the master.
> 
> Now there are cases where i have multiple virtual IP's ( around 20 )
> and for failover time gets substantially increased in these cases.
> Based on the logs what i have observed is the IPaddr resources are
> moved sequentially. Is this really the case ? Also is it possible to
> specify that they can be added simultaneously, since none of them have
> any sort of corelation with the other ?

While pacemaker will of course initiate the moves one by one, it
shouldn't wait for one to be completed before initiating the next one,
unless you have ordering constraints between them or they are in a group
together.

> If it's sequential what is the reason behind it ?
> 
> 
> Thanks in advance.
> 
> Regards
> Arjun


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Automatic Recover for stonith:external/libvirt

[Ken Gaillot]

On 01/08/2016 08:56 AM, m...@inwx.de wrote:
> Hello List,
> 
> I have here a test environment for checking pacemaker. Sometimes our
> kvm-hosts with libvirt have trouble with responding the stonith/libvirt
> resource, so I like to configure the service to realize as failed after
> three failed monitoring attempts. I was searching for a configuration 
> here:
> 
> 
> http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html/Pacemaker_Explained/index.html
> 
> 
> But I failed after hours.
> 
> That's the configuration line for stonith/libvirt:
> 
> crm configure primitive p_fence_ha3 stonith:external/libvirt  params
> hostlist="ha3" hypervisor_uri="qemu+tls://debian1/system" op monitor
> interval="60"
> 
> Every 60 seconds pacemaker makes something like this:
> 
>  stonith -t external/libvirt hostlist="ha3"
> hypervisor_uri="qemu+tls://debian1/system" -S
>  ok
> 
> To simulate the unavailability of the kvm host I remove the certificate
> in /etc/libvirt/libvirtd.conf and restart libvirtd. After 60 seconds or
> less I can see the error with "crm status". On the kvm host I add
> certificate again to /etc/libvirt/libvirtd.conf and restart libvirt
> again. Although libvirt is again available the stonith-resource did not
> start again.
> 
> I altered the configuration line for stonith/libvirt with following parts:
> 
>  op monitor interval="60" pcmk_status_retries="3"
>  op monitor interval="60" pcmk_monitor_retries="3"
>  op monitor interval="60" start-delay=180
>  meta migration-threshold="200" failure-timeout="120"
> 
> But always with first failed monitor check after 60 or less seconds
> pacemakers did not resume stonith-libvirt after libvirt is again available.

Is there enough time left in the timeout for the cluster to retry? (The
interval is not the same as the timeout.) Check your pacemaker.log for
messages like "Attempted to execute agent ... the maximum number of
times (...) allowed". That will tell you whether it is retrying.

You definitely don't want start-delay, and migration-threshold doesn't
really mean much for fence devices.

Of course, you also want to fix the underlying problem of libvirt not
being responsive. That doesn't sound like something that should
routinely happen.

BTW I haven't used stonith/external agents (which rely on the
cluster-glue package) myself. I use the fence_virtd daemon on the host
with fence_xvm as the configured fence agent.

> Here is the "crm status"-output on debian 8 (Jessie):
> 
>  root@ha4:~# crm status
>  Last updated: Tue Jan  5 10:04:18 2016
>  Last change: Mon Jan  4 18:18:12 2016
>  Stack: corosync
>  Current DC: ha3 (167772400) - partition with quorum
>  Version: 1.1.12-561c4cf
>  2 Nodes configured
>  2 Resources configured
>  Online: [ ha3 ha4 ]
>  Service-IP (ocf::heartbeat:IPaddr2):   Started ha3
>  haproxy(lsb:haproxy):  Started ha3
>  p_fence_ha3(stonith:external/libvirt): Started ha4
> 
> Kind regards
> 
> Michael R.


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: Antw: Re: Early VM resource migration

[Ken Gaillot]

On 01/08/2016 07:03 AM, Klechomir wrote:
> Here is what pacemaker says right after node1 comes back after standby:
> 
> Dec 16 16:11:41 [4512] CLUSTER-2pengine:debug:
> native_assign_node: All nodes for resource VM_VM1 are unavailable,
> unclean or shutting down (CLUSTER-1: 1, -100)
> 
> Dec 16 16:11:41 [4512] CLUSTER-2pengine:debug:
> native_assign_node:  Could not allocate a node for VM_VM1
> 
> Dec 16 16:11:41 [4512] CLUSTER-2pengine:debug:
> native_assign_node:  Processing VM_VM1_monitor_1
> 
> Dec 16 16:11:41 [4512] CLUSTER-2pengine: info: native_color:
> Resource VM_VM1 cannot run anywhere
> 
> 
> 
> VM_VM1 gets immediately stopped as soon as node1 re-appears and stays
> down until its "order/colocation AA resource" comes up on node1.
> 
> The curious part is that in the opposite case (node2 comes from
> standby), the failback is ok.
> 
> Any ideas?

This might be a bug. Can you open a report at
http://bugs.clusterlabs.org/ and attach your full CIB and logs from all
nodes both when the issue occurs and when node2 handles it correctly?

> Regards,
> 
> On 17.12.2015 14:51:21 Ulrich Windl wrote:
>> >>> Klechomir <kle...@gmail.com> schrieb am 17.12.2015 um 14:16 in
>> Nachricht
>>
>> <2102747.TPh6pTdk8c@bobo>:
>> > Hi Ulrich,
>> > This is only a part of the config, which concerns the problem.
>> > Even with dummy resources, the behaviour will be identical, so don't
>> think
>> > that dlm/clvmd res. config will help solving the problem.
>>
>> You could send logs with the actual startup sequence then.
>>
>> > Regards,
>> > KIecho
>> >
>> > On 17.12.2015 08:19:43 Ulrich Windl wrote:
>> >> >>> Klechomir <kle...@gmail.com> schrieb am 16.12.2015 um 17:30 in
>> >> >>> Nachricht
>> >>
>> >> <5671918e.40...@gmail.com>:
>> >> > On 16.12.2015 17:52, Ken Gaillot wrote:
>> >> >> On 12/16/2015 02:09 AM, Klechomir wrote:
>> >> >>> Hi list,
>> >> >>> I have a cluster with VM resources on a cloned active-active
>> storage.
>> >> >>>
>> >> >>> VirtualDomain resource migrates properly during failover (node
>> >> >>> standby),
>> >> >>> but tries to migrate back too early, during failback, ignoring the
>> >> >>> "order" constraint, telling it to start when the cloned storage is
>> >> >>> up.
>> >> >>> This causes unnecessary VM restart.
>> >> >>>
>> >> >>> Is there any way to make it wait, until its storage resource is
>> up?
>> >> >>
>> >> >> Hi Klecho,
>> >> >>
>> >> >> If you have an order constraint, the cluster will not try to
>> start the
>> >> >> VM until the storage resource agent returns success for its
>> start. If
>> >> >> the storage isn't fully up at that point, then the agent is faulty,
>> >> >> and
>> >> >> should be modified to wait until the storage is truly available
>> before
>> >> >> returning success.
>> >> >>
>> >> >> If you post all your constraints, I can look for anything that
>> might
>> >> >> affect the behavior.
>> >> >
>> >> > Thanks for the reply, Ken
>> >> >
>> >> > Seems to me that that the constraints for a cloned resources act a a
>> >> > bit
>> >> > different.
>> >> >
>> >> > Here is my config:
>> >> >
>> >> > primitive p_AA_Filesystem_CDrive1 ocf:heartbeat:Filesystem \
>> >> >
>> >> >  params device="/dev/CSD_CDrive1/AA_CDrive1"
>> >> >
>> >> > directory="/volumes/AA_CDrive1" fstype="ocfs2" options="rw,noatime"
>> >> > primitive VM_VM1 ocf:heartbeat:VirtualDomain \
>> >> >
>> >> >  params config="/volumes/AA_CDrive1/VM_VM1/VM1.xml"
>> >> >
>> >> > hypervisor="qemu:///system" migration_transport="tcp" \
>> >> >
>> >> >  meta allow-migrate="true" target-role="Started"
>> >> >
>> >> > clone AA_Filesystem_CDrive1 p_AA_Filesystem_CDrive1 \
>> >> >
>> >> >  meta interleave="true" resource-stickiness="0"
>> >> >
>> >> > target-role="Started"
>> >> > order VM_VM1_after_AA_Filesystem_CDrive1 inf: AA_Filesystem_CDrive1
>> >> > VM_VM1
>> >> >
>> >> > Every time when a node comes back from standby, the VM tries to live
>> >> > migrate to it long before the filesystem is up.
>> >>
>> >> Hi!
>> >>
>> >> To me your config looks rather incomplete: What about DLM, O2CB, cLVM,
>> >> etc.?>>


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] All IP resources deleted once a fenced node rejoins

[Ken Gaillot]

On 01/15/2016 11:08 AM, Ken Gaillot wrote:

>> Jan 13 19:33:00 [4291] oranacib: info:
>> cib_process_replace: Replacement 0.4.0 from kamet not applied to
>> 0.74.1: current epoch is greater than the replacement
>> Jan 13 19:33:00 [4291] oranacib:  warning:
>> cib_process_request: Completed cib_replace operation for section
>> 'all': Update was older than existing configuration (rc=-205,
>> origin=kamet/cibadmin/2, version=0.74.1)

I misread. Looking at it again, the above means that the old
configuration was indeed rejected for section "all". However:

>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op:
>> Diff: --- 0.74.1 2
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op:
>> Diff: +++ 0.75.0 (null)
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
>> /cib/configuration/nodes/node[@id='kamet']
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
>> /cib/configuration/nodes/node[@id='orana']
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
>> /cib/configuration/resources/primitive[@id='fence-uc-orana']/meta_attributes[@id='fence-uc-orana-meta_attributes']
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
>> /cib/configuration/resources/primitive[@id='fence-uc-kamet']
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
>> /cib/configuration/resources/primitive[@id='C-3']
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
>> /cib/configuration/resources/primitive[@id='C-FLT']
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
>> /cib/configuration/resources/primitive[@id='C-FLT2']
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
>> /cib/configuration/resources/primitive[@id='E-3']
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
>> /cib/configuration/resources/primitive[@id='MGMT-FLT']
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
>> /cib/configuration/resources/primitive[@id='M-FLT']
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
>> /cib/configuration/resources/primitive[@id='M-FLT2']
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
>> /cib/configuration/resources/primitive[@id='S-FLT']
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
>> /cib/configuration/resources/primitive[@id='S-FLT2']
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_colocation[@id='colocation-C-3-foo-master-INFINITY']
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_order[@id='order-C-3-foo-master-mandatory']
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_colocation[@id='colocation-C-FLT-foo-master-INFINITY']
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_order[@id='order-C-FLT-foo-master-mandatory']
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_colocation[@id='colocation-C-FLT2-foo-master-INFINITY']
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_order[@id='order-C-FLT2-foo-master-mandatory']
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_colocation[@id='colocation-E-3-foo-master-INFINITY']
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_order[@id='order-E-3-foo-master-mandatory']
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_colocation[@id='colocation-MGMT-FLT-foo-master-INFINITY']
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_order[@id='order-MGMT-FLT-foo-master-mandatory']
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_colocation[@id='colocation-M-FLT-foo-master-INFINITY']
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_order[@id='order-M-FLT-foo-master-mandatory']
>> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_colocation[@id='colocation-M-FLT2-foo-master-INFINITY']
>> Jan 13 

Re: [ClusterLabs] All IP resources deleted once a fenced node rejoins

[Ken Gaillot]

On 01/15/2016 05:02 AM, Arjun Pandey wrote:
> Based  on corosync logs from orana ( The node that did the actual
> fencing  and is the current master node)
> 
> I also tried looking at pengine outputs based on crm_simulate. Uptil
> the fenced node rejoins things look good.
> 
> [root@ucc1 orana]# crm_simulate -S --xml-file
> ./pengine/pe-input-1450.bz2  -u kamet
> Current cluster status:
> Node kamet: pending
> Online: [ orana ]

Above, "pending" means that the node has started to join the cluster,
but has not yet fully joined.


> Jan 13 19:32:53 [4295] oranapengine: info: probe_resources:
> Action probe_complete-kamet on kamet is unrunnable (pending)

Any action on kamet is unrunnable until it finishes joining the cluster.


> Jan 13 19:32:59 [4292] orana stonith-ng: info:
> crm_update_peer_proc: pcmk_cpg_membership: Node kamet[2] -
> corosync-cpg is now online

The pacemaker daemons on orana each report when they see kamet come up
at the corosync level. Here, stonith-ng sees it.


> Jan 13 19:32:59 [4291] oranacib: info:
> crm_update_peer_proc: pcmk_cpg_membership: Node kamet[2] -
> corosync-cpg is now online

Now, the cib sees it.


> Jan 13 19:33:00 [4296] orana   crmd: info:
> crm_update_peer_proc: pcmk_cpg_membership: Node kamet[2] -
> corosync-cpg is now online

Now, crmd sees it.


 [Arjun] Why does pengine declare that the following monitor actions are 
 now unrunnable ?
> 
> Jan 13 19:33:00 [4295] oranapengine:  warning: custom_action:
> Action foo:0_monitor_0 on kamet is unrunnable (pending)

At this point, pengine still hasn't seen kamet join yet, so actions on
it are still unrunnable.


> Jan 13 19:33:00 [4296] orana   crmd: info: join_make_offer:
> join-2: Sending offer to kamet

Having seen kamet at the corosync level, crmd now offers cluster-level
membership to kamet.


> Jan 13 19:33:00 [4291] oranacib: info:
> cib_process_replace: Replacement 0.4.0 from kamet not applied to
> 0.74.1: current epoch is greater than the replacement
> Jan 13 19:33:00 [4291] oranacib:  warning:
> cib_process_request: Completed cib_replace operation for section
> 'all': Update was older than existing configuration (rc=-205,
> origin=kamet/cibadmin/2, version=0.74.1)
> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op:
> Diff: --- 0.74.1 2
> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op:
> Diff: +++ 0.75.0 (null)
> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
> /cib/configuration/nodes/node[@id='kamet']
> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
> /cib/configuration/nodes/node[@id='orana']
> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
> /cib/configuration/resources/primitive[@id='fence-uc-orana']/meta_attributes[@id='fence-uc-orana-meta_attributes']
> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
> /cib/configuration/resources/primitive[@id='fence-uc-kamet']
> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
> /cib/configuration/resources/primitive[@id='C-3']
> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
> /cib/configuration/resources/primitive[@id='C-FLT']
> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
> /cib/configuration/resources/primitive[@id='C-FLT2']
> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
> /cib/configuration/resources/primitive[@id='E-3']
> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
> /cib/configuration/resources/primitive[@id='MGMT-FLT']
> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
> /cib/configuration/resources/primitive[@id='M-FLT']
> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
> /cib/configuration/resources/primitive[@id='M-FLT2']
> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
> /cib/configuration/resources/primitive[@id='S-FLT']
> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
> /cib/configuration/resources/primitive[@id='S-FLT2']
> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
> /cib/configuration/constraints/rsc_colocation[@id='colocation-C-3-foo-master-INFINITY']
> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
> /cib/configuration/constraints/rsc_order[@id='order-C-3-foo-master-mandatory']
> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
> /cib/configuration/constraints/rsc_colocation[@id='colocation-C-FLT-foo-master-INFINITY']
> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
> /cib/configuration/constraints/rsc_order[@id='order-C-FLT-foo-master-mandatory']
> Jan 13 19:33:00 [4291] oranacib: info: cib_perform_op: --
> /cib/configuration/constraints/rsc_colocation[@id='colocation-C-FLT2-foo-master-INFINITY']
> Jan 13 19:33:00 [4291] oranacib: 

[ClusterLabs] Pacemaker 1.1.14 released

[Ken Gaillot]

ClusterLabs is proud to announce the latest release of the Pacemaker
cluster resource manager, version 1.1.14. The source code is available at:

https://github.com/ClusterLabs/pacemaker/releases/tag/Pacemaker-1.1.14

This version introduces some valuable new features:

* Resources will now start as soon as their state has been confirmed on
all nodes and all dependencies have been satisfied, rather than waiting
for the state of all resources to be confirmed. This allows for faster
startup of some services, and more even startup load.

* Fencing topology levels can now be applied to all nodes whose name
matches a configurable pattern, or that have a configurable node attribute.

* When a fencing topology level has multiple devices, reboots are now
automatically mapped to all-off-then-all-on, allowing much simplified
configuration of redundant power supplies.

* Guest nodes can now be included in groups, which simplifies the common
Pacemaker Remote use case of a grouping a storage device, filesystem and VM.

* Clone resources have a new clone-min metadata option, specifying that
a certain number of instances must be running before any dependent
resources can run. This is particularly useful for services behind a
virtual IP and haproxy, as is often done with OpenStack.

As usual, the release includes many bugfixes and minor enhancements. For
a more detailed list of changes, see the change log:

https://github.com/ClusterLabs/pacemaker/blob/1.1/ChangeLog

Feedback is invited and welcome.
-- 
Ken Gaillot <kgail...@redhat.com>

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Pacemaker and Corosync versions compatibility

[Ken Gaillot]

On 06/24/2016 08:46 PM, Maciej Kopczyński wrote:
> Hello,
> 
> I've been following a tutorial to set up a simple HA cluster using
> Pacemaker and Corosync on CentOS 6.x while I have noticed that in the
> original documentation it is stated that:
> 
> "Since |pcs| has the ability to manage all aspects of the cluster (both
> corosync and pacemaker), it requires a specific cluster stack to be in
> use: corosync 2.0 or later with votequorum plus Pacemaker 1.1.8 or later."

That's incorrect -- pcs has supported RHEL 6 and its derivatives (with
corosync 1 and CMAN) since at least 6.5. Thanks for bringing it up; I'll
delete that note.

> Here are the versions of packages installed in my system (CentOS 6.7):
> pacemaker-1.1.14-8.el6.x86_64
> corosync-1.4.7-5.el6.x86_64
> 
> I did not do that much of testing, but my cluster seems to be more or
> less working so far, what are the compatibility issues then? What will
> not work with corosync in version lower than 2.0?

corosync 2 is a cleaner implementation and has better message delivery
guarantees, but most of that is transparent to the user. With corosync
1, pacemaker will use a "legacy" version of the node attribute daemon,
so the attrd_updater command has fewer options. Most of the other
differences are in implementation, and may be less efficient for
example, but are not readily visible.

> Thanks in advance for your answers.
> 
> MK

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Pacemaker (remote) component relations

[Ken Gaillot]

On 02/08/2016 07:55 AM, Ferenc Wágner wrote:
> Hi,
> 
> I'm looking for information about the component interdependencies,
> because I'd like to split the Pacemaker packages in Debian properly.
> The current idea is to create two daemon packages, pacemaker and
> pacemaker-remote, which exclude each other, as they contain daemons
> listening on the same sockets.
> 
> 1. Is the socket names configurable?  Are there reasonable use cases
>requiring both daemons running concurrently?

No, they are exclusive. Pacemaker Remote simulates the cluster services,
so they listen on the same sockets. They can be installed on the same
machine, just not running at the same time.

> These two daemon packages would depend on a package providing the common
> hacluster user, the haclient group, the sysconfig and logrotate config.
> 
> What else should go here?
> 
> 2. Are the various RNG and XSL files under /usr/share/pacemaker used
>equally by pacemakerd and pacemaker_remoted?  Or by the CLI utils?

Yes, the RNG and XSL files should be considered common.

> 3. Maybe the ocf:pacemaker RAs and their man pages?  Or are they better
>in a separate pacemaker-resource-agents package?

If I were starting from scratch, I would consider a separate package.
Upstream, we recently moved most of the agents to the CLI package, which
is also a good alternative.

Note that a few agents can't run on remote nodes, and we've left those
in the pacemaker package upstream. These are remote (obviously),
controld and o2cb (I believe because the services they start require
direct access to fencing).

> 4. Is /usr/share/snmp/mibs/PCMK-MIB.txt used by any component, or is it
>only for decyphering SNMP traps at their destination?  I guess it can
>go anywhere, it will be copied by the admin anyway.

The ClusterMon resource uses it, and user scripts can use it. I'd vote
for common, as it's architecture-neutral and theoretically usable by
anything.

> There's also a separate package for the various command line utilities,
> which would depend on pacemaker OR pacemaker-remote.
> 
> 5. crm_attribute is in the pacemaker RPM package, unlike the other
>utilites, which are in pacemaker-cli.  What's the reason for this?
> 
> 6. crm_node connects to corosync directly, so it won't work with
>pacemaker_remote.  crm_master uses crm_node, thus it won't either (at
>least without -N).  But 4.3.5. of the Remote book explicitly mentions
>crm_master amongst the tools usable with pacemaker_remote.  Where's
>the mistake?

crm_attribute and crm_node both depend on the cluster-layer libraries,
which won't necessarily be available on a remote node. We hope to remove
that dependency at some point.

It is possible to install the cluster libraries on a remote node, and
some of the crm_attribute/crm_node functionality will work, though not all.

> 7. According to its man page, crm_master should be invoked from an OCF
>resource agent, which could happen under pacemaker_remote.  This is
>again problematic due to the previous point.

I'd have to look into this to be sure, but it's possible that if the
cluster libs are installed, the particular options that crm_master uses
are functional.

> 8. Do fence_legacy, fence_pcmk and stonith_admin make any sense outside
>the pacemaker package?  Even if they can run on top of
>pacemaker_remote, the cluster would never used them there, right?
>And what about attrd_updater?  To be honest, I don't know what that's
>for anyway...

No fence agents run on remote nodes.

I'd expect stonith_admin and attrd_updater to work, as the remote node
will proxy socket connections to stonithd and attrd. attrd_updater is an
alternative interface similar to crm_attribute.

> 9. What's the purpose of chgrp haclient /etc/pacemaker if
>pacemaker_remoted is running as root?  What other haclients may need
>access to the authkey file?

The crmd (which runs as hacluster:haclient) manages the cluster side of
a remote node connection, so on the cluster nodes, the key has to be
readable by that process. In the how-to's, I kept it consistent on the
remote nodes to reduce confusion and the chance for errors.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Working with 2 VIPs

[Ken Gaillot]

On 02/08/2016 04:24 AM, Louis Chanouha wrote:
> Hello,
> I'm not sure if this mailign is the proper place to send ma request, please 
> tell 
> me where i should send it if not :)

This is the right place :)

> I have an use case that i can't run acutally with corosync + pacemaker.
> 
> I have two nodes, two VIP and two services (one dupplicated), in order to 
> provide an active/active service (2 physical sites).

By "2 physical sites", do you mean 2 physical machines on the same LAN,
or 2 geographically separate locations?

> On a normal situation, one VIP is associated to one node via a prefered 
> location, and the service is running one the two nodes (cloned).
> 
> On failing situation, i want that the working node takes the IP of the other 
> host without migrating the service (listening on 0.0.0.0), so when :
>   - the service is down - not working
>   - the node is down (network or OS layer) - working
> 
> I can't find the proper way to conceptualize this problem with 
> group/colocation/order notions of pacemaker. I would be happy in you give me 
> some thoughs on appropriate options.

I believe your current configuration already does that. What problems
are you seeing?

> 
> Thank you in advance for your help.
> Sorry for my non-native English.
> 
> Louis Chanouha
> 
> **
> 
> My current configuration is this one. I can't translate it in XML if you need 
> it.
> 
> /node Gollum//

This will likely cause a log warning that "Node names with capitals are
discouraged". It's one of those things that shouldn't matter, but better
safe than sorry ...

> //node edison//
> //primitive cups lsb:cups \//
> //op monitor interval="2s"//
> //primitive vip_edison ocf:heartbeat:IPaddr2 \//
> //params nic="eth0" ip="10.1.9.18" cidr_netmask="24" \//
> //op monitor interval="2s"//
> //primitive vip_gollum ocf:heartbeat:IPaddr2 \//
> //params nic="eth0" ip="10.1.9.23" cidr_netmask="24" \//
> //op monitor interval="2s"//
> //clone ha-cups cups//
> //location pref_edison vip_edison 50: edison//
> //location pref_gollum vip_gollum 50: Gollum//
> //property $id="cib-bootstrap-options" \//
> //dc-version="1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff" \//
> //cluster-infrastructure="openais" \//
> //expected-quorum-votes="2" \//
> //stonith-enabled="false" \//

Without stonith, the cluster will be unable to recover from certain
types of failures (for example, network failures). If both nodes are up
but can't talk to each other ("split brain"), they will both bring up
both IP addresses.

> //no-quorum-policy="ignore"/

If you can use corosync 2, you can set "two_node: 1" in corosync.conf,
and then you wouldn't need this line.

> -- 
> 
> *Louis Chanouha | Ingénieur Système et Réseaux*
> Service Numérique de l'Université de Toulouse
> *Université Fédérale Toulouse Midi-Pyrénées*
> 15 rue des Lois - BP 61321 - 31013 Toulouse Cedex 6
> Tél. : +33(0)5 61 10 80 45  / poste int. : 18045
> 
> louis.chano...@univ-toulouse.fr
> Facebook 
>  | 
> Twitter  | 
> www.univ-toulouse.fr


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] "After = syslog.service" it is not working?

[Ken Gaillot]

On 01/28/2016 10:27 PM, 飯田 雄介 wrote:
> Hi, Ken
> 
> I have to get the log.
> shutdown the -r now ran to "03:00:22".
> rsyslog also "03:00:22" to receive a TERM appears to have stopped.
> 
> Regards, Yusuke

Hello Yusuke,

I see in your logs that /var/log/messages does not have any messages at
all from pacemaker (not only at shutdown).

Meanwhile, pacemaker.log does have messages, including from shutdown.
The cluster writes directly to this file, without going through rsyslog.

Both logs end at 03:00:22. I'm guessing that systemd waited until
pacemaker exited, then stopped rsyslog. I don't think they're exiting at
the same time (or that rsyslog exited before pacemaker).

Instead, I suspect that either pacemaker is configured not to log via
syslog, or rsyslog is configured not to send pacemaker logs to
/var/log/messages.

Check the value of PCMK_logfacility and PCMK_logpriority in your
/etc/sysconfig/pacemaker. By default, pacemaker will log via syslog, but
if these are explicitly set, they might be turning that off.

Also, pacemaker will try to inherit log options from corosync, so check
/etc/corosync/corosync.conf for logging options, especially to_syslog
and syslog_facility.

If that isn't an issue, then I would check /etc/rsyslog.conf and
/etc/rsyslog.d/* to see if they do anything nonstandard.

>> -Original Message-
>> From: Ken Gaillot [mailto:kgail...@redhat.com]
>> Sent: Friday, January 29, 2016 12:16 AM
>> To: users@clusterlabs.org
>> Subject: Re: [ClusterLabs] "After = syslog.service" it is not working?
>>
>> On 01/28/2016 12:48 AM, 飯田 雄介 wrote:
>>> Hi, All
>>>
>>> I am building a cluster in the following environments.
>>> RHEL7.2
>>> Pacemaker-1.1.14
>>>
>>> The OS while it is running the Pacemaker was allowed to shutdown.
>>> Logs at this time Pacemaker in the stop was not output to the syslog.
>>>
>>> This "After = syslog.service" does not work is set to start-up script, it
>> seems pacemaker and rsyslog is stopped at the same time.
>>>
>>> Because I think it's rsyslog.service In RHEL7, whether this setting should
>> not be the "After = rsyslog.service"?
>>>
>>> Regards, Yusuke
>>
>> The "After = syslog.service" line neither helps nor hurts, and we should just
>> take it out.
>>
>> For a long time (and certainly in RHEL 7's systemd version 219), systemd
>> automatically orders the system log to start before and stop after other 
>> services,
>> so I don't think that's the cause of your problem.
>>
>> I'm not sure what would cause that behavior; can you post the messages that
>> are logged once shutdown is initiated?


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Fwd: dlm not starting

[Ken Gaillot]

> am configuring shared storage for 2 nodes (Cent 7) installed
> pcs/gfs2-utils/lvm2-cluster when creating resource unable to start dlm
> 
>  crm_verify -LV
>error: unpack_rsc_op:Preventing dlm from re-starting anywhere:
> operation start failed 'not configured' (6)

Are you using the ocf:pacemaker:controld resource agent for dlm?
Normally it logs what the problem is when returning 'not configured',
but I don't see it below. As far as I know, it will return 'not
configured' if stonith-enabled=false or globally-unique=true, as those
are incompatible with DLM.

There is also a rare cluster error condition that will be reported as
'not configured', but it will always be accompanied by "Invalid resource
definition" in the logs.

> 
> Feb 05 13:34:26 [24262] libcompute1pengine: info:
> determine_online_status:  Node libcompute1 is online
> Feb 05 13:34:26 [24262] libcompute1pengine: info:
> determine_online_status:  Node libcompute2 is online
> Feb 05 13:34:26 [24262] libcompute1pengine:  warning:
> unpack_rsc_op_failure:Processing failed op start for dlm on
> libcompute1: not configured (6)
> Feb 05 13:34:26 [24262] libcompute1pengine:error: unpack_rsc_op:
>  Preventing dlm from re-starting anywhere: operation start failed 'not
> configured' (6)
> Feb 05 13:34:26 [24262] libcompute1pengine:  warning:
> unpack_rsc_op_failure:Processing failed op start for dlm on
> libcompute1: not configured (6)
> Feb 05 13:34:26 [24262] libcompute1pengine:error: unpack_rsc_op:
>  Preventing dlm from re-starting anywhere: operation start failed 'not
> configured' (6)
> Feb 05 13:34:26 [24262] libcompute1pengine: info: native_print: dlm
> (ocf::pacemaker:controld):  FAILED libcompute1
> Feb 05 13:34:26 [24262] libcompute1pengine: info:
> get_failcount_full:   dlm has failed INFINITY times on libcompute1
> Feb 05 13:34:26 [24262] libcompute1pengine:  warning:
> common_apply_stickiness:  Forcing dlm away from libcompute1 after
> 100 failures (max=100)
> Feb 05 13:34:26 [24262] libcompute1pengine: info: native_color:
> Resource dlm cannot run anywhere
> Feb 05 13:34:26 [24262] libcompute1pengine:   notice: LogActions:
> Stopdlm (libcompute1)
> Feb 05 13:34:26 [24262] libcompute1pengine:   notice:
> process_pe_message:   Calculated Transition 59:
> /var/lib/pacemaker/pengine/pe-input-176.bz2
> Feb 05 13:34:26 [24263] libcompute1   crmd: info:
> do_state_transition:  State transition S_POLICY_ENGINE ->
> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE
> origin=handle_response ]
> Feb 05 13:34:26 [24263] libcompute1   crmd: info: do_te_invoke:
> Processing graph 59 (ref=pe_calc-dc-1454697266-177) derived from
> /var/lib/pacemaker/pengine/pe-input-176.bz2


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] "After = syslog.service" it is not working?

[Ken Gaillot]

On 02/08/2016 12:20 AM, 飯田 雄介 wrote:
> Hi, Ken
> 
> Log sent last time, it was acquired in an environment that has been set so as 
> to output the syslog to a file called ha-log.
> Because I think that this in hard to understand, we again get the log in a 
> more set close to default.
> I have collected the two types of log.
> Each attachment was, including those of the following other.
> /etc/rsyslog.conf
> /usr/lib/systemd/system/pacemaker.service
> /usr/lib/systemd/system/corosync.service
> /usr/lib/systemd/system/rsyslog.service
> 
> after-syslog.tar.bz2 is that to get the information in a set of "After = 
> syslog.service".
> 
> - Time that rsyslog has stopped (from rhel72-7 / messages)
> Feb 8 14:24:54 rhel72-7 rsyslogd: [origin software = "rsyslogd" swVersion = 
> "7.4.7" x-pid = "769" x-info = "http://www.rsyslog.com;] exiting on signal 15.
> 
> - Time that Pacemaker is stopped (from rhel72-7 / pacemaker.log)
> Feb 08 14:24:56 [8235] rhel72-7 pacemakerd: notice: pcmk_shutdown_worker: 
> Shutdown complete
> 
> From the above, Pacemaker will find that it has been stopped after a halt on 
> rsyslog.
> 
> after-rsyslog.tar.bz2 is that to get the information to change the setting of 
> "After = rsyslog.service".
> 
> - Time that Pacemaker is stopped (from rhel72-7 / messages)
> Feb 8 14:44:52 rhel72-7 pacemakerd [18528]: notice: Shutdown complete
> 
> - Time that rsyslog has stopped (from rhel72-7 / messages)
> Feb 8 14:44:52 rhel72-7 rsyslogd: [origin software = "rsyslogd" swVersion = 
> "7.4.7" x-pid = "768" x-info = "http://www.rsyslog.com;] exiting on signal 15.
> 
> From the above, rsyslog will find that it has been stopped after the 
> Pacemaker.
> 
> Is this information will help you understand?
> 
> Regards, Yusuke

Hi Yusuke,

Systemd wants all syslog implementations to use Requires=syslog.socket
and Alias=syslog.service:
https://www.freedesktop.org/wiki/Software/systemd/syslog/

But rsyslog prefers to do things differently, so those are commented out
in its rsyslog.service unit file:
http://lists.adiscon.net/pipermail/rsyslog/2015-March/039885.html

Mr. Systemd himself commented on a similar issue in
https://bugzilla.redhat.com/show_bug.cgi?id=766103#c13 : "I figure this
is a problem of the past as rsyslog will read shutdown messages (and
early-boot messages) from systemd's buffer on next reboot."

And the rsyslog folks seem to agree:
http://lists.adiscon.net/pipermail/rsyslog/2015-March/039888.html

So it may be that the rsyslog will print the later messages on the next
boot, or it may be possible to see them using journald instead of
looking in rsyslog's text logs.

Since the syslog.service alias exists for the purpose of abstracting the
specific syslog implementation from other software, it *should* be
sufficient for Pacemaker to order after it, but here we see rsyslog
doesn't use it. I suppose we could order after both syslog.service and
rsyslog.service; I don't think that would hurt.

>> -Original Message-
>> From: Ken Gaillot [mailto:kgail...@redhat.com]
>> Sent: Saturday, February 06, 2016 7:32 AM
>> To: 飯田 雄介; 'Cluster Labs - All topics related to open-source clustering
>> welcomed'
>> Cc: 島崎 勝美
>> Subject: Re: [ClusterLabs] "After = syslog.service" it is not working?
>>
>> On 01/28/2016 10:27 PM, 飯田 雄介 wrote:
>>> Hi, Ken
>>>
>>> I have to get the log.
>>> shutdown the -r now ran to "03:00:22".
>>> rsyslog also "03:00:22" to receive a TERM appears to have stopped.
>>>
>>> Regards, Yusuke
>>
>> Hello Yusuke,
>>
>> I see in your logs that /var/log/messages does not have any messages at all
>> from pacemaker (not only at shutdown).
>>
>> Meanwhile, pacemaker.log does have messages, including from shutdown.
>> The cluster writes directly to this file, without going through rsyslog.
>>
>> Both logs end at 03:00:22. I'm guessing that systemd waited until pacemaker
>> exited, then stopped rsyslog. I don't think they're exiting at the same time
>> (or that rsyslog exited before pacemaker).
>>
>> Instead, I suspect that either pacemaker is configured not to log via syslog,
>> or rsyslog is configured not to send pacemaker logs to /var/log/messages.
>>
>> Check the value of PCMK_logfacility and PCMK_logpriority in your
>> /etc/sysconfig/pacemaker. By default, pacemaker will log via syslog, but if
>> these are explicitly set, they might be turning that off.
>>
>> Also, pacemaker will try to inherit log options from corosync, so check
>> /etc/corosync/corosync.conf for logging options, esp

Re: [ClusterLabs] Cluster resources migration from CMAN to Pacemaker

[Ken Gaillot]

On 01/27/2016 02:34 AM, jaspal singla wrote:
> Hi Jan,
> 
> Thanks for your replies!!
> 
> I have couple of concerns more to answer, please help!

I'm not familiar with rgmanager, so there may be better ways that
hopefully someone else can suggest, but here are some ideas off the top
of my head:

> 1) In CMAN, there was meta attribute - autostart=0 (This parameter disables
> the start of all services when RGManager starts). Is there any way for such
> behavior in Pacemaker?
> 
> I tried to explore is-manged=0 but when I start the cluster through pcs
> cluster start OR pcs cluster start --all, my all of the resources gets
> started (even the one which has meta attribute configured as
> is-managed=false). Any clue to achieve  such behavior?
> 
> What does is-manged=false do?

Any service with is-managed=false should not be started or stopped by
the cluster. If they are already running, they should be left running;
if they are stopped, they should be left stopped.

I'm not sure why it didn't work in your test; maybe attach the output of
"pcs cluster cib" and "pcs status" with the setting in effect.

I don't think there's any exact replacement for autostart in pacemaker.
Probably the closest is to set target-role=Stopped before stopping the
cluster, and set target-role=Started when services are desired to be
started.

> 2) Please put some alternatives to exclusive=0 and __independent_subtree?
> what we have in Pacemaker instead of these?

My first though for exclusive=0 would be to configure negative
colocation constraints between that resource and all other resources.
For example:

  pcs constraint colocation add A with B "-INFINITY"

says that A should never run on a node where B is running (B is unaffected).

For __independent_subtree, each component must be a separate pacemaker
resource, and the constraints between them would depend on exactly what
you were trying to accomplish. The key concepts here are ordering
constraints, colocation constraints, kind=Mandatory/Optional (for
ordering constraints), and ordered sets.

> My cluster depends on these meta attributes and I am getting nothing in
> pacemaker as an alternative to these one. Please see, if it is feasible to
> achieve such meta attributes in Pacemaker.
> 
> 
> Thanks
> Jaspal
> 
> 
> 
> 
>> On 24/01/16 16:54 +0530, jaspal singla wrote:
>>> Thanks Digimer for letting me about the tool. I was unaware of any such
>>> tools!! And because of that I was able to search clufter-cli tool for the
>>> migration.
>>>
>>> Thanks a lot John for explaining each and everything in detailed manner.
>> I
>>> am really admired the knowledge you guys have!!
>>>
>>> I also noticed that clufter tool is written by you :). I am very thankful
>>> to you as it would save the ass of millions people like me who may have
>> had
>>> difficulties in migration of their legacy programs from CMAN to
>> Pacemaker.
>>
>> Glad to hear this, indeed :)
>>
>>> As suggested I tried to migrate my existing cluster.conf file from CMAN
>> to
>>> Pacemaker through the use of clufter. But have couple of queries going
>>> forward, would appreciate if you could answer these.
>>>
>>> Please find In-line queries:
>>
>> Answers ditto...
>>
 Date: Fri, 22 Jan 2016 21:52:17 +0100
 From: Jan Pokorn? 
 Subject: Re: [ClusterLabs] Cluster resources migration from CMAN to
 Pacemaker
 Message-ID: <20160122205217.ge28...@redhat.com>

 yes, as Digimer mentioned, clufter is the tool you may want to look
 at.  Do not expect fully automatic miracles from it, though.
 It's meant to show the conversion path, but one has to be walk it
 very carefully and make adjustments every here and there.
 In part because there is not a large overlap between resource agents
 of both kinds.

 On 22/01/16 17:32 +0530, jaspal singla wrote:
> I desperately need some help in order to migrate my cluster
>> configuration
> from CMAN (RHEL-6.5) to PACEMAKER (RHEL-7.1).
>
> I have tried to explore a lot but couldn't find similarities
>> configuring
> same resources (created in CMAN's cluster.conf file) to Pacemaker.
>
> I'd like to share cluster.conf of RHEL-6.5 and want to achieve the same
> thing through Pacemaker. Any help would be greatly appreciable!!
>
> *Cluster.conf file*
>
> ##
>

 [reformatted configuration file below for better readability and added
 some comment in-line]

> 
  ^^^
  no, this is not the way to increase config version

 This seems to be quite frequented mistake; looks like configuration
 tools should have strictly refrained from using this XML declaration
 in the first place.

> 
>   > post_join_delay="3"/>
>   
> 
>   
> 

 (I suppose that other nodes were omitted)

>>>
>>> No, its 

Re: [ClusterLabs] cron-suitable cluster status check

[Ken Gaillot]

On 02/27/2016 03:56 PM, Devin Reade wrote:
> Right now in a test cluster on CentOS 7 I'm occasionally seeing
> resource monitoring failures and, just today, a failure to start
> a fencing agent.  While I need to track those down problems, the
> issue I want to discuss here is being notified when there is a
> problem with the cluster, where there is not a nagios-type monitoring
> system in place.
> 
> On an older CentOS 5 cluster I have a cron job that periodically runs
> 'crm_verify -LV'.  If the return code is non-zero, the output of
> that command (and some other info) is mailed to the operator.  That
> mechanism has been working well for years.
> 
> However on CentOS 7, when the cluster gets into this state 'crm_verify -LV'
> returns zero, and its output claims there is no problem.  However in
> 'crm_mon -f' I can see that I've got resource failures and nonzero
> failcounts.
> 
> I tried 'pcs cluster status', however when the cluster is properly
> working (no failures), that command still has a return code of '1',
> probably because I get the 'Error: no nodes found in corosync.conf'
> which is an ignorable condition per
> .
> 
> Is there a command that I can run from cron in the current cluster
> tools to tell me the simple answer of whether there is *anything*
> failed in the cluster, preferably based on its return code?

I'm not sure about the CentOS 5 days, but at least now, crm_verify is
intended to verify the syntax of a cluster's configuration rather than
its status.

The simplest method is "crm_mon -s", which gives a one-line
nagios-compatible output with return code 0=success and 1=problem.
However. it returns 1 for cluster not running, no DC, or offline nodes.

Back in the day, I used check_crm with nagios/icinga. It's a perl script
that parses the output of crm_mon -1rf and crm configure show. It's
trivial to use such a check outside a monitoring system, and it could be
modified to work with pcs and current crm_mon output, so maybe it could
help:

https://exchange.nagios.org/directory/Plugins/Clustering-and-High-2DAvailability/Check-CRM/details

> The CentOS 7 cluster is running:
>corosync 2.3.4
>pacemaker 1.1.13
> 
> The CentOS 5 cluster is running:
>corosync 1.2.7
>pacemaker 1.0.12
> 
> The corosync.conf is included below:
> 
> - cut here and be careful of pointy scissors -
> totem {
> version: 2
> #secauth: off
> cluster_name: somecluster
> #transport: udpu
> rrp_mode: passive
> crypto_hash: sha256
> clear_node_high_bit: yes
> 
> interface {
> ringnumber: 0
> bindnetaddr: 192.168.1.0
> mcastaddr: 239.192.0.5
> mcastport: 5406
> }
> interface {
> ringnumber: 1
> bindnetaddr: 192.168.2.0
> mcastaddr: 239.192.0.6
> mcastport: 5408
> }
> }
> 
> quorum {
> provider: corosync_votequorum
> two_node: 1
> expected_votes: 2
> }
> 
> logging {
> to_syslog: yes
> }
> 
> - cut here and be careful of pointy scissors -
> 
> Devin


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Pacemaker shows false status of a resource and doesn't react on OCF_NOT_RUNNING rc.

[Ken Gaillot]

On 01/19/2016 10:30 AM, Kostiantyn Ponomarenko wrote:
> The resource that wasn't running, but was reported as running, is
> "adminServer".
> 
> Here are a brief chronological description:
> 
> [Jan 19 23:42:16] The first time Pacemaker triggers its monitor function at
> line #1107. (those lines are from its Resource Agent)
> [Jan 19 23:42:16] Then Pacemaker starts the resource - line #1191.
> [Jan 19 11:42:53] The first failure is reported by monitor operation at
> line #1543.
> [Jan 19 11:42:53] The fail-count is set, but I don't see any attempt from
> Pacemaker to "start" the resource - the start function is not called (from
> the logs) - line #1553.
> [Jan 19 12:27:56] Then adminServer's monitor operation keeps returning
> $OCF_NOT_RUNNING - starts at line #1860.
> [Jan 19 12:57:53] Then the expired failcount is cleared at line #1969.
> [Jan 19 12:57:53] Another call of the monitor function happens at line
> #2038.
> [Jan 19 12:57:53] I assume that the line #2046 means "not running" (?).
> [Jan 19 12:57:53] The "stop" function is called - line #2150
> [Jan 19 12:57:53] The "start" function is called and the resource is
> successfully started - line #2164
> 
> 
> The time change occurred while cluster was starting, I see this from
> "journalctl --since="2016-01-19" --until="2016-01-20"":
> 
> Jan 19 23:10:39 A2-2U12-302-LS ntpd[2210]: 0.0.0.0 c61c 0c clock_step
> -43193.793349 s
> Jan 19 11:10:45 A2-2U12-302-LS ntpd[2210]: 0.0.0.0 c614 04 freq_mode
> Jan 19 11:10:45 A2-2U12-302-LS systemd[1]: Time has been changed
> 
> I am attaching corosync.log.

The time change is interesting. I suspect what's happening is that
pacemaker considers the failed monitor "older" than the original
successful one, and so ignores it.

In general, we don't support large clock shifts in a running cluster (a
topic that has come up before on this list). But if you can reproduce
the behavior with 1.1.14, feel free to open a bug report. It might be
worth revisiting to see if there is anything we can do about it.

> Thank you,
> Kostia
> 
> On Tue, Jan 19, 2016 at 5:17 PM, Bogdan Dobrelya <bdobre...@mirantis.com>
> wrote:
> 
>> On 19.01.2016 16:13, Ken Gaillot wrote:
>>> On 01/19/2016 06:49 AM, Kostiantyn Ponomarenko wrote:
>>>> One of resources in my cluster is not actually running, but "crm_mon"
>> shows
>>>> it with the "Started" status.
>>>> Its resource agent's monitor function returns "$OCF_NOT_RUNNING", but
>>>> Pacemaker doesn't react on this anyhow - crm_mon show the resource as
>>>> Started.
>>>> I couldn't find an explanation to this behavior, so I suppose it is a
>> bug,
>>>> is it?
>>>
>>> That is unexpected. Can you post the configuration and logs from around
>>> the time of the issue?
>>>
>>
>> Oh, sorry, I forgot to mention the related thread [0]. That is exactly
>> the case I reported there. Looks same, so I thought you've just updated
>> my thread :)
>>
>> These may be merged perhaps.
>>
>> [0] http://clusterlabs.org/pipermail/users/2016-January/002035.html
>>
>>>
>>> ___
>>> Users mailing list: Users@clusterlabs.org
>>> http://clusterlabs.org/mailman/listinfo/users
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://bugs.clusterlabs.org


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Pacemaker shows false status of a resource and doesn't react on OCF_NOT_RUNNING rc.

[Ken Gaillot]

On 01/19/2016 11:02 AM, Kostiantyn Ponomarenko wrote:
> Just in case, this is the monitor function from the resource agent:
> ra_monitor() {
> #   ocf_log info "$RA: [monitor]"
> systemctl status ${service}
> rc=$?
> if [ "$rc" -eq "0" ]; then
> return $OCF_SUCCESS
> fi
> 
> ocf_log warn "$RA: [monitor] : got rc=$rc"
> return $OCF_NOT_RUNNING
> }

Out of curiosity, why are you wrapping systemctl with OCF when pacemaker
supports systemd resources natively? The native support works around a
number of quirks in systemd behavior. (In fact a recent commit to the
master branch handles yet another one.)

> Thank you,
> Kostia
> 
> On Tue, Jan 19, 2016 at 6:30 PM, Kostiantyn Ponomarenko <
> konstantin.ponomare...@gmail.com> wrote:
> 
>> The resource that wasn't running, but was reported as running, is
>> "adminServer".
>>
>> Here are a brief chronological description:
>>
>> [Jan 19 23:42:16] The first time Pacemaker triggers its monitor function
>> at line #1107. (those lines are from its Resource Agent)
>> [Jan 19 23:42:16] Then Pacemaker starts the resource - line #1191.
>> [Jan 19 11:42:53] The first failure is reported by monitor operation at
>> line #1543.
>> [Jan 19 11:42:53] The fail-count is set, but I don't see any attempt from
>> Pacemaker to "start" the resource - the start function is not called (from
>> the logs) - line #1553.
>> [Jan 19 12:27:56] Then adminServer's monitor operation keeps returning
>> $OCF_NOT_RUNNING - starts at line #1860.
>> [Jan 19 12:57:53] Then the expired failcount is cleared at line #1969.
>> [Jan 19 12:57:53] Another call of the monitor function happens at line
>> #2038.
>> [Jan 19 12:57:53] I assume that the line #2046 means "not running" (?).
>> [Jan 19 12:57:53] The "stop" function is called - line #2150
>> [Jan 19 12:57:53] The "start" function is called and the resource is
>> successfully started - line #2164
>>
>>
>> The time change occurred while cluster was starting, I see this from
>> "journalctl --since="2016-01-19" --until="2016-01-20"":
>>
>> Jan 19 23:10:39 A2-2U12-302-LS ntpd[2210]: 0.0.0.0 c61c 0c clock_step
>> -43193.793349 s
>> Jan 19 11:10:45 A2-2U12-302-LS ntpd[2210]: 0.0.0.0 c614 04 freq_mode
>> Jan 19 11:10:45 A2-2U12-302-LS systemd[1]: Time has been changed
>>
>> I am attaching corosync.log.
>>
>> Thank you,
>> Kostia
>>
>> On Tue, Jan 19, 2016 at 5:17 PM, Bogdan Dobrelya <bdobre...@mirantis.com>
>> wrote:
>>
>>> On 19.01.2016 16:13, Ken Gaillot wrote:
>>>> On 01/19/2016 06:49 AM, Kostiantyn Ponomarenko wrote:
>>>>> One of resources in my cluster is not actually running, but "crm_mon"
>>> shows
>>>>> it with the "Started" status.
>>>>> Its resource agent's monitor function returns "$OCF_NOT_RUNNING", but
>>>>> Pacemaker doesn't react on this anyhow - crm_mon show the resource as
>>>>> Started.
>>>>> I couldn't find an explanation to this behavior, so I suppose it is a
>>> bug,
>>>>> is it?
>>>>
>>>> That is unexpected. Can you post the configuration and logs from around
>>>> the time of the issue?
>>>>
>>>
>>> Oh, sorry, I forgot to mention the related thread [0]. That is exactly
>>> the case I reported there. Looks same, so I thought you've just updated
>>> my thread :)
>>>
>>> These may be merged perhaps.
>>>
>>> [0] http://clusterlabs.org/pipermail/users/2016-January/002035.html



___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Pacemaker shows false status of a resource and doesn't react on OCF_NOT_RUNNING rc.

[Ken Gaillot]

On 01/19/2016 12:20 PM, Kostiantyn Ponomarenko wrote:
> I've put the wrong entry from "journalctl --since="2016-01-19"
> --until="2016-01-20"".
> The correct one is:
> 
> Jan 19 23:42:24 A2-2U12-302-LS ntpd[2204]: 0.0.0.0 c61c 0c clock_step
> -43194.111405 s
> Jan 19 11:42:29 A2-2U12-302-LS ntpd[2204]: 0.0.0.0 c614 04 freq_mode
> Jan 19 11:42:29 A2-2U12-302-LS systemd[1]: Time has been changed
> 
> Yes, the really first monitor operation was successful.
> But I still have a question.
> The first monitor op was at [Jan 19 23:42:16] and the failure was
> considered by Pacemaker at [Jan 19 12:57:53].
> So, the first one still remains older and here I don't understand why
> Pacemaker considers it failed?

23:42 is later (newer) than 12:57. In a cluster, with the possibility of
nodes leaving and rejoining, and having different loads, etc., it is
possible for updates and operation results to come into the DC out of
chronological order. So pacemaker relies on the timestamps to determine
what's current.

I haven't traced this issue through the code, but my guess is that when
pacemaker receives the 12:57 failure, it will of course log that it
happened, but if it has what it thinks is a newer result (from 23:42),
it won't treat the failure as the current state. Again, that's just a
guess at this point, but I think that's what's happening.

> Thank you,
> Kostia
> 
> On Tue, Jan 19, 2016 at 8:02 PM, Ken Gaillot <kgail...@redhat.com> wrote:
> 
>> On 01/19/2016 10:30 AM, Kostiantyn Ponomarenko wrote:
>>> The resource that wasn't running, but was reported as running, is
>>> "adminServer".
>>>
>>> Here are a brief chronological description:
>>>
>>> [Jan 19 23:42:16] The first time Pacemaker triggers its monitor function
>> at
>>> line #1107. (those lines are from its Resource Agent)
>>> [Jan 19 23:42:16] Then Pacemaker starts the resource - line #1191.
>>> [Jan 19 11:42:53] The first failure is reported by monitor operation at
>>> line #1543.
>>> [Jan 19 11:42:53] The fail-count is set, but I don't see any attempt from
>>> Pacemaker to "start" the resource - the start function is not called
>> (from
>>> the logs) - line #1553.
>>> [Jan 19 12:27:56] Then adminServer's monitor operation keeps returning
>>> $OCF_NOT_RUNNING - starts at line #1860.
>>> [Jan 19 12:57:53] Then the expired failcount is cleared at line #1969.
>>> [Jan 19 12:57:53] Another call of the monitor function happens at line
>>> #2038.
>>> [Jan 19 12:57:53] I assume that the line #2046 means "not running" (?).
>>> [Jan 19 12:57:53] The "stop" function is called - line #2150
>>> [Jan 19 12:57:53] The "start" function is called and the resource is
>>> successfully started - line #2164
>>>
>>>
>>> The time change occurred while cluster was starting, I see this from
>>> "journalctl --since="2016-01-19" --until="2016-01-20"":
>>>
>>> Jan 19 23:10:39 A2-2U12-302-LS ntpd[2210]: 0.0.0.0 c61c 0c clock_step
>>> -43193.793349 s
>>> Jan 19 11:10:45 A2-2U12-302-LS ntpd[2210]: 0.0.0.0 c614 04 freq_mode
>>> Jan 19 11:10:45 A2-2U12-302-LS systemd[1]: Time has been changed
>>>
>>> I am attaching corosync.log.
>>
>> The time change is interesting. I suspect what's happening is that
>> pacemaker considers the failed monitor "older" than the original
>> successful one, and so ignores it.
>>
>> In general, we don't support large clock shifts in a running cluster (a
>> topic that has come up before on this list). But if you can reproduce
>> the behavior with 1.1.14, feel free to open a bug report. It might be
>> worth revisiting to see if there is anything we can do about it.
>>
>>> Thank you,
>>> Kostia
>>>
>>> On Tue, Jan 19, 2016 at 5:17 PM, Bogdan Dobrelya <bdobre...@mirantis.com
>>>
>>> wrote:
>>>
>>>> On 19.01.2016 16:13, Ken Gaillot wrote:
>>>>> On 01/19/2016 06:49 AM, Kostiantyn Ponomarenko wrote:
>>>>>> One of resources in my cluster is not actually running, but "crm_mon"
>>>> shows
>>>>>> it with the "Started" status.
>>>>>> Its resource agent's monitor function returns "$OCF_NOT_RUNNING", but
>>>>>> Pacemaker doesn't react on this anyhow - crm_mon show the resource as
>>>>>> Started.
>>>>>> I couldn't find an explanation to this behavior, so I suppose it is a
>>>> bug,
>>>>>> is it?
>>>>>
>>>>> That is unexpected. Can you post the configuration and logs from around
>>>>> the time of the issue?
>>>>>
>>>>
>>>> Oh, sorry, I forgot to mention the related thread [0]. That is exactly
>>>> the case I reported there. Looks same, so I thought you've just updated
>>>> my thread :)
>>>>
>>>> These may be merged perhaps.
>>>>
>>>> [0] http://clusterlabs.org/pipermail/users/2016-January/002035.html


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Wait until resource is really ready before moving clusterip

[Ken Gaillot]

On 01/26/2016 05:06 AM, Joakim Hansson wrote:
> Thanks for the help guys.
> I ended up patching together my own RA from the Delay and Dummy RA's and
> using curl to request the header of solr's ping request handler on
> localhost, which made the resource start return a bit more dynamic.
> However, now I have another problem which I don't think is related to my RA.
> For some reason when failing over the nodes, the ClusterIP (vIP below)
> seems to avoid the node running the fencing agent:
> 
> pcs status
> 
> Online: [ node01 node02 ]
> OFFLINE: [ node03 ]
> 
> Full list of resources:
> 
>  VMWare-fence   (stonith:fence_vmware_soap):Started node02
>  Clone Set: dlm-clone [dlm]
>  Started: [ node01 node02 ]
>  Stopped: [ node03 ]
>  Clone Set: GFS2-clone [GFS2] (unique)
>  GFS2:0 (ocf::heartbeat:Filesystem):Started node01
>  GFS2:1 (ocf::heartbeat:Filesystem):Stopped
>  GFS2:2 (ocf::heartbeat:Filesystem):Started node02
>  Clone Set: Tomcat-clone [Tomcat]
>  Started: [ node02 ]
>  Stopped: [ node01 node03 ]
>  vIP(ocf::heartbeat:IPaddr2): Stopped
> 
> Notice how the tomcat-clone is started on node02 but the vIP remains
> stopped.
> If I start the fence agent on any of the other nodes the same thing happens
> (ie, vIP avoiding the fencing node)
> Any idea why this happens?
> 
> Output of 'pcs config show':
> https://github.com/apepojken/pacemaker/blob/master/Config

I notice you have mutliple ordering constraints but only one colocation
constraint. That means, for example, that tomcat-clone must be started
after GFS2, but it does not have to be on the same node. I'm pretty sure
you want colocation constraints as well, to make them start on the same
node.

FYI, a group is like a shorthand for ordering and constraint constraints
for multiple resources that need to be kept together and started/stopped
in order.

I also see you have globally-unique=true on GFS2-clone. You probably do
not want this. globally-unique=false (the default) is more common, and
means that all clone instances are interchangeable, and is usually
configured with clone-node-max=1, because only one instance is ever
needed on any one node. globally-unique=true means that each clone
instance handles a different subset of requests, and is usually
configured with clone-node-max > 1 so that multiple clone instances can
run on a single node if needed.

I don't see from that alone why vIP wouldn't start, but take care of the
above issues first, and see what the behavior is then.

> Thanks again!
> 
> 2016-01-20 1:14 GMT+01:00 Jan Pokorný :
> 
>> On 14/01/16 14:46 +0100, Kristoffer Grönlund wrote:
>>> Joakim Hansson  writes:
 When adding the Delay RA it starts throwing a bunch of errors and the
 cluster starts fencing the nodes one by one.

 The error's I get with "pcs status":

 Failed Actions:
 * Delay_monitor_0 on node03 'unknown error' (1): call=51, status=Timed
>> Out,
 exit
 reason='none',
 last-rc-change='Thu Jan 14 13:30:14 2016', queued=0ms, exec=30002ms
 * Delay_monitor_0 on node01 'unknown error' (1): call=53, status=Timed
>> Out,
 exit
 reason='none',
 last-rc-change='Thu Jan 14 13:30:14 2016', queued=0ms, exec=30002ms
 * Delay_monitor_0 on node02 'unknown error' (1): call=51, status=Timed
>> Out,
 exit
 reason='none',
 last-rc-change='Thu Jan 14 13:30:14 2016', queued=0ms, exec=30006ms

 and in the /var/log/pacemaker.log:


>> https://github.com/apepojken/pacemaker-errors/blob/master/ocf:heartbeat:Delay

 I added the Delay RA with:

 pcs resource create Delay ocf:heartbeat:Delay \
 startdelay="120" meta target-role=Started \
 op start timeout="180"

 and my config looks like this:

 https://github.com/apepojken/pacemaker/blob/master/Config

 Am I missing something obvious here?
>>>
>>> It looks like you have a monitor operation configured for the Delay
>>> resource, but you haven't set the mondelay parameter. But either way,
>>> there is no reason to monitor the Delay resource, so remove that. Same
>>> thing for the stop operation, just remove it.
>>>
>>> I'm guessing pcs adds these by default.
>>
>> It's true that pcs adds equivalent of "op monitor interval=60s"
>> as an unconditional fallback when defining a new resource.
>> Other operations are driven solely by explicit values or by
>> defaults for particular resource, and this can be turned off
>> via "--no-default-ops" option to pcs.
>>
>> FWIW, this could be a way to have monitor explicitly deactivated:
>>
>> pcs resource create   ... op monitor interval=0s
>>
>> --
>> Jan (Poki)


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: 

Re: [ClusterLabs] ClusterIP location constraint reappears after reboot

[Ken Gaillot]

On 02/18/2016 01:07 PM, Jeremy Matthews wrote:
> Hi,
> 
> We're having an issue with our cluster where after a reboot of our system a 
> location constraint reappears for the ClusterIP. This causes a problem, 
> because we have a daemon that checks the cluster state and waits until the 
> ClusterIP is started before it kicks off our application. We didn't have this 
> issue when using an earlier version of pacemaker. Here is the constraint as 
> shown by pcs:
> 
> [root@g5se-f3efce cib]# pcs constraint
> Location Constraints:
>   Resource: ClusterIP
> Disabled on: g5se-f3efce (role: Started)
> Ordering Constraints:
> Colocation Constraints:
> 
> ...and here is our cluster status with the ClusterIP being Stopped:
> 
> [root@g5se-f3efce cib]# pcs status
> Cluster name: cl-g5se-f3efce
> Last updated: Thu Feb 18 11:36:01 2016
> Last change: Thu Feb 18 10:48:33 2016 via crm_resource on g5se-f3efce
> Stack: cman
> Current DC: g5se-f3efce - partition with quorum
> Version: 1.1.11-97629de
> 1 Nodes configured
> 4 Resources configured
> 
> 
> Online: [ g5se-f3efce ]
> 
> Full list of resources:
> 
> sw-ready-g5se-f3efce   (ocf::pacemaker:GBmon): Started g5se-f3efce
> meta-data  (ocf::pacemaker:GBmon): Started g5se-f3efce
> netmon (ocf::heartbeat:ethmonitor):Started g5se-f3efce
> ClusterIP  (ocf::heartbeat:IPaddr2):   Stopped
> 
> 
> The cluster really just has one node at this time.
> 
> I retrieve the constraint ID, remove the constraint, verify that ClusterIP is 
> started,  and then reboot:
> 
> [root@g5se-f3efce cib]# pcs constraint ref ClusterIP
> Resource: ClusterIP
>   cli-ban-ClusterIP-on-g5se-f3efce
> [root@g5se-f3efce cib]# pcs constraint remove cli-ban-ClusterIP-on-g5se-f3efce
> 
> [root@g5se-f3efce cib]# pcs status
> Cluster name: cl-g5se-f3efce
> Last updated: Thu Feb 18 11:45:09 2016
> Last change: Thu Feb 18 11:44:53 2016 via crm_resource on g5se-f3efce
> Stack: cman
> Current DC: g5se-f3efce - partition with quorum
> Version: 1.1.11-97629de
> 1 Nodes configured
> 4 Resources configured
> 
> 
> Online: [ g5se-f3efce ]
> 
> Full list of resources:
> 
> sw-ready-g5se-f3efce   (ocf::pacemaker:GBmon): Started g5se-f3efce
> meta-data  (ocf::pacemaker:GBmon): Started g5se-f3efce
> netmon (ocf::heartbeat:ethmonitor):Started g5se-f3efce
> ClusterIP  (ocf::heartbeat:IPaddr2):   Started g5se-f3efce
> 
> 
> [root@g5se-f3efce cib]# reboot
> 
> after reboot, log in, and the constraint is back and ClusterIP has not 
> started.
> 
> 
> I have noticed in /var/lib/pacemaker/cib that the cib-x.raw files get created 
> when there are changes to the cib (cib.xml). After a reboot, I see the 
> constraint being added in a diff between .raw files:
> 
> [root@g5se-f3efce cib]# diff cib-7.raw cib-8.raw
> 1c1
> <  validate-with="pacemaker-1.2" cib-last-written="Thu Feb 18 11:44:53 2016" 
> update-origin="g5se-f3efce" update-client="crm_resource" 
> crm_feature_set="3.0.9" have-quorum="1" dc-uuid="g5se-f3efce">
> ---
>> > validate-with="pacemaker-1.2" cib-last-written="Thu Feb 18 11:46:49 2016" 
>> update-origin="g5se-f3efce" update-client="crm_resource" 
>> crm_feature_set="3.0.9" have-quorum="1" dc-uuid="g5se-f3efce">
> 50c50,52
> < 
> ---
>> 
>>   > role="Started" node="g5se-f3efce" score="-INFINITY"/>
>> 
> 
> 
> I have also looked in /var/log/cluster/corosync.log and seen logs where it 
> seems the cib is getting updated. I'm not sure if the constraint is being put 
> back in at shutdown or at start up. I just don't understand why it's being 
> put back in. I don't think our daemon code or other scripts are doing this,  
> but it is something I could verify.

I would look at any scripts running around that time first. Constraints
that start with "cli-" were created by one of the CLI tools, so
something must be calling it. The most likely candidates are pcs
resource move/ban or crm_resource -M/--move/-B/--ban.

> 
> 
> From "yum info pacemaker", my current version is:
> 
> Name: pacemaker
> Arch: x86_64
> Version : 1.1.12
> Release : 8.el6_7.2
> 
> My earlier version was:
> 
> Name: pacemaker
> Arch: x86_64
> Version : 1.1.10
> Release : 1.el6_4.4
> 
> I'm still using an earlier version pcs, because the new one seems to have 
> issues with python:
> 
> Name: pcs
> Arch: noarch
> Version : 0.9.90
> Release : 1.0.1.el6.centos
> 
> ***
> 
> If anyone has ideas on the cause or thoughts on this, anything would be 
> greatly appreciated.
> 
> Thanks!
> 
> 
> 
> Jeremy Matthews


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] Coming in Pacemaker 1.1.15: graceful Pacemaker Remote node stops

[Ken Gaillot]

Pacemaker's upstream master branch has a new feature that will be part
of the eventual 1.1.15 release.

Since version 1.1.10, Pacemaker has supported Pacemaker Remote nodes to
scale a cluster beyond the limits of a full cluster stack (and to manage
resources inside a VM that is itself managed by the cluster). Pacemaker
Remote nodes can run resources, but do not participate in quorum or
other activities of the underlying messaging layer.

In current released versions of Pacemaker, if the pacemaker_remoted
daemon is stopped on an active Pacemaker Remote node, the cluster will
fence the node. Because updating the software requires a restart of the
service, this means that resources creating Pacemaker Remote nodes must
be stopped before the node can be updated.

With the new feature, pacemaker_remoted will notify the cluster when it
needs to stop, and the cluster will gracefully move all resources off
the node, then wait for the node to come back up. This simplifies
package updates on Pacemaker Remote nodes, and allows the process to be
scripted more easily.

This new feature makes updates of Pacemaker Remote nodes more similar to
that of cluster nodes -- simply stop cluster services (in this case
pacemaker_remoted) before updating.
-- 
Ken Gaillot <kgail...@redhat.com>

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Clone Issue

[Ken Gaillot]

On 02/13/2016 08:09 PM, Frank D. Engel, Jr. wrote:
> Hi,
> 
> I'm new to the software, and with the list - just started experimenting
> with trying to get a cluster working using CentOS 7 and the pcs utility,
> and I've made some progress, but I can't quite figure out why I'm seeing
> this one behavior - hoping someone can help, might be something simple I
> haven't picked up on yet.
> 
> I have three nodes configured (running under VirtualBox) with shared
> storage using GFS2 - that much seems to be working ok.
> 
> I have a service called "WebSite" representing the Apache configuration,
> and I cloned that to create "WebSite-clone", which I would expect to run
> instances of on all three nodes.
> 
> However, if I leave "globally-unique" off, it will only run on one node,
> where if I turn it on, it will run on two, but never on all three.  I've
> tried a number of things to get this working.  I did verify that I can
> manually start and stop Apache on all three nodes and it works on any of
> them that way.

You don't want globally-unique=true; that's for cases where you want to
be able to run multiple instances of the service on the same machine if
necessary, because each clone handles different requests.

> Currently my status looks like this (with globally-unique set to false;
> "cluster-data" is my GFS2 filesystem):
> 
> Cluster name: lincl
> Last updated: Sat Feb 13 20:58:26 2016Last change: Sat Feb 13
> 20:45:08 2016 by root via crm_resource on lincl2-hb
> Stack: corosync
> Current DC: lincl2-hb (version 1.1.13-10.el7-44eb2dd) - partition with
> quorum
> 3 nodes and 13 resources configured
> 
> Online: [ lincl0-hb lincl1-hb lincl2-hb ]
> 
> Full list of resources:
> 
>  kdump(stonith:fence_kdump):Started lincl0-hb
>  Clone Set: dlm-clone [dlm]
>  Started: [ lincl0-hb lincl1-hb lincl2-hb ]
>  Master/Slave Set: cluster-data-clone [cluster-data]
>  Slaves: [ lincl0-hb lincl1-hb lincl2-hb ]
>  Clone Set: ClusterIP-clone [ClusterIP] (unique)
>  ClusterIP:0(ocf::heartbeat:IPaddr2):Started lincl2-hb
>  ClusterIP:1(ocf::heartbeat:IPaddr2):Started lincl0-hb
>  ClusterIP:2(ocf::heartbeat:IPaddr2):Started lincl1-hb
>  Clone Set: WebSite-clone [WebSite]
>  Started: [ lincl0-hb ]
>  Stopped: [ lincl1-hb lincl2-hb ]

The above says that the cluster successfully started a WebSite instance
on lincl0-hb, but it is for some reason prevented from doing so on the
other two nodes.

> Failed Actions:
> * WebSite:0_start_0 on lincl2-hb 'unknown error' (1): call=142,
> status=Timed Out, exitreason='Failed to access httpd status page.',
> last-rc-change='Sat Feb 13 19:55:45 2016', queued=0ms, exec=120004ms

This gives a good bit of info:

* The "start" action on the "WebSite" resource failed no node lincl2-hb.

* The failure was a timeout. The start action did not return in the
configured (or default) time.

* The reason given by the apache resource agent was "Failed to access
httpd status page".

> * WebSite:2_start_0 on lincl2-hb 'unknown error' (1): call=130,
> status=Timed Out, exitreason='none',
> last-rc-change='Sat Feb 13 19:33:49 2016', queued=0ms, exec=40003ms
> * WebSite:1_monitor_6 on lincl0-hb 'unknown error' (1): call=101,
> status=complete, exitreason='Failed to access httpd status page.',
> last-rc-change='Sat Feb 13 19:53:53 2016', queued=0ms, exec=0ms
> * WebSite:0_monitor_6 on lincl0-hb 'not running' (7): call=77,
> status=complete, exitreason='none',
> last-rc-change='Sat Feb 13 19:34:48 2016', queued=0ms, exec=0ms
> * WebSite:2_start_0 on lincl1-hb 'unknown error' (1): call=41,
> status=Timed Out, exitreason='none',
> last-rc-change='Sat Feb 13 19:53:41 2016', queued=1ms, exec=120004ms
> 
> 
> PCSD Status:
>   lincl0-hb: Online
>   lincl1-hb: Online
>   lincl2-hb: Online
> 
> Daemon Status:
>   corosync: active/enabled
>   pacemaker: active/enabled
>   pcsd: active/enabled
> 
> 
> 
> I'm not sure how to further troubleshoot those "Failed Actions" or how
> to clear them from the display?

Pacemaker relies on what the resource agent tells it, so when the
resource agent fails, you'll have to look at that rather than pacemaker
itself. Often, agents will print more detailed messages to the system
log. Otherwise, just verifying the resource configuration and so forth
is a good idea.

In this case, the big hint is the status page. The apache resource agent
relies on the /server-status URL to verify that apache is running.
Double-check that apache's configuration is identical on all nodes,
particularly the /server-status configuration.

Once you've addressed the root cause of a failed action, you can clear
it from the display with "pcs resource cleanup" -- see "man pcs" for the
options it takes.

Another good idea is (with the cluster stopped) to ensure you can start
apache manually on each node and see the server-status URL from that
node (using curl or wget or whatever).

> 
> Configuration of the WebSite-clone 

Re: [ClusterLabs] pacemaker remote configuration on ubuntu 14.04

[Ken Gaillot]

On 03/08/2016 11:38 PM, Сергей Филатов wrote:
> ssh -p 3121 compute-1
> ssh_exchange_identification: read: Connection reset by peer
> 
> That’s what I get in /var/log/pacemaker.log after restarting pacemaker_remote:
> Mar 09 05:30:27 [28031] compute-1.domain.com   lrmd: info: 
> crm_signal_dispatch:  Invoking handler for signal 15: Terminated
> Mar 09 05:30:27 [28031] compute-1.domain.com   lrmd: info: 
> lrmd_shutdown:Terminating with  0 clients
> Mar 09 05:30:27 [28031] compute-1.domain.com   lrmd: info: 
> qb_ipcs_us_withdraw:  withdrawing server sockets
> Mar 09 05:30:27 [28031] compute-1.domain.com   lrmd: info: 
> crm_xml_cleanup:  Cleaning up memory from libxml2
> Mar 09 05:30:27 [28193] compute-1.domain.com   lrmd: info: 
> crm_log_init: Changed active directory to 
> /var/lib/heartbeat/cores/root
> Mar 09 05:30:27 [28193] compute-1.domain.com   lrmd: info: 
> qb_ipcs_us_publish:   server name: lrmd
> Mar 09 05:30:27 [28193] compute-1.domain.com   lrmd:   notice: 
> lrmd_init_remote_tls_server:  Starting a tls listener on port 3121.
> Mar 09 05:30:28 [28193] compute-1.domain.com   lrmd:   notice: 
> bind_and_listen:  Listening on address ::
> Mar 09 05:30:28 [28193] compute-1.domain.com   lrmd: info: 
> qb_ipcs_us_publish:   server name: cib_ro
> Mar 09 05:30:28 [28193] compute-1.domain.com   lrmd: info: 
> qb_ipcs_us_publish:   server name: cib_rw
> Mar 09 05:30:28 [28193] compute-1.domain.com   lrmd: info: 
> qb_ipcs_us_publish:   server name: cib_shm
> Mar 09 05:30:28 [28193] compute-1.domain.com   lrmd: info: 
> qb_ipcs_us_publish:   server name: attrd
> Mar 09 05:30:28 [28193] compute-1.domain.com   lrmd: info: 
> qb_ipcs_us_publish:   server name: stonith-ng
> Mar 09 05:30:28 [28193] compute-1.domain.com   lrmd: info: 
> qb_ipcs_us_publish:   server name: crmd
> Mar 09 05:30:28 [28193] compute-1.domain.com   lrmd: info: main:  
>Starting

It looks like the cluster is not even trying to connect to the remote
node. pacemaker_remote here is binding only to IPv6, so the cluster will
need to contact it on that address.

What is your ocf:pacemaker:remote resource configuration?

Check your cluster node logs for the start action -- if your resource is
named R, the start action will be R_start_0. There will be two nodes of
interest: the node assigned the remote node resource, and the DC.

> I got only pacemaker-remote resource-agents pcs installed, so no 
> /etc/default/pacemaker file on remote node
> selinux is disabled and I specifically opened firewall on 2224, 3121 and 
> 21064 tcp and 5405 udp
> 
>> On 08 Mar 2016, at 08:51, Ken Gaillot <kgail...@redhat.com> wrote:
>>
>> On 03/07/2016 09:10 PM, Сергей Филатов wrote:
>>> Thanks for an answer. Turned out the problem was not in ipv6.
>>> Remote node is listening on 3121 port and it’s name is resolving fine.
>>> Got authkey file at /etc/pacemaker on both remote and cluster nodes.
>>> What can I check in addition? Is there any walkthrough for ubuntu?
>>
>> Nothing specific to ubuntu, but there's not much distro-specific to it.
>>
>> If you "ssh -p 3121" to the remote node from a cluster node, what do you
>> get?
>>
>> pacemaker_remote will use the usual log settings for pacemaker (probably
>> /var/log/pacemaker.log, probably configured in /etc/default/pacemaker on
>> ubuntu). You should see "New remote connection" in the remote node's log
>> when the cluster tries to connect, and "LRMD client connection
>> established" if it's successful.
>>
>> As always, check for firewall and SELinux issues.
>>
>>>
>>>> On 07 Mar 2016, at 09:40, Ken Gaillot <kgail...@redhat.com> wrote:
>>>>
>>>> On 03/06/2016 07:43 PM, Сергей Филатов wrote:
>>>>> Hi,
>>>>> I’m trying to set up pacemaker_remote resource on ubuntu 14.04
>>>>> I followed "remote node walkthrough” guide 
>>>>> (http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html-single/Pacemaker_Remote/#idm140473081667280
>>>>>  
>>>>> <http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html-single/Pacemaker_Remote/#idm140473081667280>)
>>>>> After creating ocf:pacemaker:remote resource on cluster node, remote node 
>>>>> doesn’t show up as online.
>>>>> I guess I need to configure remote agent to listen on ipv4, where can I 
>>>>> configure it?
>>>>> Or is there any other steps to set up remote node besides the ones 
>>>>> mentioned in guide?
>

Re: [ClusterLabs] FLoating IP failing over but not failing back with active/active LDAP (dirsrv)

[Ken Gaillot]

On 03/10/2016 09:38 AM, Bernie Jones wrote:
> Hi Ken,
> Thanks for your response, I've now corrected the constraint order but the
> behaviour is still the same, the IP does not fail over (after the first
> time) unless I issue a pcs resource cleanup command on dirsrv-daemon.
> 
> Also, I'm not sure why you advise against using is-managed=false in
> production. We are trying to use pacemaker purely to fail over on detection
> of a failure and not to control starting or stopping of the instances. It is
> essential that in normal operation we have both instances up as we are using
> MMR.
> 
> Thanks,
> Bernie

I think you misunderstand is-managed. It is used to be able to perform
maintenance on a service without pacemaker fencing the node when the
service is stopped/restarted. Failover won't work with is-managed=false,
because failover involves stopping and starting the service.

Your goal is already accomplished by using a clone with master-max=2.
With the clone, pacemaker will run the service on both nodes, and with
master-max=2, it will be master/master.

> -Original Message-
> From: Ken Gaillot [mailto:kgail...@redhat.com] 
> Sent: 10 March 2016 15:01
> To: users@clusterlabs.org
> Subject: Re: [ClusterLabs] FLoating IP failing over but not failing back
> with active/active LDAP (dirsrv)
> 
> On 03/10/2016 08:48 AM, Bernie Jones wrote:
>> A bit more info..
>>
>>  
>>
>> If, after I restart the failed dirsrv instance, I then perform a "pcs
>> resource cleanup dirsrv-daemon" to clear the FAIL messages then the
> failover
>> will work OK.
>>
>> So it's as if the cleanup is changing the status in some way..
>>
>>  
>>
>> From: Bernie Jones [mailto:ber...@securityconsulting.ltd.uk] 
>> Sent: 10 March 2016 08:47
>> To: 'Cluster Labs - All topics related to open-source clustering welcomed'
>> Subject: [ClusterLabs] FLoating IP failing over but not failing back with
>> active/active LDAP (dirsrv)
>>
>>  
>>
>> Hi all, could you advise please?
>>
>>  
>>
>> I'm trying to configure a floating IP with an active/active deployment of
>> 389 directory server. I don't want pacemaker to manage LDAP but just to
>> monitor and switch the IP as required to provide resilience. I've seen
> some
>> other similar threads and based my solution on those.
>>
>>  
>>
>> I've amended the ocf for slapd to work with 389 DS and this tests out OK
>> (dirsrv).
>>
>>  
>>
>> I've then created my resources as below:
>>
>>  
>>
>> pcs resource create dirsrv-ip ocf:heartbeat:IPaddr2 ip="192.168.26.100"
>> cidr_netmask="32" op monitor timeout="20s" interval="5s" op start
>> interval="0" timeout="20" op stop interval="0" timeout="20"
>>
>> pcs resource create dirsrv-daemon ocf:heartbeat:dirsrv op monitor
>> interval="10" timeout="5" op start interval="0" timeout="5" op stop
>> interval="0" timeout="5" meta "is-managed=false"
> 
> is-managed=false means the cluster will not try to start or stop the
> service. It should never be used in regular production, only when doing
> maintenance on the service.
> 
>> pcs resource clone dirsrv-daemon meta globally-unique="false"
>> interleave="true" target-role="Started" "master-max=2"
>>
>> pcs constraint colocation add dirsrv-daemon-clone with dirsrv-ip
>> score=INFINITY
> 
> This constraint means that dirsrv is only allowed to run where dirsrv-ip
> is. I suspect you want the reverse, dirsrv-ip with dirsrv-daemon-clone,
> which means keep the IP with a working dirsrv instance.
> 
>> pcs property set no-quorum-policy=ignore
> 
> If you're using corosync 2, you generally don't need or want this.
> Instead, ensure corosync.conf has two_node: 1 (which will be done
> automatically if you used pcs cluster setup).
> 
>> pcs resource defaults migration-threshold=1
>>
>> pcs property set stonith-enabled=false
>>
>>  
>>
>> On startup all looks well:
>>
>>
> 
>> 
>>
>>  
>>
>> Last updated: Thu Mar 10 08:28:03 2016
>>
>> Last change: Thu Mar 10 08:26:14 2016
>>
>> Stack: cman
>>
>> Current DC: ga2.idam.com - partition with quorum
>>
>> Version: 1.1.11-97629de
>>
>> 2 Nodes configured
>>
>> 3 Resources configured
>

Re: [ClusterLabs] FLoating IP failing over but not failing back with active/active LDAP (dirsrv)

[Ken Gaillot]

On 03/10/2016 08:48 AM, Bernie Jones wrote:
> A bit more info..
> 
>  
> 
> If, after I restart the failed dirsrv instance, I then perform a "pcs
> resource cleanup dirsrv-daemon" to clear the FAIL messages then the failover
> will work OK.
> 
> So it's as if the cleanup is changing the status in some way..
> 
>  
> 
> From: Bernie Jones [mailto:ber...@securityconsulting.ltd.uk] 
> Sent: 10 March 2016 08:47
> To: 'Cluster Labs - All topics related to open-source clustering welcomed'
> Subject: [ClusterLabs] FLoating IP failing over but not failing back with
> active/active LDAP (dirsrv)
> 
>  
> 
> Hi all, could you advise please?
> 
>  
> 
> I'm trying to configure a floating IP with an active/active deployment of
> 389 directory server. I don't want pacemaker to manage LDAP but just to
> monitor and switch the IP as required to provide resilience. I've seen some
> other similar threads and based my solution on those.
> 
>  
> 
> I've amended the ocf for slapd to work with 389 DS and this tests out OK
> (dirsrv).
> 
>  
> 
> I've then created my resources as below:
> 
>  
> 
> pcs resource create dirsrv-ip ocf:heartbeat:IPaddr2 ip="192.168.26.100"
> cidr_netmask="32" op monitor timeout="20s" interval="5s" op start
> interval="0" timeout="20" op stop interval="0" timeout="20"
> 
> pcs resource create dirsrv-daemon ocf:heartbeat:dirsrv op monitor
> interval="10" timeout="5" op start interval="0" timeout="5" op stop
> interval="0" timeout="5" meta "is-managed=false"

is-managed=false means the cluster will not try to start or stop the
service. It should never be used in regular production, only when doing
maintenance on the service.

> pcs resource clone dirsrv-daemon meta globally-unique="false"
> interleave="true" target-role="Started" "master-max=2"
> 
> pcs constraint colocation add dirsrv-daemon-clone with dirsrv-ip
> score=INFINITY

This constraint means that dirsrv is only allowed to run where dirsrv-ip
is. I suspect you want the reverse, dirsrv-ip with dirsrv-daemon-clone,
which means keep the IP with a working dirsrv instance.

> pcs property set no-quorum-policy=ignore

If you're using corosync 2, you generally don't need or want this.
Instead, ensure corosync.conf has two_node: 1 (which will be done
automatically if you used pcs cluster setup).

> pcs resource defaults migration-threshold=1
> 
> pcs property set stonith-enabled=false
> 
>  
> 
> On startup all looks well:
> 
> 
> 
> 
>  
> 
> Last updated: Thu Mar 10 08:28:03 2016
> 
> Last change: Thu Mar 10 08:26:14 2016
> 
> Stack: cman
> 
> Current DC: ga2.idam.com - partition with quorum
> 
> Version: 1.1.11-97629de
> 
> 2 Nodes configured
> 
> 3 Resources configured
> 
>  
> 
>  
> 
> Online: [ ga1.idam.com ga2.idam.com ]
> 
>  
> 
> dirsrv-ip   (ocf::heartbeat:IPaddr2): Started ga1.idam.com
> 
>  Clone Set: dirsrv-daemon-clone [dirsrv-daemon]
> 
>  dirsrv-daemon  (ocf::heartbeat:dirsrv):Started ga2.idam.com
> (unmanaged)
> 
>  dirsrv-daemon  (ocf::heartbeat:dirsrv):Started ga1.idam.com
> (unmanaged)
> 
>  
> 
>  
> 
> 
> 
> 
>  
> 
> Stop dirsrv on ga1:
> 
>  
> 
> Last updated: Thu Mar 10 08:28:43 2016
> 
> Last change: Thu Mar 10 08:26:14 2016
> 
> Stack: cman
> 
> Current DC: ga2.idam.com - partition with quorum
> 
> Version: 1.1.11-97629de
> 
> 2 Nodes configured
> 
> 3 Resources configured
> 
>  
> 
>  
> 
> Online: [ ga1.idam.com ga2.idam.com ]
> 
>  
> 
> dirsrv-ip   (ocf::heartbeat:IPaddr2): Started ga2.idam.com
> 
>  Clone Set: dirsrv-daemon-clone [dirsrv-daemon]
> 
>  dirsrv-daemon  (ocf::heartbeat:dirsrv):Started ga2.idam.com
> (unmanaged)
> 
>  dirsrv-daemon  (ocf::heartbeat:dirsrv):FAILED ga1.idam.com
> (unmanaged)
> 
>  
> 
> Failed actions:
> 
> dirsrv-daemon_monitor_1 on ga1.idam.com 'not running' (7): call=12,
> status=complete, last-rc-change='Thu Mar 10 08:28:41 2016', queued=0ms,
> exec=0ms
> 
>  
> 
> IP fails over to ga2 OK:
> 
>  
> 
> 
> 
> 
>  
> 
> Restart dirsrv on ga1
> 
>  
> 
> Last updated: Thu Mar 10 08:30:01 2016
> 
> Last change: Thu Mar 10 08:26:14 2016
> 
> Stack: cman
> 
> Current DC: ga2.idam.com - partition with quorum
> 
> Version: 1.1.11-97629de
> 
> 2 Nodes configured
> 
> 3 Resources configured
> 
>  
> 
>  
> 
> Online: [ ga1.idam.com ga2.idam.com ]
> 
>  
> 
> dirsrv-ip   (ocf::heartbeat:IPaddr2): Started ga2.idam.com
> 
>  Clone Set: dirsrv-daemon-clone [dirsrv-daemon]
> 
>  dirsrv-daemon  (ocf::heartbeat:dirsrv):Started ga2.idam.com
> (unmanaged)
> 
>  dirsrv-daemon  (ocf::heartbeat:dirsrv):Started ga1.idam.com
> (unmanaged)
> 
>  
> 
> Failed actions:
> 
> dirsrv-daemon_monitor_1 on ga1.idam.com 'not running' 

Re: [ClusterLabs] Stonith ignores resource stop errors

[Ken Gaillot]

On 03/10/2016 04:42 AM, Klechomir wrote:
> Hi List
> 
> I'm testing stonith now (pacemaker 1.1.8), and noticed that it properly kills 
> a node with stopped pacemaker, but ignores resource stop errors.
> 
> I'm pretty sure that the same version worked properly with stonith before.
> Maybe I'm missing some setting?
> 
> Rgards,
> Klecho

The only setting that should be relevant there is on-fail for the
resource's stop operation, which defaults to fence but can be set to
other actions.

That said, 1.1.8 is pretty old at this point, so I'm not sure of its
behavior.


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] ClusterIP location constraint reappears after reboot

[Ken Gaillot]

 Feb 21 23:10:42 g5se-dea2b1 crmd[1562]:   notice: process_lrm_event: LRM 
> operation ClusterIP_start_0 (call=68, rc=0, cib-update=76, confirmed=true) ok
> Feb 21 23:10:42 g5se-dea2b1 crmd[1562]:   notice: te_rsc_command: Initiating 
> action 7: monitor ClusterIP_monitor_3 on g5se-dea2b1 (local)
> Feb 21 23:10:42 g5se-dea2b1 crmd[1562]:   notice: process_lrm_event: LRM 
> operation ClusterIP_monitor_3 (call=71, rc=0, cib-update=77, 
> confirmed=false) ok
> Feb 21 23:10:42 g5se-dea2b1 crmd[1562]:   notice: run_graph: Transition 36 
> (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0, 
> Source=/var/lib/pacemaker/pengine/pe-input-138.bz2): Complete
> Feb 21 23:10:42 g5se-dea2b1 crmd[1562]:   notice: do_state_transition: State 
> transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS 
> cause=C_FSA_INTERNAL origin=notify_crmd ]
> Feb 21 23:10:43 g5se-dea2b1 crmd[1562]:   notice: do_state_transition: State 
> transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL 
> origin=abort_transition_graph ]
> Feb 21 23:10:43 g5se-dea2b1 stonith-ng[1558]:   notice: unpack_config: On 
> loss of CCM Quorum: Ignore
> Feb 21 23:10:43 g5se-dea2b1 cib[1557]:   notice: cib:diff: Diff: --- 0.71.3
> Feb 21 23:10:43 g5se-dea2b1 cib[1557]:   notice: cib:diff: Diff: +++ 0.72.1 
> 4e5a3b6259a59f84bcfec6d0f16ad3ba
> Feb 21 23:10:43 g5se-dea2b1 cib[1557]:   notice: cib:diff: --  admin_epoch="0" epoch="71" num_updates="3"/>
> Feb 21 23:10:43 g5se-dea2b1 cib[1557]:   notice: cib:diff: ++   
>  role="Started" node="g5se-dea2b1" score="-INFINITY"/>
> Feb 21 23:10:43 g5se-dea2b1 pengine[1561]:   notice: unpack_config: On loss 
> of CCM Quorum: Ignore
> Feb 21 23:10:43 g5se-dea2b1 pengine[1561]:   notice: LogActions: Stop
> ClusterIP#011(g5se-dea2b1)
> Feb 21 23:10:43 g5se-dea2b1 pengine[1561]:   notice: process_pe_message: 
> Calculated Transition 37: /var/lib/pacemaker/pengine/pe-input-139.bz2
> Feb 21 23:10:43 g5se-dea2b1 crmd[1562]:   notice: te_rsc_command: Initiating 
> action 7: stop ClusterIP_stop_0 on g5se-dea2b1 (local)
> Feb 21 23:10:43 g5se-dea2b1 IPaddr2[13372]: INFO: IP status = ok, IP_CIP=
> Feb 21 23:10:43 g5se-dea2b1 crmd[1562]:   notice: process_lrm_event: LRM 
> operation ClusterIP_stop_0 (call=75, rc=0, cib-update=79, confirmed=true) ok
> Feb 21 23:10:43 g5se-dea2b1 crmd[1562]:   notice: run_graph: Transition 37 
> (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0, 
> Source=/var/lib/pacemaker/pengine/pe-input-139.bz2): Complete
> Feb 21 23:10:43 g5se-dea2b1 crmd[1562]:   notice: do_state_transition: State 
> transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS 
> cause=C_FSA_INTERNAL origin=notify_crmd ]
> Feb 21 23:10:43 g5se-dea2b1 stonith-ng[1558]:   notice: unpack_config: On 
> loss of CCM Quorum: Ignore
> Feb 21 23:10:43 g5se-dea2b1 cib[1557]:   notice: cib:diff: Diff: --- 0.72.2
> Feb 21 23:10:43 g5se-dea2b1 cib[1557]:   notice: cib:diff: Diff: +++ 0.73.1 
> 93f902fd51a6750b828144d42f8c7a6e
> Feb 21 23:10:43 g5se-dea2b1 cib[1557]:   notice: cib:diff: --   
>  role="Started" node="g5se-dea2b1" score="-INFINITY"/>
> Feb 21 23:10:43 g5se-dea2b1 cib[1557]:   notice: cib:diff: ++  num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2" 
> cib-last-written="Sun Feb 21 23:10:43 2016" update-origin="g5se-dea2b1" 
> update-client="crm_resource" crm_feature_set="3.0.7" have-quorum="1" 
> dc-uuid="g5se-dea2b1"/>
> Feb 21 23:10:43 g5se-dea2b1 crmd[1562]:   notice: do_state_transition: State 
> transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL 
> origin=abort_transition_graph ]
> Feb 21 23:10:43 g5se-dea2b1 pengine[1561]:   notice: unpack_config: On loss 
> of CCM Quorum: Ignore
> Feb 21 23:10:43 g5se-dea2b1 pengine[1561]:   notice: LogActions: Start   
> ClusterIP#011(g5se-dea2b1)
> Feb 21 23:10:43 g5se-dea2b1 pengine[1561]:   notice: process_pe_message: 
> Calculated Transition 38: /var/lib/pacemaker/pengine/pe-input-140.bz2
> Feb 21 23:10:43 g5se-dea2b1 crmd[1562]:   notice: te_rsc_command: Initiating 
> action 6: start ClusterIP_start_0 on g5se-dea2b1 (local)
> 
> 
> 
> -Original Message-
> From: users-requ...@clusterlabs.org [mailto:users-requ...@clusterlabs.org] 
> Sent: Monday, February 22, 2016 11:42 AM
> To: users@clusterlabs.org
> Subject: Users Digest, Vol 13, Issue 44
> 
> Send Users mailing list submissions to
>   users@clusterlabs.org
> 
> To subscribe or unsubscribe via the World Wide Web, visit
>   http://clusterlabs.org/mailman/listinfo/users
> or, via email, send a me

Re: [ClusterLabs] Cluster failover failure with Unresolved dependency

[Ken Gaillot]

engine: info:
> LogActions:   Leave   fs-spool(Started mail1)
> Mar 16 11:38:06 [7419] HWJ-626.domain.localpengine: info:
> LogActions:   Leave   fs-mail (Started mail1)
> Mar 16 11:38:06 [7419] HWJ-626.domain.localpengine:   notice:
> LogActions:   Stoppostfix (mail1)
> Mar 16 11:38:06 [7420] HWJ-626.domain.local   crmd: info:
> do_state_transition:  State transition S_POLICY_ENGINE ->
> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE
> origin=handle_response ]
> Mar 16 11:38:06 [7419] HWJ-626.domain.localpengine:   notice:
> process_pe_message:   Calculated Transition 2964:
> /var/lib/pacemaker/pengine/pe-input-331.bz2
> Mar 16 11:38:06 [7420] HWJ-626.domain.local   crmd: info:
> do_te_invoke: Processing graph 2964 (ref=pe_calc-dc-1458124686-5542)
> derived from /var/lib/pacemaker/pengine/pe-input-331.bz2
> Mar 16 11:38:06 [7420] HWJ-626.domain.local   crmd:   notice:
> te_rsc_command:   Initiating action 5: stop postfix_stop_0 on mail1
> Mar 16 11:38:07 [7415] HWJ-626.domain.localcib: info:
> cib_perform_op:   Diff: --- 0.215.10 2
> Mar 16 11:38:07 [7415] HWJ-626.domain.localcib: info:
> cib_perform_op:   Diff: +++ 0.215.11 (null)
> Mar 16 11:38:07 [7415] HWJ-626.domain.localcib: info:
> cib_perform_op:   +  /cib:  @num_updates=11
> Mar 16 11:38:07 [7415] HWJ-626.domain.localcib: info:
> cib_perform_op:   +
>  
> /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='postfix']/lrm_rsc_op[@id='postfix_last_0']:
>  @operation_key=postfix_stop_0, @operation=stop,
> @transition-key=5:2964:0:ae755a85-c250-498f-9c94-ddd8a7e2788a,
> @transition-magic=0:0;5:2964:0:ae755a85-c250-498f-9c94-ddd8a7e2788a,
> @call-id=1335, @last-run=1458124686, @last-rc-change=1458124686,
> @exec-time=435
> Mar 16 11:38:07 [7420] HWJ-626.domain.local   crmd: info:
> match_graph_event:Action postfix_stop_0 (5) confirmed on mail1 (rc=0)
> Mar 16 11:38:07 [7415] HWJ-626.domain.localcib: info:
> cib_process_request:  Completed cib_modify operation for section status: OK
> (rc=0, origin=mail1/crmd/254, version=0.215.11)
> Mar 16 11:38:07 [7420] HWJ-626.domain.local   crmd:   notice:
> te_rsc_command:   Initiating action 12: stop virtualip-1_stop_0 on mail1
> Mar 16 11:38:07 [7415] HWJ-626.domain.localcib: info:
> cib_perform_op:   Diff: --- 0.215.11 2
> Mar 16 11:38:07 [7415] HWJ-626.domain.localcib: info:
> cib_perform_op:   Diff: +++ 0.215.12 (null)
> Mar 16 11:38:07 [7415] HWJ-626.domain.localcib: info:
> cib_perform_op:   +  /cib:  @num_updates=12
> Mar 16 11:38:07 [7415] HWJ-626.domain.localcib: info:
> cib_perform_op:   +
>  
> /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='virtualip-1']/lrm_rsc_op[@id='virtualip-1_last_0']:
>  @operation_key=virtualip-1_stop_0, @operation=stop,
> @transition-key=12:2964:0:ae755a85-c250-498f-9c94-ddd8a7e2788a,
> @transition-magic=0:0;12:2964:0:ae755a85-c250-498f-9c94-ddd8a7e2788a,
> @call-id=1337, @last-run=1458124687, @last-rc-change=1458124687,
> @exec-time=56
> Mar 16 11:38:07 [7420] HWJ-626.domain.local   crmd: info:
> match_graph_event:Action virtualip-1_stop_0 (12) confirmed on mail1
> (rc=0)
> Mar 16 11:38:07 [7415] HWJ-626.domain.localcib: info:
> cib_process_request:  Completed cib_modify operation for section status: OK
> (rc=0, origin=mail1/crmd/255, version=0.215.12)
> Mar 16 11:38:07 [7420] HWJ-626.domain.local   crmd:   notice:
> run_graph:Transition 2964 (Complete=7, Pending=0, Fired=0, Skipped=0,
> Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-331.bz2): Complete
> Mar 16 11:38:07 [7420] HWJ-626.domain.local   crmd: info: do_log:
> FSA: Input I_TE_SUCCESS from notify_crmd() received in state
> S_TRANSITION_ENGINE
> Mar 16 11:38:07 [7420] HWJ-626.domain.local   crmd:   notice:
> do_state_transition:  State transition S_TRANSITION_ENGINE -> S_IDLE [
> input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
> Mar 16 11:38:12 [7415] HWJ-626.domain.localcib: info:
> cib_process_ping: Reporting our current digest to mail2:
> ed43bc3ecf0f15853900ba49fc514870 for 0.215.12 (0x152b110 0)
> 
> 
> On Mon, Mar 14, 2016 at 6:44 PM, Ken Gaillot <kgail...@redhat.com> wrote:
> 
>> On 03/10/2016 09:49 AM, Lorand Kelemen wrote:
>>> Dear List,
>>>
>>> After the creation and testing of a simple 2 node active-passive
>>> drbd+postfix cluster nearly everything works flawlessly (standby, failure
>>> of a filesystem resource + failover, splitbrain + manual recovery)
>> however

Re: [ClusterLabs] Unable to create HAProxy resource: no such resource agent

[Ken Gaillot]

On 03/11/2016 02:18 PM, Matthew Mucker wrote:
> I've created a Pacemaker cluster and have created a virtual IP address 
> resource that works properly. I am now attempting to add HAProxy as a 
> resource and I'm having problems.
> 
> 
> - I installed HAProxy on all nodes of the cluster
> 
> - I downloaded http://github.com/russki/cluster-agents/raw/master/haproxy to 
> /usr/lib/ocf/resource.d/heartbeat/haproxy on each node
> 
> - I ran chmod 755 on /usr/lib/ocf/resource.d/heartbeat/haproxy on each node
> 
> - I configured HAProxy.cfg on each node
> 
> - I edited /etc/default/haproxy to enable haproxy

FYI, this file will be ignored when the service is managed by the
cluster (unless the RA specifically reads it, which I've rarely seen).
That won't cause any problems, but any desired settings should be made
in the resource's cluster configuration rather than here.

> - I've tested and confirmed that HAProxy will start as a service on the node 
> hosting the virtual IP address

So far, so good. Good prep work.

> 
> However, when I run the command:
> 
> 
> crm configure primitive haproxy ocf:heartbeat:haproxy op monitor interval=15s
> 
> 
> I get output:
> 
> 
> ERROR: None
> ERROR: ocf:heartbeat:haproxy: meta-data contains no resource-agent element
> ERROR: None
> ERROR: ocf:heartbeat:haproxy: meta-data contains no resource-agent element
> ERROR: ocf:heartbeat:haproxy: no such resource agent
> 
> 
> I've been unable to find a solution to this problem in the searching I've 
> done online. Does anyone have any idea what the cause and the solution to 
> this problem might be?

What does this command return:

crm_resource --show-metadata ocf:heartbeat:haproxy

> 
> Thanks,
> 
> 
> -Matthew


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Unable to create HAProxy resource: no such resource agent

[Ken Gaillot]

On 03/11/2016 03:25 PM, Matthew Mucker wrote:
> I found the problem. When I used wget to retrieve the file, I was actually 
> downloading an HTML error page from my proxy server instead of the intended 
> file.
> 
> 
> Oops.

:-) I've done that before too ...

> 
> 
> 
> 
> I've created a Pacemaker cluster and have created a virtual IP address 
> resource that works properly. I am now attempting to add HAProxy as a 
> resource and I'm having problems.
> 
> 
> - I installed HAProxy on all nodes of the cluster
> 
> - I downloaded http://github.com/russki/cluster-agents/raw/master/haproxy to 
> /usr/lib/ocf/resource.d/heartbeat/haproxy on each node
> 
> - I ran chmod 755 on /usr/lib/ocf/resource.d/heartbeat/haproxy on each node
> 
> - I configured HAProxy.cfg on each node
> 
> - I edited /etc/default/haproxy to enable haproxy
> 
> - I've tested and confirmed that HAProxy will start as a service on the node 
> hosting the virtual IP address
> 
> 
> However, when I run the command:
> 
> 
> crm configure primitive haproxy ocf:heartbeat:haproxy op monitor interval=15s
> 
> 
> I get output:
> 
> 
> ERROR: None
> ERROR: ocf:heartbeat:haproxy: meta-data contains no resource-agent element
> ERROR: None
> ERROR: ocf:heartbeat:haproxy: meta-data contains no resource-agent element
> ERROR: ocf:heartbeat:haproxy: no such resource agent
> 
> 
> I've been unable to find a solution to this problem in the searching I've 
> done online. Does anyone have any idea what the cause and the solution to 
> this problem might be?
> 
> 
> Thanks,
> 
> 
> -Matthew


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Problems with pcs/corosync/pacemaker/drbd/vip/nfs

[Ken Gaillot]

On 03/14/2016 12:47 PM, Todd Hebert wrote:
> Hello,
> 
> I'm working on setting up a test-system that can handle NFS failover.
> 
> The base is CentOS 7.
> I'm using ZVOL block devices out of ZFS to back DRBD replicated volumes.
> 
> I have four DRBD resources (r0, r1, r2, r3, which are /dev/drbd1 drbd2 drbd3 
> and drbd4 respectively)
> 
> These all have XFS filesystems on them that mount properly and serve content 
> etc..
> 
> I tried using corosync/pacemaker/drbd/lsb:nfs-kernel-server on Ubuntu, and it 
> would serve content on the primary server without issue.  Any attempt to 
> failover or migrate services.. .everything would say it migrated fine, and 
> the filesystems would be mounted and readable/writable etc.., but the NFS 
> clients access to them would "pause"
> 
> This appears to be an issue with the nfs-kernel-server in Ubuntu, where it 
> simply would not recognise the NFS session information, which was on a 
> replicated volume.
> 
> If the primary node is put back online, everything migrates back "perfect" 
> and traffic that had "paused" on failover to the secondary system resumes, 
> even if it's been sitting there for 15-20 minutes not working.
> 
> There is no difference in behaviour between offlining the primary node, and 
> migrating lsb:nfs-kernel-server to another node (by it's primitive name, not 
> as lsb:nfs-kernel-server, obviously)
> 
> If I create new connections into NFS while test-sanb is active, they work, 
> only to "freeze" as above with an offline or migrate away from test-sanb, so 
> symptoms are the same in both "directions"
> 
> 
> 
> After not being able to get the lsb:nfs-kernel-server working properly in 
> Ubuntu, and reading similar stories from other users after a series of 
> googles, I switched over to CentOS 7.
> CentOS 7, instead of lsb:nfs-kernel-server, I am trying to use 
> systemd:nfs-server, since CentOS 7 uses systemd, rather than sysinitv for 
> managing services.

I'm not very familiar with NFS in a cluster, but there is an
ocf:heartbeat:nfsserver resource agent in the resource-agents package.
OCF agents are generally preferable to lsb/systemd because they give
more detailed information to the cluster, and it looks like in this
case, the RA does some RPC commands that the system scripts don't.

I'd give it a shot and see if it helps.

> Pretty much everything in the configuration except lsb:nfs-kernel-server came 
> right over.
> 
> Now, everything will run properly on the primary node (as was the case with 
> Ubuntu) but...
> If I put the "test-sana" node into standby, first NFS stops, then the VIP 
> stops, then the three NFS-shared filesystems get umounted (perfect so far)
> Then.. it appears that parts of the NFS service, either idmapd or rpcbind 
> haven't released their hold on the rpc_pipefs filesystem, so it's still 
> mounted... it's mounted inside /var/lib/nfs, which is on the last drbd volume.
> Pacemaker, or some other element detects that rpc_pipefs was still mounted, 
> umounts it, then umounts /var/lib/nfs, which should clear the way for 
> everything else to work.. but that's not what happens.
> 
> At this point, the ms_drbd_r should demote to "Secondary" on the primary 
> mode, allowing for the secondary node to promote to "Primary" and services to 
> start on "test-sanb", but instead the drbd processes on "test-sana" end up 
> marked as "Stopped" and checking `cat /proc/drbd` shows that the volumes are 
> still Primary/Secondary UpToDate/UpToDate on test-sana (and the opposite on 
> test-sanb)
> 
> It takes AGES (several minutes) for things to reach this state.
> 
> They stay this way indefinitely.  If I manually demote DRBD resources on 
> test-sana, they end up listed as "Master" in a "crm status" or "pcs status" 
> again, and eventually the status changes to "Primary/Secondary" in /proc/drbd 
> as well.
> 
> If I put node test-sana back online (node online test-sana) it takes a few 
> seconds for services to start back up and serve content again.
> 
> Since I cannot get services to run on test-sanb at all thus far, I don't know 
> if the symptoms would be the same in both directions.
> I can't find any differences in the two nodes that should account for this.
> 
> ---
> 
> In any case, what I need to arrive at is a working solution for NFS failure 
> across two nodes.
> 
> I have several systems where I'm using just heartbeat in order to failover IP 
> address, drbd, and nfs, but for single instances of drbd/nfs
> 
> I cannot find any working examples for either Ubuntu 14.04, nor CentOS 7 for 
> this scenario.  (There are some out there for Ubuntu, but they do not appear 
> to actually work with modern pacemaker et al.)
> 
> Does anyone have an example of working configurations for this?
> 
> My existing pacemaker configuration can be found here:  
> http://paste.ie/view/c766a4ff
> 
> As I mentioned, the configurations are nearly identical for both the Ubuntu 
> 14.04 and CentOS 7 setups, and the hardware used is the same 

Re: [ClusterLabs] pacemaker remote configuration on ubuntu 14.04

[Ken Gaillot]

On 03/10/2016 11:36 PM, Сергей Филатов wrote:
> This one is the right log

Something in the cluster configuration and state (for example, an
unsatisfied constraint) is preventing the cluster from starting the
resource:

Mar 10 04:00:53 [11785] controller-1.domain.compengine: info:
native_print: compute-1   (ocf::pacemaker:remote):Stopped
Mar 10 04:00:53 [11785] controller-1.domain.compengine: info:
native_color: Resource compute-1 cannot run anywhere


> 
> 
> 
>> On 10 Mar 2016, at 08:17, Сергей Филатов <filat...@gmail.com 
>> <mailto:filat...@gmail.com>> wrote:
>>
>> pcs resource show compute-1
>>
>>  Resource: compute-1 (class=ocf provider=pacemaker type=remote)
>>  Operations: monitor interval=60s (compute-1-monitor-interval-60s)
>>
>> Can’t find _start_0 template in pacemaker logs
>> I don’t have ipv6 address for remote node, but I guess it should be 
>> listening 
>> on both
>>
>> attached pacemaker.log for cluster node
>> 
>>
>>
>>> On 09 Mar 2016, at 10:23, Ken Gaillot <kgail...@redhat.com 
>>> <mailto:kgail...@redhat.com>> wrote:
>>>
>>> On 03/08/2016 11:38 PM, Сергей Филатов wrote:
>>>> ssh -p 3121 compute-1
>>>> ssh_exchange_identification: read: Connection reset by peer
>>>>
>>>> That’s what I get in /var/log/pacemaker.log after restarting 
>>>> pacemaker_remote:
>>>> Mar 09 05:30:27 [28031] compute-1.domain.com 
>>>> <http://compute-1.domain.com/> 
>>>>   lrmd: info: crm_signal_dispatch:  Invoking handler for signal 
>>>> 15: 
>>>> Terminated
>>>> Mar 09 05:30:27 [28031] compute-1.domain.com 
>>>> <http://compute-1.domain.com/> 
>>>>   lrmd: info: lrmd_shutdown:Terminating with  0 clients
>>>> Mar 09 05:30:27 [28031] compute-1.domain.com 
>>>> <http://compute-1.domain.com/> 
>>>>   lrmd: info: qb_ipcs_us_withdraw:  withdrawing server sockets
>>>> Mar 09 05:30:27 [28031] compute-1.domain.com 
>>>> <http://compute-1.domain.com/> 
>>>>   lrmd: info: crm_xml_cleanup:  Cleaning up memory from libxml2
>>>> Mar 09 05:30:27 [28193] compute-1.domain.com 
>>>> <http://compute-1.domain.com/> 
>>>>   lrmd: info: crm_log_init: Changed active directory to 
>>>> /var/lib/heartbeat/cores/root
>>>> Mar 09 05:30:27 [28193] compute-1.domain.com 
>>>> <http://compute-1.domain.com/> 
>>>>   lrmd: info: qb_ipcs_us_publish:   server name: lrmd
>>>> Mar 09 05:30:27 [28193] compute-1.domain.com 
>>>> <http://compute-1.domain.com/> 
>>>>   lrmd:   notice: lrmd_init_remote_tls_server:  Starting a tls 
>>>> listener 
>>>> on port 3121.
>>>> Mar 09 05:30:28 [28193] compute-1.domain.com 
>>>> <http://compute-1.domain.com/> 
>>>>   lrmd:   notice: bind_and_listen:  Listening on address ::
>>>> Mar 09 05:30:28 [28193] compute-1.domain.com 
>>>> <http://compute-1.domain.com/> 
>>>>   lrmd: info: qb_ipcs_us_publish:   server name: cib_ro
>>>> Mar 09 05:30:28 [28193] compute-1.domain.com 
>>>> <http://compute-1.domain.com/> 
>>>>   lrmd: info: qb_ipcs_us_publish:   server name: cib_rw
>>>> Mar 09 05:30:28 [28193] compute-1.domain.com 
>>>> <http://compute-1.domain.com/> 
>>>>   lrmd: info: qb_ipcs_us_publish:   server name: cib_shm
>>>> Mar 09 05:30:28 [28193] compute-1.domain.com 
>>>> <http://compute-1.domain.com/> 
>>>>   lrmd: info: qb_ipcs_us_publish:   server name: attrd
>>>> Mar 09 05:30:28 [28193] compute-1.domain.com 
>>>> <http://compute-1.domain.com/> 
>>>>   lrmd: info: qb_ipcs_us_publish:   server name: stonith-ng
>>>> Mar 09 05:30:28 [28193] compute-1.domain.com 
>>>> <http://compute-1.domain.com/> 
>>>>   lrmd: info: qb_ipcs_us_publish:   server name: crmd
>>>> Mar 09 05:30:28 [28193] compute-1.domain.com 
>>>> <http://compute-1.domain.com/> 
>>>>   lrmd: info: main: Starting
>>>
>>> It looks like the cluster is not even trying to connect to the remote
>>> node. pacemaker_remote here is binding only to IPv6, so the cluster will
>>> need to contact it on that address.
>>>
>&g

Re: [ClusterLabs] PCS, Corosync, Pacemaker, and Bind

[Ken Gaillot]

On 03/15/2016 06:47 PM, Mike Bernhardt wrote:
> Not sure if this is a BIND question or a PCS/Corosync question, but
> hopefully someone has done this before:
> 
>  
> 
> I'm setting up a new CentOS 7 DNS server cluster to replace our very old
> CentOS 4 cluster. The old one uses heartbeat which is no longer supported,
> so I'm now using pcs, corosync, and pacemaker.  The new one is running the
> latest 9.10.x production release of BIND. I want BIND to listen on, query
> from, etc on a particular IP address, which is virtualized with pacemaker. 
> 
>  
> 
> This worked fine on the old cluster. But whereas heartbeat would create a
> virtual subinterface (i.e. eth0:0) to support the virtual IP, corosync does
> not do that; at least it doesn't by default. So although the virtual IP
> exists and is pingable, it is not tied to a "physical" interface- ifconfig
> does not find it. And when BIND tries to start up, it fails because it can't
> find the virtual IP it's configured to run on, even though it is reachable.
> I only need IPv4, not IPv6.

The old subinterfaces are no longer neded in linux-land for "virtual"
IPs, which are now actually full-class citizens, just one of multiple
IPs assigned to the interface. ifconfig (or its fashionably new
alternative, "ip addr") should show both addresses on the same interface.

BIND shouldn't have any problems finding the IP. Can you show the error
messages that come up, and your pacemaker configuration?

>  
> 
> So, I'm hoping that there is a way to tell corosync (hopefully using pcsd)
> to create a virtual interface, not just a virtual address, so BIND can find
> it.
> 
>  
> 
> Thanks in advance!


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Cluster failover failure with Unresolved dependency

[Ken Gaillot]

On 03/16/2016 11:20 AM, Lorand Kelemen wrote:
> Dear Ken,
> 
> I already modified the startup as suggested during testing, thanks! I
> swapped the postfix ocf resource to the amavisd systemd resource, as latter
> controls postfix startup also as it turns out and having both resouces in
> the mail-services group causes conflicts (postfix is detected as not
> running).
> 
> Still experiencing the same behaviour, killing amavisd returns an rc=7 for
> the monitoring operation on the "victim" node, this soungs logical, but the
> logs contain the same: amavisd and virtualip cannot run anywhere.
> 
> I made sure systemd is clean (amavisd = inactive, not running instead of
> failed) and also reset the failcount on all resources before killing
> amavisd.
> 
> How can I make sure to have a clean state for the resources beside above
> actions?

What you did is fine. I'm not sure why amavisd and virtualip can't run.
Can you show the output of "cibadmin -Q" when the cluster is in that state?

> Also note: when causing a filesystem resource to fail (e.g. with unmout),
> the failover happens successfully, all resources are started on the
> "survivor" node.
> 
> Best regards,
> Lorand
> 
> 
> On Wed, Mar 16, 2016 at 4:34 PM, Ken Gaillot <kgail...@redhat.com> wrote:
> 
>> On 03/16/2016 05:49 AM, Lorand Kelemen wrote:
>>> Dear Ken,
>>>
>>> Thanks for the reply! I lowered migration-threshold to 1 and rearranged
>>> contraints like you suggested:
>>>
>>> Location Constraints:
>>> Ordering Constraints:
>>>   promote mail-clone then start fs-services (kind:Mandatory)
>>>   promote spool-clone then start fs-services (kind:Mandatory)
>>>   start fs-services then start network-services (kind:Mandatory)
>>
>> Certainly not a big deal, but I would change the above constraint to
>> start fs-services then start mail-services. The IP doesn't care whether
>> the filesystems are up yet or not, but postfix does.
>>
>>>   start network-services then start mail-services (kind:Mandatory)
>>> Colocation Constraints:
>>>   fs-services with spool-clone (score:INFINITY) (rsc-role:Started)
>>> (with-rsc-role:Master)
>>>   fs-services with mail-clone (score:INFINITY) (rsc-role:Started)
>>> (with-rsc-role:Master)
>>>   network-services with mail-services (score:INFINITY)
>>>   mail-services with fs-services (score:INFINITY)
>>>
>>> Now virtualip and postfix becomes stopped, I guess these are relevant
>> but I
>>> attach also full logs:
>>>
>>> Mar 16 11:38:06 [7419] HWJ-626.domain.localpengine: info:
>>> native_color: Resource postfix cannot run anywhere
>>> Mar 16 11:38:06 [7419] HWJ-626.domain.localpengine: info:
>>> native_color: Resource virtualip-1 cannot run anywhere
>>>
>>> Interesting, will try to play around with ordering - colocation, the
>>> solution must be in these settings...
>>>
>>> Best regards,
>>> Lorand
>>>
>>> Mar 16 11:38:06 [7415] HWJ-626.domain.localcib: info:
>>> cib_perform_op:   Diff: --- 0.215.7 2
>>> Mar 16 11:38:06 [7415] HWJ-626.domain.localcib: info:
>>> cib_perform_op:   Diff: +++ 0.215.8 (null)
>>> Mar 16 11:38:06 [7415] HWJ-626.domain.localcib: info:
>>> cib_perform_op:   +  /cib:  @num_updates=8
>>> Mar 16 11:38:06 [7415] HWJ-626.domain.localcib: info:
>>> cib_perform_op:   ++
>>>
>> /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='postfix']:
>>>  >> operation_key="postfix_monitor_45000" operation="monitor"
>>> crm-debug-origin="do_update_resource" crm_feature_set="3.0.10"
>>> transition-key="86:2962:0:ae755a85-c250-498f-9c94-ddd8a7e2788a"
>>> transition-magic="0:7;86:2962:0:ae755a85-c250-498f-9c94-ddd8a7e2788a"
>>> on_node="mail1" call-id="1333" rc-code="7"
>>> Mar 16 11:38:06 [7420] HWJ-626.domain.local   crmd: info:
>>> abort_transition_graph:   Transition aborted by postfix_monitor_45000
>>> 'create' on mail1: Inactive graph
>>> (magic=0:7;86:2962:0:ae755a85-c250-498f-9c94-ddd8a7e2788a, cib=0.215.8,
>>> source=process_graph_event:598, 1)
>>> Mar 16 11:38:06 [7420] HWJ-626.domain.local   crmd: info:
>>> update_failcount: Updating failcount for postfix on mail1 after
>> failed
>>> monitor: rc=7 (update=value++, time=1458124686)

Re: [ClusterLabs] documentation on STONITH with remote nodes?

[Ken Gaillot]

On 03/12/2016 05:07 AM, Adam Spiers wrote:
> Is there any documentation on how STONITH works on remote nodes?  I
> couldn't find any on clusterlabs.org, and it's conspicuously missing
> from:
> 
> http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html-single/Pacemaker_Remote/
> 
> I'm guessing the answer is more or less "it works exactly the same as
> for corosync nodes", however I expect there are nuances which might be
> worth documenting.  In particular I'm looking for confirmation that
> STONITH resources for remote nodes will only run on the corosync
> nodes, and can't run on (other) remote nodes.  My empirical tests seem
> to confirm this, but reassurance from an expert would be appreciated :-)

The above link does have some information -- search it for "fencing".

You are correct, only full cluster nodes can run fence devices or
initiate fencing actions.

Fencing of remote nodes (configured via ocf:pacemaker:remote resource)
is indeed identical to fencing of full cluster nodes. You can configure
fence devices for them the same way, and the cluster fences them the
same way.

Fencing of guest nodes (configured via remote-node property of a
resource such as VirtualDomain) is different. For those, fence devices
are ignored, and the cluster fences them by stopping and starting the
resource.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Antw: notice: throttle_handle_load: High CPU load detected

[Ken Gaillot]

On 03/29/2016 08:22 AM, Kostiantyn Ponomarenko wrote:
> Ken, thank you for the answer.
> 
> Every node in my cluster under normal conditions has "load average" of
> about 420. It is mainly connected to the high disk IO on the system.
> My system is designed to use almost 100% of its hardware (CPU/RAM/disks),
> so the situation when the system consumes almost all HW resources is
> normal.

420 suggests that HW resources are outstripped -- anything above the
system's number of cores means processes are waiting for some resource.
(Although with an I/O-bound workload like this, the number of cores
isn't very important -- most will be sitting idle despite the high
load.) And if that's during normal conditions, what will happen during a
usage spike? It sounds like a recipe for less-than-HA.

Under high load, there's a risk of negative feedback, where monitors
time out, causing pacemaker to schedule recovery actions, which cause
load to go higher and more monitors to time out, etc. That's why
throttling is there.

> I would like to get rid of "High CPU load detected" messages in the
> log, because
> they flood corosync.log as well as system journal.
> 
> Maybe you can give an advice what would be the best way do to it?
> 
> So far I came up with the idea of setting "load-threshold" to 1000% ,
> because of:
> 420(load average) / 24 (cores) = 17.5 (adjusted_load);
> 2 (THROTLE_FACTOR_HIGH) * 10 (throttle_load_target) = 20
> 
> if(adjusted_load > THROTTLE_FACTOR_HIGH * throttle_load_target) {
> crm_notice("High %s detected: %f", desc, load);

That should work, as far as reducing the log messages, though of course
it also reduces the amount of throttling pacemaker will do.

> In this case do I need to set "node-action-limit" to something less than "2
> x cores" (which is default).

It's not necessary, but it would help compensate for the reduced
throttling by imposing a maximum number of actions run at one time.

I usually wouldn't recommend reducing log verbosity, because detailed
logs are often necessary for troubleshooting cluster issues, but if your
logs are on the same I/O controller that is overloaded, you might
consider logging only to syslog and not to an additional detail file.
That would cut back on the amount of I/O due to pacemaker itself. You
could even drop PCMK_logpriority to warning, but then you're losing even
more information.

> Because the logic is (crmd/throttle.c):
> 
> switch(r->mode) {
> case throttle_extreme:
> case throttle_high:
> jobs = 1; /* At least one job must always be allowed */
> break;
> case throttle_med:
> jobs = QB_MAX(1, r->max / 4);
> break;
> case throttle_low:
> jobs = QB_MAX(1, r->max / 2);
> break;
> case throttle_none:
> jobs = QB_MAX(1, r->max);
> break;
> default:
> crm_err("Unknown throttle mode %.4x on %s", r->mode, node);
> break;
> }
> return jobs;
> 
> 
> The thing is, I know that there is "High CPU load" and this is normal
> state, but I wont Pacemaker to not saying it to me and treat this state the
> best it can.

If you can't improve your I/O performance, what you suggested is
probably the best that can be done.

When I/O is that critical to you, there are many tweaks that can make a
big difference in performance. I'm not sure how familiar you are with
them already. Options depend on what your storage is (local or network,
hardware/software/no RAID, etc.) and what your I/O-bound application is
(database, etc.), but I'd look closely at cache/buffer settings at all
levels from hardware to application, RAID stripe alignment, filesystem
choice and tuning, log verbosity, etc.

> 
> Thank you,
> Kostia
> 
> On Mon, Mar 14, 2016 at 7:18 PM, Ken Gaillot <kgail...@redhat.com> wrote:
> 
>> On 02/29/2016 07:00 AM, Kostiantyn Ponomarenko wrote:
>>> I am back to this question =)
>>>
>>> I am still trying to understand the impact of "High CPU load detected"
>>> messages in the log.
>>> Looking in the code I figured out that setting "load-threshold" parameter
>>> to something higher than 100% solves the problem.
>>> And actually for 8 cores (12 with Hyper Threading) load-threshold=400%
>> kind
>>> of works.
>>>
>>> Also I noticed that this parameter may have an impact on the number of
>> "the
>>> maximum number of jobs that can be scheduled per node". As there is a
>>> formula to limit F_CRM_THROTTLE_MAX based on F_CRM_THROTTLE_MODE.
>>>
>>> Is my 

Re: [ClusterLabs] Pacemaker on-fail standby recovery does not start DRBD slave resource

[Ken Gaillot]

On 03/30/2016 11:20 AM, Sam Gardner wrote:
> I have configured some network resources to automatically standby their node 
> if the system detects a failure on them. However, the DRBD slave that I have 
> configured does not automatically restart after the node is "unstandby-ed" 
> after the failure-timeout expires.
> Is there any way to make the "stopped" DRBDSlave resource automatically start 
> again once the node is recovered?
> 
> See the  progression of events below:
> 
> Running cluster:
> Wed Mar 30 16:04:20 UTC 2016
> Cluster name:
> Last updated: Wed Mar 30 16:04:20 2016
> Last change: Wed Mar 30 16:03:24 2016
> Stack: classic openais (with plugin)
> Current DC: ha-d1.tw.com - partition with quorum
> Version: 1.1.12-561c4cf
> 2 Nodes configured, 2 expected votes
> 7 Resources configured
> 
> 
> Online: [ ha-d1.tw.com ha-d2.tw.com ]
> 
> Full list of resources:
> 
>  Resource Group: network
>  inif   (ocf::custom:ip.sh):   Started ha-d1.tw.com
>  outif  (ocf::custom:ip.sh):   Started ha-d1.tw.com
>  dmz1   (ocf::custom:ip.sh):   Started ha-d1.tw.com
>  Master/Slave Set: DRBDMaster [DRBDSlave]
>  Masters: [ ha-d1.tw.com ]
>  Slaves: [ ha-d2.tw.com ]
>  Resource Group: filesystem
>  DRBDFS (ocf::heartbeat:Filesystem):Started ha-d1.tw.com
>  Resource Group: application
>  service_failover   (ocf::custom:service_failover):Started 
> ha-d1.tw.com
> 
> 
> version: 8.4.5 (api:1/proto:86-101)
> srcversion: 315FB2BBD4B521D13C20BF4
> 
>  1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-
> ns:4 nr:0 dw:4 dr:757 al:1 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
> [153766.565352] block drbd1: send bitmap stats [Bytes(packets)]: plain 0(0), 
> RLE 21(1), total 21; compression: 100.0%
> [153766.568303] block drbd1: receive bitmap stats [Bytes(packets)]: plain 
> 0(0), RLE 21(1), total 21; compression: 100.0%
> [153766.568316] block drbd1: helper command: /sbin/drbdadm 
> before-resync-source minor-1
> [153766.568356] block drbd1: helper command: /sbin/drbdadm 
> before-resync-source minor-1 exit code 255 (0xfffe)
> [153766.568363] block drbd1: conn( WFBitMapS -> SyncSource ) pdsk( Consistent 
> -> Inconsistent )
> [153766.568374] block drbd1: Began resync as SyncSource (will sync 4 KB [1 
> bits set]).
> [153766.568444] block drbd1: updated sync UUID 
> B0DA745C79C56591:36E0631B6F022952:36DF631B6F022952:133127197CF097C6
> [153766.577695] block drbd1: Resync done (total 1 sec; paused 0 sec; 4 K/sec)
> [153766.577700] block drbd1: updated UUIDs 
> B0DA745C79C56591::36E0631B6F022952:36DF631B6F022952
> [153766.577705] block drbd1: conn( SyncSource -> Connected ) pdsk( 
> Inconsistent -> UpToDate )¯
> 
> Failure detected:
> Wed Mar 30 16:08:22 UTC 2016
> Cluster name:
> Last updated: Wed Mar 30 16:08:22 2016
> Last change: Wed Mar 30 16:03:24 2016
> Stack: classic openais (with plugin)
> Current DC: ha-d1.tw.com - partition with quorum
> Version: 1.1.12-561c4cf
> 2 Nodes configured, 2 expected votes
> 7 Resources configured
> 
> 
> Node ha-d1.tw.com: standby (on-fail)
> Online: [ ha-d2.tw.com ]
> 
> Full list of resources:
> 
>  Resource Group: network
>  inif   (ocf::custom:ip.sh):   Started ha-d1.tw.com
>  outif  (ocf::custom:ip.sh):   Started ha-d1.tw.com
>  dmz1   (ocf::custom:ip.sh):   FAILED ha-d1.tw.com
>  Master/Slave Set: DRBDMaster [DRBDSlave]
>  Masters: [ ha-d1.tw.com ]
>  Slaves: [ ha-d2.tw.com ]
>  Resource Group: filesystem
>  DRBDFS (ocf::heartbeat:Filesystem):Started ha-d1.tw.com
>  Resource Group: application
>  service_failover   (ocf::custom:service_failover):Started 
> ha-d1.tw.com
> 
> Failed actions:
> dmz1_monitor_7000 on ha-d1.tw.com 'not running' (7): call=156, 
> status=complete, last-rc-change='Wed Mar 30 16:08:19 2016', queued=0ms, 
> exec=0ms
> 
> 
> 
> version: 8.4.5 (api:1/proto:86-101)
> srcversion: 315FB2BBD4B521D13C20BF4
> 
>  1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-
> ns:4 nr:0 dw:4 dr:765 al:1 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
> [153766.568356] block drbd1: helper command: /sbin/drbdadm 
> before-resync-source minor-1 exit code 255 (0xfffe)
> [153766.568363] block drbd1: conn( WFBitMapS -> SyncSource ) pdsk( Consistent 
> -> Inconsistent )
> [153766.568374] block drbd1: Began resync as SyncSource (will sync 4 KB [1 
> bits set]).
> [153766.568444] block drbd1: updated sync UUID 
> B0DA745C79C56591:36E0631B6F022952:36DF631B6F022952:133127197CF097C6
> [153766.577695] block drbd1: Resync done (total 1 sec; paused 0 sec; 4 K/sec)
> [153766.577700] block drbd1: updated UUIDs 
> B0DA745C79C56591::36E0631B6F022952:36DF631B6F022952
> [153766.577705] block drbd1: conn( SyncSource -> Connected ) pdsk( 
> Inconsistent -> UpToDate )
> [154057.455270] e1000: eth2 NIC Link is Down
> [154057.455451] e1000 :02:02.0 eth2: Reset adapter
> 
> Failover complete:
> Wed Mar 

Re: [ClusterLabs] Freezing/Unfreezing in Pacemaker ?

[Ken Gaillot]

On 04/07/2016 06:40 AM, jaspal singla wrote:
> Hello,
> 
> As we have clusvcadm -U  and clusvcadm -Z 
>  to freeze and unfreeze resource in CMAN. Would really appreciate if
> someone please give some pointers for freezing/unfreezing a resource in
> Pacemaker (pcs) as well.
> 
> Thanks,
> Jaspal Singla

Hi,

The equivalent in pacemaker is "managed" and "unmanaged" resources.

The usage depends on what tools you are using. For pcs, it's "pcs
resource unmanage " to freeze, and "manage" to unfreeze.
At a lower level, it's setting the is-managed meta-attribute of the
resource.

It's also possible to set the maintenance-mode cluster property to
"freeze" all resources.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] DRBD on asymmetric-cluster

[Ken Gaillot]

On 04/06/2016 09:29 PM, Jason Voorhees wrote:
> Hey guys:
> 
> I've been reading a little bit more about rules but there are certain
> things that are not so clear to me yet. First, I've created 3 normal
> resources and one master/slave resource (clusterdataClone). My
> resources and constraints look like this:
> 
> # pcs resource
>  MTA(systemd:postfix):  Started nodo1
>  Web(systemd:httpd):Started nodo1
>  IPService  (ocf::heartbeat:IPaddr2):   Started nodo1
>  Master/Slave Set: clusterdataClone [clusterdata]
>  Masters: [ nodo1 ]
>  Slaves: [ nodo2 ]
> 
> # pcs constraint show --full
> Location Constraints:
>   Resource: IPService
> Enabled on: nodo1 (score:10) (id:location-IPService-nodo1-10)
> Enabled on: nodo2 (score:9) (id:location-IPService-nodo2-9)
> Enabled on: nodo1 (score:INFINITY) (role: Started) 
> (id:cli-prefer-IPService)

FYI, commands that "move" a resource do so by adding location
constraints. The ID of these constraints will start with "cli-". They
override the normal behavior of the cluster, and stay in effect until
you explicitly remove them. (With pcs, you can remove them with "pcs
resource clear".)

>   Resource: MTA
> Enabled on: nodo1 (score:10) (id:location-MTA-nodo1-10)
> Enabled on: nodo2 (score:9) (id:location-MTA-nodo2-9)
>   Resource: Web
> Enabled on: nodo1 (score:10) (id:location-Web-nodo1-10)
> Enabled on: nodo2 (score:9) (id:location-Web-nodo2-9)
>   Resource: clusterdataClone
> Constraint: location-clusterdataClone
>   Rule: score=INFINITY boolean-op=or  (id:location-clusterdataClone-rule)
> Expression: #uname eq nodo1  (id:location-clusterdataClone-rule-expr)
> Expression: #uname eq nodo2  
> (id:location-clusterdataClone-rule-expr-1)
> Ordering Constraints:
> Colocation Constraints:
>   Web with IPService (score:INFINITY) (id:colocation-Web-IPService-INFINITY)
>   MTA with IPService (score:INFINITY) (id:colocation-MTA-IPService-INFINITY)
>   clusterdataClone with IPService (score:INFINITY) (rsc-role:Master)
> (with-rsc-role:Started)
> (id:colocation-clusterdataClone-IPService-INFINITY)

Note that colocation constraints only specify that the resources must
run together. It does not imply any order in which they must be started.
If Web and/or MTA should be started after clusterdataClone, configure
explicit ordering constraints for that.

> These are the commands I run to create the master/slave resource and
> its contraints:
> 
> # pcs cluster cib myfile
> # pcs -f myfile resource create clusterdata ocf:linbit:drbd
> drbd_resource=clusterdb op monitor interval=30s role=Master op monitor
> interval=31s role=Slave
> # pcs -f myfile resource master clusterdataClone clusterdata
> master-max=1 master-node-max=1 clone-max=2 clone-node-max=1
> notify=true
> # pcs -f myfile constraint location clusterdataClone rule
> score=INFINITY \#uname eq nodo1 or \#uname eq nodo2

The above constraint as currently worded will have no effect. It says
that clusterdataClone must be located on either nodo1 or nodo2. Since
those are your only nodes, it doesn't really constrain anything.

If you want to prefer one node for the master role, you want to add
role=master, take out the node you don't want to prefer, and set score
to something less than INFINITY.

> # pcs -f myfile constraint colocation add master clusterdataClone with 
> IPService
> # pcs cluster cib-push myfile
> 
> So now, my master/slave resource is started as Master in the same node
> where IPService is already active. So far so good. But the problem is
> that I can't move IPService from nodo1 to nodo2. When I run...
> 
> # pcs resource move IPService nodo2
> 
> it does nothing but... IPService keeps active on nodo1.
> 
> Then I tried to remove all my clusterdataClone constraints and repeat
> the same commands shows lines above (# pcs -f myfile ...) but this
> time without creating a colocation constraint between clusterdataClone
> and IPService.  When I do some tests again running...
> 
> # pcs resource move IPService nodo2
> 
> well, IPService is moved to nodo2, but clusterdataClone keeps active
> as Master in node1. I thought it would be promoted as Master in nodo2
> and demoted to Slave in nodo1.
> 
> Do you know why my master/slave resource is not being "moved as
> master" between nodes?
> 
> How do I "move" the Master role from nodo1 to nodo2 for
> clusterdataClone? I want to make  nodo2 Primary and nodo1 Secondary
> but I have no idea how to do this manually (only for testing)
> 
> I hope someone can help :(
> 
> Thanks in advance
> 
> On Mon, Apr 4, 2016 at 4:50 PM, Jason Voorhees <jvoorhe...@gmail.com> wrote:
>>

Re: [ClusterLabs] DRBD on asymmetric-cluster

[Ken Gaillot]

On 04/07/2016 10:30 AM, Jason Voorhees wrote:
>> FYI, commands that "move" a resource do so by adding location
>> constraints. The ID of these constraints will start with "cli-". They
>> override the normal behavior of the cluster, and stay in effect until
>> you explicitly remove them. (With pcs, you can remove them with "pcs
>> resource clear".)
> 
> Agree :)
> 
> 
>> Note that colocation constraints only specify that the resources must
>> run together. It does not imply any order in which they must be started.
>> If Web and/or MTA should be started after clusterdataClone, configure
>> explicit ordering constraints for that.
>>
> 
> Agree. So far I haven't created any ordering constraints because it
> isn't important to me, YET, the order for starting services. However I
> have a question... if I don't have any ordering constraints at all, am
> I still able to activate resources no matter the order?

Sort of, but not exactly.

With a colocation constraint "A with B", the cluster must assign B to a
node before it can place A. B does not have to be started, but it does
have to be *able* to be started, in order to be assigned to a node. So
if something prevents B from being started (disabled in config, not
allowed on any online node, etc.), it will not be assigned to a node,
and A will not run.

That doesn't mean that B will be started first, though. If the cluster
needs to start both A and B, it can start them in any order. With an
ordering constraint "B then A", B must be started first, and the start
must complete successfully, before A can be started.

>>> These are the commands I run to create the master/slave resource and
>>> its contraints:
>>>
>>> # pcs cluster cib myfile
>>> # pcs -f myfile resource create clusterdata ocf:linbit:drbd
>>> drbd_resource=clusterdb op monitor interval=30s role=Master op monitor
>>> interval=31s role=Slave
>>> # pcs -f myfile resource master clusterdataClone clusterdata
>>> master-max=1 master-node-max=1 clone-max=2 clone-node-max=1
>>> notify=true
>>> # pcs -f myfile constraint location clusterdataClone rule
>>> score=INFINITY \#uname eq nodo1 or \#uname eq nodo2
>>
>> The above constraint as currently worded will have no effect. It says
>> that clusterdataClone must be located on either nodo1 or nodo2. Since
>> those are your only nodes, it doesn't really constrain anything.
> Ok, the last command (location with rule) was created to allow
> clusterdataClone start at both nodes, because without this rule the
> resource was always in "stopped" status in both nodes. Once I added
> this rule my clusterdataClone resource started automatically but I
> don't understand why it choosed a node to run as Master and the other
> one as Slave. Is it random?

I don't know why the resource would be stopped without this constraint.
Maybe you have an opt-in cluster? But in that case you can use a normal
location constraint, you don't need a rule.

It will choose one as master and one as slave because you have
master-max=1. The choice, as with everything else in pacemaker, is based
on scores, but these are transparent to the user and appears "random".

>> If you want to prefer one node for the master role, you want to add
>> role=master, take out the node you don't want to prefer, and set score
>> to something less than INFINITY.
> Well, I could add a rule to prefer nodo1 over nodo2 to run the Master
> role (in fact, I think I already did it) but what I want it's
> something different: I would like the Master role to follow IPService,
> I mean, clusterdataClone become Master where IPService was previously
> activated.
> 
> Is this possible? Or the only way to configure constraints is that my
> resources (IPService, Web, MTA) follow the Master role of
> clusterdataClone?

I think the latter approach makes more sense and is common. Storage is
more complicated than an IP and thus more likely to break, so it would
seem to be more reliable to follow where storage can successfully start.
The exception would be if the IP is much more important to you than the
storage and is useful without it.

You might want to look at resource sets. The syntax is a bit difficult
to follow but it's very flexible. See pcs constraint colocation/order set.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] HA meetup at OpenStack Summit

[Ken Gaillot]

Hi everybody,

The upcoming OpenStack Summit is April 25-29 in Austin, Texas (US). Some
regular ClusterLabs contributors are going, so I was wondering if anyone
would like to do an informal meetup sometime during the summit.

It looks like the best time would be that Wednesday, either lunch (at
the venue) or dinner (offsite). It might also be possible to reserve a
small (10-person) meeting room, or just meet informally in the expo hall.

Anyone interested? Preferences/conflicts?
-- 
Ken Gaillot <kgail...@redhat.com>

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] attrd: Fix sigsegv on exit if initialization failed

[Ken Gaillot]

On 10/12/2015 06:08 AM, Vladislav Bogdanov wrote:
> Hi,
> 
> This was caught with 0.17.1 libqb, which didn't play well with long pids.
> 
> commit 180a943846b6d94c27b9b984b039ac0465df64da
> Author: Vladislav Bogdanov 
> Date:   Mon Oct 12 11:05:29 2015 +
> 
> attrd: Fix sigsegv on exit if initialization failed
> 
> diff --git a/attrd/main.c b/attrd/main.c
> index 069e9fa..94e9212 100644
> --- a/attrd/main.c
> +++ b/attrd/main.c
> @@ -368,8 +368,12 @@ main(int argc, char **argv)
>  crm_notice("Cleaning up before exit");
> 
>  election_fini(writer);
> -crm_client_disconnect_all(ipcs);
> -qb_ipcs_destroy(ipcs);
> +
> +if (ipcs) {
> +crm_client_disconnect_all(ipcs);
> +qb_ipcs_destroy(ipcs);
> +}
> +
>  g_hash_table_destroy(attributes);
> 
>  if (the_cib) {

I set aside this message to merge it, then promptly lost it ... finally
ran across it again. It's merged into master now. Thanks for reporting
the problem and a patch.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] reproducible split brain

[Ken Gaillot]

On 03/16/2016 03:04 PM, Christopher Harvey wrote:
> On Wed, Mar 16, 2016, at 04:00 PM, Digimer wrote:
>> On 16/03/16 03:59 PM, Christopher Harvey wrote:
>>> I am able to create a split brain situation in corosync 1.1.13 using
>>> iptables in a 3 node cluster.
>>>
>>> I have 3 nodes, vmr-132-3, vmr-132-4, and vmr-132-5
>>>
>>> All nodes are operational and form a 3 node cluster with all nodes are
>>> members of that ring.
>>> vmr-132-3 ---> Online: [ vmr-132-3 vmr-132-4 vmr-132-5 ]
>>> vmr-132-4 ---> Online: [ vmr-132-3 vmr-132-4 vmr-132-5 ]
>>> vmr-132-5 ---> Online: [ vmr-132-3 vmr-132-4 vmr-132-5 ]
>>> so far so good.
>>>
>>> running the following on vmr-132-4 drops all incoming (but not outgoing)
>>> packets from vmr-132-3:
>>> # iptables -I INPUT -s 192.168.132.3 -j DROP
>>> # iptables -L
>>> Chain INPUT (policy ACCEPT)
>>> target prot opt source   destination
>>> DROP   all  --  192.168.132.3anywhere
>>>
>>> Chain FORWARD (policy ACCEPT)
>>> target prot opt source   destination
>>>
>>> Chain OUTPUT (policy ACCEPT)
>>> target prot opt source   destination
>>>
>>> vmr-132-3 ---> Online: [ vmr-132-3 vmr-132-4 vmr-132-5 ]
>>> vmr-132-4 ---> Online: [ vmr-132-4 vmr-132-5 ]
>>> vmr-132-5 ---> Online: [ vmr-132-4 vmr-132-5 ]
>>>
>>> vmr-132-3 thinks everything is normal and continues to provide service,
>>> vmr-132-4 and 5 form a new ring, achieve quorum and provide the same
>>> service. Splitting the link between 3 and 4 in both directions isolates
>>> vmr 3 from the rest of the cluster and everything fails over normally,
>>> so only a unidirectional failure causes problems.
>>>
>>> I don't have stonith enabled right now, and looking over the
>>> pacemaker.log file closely to see if 4 and 5 would normally have fenced
>>> 3, but I didn't see any fencing or stonith logs.
>>>
>>> Would stonith solve this problem, or does this look like a bug?
>>
>> It should, that is its job.
> 
> is there some log I can enable that would say
> "ERROR: hey, I would use stonith here, but you have it disabled! your
> warranty is void past this point! do not pass go, do not file a bug"?

Enable fencing, and create a fence device with a static host list that
doesn't match any of your nodes. Pacemaker will think fencing is
configured, but when it tries to actually fence a node, no devices will
be capable of it, and there will be errors to that effect (including "No
such device"). The cluster will block at that point. You can use
stonith_admin --confirm to manually indicate the node is down and
unblock the cluster (but be absolutely sure the node really is down!).

>> -- 
>> Digimer
>> Papers and Projects: https://alteeve.ca/w/
>> What if the cure for cancer is trapped in the mind of a person without
>> access to education?


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Cluster goes to unusable state if fencing resource is down

[Ken Gaillot]

On 03/18/2016 02:58 AM, Arjun Pandey wrote:
> Hi
> 
> I am running a 2 node cluster with this config on centos 6.6  where i
> have a multi-state resource foo being run in master/slave mode and  a
> bunch of floating IP addresses configured. Additionally i have a
> collocation constraint for the IP addr to be collocated with the
> master.
> 
> When i configure fencing using fence_ilo4 agents things work fine.
> However during testing i was trying out a case where the ilo cable is
> plugged out. In this case the entire cluster is brought down.
> 
> I understand that this seems to be a safer solution to ensure
> correctness and consistency of the systems. However my requirement was

Exactly. Without working fencing, the cluster can't know whether the
node is really down, or just malfunctioning and possibly still accessing
shared resources.

> to still keep it operational since the application and the floating ip
> are still up. Is there a way to acheive this ?

If fencing fails, and the node is really down, you'd be fine ignoring
the failure. But if the node is actually up, ignoring the failure means
both nodes will activate the floating IP, which will not be operational
(packets will sometimes go to one node, sometimes the other, disrupting
any reliable communication).

> Also considering a case where there is a multi node cluster ( more
> than 10 nodes )  and one of the machines just goes down along with the
> ilo resource for that node. Does it really make sense to bring the
> services down even when the rest of nodes are up ?

It makes sense if data integrity is your highest priority. Imagine a
cluster used by a bank for customer's account balances -- it's far
better to lock up the entire cluster than risk corrupting that data.

The best solution that pacemaker offers in this situation is fencing
topology. You can have multiple fence devices, and if one fails,
pacemaker will try the next.

One common deployment is IPMI as the first level (as you have now), with
an intelligent power switch as the second (backup) level. If IPMI
doesn't respond, the cluster will cut power to the host. Another
possibility is to use an intelligent network switch to cut off network
access to the failed node (if that is sufficient to prevent the node
from accessing any shared resources). If the services being offered are
important enough to require high availability, the relatively small cost
of an intelligent power switch should be easily justified, serving as a
type of insurance.

Not having fencing has such a high chance of making a huge mess that no
company I know of that supports clusters will support a cluster without it.

That said, if you are supporting your own clusters, understand the
risks, and are willing to deal with the worst-case scenario manually,
pacemaker does offer the option to disable stonith. There is no built-in
option to try stonith but ignore any failures. However, it is possible
to configure a fencing topology that does the same thing, if the second
level simply pretends that the fencing succeeded. I'm not going to
encourage that by describing how ;)

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] no clone for pcs-based cluster fencing?

[Ken Gaillot]

On 03/20/2016 06:20 PM, Devin Reade wrote:
> I'm looking at a new pcs-style two node cluster running on CentOS 7
> (pacemaker 1.1.13, corosync 2.3.4) and crm_mon shows this line
> for my fencing resource, that is the resource running on only one of
> the two nodes:
> 
>fence_cl2 (stonith:fence_apc_snmp):Started nodeB
> 
> On an older CentOS 5 cluster (pacemaker 1.0.12, corosync 1.2.7),
> crm_mon shows as the comparable line the following, that is the
> fencing agent cloned across both nodes:
> 
>Clone Set: pdu-fence
>Started: [ node1 node2 ]
> 
> With the new pcs-based clusters, are the (APC SNMP) fencing agents not
> supposed to be cloned anymore, or do I just have things misconfigured?

It's actually newer pacemaker versions rather than pcs itself. Fence
agents do not need to be cloned, or even running -- as long as they're
configured and enabled, any node can use the resource. The node that's
"running" the fence device will be the one that monitors it, and as a
result it will be preferred to execute the fencing if possible.

For fence_apc_snmp in particular, you want to use pcmk_host_map instead
of pcmk_host_list/pcmk_host_check, to map node names to APC ports. For
example, pcmk_host_map="nodeA:1;nodeB:2" if those are the relevant APC
ports.

> Looking at the pcs man page, it looks like cloning a fencing agent isn't
> supported.
> 
> I was pretty sure I tested fencing on both nodes when I first configured
> the cluster, but I don't have a record of where the fencing agent was
> running at the time.  I was using the RHEL7 cluster configuration guide
> and I don't seem to see anything in there talking about cloning the fencing
> agent, so I'm wondering if it's no longer a recommended/required procedure.
> 
> New cluster config:
> 
>  Stonith Devices:
>Resource: fence_cl2 (class=stonith type=fence_apc_snmp)
> Attributes: pcmk_host_list=nodeA,nodeB ipaddr=192.168.1.200
> community=somepass pcmk_host_check=static-list
> Operations: monitor interval=60s (fence_cl2-monitor-interval-60s)
> 
> Old cluster config:
> 
>   primitive st-pdu stonith:apcmastersnmp \
>params ipaddr="192.168.1.200" community="somepass" port="161" \
>op start interval="0" timeout="60s" \
>op stop interval="0" timeout="15s" \
>op monitor interval="3600" timeout="60s"
>   clone pdu-fence st-pdu
> 
> 
> Devin


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: no clone for pcs-based cluster fencing?

[Ken Gaillot]

On 03/21/2016 09:34 AM, Ulrich Windl wrote:
>>>> Ken Gaillot <kgail...@redhat.com> schrieb am 21.03.2016 um 15:22 in 
>>>> Nachricht
> <56f003b0.4020...@redhat.com>:
> 
> [...]
>> It's actually newer pacemaker versions rather than pcs itself. Fence
>> agents do not need to be cloned, or even running -- as long as they're
>> configured and enabled, any node can use the resource. The node that's
>> "running" the fence device will be the one that monitors it, and as a
>> result it will be preferred to execute the fencing if possible.
> 
> Q: How can I find out which node is running the fencing device when not 
> having configured a monitor?

Fence devices show up in status output like any other resource. If
there's no monitor configured, "running" a fence device has no real
effect on anything.

> 
> [...]
> 
> Regards,
> Ulrich
> 
> 


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] fence_scsi no such device

[Ken Gaillot]

On 03/21/2016 08:39 AM, marvin wrote:
> 
> 
> On 03/15/2016 03:39 PM, Ken Gaillot wrote:
>> On 03/15/2016 09:10 AM, marvin wrote:
>>> Hi,
>>>
>>> I'm trying to get fence_scsi working, but i get "no such device" error.
>>> It's a two node cluster with nodes called "node01" and "node03". The OS
>>> is RHEL 7.2.
>>>
>>> here is some relevant info:
>>>
>>> # pcs status
>>> Cluster name: testrhel7cluster
>>> Last updated: Tue Mar 15 15:05:40 2016  Last change: Tue Mar 15
>>> 14:33:39 2016 by root via cibadmin on node01
>>> Stack: corosync
>>> Current DC: node03 (version 1.1.13-10.el7-44eb2dd) - partition with
>>> quorum
>>> 2 nodes and 23 resources configured
>>>
>>> Online: [ node01 node03 ]
>>>
>>> Full list of resources:
>>>
>>>   Clone Set: dlm-clone [dlm]
>>>   Started: [ node01 node03 ]
>>>   Clone Set: clvmd-clone [clvmd]
>>>   Started: [ node01 node03 ]
>>>   fence-node1(stonith:fence_ipmilan):Started node03
>>>   fence-node3(stonith:fence_ipmilan):Started node01
>>>   Resource Group: test_grupa
>>>   test_ip(ocf::heartbeat:IPaddr):Started node01
>>>   lv_testdbcl(ocf::heartbeat:LVM):   Started node01
>>>   fs_testdbcl(ocf::heartbeat:Filesystem):Started node01
>>>   oracle11_baza  (ocf::heartbeat:oracle):Started node01
>>>   oracle11_lsnr  (ocf::heartbeat:oralsnr):   Started node01
>>>   fence-scsi-node1   (stonith:fence_scsi):   Started node03
>>>   fence-scsi-node3   (stonith:fence_scsi):   Started node01
>>>
>>> PCSD Status:
>>>node01: Online
>>>node03: Online
>>>
>>> Daemon Status:
>>>corosync: active/enabled
>>>pacemaker: active/enabled
>>>pcsd: active/enabled
>>>
>>> # pcs stonith show
>>>   fence-node1(stonith:fence_ipmilan):Started node03
>>>   fence-node3(stonith:fence_ipmilan):Started node01
>>>   fence-scsi-node1   (stonith:fence_scsi):   Started node03
>>>   fence-scsi-node3   (stonith:fence_scsi):   Started node01
>>>   Node: node01
>>>Level 1 - fence-scsi-node3
>>>Level 2 - fence-node3
>>>   Node: node03
>>>Level 1 - fence-scsi-node1
>>>Level 2 - fence-node1
>>>
>>> # pcs stonith show fence-scsi-node1 --all
>>>   Resource: fence-scsi-node1 (class=stonith type=fence_scsi)
>>>Attributes: pcmk_host_list=node01 pcmk_monitor_action=metadata
>>> pcmk_reboot_action=off
>>>Meta Attrs: provides=unfencing
>>>Operations: monitor interval=60s
>>> (fence-scsi-node1-monitor-interval-60s)
>>>
>>> # pcs stonith show fence-scsi-node3 --all
>>>   Resource: fence-scsi-node3 (class=stonith type=fence_scsi)
>>>Attributes: pcmk_host_list=node03 pcmk_monitor_action=metadata
>>> pcmk_reboot_action=off
>>>Meta Attrs: provides=unfencing
>>>Operations: monitor interval=60s
>>> (fence-scsi-node3-monitor-interval-60s)
>>>
>>> node01 # pcs stonith fence node03
>>> Error: unable to fence 'node03'
>>> Command failed: No such device
>>>
>>> node01 # tail /var/log/messages
>>> Mar 15 14:54:04 node01 stonith-ng[20024]:  notice: Client
>>> stonith_admin.29191.2b7fe910 wants to fence (reboot) 'node03' with
>>> device '(any)'
>>> Mar 15 14:54:04 node01 stonith-ng[20024]:  notice: Initiating remote
>>> operation reboot for node03: d1df9201-5bb1-447f-9b40-d3d7235c3d0a (0)
>>> Mar 15 14:54:04 node01 stonith-ng[20024]:  notice: fence-scsi-node3 can
>>> fence (reboot) node03: static-list
>>> Mar 15 14:54:04 node01 stonith-ng[20024]:  notice: fence-node3 can fence
>>> (reboot) node03: static-list
>>> Mar 15 14:54:04 node01 stonith-ng[20024]:  notice: All fencing options
>>> to fence node03 for stonith_admin.29191@node01.d1df9201 failed
>> The above line is the key. Both of the devices registered for node03
>> returned failure. Pacemaker then looked for any other device capable of
>> fencing node03 and there is none, so that's why it reported "No such
>> device" (an admittedly obscure message).
>>
>> It looks like the fence agents require more configuration options than
>> you have set. If you run "/path/to/fence/agent -o metadata", y

Re: [ClusterLabs] "No such device" with fence_pve agent

[Ken Gaillot]

On 03/22/2016 06:32 AM, Stanislav Kopp wrote:
> Hi,
> 
> I have problem with using "fence_pve" agent with pacemaker, the agent
> works fine from command line, but if I simulate stonith action or use
> "crm node fence ", it doesn't work:
> 
>  Mar 22 10:38:06 [675] redis2 stonith-ng: debug:
> xml_patch_version_check: Can apply patch 0.50.22 to 0.50.21
> Mar 22 10:38:06 [675] redis2 stonith-ng: debug: stonith_command:
> Processing st_query 0 from redis1 ( 0)
> Mar 22 10:38:06 [675] redis2 stonith-ng: debug: stonith_query: Query
>  st_async_id="da507055-afc3-4b5c-bcc0-f887dd1736d3" st_op="st_query"
> st_callid="2" st_callopt="0"
> st_remote_op="da507055-afc3-4b5c-bcc0-f887dd1736d3" st_target="redis1"
> st_device_action="reboot" st_origin="redis1"
> st_clientid="95f8f271-e698-4a9b-91ae-950c55c230a5"
> st_clientname="crmd.674" st_timeout="60" src="redis1"/>
> Mar 22 10:38:06 [675] redis2 stonith-ng: debug: get_capable_devices:
> Searching through 1 devices to see what is capable of action (reboot)
> for target redis1
> Mar 22 10:38:06 [675] redis2 stonith-ng: debug:
> schedule_stonith_command: Scheduling list on stonith-redis1 for
> stonith-ng (timeout=60s)
> Mar 22 10:38:06 [675] redis2 stonith-ng: debug: stonith_command:
> Processed st_query from redis1: OK (0)
> Mar 22 10:38:06 [675] redis2 stonith-ng: debug: stonith_action_create:
> Initiating action list for agent fence_pve (target=(null))
> Mar 22 10:38:06 [675] redis2 stonith-ng: debug:
> internal_stonith_action_execute: forking
> Mar 22 10:38:06 [675] redis2 stonith-ng: debug:
> internal_stonith_action_execute: sending args
> Mar 22 10:38:06 [675] redis2 stonith-ng: debug:
> stonith_device_execute: Operation list on stonith-redis1 now running
> with pid=8707, timeout=60s
> Mar 22 10:38:07 [675] redis2 stonith-ng: debug:
> stonith_action_async_done: Child process 8707 performing action 'list'
> exited with rc 0

There is a fence parameter pcmk_host_check that specifies how pacemaker
determines which fence devices can fence which nodes. The default is
dynamic-list, which means to run the fence agent's list command to get
the nodes. So that's what we're seeing above ...

> Mar 22 10:38:07 [675] redis2 stonith-ng: info: dynamic_list_search_cb:
> Refreshing port list for stonith-redis1
> Mar 22 10:38:07 [675] redis2 stonith-ng: debug:
> search_devices_record_result: Finished Search. 0 devices can perform
> action (reboot) on node redis1

... however not all fence agents can figure out their targets
dynamically. Above, we can see either that's the case, or the device
really can't fence redis1.

> Mar 22 10:38:07 [675] redis2 stonith-ng: debug:
> stonith_query_capable_device_cb: Found 0 matching devices for 'redis1'
> Mar 22 10:38:07 [675] redis2 stonith-ng: debug: stonith_command:
> Processing st_notify reply 0 from redis1 ( 0)
> Mar 22 10:38:07 [675] redis2 stonith-ng: debug:
> process_remote_stonith_exec: Marking call to reboot for redis1 on
> behalf of crmd.674@da507055-afc3-4b5c-bcc0-f887dd1736d3.redis1: No
> such device (-19)
> Mar 22 10:38:07 [675] redis2 stonith-ng: notice: remote_op_done:
> Operation reboot of redis1 by  for crmd.674@redis1.da507055:
> No such device
> Mar 22 10:38:07 [675] redis2 stonith-ng: debug: stonith_command:
> Processed st_notify reply from redis1: OK (0)
> Mar 22 10:38:07 [679] redis2 crmd: notice: tengine_stonith_notify:
> Peer redis1 was not terminated (reboot) by  for redis1: No
> such device (ref=da507055-afc3-4b5c-bcc0-f887dd1736d3) by client
> crmd.674
> Connection to 192.168.122.137 closed by remote host.
> 
> I already read similar thread ("fence_scsi no such device"), but
> didn't find anything what can help me. I'm using pacemaker
> 1.1.14-2~bpo8+1 with corosync 2.3.5-3~bpo8+1 on Debian jessie.
> 
> some info:
> 
> redis1:~# stonith_admin -L
>  stonith-redis2
> 1 devices found
> 
> redis1:~# crm configure show
> node 3232266889: redis2
> node 3232266923: redis1
> primitive ClusterIP IPaddr2 \
> params ip=192.168.122.10 nic=eth0 \
> op monitor interval=10s \
> meta is-managed=true
> primitive stonith-redis1 stonith:fence_pve \
> params ipaddr=192.168.122.6 \
> params login="root@pam" passwd=secret port=100 \
> op start interval=0 timeout=60s \
> meta target-role=Started is-managed=true

You can specify pcmk_host_list or pcmk_host_map to use a static target
list for the device. For example pcmk_host_list=redis1 would say this
fence device can target redis1 only. pcmk_host_map is the same but lets
you specify a different name for the target when calling the device --
for example, pcmk_host_map=redis1:1 would target redis1, but send just
"1" to the device.

> primitive stonith-redis2 stonith:fence_pve \
> params ipaddr=192.168.122.7 \
> params login="root@pam" passwd=secret port=101 \
> op start interval=0 timeout=60s \
> meta target-role=Started is-managed=true
> location loc_stonith-redis1 stonith-redis1 -inf: redis1
> location 

Re: [ClusterLabs] "No such device" with fence_pve agent

[Ken Gaillot]

On 03/23/2016 06:41 AM, Ferenc Wágner wrote:
> Ken Gaillot <kgail...@redhat.com> writes:
> 
>> There is a fence parameter pcmk_host_check that specifies how pacemaker
>> determines which fence devices can fence which nodes. The default is
>> dynamic-list, which means to run the fence agent's list command to get
>> the nodes.  [...]
>>
>> You can specify pcmk_host_list or pcmk_host_map to use a static target
>> list for the device.
> 
> I meant to research this, but now that you brought it up: does the
> default of pcmk_host_check automatically change to static-list if
> pcmk_host_list is defined?

If pcmk_host_check is specified, it is used;

otherwise, if pcmk_host_list and/or pcmk_host_map are specified,
pcmk_host_check=static-list;

otherwise, if the device supports the list or status commands,
pcmk_host_check=dynamic-list;

otherwise, pcmk_host_check=none.


> Does pcmk_host_map override pcmk_host_list?  Does it play together with
> pcmk_host_check=dynamic-list?

If pcmk_host_check=none, all devices can fence all nodes (i.e. if you
explicitly specify none, pcmk_host_list/pcmk_host_map are ignored);

otherwise, if pcmk_host_check=static-list, the values of pcmk_host_list
and/or pcmk_host_map are allowed targets (i.e. they combine if both are
specified, though that would be poor practice as it's confusing to read);

otherwise, the device is queried (i.e. static and dynamic never combine).

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Help required for N+1 redundancy setup

[Ken Gaillot]

On 03/16/2016 05:22 AM, Nikhil Utane wrote:
> I see following info gets updated in CIB. Can I use this or there is better
> way?
> 
>  crm-debug-origin="peer_update_callback" join="*down*" expected="member">

in_ccm/crmd/join reflect the current state of the node (as known by the
partition that you're looking at the CIB on), so if the node went down
and came back up, it won't tell you anything about being down.

- in_ccm indicates that the node is part of the underlying cluster layer
(heartbeat/cman/corosync)

- crmd indicates that the node is communicating at the pacemaker layer

- join indicates what phase of the join process the node is at

There's not a direct way to see what node went down after the fact.
There are ways however:

- if the node was running resources, those will be failed, and those
failures (including node) will be shown in the cluster status

- the logs show all node membership events; you can search for logs such
as "state is now lost" and "left us"

- "stonith -H $NODE_NAME" will show the fence history for a given node,
so if the node went down due to fencing, it will show up there

- you can configure an ocf:pacemaker:ClusterMon resource to run crm_mon
periodically and run a script for node events, and you can write the
script to do whatever you want (email you, etc.) (in the upcoming 1.1.15
release, built-in notifications will make this more reliable and easier,
but any script you use with ClusterMon will still be usable with the new
method)

> On Wed, Mar 16, 2016 at 12:40 PM, Nikhil Utane <nikhil.subscri...@gmail.com>
> wrote:
> 
>> Hi Ken,
>>
>> Sorry about the long delay. This activity was de-focussed but now it's
>> back on track.
>>
>> One part of question that is still not answered is on the newly active
>> node, how to find out which was the node that went down?
>> Anything that gets updated in the status section that can be read and
>> figured out?
>>
>> Thanks.
>> Nikhil
>>
>> On Sat, Jan 9, 2016 at 3:31 AM, Ken Gaillot <kgail...@redhat.com> wrote:
>>
>>> On 01/08/2016 11:13 AM, Nikhil Utane wrote:
>>>>> I think stickiness will do what you want here. Set a stickiness higher
>>>>> than the original node's preference, and the resource will want to stay
>>>>> where it is.
>>>>
>>>> Not sure I understand this. Stickiness will ensure that resources don't
>>>> move back when original node comes back up, isn't it?
>>>> But in my case, I want the newly standby node to become the backup node
>>> for
>>>> all other nodes. i.e. it should now be able to run all my resource
>>> groups
>>>> albeit with a lower score. How do I achieve that?
>>>
>>> Oh right. I forgot to ask whether you had an opt-out
>>> (symmetric-cluster=true, the default) or opt-in
>>> (symmetric-cluster=false) cluster. If you're opt-out, every node can run
>>> every resource unless you give it a negative preference.
>>>
>>> Partly it depends on whether there is a good reason to give each
>>> instance a "home" node. Often, there's not. If you just want to balance
>>> resources across nodes, the cluster will do that by default.
>>>
>>> If you prefer to put certain resources on certain nodes because the
>>> resources require more physical resources (RAM/CPU/whatever), you can
>>> set node attributes for that and use rules to set node preferences.
>>>
>>> Either way, you can decide whether you want stickiness with it.
>>>
>>>> Also can you answer, how to get the values of node that goes active and
>>> the
>>>> node that goes down inside the OCF agent?  Do I need to use
>>> notification or
>>>> some simpler alternative is available?
>>>> Thanks.
>>>>
>>>>
>>>> On Fri, Jan 8, 2016 at 9:30 PM, Ken Gaillot <kgail...@redhat.com>
>>> wrote:
>>>>
>>>>> On 01/08/2016 06:55 AM, Nikhil Utane wrote:
>>>>>> Would like to validate my final config.
>>>>>>
>>>>>> As I mentioned earlier, I will be having (upto) 5 active servers and 1
>>>>>> standby server.
>>>>>> The standby server should take up the role of active that went down.
>>> Each
>>>>>> active has some unique configuration that needs to be preserved.
>>>>>>
>>>>>> 1) So I will create total 5 groups. Each group has a
>>> "heartbeat::IPaddr2
>>>>>> resource (for virtual IP) an

Re: [ClusterLabs] attrd does not clean per-node cache after node removal

[Ken Gaillot]

On 03/23/2016 07:35 AM, Vladislav Bogdanov wrote:
> Hi!
> 
> It seems like atomic attrd in post-1.1.14 (eb89393) does not
> fully clean node cache after node is removed.

Is this a regression? Or have you only tried it with this version?

> After our QA guys remove node wa-test-server-ha-03 from a two-node cluster:
> * stop pacemaker and corosync on wa-test-server-ha-03
> * remove node wa-test-server-ha-03 from corosync nodelist on 
> wa-test-server-ha-04
> * tune votequorum settings
> * reload corosync on wa-test-server-ha-04
> * remove node from pacemaker on wa-test-server-ha-04
> * delete everything from /var/lib/pacemaker/cib on wa-test-server-ha-03
> , and then join it with the different corosync ID (but with the same node 
> name),
> we see the following in logs:
> 
> Leave node 1 (wa-test-server-ha-03):
> Mar 23 04:19:53 wa-test-server-ha-04 attrd[25962]:   notice: 
> crm_update_peer_proc: Node wa-test-server-ha-03[1] - state is now lost (was 
> member)
> Mar 23 04:19:53 wa-test-server-ha-04 attrd[25962]:   notice: Removing all 
> wa-test-server-ha-03 (1) attributes for attrd_peer_change_cb
> Mar 23 04:19:53 wa-test-server-ha-04 attrd[25962]:   notice: Lost attribute 
> writer wa-test-server-ha-03
> Mar 23 04:19:53 wa-test-server-ha-04 attrd[25962]:   notice: Removing 
> wa-test-server-ha-03/1 from the membership list
> Mar 23 04:19:53 wa-test-server-ha-04 attrd[25962]:   notice: Purged 1 peers 
> with id=1 and/or uname=wa-test-server-ha-03 from the membership cache
> Mar 23 04:19:56 wa-test-server-ha-04 attrd[25962]:   notice: Processing 
> peer-remove from wa-test-server-ha-04: wa-test-server-ha-03 0
> Mar 23 04:19:56 wa-test-server-ha-04 attrd[25962]:   notice: Removing all 
> wa-test-server-ha-03 (0) attributes for wa-test-server-ha-04
> Mar 23 04:19:56 wa-test-server-ha-04 attrd[25962]:   notice: Removing 
> wa-test-server-ha-03/1 from the membership list
> Mar 23 04:19:56 wa-test-server-ha-04 attrd[25962]:   notice: Purged 1 peers 
> with id=0 and/or uname=wa-test-server-ha-03 from the membership cache
> 
> Join node 3 (the same one, wa-test-server-ha-03, but ID differs):
> Mar 23 04:21:23 wa-test-server-ha-04 attrd[25962]: notice: 
> crm_update_peer_proc: Node wa-test-server-ha-03[3] - state is now member (was 
> (null))
> Mar 23 04:21:26 wa-test-server-ha-04 attrd[25962]:  warning: crm_find_peer: 
> Node 3/wa-test-server-ha-03 = 0x201bf30 - a4cbcdeb-c36a-4a0e-8ed6-c45b3db89296
> Mar 23 04:21:26 wa-test-server-ha-04 attrd[25962]:  warning: crm_find_peer: 
> Node 2/wa-test-server-ha-04 = 0x1f90e20 - 6c18faa1-f8c2-4b0c-907c-20db450e2e79
> Mar 23 04:21:26 wa-test-server-ha-04 attrd[25962]: crit: Node 1 and 3 
> share the same name 'wa-test-server-ha-03'

It took me a while to understand the above combination of messages. This
is not node 3 joining. This is node 1 joining after node 3 has already
been seen.

The warnings are a complete dump of the peer cache. So you can see that
wa-test-server-ha-03 is listed only once, with id 3.

The critical message ("Node 1 and 3") lists the new id first and the
found ID second. So id 1 is what it's trying to add to the cache.

Did you update the node ID in corosync.conf on *both* nodes?

> Mar 23 04:21:29 wa-test-server-ha-04 attrd[25962]:   notice: Node 
> 'wa-test-server-ha-03' has changed its ID from 1 to 3
> Mar 23 04:21:29 wa-test-server-ha-04 attrd[25962]:  warning: crm_find_peer: 
> Node 3/wa-test-server-ha-03 = 0x201bf30 - a4cbcdeb-c36a-4a0e-8ed6-c45b3db89296
> Mar 23 04:21:29 wa-test-server-ha-04 attrd[25962]:  warning: crm_find_peer: 
> Node 2/wa-test-server-ha-04 = 0x1f90e20 - 6c18faa1-f8c2-4b0c-907c-20db450e2e79
> Mar 23 04:21:29 wa-test-server-ha-04 attrd[25962]: crit: Node 1 and 3 
> share the same name 'wa-test-server-ha-03'
> Mar 23 04:21:31 wa-test-server-ha-04 attrd[25962]:   notice: Node 
> 'wa-test-server-ha-03' has changed its ID from 1 to 3
> Mar 23 04:21:31 wa-test-server-ha-04 attrd[25962]:  warning: crm_find_peer: 
> Node 3/wa-test-server-ha-03 = 0x201bf30 - a4cbcdeb-c36a-4a0e-8ed6-c45b3db89296
> Mar 23 04:21:31 wa-test-server-ha-04 attrd[25962]:  warning: crm_find_peer: 
> Node 2/wa-test-server-ha-04 = 0x1f90e20 - 6c18faa1-f8c2-4b0c-907c-20db450e2e79
> Mar 23 04:21:31 wa-test-server-ha-04 attrd[25962]: crit: Node 1 and 3 
> share the same name 'wa-test-server-ha-03'
> Mar 23 04:21:31 wa-test-server-ha-04 attrd[25962]:   notice: Node 
> 'wa-test-server-ha-03' has changed its ID from 3 to 1
> ...
> 
> On the node being joined:
> Mar 23 04:21:23 wa-test-server-ha-03 attrd[15260]:   notice: Connecting to 
> cluster infrastructure: corosync
> Mar 23 04:21:23 wa-test-server-ha-03 attrd[15260]:   notice: 
> crm_update_peer_proc: Node wa-test-server-ha-03[3] - state is now member (was 
> (null))
> Mar 23 04:21:24 wa-test-server-ha-03 attrd[15260]:   notice: 
> crm_update_peer_proc: Node wa-test-server-ha-04[2] - state is now member (was 
> (null))
> Mar 23 04:21:24 wa-test-server-ha-03 attrd[15260]:   notice: Recorded 
> 

Re: [ClusterLabs] pacemaker remote configuration on ubuntu 14.04

[Ken Gaillot]

On 03/07/2016 09:10 PM, Сергей Филатов wrote:
> Thanks for an answer. Turned out the problem was not in ipv6.
> Remote node is listening on 3121 port and it’s name is resolving fine.
> Got authkey file at /etc/pacemaker on both remote and cluster nodes.
> What can I check in addition? Is there any walkthrough for ubuntu?

Nothing specific to ubuntu, but there's not much distro-specific to it.

If you "ssh -p 3121" to the remote node from a cluster node, what do you
get?

pacemaker_remote will use the usual log settings for pacemaker (probably
/var/log/pacemaker.log, probably configured in /etc/default/pacemaker on
ubuntu). You should see "New remote connection" in the remote node's log
when the cluster tries to connect, and "LRMD client connection
established" if it's successful.

As always, check for firewall and SELinux issues.

> 
>> On 07 Mar 2016, at 09:40, Ken Gaillot <kgail...@redhat.com> wrote:
>>
>> On 03/06/2016 07:43 PM, Сергей Филатов wrote:
>>> Hi,
>>> I’m trying to set up pacemaker_remote resource on ubuntu 14.04
>>> I followed "remote node walkthrough” guide 
>>> (http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html-single/Pacemaker_Remote/#idm140473081667280
>>>  
>>> <http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html-single/Pacemaker_Remote/#idm140473081667280>)
>>> After creating ocf:pacemaker:remote resource on cluster node, remote node 
>>> doesn’t show up as online.
>>> I guess I need to configure remote agent to listen on ipv4, where can I 
>>> configure it?
>>> Or is there any other steps to set up remote node besides the ones 
>>> mentioned in guide?
>>> tcp6   0  0 :::3121 :::*LISTEN  
>>> 21620/pacemaker_rem off (0.00/0/0)
>>>
>>> pacemaker and pacemaker_remote are 1.12 version
>>
>>
>> pacemaker_remote will try to bind to IPv6 addresses first, and only if
>> that fails, will it bind to IPv4. There is no way to configure this
>> behavior currently, though it obviously would be nice to have.
>>
>> The only workarounds I can think of are to make IPv6 connections work
>> between the cluster and the remote node, or disable IPv6 on the remote
>> node. Using IPv6, there could be an issue if your name resolution
>> returns both IPv4 and IPv6 addresses for the remote host; you could
>> potentially work around that by adding an IPv6-only name for it, and
>> using that as the server option to the remote resource.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Connectivity is degraded (Expected=300)

[Ken Gaillot]

On 03/01/2016 04:33 PM, Rafał Sanocki wrote:
> but my score from pingd should be 200 because my config is
> 
> node nodeA
> node nodeB
> primitive failover-ip1 ocf:pacemaker:node \
> op monitor interval="100s" \
> op start interval="0" timeout="2m" \
> op stop interval="0" timeout="1m" \
> meta target-role="Started"
> primitive my-arping ocf:pacemaker:arping \
> params name="arping" dampen="30s" multiplier="100"
> host_list="neteth1;192.168.1.10" \
> op monitor interval="5s"
> primitive my-pingd ocf:pacemaker:pingd \
> params name="pingd" dampen="15s" multiplier="100"
> host_list="192.168.2.10  193.168.10.50 " interval="2"
> clone my-conn my-arping \
> meta globally-unique="false" clone-max="2" clone-node-max="1"
> target-role="Started"
> clone my-connp my-pingd \
> meta globally-unique="false" clone-max="2" clone-node-max="1"
> dampen="15s"
> location pref-ip1 failover-ip1 \
> rule $id="pref-failover-ip1-rule" pingd: defined pingd \
> rule $id="pref-failover-ip1-rule2" arping: defined arping
> property $id="cib-bootstrap-options" \
> dc-version="1.0.11-614a4bc7ea748b7583fc7fe164eddbcdf33b03ba" \

Hi Rafał,

1.0.11 is old (2011) and no longer supported, and there have been 14
releases in the 1.1 line. I have no idea whether the pingd agent from
that works or not. In recent versions, the ocf:pacemaker:pingd agent is
obsolete, and is now just a wrapper for ocf:pacemaker:ping.

If you have any possibility of upgrading, I would strongly recommend it.
You would also need to switch from openais to corosync >=2, and libqb >=
0.17.0. Most Linux distributions have recent versions in their regular
package repositories.

> cluster-infrastructure="openais" \
> stonith-enabled="false" \
> no-quorum-policy="ignore" \
> expected-quorum-votes="2" \
> cluster-delay="30s" \
> symmetric-cluster="true" \
> last-lrm-refresh="1373962109"
> rsc_defaults $id="rsc-options" \
> resource-stickiness="0"
> op_defaults $id="op_defaults-options" \
> timeout="20s"
> 
> 2xhost x 100 = 200 score soo why he expect 300, is that 100 from arping
> expecting?
>
> Rafał
> 
> 
> 
> 
> 
> 
> W dniu 2016-03-01 o 16:09, Ken Gaillot pisze:
>> On 03/01/2016 08:24 AM, Rafał Sanocki wrote:
>>> Hello
>>> Can you tell if that message is correct "
>>>
>>>   #crm_mon -A
>>> Online: [ nodeA nodeB ]
>>> failover-ip1(ocf::pacemaker:wall):  Started nodeB
>>>   Clone Set: my-conn
>>>   Started: [ nodeA nodeB ]
>>>   Clone Set: my-connp
>>>   Started: [ nodeA nodeB ]
>>> Node Attributes:
>>> * Node nodeA:
>>>  + arping: 0
>>>  + pingd : 200   : Connectivity
>>> is degraded (Expected=300)
>> crm_mon prints messages like this for ping/pingd resources. The expected
>> value is the number of nodes in the resource's host_list parameter
>> multiplied by its multiplier parameter (default 1 if not specified). If
>> the actual value is less than the expected value, it will print the
>> message.
>>
>> Whether that's "correct" depends on what you mean by that. It's correct
>> in that the actual value is less than the expected value.
>>
>>> * Node nodeB:
>>>  + arping: 100
>>>  + pingd : 200   : Connectivity
>>> is degraded (Expected=300)
>>
>> ___
>> Users mailing list: Users@clusterlabs.org
>> http://clusterlabs.org/mailman/listinfo/users
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
> 


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] ELEMENTARY :: Please Help :: Getting error when building Pacemaker-1.1 from source

[Ken Gaillot]

On 03/01/2016 11:24 PM, Sharat Joshi wrote:
> Hi List Folk,
> 
> I am very new to Pacemaker and I am trying to build using sources.
> After successfully installing libqb and corosync under
> /disk1/software/libqb and /disk1/software/corosync and setting
> 
> $ export
> PKG_CONFIG_PATH=/disk1/software/libqb/lib/pkgconfig:/disk1/software/corosync/lib/pkgconfig
> $./autogen.sh && ./configure --prefix=/disk1/software/pacemaker
> 
> I try building the final peice Pacemaker and I get the following...
> 
> configure: WARNING: Unable to support Corosync: corosync/libqb IPC
> libraries not found by pkg_config
> 
> checking for supported stacks... configure: error: in
> `/disk1/software/pacemaker-1.1':
> 
> configure: error: You must support at least one cluster stack (heartbeat or
> corosync)
> 
> 
> I am stuck in this error since a day now, Please help.
> 
> Thanks,
> 
> Sharat Joshi


Hi Sharat,

A few things to check:

Are you using recent versions of corosync (>=2) and libqb (>=0.17.0)?

Is a valid libqb.pc generated in /disk1/software/libqb/lib/pkgconfig?

If you run "pkg-config --cflags --libs libqb" manually after setting
PKG_CONFIG_PATH, does it print output?

Check the ./configure output for "checking for libqb..." and see if
there are any errors around there.

One thing to watch out for: you can install libqb, corosync, and
pacemaker anywhere, but the OCF standard mandates that resource agents
are installed under /usr/lib/ocf, and that path is hardcoded in multiple
places.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] crm_mon change in behaviour PM 1.1.12 -> 1.1.14: crm_mon -XA filters #health.* node attributes

[Ken Gaillot]

On 03/03/2016 10:07 AM, Martin Schlegel wrote:
> Hello everybody
> 
> 
> This is my first post on this mailing list and I am only using Pacemaker 
> since 
> fall 2015 ... please be gentle :-) and I will do the same.
> 
> 
> Our cluster is using multiple resource agents that update various node 
> attributes (via RAs like sysinfo, healthcpu, etc.) in the form of #health.* 
> and 
> we rely on the mechanism enabled via the 
> property node-health-strategy=migrate-on-red to trigger resource migrations.
> 
> In Pacemaker version 1.1.12 crm_mon -A or -XA would still display these 
> #health.* attributes, but not since we have moved up to 1.1.14 and I am not 
> sure 
> why this needed to be changed :
> 
> root@ys0-resiliency-test-1:~# crm node status-attr pg1 show \#health-cpu
> *scope=status name=#health-cpu value=green*
> 
> root@ys0-resiliency-test-1:~# crm_mon -XrRAf1 | grep -i '#health' ; echo $?
> *1*
> 
> *
> *
> 
> This seems to be due to this part of the crm_mon.c code:
> 
> /* Never display node attributes whose name starts with one of these prefixes 
> */
> #define FILTER_STR { "shutdown", "terminate", "standby", "fail-count", \
>  "last-failure", "probe_complete", *“#”*, 
> NULL }

You are correct. The goal is to hide attributes that are sort of
"internal" to the cluster or resource agents, and just show attributes
set by the user.

> *
> *
> 
> I would like to know if anybody is sharing my opinions on that:
> 
> 1. From an operations point of view it would be easier to get crm_mon to 
> include #health.* in the general output or at least in the XML output via 
> crm_mon -XA, so that I can get a comprehensive status view in one shot.

It does make sense to include all attributes in XML output since that is
intended for automated parsing. It should be fairly easy to move the
filtering from create_attr_list() to print_node_attribute().

> 2. Because the node attributes list can be extensive and clutters up the 
> output it would make sense to allow a user-defined filter for node attributes 
> in 
> general

Feel free to open a feature request on bugs.clusterlabs.org. We also
gladly accept code submissions at https://github.com/ClusterLabs/pacemaker

> 
> Regards,
> 
> Martin Schlegel


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Removing node from pacemaker.

[Ken Gaillot]

On 03/03/2016 06:04 AM, Debabrata Pani wrote:
> http://clusterlabs.org/doc/en-US/Pacemaker/1.0/html/Pacemaker_Explained/s-n
> ode-delete.html
> 
> 
> 
> Are we missing the deletion of the nodes from the cib ?

That documentation is old; crm_node -R does remove the node from the CIB.

> Regards,
> Debabrata 
> 
> 
> From:  Somanath Jeeva 
> Reply-To:  Cluster Labs - All topics related to open-source clustering
> welcomed 
> Date:  Thursday, 3 March 2016 17:14
> To:  "users@clusterlabs.org" 
> Subject:  [ClusterLabs]  Removing node from pacemaker.
> 
> 
> Hi,
>  
> I am trying to remove a node from the pacemaker¹/corosync cluster, using
> the command ³crm_node -R dl360x4061 ­force².

The above command removes the node from pacemaker itself, but not the
underlying cluster layer, which should be done first.

High-level tools often offer simple commands to handle both the
pacemaker and cluster layer parts for you. For pcs, it's "pcs cluster
node remove ".

Otherwise, the process for removing the node from the cluster layer
varies by what you're using -- heartbeat, corosync 1 with plugin,
corosync 1 with CMAN, or corosync 2. Generally, you want to stop the
cluster software on the node to be removed first, then remove the node
from the layer's configuration on the remaining nodes if it is
explicitly specified there.

> Though this command removes the node from the cluster, it is appearing as
> offline after pacemaker/corosync restart in the nodes that are online.
>  
> Is there any other command to completely delete the node from the
> pacemaker/corosync cluster.
>  
> Pacemaker and Corosync Versions.
> PACEMAKER=1.1.10
> COROSYNC=1.4.1
>  
> Regards
> Somanath Thilak J


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] pacemaker remote configuration on ubuntu 14.04

[Ken Gaillot]

On 03/06/2016 07:43 PM, Сергей Филатов wrote:
> Hi,
> I’m trying to set up pacemaker_remote resource on ubuntu 14.04
> I followed "remote node walkthrough” guide 
> (http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html-single/Pacemaker_Remote/#idm140473081667280
>  
> )
> After creating ocf:pacemaker:remote resource on cluster node, remote node 
> doesn’t show up as online.
> I guess I need to configure remote agent to listen on ipv4, where can I 
> configure it?
> Or is there any other steps to set up remote node besides the ones mentioned 
> in guide?
> tcp6   0  0 :::3121 :::*LISTEN
>   21620/pacemaker_rem off (0.00/0/0)
> 
> pacemaker and pacemaker_remote are 1.12 version


pacemaker_remote will try to bind to IPv6 addresses first, and only if
that fails, will it bind to IPv4. There is no way to configure this
behavior currently, though it obviously would be nice to have.

The only workarounds I can think of are to make IPv6 connections work
between the cluster and the remote node, or disable IPv6 on the remote
node. Using IPv6, there could be an issue if your name resolution
returns both IPv4 and IPv6 addresses for the remote host; you could
potentially work around that by adding an IPv6-only name for it, and
using that as the server option to the remote resource.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Issues with crm_mon or ClusterMon resource agent

[Ken Gaillot]

On 03/06/2016 08:36 AM, Debabrata Pani wrote:
> Hi,
> 
> I would like to understand if anybody has got this working recently.
> 
> Looks like I have missed something in the description and hence the
> problem statement is not clear to the group.
> 
> Can I enable some logs in crm_mon to improve the description of the
> problem ?
> 
> Regards,
> Debabrata Pani
> 
> 
> 
> On 04/03/16 11:48, "Debabrata Pani"  wrote:
> 
>> Hi,
>>
>> I wanted  to configure ClusterMon resource agent so that I can get
>> information about events in the pacemaker cluster.
>> *Objective is to generate traps for some specific resource agents and/or
>> conditions*
>>
>>
>> My cluster installation details :
>> pacemakerd --version
>> Pacemaker 1.1.11
>> Written by Andrew Beekhof
>>
>>
>> corosync -v
>> Corosync Cluster Engine, version '1.4.7'
>> Copyright (c) 2006-2009 Red Hat, Inc.
>>
>>
>> crm_mon --version
>> Pacemaker 1.1.11
>> Written by Andrew Beekhof
>>
>>
>>
>> I followed the following documentation
>> http://clusterlabs.org/doc/en-US/Pacemaker/1.1-crmsh/html/Pacemaker_Explai
>> n
>> ed/ch07.html
>> http://floriancrouzat.net/2013/01/monitor-a-pacemaker-cluster-with-ocfpace
>> m
>> akerclustermon-andor-external-agent/
>>
>> My current cluster status shows:
>>
>> MonitorClusterChange(ocf::pacemaker:ClusterMon):Started stvm4
>>
>> The resource agent is configure
>>
>> The crm_mon is indeed running as a daemon on the node stvm4
>>
>> [root@stvm4 panidata]# ps -ef | grep crm_mon | grep -v grep
>> root  2908 1  0 10:59 ?00:00:00 /usr/sbin/crm_mon -p
>> /tmp/ClusterMon_MonitorClusterChange.pid -d -i 0 -E
>> /root/panidata/testscript.sh -e anonymous -h
>> /tmp/ClusterMon_MonitorClusterChange.html

The "-i 0" refers to the "update" property of your resource, which is
how often crm_mon should recheck the cluster. I'm not sure what zero
would do, but it would be better around 10-30 seconds.

Is /root/panidata/testscript.sh executable? Does it work when run from
the command line?

>> My test script is the following
>> #!/bin/bash
>> echo "running" >> /root/running.log
>> echo "CRM_notify_recipient=$CRM_notify_recipient"
>> ..
>>
>>
>>
>> As I trigger events by shutting down one or the other service, I see the
>> html file "/tmp/ClusterMon_MonitorClusterChange.html² getting updated each
>> time an event is triggered.
>> So the timestamp of the file keeps on changing.
>>
>> But I am not sure if the script is getting executed. Because I don¹t see
>> any ³/root/running.log² file.
>>
>> Things I have tried:
>> * Using ³logger² command instead of echo.
>> * Running the crm_mon command with -d and other parameters manually to
>> check , if it is the problem with resource agent etc.
>>
>> Queries:
>> * Is this a know issue ?
>> * Am I doing something incorrect ?
>>
>>
>>
>>
>> Regards,
>> Debabrata Pani


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Connectivity is degraded (Expected=300)

[Ken Gaillot]

On 03/01/2016 08:24 AM, Rafał Sanocki wrote:
> Hello
> Can you tell if that message is correct "
> 
>  #crm_mon -A
> Online: [ nodeA nodeB ]
> failover-ip1(ocf::pacemaker:wall):  Started nodeB
>  Clone Set: my-conn
>  Started: [ nodeA nodeB ]
>  Clone Set: my-connp
>  Started: [ nodeA nodeB ]
> Node Attributes:
> * Node nodeA:
> + arping: 0
> + pingd : 200   : Connectivity
> is degraded (Expected=300)

crm_mon prints messages like this for ping/pingd resources. The expected
value is the number of nodes in the resource's host_list parameter
multiplied by its multiplier parameter (default 1 if not specified). If
the actual value is less than the expected value, it will print the message.

Whether that's "correct" depends on what you mean by that. It's correct
in that the actual value is less than the expected value.

> * Node nodeB:
> + arping: 100
> + pingd : 200   : Connectivity
> is degraded (Expected=300)


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] DRBD on asymmetric-cluster

[Ken Gaillot]

On 04/02/2016 01:16 AM, Jason Voorhees wrote:
> Hello guys:
> 
> I've been recently reading "Pacemaker - Clusters from scratch" and
> working on a CentOS 7 system with pacemaker 1.1.13, corosync-2.3.4 and
> drbd84-utils-8.9.5.
> 
> The PDF instructs how to create a DRBD resource that seems to be
> automatically started due to a symmetric-cluster setup.
> 
> However I want to setup an asymmetric-cluster/opt-in
> (symmetric-cluster=false) but I don't know how to configure a
> constraint to prefer node1 over node2 to start my DRBD resource as
> Master (Primary).

I thought location constraints supported role, but that isn't
documented, so I'm not sure. But it is documented with regard to rules,
which using pcs might look like:

pcs location clusterdataClone rule \
  role=master \
  score=50 \
  '#uname' eq nodo1

For a lower-level explanation of rules, see
http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#idm140617356537136

> So far this are my resources and constraints:
> 
> [root@nodo1 ~]# pcs resource
>  IPService  (ocf::heartbeat:IPaddr2):   Started nodo1
>  Web(systemd:httpd):Started nodo1
>  Master/Slave Set: clusterdataClone [clusterdata]
>  Stopped: [ nodo1 nodo2 ]
> 
> [root@nodo1 ~]# pcs constraint
> Location Constraints:
>   Resource: IPService
> Enabled on: nodo2 (score:50)
> Enabled on: nodo1 (score:100)
>   Resource: Web
> Enabled on: nodo2 (score:50)
> Enabled on: nodo1 (score:100)
> Ordering Constraints:
>   start IPService then start Web (kind:Mandatory)
> Colocation Constraints:
>   Web with IPService (score:INFINITY)
> 
> My current DRBD status:
> 
> [root@nodo1 ~]# drbdadm role clusterdb
> 0: Failure: (127) Device minor not allocated
> additional info from kernel:
> unknown minor
> Command 'drbdsetup-84 role 0' terminated with exit code 10
> 
> 
> [root@nodo2 ~]# drbdadm role clusterdb
> 0: Failure: (127) Device minor not allocated
> additional info from kernel:
> unknown minor
> Command 'drbdsetup-84 role 0' terminated with exit code 10
> 
> 
> I know that it's possible to configure my cluster as asymmetric and
> use constraints to avoid a resource running (or becoming master) on
> certain nodes, but this time I would like to learn how to do it with
> an opt-in scenario.
> 
> Thanks in advance for your help.
> 
> P.D. nodo1 & nodo2 are spanish names for node1 and node2
> 
> ___
> Users mailing list: Users@clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
> 


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] cloned pingd resource problem

[Ken Gaillot]

On 03/30/2016 08:38 AM, fatcha...@gmx.de wrote:
> Hi,
> 
> I`m running a two node cluster on a fully updated CentOS 7 
> (pacemaker-1.1.13-10.el7_2.2.x86_64 pcs-0.9.143-15.el7.x86_64) . I see on one 
> of our nodes a lot of this in the logfiles:
> 
> Mar 30 12:32:13 localhost crmd[12986]:  notice: State transition S_IDLE -> 
> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED 
> origin=crm_timer_popped ]
> Mar 30 12:32:13 localhost pengine[12985]:  notice: On loss of CCM Quorum: 
> Ignore
> Mar 30 12:32:13 localhost pengine[12985]: warning: Processing failed op 
> monitor for ping_fw:0 on kathie2: unknown error (1)
> Mar 30 12:32:13 localhost pengine[12985]: warning: Processing failed op start 
> for ping_fw:1 on stacy2: unknown error (1)
> Mar 30 12:32:13 localhost pengine[12985]: warning: Forcing ping_fw-clone away 
> from stacy2 after 100 failures (max=100)
> Mar 30 12:32:13 localhost pengine[12985]: warning: Forcing ping_fw-clone away 
> from stacy2 after 100 failures (max=100)

Pacemaker monitors the resource by calling its resource agent's status
action every 45 seconds. The first warning above indicates that the
resource agent returned a generic error code on kathie2, which in this
case (ocf:pacemaker:ping) means that the specified IP (192.168.16.1) did
not respond to ping.

The second warning indicates that the instance on stacy2 failed to
start, which again in this case means that the IP did not respond to a
ping from that node. The last two warnings indicate that pacemaker
retried the start continuously and eventually gave up.

> Mar 30 12:32:13 localhost pengine[12985]:  notice: Calculated Transition 
> 1823: /var/lib/pacemaker/pengine/pe-input-355.bz2
> Mar 30 12:32:13 localhost crmd[12986]:  notice: Transition 1823 (Complete=0, 
> Pending=0, Fired=0, Skipped=0, Incomplete=0, 
> Source=/var/lib/pacemaker/pengine/pe-input-355.bz2): Complete
> Mar 30 12:32:13 localhost crmd[12986]:  notice: State transition 
> S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL 
> origin=notify_crmd ]
> 
> 
> The configuration looks like this:
> 
> Clone: ping_fw-clone
>   Resource: ping_fw (class=ocf provider=pacemaker type=ping)
>Attributes: dampen=5s multiplier=1000 host_list=192.168.16.1 timeout=60
>Operations: start interval=0s timeout=60 (ping_fw-start-interval-0s)
>stop interval=0s timeout=20 (ping_fw-stop-interval-0s)
>monitor interval=45 (ping_fw-monitor-interval-45)
> 
> 
> What can I do to resolve the problem ? 

The problem is that ping from the nodes to 192.168.16.1 does not always
work. This could be expected in your environment, or could indicate a
networking issue. But it's outside pacemaker's control; pacemaker is
simply monitoring it and reporting when there's a problem.

> Any suggestions are welcome
> 
> Kind regards
> 
> fatcharly


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] service flap as nodes join and leave

[Ken Gaillot]

On 04/13/2016 11:23 AM, Christopher Harvey wrote:
> I have a 3 node cluster (see the bottom of this email for 'pcs config'
> output) with 3 nodes. The MsgBB-Active and AD-Active service both flap
> whenever a node joins or leaves the cluster. I trigger the leave and
> join with a pacemaker service start and stop on any node.

That's the default behavior of clones used in ordering constraints. If
you set interleave=true on your clones, each dependent clone instance
will only care about the depended-on instances on its own node, rather
than all nodes.

See
http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#_clone_options

While the interleave=true behavior is much more commonly used,
interleave=false is the default because it's safer -- the cluster
doesn't know anything about the cloned service, so it can't assume the
service is OK with it. Since you know what your service does, you can
set interleave=true for services that can handle it.

> Here is the happy steady state setup:
> 
> 3 nodes and 4 resources configured
> 
> Online: [ vmr-132-3 vmr-132-4 vmr-132-5 ]
> 
>  Clone Set: Router-clone [Router]
>  Started: [ vmr-132-3 vmr-132-4 ]
> MsgBB-Active(ocf::solace:MsgBB-Active): Started vmr-132-3
> AD-Active   (ocf::solace:AD-Active):Started vmr-132-3
> 
> [root@vmr-132-4 ~]# supervisorctl stop pacemaker
> no change, except vmr-132-4 goes offline
> [root@vmr-132-4 ~]# supervisorctl start pacemaker
> vmr-132-4 comes back online
> MsgBB-Active and AD-Active flap very quickly (<1s)
> Steady state is resumed.
> 
> Why should the fact that vmr-132-4 coming and going affect the service
> on any other node?
> 
> Thanks,
> Chris
> 
> Cluster Name:
> Corosync Nodes:
>  192.168.132.5 192.168.132.4 192.168.132.3
> Pacemaker Nodes:
>  vmr-132-3 vmr-132-4 vmr-132-5
> 
> Resources:
>  Clone: Router-clone
>   Meta Attrs: clone-max=2 clone-node-max=1
>   Resource: Router (class=ocf provider=solace type=Router)
>Meta Attrs: migration-threshold=1 failure-timeout=1s
>Operations: start interval=0s timeout=2 (Router-start-timeout-2)
>stop interval=0s timeout=2 (Router-stop-timeout-2)
>monitor interval=1s (Router-monitor-interval-1s)
>  Resource: MsgBB-Active (class=ocf provider=solace type=MsgBB-Active)
>   Meta Attrs: migration-threshold=2 failure-timeout=1s
>   Operations: start interval=0s timeout=2 (MsgBB-Active-start-timeout-2)
>   stop interval=0s timeout=2 (MsgBB-Active-stop-timeout-2)
>   monitor interval=1s (MsgBB-Active-monitor-interval-1s)
>  Resource: AD-Active (class=ocf provider=solace type=AD-Active)
>   Meta Attrs: migration-threshold=2 failure-timeout=1s
>   Operations: start interval=0s timeout=2 (AD-Active-start-timeout-2)
>   stop interval=0s timeout=2 (AD-Active-stop-timeout-2)
>   monitor interval=1s (AD-Active-monitor-interval-1s)
> 
> Stonith Devices:
> Fencing Levels:
> 
> Location Constraints:
>   Resource: AD-Active
> Disabled on: vmr-132-5 (score:-INFINITY) (id:ADNotOnMonitor)
>   Resource: MsgBB-Active
> Enabled on: vmr-132-4 (score:100) (id:vmr-132-4Priority)
> Enabled on: vmr-132-3 (score:250) (id:vmr-132-3Priority)
> Disabled on: vmr-132-5 (score:-INFINITY) (id:MsgBBNotOnMonitor)
>   Resource: Router-clone
> Disabled on: vmr-132-5 (score:-INFINITY) (id:RouterNotOnMonitor)
> Ordering Constraints:
>   Resource Sets:
> set Router-clone MsgBB-Active sequential=true
> (id:pcs_rsc_set_Router-clone_MsgBB-Active) setoptions kind=Mandatory
> (id:pcs_rsc_order_Router-clone_MsgBB-Active)
> set MsgBB-Active AD-Active sequential=true
> (id:pcs_rsc_set_MsgBB-Active_AD-Active) setoptions kind=Mandatory
> (id:pcs_rsc_order_MsgBB-Active_AD-Active)
> Colocation Constraints:
>   MsgBB-Active with Router-clone (score:INFINITY)
>   (id:colocation-MsgBB-Active-Router-clone-INFINITY)
>   AD-Active with MsgBB-Active (score:1000)
>   (id:colocation-AD-Active-MsgBB-Active-1000)
> 
> Resources Defaults:
>  No defaults set
> Operations Defaults:
>  No defaults set
> 
> Cluster Properties:
>  cluster-infrastructure: corosync
>  cluster-recheck-interval: 1s
>  dc-version: 1.1.13-10.el7_2.2-44eb2dd
>  have-watchdog: false
>  maintenance-mode: false
>  start-failure-is-fatal: false
>  stonith-enabled: false


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Monitoring action of Pacemaker resources fail because of high load on the nodes

[Ken Gaillot]

On 04/22/2016 01:13 PM, Dimitri Maziuk wrote:
> On 04/22/2016 12:58 PM, Ken Gaillot wrote:
> 
>>> Consider that monitoring - at least as part of the action -
>>> should check if what your service is actually providing is
>>> working according to some functional and nonfunctional
>>> constraints as to simulate the experience of the consumer of
>>> your services.
> 
> Goedel and Turing say the only one who can answer that is the
> actual consumer. So a simple check for what you *can* check would
> be very nice indeed.
> 
>> Also, you can provide multiple levels of monitoring:
>> 
>> http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#_multiple_monitor_operations
>>
>>
>> 
>> For example, you could provide a very simple check that just makes sure
>> MySQL is responding on its port, and run that frequently with a
>> low timeout. And your existing thorough monitor could be run less
>> frequently with a high timeout.
> 
> Looking at this, it seems you have to actually rewrite the RA to
> switch on $OCF_CHECK_LEVEL -- unless the stock RA already provides
> the "simple check" you need, is that correct?
> 
> E.g. this page:
> http://linux-ha.org/doc/man-pages/re-ra-apache.html suggests that
> apache RA does not and all you can do in practice is run the same
> curl http:/localhost/server-status check with different 
> frequencies. Would that be what we actually have ATM?

Correct, you would need to customize the RA. Given how long you said a
check can take, I assumed you already had a custom check that did
something more detailed than the stock mysql RA.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] Pacemaker 1.1.15 - Release Candidate 1

[Ken Gaillot]

ClusterLabs is happy to announce the first release candidate for
Pacemaker version 1.1.15. Source code is available at:

https://github.com/ClusterLabs/pacemaker/releases/tag/Pacemaker-1.1.15-rc1

The most significant enhancements in this release are:

* A new "alerts" section of the CIB allows you to configure scripts that
will be called after significant cluster events. (For details, see the
recent "Coming in 1.1.15: Event-driven alerts" thread on this mailing list.)

* A new pcmk_action_limit option for fence devices allows multiple fence
actions to be executed concurrently. It defaults to 1 to preserve
existing behavior (i.e. serial execution of fence actions).

* Pacemaker Remote support has been improved. Most noticeably, if
pacemaker_remote is stopped without disabling the remote resource first,
any resources will be moved off the node (previously, the node would get
fenced). This allows easier software updates on remote nodes, since
updates often involve restarting the daemon.

* You may notice some files have moved from the pacemaker package to
pacemaker-cli, including most ocf:pacemaker resource agents, the
logrotate configuration, the XML schemas and the SNMP MIB. This allows
Pacemaker Remote nodes to work better when the full pacemaker package is
not installed.

* Have you ever wondered why a resource is not starting when you think
it should? crm_mon will now show why a resource is stopped, for example,
because it is unmanaged, or disabled in the configuration.

* Three significant regressions have been fixed. Compressed CIBs larger
than 1MB are again supported (a regression since 1.1.14), fenced unseen
nodes properly are not marked as unclean (also a regression since
1.1.14), and failures of multiple-level monitor checks should again
cause the resource to fail (a regression since 1.1.10).

As usual, the release includes many bugfixes and minor enhancements. For
a more detailed list of changes, see the change log:

https://github.com/ClusterLabs/pacemaker/blob/1.1/ChangeLog

Everyone is encouraged to download, compile and test the new release. We
do many regression tests and simulations, but we can't cover all
possible use cases, so your feedback is important and appreciated.

Many thanks to all contributors of source code to this release,
including Andrew Beekhof, Bin Liu, David Shane Holden, Ferenc Wágner,
Gao Yan, Hideo Yamauchi, Jan Pokorný, Ken Gaillot, Klaus Wenninger,
Kristoffer Grönlund, Lars Ellenberg, Michal Koutný, Nakahira Kazutomo,
Ruben Kerkhof, and Yusuke Iida. Apologies if I have overlooked anyone.
-- 
Ken Gaillot <kgail...@redhat.com>

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Coming in 1.1.15: Event-driven alerts

[Ken Gaillot]

On 04/22/2016 02:43 AM, Klaus Wenninger wrote:
> On 04/22/2016 08:16 AM, Ulrich Windl wrote:
>>>>> Ken Gaillot <kgail...@redhat.com> schrieb am 21.04.2016 um 19:50 in 
>>>>> Nachricht
>> <571912f3.2060...@redhat.com>:
>>
>> [...]
>>> The alerts section can have any number of alerts, which look like:
>>>
>>>>>   path="/srv/pacemaker/pcmk_alert_sample.sh">
>>>
>>>   >>  value="/var/log/cluster-alerts.log" />
>>>
>>>
>> Are there any parameters supplied for the script? For the XML: I think 
>> "path" for the script to execute is somewhat generic: Why not call it "exec" 
>> or something like that? Likewise for "value": Isn't "logfile" a better name?
> exec has a certain appeal...
> but recipient can actually be anything like email-address, logfile, ... so
> keeping it general like value makes sense in my mind
>>
>>> As always, id is simply a unique label for the entry. The path is an
>>> arbitrary file path to an alert script. Existing external scripts used
>>> with ClusterMon resources will work as alert scripts, because the
>>> interface is compatible.
>>>
>>> We intend to provide sample scripts in the extra/alerts source
>>> directory. The existing pcmk_notify_sample.sh script has been moved
>>> there (as pcmk_alert_sample.sh), and so has pcmk_snmp_helper.sh.
>>>
>>> Each alert may have any number of recipients configured. These values
>> What I did not understand is how an "alert" is related to some cluster 
>> "event": By ID, or by some explict configuration?
> There are "node", "fencing" and "resource" (CRM_alert_kind tells you
> if you want to know inside a script) alerts and alerts was chosen
> as it is in sync with other frameworks like nagios, ... but you can choose
> it a synonym for event ... meaning it is not necessarily anything bad
> or good just something you might be interested in.
> 
> You get set a bunch of environment variables when your executable is
> called you can use to get more info and add intelligence if you like:
> 
> CRM_alert_node, CRM_alert_nodeid, CRM_alert_rsc, CRM_alert_task,
> CRM_alert_interval, CRM_alert_desc, CRM_alert_status,
> CRM_alert_target_rc, CRM_alert_rc, CRM_alert_kind,
> CRM_alert_version, CRM_alert_node_sequence
> CRM_alert_timestamp
> 
> Referencing is done via node-names, resource-ids as throughout
> the pacemaker-config in the cib.
> 
> 
>>
>>> will simply be passed to the script as arguments. The first recipient
>>> will also be passed as the CRM_alert_recipient environment variable, for
>>> compatibility with existing scripts that only support one recipient.
>>> (All CRM_alert_* variables will also be passed as CRM_notify_* for
>>> compatibility with existing ClusterMon scripts.)
>>>
>>> An alert may also have instance attributes and meta-attributes, for example:
>>>
>>>>>   path="/srv/pacemaker/pcmk_alert_sample.sh">
>>>
>>>   
>>>  
>>>   
>>>
>>>   
>>> 
>>> 
>>>   
>>>
>>>   >>  value="/var/log/cluster-alerts.log" />
>>>
>>>
>>>
>>> The meta-attributes are optional properties used by the cluster.
>>> Currently, they include "timeout" (which defaults to 30s) and
>>> "tstamp_format" (which defaults to "%H:%M:%S.%06N", and is a
>>> microsecond-resolution timestamp provided to the alert script as the
>>> CRM_alert_timestamp environment variable).
>>>
>>> The instance attributes are arbitrary values that will be passed as
>>> environment variables to the alert script. This provides you a
>>> convenient way to configure your scripts in the cluster, so you can
>>> easily reuse them.
>> At the moment this sounds quite abstract, yet.
> meta-attributes and instance-attributes as used as with
> resources, where meta-attributes reflect config-parameters
> you pass rather to pacemaker like in this case for the timeout
> observation when the script is executed, and the format
> string that tells pacemaker in which style you would like
> CRM_alert_timestamp to be filled.
> By the way this timestamp is created immediately before all alerts
> are fired off in parallel so to be usable for analysis of what happened
> in

Re: [ClusterLabs] Monitoring action of Pacemaker resources fail because of high load on the nodes

[Ken Gaillot]

On 04/22/2016 08:57 AM, Klaus Wenninger wrote:
> On 04/22/2016 03:29 PM, John Gogu wrote:
>> Hello community,
>> I am facing following situation with a Pacemaker 2 nodes DB cluster 
>> (3 resources configured into the cluster - 1 MySQL DB resource, 1
>> Apache resource, 1 IP resource )
>> -at every 61 seconds an MySQL monitoring action is started and have a
>> 1200 sec timeout.
> You can increase the timeout for monitoring.
>>
>> In some situation due to high load on the machines, monitoring action
>> run into a timeout, and the cluster is performing a fail over even if
>> the DB is up and running. Do you have a hint how can  be prioritized
>> automatically monitoring actions?
>>
> Consider that monitoring - at least as part of the action - should check
> if what your service is actually providing
> is working according to some functional and nonfunctional constraints as
> to simulate the experience of the
> consumer of your services. So you probably don't want that to happen
> prioritized.
> So if you relaxed the timing requirements of your monitoring to
> something that would be acceptable in terms
> of the definition of the service you are providing and you are still
> running into troubles the service quality you
> are providing wouldn't be that spiffing either...

Also, you can provide multiple levels of monitoring:

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#_multiple_monitor_operations

For example, you could provide a very simple check that just makes sure
MySQL is responding on its port, and run that frequently with a low
timeout. And your existing thorough monitor could be run less frequently
with a high timeout.

FYI there was a bug related to multiple monitors apparently introduced
in 1.1.10, such that a higher-level monitor failure might not trigger a
resource failure. It was recently fixed in the upstream master branch
(which will be in the soon-to-be-released 1.1.15-rc1).

>> Thank you and best regards,
>> John

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] [ClusterLab] : Unable to bring up pacemaker

[Ken Gaillot]

On 04/27/2016 11:25 AM, emmanuel segura wrote:
> you need to use pcs to do everything, pcs cluster setup and pcs
> cluster start, try to use the redhat docs for more information.

Agreed -- pcs cluster setup will create a proper corosync.conf for you.
Your corosync.conf below uses corosync 1 syntax, and there were
significant changes in corosync 2. In particular, you don't need the
file created in step 4, because pacemaker is no longer launched via a
corosync plugin.

> 2016-04-27 17:28 GMT+02:00 Sriram :
>> Dear All,
>>
>> I m trying to use pacemaker and corosync for the clustering requirement that
>> came up recently.
>> We have cross compiled corosync, pacemaker and pcs(python) for ppc
>> environment (Target board where pacemaker and corosync are supposed to run)
>> I m having trouble bringing up pacemaker in that environment, though I could
>> successfully bring up corosync.
>> Any help is welcome.
>>
>> I m using these versions of pacemaker and corosync
>> [root@node_cu pacemaker]# corosync -v
>> Corosync Cluster Engine, version '2.3.5'
>> Copyright (c) 2006-2009 Red Hat, Inc.
>> [root@node_cu pacemaker]# pacemakerd -$
>> Pacemaker 1.1.14
>> Written by Andrew Beekhof
>>
>> For running corosync, I did the following.
>> 1. Created the following directories,
>> /var/lib/pacemaker
>> /var/lib/corosync
>> /var/lib/pacemaker
>> /var/lib/pacemaker/cores
>> /var/lib/pacemaker/pengine
>> /var/lib/pacemaker/blackbox
>> /var/lib/pacemaker/cib
>>
>>
>> 2. Created a file called corosync.conf under /etc/corosync folder with the
>> following contents
>>
>> totem {
>>
>> version: 2
>> token:  5000
>> token_retransmits_before_loss_const: 20
>> join:   1000
>> consensus:  7500
>> vsftype:none
>> max_messages:   20
>> secauth:off
>> cluster_name:   mycluster
>> transport:  udpu
>> threads:0
>> clear_node_high_bit: yes
>>
>> interface {
>> ringnumber: 0
>> # The following three values need to be set based on your
>> environment
>> bindnetaddr: 10.x.x.x
>> mcastaddr: 226.94.1.1
>> mcastport: 5405
>> }
>>  }
>>
>>  logging {
>> fileline: off
>> to_syslog: yes
>> to_stderr: no
>> to_syslog: yes
>> logfile: /var/log/corosync.log
>> syslog_facility: daemon
>> debug: on
>> timestamp: on
>>  }
>>
>>  amf {
>> mode: disabled
>>  }
>>
>>  quorum {
>> provider: corosync_votequorum
>>  }
>>
>> nodelist {
>>   node {
>> ring0_addr: node_cu
>> nodeid: 1
>>}
>> }
>>
>> 3.  Created authkey under /etc/corosync
>>
>> 4.  Created a file called pcmk under /etc/corosync/service.d and contents as
>> below,
>>   cat pcmk
>>   service {
>>  # Load the Pacemaker Cluster Resource Manager
>>  name: pacemaker
>>  ver:  1
>>   }
>>
>> 5. Added the node name "node_cu" in /etc/hosts with 10.X.X.X ip
>>
>> 6. ./corosync -f -p & --> this step started corosync
>>
>> [root@node_cu pacemaker]# netstat -alpn | grep -i coros
>> udp0  0 10.X.X.X:61841 0.0.0.0:*
>> 9133/corosync
>> udp0  0 10.X.X.X:5405  0.0.0.0:*
>> 9133/corosync
>> unix  2  [ ACC ] STREAM LISTENING 14 9133/corosync
>> @quorum
>> unix  2  [ ACC ] STREAM LISTENING 148884 9133/corosync
>> @cmap
>> unix  2  [ ACC ] STREAM LISTENING 148887 9133/corosync
>> @votequorum
>> unix  2  [ ACC ] STREAM LISTENING 148885 9133/corosync
>> @cfg
>> unix  2  [ ACC ] STREAM LISTENING 148886 9133/corosync
>> @cpg
>> unix  2  [ ] DGRAM148840 9133/corosync
>>
>> 7. ./pacemakerd -f & gives the following error and exits.
>> [root@node_cu pacemaker]# pacemakerd -f
>> cmap connection setup failed: CS_ERR_TRY_AGAIN.  Retrying in 1s
>> cmap connection setup failed: CS_ERR_TRY_AGAIN.  Retrying in 2s
>> cmap connection setup failed: CS_ERR_TRY_AGAIN.  Retrying in 3s
>> cmap connection setup failed: CS_ERR_TRY_AGAIN.  Retrying in 4s
>> cmap connection setup failed: CS_ERR_TRY_AGAIN.  Retrying in 5s
>> Could not connect to Cluster Configuration Database API, error 6
>>
>> Can you please point me, what is missing in these steps ?
>>
>> Before trying these steps, I tried running "pcs cluster start", but that
>> command fails with "service" script not found. As the root filesystem
>> doesn't contain either /etc/init.d/ or /sbin/service
>>
>> So, the plan is to bring up corosync and pacemaker manually, later do the
>> cluster configuration using "pcs" commands.
>>
>> Regards,
>> Sriram
>>
>> ___
>> Users mailing list: Users@clusterlabs.org
>> http://clusterlabs.org/mailman/listinfo/users
>>
>> Project Home: 

Re: [ClusterLabs] Coming in 1.1.15: Event-driven alerts

[Ken Gaillot]

On 04/28/2016 10:24 AM, Lars Marowsky-Bree wrote:
> On 2016-04-27T12:10:10, Klaus Wenninger  wrote:
> 
>>> Having things in ARGV[] is always risky due to them being exposed more
>>> easily via ps. Environment variables or stdin appear better.
>> What made you assume the recipient is being passed as argument?
>>
>> The environment variable CRM_alert_recipient is being used to pass it.
> 
> Ah, excellent! But what made me think that this would be passed as
> arguments is that your announcement said: "Each alert may have any
> number of recipients configured. These values will simply be passed to
> the script as *arguments*." ;-)

Yes, that was my mistake in the original announcement. :-/

An early design had the script called once with all recipients as
arguments, but in the implementation we went with the script being
called once per recipient (and using only the environment variable).


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Monitoring action of Pacemaker resources fail because of high load on the nodes

[Ken Gaillot]

On 04/25/2016 10:23 AM, Dmitri Maziuk wrote:
> On 2016-04-24 16:20, Ken Gaillot wrote:
> 
>> Correct, you would need to customize the RA.
> 
> Well, you wouldn't because your custom RA will be overwritten by the
> next RPM update.

Correct again :)

I should have mentioned that the convention is to copy the script to a
different name before editing it. The recommended approach is to create
a new provider for your organization. For example, copy the RA to a new
directory /usr/lib/ocf/resource.d/local, so it would be used in
pacemaker as ocf:local:mysql. You can use anything in place of "local".

> Dimitri
> 
> 
> 
> ___
> Users mailing list: Users@clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] operation parallelism

[Ken Gaillot]

On 04/22/2016 09:05 AM, Ferenc Wágner wrote:
> Hi,
> 
> Are recurring monitor operations constrained by the batch-limit cluster
> option?  I ask because I'd like to limit the number of parallel start
> and stop operations (because they are resource hungry and potentially
> take long) without starving other operations, especially monitors.

No, they are not. The batch-limit only affects actions initiated by the
DC. The DC will initiate the first run of a monitor, so that will be
affected, but the local resource manager (lrmd) on the target node will
remember the monitor and run on it on schedule without further prompting
by the DC.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] attrd does not clean per-node cache after node removal

[Ken Gaillot]

On 03/23/2016 12:01 PM, Vladislav Bogdanov wrote:
> 23.03.2016 19:52, Vladislav Bogdanov wrote:
>> 23.03.2016 19:39, Ken Gaillot wrote:
>>> On 03/23/2016 07:35 AM, Vladislav Bogdanov wrote:
>>>> Hi!
>>>>
>>>> It seems like atomic attrd in post-1.1.14 (eb89393) does not
>>>> fully clean node cache after node is removed.

I haven't forgotten, this was a tricky one :-)

I believe this regression was introduced in da17fd0, which clears the
node's attribute *values* when purging the node, but not the value
*structures* that contain the node name and ID. That was intended as a
fix for when nodes leave and rejoin. However the same affected function
is used to handle "crm_node -R" requests, which should cause complete
removal.

I hope to have a fix soon.

Note that the behavior may still occur if "crm_node -R" is not called
after reloading corosync.

>>> Is this a regression? Or have you only tried it with this version?
>>
>> Only with this one.
>>
>>>
>>>> After our QA guys remove node wa-test-server-ha-03 from a two-node 
>>>> cluster:
>>>> * stop pacemaker and corosync on wa-test-server-ha-03
>>>> * remove node wa-test-server-ha-03 from corosync nodelist on 
>>>> wa-test-server-ha-04
>>>> * tune votequorum settings
>>>> * reload corosync on wa-test-server-ha-04
>>>> * remove node from pacemaker on wa-test-server-ha-04
>>>> * delete everything from /var/lib/pacemaker/cib on wa-test-server-ha-03
>>>> , and then join it with the different corosync ID (but with the same 
>>>> node name),
>>>> we see the following in logs:
>>>>
>>>> Leave node 1 (wa-test-server-ha-03):
>>>> Mar 23 04:19:53 wa-test-server-ha-04 attrd[25962]:   notice: 
>>>> crm_update_peer_proc: Node wa-test-server-ha-03[1] - state is now 
>>>> lost (was member)
>>>> Mar 23 04:19:53 wa-test-server-ha-04 attrd[25962]:   notice: Removing 
>>>> all wa-test-server-ha-03 (1) attributes for attrd_peer_change_cb
>>>> Mar 23 04:19:53 wa-test-server-ha-04 attrd[25962]:   notice: Lost 
>>>> attribute writer wa-test-server-ha-03
>>>> Mar 23 04:19:53 wa-test-server-ha-04 attrd[25962]:   notice: Removing 
>>>> wa-test-server-ha-03/1 from the membership list
>>>> Mar 23 04:19:53 wa-test-server-ha-04 attrd[25962]:   notice: Purged 1 
>>>> peers with id=1 and/or uname=wa-test-server-ha-03 from the membership 
>>>> cache
>>>> Mar 23 04:19:56 wa-test-server-ha-04 attrd[25962]:   notice: 
>>>> Processing peer-remove from wa-test-server-ha-04: wa-test-server-ha-03 0
>>>> Mar 23 04:19:56 wa-test-server-ha-04 attrd[25962]:   notice: Removing 
>>>> all wa-test-server-ha-03 (0) attributes for wa-test-server-ha-04
>>>> Mar 23 04:19:56 wa-test-server-ha-04 attrd[25962]:   notice: Removing 
>>>> wa-test-server-ha-03/1 from the membership list
>>>> Mar 23 04:19:56 wa-test-server-ha-04 attrd[25962]:   notice: Purged 1 
>>>> peers with id=0 and/or uname=wa-test-server-ha-03 from the membership 
>>>> cache
>>>>
>>>> Join node 3 (the same one, wa-test-server-ha-03, but ID differs):
>>>> Mar 23 04:21:23 wa-test-server-ha-04 attrd[25962]: notice: 
>>>> crm_update_peer_proc: Node wa-test-server-ha-03[3] - state is now 
>>>> member (was (null))
>>>> Mar 23 04:21:26 wa-test-server-ha-04 attrd[25962]:  warning: 
>>>> crm_find_peer: Node 3/wa-test-server-ha-03 = 0x201bf30 - 
>>>> a4cbcdeb-c36a-4a0e-8ed6-c45b3db89296
>>>> Mar 23 04:21:26 wa-test-server-ha-04 attrd[25962]:  warning: 
>>>> crm_find_peer: Node 2/wa-test-server-ha-04 = 0x1f90e20 - 
>>>> 6c18faa1-f8c2-4b0c-907c-20db450e2e79
>>>> Mar 23 04:21:26 wa-test-server-ha-04 attrd[25962]: crit: Node 1 
>>>> and 3 share the same name 'wa-test-server-ha-03'
>>>
>>> It took me a while to understand the above combination of messages. This
>>> is not node 3 joining. This is node 1 joining after node 3 has already
>>> been seen.
>>
>> Hmmm...
>> corosync.conf and corosync-cmapctl both say it is 3
>> Also, cib lists it as 3 and lrmd puts its status records under 3.
> 
> I mean:
> 
>  crm-debug-origin="do_update_resource" in_ccm="true" join="member" 
> expected="member">
>   
> 
> ...
> 
>   
> 
> 
>   
> 
>   
>name="master-rabbi

Re: [ClusterLabs] Antw: Re: FR: send failcount to OCF RA start/stop actions

[Ken Gaillot]

On 05/20/2016 10:40 AM, Adam Spiers wrote:
> Ken Gaillot <kgail...@redhat.com> wrote:
>> Just musing a bit ... on-fail + migration-threshold could have been
>> designed to be more flexible:
>>
>>   hard-fail-threshold: When an operation fails this many times, the
>> cluster will consider the failure to be a "hard" failure. Until this
>> many failures, the cluster will try to recover the resource on the same
>> node.
> 
> How is this different to migration-threshold, other than in name?
> 
>>   hard-fail-action: What to do when the operation reaches
>> hard-fail-threshold ("ban" would work like current "restart" i.e. move
>> to another node, and ignore/block/stop/standby/fence would work the same
>> as now)
> 
> And I'm not sure I understand how this is different to / more flexible
> than what we can do with on-fail now?
> 
>> That would allow fence etc. to be done only after a specified number of
>> retries. Ah, hindsight ...
> 
> Isn't that possible now, e.g. with migration-threshold=3 and
> on-fail=fence?  I feel like I'm missing something.

migration-threshold only applies when on-fail=restart. If on-fail=fence
or something else, that action always applies after the first failure.

So hard-fail-threshold would indeed be the same as migration-threshold,
but applied to all actions (and would be renamed, since the resource
won't migrate in the other cases).

>>> - neutron-l3-agent RA detects that the agent is unhealthy, and iff it
>>>   fails to restart it, we want to trigger migration of any routers on
>>>   that l3-agent to a healthy l3-agent.  Currently we wait for the
>>>   connection between the agent and the neutron server to time out,
>>>   which is unpleasantly slow.  This case is more of a requirement than
>>>   an optimization, because we really don't want to migrate routers to
>>>   another node unless we have to, because a) it takes time, and b) is
>>>   disruptive enough that we don't want to have to migrate them back
>>>   soon after if we discover we can successfully recover the unhealthy
>>>   l3-agent.
>>>
>>> - Remove a failed backend from an haproxy-fronted service if
>>>   it can't be restarted.
>>>
>>> - Notify any other service (OpenStack or otherwise) where the failing
>>>   local resource is a backend worker for some central service.  I
>>>   guess ceilometer, cinder, mistral etc. are all potential
>>>   examples of this.
> 
> Any thoughts on the sanity of these?

Beyond my expertise. But sounds reasonable.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Node attributes

[Ken Gaillot]

On 05/18/2016 10:49 PM, ‪H Yavari‬ ‪ wrote:
> Hi,
> 
> How can I define a constraint for two resource based on one nodes
> attribute?
> 
> For example resource X and Y are co-located based on node attribute Z.
> 
> 
> 
> Regards,
> H.Yavari

Hi,

See
http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#idm140617356537136

High-level tools such as pcs and crm provide a simpler interface, but
the concepts will be the same.

This works for location constraints, not colocation, but you can easily
accomplish what you want. If your goal is that X and Y each can only run
on a node with attribute Z, then set up a location constraint for each
one using the appropriate rule. If you goal is that X and Y must be
colocated together, on a node with attribute Z, then set up a regular
colocation constraint between them, and a location constraint for one of
them with the appropriate rule; or, put them in a group, and set up a
location constraint for the group with the appropriate rule.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] Informing RAs about recovery: failed resource recovery, or any start-stop cycle?

[Ken Gaillot]

A recent thread discussed a proposed new feature, a new environment
variable that would be passed to resource agents, indicating whether a
stop action was part of a recovery.

Since that thread was long and covered a lot of topics, I'm starting a
new one to focus on the core issue remaining:

The original idea was to pass the number of restarts remaining before
the resource will no longer tried to be started on the same node. This
involves calculating (fail-count - migration-threshold), and that
implies certain limitations: (1) it will only be set when the cluster
checks migration-threshold; (2) it will only be set for the failed
resource itself, not for other resources that may be recovered due to
dependencies on it.

Ulrich Windl proposed an alternative: setting a boolean value instead. I
forgot to cc the list on my reply, so I'll summarize now: We would set a
new variable like OCF_RESKEY_CRM_recovery=true whenever a start is
scheduled after a stop on the same node in the same transition. This
would avoid the corner cases of the previous approach; instead of being
tied to migration-threshold, it would be set whenever a recovery was
being attempted, for any reason. And with this approach, it should be
easier to set the variable for all actions on the resource
(demote/stop/start/promote), rather than just the stop.

I think the boolean approach fits all the envisioned use cases that have
been discussed. Any objections to going that route instead of the count?
-- 
Ken Gaillot <kgail...@redhat.com>

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Issue in resource constraints and fencing - RHEL7 - AWS EC2

[Ken Gaillot]

On 05/20/2016 10:02 AM, Pratip Ghosh wrote:
> Hi All,
> 
> I am implementing 2 node RedHat (RHEL 7.2) HA cluster on Amazon EC2
> instance. For floating IP I am using a shell script provided by AWS so
> that virtual IP float to another instance if any one server failed with
> health check. In basic level cluster is working but I have 2 issues on
> that which I describe in bellow.
> 
> ISSUE 1
> =
> Now I need to configure fencing/STONITH to avoid split brain scenario in
> storage cluster. I want to use multi-primari (Active/Active) DRBD in my
> cluster for distributed storage. Is it possible to configure power
> fencing on AWS EC2 instance? Can any one please guide me on this?

There has been some discussion about this on this list before -- see
http://search.gmane.org/?query=ec2=gmane.comp.clustering.clusterlabs.user

Basically, there is an outdated agent available at
https://github.com/beekhof/fence_ec2 and a newer fork of it in the
(RHEL-incompatible) cluster-glue package. So with some work you may be
able to get something working.

> 
> ISSUE2
> =
> Currently I am using single  primary DRBD distributed storage. I added
> cluster resources so that if any cluster node goes down then another
> cluster node will promoted DRBD volume as primary and mount it on
> /var/www/html.
> 
> This configuration is working but for only if cluster node1 goes down.
> If cluster node2 goes down all cluster resources fails over towards
> cluster node1 but whenever cluster node2 again become on-line then
> virtual_ip (cluster ip) ownership automatically goes towards cluster
> node2 again. All the remaining resources not failed over like that. In
> that case secondary IP stays with Node1 and ownership goes to Node2.
> 
> I think this is an issue with resource stickiness or resource constraint
> but here I am totally clueless. Can any one please help me on this?
> 
> 
> My cluster details:
> ===
> 
> [root@drbd01 ~]# pcs config
> Cluster Name: web_cluster
> Corosync Nodes:
>  ec2-52-24-8-124.us-west-2.compute.amazonaws.com
> ec2-52-27-70-12.us-west-2.compute.amazonaws.com
> Pacemaker Nodes:
>  ec2-52-24-8-124.us-west-2.compute.amazonaws.com
> ec2-52-27-70-12.us-west-2.compute.amazonaws.com
> 
> Resources:
>  Resource: virtual_ip (class=ocf provider=heartbeat type=IPaddr2)
>   Attributes: ip=10.98.70.100 cidr_netmask=24
>   Operations: start interval=0s timeout=20s (virtual_ip-start-interval-0s)
>   stop interval=0s timeout=20s (virtual_ip-stop-interval-0s)
>   monitor interval=30s (virtual_ip-monitor-interval-30s)
>  Resource: WebSite (class=ocf provider=heartbeat type=apache)
>   Attributes: configfile=/etc/httpd/conf/httpd.conf
> statusurl=http://10.98.70.100/server-status
>   Operations: start interval=0s timeout=40s (WebSite-start-interval-0s)
>   stop interval=0s timeout=60s (WebSite-stop-interval-0s)
>   monitor interval=1min (WebSite-monitor-interval-1min)
>  Master: WebDataClone
>   Meta Attrs: master-max=1 master-node-max=1 clone-max=2
> clone-node-max=1 notify=true
>   Resource: WebData (class=ocf provider=linbit type=drbd)
>Attributes: drbd_resource=r1
>Operations: start interval=0s timeout=240 (WebData-start-interval-0s)
>promote interval=0s timeout=90 (WebData-promote-interval-0s)
>demote interval=0s timeout=90 (WebData-demote-interval-0s)
>stop interval=0s timeout=100 (WebData-stop-interval-0s)
>monitor interval=60s (WebData-monitor-interval-60s)
>  Resource: WebFS (class=ocf provider=heartbeat type=Filesystem)
>   Attributes: device=/dev/drbd1 directory=/var/www/html fstype=xfs
>   Operations: start interval=0s timeout=60 (WebFS-start-interval-0s)
>   stop interval=0s timeout=60 (WebFS-stop-interval-0s)
>   monitor interval=20 timeout=40 (WebFS-monitor-interval-20)
> 
> Stonith Devices:
> Fencing Levels:
> 
> Location Constraints:
> Ordering Constraints:
>   promote WebDataClone then start WebFS (kind:Mandatory)
> (id:order-WebDataClone-WebFS-mandatory)
>   start WebFS then start virtual_ip (kind:Mandatory)
> (id:order-WebFS-virtual_ip-mandatory)
>   start virtual_ip then start WebSite (kind:Mandatory)
> (id:order-virtual_ip-WebSite-mandatory)
> Colocation Constraints:
>   WebSite with virtual_ip (score:INFINITY)
> (id:colocation-WebSite-virtual_ip-INFINITY)
>   WebFS with WebDataClone (score:INFINITY) (with-rsc-role:Master)
> (id:colocation-WebFS-WebDataClone-INFINITY)
>   WebSite with WebFS (score:INFINITY)
> (id:colocation-WebSite-WebFS-INFINITY)
> 
> Resources Defaults:
>  resource-stickiness: INFINITY

You don't have any constraints requiring virtual_ip to stay with any
other resource. So it doesn't.

You could colocate virtual_ip with WebFS, and drop the colocation of
WebSite with WebFS, but it would probably be easier to configure a group
with WebFS, virtual_ip, WebSite, and WebFS. Then you would only need
promote WebDataClone then start the new 

Re: [ClusterLabs] Resource seems to not obey constraint

[Ken Gaillot]

On 05/20/2016 10:29 AM, Leon Botes wrote:
> I push the following config.
> The iscsi-target fails as it tries to start on iscsiA-node1
> This is because I have no target installed on iscsiA-node1 which is by
> design. All services listed here should only start on  iscsiA-san1
> iscsiA-san2.
> I am using using the iscsiA-node1 basically for quorum and some other
> minor functions.
> 
> Can someone please show me where I am going wrong?
> All services should start on the same node, order is drbd-master
> vip-blue vip-green iscsi-target iscsi-lun
> 
> pcs -f ha_config property set symmetric-cluster="true"
> pcs -f ha_config property set no-quorum-policy="stop"
> pcs -f ha_config property set stonith-enabled="false"
> pcs -f ha_config resource defaults resource-stickiness="200"
> 
> pcs -f ha_config resource create drbd ocf:linbit:drbd drbd_resource=r0
> op monitor interval=60s
> pcs -f ha_config resource master drbd master-max=1 master-node-max=1
> clone-max=2 clone-node-max=1 notify=true
> pcs -f ha_config resource create vip-blue ocf:heartbeat:IPaddr2
> ip=192.168.101.100 cidr_netmask=32 nic=blue op monitor interval=20s
> pcs -f ha_config resource create vip-green ocf:heartbeat:IPaddr2
> ip=192.168.102.100 cidr_netmask=32 nic=green op monitor interval=20s
> pcs -f ha_config resource create iscsi-target ocf:heartbeat:iSCSITarget
> params iqn="iqn.2016-05.trusc.net" implementation="lio-t" op monitor
> interval="30s"
> pcs -f ha_config resource create iscsi-lun
> ocf:heartbeat:iSCSILogicalUnit params target_iqn="iqn.2016-05.trusc.net"
> lun="1" path="/dev/drbd0"
> 
> pcs -f ha_config constraint colocation add vip-blue drbd-master INFINITY
> with-rsc-role=Master
> pcs -f ha_config constraint colocation add vip-green drbd-master
> INFINITY with-rsc-role=Master
> 
> pcs -f ha_config constraint location drbd-master prefers stor-san1=500
> pcs -f ha_config constraint location drbd-master avoids stor-node1=INFINITY

The above constraint is an example of how to ban a resource from a node.
However stor-node1 is not a valid node name in your setup (maybe an
earlier design?), so this particular constraint won't have any effect.
If you want to ban certain resources from iscsiA-node1, add constraints
like the above for each resource, using the correct node name.

> pcs -f ha_config constraint order promote drbd-master then start vip-blue
> pcs -f ha_config constraint order start vip-blue then start vip-green
> pcs -f ha_config constraint order start vip-green then start iscsi-target
> pcs -f ha_config constraint order start iscsi-target then start iscsi-lun
> 
> Results:
> 
> [root@san1 ~]# pcs status
> Cluster name: storage_cluster
> Last updated: Fri May 20 17:21:10 2016  Last change: Fri May 20
> 17:19:43 2016 by root via cibadmin on iscsiA-san1
> Stack: corosync
> Current DC: iscsiA-san1 (version 1.1.13-10.el7_2.2-44eb2dd) - partition
> with quorum
> 3 nodes and 6 resources configured
> 
> Online: [ iscsiA-node1 iscsiA-san1 iscsiA-san2 ]
> 
> Full list of resources:
> 
>  Master/Slave Set: drbd-master [drbd]
>  Masters: [ iscsiA-san1 ]
>  Slaves: [ iscsiA-san2 ]
>  vip-blue   (ocf::heartbeat:IPaddr2):   Started iscsiA-san1
>  vip-green  (ocf::heartbeat:IPaddr2):   Started iscsiA-san1
>  iscsi-target   (ocf::heartbeat:iSCSITarget):   FAILED iscsiA-node1
> (unmanaged)
>  iscsi-lun  (ocf::heartbeat:iSCSILogicalUnit):  Stopped
> 
> Failed Actions:
> * drbd_monitor_0 on iscsiA-node1 'not installed' (5): call=6, status=Not
> installed, exitreason='none',
> last-rc-change='Fri May 20 17:19:44 2016', queued=0ms, exec=0ms
> * iscsi-target_stop_0 on iscsiA-node1 'not installed' (5): call=24,
> status=complete, exitreason='Setup problem: couldn't find command:
> targetcli',
> last-rc-change='Fri May 20 17:19:45 2016', queued=0ms, exec=18ms
> * iscsi-lun_monitor_0 on iscsiA-node1 'not installed' (5): call=22,
> status=complete, exitreason='Undefined iSCSI target implementation',
> last-rc-change='Fri May 20 17:19:44 2016', queued=0ms, exec=27ms

The above failures will still occur even if you add the proper
constraints, because these are probes. Before starting a resource,
Pacemaker probes it on all nodes, to make sure it's not already running
somewhere. You can prevent this when you know it is impossible that the
resource could be running on a particular node, by adding
resource-discovery=never when creating the constraint banning it from
that node.

> 
> PCSD Status:
>   iscsiA-san1: Online
>   iscsiA-san2: Online
>   iscsiA-node1: Online
> 
> Daemon Status:
>   corosync: active/disabled
>   pacemaker: active/disabled
>   pcsd: active/disabled
> 


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] pacemaker and fence_sanlock

[Ken Gaillot]

On 05/11/2016 09:14 PM, Da Shi Cao wrote:
> Dear all,
> 
> I'm just beginning to use pacemaker+corosync as our HA solution on
> Linux, but I got stuck at the stage of configuring fencing.
> 
> Pacemaker 1.1.15,  Corosync Cluster Engine, version '2.3.5.46-d245', and
> sanlock 3.3.0 (built May 10 2016 05:13:12)
> 
> I have the following questions:
> 
> 1. stonith_admin --list-installed will only list two agents: fence_pcmk,
> fence_legacy before sanlock is compiled and installed under /usr/local.
> But after "make install" of sanlock, stonith_admin --list-installed will
> list: 
> 
>  fence_sanlockd
>  fence_sanlock
>  fence_pcmk
>  fence_legacy
>  It is weird and I wonder what makes stonith_admin know about fence_sanlock?

I'm guessing you also installed pacemaker under /usr/local;
stonith_admin will simply list $installdir/sbin/fence_*

> 2. How to configure the fencing by fence_sanlock into pacemaker? I've
> tried to create a new resource to do the unfencing for each node, but
> the resource start will fail since there is no monitor operation of
> fence_sanlock agent, because resource manager will fire monitor once
> after the start to make sure it has been started OK.

I'm not familiar with fence_sanlock, but it should be fine to do what
you describe. There's probably an issue with your configuration. What
command did you use to configure the resource?

> 3. How to create a fencing resource to do the fencing by sanlock. This
> I've not tried yet. But I wonder which node/nodes of the majority will
> initiate the fence operations to the nodes without quorum.

Once you've defined the resource in the pacemaker configuration, the
cluster will intelligently decide when and how to call it.

When you check the cluster status, you'll see that the fence device is
"running" on one node. In fact, any node can use the fence device
(assuming the configuration doesn't specifically ban it); the listed
node is the one running the recurring monitor on the resource. The
cluster considers that node to have "verified" access to the device, so
it will prefer that node when fencing using the device -- but it may
decide to choose another node when appropriate.

You may be interested to know that pacemaker has recently gained native
support for watchdog-based fencing via the "sbd" software package. See:

  http://blog.clusterlabs.org/blog/2015/sbd-fun-and-profit/
  http://clusterlabs.org/wiki/Using_SBD_with_Pacemaker

Some discussion of common configuration issues can be seen at:

  https://bugzilla.redhat.com/show_bug.cgi?id=1221680

If you have a Red Hat subscription, Red Hat has a simple walk-through
for configuring sbd with pacemaker on RHEL 6.8+/7.1+ (using watchdog
only, no "poison pill" shared storage):

  https://access.redhat.com/articles/2212861

> Thank you very much.
> Dashi Cao

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: FR: send failcount to OCF RA start/stop actions

[Ken Gaillot]

On 05/12/2016 06:21 AM, Adam Spiers wrote:
> Hi Ken,
> 
> Firstly thanks a lot not just for working on this, but also for being
> so proactive in discussing the details.  A perfect example of
> OpenStack's "Open Design" philosophy in action :-)
> 
> Ken Gaillot <kgail...@redhat.com> wrote:
>> On 05/10/2016 02:29 AM, Ulrich Windl wrote:
>>>>>> Ken Gaillot <kgail...@redhat.com> schrieb am 10.05.2016 um 00:40 in 
>>>>>> Nachricht
>>> <573111d3.7060...@redhat.com>:
> 
> [snipped]
> 
>>>> Here is what I'm testing currently:
>>>>
>>>> - When the cluster recovers a resource, the resource agent's stop action
>>>> will get a new variable, OCF_RESKEY_CRM_meta_recovery_left =
>>>> migration-threshold - fail-count on the local node.
>>>
>>> With that mechanism RA testingwill be more complicated as it is
>>> now, and I cannot see the benefit yet.
>>
>> Testing will be more complicated for RAs that choose to behave
>> differently depending on the variable value, but the vast, vast majority
>> won't, so it will have no effect on most users. No pacemaker behavior
>> changes.
>>
>> BTW I should have explicitly mentioned that the variable name is up for
>> discussion; I had a hard time coming up with something meaningful that
>> didn't span an entire line of text.
> 
> I'd prefer plural (OCF_RESKEY_CRM_meta_recoveries_left) but other than
> that I think it's good.  OCF_RESKEY_CRM_meta_retries_left is shorter;
> not sure whether it's marginally worse or better though.

I'm now leaning to restart_remaining (restarts_remaining would be just
as good).

>>>> - The variable is not added for any action other than stop.
>>>>
>>>> - I'm preferring simplicity over flexibility by providing only a single
>>>> variable. The RA theoretically can already get the migration-threshold
>>>> from the CIB and fail-count from attrd -- what we're adding is the
>>>> knowledge that the stop is part of a recovery.
>>>>
>>>> - If the stop is final (the cluster does not plan to start the resource
>>>> anywhere), the variable may be set to 0, or unset. The RA should treat 0
>>>> and unset as equivalent.
>>>>
>>>> - So, the variable will be 1 for the stop before the last time the
>>>> cluster will try to start the resource on the same node, and 0 or unset
>>>> for the last stop on this node before trying to start on another node.
> 
> OK, so the RA code would typically be something like this?
> 
> if [ ${OCF_RESKEY_CRM_meta_retries_left:-0} = 0 ]; then
> # This is the final stop, so tell the external service
> # not to send any more work our way.
> disable_service
> fi

I'd use -eq :) but yes

>>> Be aware that the node could be fenced (for reasons ouside of your
>>> RA) even before all these attempts are carried out.
>>
>> Yes, by listing such scenarios and the ones below, I am hoping the
>> potential users of this feature can think through whether it will be
>> sufficient for their use cases.
> 
> That's a good point, but I think it's OK because if the node gets
> fenced, we have one and shortly two different mechanisms for achieving
> the same thing:
> 
>   1. add another custom fencing agent to fencing_topology
>   2. use the new events mechanism
> 
>>>> - The variable will be set only in situations when the cluster will
>>>> consider migration-threshold. This makes sense, but some situations may
>>>> be unintuitive:
>>>>
>>>> -- If a resource is being recovered, but the fail-count is being cleared
>>>> in the same transition, the cluster will ignore migration-threshold (and
>>>> the variable will not be set). The RA might see recovery_left=5, 4, 3,
>>>> then someone clears the fail-count, and it won't see recovery_left even
>>>> though there is a stop and start being attempted.
> 
> Hmm.  So how would the RA distinguish that case from the one where
> the stop is final?

That's the main question in all this. There are quite a few scenarios
where there's no meaningful distinction between 0 and unset. With the
current implementation at least, the ideal approach is for the RA to
treat the last stop before a restart the same as a final stop.

>>>> -- Migration-threshold will be considered (and the variable will be set)
>>>> only if the resource is being recovered due to failure, not if the
>>>> resource is being restarted or moved for some other reason (cons

Re: [ClusterLabs] notify action asynchronous ?

[Ken Gaillot]

On 05/12/2016 04:37 AM, Jehan-Guillaume de Rorthais wrote:
> Le Sun, 8 May 2016 16:35:25 +0200,
> Jehan-Guillaume de Rorthais <j...@dalibo.com> a écrit :
> 
>> Le Sat, 7 May 2016 00:27:04 +0200,
>> Jehan-Guillaume de Rorthais <j...@dalibo.com> a écrit :
>>
>>> Le Wed, 4 May 2016 09:55:34 -0500,
>>> Ken Gaillot <kgail...@redhat.com> a écrit :
>> ...
>>>> There would be no point in the pre-promote notify waiting for the
>>>> attribute value to be retrievable, because the cluster isn't going to
>>>> wait for the pre-promote notify to finish before calling promote.
>>>
>>> Oh, this is surprising. I thought the pseudo action
>>> "*_confirmed-pre_notify_demote_0" in the transition graph was a wait for
>>> each resource clone return code before going on with the transition. The
>>> graph is confusing, if the cluster isn't going to wait for the pre-promote
>>> notify to finish before calling promote, I suppose some arrows should point
>>> directly from start (or post-start-notify?) action directly to the promote
>>> action then, isn't it?
>>>
>>> This is quite worrying as our RA rely a lot on notifications. As instance,
>>> we try to recover a PostgreSQL instance during pre-start or pre-demote if we
>>> detect a recover action...
>>
>> I'm coming back on this point.
>>
>> Looking at this documentation page:
>> http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-config-testing-changes.html
>>
>> I can read "Arrows indicate ordering dependencies".
>>
>> Looking at the transition graph I am studying (see attachment, a simple
>> master resource move), I still don't understand how the cluster isn't going 
>> to
>> wait for a pre-promote notify to finish before calling promote.
>>
>> So either I misunderstood your words or I miss something else important, 
>> which
>> is quite possible as I am fairly new to this word. Anyway, I try to make a
>> RA as robust as possible and any lights/docs are welcome!
> 
> I tried to trigger this potential asynchronous behavior of the notify action,
> but couldn't observe it.
> 
> I added different sleep period in the notify action for each node of my 
> cluster:
>   * 10s for hanode1
>   * 15s for hanode2
>   * 20s for hanode3
> 
> The master was on hanode1 and  the DC was hanode1. While moving the master
> resource to hanode2, I can see in the log files that the DC is always
> waiting for the rc of hanode3 before triggering the next action in the
> transition.
> 
> So, **in pratice**, it seems the notify action is synchronous. In theory now, 
> I
> still wonder if I misunderstood your words...

I think you're right, and I was mistaken. The asynchronicity most likely
comes purely from crm_attribute not waiting for the value to be set and
propagated to all nodes.

I think I was confusing clone notifications with the new alerts feature,
which is asynchronous. We named that "alerts" to try to avoid such
confusion, but my brain hasn't gotten the memo yet ;)

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Q: monitor and probe result codes and consequences

[Ken Gaillot]

On 05/12/2016 02:56 AM, Ulrich Windl wrote:
> Hi!
> 
> I have a question regarding an RA written by myself and pacemaker 
> 1.1.12-f47ea56 (SLES11 SP4):
> 
> During "probe" all resources' "monitor" actions are executed (regardless of 
> any ordering constraints). Therefore my RA considers a parameter as invalid 
> ("file does not exist") (the file will be provided once some supplying 
> resource is up) and returns rc=2.
> OK, this may not be optimal, but pacemaker makes it worse: It does not repeat 
> the probe once the resource would start, but keeps the state, preventing a 
> resource start:
> 
>  primitive_monitor_0 on h05 'invalid parameter' (2): call=73, 
> status=complete, exit-reason='none', last-rc-change='Wed May 11 17:03:39 
> 2016', queued=0ms, exec=82ms

Correct, OCF_ERR_CONFIGURED is a "fatal" error:

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#_how_are_ocf_return_codes_interpreted

> So you would say that monitor may only return "success" or "not running", but 
> I feel the RA should detect the condition that the resource could not run at 
> all at the present state.

OCF_ERR_CONFIGURED is meant to indicate that the resource could not
possibly run *as configured*, regardless of the system's current state.
So for example, a required parameter is missing or invalid.

You could possibly use OCF_ERR_ARGS in this case (a "hard" error that
bans the particular node, and means that the resource's configuration is
not valid on this particular node).

But, I suspect the right answer here is simply an order constraint
between the supplying resource and this resource. This resource's start
action, not monitor, should be the one that checks for the existence of
the supplied file.

> Shouldn't pacemaker reprobe resources before it tries to start them?

Probes are meant to check whether the resource is already active
anywhere. The decision of whether and where to start the resource takes
into account the result of the probes, so it doesn't make sense to
re-probe -- that's what the initial probe was for.

> Before my RA had passed all the ocf-tester checks, so this situation is hard 
> to test (unless you have a test cluster you can restart any time).
> 
> (After manual resource cleanup the resource started as usual)
> 
> My monitor uses the following logic:
> ---
> monitor|status)
> if validate; then
> set_variables
> check_resource || exit $OCF_NOT_RUNNING
> status=$OCF_SUCCESS
> else # cannot check status with invalid parameters
> status=$?
> fi
> exit $status
> ;;
> ---
> 
> Should I mess with ocf_is_probe?
> 
> Regards,
> Ulrich

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] start a resource

[Ken Gaillot]

On 05/16/2016 12:22 PM, Dimitri Maziuk wrote:
> On 05/13/2016 04:31 PM, Ken Gaillot wrote:
> 
>> That is definitely not a properly functioning cluster. Something
>> is going wrong at some level.
> 
> Yeah, well... how do I find out what/where?

What happens after "pcs resource cleanup"? "pcs status" reports the
time associated with each failure, so you can check whether you are
seeing the same failure or a new one.

The system log is usually the best starting point, as it will have
messages from pacemaker, corosync and the resource agents. You can
look around the time of the failure(s) to look for details or anything
unusual.

Pacemaker also has a detail log (by default, /var/log/pacemaker.log).
In general, this is more useful to developers than administrators, but
if the system log doesn't help, it can sometimes shed a little more light.

> One question: in corosync.conf I have nodelist { node { ring0_addr:
> node1_name nodeid: 1 } node { ring0_addr: node2_name nodeid: 2 } }
> 
> Could 'pcs cluster stop/start' reset the interface that resolves
> to nodeX_name? If so, that would answer why ssh connections get
> killed.

No, Pacemaker and pcs don't touch the interfaces (unless of course you
explicitly add a cluster resource to do so, which wouldn't work anyway
for the interface(s) that corosync itself needs to use).


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Pacemaker with Zookeeper??

[Ken Gaillot]

On 05/17/2016 06:50 AM, Bogdan Dobrelya wrote:
> On 05/17/2016 01:17 PM, Adam Spiers wrote:
>> Bogdan Dobrelya  wrote:
>>> On 05/16/2016 09:23 AM, Jan Friesse wrote:
> Hi,
>
> I have an idea: use Pacemaker with Zookeeper (instead of Corosync). Is
> it possible?
> Is there any examination about that?
>>>
>>> Indeed, would be *great* to have a Pacemaker based control plane on top
>>> of other "pluggable" distributed KVS & messaging systems, for example
>>> etcd as well :)
>>> I'm looking forward to joining any dev efforts around that, although I'm
>>> not a Java or Go developer.
>>
>> Pacemaker and corosync are written in C, so I suspect it would be
>> pretty awkward to replace corosync with something which isn't.
>>
>> ___
>> Users mailing list: Users@clusterlabs.org
>> http://clusterlabs.org/mailman/listinfo/users
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
> 
> Would be nice to see a clear picture for what is the coupling between
> those separate services making the Idea impossible? Is it libqb with its
> C-only API (wild guessing, sorry!), anything else? Theoretically, if
> there were no loose coupling, one could replace storage or messaging
> components or both and still leverage a Pacemaker CP :)

The only language dependency is that Pacemaker is written in C and needs
to hook into the messaging layer, so the messaging layer needs some C
API. Since most languages offer some way for library functions to be
called by a C program, it shouldn't be a deal-breaker.

Two issues make this challenging:

First, all the places in Pacemaker code that currently can call either
heartbeat or corosync functions would need equivalents for the new
layer, which would require a large investment of developer time. Search
the code for "SUPPORT_COROSYNC" to get an idea.

Second, corosync has been optimized over many years for the HA cluster
use case, and any replacement would have a lot of catching up to do, to
be truly equivalent. Communication layers that haven't been forged by
experience with HA clusters are highly unlikely to handle the known
corner cases and tricky situations. See the corosync and votequorum man
pages for just the user-tunable parameters that have evolved.

It is possible, just difficult.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: Using different folder for /var/lib/pacemaker and usage of /dev/shm files

[Ken Gaillot]

On 05/17/2016 12:02 PM, Nikhil Utane wrote:
> OK. Will do that.
> 
> Actually I gave the /dev/shm usage when the cluster wasn't up.
> When it is up, I see it occupies close to 300 MB (it's also the DC).

Hmmm, there should be no usage if the cluster is stopped. Any memory
used by the cluster will start with "qb-", so anything else is from
something else.

If all executables using libqb (including corosync and pacemaker) are
stopped, it's safe to remove any /dev/shm/qb-* files that remain. That
should be rare, probably only after a core dump or such.

> tmpfs   500.0M329.4M170.6M  66% /dev/shm
> 
> On another node the same is 115 MB.
> 
> Anyways, I'll monitor the usage to know what size is needed. 
> 
> Thank you Ken and Ulrich. 
> 
> On Tue, May 17, 2016 at 8:23 PM, Ken Gaillot <kgail...@redhat.com
> <mailto:kgail...@redhat.com>> wrote:
> 
> On 05/17/2016 04:07 AM, Nikhil Utane wrote:
> > What I would like to understand is how much total shared memory
> > (approximately) would Pacemaker need so that accordingly I can define
> > the partition size. Currently it is 300 MB in our system. I recently ran
> > into insufficient shared memory issue because of improper clean-up. So
> > would like to understand how much Pacemaker would need for a 6-node
> > cluster so that accordingly I can increase it.
> 
> I have no idea :-)
> 
> I don't think there's any way to pre-calculate it. The libqb library is
> the part of the software stack that actually manages the shared memory,
> but it's used by everything -- corosync (including its cpg and
> votequorum components) and each pacemaker daemon.
> 
> The size depends directly on the amount of communication activity in the
> cluster, which is only indirectly related to the number of
> nodes/resources/etc., the size of the CIB, etc. A cluster with nodes
> joining/leaving frequently and resources moving around a lot will use
> more shared memory than a cluster of the same size that's quiet. Cluster
> options such as cluster-recheck-interval would also matter.
> 
> Practically, I think all you can do is simulate expected cluster
> configurations and loads, and see what it comes out to be.
> 
> > # df -kh
> > tmpfs   300.0M 27.5M272.5M   9% /dev/shm
> >
> > Thanks
> > Nikhil
> >
> > On Tue, May 17, 2016 at 12:09 PM, Ulrich Windl
> > <ulrich.wi...@rz.uni-regensburg.de
> <mailto:ulrich.wi...@rz.uni-regensburg.de>
> > <mailto:ulrich.wi...@rz.uni-regensburg.de
> <mailto:ulrich.wi...@rz.uni-regensburg.de>>> wrote:
> >
> > Hi!
> >
> > One of the main problems I identified with POSIX shared memory
> > (/dev/shm) in Linux is that changes to the shared memory don't
> > affect the i-node, so you cannot tell from a "ls -rtl" which
> > segments are still active and which are not. You can only see the
> > creation time.
> >
> > Maybe there should be a tool that identifies and cleans up obsolete
> > shared memory.
> > I don't understand the part talking about the size of /dev/shm: It's
> > shared memory. See "kernel.shmmax" and "kernel.shmall" in you sysctl
> > settings (/etc/sysctl.conf).
> >
> > Regards,
> > Ulrich
> >
> > >>> Nikhil Utane <nikhil.subscri...@gmail.com 
> <mailto:nikhil.subscri...@gmail.com>
> > <mailto:nikhil.subscri...@gmail.com
> <mailto:nikhil.subscri...@gmail.com>>> schrieb am 16.05.2016 um 14:31 in
> > Nachricht
> >   
>  <CAGNWmJVSye5PJgkdbFAi5AzO+Qq-j=2fs1c+0rgnqs994vv...@mail.gmail.com
> <mailto:2fs1c%2b0rgnqs994vv...@mail.gmail.com>
> > <mailto:2fs1c%2b0rgnqs994vv...@mail.gmail.com
> <mailto:2fs1c%252b0rgnqs994vv...@mail.gmail.com>>>:
> > > Thanks Ken.
> > >
> > > Could you also respond on the second question?
> > >
> > >> Also, in /dev/shm I see that it created around 300+ files of
> > around
> > >> 250 MB.
> > >>
> > >> For e.g.
> > >> -rw-rw1 hacluste hacluste  8232 May  6 13:03
> > >> qb-cib_rw-response-25035-25038-10-header
> > >> -rw-rw1 hacluste hacluste540672 May  6 1

Re: [ClusterLabs] Using different folder for /var/lib/pacemaker and usage of /dev/shm files

[Ken Gaillot]

On 05/08/2016 11:19 PM, Nikhil Utane wrote:
> Moving these questions to a different thread.
> 
> Hi,
> 
> We have limited storage capacity in our system for different folders. 
> How can I configure to use a different folder for /var/lib/pacemaker?

./configure --localstatedir=/wherever (defaults to /var or ${prefix}/var)

That will change everything that normally is placed or looked for under
/var (/var/lib/pacemaker, /var/lib/heartbeat, /var/run, etc.).

Note that while ./configure lets you change the location of nearly
everything, /usr/lib/ocf/resource.d is an exception, because it is
specified in the OCF standard.

> 
> 
> Also, in /dev/shm I see that it created around 300+ files of around
> 250 MB.
> 
> For e.g.
> -rw-rw1 hacluste hacluste  8232 May  6 13:03
> qb-cib_rw-response-25035-25038-10-header
> -rw-rw1 hacluste hacluste540672 May  6 13:03
> qb-cib_rw-response-25035-25038-10-data
> -rw---1 hacluste hacluste  8232 May  6 13:03
> qb-cib_rw-response-25035-25036-12-header
> -rw---1 hacluste hacluste540672 May  6 13:03
> qb-cib_rw-response-25035-25036-12-data
> And many more..
> 
> We have limited space in /dev/shm and all these files are filling it
> up. Are these all needed? Any way to limit? Do we need to do any
> clean-up if pacemaker termination was not graceful? 
> 
> -Thanks
> Nikhil
> 
> 
> 
> 
> ___
> Users mailing list: Users@clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
> 


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: Antw: Re: Q: monitor and probe result codes and consequences

[Ken Gaillot]

On 05/13/2016 06:00 AM, Ulrich Windl wrote:
>>>> Dejan Muhamedagic <deja...@fastmail.fm> schrieb am 13.05.2016 um 12:16 in
> Nachricht <20160513101626.GA12493@walrus.homenet>:
>> Hi,
>>
>> On Fri, May 13, 2016 at 09:05:54AM +0200, Ulrich Windl wrote:
>>>>>> Ken Gaillot <kgail...@redhat.com> schrieb am 12.05.2016 um 16:41 in 
>>>>>> Nachricht
>>> <57349629.40...@redhat.com>:
>>>> On 05/12/2016 02:56 AM, Ulrich Windl wrote:
>>>>> Hi!
>>>>>
>>>>> I have a question regarding an RA written by myself and pacemaker 
>>>> 1.1.12-f47ea56 (SLES11 SP4):
>>>>>
>>>>> During "probe" all resources' "monitor" actions are executed (regardless 
>>>>> of 
>>
>>>> any ordering constraints). Therefore my RA considers a parameter as 
>>>> invalid 
>>
>>>> ("file does not exist") (the file will be provided once some supplying 
>>>> resource is up) and returns rc=2.
>>>>> OK, this may not be optimal, but pacemaker makes it worse: It does not 
>>>> repeat the probe once the resource would start, but keeps the state, 
>>>> preventing a resource start:
>>>>>
>>>>>  primitive_monitor_0 on h05 'invalid parameter' (2): call=73, 
>>>> status=complete, exit-reason='none', last-rc-change='Wed May 11 17:03:39 
>> 2016', 
>>>> queued=0ms, exec=82ms
>>>>
>>>> Correct, OCF_ERR_CONFIGURED is a "fatal" error:
>>>>
>>>>
>> http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explai
>>  
>>
>>>> ned/index.html#_how_are_ocf_return_codes_interpreted
>>>
>>> I think the mistake here is assuming that OCF_ERR_CONFIGURED only depends 
>>> on 
>> the RA parameters, when in fact the validity of RA params may depend on the 
>> environment found at the time of checking. And as we all now the environment 
>> changes, especially when resources are started and stopped.

It's not a mistake; it's the definition of OCF_ERR_CONFIGURED. The
various documentation could be more clear, but the intent is: the
resource's configuration *in the cluster configuration* is inherently
invalid, and could never work, regardless of what system it was on.
Examples listed include a required parameter not being specified, or a
nonnumeric value supplied for an integer parameter.

OCF_ERR_CONFIGURED is the *only* fatal error (fatal = the operation will
not be retried anywhere).

Of course, as you mentioned later, here you actually have OCF_ERR_ARGS
(2) -- I saw "invalid parameter" and mistakenly thought
OCF_ERR_CONFIGURED ...

>>>>> So you would say that monitor may only return "success" or "not running", 
>>>> but I feel the RA should detect the condition that the resource could not 
>> run 
>>>> at all at the present state.
>>>>
>>>> OCF_ERR_CONFIGURED is meant to indicate that the resource could not
>>>> possibly run *as configured*, regardless of the system's current state.
>>>
>>> But how do you handle parameters that describe file or host names (which 
>>> may 
>> exist or not independently of a change in the param's value)?
>>
>> The RA should've exited with OCF_ERR_INSTALLED. That's the code
>> which means that there's something wrong with the environment on
>> this node, but that the resource could be started on another one.
> 
> Really, besides implementation, I don't see why OCF_ERR_INSTALLED is less 
> permanent than  OCF_ERR_ARGS.
> 
>>
>>>> So for example, a required parameter is missing or invalid.
>>>>
>>>> You could possibly use OCF_ERR_ARGS in this case (a "hard" error that
>>>> bans the particular node, and means that the resource's configuration is
>>>> not valid on this particular node).
>>>
>>> ("rc=2" _is_ OCF_ERR_ARGS)

OCF_ERR_ARGS and OCF_ERR_INSTALLED are both hard errors (hard = the
operation will not be retried on this node).

Looking at the code, OCF_ERR_INSTALLED is treated differently for probes
(it's not recorded as a failed op), but I think the node would still get
banned from running the resource.

I totally understand your point now: The probe may hit one of these
conditions only because it is called before depended-on resources are
up, but the cluster doesn't really care -- it just wants to know whether
the resource is running. Using ocf_is_probe to mask these errors works,
but could be considered a less-than-ideal wor

Re: [ClusterLabs] start a resource

[Ken Gaillot]

On 05/06/2016 01:01 PM, Dimitri Maziuk wrote:
> On 05/06/2016 12:05 PM, Ian wrote:
>> Are you getting any other errors now that you've fixed the
>> config?
> 
> It's running now that I did the cluster stop/start, but no: I
> wasn't getting any other errors. I did have a symlink resource
> "stopped" for no apparent reason and with no errors logged.
> 
> The cluster is a basic active-passive pair. The relevant part of
> the setup is:
> 
> drbd filesystem floating ip colocated with drbd filesystem +inf 
> order drbd filesystem then floating ip
> 
> ocf:heartbeat:symlink resource that does /etc/rsyncd.conf ->
> /drbd/etc/rsyncd.conf colocated with drbd filesystem +inf order
> drbd filesystem then the symlink
> 
> ocf:heartbeat:rsyncd resource that is colocated with the symlink 
> order symlink then rsyncd order floating ip then rsyncd
> 
> (Looking at this, maybe I should also colocate rsyncd with floating
> ip to avoid any confusion in pacemaker's little brain.)

Not strictly necessary, since rsync is colocated with symlink which is
colocated with filesystem, and ip is also colocated with filesystem.

But it is a good idea to model all logical dependencies, since you
don't know what changes you might make to the configuration in the
future. If you want rsyncd to always be with the floating ip, then by
all means add a colocation constraint.

> But this is not specific to rsyncd: the behaviour was exactly the
> same when a co-worker made a typo in apache config (which is
> another resource on the same cluster). The only way to restart
> apache was to "pcs cluster stop ; pcs cluster start" and that
> randomly killed ssh connections to the nodes' "proper" IPs.

That is definitely not a properly functioning cluster. Something is
going wrong at some level.

When you say that "pcs resource cleanup" didn't fix the issue, what
happened after that? Did "pcs status" still show an error for the
resource? If so, there was an additional failure.


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Resource failure-timeout does not reset when resource fails to connect to both nodes

[Ken Gaillot]

On 03/28/2016 11:44 AM, Sam Gardner wrote:
> I have a simple resource defined:
> 
> [root@ha-d1 ~]# pcs resource show dmz1
>  Resource: dmz1 (class=ocf provider=internal type=ip-address)
>   Attributes: address=172.16.10.192 monitor_link=true
>   Meta Attrs: migration-threshold=3 failure-timeout=30s
>   Operations: monitor interval=7s (dmz1-monitor-interval-7s)
> 
> This is a custom resource which provides an ethernet alias to one of the
> interfaces on our system.
> 
> I can unplug the cable on either node and failover occurs as expected,
> and 30s after re-plugging it I can repeat the exercise on the opposite
> node and failover will happen as expected.
> 
> However, if I unplug the cable from both nodes, the failcount goes up,
> and the 30s failure-timeout does not reset the failcounts, meaning that
> pacemaker never tries to start the failed resource again.

Apologies for the late response, but:

Time-based actions in Pacemaker, including failure-timeout, are not
guaranteed to be checked more frequently than the value of the
cluster-recheck-interval cluster property, which defaults to 15 minutes.

If the cluster responds to an event (node joining/leaving, monitor
failure, etc.), it will check time-based actions at that point, but
otherwise it doesn't. So cluster-recheck-interval acts as a maximum time
between such checks.

Try lowering your cluster-recheck-interval. Personally, I would think
30s for a failure-timeout is awfully quick; it would lead to continuous
retries. And it would require setting cluster-recheck-interval to
something similar, which would add a lot of computational overhead to
the cluster.

I'm curious what values of cluster-recheck-interval and failure-timeout
people are commonly using "in the wild". On a small, underutilized
cluster, you could probably get away with setting them quite low, but on
larger clusters, I would expect it would be too much overhead.

> Full list of resources:
> 
>  Resource Group: network
>  inif   (off::internal:ip.sh):   Started ha-d1.dev.com
>  outif  (off::internal:ip.sh):   Started ha-d2.dev.com
>  dmz1   (off::internal:ip.sh):   Stopped
>  Master/Slave Set: DRBDMaster [DRBDSlave]
>  Masters: [ ha-d1.dev.com ]
>  Slaves: [ ha-d2.dev.com ]
>  Resource Group: filesystem
>  DRBDFS (ocf::heartbeat:Filesystem):Stopped
>  Resource Group: application
>  service_failover   (off::internal:service_failover):Stopped
> 
> Failcounts for dmz1
>  ha-d1.dev.com: 4
>  ha-d2.dev.com: 4
> 
> Is there any way to automatically recover from this scenario, other than
> setting an obnoxiously high migration-threshold? 
> 
> -- 
> 
> *Sam Gardner   *
> 
> Software Engineer
> 
> *Trustwave** *| SMART SECURITY ON DEMAND

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] unable to start fence_scsi

[Ken Gaillot]

 fence (on) 'apache-up002.ring0' with device 
>> '(any)'
>> May 18 10:37:03 apache-up001 stonith-ng[15914]:  notice: Initiating remote 
>> operation on for apache-up002.ring0: 76aba815-280e-491a-bd17-40776c8169e9 (0)
>> May 18 10:37:03 apache-up001 crmd[15918]:  notice: Initiating action 3: 
>> probe_complete probe_complete-apache-up001.ring0 on apache-up001.ring0 
>> (local) - no waiting
>> May 18 10:37:03 apache-up001 stonith-ng[15914]:  notice: Client 
>> crmd.15918.697c495e wants to fence (on) 'apache-up001.ring0' with device 
>> '(any)'
>> May 18 10:37:03 apache-up001 stonith-ng[15914]:  notice: Initiating remote 
>> operation on for apache-up001.ring0: e50d7e16-9578-4964-96a3-7b36bdcfba46 (0)
>> May 18 10:37:03 apache-up001 stonith-ng[15914]:  notice: Couldn't find 
>> anyone to fence (on) apache-up003.ring0 with any device
>> May 18 10:37:03 apache-up001 stonith-ng[15914]:  notice: Couldn't find 
>> anyone to fence (on) apache-up002.ring0 with any device
>> May 18 10:37:03 apache-up001 stonith-ng[15914]:   error: Operation on of 
>> apache-up003.ring0 by  for crmd.15918@apache-up001.ring0.0599387e: 
>> No such device
>> May 18 10:37:03 apache-up001 stonith-ng[15914]:   error: Operation on of 
>> apache-up002.ring0 by  for crmd.15918@apache-up001.ring0.76aba815: 
>> No such device
>> May 18 10:37:03 apache-up001 stonith-ng[15914]:  notice: Couldn't find 
>> anyone to fence (on) apache-up001.ring0 with any device
>> May 18 10:37:03 apache-up001 crmd[15918]:  notice: Stonith operation 
>> 5/11:11:0:8248cebf-c198-4ff2-bd43-7415533ce50f: No such device (-19)
>> May 18 10:37:03 apache-up001 stonith-ng[15914]:   error: Operation on of 
>> apache-up001.ring0 by  for crmd.15918@apache-up001.ring0.e50d7e16: 
>> No such device
>> May 18 10:37:03 apache-up001 crmd[15918]:  notice: Stonith operation 5 for 
>> apache-up003.ring0 failed (No such device): aborting transition.
>> May 18 10:37:03 apache-up001 crmd[15918]:  notice: Transition aborted: 
>> Stonith failed (source=tengine_stonith_callback:733, 0)
>> May 18 10:37:03 apache-up001 crmd[15918]:   error: Unfencing of 
>> apache-up003.ring0 by  failed: No such device (-19)
>> May 18 10:37:03 apache-up001 crmd[15918]:  notice: Stonith operation 
>> 6/8:11:0:8248cebf-c198-4ff2-bd43-7415533ce50f: No such device (-19)
>> May 18 10:37:03 apache-up001 crmd[15918]:  notice: Stonith operation 6 for 
>> apache-up002.ring0 failed (No such device): aborting transition.
>> May 18 10:37:03 apache-up001 crmd[15918]:   error: Unfencing of 
>> apache-up002.ring0 by  failed: No such device (-19)
>> May 18 10:37:03 apache-up001 crmd[15918]:  notice: Stonith operation 
>> 7/5:11:0:8248cebf-c198-4ff2-bd43-7415533ce50f: No such device (-19)
>> May 18 10:37:03 apache-up001 crmd[15918]:  notice: Stonith operation 7 for 
>> apache-up001.ring0 failed (No such device): aborting transition.
>> May 18 10:37:03 apache-up001 crmd[15918]:   error: Unfencing of 
>> apache-up001.ring0 by  failed: No such device (-19)
>> May 18 10:37:03 apache-up001 crmd[15918]:  notice: Initiating action 10: 
>> monitor scsia_monitor_0 on apache-up003.ring0
>> May 18 10:37:03 apache-up001 crmd[15918]:  notice: Initiating action 7: 
>> monitor scsia_monitor_0 on apache-up002.ring0
>> May 18 10:37:03 apache-up001 crmd[15918]:  notice: Initiating action 4: 
>> monitor scsia_monitor_0 on apache-up001.ring0 (local)
>> May 18 10:37:03 apache-up001 crmd[15918]:  notice: Operation 
>> scsia_monitor_0: not running (node=apache-up001.ring0, call=19, rc=7, 
>> cib-update=59, confirmed=true)
>> May 18 10:37:03 apache-up001 crmd[15918]:  notice: Transition 11 
>> (Complete=10, Pending=0, Fired=0, Skipped=1, Incomplete=2, 
>> Source=/var/lib/pacemaker/pengine/pe-input-95.bz2): Stopped
>> May 18 10:37:03 apache-up001 crmd[15918]:  notice: No devices found in 
>> cluster to fence apache-up001.ring0, giving up
>> May 18 10:37:03 apache-up001 crmd[15918]:  notice: State transition 
>> S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL 
>> origin=notify_crmd ]
>>
>>
>>
>>
>>
>>
>>
>>> On 16 May 2016, at 16:22, Ken Gaillot <kgail...@redhat.com> wrote:
>>>
>>> On 05/14/2016 08:54 AM, Marco A. Carcano wrote:
>>>> I hope to find here someone who can help me:
>>>>
>>>> I have a 3 node cluster and I’m struggling to create a GFSv2 shared 
>>>> storage. The  weird thing is that despite cluster seems OK, I’m not able 
>>>> to have the fence_scsi stonith device managed, and this prevent CLVMD and 
>>>&g

Re: [ClusterLabs] Unable to run Pacemaker: pcmk_child_exit

[Ken Gaillot]

On 05/05/2016 11:25 AM, Nikhil Utane wrote:
> Thanks Ken for your quick response as always.
> 
> But what if I don't want to use quorum? I just want to bring up
> pacemaker + corosync on 1 node to check that it all comes up fine.
> I added corosync_votequorum as you suggested. Additionally I also added
> these 2 lines:
> 
> expected_votes: 2
> two_node: 1

There's actually nothing wrong with configuring a single-node cluster.
You can list just one node in corosync.conf and leave off the above.

> However still pacemaker is not able to run.

There must be other issues involved. Even if pacemaker doesn't have
quorum, it will still run, it just won't start resources.

> [root@airv_cu root]# pcs cluster start
> Starting Cluster...
> Starting Pacemaker Cluster Manager[FAILED]
> 
> Error: unable to start pacemaker
> 
> Corosync.log:
> *May 05 16:15:20 [16294] airv_cu pacemakerd: info:
> pcmk_quorum_notification: Membership 240: quorum still lost (1)*
> May 05 16:15:20 [16259] airv_cu corosync debug   [QB] Free'ing
> ringbuffer: /dev/shm/qb-cmap-request-16259-16294-21-header
> May 05 16:15:20 [16294] airv_cu pacemakerd:   notice:
> crm_update_peer_state_iter:   pcmk_quorum_notification: Node
> airv_cu[181344357] - state is now member (was (null))
> May 05 16:15:20 [16294] airv_cu pacemakerd: info:
> pcmk_cpg_membership:  Node 181344357 joined group pacemakerd
> (counter=0.0)
> May 05 16:15:20 [16294] airv_cu pacemakerd: info:
> pcmk_cpg_membership:  Node 181344357 still member of group
> pacemakerd (peer=airv_cu, counter=0.0)
> May 05 16:15:20 [16294] airv_cu pacemakerd:  warning: pcmk_child_exit:
>  The cib process (16353) can no longer be respawned, shutting the
> cluster down.
> May 05 16:15:20 [16294] airv_cu pacemakerd:   notice:
> pcmk_shutdown_worker: Shutting down Pacemaker
> 
> The log and conf file is attached.
> 
> -Regards
> Nikhil
> 
> On Thu, May 5, 2016 at 8:04 PM, Ken Gaillot <kgail...@redhat.com
> <mailto:kgail...@redhat.com>> wrote:
> 
> On 05/05/2016 08:36 AM, Nikhil Utane wrote:
> > Hi,
> >
> > Continuing with my adventure to run Pacemaker & Corosync on our
> > big-endian system, I managed to get past the corosync issue for now. But
> > facing an issue in running Pacemaker.
> >
> > Seeing following messages in corosync.log.
> >  pacemakerd:  warning: pcmk_child_exit:  The cib process (2) can no
> > longer be respawned, shutting the cluster down.
> >  pacemakerd:  warning: pcmk_child_exit:  The stonith-ng process (20001)
> > can no longer be respawned, shutting the cluster down.
> >  pacemakerd:  warning: pcmk_child_exit:  The lrmd process (20002) can no
> > longer be respawned, shutting the cluster down.
> >  pacemakerd:  warning: pcmk_child_exit:  The attrd process (20003) can
> > no longer be respawned, shutting the cluster down.
> >  pacemakerd:  warning: pcmk_child_exit:  The pengine process (20004) can
> > no longer be respawned, shutting the cluster down.
> >  pacemakerd:  warning: pcmk_child_exit:  The crmd process (20005) can no
> > longer be respawned, shutting the cluster down.
> >
> > I see following error before these messages. Not sure if this is the 
> cause.
> > May 05 11:26:24 [19998] airv_cu pacemakerd:error:
> > cluster_connect_quorum:   Corosync quorum is not configured
> >
> > I tried removing the quorum block (which is anyways blank) from the conf
> > file but still had the same error.
> 
> Yes, that is the issue. Pacemaker can't do anything if it can't ask
> corosync about quorum. I don't know what the issue is at the corosync
> level, but your corosync.conf should have:
> 
> quorum {
> provider: corosync_votequorum
> }
> 
> 
> > Attaching the log and conf files. Please let me know if there is any
> > obvious mistake or how to investigate it further.
> >
> > I am using pcs cluster start command to start the cluster
> >
> > -Thanks
> > Nikhil

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Coming in 1.1.15: Event-driven alerts

[Ken Gaillot]

On 04/25/2016 07:28 AM, Lars Ellenberg wrote:
> On Thu, Apr 21, 2016 at 12:50:43PM -0500, Ken Gaillot wrote:
>> Hello everybody,
>>
>> The release cycle for 1.1.15 will be started soon (hopefully tomorrow)!
>>
>> The most prominent feature will be Klaus Wenninger's new implementation
>> of event-driven alerts -- the ability to call scripts whenever
>> interesting events occur (nodes joining/leaving, resources
>> starting/stopping, etc.).
> 
> What exactly is "etc." here?
> What is the comprehensive list
> of which "events" will trigger "alerts"?

The exact list should be documented in Pacemaker Explained before the
final 1.1.15 release. I think it's comparable to what crm_mon -E does
currently. The basic categories are node events, fencing events, and
resource events.

> My guess would be
>  DC election/change
>which does not necessarily imply membership change
>  change in membership
>which includes change in quorum
>  fencing events
>(even failed fencing?)
>  resource start/stop/promote/demote
>   (probably) monitor failure?
>maybe only if some fail-count changes to/from infinity?
>or above a certain threshold?
> 
>  change of maintenance-mode?
>  node standby/online (maybe)?
>  maybe "resource cannot be run anywhere"?

It would certainly be possible to expand alerts to more situations if
there is a need. I think the existing ones will be sufficient for common
use cases though.

> would it be useful to pass in the "transaction ID"
> or other pointer to the recorded cib input at the time
> the "alert" was triggered?

Possibly, though it isn't currently. We do pass a node-local counter and
a subsecond-resolution timestamp, to help with ordering.

> can an alert "observer" (alert script) "register"
> for only a subset of the "alerts"?

Not explicitly, but the alert type is passed in as an environment
variable, so the script can simply exit for "uninteresting" event types.
That's not as efficient since the process must still be spawned, but it
simplifies things.

> if so, can this filter be per alert script,
> or per "recipient", or both?
> 
> Thanks,
> 
> Lars
> 

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] FR: send failcount to OCF RA start/stop actions

[Ken Gaillot]

On 05/04/2016 08:49 AM, Klaus Wenninger wrote:
> On 05/04/2016 02:09 PM, Adam Spiers wrote:
>> Hi all,
>>
>> As discussed with Ken and Andrew at the OpenStack summit last week, we
>> would like Pacemaker to be extended to export the current failcount as
>> an environment variable to OCF RA scripts when they are invoked with
>> 'start' or 'stop' actions.  This would mean that if you have
>> start-failure-is-fatal=false and migration-threshold=3 (say), then you
>> would be able to implement a different behaviour for the third and
>> final 'stop' of a service executed on a node, which is different to
>> the previous 'stop' actions executed just prior to attempting a
>> restart of the service.  (In the non-clone case, this would happen
>> just before migrating the service to another node.)
> So what you actually want to know is how much headroom
> there still is till the resource would be migrated.
> So wouldn't it then be much more catchy if we don't pass
> the failcount but rather the headroom?

Yes, that's the plan: pass a new environment variable with
(migration-threshold - fail-count) when recovering a resource. I haven't
worked out the exact behavior yet, but that's the idea. I do hope to get
this in 1.1.15 since it's a small change.

The advantage over using crm_failcount is that it will be limited to the
current recovery attempt, and it will calculate the headroom as you say,
rather than the raw failcount.

>> One use case for this is to invoke "nova service-disable" if Pacemaker
>> fails to restart the nova-compute service on an OpenStack compute
>> node.
>>
>> Is it feasible to squeeze this in before the 1.1.15 release?
>>
>> Thanks a lot!
>> Adam

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] ringid interface FAULTY no resource move

[Ken Gaillot]

On 05/04/2016 07:14 AM, Rafał Sanocki wrote:
> Hello,
> I cant find what i did wrong. I have 2 node cluster, Corosync ,Pacemaker
> , DRBD .  When i plug out cable nothing happend.
> 
> Corosync.conf
> 
> # Please read the corosync.conf.5 manual page
> totem {
> version: 2
> crypto_cipher: none
> crypto_hash: none
> rrp_mode: passive
> 
> interface {
> ringnumber: 0
> bindnetaddr: 172.17.10.0
> mcastport: 5401
> ttl: 1
> }
> interface {
> ringnumber: 1
> bindnetaddr: 255.255.255.0
> mcastport: 5409
> ttl: 1
> }

255.255.255.0 is not a valid bindnetaddr. bindnetaddr should be the IP
network address (not netmask) of the desired interface.

Also, the point of rrp is to have two redundant network links. So
unplugging one shouldn't cause problems, if the other is still up.

> 
> transport: udpu
> }
> 
> logging {
> fileline: off
> to_logfile: yes
> to_syslog: yes
> logfile: /var/log/cluster/corosync.log
> debug: off
> timestamp: on
> logger_subsys {
> subsys: QUORUM
> debug: off
> }
> }
> 
> nodelist {
> node {
> ring0_addr: 172.17.10.81
> ring1_addr: 255.255.255.1
> nodeid: 1
> }
> node {
> ring0_addr: 172.17.10.89
> ring1_addr: 255.255.255.9
> nodeid: 2
> }
> 
> }
> quorum {
> # Enable and configure quorum subsystem (default: off)
> # see also corosync.conf.5 and votequorum.5
> provider: corosync_votequorum
> }
> 
> crm config
> 
> crm(live)configure# show
> node 1: cs01A
> node 2: cs01B
> primitive p_drbd2dev ocf:linbit:drbd \
> params drbd_resource=b1 \
> op monitor interval=29s role=Master \
> op monitor interval=31s role=Slave \
> meta target-role=Started
> primitive p_exportfs_fs2 exportfs \
> params fsid=101 directory="/data1/b1"
> options="rw,sync,no_root_squash,insecure,anonuid=100,anongid=101,nohide"
> clientspec="172.17.10.0/255.255.255.0" wait_for_leasetime_on_stop=false \
> op monitor interval=30s \
> op start interval=0 timeout=240s \
> op stop interval=0 timeout=100s \
> meta target-role=Started
> primitive p_ip_2 IPaddr2 \
> params ip=172.17.10.97 nic=neteth0 cidr_netmask=24 \
> op monitor interval=30s timeout=5s \
> meta target-role=Started
> primitive p_mount_fs2 Filesystem \
> params fstype=reiserfs options="noatime,nodiratime,notail"
> directory="/data1" device="/dev/drbd2" \
> op start interval=0 timeout=400s \
> op stop interval=0 timeout=100s \
> op monitor interval=30s \
> meta target-role=Started
> group g_nfs2 p_ip_2 p_mount_fs2 p_exportfs_fs2
> ms ms_drbd2 p_drbd2dev \
> meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1
> notify=true is-managed=true target-role=Slave
> colocation co_drbd2 inf: g_nfs2 ms_drbd2:Master
> order ms_drbd2_order Mandatory: ms_drbd2:promote g_nfs2:start
> property cib-bootstrap-options: \
> stonith-enabled=false \
> have-watchdog=true \
> dc-version=1.1.14-535193a \
> cluster-infrastructure=corosync \
> maintenance-mode=false \
> no-quorum-policy=ignore \
> last-lrm-refresh=1460627538
> 
> 
> # ip addr show
> neteth1:  mtu 1500 qdisc mq portid
> d8d385bda90c state DOWN group default qlen 1000
> link/ether d8:d3:85:aa:aa:aa brd ff:ff:ff:ff:ff:ff
> inet 255.255.255.1/24 brd 255.255.255.255 scope global neteth1
>valid_lft forever preferred_lft forever
> 
> # corosync-cfgtool -s
> Printing ring status.
> Local node ID 1
> RING ID 0
> id  = 172.17.10.81
> status  = ring 0 active with no faults
> RING ID 1
> id  = 255.255.255.1
> status  = Marking ringid 1 interface 255.255.255.1 FAULTY
> 
> #crm_mon -A
> 
> Stack: corosync
> Current DC: csb01A (version 1.1.14-535193a) - partition with quorum
> Last updated: Wed May  4 14:11:34 2016  Last change: Thu Apr 14
> 13:06:15 2016 by root via crm_resource on csb01B
> 
> 2 nodes and 5 resources configured: 2 resources DISABLED and 0 BLOCKED
> from being started due to failures
> 
> Online: [ cs01A cs01B ]
> 
>  Resource Group: g_nfs2
>  p_ip_2 (ocf::heartbeat:IPaddr2):   Started csb01A
>  p_mount_fs2(ocf::heartbeat:Filesystem):Started csb01A
>  p_exportfs_fs2 (ocf::heartbeat:exportfs):  Started csb01A
>  Master/Slave Set: ms_drbd2 [p_drbd2dev]
>  Masters: [ csb01A ]
>  Slaves (target-role): [ csb01B ]
> 
> Node Attributes:
> * Node csb01A:
> + master-p_drbd2dev : 1
> * Node csb01B:
> + 

Re: [ClusterLabs] why and when a call of crm_attribute can be delayed ?

[Ken Gaillot]

On 04/25/2016 05:02 AM, Jehan-Guillaume de Rorthais wrote:
> Hi all,
> 
> I am facing a strange issue with attrd while doing some testing on a three 
> node
> cluster with the pgsqlms RA [1].
> 
> pgsqld is my pgsqlms resource in the cluster. pgsql-ha is the master/slave
> setup on top of pgsqld.
> 
> Before triggering a failure, here was the situation:
> 
>   * centos1: pgsql-ha slave
>   * centos2: pgsql-ha slave
>   * centos3: pgsql-ha master
> 
> Then we triggered a failure: the node centos3 has been kill using 
> 
>   echo c > /proc/sysrq-trigger
> 
> In this situation, PEngine provide a transition where :
> 
>   * centos3 is fenced 
>   * pgsql-ha on centos2 is promoted
> 
> During the pre-promote notify action in the pgsqlms RA, each remaining slave 
> are
> setting a node attribute called lsn_location, see: 
> 
>   https://github.com/dalibo/PAF/blob/master/script/pgsqlms#L1504
> 
>   crm_attribute -l reboot -t status --node "$nodename" \
> --name lsn_location --update "$node_lsn"
> 
> During the promotion action in the pgsqlms RA, the RA check the lsn_location 
> of
> the all the nodes to make sure the local one is higher or equal to all others.
> See:
> 
>   https://github.com/dalibo/PAF/blob/master/script/pgsqlms#L1292
> 
> This is where we face a attrd behavior we don't understand.
> 
> Despite we can see in the log the RA was able to set its local
> "lsn_location", during the promotion action, the RA was unable to read its
> local lsn_location":
> 
>   pgsqlms(pgsqld)[9003]:  2016/04/22_14:46:16  
> INFO: pgsql_notify: promoting instance on node "centos2" 
> 
>   pgsqlms(pgsqld)[9003]:  2016/04/22_14:46:16  
> INFO: pgsql_notify: current node LSN: 0/1EE24000 
> 
>   [...]
> 
>   pgsqlms(pgsqld)[9023]:  2016/04/22_14:46:16
> CRIT: pgsql_promote: can not get current node LSN location
> 
>   Apr 22 14:46:16 [5864] centos2   lrmd:
> notice: operation_finished: pgsqld_promote_0:9023:stderr 
> [ Error performing operation: No such device or address ] 
> 
>   Apr 22 14:46:16 [5864] centos2   lrmd: 
> info: log_finished:  finished - rsc:pgsqld
> action:promote call_id:211 pid:9023 exit-code:1 exec-time:107ms
> queue-time:0ms
> 
> The error comes from:
> 
>   https://github.com/dalibo/PAF/blob/master/script/pgsqlms#L1320
> 
> **After** this error, we can see in the log file attrd set the "lsn_location" 
> of
> centos2:
> 
>   Apr 22 14:46:16 [5865] centos2
> attrd: info: attrd_peer_update:
> Setting lsn_location[centos2]: (null) -> 0/1EE24000 from centos2 
> 
>   Apr 22 14:46:16 [5865] centos2
> attrd: info: write_attribute:   
> Write out of 'lsn_location' delayed:update 189 in progress
> 
> 
> As I understand it, the call of crm_attribute during pre-promote notification
> has been taken into account AFTER the "promote" action, leading to this error.
> Am I right?
> 
> Why and how this could happen? Could it comes from the dampen parameter? We 
> did
> not set any dampen anywhere, is there a default value in the cluster setup?
> Could we avoid this behavior?

Unfortunately, that is expected. Both the cluster's call of the RA's
notify action, and the RA's call of crm_attribute, are asynchronous. So
there is no guarantee that anything done by the pre-promote notify will
be complete (or synchronized across other cluster nodes) by the time the
promote action is called.

There would be no point in the pre-promote notify waiting for the
attribute value to be retrievable, because the cluster isn't going to
wait for the pre-promote notify to finish before calling promote.

Maybe someone else can come up with a better idea, but I'm thinking
maybe the attribute could be set as timestamp:lsn, and the promote
action could poll attrd repeatedly (for a small duration lower than the
typical promote timeout) until it gets lsn's with a recent timestamp
from all nodes. One error condition to handle would be if one of the
other slaves happens to fail or be unresponsive at that time.

> Please, find in attachment a tarball with :
>   * all cluster logfiles from the three nodes
>   * the content of /var/lib/pacemaker from the three nodes:
> * CIBs
> * PEngine transitions
> 
> 
> Regards,
> 
> [1] https://github.com/dalibo/PAF
> 


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Running several instances of a Corosync/Pacemaker cluster on a node

[Ken Gaillot]

On 04/26/2016 03:33 AM, Bogdan Dobrelya wrote:
> Is it possible to run several instances of a Corosync/Pacemaker clusters
> on a node? Can a node be a member of several clusters, so they could put
> resources there? I'm sure it's doable with separate nodes or containers,
> but that's not the case.
> 
> My case is to separate data-critical resources, like storage or VIPs,
> from the complex resources like DB or MQ clusters.
> 
> The latter should run with no-quorum-policy=ignore as they know how to
> deal with network partitions/split-brain, use own techniques to protect
> data and don't want external fencing from a Pacemaker, which
> no-quorum-policy/STONITH is.
> 
> The former must use STONITH (or a stop policy, if it's only a VIP), as
> they don't know how to deal with split-brain, for example.

I don't think it's possible, though I could be wrong, if separate
IPs/ports, chroots and node names are used (just shy of a container ...).

However I suspect it would not meet your goal in any case. DB and MQ
software generally do NOT have sufficient techniques to deal with a
split-brain situation -- either you lose high availability or you
corrupt data. Using no-quorum-policy=stop is fine for handling network
splits, but it does not help if a node becomes unresponsive.

Also note that pacemaker does provide the ability to treat different
resources differently with respect to quorum and fencing, without
needing to run separate clusters. See the "required" meta-attribute:

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#_resource_meta_attributes

I suspect your motive for this is to be able to run a cluster without
fencing. There are certain failure scenarios that simply are not
recoverable without fencing, regardless of what the application software
can do. There is really only one case in which doing without fencing is
reasonable: when you're willing to lose your data and/or have downtime
when a situation arises that requires fencing.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Node is silently unfenced if transition is very long

[Ken Gaillot]

On 04/19/2016 10:47 AM, Vladislav Bogdanov wrote:
> Hi,
> 
> Just found an issue with node is silently unfenced.
> 
> That is quite large setup (2 cluster nodes and 8 remote ones) with
> a plenty of slowly starting resources (lustre filesystem).
> 
> Fencing was initiated due to resource stop failure.
> lustre often starts very slowly due to internal recovery, and some such
> resources were starting in that transition where another resource failed to 
> stop.
> And, as transition did not finish in time specified by the
> "failure-timeout" (set to 9 min), and was not aborted, that stop failure was 
> successfully cleaned.
> There were transition aborts due to attribute changes, after that stop 
> failure happened, but fencing
> was not initiated for some reason.

Unfortunately, that makes sense with the current code. Failure timeout
changes the node attribute, which aborts the transition, which causes a
recalculation based on the new state, and the fencing is no longer
needed. I'll make a note to investigate a fix, but feel free to file a
bug report at bugs.clusterlabs.org for tracking purposes.

> Node where stop failed was a DC.
> pacemaker is 1.1.14-5a6cdd1 (from fedora, built on EL7)
> 
> Here is log excerpt illustrating the above:
> Apr 19 14:57:56 mds1 pengine[3452]:   notice: Movemdt0-es03a-vg
> (Started mds1 -> mds0)
> Apr 19 14:58:06 mds1 pengine[3452]:   notice: Movemdt0-es03a-vg
> (Started mds1 -> mds0)
> Apr 19 14:58:10 mds1 crmd[3453]:   notice: Initiating action 81: monitor 
> mdt0-es03a-vg_monitor_0 on mds0
> Apr 19 14:58:11 mds1 crmd[3453]:   notice: Initiating action 2993: stop 
> mdt0-es03a-vg_stop_0 on mds1 (local)
> Apr 19 14:58:11 mds1 LVM(mdt0-es03a-vg)[6228]: INFO: Deactivating volume 
> group vg_mdt0_es03a
> Apr 19 14:58:12 mds1 LVM(mdt0-es03a-vg)[6541]: ERROR: Logical volume 
> vg_mdt0_es03a/mdt0 contains a filesystem in use. Can't deactivate volume 
> group "vg_mdt0_es03a" with 1 open logical volume(s)
> [...]
> Apr 19 14:58:30 mds1 LVM(mdt0-es03a-vg)[9939]: ERROR: LVM: vg_mdt0_es03a did 
> not stop correctly
> Apr 19 14:58:30 mds1 LVM(mdt0-es03a-vg)[9943]: WARNING: vg_mdt0_es03a still 
> Active
> Apr 19 14:58:30 mds1 LVM(mdt0-es03a-vg)[9947]: INFO: Retry deactivating 
> volume group vg_mdt0_es03a
> Apr 19 14:58:31 mds1 lrmd[3450]:   notice: mdt0-es03a-vg_stop_0:5865:stderr [ 
> ocf-exit-reason:LVM: vg_mdt0_es03a did not stop correctly ]
> [...]
> Apr 19 14:58:31 mds1 lrmd[3450]:   notice: mdt0-es03a-vg_stop_0:5865:stderr [ 
> ocf-exit-reason:LVM: vg_mdt0_es03a did not stop correctly ]
> Apr 19 14:58:31 mds1 crmd[3453]:   notice: Operation mdt0-es03a-vg_stop_0: 
> unknown error (node=mds1, call=324, rc=1, cib-update=1695, confirmed=true)
> Apr 19 14:58:31 mds1 crmd[3453]:   notice: mds1-mdt0-es03a-vg_stop_0:324 [ 
> ocf-exit-reason:LVM: vg_mdt0_es03a did not stop 
> correctly\nocf-exit-reason:LVM: vg_mdt0_es03a did not stop 
> correctly\nocf-exit-reason:LVM: vg_mdt0_es03a did not stop 
> correctly\nocf-exit-reason:LVM: vg_mdt0_es03a did not stop 
> correctly\nocf-exit-reason:LVM: vg_mdt0_es03a did not stop 
> correctly\nocf-exit-reason:LVM: vg_mdt0_es03a did not stop 
> correctly\nocf-exit-reason:LVM: vg_mdt0_es03a did not stop 
> correctly\nocf-exit-reason:LVM: vg_mdt0_es03a did not stop correctl
> Apr 19 14:58:31 mds1 crmd[3453]:  warning: Action 2993 (mdt0-es03a-vg_stop_0) 
> on mds1 failed (target: 0 vs. rc: 1): Error
> Apr 19 14:58:31 mds1 crmd[3453]:  warning: Action 2993 (mdt0-es03a-vg_stop_0) 
> on mds1 failed (target: 0 vs. rc: 1): Error
> Apr 19 15:02:03 mds1 pengine[3452]:  warning: Processing failed op stop for 
> mdt0-es03a-vg on mds1: unknown error (1)
> Apr 19 15:02:03 mds1 pengine[3452]:  warning: Processing failed op stop for 
> mdt0-es03a-vg on mds1: unknown error (1)
> Apr 19 15:02:03 mds1 pengine[3452]:  warning: Node mds1 will be fenced 
> because of resource failure(s)
> Apr 19 15:02:03 mds1 pengine[3452]:  warning: Forcing mdt0-es03a-vg away from 
> mds1 after 100 failures (max=100)
> Apr 19 15:02:03 mds1 pengine[3452]:  warning: Scheduling Node mds1 for STONITH
> Apr 19 15:02:03 mds1 pengine[3452]:   notice: Stop of failed resource 
> mdt0-es03a-vg is implicit after mds1 is fenced
> Apr 19 15:02:03 mds1 pengine[3452]:   notice: Recover mdt0-es03a-vg
> (Started mds1 -> mds0)
> [... many of these ]
> Apr 19 15:07:22 mds1 pengine[3452]:  warning: Processing failed op stop for 
> mdt0-es03a-vg on mds1: unknown error (1)
> Apr 19 15:07:22 mds1 pengine[3452]:  warning: Processing failed op stop for 
> mdt0-es03a-vg on mds1: unknown error (1)
> Apr 19 15:07:22 mds1 pengine[3452]:  warning: Node mds1 will be fenced 
> because of resource failure(s)
> Apr 19 15:07:22 mds1 pengine[3452]:  warning: Forcing mdt0-es03a-vg away from 
> mds1 after 100 failures (max=100)
> Apr 19 15:07:23 mds1 pengine[3452]:  warning: Scheduling Node mds1 for STONITH
> Apr 19 15:07:23 mds1 pengine[3452]:   notice: Stop of failed resource 
> 

Re: [ClusterLabs] Coming in 1.1.15: Event-driven alerts

[Ken Gaillot]

On 04/22/2016 05:55 PM, Adam Spiers wrote:
> Ken Gaillot <kgail...@redhat.com> wrote:
>> On 04/21/2016 06:09 PM, Adam Spiers wrote:
>>> Ken Gaillot <kgail...@redhat.com> wrote:
>>>> Hello everybody,
>>>>
>>>> The release cycle for 1.1.15 will be started soon (hopefully tomorrow)!
>>>>
>>>> The most prominent feature will be Klaus Wenninger's new implementation
>>>> of event-driven alerts -- the ability to call scripts whenever
>>>> interesting events occur (nodes joining/leaving, resources
>>>> starting/stopping, etc.).
>>>
>>> Ooh, that sounds cool!  Can it call scripts after fencing has
>>> completed?  And how is it determined which node the script runs on,
>>> and can that be limited via constraints or similar?
>>
>> Yes, it called after all "interesting" events (including fencing), and
>> the script can use the provided environment variables to determine what
>> type of event it was.
> 
> Great.  Does the script run on the DC, or is that configurable somehow?

The script runs on all cluster nodes, to give maximum flexibility and
resiliency (during partitions etc.). Scripts must handle ordering and
de-duplication themselves, if needed.

A script that isn't too concerned about partitions might simply check
whether the local node is the DC, and only take action if so, to avoid
duplicates.

We're definitely interested in hearing how people approach these issues.
The possibilities for what an alert script might do are wide open, and
we want to be as flexible as possible at this stage. If the community
settles on certain approaches or finds certain gaps, we can enhance the
support in those areas as needed.

>> We don't notify before events, because at that moment we don't know
>> whether the event will really happen or not. We might try but fail.
> 
> You lost me here ;-)

We only call alert scripts after an event occurs, because we can't
predict the future. :-) For example, we don't know whether a node is
about to join or leave the cluster. Or for fencing, we might try to
fence but be unsuccessful -- and the part of pacemaker that calls the
alert scripts won't even know about fencing initiated outside cluster
control, such as by DLM or a human running stonith_admin.

>>> I'm wondering if it could replace the current fencing_topology hack we
>>> use to invoke fence_compute which starts the workflow for recovering
>>> VMs off dead OpenStack nova-compute nodes.
>>
>> Yes, that is one of the reasons we did this!
> 
> Haha, at this point can I say great minds think alike? ;-)
> 
>> The initial implementation only allowed for one script to be called (the
>> "notification-agent" property), but we quickly found out that someone
>> might need to email an administrator, notify nova-compute, and do other
>> types of handling as well. Making someone write one script that did
>> everything would be too complicated and error-prone (and unsupportable).
>> So we abandoned "notification-agent" and went with this new approach.
>>
>> Coordinate with Andrew Beekhof for the nova-compute alert script, as he
>> already has some ideas for that.
> 
> OK.  I'm sure we'll be able to talk about this more next week in Austin!
> 
>>> Although even if that's possible, maybe there are good reasons to stay
>>> with the fencing_topology approach?
>>>
>>> Within the same OpenStack compute node HA scenario, it strikes me that
>>> this could be used to invoke "nova service-disable" when the
>>> nova-compute service crashes on a compute node and then fails to
>>> restart.  This would eliminate the window in between the crash and the
>>> nova server timing out the nova-compute service - during which it
>>> would otherwise be possible for nova-scheduler to attempt to schedule
>>> new VMs on the compute node with the crashed nova-compute service.
>>>
>>> IIUC, this is one area where masakari is currently more sophisticated
>>> than the approach based on OCF RAs:
>>>
>>> https://github.com/ntt-sic/masakari/blob/master/docs/evacuation_patterns.md#evacuation-patterns
>>>
>>> Does that make sense?
>>
>> Maybe. The script would need to be able to determine based on the
>> provided environment variables whether it's in that situation or not.
> 
> Yep.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Moving Related Servers

[Ken Gaillot]

On 04/18/2016 10:05 PM, ‪H Yavari‬ ‪ wrote:
> Hi,
> 
> This is servers maps:
> 
> App 3-> App 1(Active)
> 
> App 4 -> App 2   (Standby)
> 
> 
> Now App1 and App2 are in a cluster with IP failover.
> 
> I need when IP failover will run and App2 will be Active node, service
> "X" on server App3 will be stop and App 4 will be Active node.
> In the other words, App1 works only with App3 and App 2 works with App 4.
> 
> I have a web application on App1 and some services on App 3 (this is
> same for App2 and App 4)

This is a difficult situation to model. In particular, you could only
have a dependency one way -- so if we could get App 3 to fail over if
App 1 fails, we couldn't model the other direction (App 1 failing over
if App 3 fails). If each is dependent on the other, there's no way to
start one first.

Is there a technical reason App 3 can work only with App 1?

Is it possible for service "X" to stay running on both App 3 and App 4
all the time? If so, this becomes easier.

> 
> Sorry for heavy description.
> 
> 
> ----
> *From:* Ken Gaillot <kgail...@redhat.com>
> *To:* users@clusterlabs.org
> **
> On 04/18/2016 02:34 AM, ‪H Yavari‬ ‪ wrote:
> 
>> Hi,
>>
>> I have 4 CentOS servers (App1,App2.App3 and App4). I created a cluster
>> for App1 and App2 with a IP float and it works well.
>> In our infrastructure App1 works only with App3 and App2 only works with
>> App4. I mean we have 2 server sets (App1 and App3) , (App2 and App4).
>> So I want when server app1 is down and app2 will Online node, App3 will
>> offline too and App4 will Online and vice versa, I mean when App3 is
>> down and App4 will Online, App1 will offline too.
>>
>>
>> How can I do with pacemaker ? we have our self service on servers so how
>> can I user Pacemaker for monitoring these services?
>>
>> Thanks for reply.
>>
>> Regards.
>> H.Yavari
> 
> 
> I'm not sure I understand your requirements.
> 
> There's no way to tell one node to leave the cluster when another node
> is down, and it would be a bad idea if you could: the nodes could never
> start up, because each would wait to see the other before starting; and
> in your cluster, two nodes shutting down would make the cluster lose
> quorum, so the other nodes would refuse to run any resources.
> 
> However, it is usually possible to use constraints to enforce any
> desired behavior. So even those the node might not leave the cluster,
> you could make the cluster not place any resources on that node.
> 
> Can you give more information about your resources and what nodes they
> are allowed to run on? What makes App1 and App3 dependent on each other?


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org