Re: [ClusterLabs] PCS security vulnerability
Hello Sathish, The CVEs you mentioned (CVE-2024-25126, CVE-2024-26141, CVE-2024-26146) were filed against the rack rubygem and not PCS itself. Therefore, the PCS upstream project is not directly impacted by these CVEs and doesn't require a change. However, PCS does rely on and uses the rack rubygem at runtime. So, if you're using PCS from the upstream source, it's important to ensure you have up-to-date rubygems installed to avoid using vulnerable versions of rack. The advisory you linked (RHSA-2024:3431) addresses these CVEs in the PCS package for RHEL 8.6. This is because the PCS package shipped with RHEL includes some bundled rubygems, including rack. Upgrading the rack rubygem and rebuilding the PCS package were necessary to resolve the CVEs in that specific scenario. Regards, Ondrej On Tue, 11 Jun 2024 at 15:18, S Sathish S wrote: > > Hi Tomas/Team, > > > > In our application we are using pcs-0.10.16 version and that module has > vulnerability(CVE-2024-25126,CVE-2024-26141,CVE-2024-26146) reported and > fixed on below RHSA Errata. can you check and provided fixed on PCS 0.10.x > latest version on upstream also. > > > > https://access.redhat.com/errata/RHSA-2024:3431 > > > > Thanks and Regards, > S Sathish S ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] cluster okey but errors when tried to move resource - ?
To me, this seems like an issue in `crm_resource` as the error message comes from it. Pcs is actually using `crm_resource --move` when moving resources. In this case, pcs should call `crm_resource --move REDIS-clone --node podnode3 --master`, you can see that if you run pcs with `--debug` option. I guess `crm_resource --move --master` creates a location constraint with `role="Promoted"` and doesn't take into account the currently used schema. However, I'm unable to test this theory as I don't have any testing environment available at the moment. Ondrej On Fri, 9 Jun 2023 at 01:39, Reid Wahl wrote: > > On Thu, Jun 8, 2023 at 2:24 PM lejeczek via Users > wrote: > > > > > > > > > Ouch. > > > > > > Let's see the full output of the move command, with the whole CIB that > > > failed to validate. > > > > > For a while there I thought perhaps it was just that one > > pglsq resource, but it seems that any - though only a few > > are set up - (only clone promoted?)resource fails to move. > > Perhaps primarily to do with 'pcs' > > > > -> $ pcs resource move REDIS-clone --promoted podnode3 > > Error: cannot move resource 'REDIS-clone' > > 1 > validate-with="pacemaker-3.6" epoch="8212" num_updates="0" > > admin_epoch="0" cib-last-written="Thu Jun 8 21:59:53 2023" > > update-origin="podnode1" update-client="crm_attribute" > > have-quorum="1" update-user="root" dc-uuid="1"> > > This is the problem: `validate-with="pacemaker-3.6"`. That old schema > doesn't support role="Promoted" in a location constraint. Support > begins with version 3.7 of the schema: > https://github.com/ClusterLabs/pacemaker/commit/e7f1424df49ac41b2d38b72af5ff9ad5121432d2. > > You'll need at least Pacemaker 2.1.0. > > > 2 > > 3 > > 4 > > 5 > id="cib-bootstrap-options-have-watchdog" > > name="have-watchdog" value="false"/> > > 6 > name="dc-version" value="2.1.6-2.el9-6fdc9deea29"/> > > 7 > id="cib-bootstrap-options-cluster-infrastructure" > > name="cluster-infrastructure" value="corosync"/> > > > > crm_resource: Error performing operation: Invalid configuration > > > > ___ > > Manage your subscription: > > https://lists.clusterlabs.org/mailman/listinfo/users > > > > ClusterLabs home: https://www.clusterlabs.org/ > > > > -- > Regards, > > Reid Wahl (He/Him) > Senior Software Engineer, Red Hat > RHEL High Availability - Pacemaker > > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] VirtualDomain - unable to migrate
On Tue, 4 Jan 2022 at 16:20, Ken Gaillot wrote: > > On Wed, 2021-12-29 at 17:28 +, lejeczek via Users wrote: > > Hi guys > > > > I'm having problems with cluster on new CentOS Stream 9 and > > I'd be glad if you can share your thoughts. > > > > -> $ pcs resource move c8kubermaster2 swir > > Location constraint to move resource 'c8kubermaster2' has > > been created > > Waiting for the cluster to apply configuration changes... > > Location constraint created to move resource > > 'c8kubermaster2' has been removed > > Waiting for the cluster to apply configuration changes... > > Error: resource 'c8kubermaster2' is running on node 'whale' > > Error: Errors have occurred, therefore pcs is unable to continue > > pcs on CS9 moves the resource as normal, then runs a simulation to see > what would happen if it removed the move-related constraint. If nothing > would change (e.g. stickiness will keep the resource where it is), then > it goes ahead and removes the constraint. > > I'm not sure what the error messages mean for that new approach. The > pcs devs may be able to chime in. I believe this is caused by a bug in the new implementation of `pcs resource move` command. We have a fix for this almost ready, though, some more testing is still needed. As you already filed a bugzilla [0], let's track it there. And thanks for bringing this up, it helped us to uncover a weird edge case in the new implementation. [0]: https://bugzilla.redhat.com/show_bug.cgi?id=2037218 > > As a workaround, you can use crm_resource --move, which always leaves > the constraint there. Old move command is also still available as `pcs resource move-with-constraint` > > > Not much in pacemaker logs, perhaps nothing at all. > > VM does migrate with 'virsh' just fine. > > > > -> $ pcs resource config c8kubermaster1 > > Resource: c8kubermaster1 (class=ocf provider=heartbeat > > type=VirtualDomain) > >Attributes: > > config=/var/lib/pacemaker/conf.d/c8kubermaster1.xml > > hypervisor=qemu:///system migration_transport=ssh > >Meta Attrs: allow-migrate=true failure-timeout=30s > >Operations: migrate_from interval=0s timeout=90s > > (c8kubermaster1-migrate_from-interval-0s) > >migrate_to interval=0s timeout=90s > > (c8kubermaster1-migrate_to-interval-0s) > >monitor interval=30s > > (c8kubermaster1-monitor-interval-30s) > >start interval=0s timeout=60s > > (c8kubermaster1-start-interval-0s) > >stop interval=0s timeout=60s > > (c8kubermaster1-stop-interval-0s) > > > > Any and all suggestions & thoughts are much appreciated. > > many thanks, L > > -- > Ken Gaillot > > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ > ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Q: rulke-based operation pause/freeze?
On 3/5/20 9:24 PM, Ulrich Windl wrote: Hi! I'm wondering whether it's possible to pause/freeze specific resource operations through rules. The idea is something like this: If your monitor operation needes (e.g.) some external NFS server, and thst NFS server is known to be down, it seems better to delay the monitor operation until NFS is up again, rather than forcing a monitor timeout that will most likely be followed by a stop operation that will also time out, eventually killing the node (which has no problem itself). As I guess it's not possible right now, what would be needed to make this work? In case it's possible, how would an example scenario look like? Regards, Ulrich Hi Ulrich, For 'monitor' operation you can disable it with approach described here at https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/_disabling_a_monitor_operation.html > "followed by a stop operation that will also time out, eventually killing the node (which has no problem itself)" This sounds to me as the resource agent "feature" and I would expect that different resources agents would have different behavior when something is lost/not present. To me the idea here looks like "maintenance period" for some resource. Is your expectation that cluster would not for some time do anything with some resources? (In such case I would consider 'is-managed'=false + disabling monitor) https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-resource-options.html#_resource_meta_attributes To determine _when_ this state should be enabled and disabled would be a different story. -- Ondrej Famera ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Finding attributes of a past resource agent invocation
On 3/3/20 11:22 PM, wf...@niif.hu wrote: Hi, I suffered unexpected fencing under Pacemaker 2.0.1. I set a resource to unmanaged (crm_resource -r vm-invtest -m -p is-managed -v false), then played with ocf-tester, which left the resource stopped. Finally I deleted the resource (crm_resource -r vm-invtest --delete -t primitive), which led to: pacemaker-controld[11670]: notice: State transition S_IDLE -> S_POLICY_ENGINE pacemaker-schedulerd[11669]: notice: Clearing failure of vm-invtest on inv1 because resource parameters have changed pacemaker-schedulerd[11669]: warning: Processing failed monitor of vm-invtest on inv1: not running pacemaker-schedulerd[11669]: warning: Detected active orphan vm-invtest running on inv1 pacemaker-schedulerd[11669]: notice: Clearing failure of vm-invtest on inv1 because it is orphaned pacemaker-schedulerd[11669]: notice: * Stop vm-invtest ( inv1 ) due to node availability pacemaker-schedulerd[11669]: notice: Calculated transition 959, saving inputs in /var/lib/pacemaker/pengine/pe-input-87.bz2 pacemaker-controld[11670]: notice: Initiating stop operation vm-invtest_stop_0 on inv1 pacemaker-controld[11670]: notice: Transition 959 aborted by deletion of lrm_rsc_op[@id='vm-invtest_last_failure_0']: Resource operation removal pacemaker-controld[11670]: warning: Action 6 (vm-invtest_stop_0) on inv1 failed (target: 0 vs. rc: 6): Error pacemaker-controld[11670]: notice: Transition 959 (Complete=5, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-87.bz2): Complete pacemaker-schedulerd[11669]: warning: Processing failed stop of vm-invtest on inv1: not configured pacemaker-schedulerd[11669]: error: Preventing vm-invtest from re-starting anywhere: operation stop failed 'not configured' (6) pacemaker-schedulerd[11669]: warning: Processing failed stop of vm-invtest on inv1: not configured pacemaker-schedulerd[11669]: error: Preventing vm-invtest from re-starting anywhere: operation stop failed 'not configured' (6) pacemaker-schedulerd[11669]: warning: Cluster node inv1 will be fenced: vm-invtest failed there pacemaker-schedulerd[11669]: warning: Detected active orphan vm-invtest running on inv1 pacemaker-schedulerd[11669]: warning: Scheduling Node inv1 for STONITH pacemaker-schedulerd[11669]: notice: Stop of failed resource vm-invtest is implicit after inv1 is fenced pacemaker-schedulerd[11669]: notice: * Fence (reboot) inv1 'vm-invtest failed there' pacemaker-schedulerd[11669]: notice: * Move fencing-inv3 ( inv1 -> inv2 ) pacemaker-schedulerd[11669]: notice: * Stop vm-invtest ( inv1 ) due to node availability The OCF resource agent (on inv1) reported that it failed to validate one of the attributes passed to it for the stop operation, hence the "not configured" error, which caused the fencing. Is there a way to find out what attributes were passed to the OCF agent in that fateful invocation? I've got pe-input files, Pacemaker detail logs and a hard time wading through them. I failed to reproduce the issue till now (but I haven't rewound the CIB yet). Hi Feri, > Is there a way to find out what attributes were passed to the OCF agent in that fateful invocation? Basically same as with any other operation while the resource was configured (with exception of ACTION which was 'stop' in case of stopping resource). As you have the pe-input files which contains the attributes of the resource you can get the attributes and their values from there. == For example if I have tried to delete my test resource with same name, the following can be found in pe-input file ... type="Dummy"> name="target-role" value="Stopped"/> value="some_value"/> name="migrate_from" timeout="20s"/> name="migrate_to" timeout="20s"/> name="monitor" timeout="20s"/> name="reload" timeout="20s"/> name="start" timeout="20s"/> name="stop" timeout="20s"/> ... From above you can see that cluster will be stopping it because of the 'name="target-role" value="Stopped"'. Also you can see that this resource has one attribute (nvpair) with value - name="fake value="some_value"'. Taking inspiration from /usr/lib/ocf/resource.d/pacemaker/Dummy I can see that resource agent will be called like "/usr/lib/ocf/resource.d/pacemaker/Dummy stop" and there will be at minimum $OCF_RESKEY_fake variable passed to it. If you can reproduce the same issue you can try to dump all variables to file when validation fails (take inspiration from function 'dump_env()' of Dummy resource). So if you wanna
Re: [ClusterLabs] Coming in Pacemaker 2.0.4: shutdown locks
Hi Ken, On 2/26/20 7:30 AM, Ken Gaillot wrote: The use case is a large organization with few cluster experts and many junior system administrators who reboot hosts for OS updates during planned maintenance windows, without any knowledge of what the host does. The cluster runs services that have a preferred node and take a very long time to start. In this scenario, pacemaker's default behavior of moving the service to a failover node when the node shuts down, and moving it back when the node comes back up, results in needless downtime compared to just leaving the service down for the few minutes needed for a reboot. 1. Do I understand it correctly that scenario will be when system gracefully reboots (pacemaker service is stopped by system shutting down) and also in case that users for example manually stop cluster but doesn't reboot the node - something like `pcs cluster stop`? If you decide while the node is down that you need the resource to be recovered, you can manually clear a lock with "crm_resource --refresh" specifying both --node and --resource. 2. I'm interested how the situation will look like in the 'crm_mon' output or in 'crm_simulate'. Will there be some indication why the resources are not moving like 'blocked-shutdown-lock' or they will just appear as not moving (Stopped)? Will this look differently from situation where for example the resource is just not allowed by constraint to run on other nodes? Thanks for heads up -- Ondrej Famera ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] 2 node clusters - ipmi fencing
On 2/21/20 10:51 PM, Ricardo Esteves wrote: Hi, I'm trying to understand what is the objective of the constraints to have the fencing devices running on opposite node or on its own node or running all on the same node. Can you explain the difference? Hi Ricardo, Using the constraints you can have a better monitoring of the IPMI device readiness. If the 'monitor' of fence device fails then most probably that device will be not able to fence node when needed. Constraints for fence devices are not required by pacemaker and pacemaker should be able to fence the nodes without them (if fence devices are configured properly). A) Having IPMI device of nodeX monitored from nodeY gives you following check: 'nodeY can communicate with nodeX IPMI device and check its status'. If in the future the nodeY would need to fence nodeX then it should be able to connect to IPMI device. B) Having IPMI device of nodeX monitored from nodeX gives you following check: 'nodeX can communicate with nodeX IPMI device and check its status'. nodeX should not fence itself for reasons mentioned in my previous email and also in Dan's response to this. So this monitoring doesn't provides you with useful information for future fencing. It is not required that nodeX can communicate to nodeX IPMI device for fencing. Most important is that fencing works for both nodes when properly tested. Ideally try if fencing works properly. Optionally if you want cluster to monitor IPMI devices from node that will be using them you can use constraints to move fence devices to opposite node. -- Ondrej Famera ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] 2 node clusters - ipmi fencing
Hello Ricardo, On 2/21/20 9:59 AM, Dan Swartzendruber wrote: I believe you in fact want each fence agent to run on the other node, yes. On February 20, 2020, at 6:23 PM, Ricardo Esteves wrote: Hi, I have a question regarding fencing, i have 2 physical servers: node01, node02, each one has an ipmi card so i create 2 fence devices: fence_ipmi_node01 (with ip of ipmi card of server node01) - with constraint to prefer to run on node01 fence_ipmi_node02 (with ip of ipmi card of server node02) - with constraint to prefer to run on node02 - configured also a 20s delay on this one with 20s delay: make sure to test that this behaves well with your cluster. This value heavily depends on the hardware (IPMI device responsiveness) so proper testing is essential. Is this the best practice? Like this node01 can only fence itself right? and node02 also can only fence itself right? Not exactly: - node2 will use node1 IPMI device to fence node1 - node1 will use node2 IPMI device to fence node2 Both above should happen regardless of constraints. Node should never "fence itself" with IPMI device because fencing is typically following procedure: - 1. check state - 2. turn off - 3. check state - 4. turn on If the node would "fence itself" then there would be no one to continue after step 2. :) All actions during fencing with fence_ipmilan or similar are taking place on one node. (agents that have unfencing like fence_scsi are a different story) Shouldn't i configure fence_ipmi_node01 location constraint to be placed on node02 and fence_ipmi_node02 location constraint to be placed on node01 ? So that node01 can fence node02 and vice versa? IPMI device of node1 prefering node2 and IPMI device of node2 prefering the node1 might be considered a good practice. Important to understand is that you configure where the cluster will run 'monitor' check of IPMI device. That means you will be monitoring readiness of node1 to use node2 IPMI device and readiness of node2 to use node1 IPMI device. -- Ondrej Famera ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] How to unfence without reboot (fence_mpath)
Hello Strahil, On 2/17/20 3:39 PM, Strahil Nikolov wrote: Hello Ondrej, thanks for your reply. I really appreciate that. I have picked fence_multipath as I'm preparing for my EX436 and I can't know what agent will be useful on the exam. Also ,according to https://access.redhat.com/solutions/3201072 , there could be a race condition with fence_scsi. I believe that exam is about testing knowledge in configuration and not testing knowledge in knowing which race condition bugs are present and how to handle them :) If you have access to learning materials for EX436 exam I would recommend trying those ones out - they have labs and comprehensive review exercises that are useful in preparation for exam. So, I've checked the cluster when fencing and the node immediately goes offline. Last messages from pacemaker are: Feb 17 08:21:57 node1.localdomain stonith-ng[23808]: notice: Client stonith_admin.controld.23888.b57ceee7 wants to fence (reboot) 'node1.localdomain' with device '(any)' Feb 17 08:21:57 node1.localdomain stonith-ng[23808]: notice: Requesting peer fencing (reboot) of node1.localdomain Feb 17 08:21:57 node1.localdomain stonith-ng[23808]: notice: FENCING can fence (reboot) node1.localdomain (aka. '1'): static-list Feb 17 08:21:58 node1.localdomain stonith-ng[23808]: notice: Operation reboot of node1.localdomain by node2.localdomain for stonith_admin.controld.23888@node1.localdomain.ede38ffb: OK - This part looks OK - meaning the fencing looks like a success. Feb 17 08:21:58 node1.localdomain crmd[23812]: crit: We were allegedly just fenced by node2.localdomain for node1.localdomai - this is also normal as node just announces that it was fenced by other node Which for me means - node1 just got fenced again. Actually fencing works ,as I/O is immediately blocked and the reservation is removed. I've used https://access.redhat.com/solutions/2766611 to setup the fence_mpath , but I could have messed up something. - note related to exam: you will not have Internet on exam, so I would expect that you would have to configure something that would not require access to this (and as Dan Swartzendruber pointed out in other email - we cannot* even see RH links without account) * you can get free developers account to read them, but ideally that should be not needed and is certainly inconvenient for wide public audience Cluster config is: [root@node3 ~]# pcs config show Cluster Name: HACLUSTER2 Corosync Nodes: node1.localdomain node2.localdomain node3.localdomain Pacemaker Nodes: node1.localdomain node2.localdomain node3.localdomain Resources: Clone: dlm-clone Meta Attrs: interleave=true ordered=true Resource: dlm (class=ocf provider=pacemaker type=controld) Operations: monitor interval=30s on-fail=fence (dlm-monitor-interval-30s) start interval=0s timeout=90 (dlm-start-interval-0s) stop interval=0s timeout=100 (dlm-stop-interval-0s) Clone: clvmd-clone Meta Attrs: interleave=true ordered=true Resource: clvmd (class=ocf provider=heartbeat type=clvm) Operations: monitor interval=30s on-fail=fence (clvmd-monitor-interval-30s) start interval=0s timeout=90s (clvmd-start-interval-0s) stop interval=0s timeout=90s (clvmd-stop-interval-0s) Clone: TESTGFS2-clone Meta Attrs: interleave=true Resource: TESTGFS2 (class=ocf provider=heartbeat type=Filesystem) Attributes: device=/dev/TEST/gfs2 directory=/GFS2 fstype=gfs2 options=noatime run_fsck=no Operations: monitor interval=15s on-fail=fence OCF_CHECK_LEVEL=20 (TESTGFS2-monitor-interval-15s) notify interval=0s timeout=60s (TESTGFS2-notify-interval-0s) start interval=0s timeout=60s (TESTGFS2-start-interval-0s) stop interval=0s timeout=60s (TESTGFS2-stop-interval-0s) Stonith Devices: Resource: FENCING (class=stonith type=fence_mpath) Attributes: devices=/dev/mapper/36001405cb123d000 pcmk_host_argument=key pcmk_host_map=node1.localdomain:1;node2.localdomain:2;node3.localdomain:3 pcmk_monitor_action=metadata pcmk_reboot_action=off Meta Attrs: provides=unfencing Operations: monitor interval=60s (FENCING-monitor-interval-60s) Fencing Levels: Location Constraints: Ordering Constraints: start dlm-clone then start clvmd-clone (kind:Mandatory) (id:order-dlm-clone-clvmd-clone-mandatory) start clvmd-clone then start TESTGFS2-clone (kind:Mandatory) (id:order-clvmd-clone-TESTGFS2-clone-mandatory) Colocation Constraints: clvmd-clone with dlm-clone (score:INFINITY) (id:colocation-clvmd-clone-dlm-clone-INFINITY) TESTGFS2-clone with clvmd-clone (score:INFINITY) (id:colocation-TESTGFS2-clone-clvmd-clone-INFINITY) Ticket Constraints: Alerts: No alerts defined Resources Defaults: No defaults set [root@node3 ~]# crm_mon -r1 Stack: corosync Current DC: node3.localdomain (version 1.1.20-5.el7_7.2-3c4c782f70) - partition with quorum Last
Re: [ClusterLabs] How to unfence without reboot (fence_mpath)
Hello Strahil, On 2/17/20 11:54 AM, Strahil Nikolov wrote: Hello Community, This is my first interaction with pacemaker and SCSI reservations and I was wondering how to unfence a node without rebooting it ? For first encounter with SCSI reservation I would recommend 'fence_scsi' over 'fence_mpath' for the reason that it is easier to configure :) If everything works correctly then simple restart of cluster on fenced node should be enough. Side NOTE: There was discussion previous year about change that introduced ability to choose what happens when node is fenced by storage-based fence agent (like fence_mpath/fence_scsi) that defaults as of now to 'shutdown the cluster'. In newer pacemaker versions is option that can change this to 'shutdown the cluster and panic the node making it to reboot'. I tried to stop & start the cluster stack - it just powers off itself. Adding the reservation before starting the cluster stack - same. It sounds like maybe after start the node was fenced again or at least the fencing was attempted. Are there any errors (/var/log/cluster/corosync.log or similar) in logs about fencing/stonith from around the time when the cluster is started again on node? Only a reboot works. What does the state of cluster looks like on living node when other nodes is fenced? I wonder if the fenced node is reported as Offline or UNCLEAN - you can use the 'crm_mon -1f' to get current cluster state on living node for this including the failures. Thank for answering my question. Best Regards, Strahil Nikolov -- Ondrej Famera ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
[ClusterLabs] When does pacemaker call 'restart'/'force-reload' operations on LSB resource?
Hi ClusterLabs list, When adding 'LSB' script to pacemaker cluster I can see that pacemaker advertises 'restart' and 'force-reload' operations to be present - regardless if the LSB script supports it or not. This seems to be coming from following piece of code. https://github.com/ClusterLabs/pacemaker/blob/92b0c1d69ab1feb0b89e141b5007f8792e69655e/lib/services/services_lsb.c#L39-L40 Questions: 1. When the 'restart' and 'force-reload' operations are called on the LSB script cluster resource? 2. How can I trigger 'restart' and 'force-reload' operation on LSB script cluster resource in pacemaker? Cluster resource definition looks like this: name="force-reload" timeout="15s"/> name="monitor" timeout="15"/> name="restart" timeout="15"/> timeout="15"/> timeout="15"/> I would have expected that 'restart' operation would be called when using 'crm_resource --restart --resource myResource', but I can see that 'stop' and 'start' operations are used in that case instead. For 'force-reload' I have no idea on how to try trigger it looking at 'crm_resource --help' output. I want to make sure that cluster will not attempt running 'restart' nor 'force-reload' on script that is not implementing them. As for now I'm considering to return exit code '3' from script when these actions are called to indicate that they are 'unimplemented feature' as suggested by LSB specification below. However I would like to verify that this works as expected. http://refspecs.linuxfoundation.org/LSB_5.0.0/LSB-Core-generic/LSB-Core-generic/iniscrptact.html -- Ondrej ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Feedback wanted: Node reaction to fabric fencing
On 7/25/19 2:33 AM, Ken Gaillot wrote: Hi all, A recent bugfix (clbz#5386) brings up a question. A node may receive notification of its own fencing when fencing is misconfigured (for example, an APC switch with the wrong plug number) or when fabric fencing is used that doesn't cut the cluster network (for example, fence_scsi). Previously, the *intended* behavior was for the node to attempt to reboot itself in that situation, falling back to stopping pacemaker if that failed. However, due to the bug, the reboot always failed, so the behavior effectively was to stop pacemaker. Now that the bug is fixed, the node will indeed reboot in that situation. It occurred to me that some users configure fabric fencing specifically so that nodes aren't ever intentionally rebooted. Therefore, I intend to make this behavior configurable. My question is, what do you think the default should be? 1. Default to the correct behavior (reboot) 2. Default to the current behavior (stop) 3. Default to the current behavior for now, and change it to the correct behavior whenever pacemaker 2.1 is released (probably a few years from now) As long as there is option to change it I'm OK with change from next minor(?) version (2.0.3) to 'reboot'. But it should be pointed out in RC stage that this is going to occur and to get ready for it. Is there any plan on getting this also into 1.1 branch? If yes, then I would be for just introducing the configuration option in 1.1.x with default to 'stop'. -- Ondrej ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
[ClusterLabs] up-to-date Gentoo ebuilds of pacemaker-2/corosync-3/pcs-0-10
Hi Everyone, I was searching for some up-to-date ebuilds of pacemaker(2.x) and corosync(3.x) on Gentoo. Currently available ones are pacemaker-1.1.19 and corosync-2.4.2 as seen from pages below. https://packages.gentoo.org/packages/sys-cluster/pacemaker https://packages.gentoo.org/packages/sys-cluster/corosync Some time ago (few months ago) I have started in my custom overlay (link below) creating updated ones to fill this gap. https://github.com/OndrejHome/ondrejhome-gentoo-overlay (they should work on both systemd and openrc systems) == Questions: 1. Is anyone aware of new version (pacemaker-2.x, corosync-3.x) ebuilds being maintained somewhere? (links for such projects/efforts welcomed) 2. Is anyone interested in testing out ebuilds I have made and giving a feedback? (via email, github issues, ...) 3. Is anyone using actively pacemaker/corosync/pcs on gentoo? :) Thanks for any info -- Ondrej ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Issue in fence_ilo4 with IPv6 ILO IPs
On 4/5/19 8:18 PM, Rohit Saini wrote: *Further update on this:* This issue is resolved now. ILO was discarding "Neighbor Advertisement" (NA) as Solicited flag was set in NA message. Hence it was not updating its local neighbor table. As per RFC, Solicited flag should be set only in NA message when it is a response to Neighbor Solicitation. After disabling the Solicited flag in NA message, ILO started updating the local neighbor cache. Hi Rohit, Sounds great that after change you get a consistent behaviour. As I had not worked with IPv6 for quite some time I wonder how did you disable the 'Solicited flag'. Was this done on the OS (cluster node) or on the iLO? My guess is the OS but I have no idea how that can be accomplished. Can you share which setting you have changed to accomplish this? :) One additional note the observation here is that you are using the "floating IP" that relocated to other machine, while the configuration of cluster seems to be not containing any IPaddr2 resources that would be representing this address. I would guess that cluster without the floating address would not have issue as it would use the addresses assigned to the nodes and therefore the mapping between IP address and MAC address will be not changing even when the fence_ilo4 resource are moving between nodes. If there is intention to use the floating address in this cluster I would suggest checking if there is also no issue when "not using the floating address" or when it is disabled to see how the fence_ilo4 communicates. I think that there might be way in routing tables to set which IPv6 address should communicate with iLO IPv6 address so you get consistent behaviour instead of using the floating IP address. Anyway I'm glad that mystery is resolved. -- Ondrej On Fri, Apr 5, 2019 at 2:23 PM Rohit Saini mailto:rohitsaini111.fo...@gmail.com>> wrote: Hi Ondrej, Finally found some lead on this.. We started tcpdump on my machine to understand the IPMI traffic. Attaching the capture for your reference. fd00:1061:37:9021:: is my floating IP and fd00:1061:37:9002:: is my ILO IP. When resource movement happens, we are initiating the "Neighbor Advertisement" for fd00:1061:37:9021:: (which is on new machine now) so that peers can update their neighbor table and starts communication with new MAC address. Looks like ILO is not updating its neighbor table, as it is still sending responding to older MAC. After sometime, "Neighbor Solicitation" happens and ILO updates the neighbor table. Now this ILO becomes reachable and starts responding towards new MAC address. My ILO firmware is 2.60. We will try again the issue post upgrading my firmware. To verify this theory, after resource movement, I flushed the local neighbor table due to which "Neighbor Solicitation" was initiated early and this delay in getting ILO response was not seen. This fixed the issue. We are now more interested in understanding why ILO couldnot update its neighbor table on receiving "Neighbor Advertisement". FYI, Override flag in "Neighbor Advertisement" is already set. Thanks, Rohit On Thu, Apr 4, 2019 at 8:37 AM Ondrej mailto:ondrej-clusterl...@famera.cz>> wrote: On 4/3/19 6:10 PM, Rohit Saini wrote: > Hi Ondrej, > Please find my reply below: > > 1. > *Stonith configuration:* > [root@orana ~]# pcs config > Resource: fence-uc-orana (class=stonith type=fence_ilo4) > Attributes: delay=0 ipaddr=fd00:1061:37:9002:: lanplus=1 login=xyz > passwd=xyz pcmk_host_list=orana pcmk_reboot_action=off > Meta Attrs: failure-timeout=3s > Operations: monitor interval=5s on-fail=ignore > (fence-uc-orana-monitor-interval-5s) > start interval=0s on-fail=restart > (fence-uc-orana-start-interval-0s) > Resource: fence-uc-tigana (class=stonith type=fence_ilo4) > Attributes: delay=10 ipaddr=fd00:1061:37:9001:: lanplus=1 login=xyz > passwd=xyz pcmk_host_list=tigana pcmk_reboot_action=off > Meta Attrs: failure-timeout=3s > Operations: monitor interval=5s on-fail=ignore > (fence-uc-tigana-monitor-interval-5s) > start interval=0s on-fail=restart > (fence-uc-tigana-start-interval-0s) > > Fencing Levels: > > Location Constraints: > Ordering Constraints: > start fence-uc-orana then promote unicloud-master (kind:Mandatory) > start fence-uc-tigana then promote unicloud-master (kind:Mandatory) > Colocation C
Re: [ClusterLabs] Issue in fence_ilo4 with IPv6 ILO IPs
On 4/3/19 6:10 PM, Rohit Saini wrote: Hi Ondrej, Please find my reply below: 1. *Stonith configuration:* [root@orana ~]# pcs config Resource: fence-uc-orana (class=stonith type=fence_ilo4) Attributes: delay=0 ipaddr=fd00:1061:37:9002:: lanplus=1 login=xyz passwd=xyz pcmk_host_list=orana pcmk_reboot_action=off Meta Attrs: failure-timeout=3s Operations: monitor interval=5s on-fail=ignore (fence-uc-orana-monitor-interval-5s) start interval=0s on-fail=restart (fence-uc-orana-start-interval-0s) Resource: fence-uc-tigana (class=stonith type=fence_ilo4) Attributes: delay=10 ipaddr=fd00:1061:37:9001:: lanplus=1 login=xyz passwd=xyz pcmk_host_list=tigana pcmk_reboot_action=off Meta Attrs: failure-timeout=3s Operations: monitor interval=5s on-fail=ignore (fence-uc-tigana-monitor-interval-5s) start interval=0s on-fail=restart (fence-uc-tigana-start-interval-0s) Fencing Levels: Location Constraints: Ordering Constraints: start fence-uc-orana then promote unicloud-master (kind:Mandatory) start fence-uc-tigana then promote unicloud-master (kind:Mandatory) Colocation Constraints: fence-uc-orana with unicloud-master (score:INFINITY) (rsc-role:Started) (with-rsc-role:Master) fence-uc-tigana with unicloud-master (score:INFINITY) (rsc-role:Started) (with-rsc-role:Master) 2. This is seen randomly. Since I am using colocation, stonith resources are stopped and started on new master. That time, starting of stonith is taking variable amount of time. No other IPv6 issues are seen in the cluster nodes. 3. fence_agent version [root@orana ~]# rpm -qa|grep fence-agents-ipmilan fence-agents-ipmilan-4.0.11-66.el7.x86_64 *NOTE:* Both IPv4 and IPv6 are configured on my ILO, with "iLO Client Applications use IPv6 first" turned on. Attaching corosync logs also. Thanks, increasing timeout to 60 worked. But thats not what exactly I am looking for. I need to know exact reason behind delay of starting these IPv6 stonith resources. Regards, Rohit Hi Rohit, Thank you for response. From configuration it is clear that we are using directly IP addresses so the DNS resolution issue can be rules out. There are no messages from fence_ilo4 that would indicate reason why it timed out. So we cannot tell yet what caused the issue. I see that you have enabled PCMK_debug=stonith-ng most probably (or PCMK_debug=yes), It is nice that increased the timeout worked, but as said in previous email it may just mask the real reason why it takes longer to do monitor/start operation. > Both IPv4 and IPv6 are configured on my ILO, with "iLO Client > Applications use IPv6 first" turned on. This seems to me to be more related to SNMP communication which we don't use with fence_ilo4 as far as I know. We use the ipmitool on port 623/udp. https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00026111en_us=en_US#N104B2 > 2. This is seen randomly. Since I am using colocation, stonith resources > are stopped and started on new master. That time, starting of stonith is > taking variable amount of time. This is a good observation. Which leads me to question if the iLO has set any kind of session limits for the user that is used here. If there is any session limit it may be worth trying to increase it and test if the same delay can be observed. One situation when this can happen is that when one node communicates with iLO and during that time the communication from other node needs to happen while the limit is 1 connection. The relocation of resource from one note to another might fit this, but this is just speculation and fastest way to prove/reject it would be to increase limit, if there is one, and test it. # What more can be done to figure out on what is causing delay? 1. The fence_ilo4 can be configured with attribute 'verbose=1' to print additional information when it is run. These data looks similar to ones below and they seems to provide the timestamps which is great as we should be able to see when what command was run. I don't have a testing machine on which to run fence_ilo4 so the below example just shows how it looks when it fails on timeout connecting. Apr 03 12:34:11 [4025] fastvm-centos-7-6-31 stonith-ng: notice: stonith_action_async_done: Child process 4252 performing action 'monitor' timed out with signal 15 Apr 03 12:34:11 [4025] fastvm-centos-7-6-31 stonith-ng: warning: log_action: fence_ilo4[4252] stderr: [ 2019-04-03 12:33:51,193 INFO: Executing: /usr/bin/ipmitool -I lanplus -H fe80::f6bd:8a67:7eb5:214f -p 623 -U xyz -P [set] -L ADMINISTRATOR chassis power status ] Apr 03 12:34:11 [4025] fastvm-centos-7-6-31 stonith-ng: warning: log_action: fence_ilo4[4252] stderr: [ ] # pcs stonith update fence-uc-orana verbose=1 Note: That above shows that some private data are included in logs, so in case that you have there something interesting for sharing make sure to strip out the sensit
Re: [ClusterLabs] Issues with DB2 HADR Resource Agent
On 02/19/2018 11:25 PM, Dileep V Nair wrote: > Hello Ondrej, > > I am still having issues with my DB2 HADR on Pacemaker. When I do a > db2_kill on Primary for testing, initially it does a restart of DB2 on > the same node. But if I let it run for some days and then try the same > test, it goes into fencing and then reboots the Primary Node. > > I am not sure how exactly it should behave in case my DB2 crashes on > Primary. > > Also if I crash the Node 1 (the node itself, not only DB2), it promotes > Node 2 to Primary, but once the Pacemaker is started again on Node 1, > the DB on Node 1 is also promoted to Primary. Is that expected behaviour ? > Regards, > > *Dileep V Nair* > Senior AIX Administrator > Cloud Managed Services Delivery (MSD), India > IBM Cloud > > > *E-mail:*_dilen...@in.ibm.com_ <mailto:dilen...@in.ibm.com> > Outer Ring Road, Embassy Manya > Bangalore, KA 560045 > India Hello Dileep, Sorry for later reply. (my email filters sometimes misbehaves) Seeing a fencing after db2_kill is interesting but questions is what has triggered the fencing. Was is failure of DB2 to stop or some other resource failure? When DB2 was successfully promoted on one node while previous has crashed, the one that was crashed should detect that it is 'outdated Primary' in DB2. When this happens the cluster will not attempt to promote it to Master and will leave it as slave. Investigation on DB2 side might be needed to determine if this didn't happen. In case that you have some procedure that results in this behavior constantly I can check on my testing machine to see if I can reproduce it - this may give a hint if it is more cluster issue or DB2 issue that needs to be addressed. -- Ondrej Faměra @Red Hat ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Issues with DB2 HADR Resource Agent
On 02/01/2018 07:24 PM, Dileep V Nair wrote: > Thanks Ondrej for the response. I have set the PEER_WINDOWto 1000 which > I guess is a reasonable value. What I am noticing is it does not wait > for the PEER_WINDOW. Before that itself the DB goes into a > REMOTE_CATCHUP_PENDING state and Pacemaker give an Error saying a DB in > STANDBY/REMOTE_CATCHUP_PENDING/DISCONNECTED can never be promoted. > > > Regards, > > *Dileep V Nair* Hi Dileep, sorry for later response. The DB2 should not get into the 'REMOTE_CATCHUP' phase or the DB2 resource agent will indeed not promote. From my experience it usually gets into that state when the DB2 on standby was restarted during or after PEER_WINDOW timeout. When the primary DB2 fails then standby should end up in some state that would match the one on line 770 of DB2 resource agent and the promote operation is attempted. 770 STANDBY/*PEER/DISCONNECTED|Standby/DisconnectedPeer) https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/db2#L770 The DB2 on standby can get restarted when the 'promote' operation times out, so you can try increasing the 'promote' timeout to something higher if this was the case. So if you see that DB2 was restarted after Primary failed, increase the promote timeout. If DB2 was not restarted then question is why DB2 has decided to change the status in this way. Let me know if above helped. -- Ondrej Faměra @Red Hat ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Issues with DB2 HADR Resource Agent
On 02/01/2018 05:57 PM, Dileep V Nair wrote: > Now the second issue I am facing is that when I crash the node were DB > is primary, the STANDBY DB is not getting promoted to PRIMARY. I could > fix that by adding below lines in db2_promote() > > 773 *) > 774 # must take over forced > 775 force="by force" > 776 > 777 ;; > > But I am not sure of the implications that this can cause. > > Can someone suggest whether what I am doing is correct OR will this lead > to any Data loss. Hi Dileep, As for the 'by force' implications you may check the documentation on what it brings. In short: the data can get corrupted. https://www.ibm.com/support/knowledgecenter/SSEPGG_11.1.0/com.ibm.db2.luw.admin.cmd.doc/doc/r0011553.html#r0011553__byforce The original 'by force peer window only' is limiting the takeover to period when DB2 is within PEER_WINDOW which gives a bit more safety. (the table in link above also explains how much safer it is) Instead of changing the resource agent I would rather suggest checking the PEER_WINDOW and HADR_TIMEOUT variables in DB2. They determine how long it is possible to do takeover 'by force peer window only'. -- Ondrej Faměra @Red Hat ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] Ansible modules for basic operation with pacemaker cluster with 'pcs'
Hi Everyone, I have developed several ansible modules for interacting with pacemaker cluster using 'pcs' utility. The modules cover enough to create cluster, authorize nodes and add/delete/update resource in it (idempotency included - for example resources are updated only if they differ). pcs-modules-2 https://galaxy.ansible.com/OndrejHome/pcs-modules-2/ Further I have also created ansible role for setting up pacemkaer cluster on CentOS/RHEL 6/7 including the fencing setup for fence_xvm on which I test it mostly and ability to specify your own fencing devices if desired. ha-cluster-pacemaker https://galaxy.ansible.com/OndrejHome/ha-cluster-pacemaker/ The role was showcased on this years DevConf 2017 in for of workshop. Links for materials and recording can be found in blog post below. https://www.famera.cz/blog/computers/devconf-2017.html Enjoy and if you hit any any issues feel free to open issue on Github. -- Ondrej Faměra ___ Users mailing list: Users@clusterlabs.org http://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
