Re: [ClusterLabs] Calling crm executables via effective uid
Have you thought about Hawk ?
Best Regards,
Strahil Nikolov
В петък, 11 декември 2020 г., 23:20:49 Гринуич+2, Alex Zarifoglu
написа:
Hello,
I have question regarding the running crm commands with the effective uid.
I am trying to create a tool to manage pacemaker resources for multiple users.
For security reasons, these users will only be able to create/delete/manage
resources that can impact that specific user only. I cannot achieve this via
ACLs because it is not possible to enforce every user to only create primitives
with certain parameters, rules etc.
Therefore, I created a user called cmadmin which has full write access to the
cib. And created an executable which is owned by this user and has the setuid
and setgid bits set.
-r-sr-s--x 1 cmadmin cmadmin 24248 Dec 11 07:04 cmexc
Within this executable I have the code:
pid_t pid;
char *const parmList[] = {"/sbin/crm_mon", "-1", "-VVV", NULL};
if ((pid = fork()) == -1)
perror("fork error");
else if (pid == 0) {
execv("/sbin/crm_mon", parmList);
printf("execv error");
}
If I run this with a user other than cmadmin, crm_mon fails. I tested with
another executable to make sure effective user id is passed in correctly and it
worked fine.
Checking the trace, we fail here with eacces permission denied:
(crm_ipc_send) trace: Sending cib_ro IPC request 5 of 191 bytes using
12ms timeout(internal_ipc_get_reply) trace: client cib_ro waiting on reply
to msg id 5(crm_ipc_send) trace: Received 179-byte reply 5 to cib_ro IPC 5:
I tested with other pacemaker commands and got similar results. I’ve also tried
adding users to haclient group (not to acls just to the group) with no success.
Is it not possible to change effective uids and call crm executables?
If so why and is there way I can achieve what I need differently?
Thank you,
Alex
Alex Zarifoglu
Software Developer | Db2 pureScale
___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users
ClusterLabs home: https://www.clusterlabs.org/
___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users
ClusterLabs home: https://www.clusterlabs.org/
[ClusterLabs] Question about portblock
Hello All, currently I'm experimenting with NFS Ganesha on CentOS8 , and the automatic setup seems tobe actively using "ocf:heartbeat:portblock", but as far as I know, portblock uses iptables while EL8 uses nftables. Does anyone managed to run "ocf:heartbeat:portblock" on EL 8 ? Best Regards, Strahil Nikolov ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Adding a node to an active cluster
pcsd is another layer ontop of the crmd and thus , you need to use the pcs commands to manage the cluster (otherwise the outer layer will be out of sync with inner layers). pcs cluster auth is designed to be used to add a node to a cluster.If you are worried of the result - put the cluster in global maintenance before that. You got 2 steps : - pcs cluster auth -> allows the pcsd on the new node to communicate with pcsd daemons on the other members of the cluster - pcs cluster node add -> adds the node to the cluster Best Regards, Strahil Nikolov В сряда, 28 октомври 2020 г., 22:04:25 Гринуич+2, Jiaqi Tian1 написа: Hi, Is "pcs cluster auth" command capable of add a host when the cluster already has resources running? Also, is there a crmsh version of this command? I didn't find this for crmsh. Thanks, Jiaqi > - Original message ----- > From: Strahil Nikolov > Sent by: "Users" > To: "users@clusterlabs.org" > Cc: > Subject: [EXTERNAL] Re: [ClusterLabs] Adding a node to an active cluster > Date: Tue, Oct 27, 2020 1:07 PM > > On RHEL, I would use "pcs cluster auth"/"pcs host auth" && "pcs cluster node > add".For cluster nodes auth , you can check : > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/considerations_in_adopting_rhel_8/high-availability-and-clusters_considerations-in-adopting-rhel-8#new_commands_for_authenticating_nodes_in_a_cluster > Best Regards,Strahil NikolovВ вторник, 27 октомври 2020 г., 18:06:06 > Гринуич+2, Jiaqi Tian1 написа:Hi Xin,Thank you. > The crmsh version is 4.1.0.0, OS is RHEL 8.0. I have tried crm cluster init > -y, but it seems it cannot be run when the cluster is already up and running > with resources running on it. is crm cluster join command used for this > situation? Thanks,Jiaqi > - Original message -> From: Xin Liang > > Sent by: "Users" > To: > "users@clusterlabs.org" > Cc:>> Subject: [EXTERNAL] > Re: [ClusterLabs] Adding a node to an active cluster> Date: Tue, Oct 27, 2020 > 3:29 AM> >> Hi Jiaqi,>> >> Which OS version do you use and which crmsh > version do you use?> I highly recommend you update your crmsh to latest > version.>> Besides that, did you mean you already have ceha03 and ceha04 > nodes running the cluster service?>> From ha-cluster-bootstrapceha03.log, I > can't see you have record of init cluster successfully.>> >> Ideally, you > should run:>> 1. on ceha03: crm cluster init -y> 2. on ceha01: > crm cluster join -c ceha03 -y>> > > From: > Users on behalf of Jiaqi Tian1 > > Sent: Tuesday, October 27, 2020 12:15 AM> To: > users@clusterlabs.org > Subject: Re: [ClusterLabs] > Adding a node to an active cluster> >> Hi Xin,>> I have ceha03 and ceha04 in > cluster and trying to add ceha01 to the cluster. Running crm cluster join -c > ceha03 -y on ceha01. Here are logs in ceha03 and ceha01. the log file in > ceha04 is empty.>> >> Thanks,>> Jiaqi>> >>> - Original message ->> > From: Xin Liang >> Sent by: "Users" > >> To: "users@clusterlabs.org" > >> Cc:>> Subject: [EXTERNAL] Re: [ClusterLabs] Adding > a node to an active cluster>> Date: Mon, Oct 26, 2020 8:43 AM>> >> Hi > Jiaqi>>>> >>>> Could you give me your > "/var/log/crmsh/ha-cluster-bootstrap.log" or > "/var/log/ha-cluster-bootstrap.log" on these 3 nodes?>>>> >>>> Thanks>>>> > >> >> From: Users > on behalf of Jiaqi Tian1 > >> Sent: Saturday, October 24, 2020 5:48 AM>> To: > users@clusterlabs.org >> Subject: Re: [ClusterLabs] > Adding a node to an active cluster>> >>>> Hi,>>>> Thank you for your > suggestion. The case I have is, I have host1 and host2 in cluster that has > resources running, then I try to join host3 to the cluster by running "crm > cluster join -c host1 -y". But I get this "Configuring csync2...ERROR: > cluster.join: Can't invoke crm cluster init init csync2_remote on host3" > issue. Are there any other requirements for running this command?>>>> >>>> > Thanks>>>> >>>> Jiaqi Tian>>>> >>>>> - Original message ->>> From: > Xin Liang >>> Sent by: "Users" > >>> To: Cluster Labs - All topics related to > open-source clustering wel
Re: [ClusterLabs] Unable to connect to node , no token available
Are you sure that the node you want to join is not one of the listed here: Online: [ SRVDRSW01 SRVDRSW02 SRVDRSW03 SRVDRSW04 SRVDRSW05 SRVDRSW06 ] The hostname looks pretty much the same. Best Regards, Strahil Nikolov В сряда, 28 октомври 2020 г., 11:18:11 Гринуич+2, Raffaele Pantaleoni написа: Hello everybody, I'm trying to add an existing cluster into the pcsd web gui. When I try to add it, I select a node (SRVDRSW01.seko.com) and right after I am requested to enter the password for the hacluster user but from the pcsd log file I can see this: I, [2020-10-28T09:58:26.213638 #12661] INFO -- : Running: id -Gn hacluster I, [2020-10-28T09:58:26.214091 #12661] INFO -- : CIB USER: hacluster, groups: I, [2020-10-28T09:58:26.222338 #12661] INFO -- : Return Value: 0 I, [2020-10-28T09:58:26.222730 #12661] INFO -- : Running: id -Gn hacluster I, [2020-10-28T09:58:26.222887 #12661] INFO -- : CIB USER: hacluster, groups: I, [2020-10-28T09:58:26.230203 #12661] INFO -- : Return Value: 0 I, [2020-10-28T09:58:26.230424 #12661] INFO -- : Running: /usr/sbin/corosync-cmapctl totem.cluster_name I, [2020-10-28T09:58:26.230587 #12661] INFO -- : CIB USER: hacluster, groups: I, [2020-10-28T09:58:26.251826 #12661] INFO -- : Return Value: 0 I, [2020-10-28T09:58:26.253144 #12661] INFO -- : SRWT Node: SRVDRSW01.seko.com Request: check_auth E, [2020-10-28T09:58:26.253603 #12661] ERROR -- : Unable to connect to node SRVDRSW01.seko.com, no token available I tried it using the node name only (SRVDRSW01) or the IP address. The error message doesn't change. Since this is a test installation inside our LAN there is no firewall configured at all. This is the actual status of the cluster: root@SRVDRSW01:~# pcs status Cluster name: debian Stack: corosync Current DC: SRVDRSW01 (version 1.1.16-94ff4df) - partition with quorum Last updated: Wed Oct 28 09:55:20 2020 Last change: Mon Oct 19 17:07:33 2020 by root via crm_attribute on SRVDRSW01 6 nodes configured 5 resources configured Online: [ SRVDRSW01 SRVDRSW02 SRVDRSW03 SRVDRSW04 SRVDRSW05 SRVDRSW06 ] Full list of resources: ClusterIP (ocf::heartbeat:IPaddr2): Started SRVDRSW01 CouchIP (ocf::heartbeat:IPaddr2): Started SRVDRSW03 NodeJSIP (ocf::heartbeat:IPaddr2): Started SRVDRSW04 FrontEnd (ocf::heartbeat:nginx): Started SRVDRSW01 ITATESTSERVER-DIP (ocf::nodejs:pm2): Started SRVDRSW04 Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/disabled I went through hundreds of tries with no luck. Is it something I am doing wrong or I forgot to do? I tried to google it but no solution was good. I have no ideas left. Thank you in advance. --- Raffaele Pantaleoni / / ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] fence_scsi problem
You need the "pcmk_reboot_action=off" for the fencing device ? Also, keep in mind that the fenced node will loose any reservations and pacemaker will shut down . You can use a workaround if you want to convert it from storage-based fencing to power fencing (there is a script that integrates with a watchdog service to reboot the node that was fenced). Best Regards, Strahil Nikolov В сряда, 28 октомври 2020 г., 14:18:05 Гринуич+2, Patrick Vranckx написа: Hi, I try yo setup an HA cluster for ZFS. I think fence_scsi is not working properly. I can reproduce the problem on two kind of hardware: iSCSI and SAS storage. Here is what I did: - set up a storage server with 3 iscsi targets to be accessed by my 2-nodes cluster - set up two nodes with centos 8.1. iscsi initiators let me see three scsi disks on each node: /dev/disk/by-id/wwn-0x60014053374288c674e4add8679947ca -> ../../sda /dev/disk/by-id/wwn-0x6001405a38892587b3f4a739046b9ff4 -> ../../sdb /dev/disk/by-id/wwn-0x6001405b249db0e617b41b19f3147af5 -> ../../sdc - configure zpool "vol1" on one node - defined resources on the cluster: pcs resource create vol1 ZFS pool="vol1" op start timeout="90" op stop timeout="90" --group=group-vol1 pcs resource create vol1-ip IPaddr2 ip=10.30.0.100 cidr_netmask=16 --group group-vol1 pcs stonith create fence-vol1 fence_scsi pcmk_monitor_action="metadata" pcmk_host_list="node1,node2" devices="/dev/disk/by-id/wwn-0x60014053374288c674e4add8679947ca,/dev/disk/by-id/wwn-0x6001405a38892587b3f4a739046b9ff4,/dev/disk/by-id/wwn-0x6001405b249db0e617b41b19f3147af5" meta provides=unfencing Everything seems ok: [root@node1 /]# pcs status Cluster name: mycluster Cluster Summary: * Stack: corosync * Current DC: node2 (version 2.0.3-5.el8_2.1-4b1f869f0f) - partition with quorum * Last updated: Wed Oct 28 12:55:52 2020 * Last change: Wed Oct 28 12:32:36 2020 by root via crm_resource on node2 * 2 nodes configured * 3 resource instances configured Node List: * Online: [ node1 node2 ] Full List of Resources: * fence-vol1 (stonith:fence_scsi): Started node1 * Resource Group: group-vol1: * vol1-ip (ocf::heartbeat:IPaddr2): Started node2 * vol1 (ocf::heartbeat:ZFS): Started node2 Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled Moving the resource to the other node runs fine and the zpool is only imported on the active node. SCSI reservations for scsi disks are as follows: LIO-ORG disk1 4.0 Peripheral device type: disk PR generation=0x2 Key=0x6c9d All target ports bit clear Relative port address: 0x1 << Reservation holder >> scope: LU_SCOPE, type: Write Exclusive, registrants only Transport Id of initiator: iSCSI world wide unique port id: iqn.2020-10.localhost.store:node1 Key=0x6c9d0001 All target ports bit clear Relative port address: 0x1 not reservation holder Transport Id of initiator: iSCSI world wide unique port id: iqn.2020-10.localhost.store:node2 QUESTION : when I move the resource to the other node, SCSI reservations do not change. I would expect that the Write Exclusive reservation would follow the active node. Is it normal ? If I fence manually the active node for group-vol1: [root@node1 /]# pcs stonith fence node1 Node: node1 fenced the resource is restarted on the other node BUT THE ZFS FILESTYSTEMS ARE MOUNTED AND WRITEABLE ON BOTH NODES even if the scsi reservations seem ok: [root@node2 by-id]# sg_persist -s /dev/disk/by-id/wwn-0x60014053374288c674e4add8679947ca LIO-ORG disk1 4.0 Peripheral device type: disk PR generation=0x3 Key=0x6c9d0001 All target ports bit clear Relative port address: 0x1 << Reservation holder >> scope: LU_SCOPE, type: Write Exclusive, registrants only Transport Id of initiator: iSCSI world wide unique port id: iqn.2020-10.localhost.store:node2 I can reproduce the same thing with two node cluster with SAS attached storage running centos 7.8. In my understanding, if node2 has write exclusive reservation, node1 shouldn't be able to write on those disks. What's the problem ? Thanks for your help, Patrick ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Upgrading/downgrading cluster configuration
Have you tried to backup the config via crmsh/pcs and when you downgrade to restore from it ? Best Regards, Strahil Nikolov В четвъртък, 22 октомври 2020 г., 15:40:43 Гринуич+3, Vitaly Zolotusky написа: Hello, We are trying to upgrade our product from Corosync 2.X to Corosync 3.X. Our procedure includes upgrade where we stopthe cluster, replace rpms and restart the cluster. Upgrade works fine, but we also need to implement rollback in case something goes wrong. When we rollback and reload old RPMs cluster says that there are no active resources. It looks like there is a problem with cluster configuration version. Here is output of the crm_mon: d21-22-left.lab.archivas.com /opt/rhino/sil/bin # crm_mon -A1 Stack: corosync Current DC: NONE Last updated: Thu Oct 22 12:39:37 2020 Last change: Thu Oct 22 12:04:49 2020 by root via crm_attribute on d21-22-left.lab.archivas.com 2 nodes configured 15 resources configured Node d21-22-left.lab.archivas.com: UNCLEAN (offline) Node d21-22-right.lab.archivas.com: UNCLEAN (offline) No active resources Node Attributes: *** What would be the best way to implement downgrade of the configuration? Should we just change crm feature set, or we need to rebuild the whole config? Thanks! _Vitaly Zolotusky ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] drbd9 and Centos 8.3
> I get this information in the cluster information: > * drbd_database_monitor_0 on lisbon 'not installed' (5): call=20, > status='complete', exitreason='DRBD kernel (module) not available?', > last-rc-change='2021-01-08 18:03:10 +01:00', queued=0ms, > exec=96ms > > The drbd-service is not started, shouldn't it be started by the > clustermanager ? Looking on my old thread at https://forums.centos.org/viewtopic.php?t=65539#p275539 , I see that the drbd should be started and initialized first before you can clusterize it. What is the output of 'drbdadm status' on both nodes ? What happens when you stop the cluster resource and start the drbd manually ? I guess it's unnecessary to mention how risky is to run a 2-node cluster and that it's far safer if you have a quorum somewhere there ;) Best Regards, Strahil Nikolov ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] DRBD 2-node M/S doesn't want to promote new master, Centos 8
В 14:10 -0700 на 15.01.2021 (пт), Brent Jensen написа: > > > > > Problem: When performing "pcs node standby" on the > current master, this node demotes fine but the slave doesn't > promote to master. It keeps looping the same error including > "Refusing to be Primary while peer is not outdated" and > "Could > not connect to the CIB." At this point the old master has > already unloaded drbd. The only way to fix it is to start > drbd > on the standby node (e.g. drbdadm r0 up). Logs contained > herein > are from the node trying to be master. > > In order to debug, stop the cluster and verify that drbd is running properly. Promote one of the nodes, then demote and promote another one... > I have done this on DRBD9/Centos7/Pacemaker1 w/o any > problems. > So I don't know were the issue is (crm-fence-peer.9.sh > > > > > > Another odd data point: On the slave if I do a "pcs > node standby" & then unstandby, DRBD is loaded again; > HOWEVER, when I do this on the master (which should then be > slave), DRBD doesn't get loaded. > > > > Stonith/Fencing doesn't seem to make a difference. Not sure > if > auto-promote is required. > > > > Quote from official documentation ( https://www.linbit.com/drbd-user-guide/drbd-guide-9_0-en/#s-pacemaker-crm-drbd-backed-service ): If you are employing the DRBD OCF resource agent, it is recommended that you defer DRBD startup, shutdown, promotion, and demotion exclusively to the OCF resource agent. That means that you should disable the DRBD init script: So remove the autopromote and disable the drbd service at all. Best Regards, Strahil Nikolov ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] CCIB migration from Pacemaker 1.x to 2.x
fence_drac5 , fence_drac (not sure about that) , SBD Best Regards,Strahil Nikolov Sent from Yahoo Mail on Android On Mon, Jan 25, 2021 at 11:23, Sharma, Jaikumar wrote: ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
[ClusterLabs] Resource migration and constraint timeout
Hi All, As you all know migrating a resource is actually manipulating the location constraint for that resource. Is there any plan for an option to control a default timeout which is valid for migrations and will remove the 'cli-ban' and 'cli-preffer' location constraints automatically after the defined timeout ? Best Regards,Strahil Nikolov Sent from Yahoo Mail on Android___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Completely disabled resource failure triggered fencing
Have you tried on-fail=ignore option ? Best Regards, Strahil Nikolov В неделя, 17 януари 2021 г., 20:45:27 Гринуич+2, Digimer написа: Hi all, I'm trying to figure out how to define a resource such that if it fails in any way, it will not cause pacemaker self self-fence. The reasoning being that there are relatively minor ways to fault a single resource (these are VMs, so for example, a bad edit to the XML definition renders it invalid, or the definition is accidentally removed). In a case like this, I fully expect that resource to enter a failed state. Of course, pacemaker won't be able to stop it, migrate it, etc. When this happens currently, it causes the host to self-fence, taking down all other hosted resources (servers). This is less than ideal. Is there a way to tell pacemaker that if it's unable to manage a resource, it flags it as failed and leaves it at that? I've been trying to do this and my config so far is; pcs resource create srv07-el6 ocf:alteeve:server name="srv07-el6" \ meta allow-migrate="true" target-role="stopped" \ op monitor interval="60" start timeout="INFINITY" \ on-fail="block" stop timeout="INFINITY" on-fail="block" \ migrate_to timeout="INFINITY" This is getting cumbersome and still, in testing, I'm finding cases where the node gets fenced when something breaks the resource in a creative way. Thanks for any insight/guidance! -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] [EXT] DRBD 2-node M/S doesn't want to promote new master, Centos 8
Can you check this old thread : https://forums.centos.org/viewtopic.php?t=65539 Theoretically El7 vs EL8 drbd cluster should not be different. Maybe you missed something. Also try with the simplest drbd conf possible and if it starst working - try to add option by option to identify which is causing thetrouble. Best Regards, Strahil Nikolov В събота, 16 януари 2021 г., 17:51:05 Гринуич+2, Brent Jensen написа: Maybe. I haven't focused on any stickiness w/ which node is generally master or not. Going standby on the master node should move the slave to master. I'm just trying to follow what's been tried and true in the past and it's no longer following what I would expect. About the only change is the way pacemaker is setup (pacemaker ver 1 uses a master resource, whereas ver 2 uses a promotable resource. I'm not 100% sure I'm doing it right because I haven't gotten it working. Working DRBD resource (pacemaker 1): Master: ms_drbd0 Meta Attrs: clone-max=2 clone-node-max=1 master-max=1 master-node-max=1 notify=true target-role=Started Resource: drbd0 (class=ocf provider=linbit type=drbd) Attributes: drbd_resource=r0 Operations: demote interval=0s timeout=90 (drbd0-demote-interval-0s) monitor interval=15 role=Master (drbd0-monitor-interval-15) monitor interval=30 role=Slave (drbd0-monitor-interval-30) notify interval=0s timeout=90 (drbd0-notify-interval-0s) promote interval=0s timeout=90 (drbd0-promote-interval-0s) reload interval=0s timeout=30 (drbd0-reload-interval-0s) start interval=0 timeout=240s (drbd0-start-interval-0) stop interval=0 timeout=100s (drbd0-stop-interval-0) This resource (pacemaker 2): Clone: drbd0-clone Meta Attrs: clone-max=2 clone-node-max=1 notify=true promotable=true promoted-max=1 promoted-node-max=1 target-role=Started Resource: drbd0 (class=ocf provider=linbit type=drbd) Attributes: drbd_resource=r0 Operations: demote interval=0s timeout=90 (drbd0-demote-interval-0s) monitor interval=20 role=Slave timeout=20 (drbd0-monitor-interval-20) monitor interval=10 role=Master timeout=20 (drbd0-monitor-interval-10) notify interval=0s timeout=90 (drbd0-notify-interval-0s) promote interval=0s timeout=90 (drbd0-promote-interval-0s) reload interval=0s timeout=30 (drbd0-reload-interval-0s) start interval=0s timeout=240 (drbd0-start-interval-0s) stop interval=0s timeout=100 (drbd0-stop-interval-0s) On 1/15/2021 5:16 PM, Ulrich Windl wrote: > On 1/15/21 10:10 PM, Brent Jensen wrote: >> pacemaker-attrd[7671]: notice: Setting master-drbd0[nfs5]: 1 -> 1000 > > I wonder: Does that mean the stickiness for master is still 1000 on nfs5? > > > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ -- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Antw: [EXT] DRBD ms resource keeps getting demoted
I have just built a test cluster (centOS 8.3) for testing DRBD and it works quite fine.Actually I followed my notes from https://forums.centos.org/viewtopic.php?t=65539 with the exception of point 8 due to the "promotable" stuff. I'm attaching the output of 'pcs cluster cib file' and I hope it helps you fix your issue. Best Regards,Strahil Nikolov В 09:32 -0500 на 19.01.2021 (вт), Stuart Massey написа: > Ulrich,Thank you for that observation. We share that concern. > We have 4 ea 1G nics active, bonded in pairs. One bonded pair serves > the "public" (to the intranet) IPs, and the other bonded pair is > private to the cluster, used for drbd replication. HA will, I hope, > be using the "public" IP, since that is the route to the IP addresses > resolved for the host names; that will certainly be the only route to > the quorum device. I can say that this cluster has run reasonably > well for quite some time with this configuration prior to the > recently developed hardware issues on one of the nodes. > Regards, > Stuart > > On Tue, Jan 19, 2021 at 2:49 AM Ulrich Windl < > ulrich.wi...@rz.uni-regensburg.de> wrote: > > >>> Stuart Massey schrieb am 19.01.2021 um > > 04:46 in > > > > Nachricht > > > > > >: > > > > > So, we have a 2-node cluster with a quorum device. One of the > > nodes (node1) > > > > > is having some trouble, so we have added constraints to prevent > > any > > > > > resources migrating to it, but have not put it in standby, so > > that drbd in > > > > > secondary on that node stays in sync. The problems it is having > > lead to OS > > > > > lockups that eventually resolve themselves - but that causes it > > to be > > > > > temporarily dropped from the cluster by the current master > > (node2). > > > > > Sometimes when node1 rejoins, then node2 will demote the drbd ms > > resource. > > > > > That causes all resources that depend on it to be stopped, > > leading to a > > > > > service outage. They are then restarted on node2, since they > > can't run on > > > > > node1 (due to constraints). > > > > > We are having a hard time understanding why this happens. It > > seems like > > > > > there may be some sort of DC contention happening. Does anyone > > have any > > > > > idea how we might prevent this from happening? > > > > > > > > I think if you are routing high-volume DRBD traffic throuch "the > > same pipe" as the cluster communication, cluster communication may > > fail if the pipe is satiated. > > > > I'm not happy with that, but it seems to be that way. > > > > > > > > Maybe running a combination of iftop and iotop could help you > > understand what's going on... > > > > > > > > Regards, > > > > Ulrich > > > > > > > > > Selected messages (de-identified) from pacemaker.log that > > illustrate > > > > > suspicion re DC confusion are below. The update_dc and > > > > > abort_transition_graph re deletion of lrm seem to always precede > > the > > > > > demotion, and a demotion seems to always follow (when not already > > demoted). > > > > > > > > > > Jan 18 16:52:17 [21938] node02.example.com crmd: info: > > > > > do_dc_takeover:Taking over DC status for this partition > > > > > Jan 18 16:52:17 [21938] node02.example.com crmd: info: > > update_dc: > > > > > Set DC to node02.example.com (3.0.14) > > > > > Jan 18 16:52:17 [21938] node02.example.com crmd: info: > > > > > abort_transition_graph:Transition aborted by deletion of > > > > > lrm[@id='1']: Resource state removal | cib=0.89.327 > > > > > source=abort_unless_down:357 > > > > > path=/cib/status/node_state[@id='1']/lrm[@id='1'] complete=true > > > > > Jan 18 16:52:19 [21937] node02.example.compengine: info: > > > > > master_color: ms_drbd_ourApp: Promoted 0 instances of a possible > > 1 to > > > > > master > > > > > Jan 18 16:52:19 [21937] node02.example.compengine: notice: > > LogAction: > > > > > * Demote drbd_ourApp:1 (Master -> Slave > > > > > node02.example.com ) > > > > > > > > > > > > > > > > > > > > ___ > > > > Manage your subscription: > > > > https://lists.clusterlabs.org/mailman/listinfo/users > > > > > > > > ClusterLabs home: https://www.clusterlabs.org/ > > > > ___Manage your > subscription:https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ drbd_cib_el83.xml Description: XML document ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Antw: Re: Antw: [EXT] DRBD ms resource keeps getting demoted
Srry for the top posting.My iSCSILogicalUnit is blocking failover on "standby" (I think it's a bug in the resource), yet without it -> drbd fails over properly: [root@drbd1 ~]# pcs resource show DRBD Resource: DRBD (class=ocf provider=linbit type=drbd) Attributes: drbd_resource=drbd0 Operations: demote interval=0s timeout=90 (DRBD-demote-interval- 0s) monitor interval=30 role=Slave (DRBD-monitor-interval- 30) monitor interval=15 role=Master (DRBD-monitor- interval-15) notify interval=0s timeout=90 (DRBD-notify- interval-0s) promote interval=0s timeout=90 (DRBD-promote- interval-0s) reload interval=0s timeout=30 (DRBD-reload- interval-0s) start interval=0s timeout=240 (DRBD-start- interval-0s) stop interval=0s timeout=100 (DRBD-stop- interval-0s)[root@drbd1 ~]# pcs resource show DRBD-CLONE Master: DRBD- CLONE Meta Attrs: clone-max=3 clone-node-max=1 master-max=1 master- node-max=1 notify=true Resource: DRBD (class=ocf provider=linbit type=drbd) Attributes: drbd_resource=drbd0 Operations: demote interval=0s timeout=90 (DRBD-demote-interval-0s) monitor interval=30 role=Slave (DRBD-monitor-interval-30) monitor interval=15 role=Master (DRBD-monitor-interval-15) notify interval=0s timeout=90 (DRBD-notify-interval-0s) promote interval=0s timeout=90 (DRBD-promote-interval-0s) reload interval=0s timeout=30 (DRBD-reload-interval-0s) start interval=0s timeout=240 (DRBD-start-interval-0s) stop interval=0s timeout=100 (DRBD-stop-interval-0s) Best Regards,Strahil Nikolov В 23:30 -0500 на 21.01.2021 (чт), Stuart Massey написа: > Hi Ulrich, > Thank you for your response. > It makes sense that this would be happening on the failing, > secondary/slave node, in which case we might expect drbd to be > restarted (entirely, since it is already demoted) on the slave. I > don't see how it would affect the master, unless the failing > secondary is causing some issue with drbd on the primary that causes > the monitor on the master to time out for some reason. This does not > (so far) seem to be the case, as the failing node has now been in > maintenance mode for a couple of days with drbd still running as > secondary, so if drbd failures on the secondary were causing the > monitor on the Master/Primary to timeout, we should still be seeing > that; we are not. The master has yet to demote the drbd resource > since we put the failing node in maintenance. > We will watch for a bit longer. > Thanks again > > > > On Thu, Jan 21, 2021, 2:23 AM Ulrich Windl < > ulrich.wi...@rz.uni-regensburg.de> wrote: > > >>> Stuart Massey schrieb am 20.01.2021 > > um 03:41 > > > > in > > > > Nachricht > > > > > >: > > > > > Strahil, > > > > > That is very kind of you, thanks. > > > > > I see that in your (feature set 3.4.1) cib, drbd is in a > > with some > > > > > meta_attributes and operations having to do with promotion, while > > in our > > > > > (feature set 3.0.14) cib, drbd is in a which does not > > have those > > > > > (maybe since promotion is implicit). > > > > > Our cluster has been working quite well for some time, too. I > > wonder what > > > > > would happen if you could hang the os in one of your nodes? If a > > VM, maybe > > > > > > > > Unless some other fencing mechanism (like watchdog timeout) kicks > > in, thge > > > > monitor operation is the only thing that can detect a problem (from > > the > > > > cluster's view): The monitor operation would timeout. Then the > > cluster would > > > > try to restart the resource (stop, then start). If stop also times > > out the node > > > > will be fenced. > > > > > > > > > the constrained secondary could be starved by setting disk IOPs > > to > > > > > something really low. Of course, you are using different versions > > of just > > > > > about everything, as we're on centos7. > > > > > Regards, > > > > > Stuart > > > > > > > > > > On Tue, Jan 19, 2021 at 6:20 PM Strahil Nikolov < > > hunter86...@yahoo.com> > > > > > wrote: > > > > > > > > > >> I have just built a test cluster (centOS 8.3) for testing DRBD > > and it > > > > >> works quite fine. > > > > >> Actually I followed my notes from > > > > >> https://forums.cent
Re: [ClusterLabs] Stopping all nodes causes servers to migrate
I think that it makes sense, as '--all' should mean 'reach all servers and shutdown there'.Yet, when you run 'pcs cluster stop' - the migration of the resources is the only option. Still, it sounds like a bug. Best Regards,Strahil Nikolov Sent from Yahoo Mail on Android On Sun, Jan 24, 2021 at 22:31, Digimer wrote: Hi all, Right off the bat; I'm using a custom RA so this behaviour might be a bug in my agent. I had a test server (srv01-test) running on node 1 (el8-a01n01), and on node 2 (el8-a01n02) I ran 'pcs cluster stop --all'. It appears like pacemaker asked the VM to migrate to node 2 instead of stopping it. Once the server was on node 2, I couldn't use 'pcs resource disable ' as it returned that that resource was unmanaged, and the cluster shut down was hung. When I directly stopped the VM and then did a 'pcs resource cleanup', the cluster shutdown completed. In my agent, I noted these environment variables had been set; OCF_RESKEY_name = srv01-test OCF_RESKEY_CRM_meta_migrate_source = el8-a01n01 OCF_RESKEY_CRM_meta_migrate_target = el8-a01n02 OCF_RESKEY_CRM_meta_on_node = el8-a01n01 So as best as I can tell, pacemaker really did ask for a migration. Is this the case? If not, what environment variables should have been set in this scenario? Thanks for any insight! -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] CentOS 8 & drbd 9, two drbd devices and colocation
> > [root@susanne ~]# drbdadm status > > drbd1 role:Primary > > disk:UpToDate > > lisbon role:Secondary > > peer-disk:UpToDate > > > > drbd2 role:Primary > > disk:UpToDate > > lisbon connection:Connecting Are you sure you got the firewall open ? This node should be connected. Try to verify that each drbd is up and running and promoting any of the 2 nodes is possible before proceeding with the cluster setup. Best Regards, Strahil Nikolov ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] CentOS 8 & drbd 9, two drbd devices and colocation
So why it is saying 'connecting' ? Best Regards, Strahil Nikolov В понеделник, 18 януари 2021 г., 23:54:02 Гринуич+2, Brent Jensen написа: Yes all works fine outside of the cluster. No firewall running nor any selinux. On 1/18/2021 11:53 AM, Strahil Nikolov wrote: >>> [root@susanne ~]# drbdadm status >>> drbd1 role:Primary >>> disk:UpToDate >>> lisbon role:Secondary >>> peer-disk:UpToDate >>> >>> drbd2 role:Primary >>> disk:UpToDate >>> lisbon connection:Connecting > Are you sure you got the firewall open ? This node should be connected. > > Try to verify that each drbd is up and running and promoting any of the 2 > nodes is possible before proceeding with the cluster setup. > > > Best Regards, > Strahil Nikolov > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ -- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Antw: [EXT] Resource migration and constraint timeout
Thanks Ken. The problem is that many junior admins forget about removing the constraint or using lifetime option (as it uses a not very common ISO format whete PT10M and P10M are easy to mistake and could lead to ...). Do you think that it is worth opening a RFE where admin can set a configuration that any migration lifetime is, by default, '8 hours' ( for example) and afterwards it expires (just like with timeout). Best Regards,Strahil Nikolov Sent from Yahoo Mail on Android On Mon, Jan 25, 2021 at 18:16, Ken Gaillot wrote: On Mon, 2021-01-25 at 13:22 +0100, Ulrich Windl wrote: > > > > Strahil Nikolov schrieb am 25.01.2021 > > > > um 12:28 in > > Nachricht <1768184755.3488991.1611574085...@mail.yahoo.com>: > > Hi All, > > As you all know migrating a resource is actually manipulating the > > location > > constraint for that resource. > > Is there any plan for an option to control a default timeout which > > is valid > > for migrations and will remove the 'cli-ban' and 'cli-preffer' > > location > > constraints automatically after the defined timeout ? It's not a default, but the option has always been there: if you're using crm_resource --move/--ban, just add a --lifetime. Under the hood, it adds a time-based rule to the constraint (which you could do if you're explicitly adding a constraint yourself). Some people are bothered that the constraint itself remains in the configuration after it expires (it just has no effect anymore). A more recent option crm_resource --clear --expired will remove all expired constraints from the configuration. > I think the problem is the implementation: It's implemented like a > constraint, not like an action. > For an action it would be enough to send to the LRMs, while for a > constraint it has to be saved to the CIB. Pacemaker's scheduler is state-based rather than action-based. If all you did was an action, the scheduler would immediately want to revert it, to meet the desired state as expressed by the configuration. (Of course something like stickiness could be used to affect that, but the general principle remains: you have to tell the scheduler what end state you want, and let it decide what actions take you there.) > It would be preferable IMHO if those move contraints were not saved > in the CIB at all (as opposed to real location constraints). The CIB is the only input to the scheduler; everything defining the desired state has to be in there. User modifies CIB -> controller feeds CIB to scheduler -> scheduler gives back a list of actions that need to be done (if any) -> controller coordinates the execution of those actions If the scheduler thinks resource R should be on node N, then the CIB has to be changed to get it to change its mind. Otherwise it will want to move it back. If your goal is "I want R to run on N2", then that is in fact a location preference, which is expressed via a constraint. > But as things are now: Could the cluster recheck take care of those? > > > > > > > Best Regards,Strahil Nikolov > > Sent from Yahoo Mail on Android -- Ken Gaillot ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Correctly stop pacemaker on 2-node cluster with SBD and failed devices?
Maybe you can try: while true ; do echo '0' > /proc/sys/kernel/nmi_watchdog ; sleep 1 ; done and in another shell stop pacemaker and sbd. I guess the only way to easily reproduce is with sbd over iscsi. Best Regards,Strahil Nikolov On Tue, Jun 15, 2021 at 21:30, Andrei Borzenkov wrote: On 15.06.2021 20:48, Strahil Nikolov wrote: > I'm using 'pcs cluster stop' (or it's crm alternative),yet I'm not sure if it > will help in this case. > No it won't. It will still stop pacemaker. > Most probably the safest way is to wait for the storage to be recovered, as > without the pacemaker<->SBD communication , sbd will stop and the watchdog > will be triggered. > What makes you think I am not aware of it? can you suggest the steps to avoid it? ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Correctly stop pacemaker on 2-node cluster with SBD and failed devices?
I'm using 'pcs cluster stop' (or it's crm alternative),yet I'm not sure if it will help in this case. Most probably the safest way is to wait for the storage to be recovered, as without the pacemaker<->SBD communication , sbd will stop and the watchdog will be triggered. Best Regards, Strahil Nikolov В вторник, 15 юни 2021 г., 18:47:06 ч. Гринуич+3, Andrei Borzenkov написа: On Tue, Jun 15, 2021 at 6:43 PM Strahil Nikolov wrote: > > How did you stop pacemaker ? systemctl stop pacemaker surprise :) > Usually I use 'pcs cluster stop' or it's crm alternative. > > Best Regards, > Strahil Nikolov > > On Tue, Jun 15, 2021 at 18:21, Andrei Borzenkov > wrote: > We had the following situation > > 2-node cluster with single device (just single external storage > available). Storage failed. So SBD lost access to the device. Cluster > was still up, both nodes were running. > > We thought that access to storage was restored, but one step was > missing so devices appeared empty. > > At this point I tried to restart the pacemaker. But as soon as I > stopped pacemaker SBD rebooted nodes - which is logical, as quorum was > now lost. > > How to cleanly stop pacemaker in this case and keep nodes up? > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Correctly stop pacemaker on 2-node cluster with SBD and failed devices?
How did you stop pacemaker ?Usually I use 'pcs cluster stop' or it's crm alternative. Best Regards,Strahil Nikolov On Tue, Jun 15, 2021 at 18:21, Andrei Borzenkov wrote: We had the following situation 2-node cluster with single device (just single external storage available). Storage failed. So SBD lost access to the device. Cluster was still up, both nodes were running. We thought that access to storage was restored, but one step was missing so devices appeared empty. At this point I tried to restart the pacemaker. But as soon as I stopped pacemaker SBD rebooted nodes - which is logical, as quorum was now lost. How to cleanly stop pacemaker in this case and keep nodes up? ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] A systemd resource monitor is still in progress: re-scheduling
Did you notice any delay in 'systemctl status openstack-cinder-scheduler' ? As far as I know the cluster will use systemd (or even maybe dbus) to get the info of the service. Also, 10s monitor intercal seems quite aggressive - have you considered increasing that ? Best Regards,Strahil Nikolov On Sun, Jun 13, 2021 at 17:45, Acewind wrote: Dear guys, I'm using pacemaker-1.1.20 to construct an openstack HA system. After I stop the cluster, pcs monitor operation always be in progress for cinder-volume & cinder-scheduler service. But the systemd service is active and openstack is working well. How does pacemaker monitor a normal systemd resource? > pcs resource show r_systemd_openstack-cinder-scheduler Resource: r_systemd_openstack-cinder-scheduler (class=systemd type=openstack-cinder-scheduler) Operations: monitor interval=10s timeout=100s (r_systemd_openstack-cinder-scheduler-monitor-interval-10s) stop interval=0s timeout=100s (r_systemd_openstack-cinder-scheduler-stop-interval-0s) 2021-06-13 20:50:42 pcs cluster stop --all2021-06-13 20:50:56 pcs cluster start --all Jun 13 20:53:16 [4057851] host001 lrmd: info: action_complete: r_systemd_openstack-cinder-scheduler monitor is still in progress: re-scheduling (elapsed=54372ms, remaining=45628ms, start_delay=2000ms) Jun 13 20:53:18 [4057851] host001 lrmd: info: action_complete: r_systemd_openstack-cinder-scheduler monitor is still in progress: re-scheduling (elapsed=56374ms, remaining=43626ms, start_delay=2000ms) Jun 13 20:53:20 [4057851] host001 lrmd: info: action_complete: r_systemd_openstack-cinder-scheduler monitor is still in progress: re-scheduling (elapsed=58375ms, remaining=41625ms, start_delay=2000ms) Jun 13 20:53:22 [4057854] host001 crmd: notice: process_lrm_event: Result of stop operation for r_systemd_openstack-cinder-scheduler on host001: 0 (ok) | call=71 key=r_systemd_openstack-cinder-scheduler_stop_0 confirmed=true cib-update=59 The whole log file is included in attachment. Thanks! ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Updating quorum configuration without restarting cluster
You can reload corosync via 'pcs' and I think that both are supported.The main question is if you did reload corosync on all nodes in the cluster ? Best Regards,Strahil Nikolov On Sat, Jun 19, 2021 at 1:22, Gerry R Sommerville wrote: Dear community, I would like to ask few questions regarding Corosync/Pacemaker quorum configuration. When updating the Corosync's quorum configuration I added last_man_standing, and auto_tie_breaker in corosync.conf on all hosts and refreshed with 'corosync-cfgtool -R'. Note that that man page included with the rpm says that the -R option with "Tell all instances of corosync in this cluster to reload corosync.conf." Next I run 'corosync-quorumtool -s', but it did not show the new quorum flags for auto tiebreaker and last man standing. Once I restarted the corosync cluster, the auto tiebreaker flags and last man standing flags appeared in the corosync-quorumtool output as I expected. So my questions are: 1. Does corosync-quorumtool actually shows the active quorum configuration? If not how can I query the active quorum config? 2. Is it possible to update the quorum configuration without restarting the cluster? Thank you,Gerry SommervilleE-mail: ge...@ca.ibm.com ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Updating quorum configuration without restarting cluster
Also, it's worth mentioning that you can still make changes without downtime. For example you can edit corosync conf and push it to all nodes, then set global maintenance, stop the cluster and then start it again. Best Regards,Strahil Nikolov On Mon, Jun 21, 2021 at 9:37, Jan Friesse wrote: Gerry, > Dear community, > > I would like to ask few questions regarding Corosync/Pacemaker quorum > configuration. > > When updating the Corosync's quorum configuration I added last_man_standing, > and > auto_tie_breaker in corosync.conf on all hosts and refreshed with > 'corosync-cfgtool -R'. > Note that that man page included with the rpm says that the -R option with > "Tell > all instances of corosync in this cluster to reload corosync.conf." > > Next I run 'corosync-quorumtool -s', but it did not show the new quorum flags > for auto tiebreaker and last man standing. > > Once I restarted the corosync cluster, the auto tiebreaker flags and last man > standing flags appeared in the corosync-quorumtool output as I expected. > > So my questions are: > 1. Does corosync-quorumtool actually shows the active quorum configuration? If > not how can I query the active quorum config? Yes, corosync-quorumtool shows quorum configuration which is really used (it's actually only source of truth, cmap is not). > > 2. Is it possible to update the quorum configuration without restarting the > cluster? Partly. Basically only quorum.two_node and quorum.expected_votes are changeable during runtime. Other options like: - quorum.allow_downscale - quorum.wait_for_all - quorum.last_man_standing - quorum.auto_tie_breaker - quorum.auto_tie_breaker_node are not (wait_for_all is a little bit more complicated - when not explicitly set/unset it follows two_node so it is possible, but only in this special case, to change it via changing two_node). Regards, Honza btw. I've already replied to Janghyuk Boo so mostly copying same answer also here. > > Thank you, > Gerry Sommerville > E-mail: ge...@ca.ibm.com <mailto:ge...@ca.ibm.com> > > > > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ > ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Antw: [EXT] Re: Antw: Hanging OCFS2 Filesystem any one else?
Thanks for the update. Could it be something local to your environment ?
Have you checked mounting the OCFS2 on a vanilla system ?
Best Regards,Strahil Nikolov
On Tue, Jun 15, 2021 at 12:01, Ulrich
Windl wrote: Hi Guys!
Just to keep you informed on the issue:
I was informed that I'm not the only one seeing this problem, and there seems
to be some "negative interference" between BtrFS reorganizing its extents
periodically and OCFS2 making reflink snapshots (a local cron job here) in
current SUSE SLES kernels. It seems that happens almost exactly at 0:00 o'
clock.
The only thing that BtrFS and OCFS2 have in common here is that BtrFS provides
the mount point for OCFS2.
Regards,
Ulrich
>>> Ulrich Windl schrieb am 02.06.2021 um 11:00 in Nachricht <60B748A4.E0C :
161 :
60728>:
>>>> Gang He schrieb am 02.06.2021 um 08:34 in Nachricht
>
om>
>
> > Hi Ulrich,
> >
> > The hang problem looks like a fix
(90bd070aae6c4fb5d302f9c4b9c88be60c8197ec
> > ocfs2: fix deadlock between setattr and dio_end_io_write), but it is not
> 100%
> > sure.
> > If possible, could you help to report a bug to SUSE, then we can work on
> > that further.
>
> Hi!
>
> Actually a service request for the issue is open at SUSE. However I don't
> know which L3 engineer is working on it.
> I have some "funny" effects, like these:
> On one node "ls" hangs, but can be interrupted with ^C; on another node "ls"
> also hangs, but cannot be stopped with ^C or ^Z
> (Most processes cannot even be killed with "kill -9")
> "ls" on the directory also hangs, just as an "rm" for a non-existent file
>
> What I really wonder is what triggered the effect, and more importantly how
> to recover from it.
> Initially I had suspected a rather full (95%) flesystem, but that means
> there are still 24GB available.
> The other suspect was concurrent creation of reflink snapshots while the
> file being snapshot did change (e.g. allocate a hole in a sparse file)
>
> Regards,
> Ulrich
>
> >
> > Thanks
> > Gang
> >
> >
> > From: Users on behalf of Ulrich Windl
> >
> > Sent: Tuesday, June 1, 2021 15:14
> > To: users@clusterlabs.org
> > Subject: [ClusterLabs] Antw: Hanging OCFS2 Filesystem any one else?
> >
> >>>> Ulrich Windl schrieb am 31.05.2021 um 12:11 in Nachricht <60B4B65A.A8F
: 161
>
> > :
> > 60728>:
> >> Hi!
> >>
> >> We have an OCFS2 filesystem shared between three cluster nodes (SLES 15
SP2,
> >> Kernel 5.3.18‑24.64‑default). The filesystem is filled up to about 95%,
and
> >> we have an odd effect:
> >> A stat() systemcall to some of the files hangs indefinitely (state "D").
> >> ("ls ‑l" and "rm" also hang, but I suspect those are calling state()
> >> internally, too).
> >> My only suspect is that the effect might be related to the 95% being
used.
> >> The other suspect is that concurrent reflink calls may trigger the
effect.
> >>
> >> Did anyone else experience something similar?
> >
> > Hi!
> >
> > I have some details:
> > It seems there is a reader/writer deadlock trying to allocate additional
> > blocks for a file.
> > The stacktrace looks like this:
> > Jun 01 07:56:31 h16 kernel: rwsem_down_write_slowpath+0x251/0x620
> > Jun 01 07:56:31 h16 kernel: ? __ocfs2_change_file_space+0xb3/0x620
[ocfs2]
> > Jun 01 07:56:31 h16 kernel: __ocfs2_change_file_space+0xb3/0x620 [ocfs2]
> > Jun 01 07:56:31 h16 kernel: ocfs2_fallocate+0x82/0xa0 [ocfs2]
> > Jun 01 07:56:31 h16 kernel: vfs_fallocate+0x13f/0x2a0
> > Jun 01 07:56:31 h16 kernel: ksys_fallocate+0x3c/0x70
> > Jun 01 07:56:31 h16 kernel: __x64_sys_fallocate+0x1a/0x20
> > Jun 01 07:56:31 h16 kernel: do_syscall_64+0x5b/0x1e0
> >
> > That is the only writer (on that host), bit there are multiple readers
like
> > this:
> > Jun 01 07:56:31 h16 kernel: rwsem_down_read_slowpath+0x172/0x300
> > Jun 01 07:56:31 h16 kernel: ? dput+0x2c/0x2f0
> > Jun 01 07:56:31 h16 kernel: ? lookup_slow+0x27/0x50
> > Jun 01 07:56:31 h16 kernel: lookup_slow+0x27/0x50
> > Jun 01 07:56:31 h16 kernel: walk_component+0x1c4/0x300
> > Jun 01 07:56:31 h16 kernel: ? path_init+0x192/0x320
> > Jun 01 07:56:31 h16 kernel: path_lookupat+0x6e/0x210
> > Jun 01 07:56:31 h16 kernel: ? __put_lkb+0x45/0xd0 [dlm]
> > Jun 01 07:56:31 h16 kernel: filename_lookup+0xb6/0x190
> > Jun 01 07:56:31 h16 kernel: ? kmem_cache_alloc+0x3d/0x250
> > Jun 01 0
Re: [ClusterLabs] Antw: [EXT] Moving multi-state resources
If something moves in/out in a non-expected way, always check:pcs constraint location --full | grep cli Best Regards,Strahil Nikolov On Thu, May 13, 2021 at 10:45, Andrei Borzenkov wrote: On Wed, May 12, 2021 at 8:15 PM Alastair Basden wrote: > > > > On 12.05.2021 20:02, Alastair Basden wrote: > >>>> Oddly enough, if I: > >>>> pcs resource move resourcedrbdClone node2 > >>>> it moves it to node 2 okay. > >>>> > >>>> But then if I > >>>> pcs resource clear resourcedrbdClone > >>>> it moves it back to node1. > >>>> > >>>> Which is odd, given its score for role Master is higher on node 2. > >>>> > >>> > >>> How do you know it? > >> Because I have set score=100 for node2, and score=50 for node1. > >> > > > > I told you several time that resource agent sets promotion scores. If > > you want to see actual scores, use crm_simulate -Ls. > > Thanks, yes, but I was talking about MY score. Pacemaker combines scores from multiple sources and it is the final total that is used for deciding resource placement. > I have set a higher score > for the node where I want it, and therefore, I would expect it to move > back there (or at least, stay there), not move away from it! > > The fact that it isn't presumably means that something is wrong in the > configuration that I posted. The question is, what. i.e. how does one > specify a preferred node for the master of a master/slave clone. > > Once I can work this out, it will all work correctly! > > >>> Every transaction is saved in a file and you can replay it with > >>> crm_simulate. Check logs for these file names. > >> > >> Which logs should be checked? > > > > pacemaker logs. > Thanks. So: > /var/log/pacemaker.log, which then suggests /var/log/cluster/corosync.log. > > Thanks, > Alastair. > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Location of High Availability Repositories?
On EL8 I think it was named policycoreutils-python-tools or something similar. Best Regards,Strahil Nikolov On Thu, May 13, 2021 at 2:45, Eric Robinson wrote: ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] DRBD + VDO HowTo?
For DRBD there is enough info, so let's focus on VDO.There is a systemd service that starts all VDOs on the system. You can create the VDO once drbs is open for writes and then you can create your own systemd '.service' file which can be used as a cluster resource. Best Regards,Strahil Nikolov On Fri, May 14, 2021 at 2:33, Eric Robinson wrote: Can anyone point to a document on how to use VDO de-duplication with DRBD? Linbit has a blog page about it, but it was last updated 6 years ago and the embedded links are dead. https://linbit.com/blog/albireo-virtual-data-optimizer-vdo-on-drbd/ -Eric Disclaimer : This email and any files transmitted with it are confidential and intended solely for intended recipients. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management. Warning: Although Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] DRBD + VDO HowTo?
There is no VDO RA according to my knowledge, but you can use systemd service
as a resource.
Yet, the VDO service that comes with thr OS is a generic one and controlls all
VDOs - so you need to create your own vdo service.
Best Regards,Strahil Nikolov
On Fri, May 14, 2021 at 6:55, Eric Robinson wrote:
#yiv6406426161 #yiv6406426161 -- _filtered {} _filtered {}#yiv6406426161
#yiv6406426161 p.yiv6406426161MsoNormal, #yiv6406426161
li.yiv6406426161MsoNormal, #yiv6406426161 div.yiv6406426161MsoNormal
{margin:0in;font-size:11.0pt;font-family:sans-serif;}#yiv6406426161 a:link,
#yiv6406426161 span.yiv6406426161MsoHyperlink
{color:blue;text-decoration:underline;}#yiv6406426161 p.yiv6406426161msonormal,
#yiv6406426161 li.yiv6406426161msonormal, #yiv6406426161
div.yiv6406426161msonormal
{margin-right:0in;margin-left:0in;font-size:11.0pt;font-family:sans-serif;}#yiv6406426161
span.yiv6406426161EmailStyle28
{font-family:sans-serif;color:windowtext;}#yiv6406426161
.yiv6406426161MsoChpDefault {font-family:sans-serif;} _filtered
{}#yiv6406426161 div.yiv6406426161WordSection1 {}#yiv6406426161
I created the VDO volumes fine on the drbd devices, formatted them as xfs
filesystems, created cluster filesystem resources, and the cluster us using
them. But the cluster won’t fail over. Is there a VDO cluster RA out there
somewhere already?
From: Strahil Nikolov
Sent: Thursday, May 13, 2021 10:07 PM
To: Cluster Labs - All topics related to open-source clustering welcomed
; Eric Robinson
Subject: Re: [ClusterLabs] DRBD + VDO HowTo?
For DRBD there is enough info, so let's focus on VDO.
There is a systemd service that starts all VDOs on the system. You can create
the VDO once drbs is open for writes and then you can create your own systemd
'.service' file which can be used as a cluster resource.
Best Regards,
Strahil Nikolov
On Fri, May 14, 2021 at 2:33, Eric Robinson
wrote:
Can anyone point to a document on how to use VDO de-duplication with DRBD?
Linbit has a blog page about it, but it was last updated 6 years ago and the
embedded links are dead.
https://linbit.com/blog/albireo-virtual-data-optimizer-vdo-on-drbd/
-Eric
Disclaimer : This email and any files transmitted with it are confidential and
intended solely for intended recipients. If you are not the named addressee you
should not disseminate, distribute, copy or alter this email. Any views or
opinions presented in this email are solely those of the author and might not
represent those of Physician Select Management. Warning: Although Physician
Select Management has taken reasonable precautions to ensure no viruses are
present in this email, the company cannot accept responsibility for any loss or
damage arising from the use of this email or attachments.
___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users
ClusterLabs home: https://www.clusterlabs.org/
Disclaimer : This email and any files transmitted with it are confidential and
intended solely for intended recipients. If you are not the named addressee you
should not disseminate, distribute, copy or alter this email. Any views or
opinions presented in this email are solely those of the author and might not
represent those of Physician Select Management. Warning: Although Physician
Select Management has taken reasonable precautions to ensure no viruses are
present in this email, the company cannot accept responsibility for any loss or
damage arising from the use of this email or attachments.
___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users
ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] DRBD + VDO HowTo?
And why don't you use your own systemd service ? Best Regards,Strahil Nikolov___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] DRBD + VDO HowTo?
Are you sure that the DRBD is working properly ?
Best Regards,Strahil Nikolov
On Mon, May 17, 2021 at 0:32, Eric Robinson wrote:
#yiv0265739749 #yiv0265739749 -- _filtered {} _filtered {} _filtered
{}#yiv0265739749 #yiv0265739749 p.yiv0265739749MsoNormal, #yiv0265739749
li.yiv0265739749MsoNormal, #yiv0265739749 div.yiv0265739749MsoNormal
{margin:0in;font-size:11.0pt;font-family:sans-serif;}#yiv0265739749 a:link,
#yiv0265739749 span.yiv0265739749MsoHyperlink
{color:blue;text-decoration:underline;}#yiv0265739749 p.yiv0265739749msonormal,
#yiv0265739749 li.yiv0265739749msonormal, #yiv0265739749
div.yiv0265739749msonormal
{margin-right:0in;margin-left:0in;font-size:11.0pt;font-family:sans-serif;}#yiv0265739749
p.yiv0265739749msonormal3, #yiv0265739749 li.yiv0265739749msonormal3,
#yiv0265739749 div.yiv0265739749msonormal3
{margin-right:0in;margin-left:0in;font-size:11.0pt;font-family:sans-serif;}#yiv0265739749
span.yiv0265739749EmailStyle22
{font-family:sans-serif;color:windowtext;}#yiv0265739749
.yiv0265739749MsoChpDefault {font-size:10.0pt;} _filtered {}#yiv0265739749
div.yiv0265739749WordSection1 {}#yiv0265739749
Okay, it turns out I was wrong. I thought I had it working, but I keep running
into problems. Sometimes when I demote a DRBD resource on Node A and promote it
on Node B, and I try to mount the filesystem, the system complains that it
cannot read the superblock. But when I move the DRBD primary back to Node A,
the file system is mountable again. Also, I have problems with filesystems not
mounting because the vdo devices are not present. All kinds of issues.
From: Users On Behalf OfEric Robinson
Sent: Friday, May 14, 2021 3:55 PM
To: Strahil Nikolov ; Cluster Labs - All topics related
to open-source clustering welcomed
Subject: Re: [ClusterLabs] DRBD + VDO HowTo?
Okay, I have it working now. The default systemd service definitions did not
work, so I created my own.
From: Strahil Nikolov
Sent: Friday, May 14, 2021 3:41 AM
To: Eric Robinson ; Cluster Labs - All topics related
to open-source clustering welcomed
Subject: RE: [ClusterLabs] DRBD + VDO HowTo?
There is no VDO RA according to my knowledge, but you can use systemd service
as a resource.
Yet, the VDO service that comes with thr OS is a generic one and controlls all
VDOs - so you need to create your own vdo service.
Best Regards,
Strahil Nikolov
On Fri, May 14, 2021 at 6:55, Eric Robinson
wrote:
I created the VDO volumes fine on the drbd devices, formatted them as xfs
filesystems, created cluster filesystem resources, and the cluster us using
them. But the cluster won’t fail over. Is there a VDO cluster RA out there
somewhere already?
From: Strahil Nikolov
Sent: Thursday, May 13, 2021 10:07 PM
To: Cluster Labs - All topics related to open-source clustering welcomed
; Eric Robinson
Subject: Re: [ClusterLabs] DRBD + VDO HowTo?
For DRBD there is enough info, so let's focus on VDO.
There is a systemd service that starts all VDOs on the system. You can create
the VDO once drbs is open for writes and then you can create your own systemd
'.service' file which can be used as a cluster resource.
Best Regards,
Strahil Nikolov
On Fri, May 14, 2021 at 2:33, Eric Robinson
wrote:
Can anyone point to a document on how to use VDO de-duplication with DRBD?
Linbit has a blog page about it, but it was last updated 6 years ago and the
embedded links are dead.
https://linbit.com/blog/albireo-virtual-data-optimizer-vdo-on-drbd/
-Eric
Disclaimer : This email and any files transmitted with it are confidential and
intended solely for intended recipients. If you are not the named addressee you
should not disseminate, distribute, copy or alter this email. Any views or
opinions presented in this email are solely those of the author and might not
represent those of Physician Select Management. Warning: Although Physician
Select Management has taken reasonable precautions to ensure no viruses are
present in this email, the company cannot accept responsibility for any loss or
damage arising from the use of this email or attachments.
___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users
ClusterLabs home: https://www.clusterlabs.org/
Disclaimer : This email and any files transmitted with it are confidential and
intended solely for intended recipients. If you are not the named addressee you
should not disseminate, distribute, copy or alter this email. Any views or
opinions presented in this email are solely those of the author and might not
represent those of Physician Select Management. Warning: Although Physician
Select Management has taken reasonable precautions to ensure no viruses are
present in this email, the company cannot accept responsibility for any loss or
damage arising from the use of this email
Re: [ClusterLabs] DRBD + VDO HowTo?
Also,pacemaker has a very fine grain control mechanisms when and where to run your resources (and even with which resourses to colocate). Best Regards,Strahil Nikolov On Tue, May 18, 2021 at 12:43, Strahil Nikolov wrote: >That was the first thing I tried. The systemd service does not work because it wants to stop and start all vdo devices, but mine are on different nodes. That's why I mentioned to create your own version of the systemd service. Best Regards,Strahil Nikolov ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] DRBD + VDO HowTo?
>That was the first thing I tried. The systemd service does not work because it >wants to stop and start all vdo devices, but mine are on different nodes. That's why I mentioned to create your own version of the systemd service. Best Regards,Strahil Nikolov___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] OCFS2 fragmentation with snapshots
Are you using KVM ? Maybe you can create a snapshot on VM level and then defragfs.ocfs2 the read-only part of the VM disk file and after the defrag -> merge it back by deleting the snapshot ? Yet, the whole idea seems wrong to me. I would freeze the FS and the application in the VM and then make a snapshot via your Virtualization tech stack. Best Regards,Strahil Nikolov On Tue, May 18, 2021 at 13:52, Ulrich Windl wrote: Hi! I thought using the reflink feature of OCFS2 would be just a nice way to make crash-consistent VM snapshots while they are running. As it is a bit tricky to find out how much data is shared between snapshots, I started to write an utility to examine the blocks allocated to the VM backing files and snapshots. Unfortunately (as it seems) OCFS2 fragments terribly under reflink snapshots. Here is an example of a rather "good" file: It has 85 extents that are rather large (not that the extents are sorted by first block; in reality it's a bit worse): DEBUG(5): update_stats: blk_list[0]: 3551627-3551632 (6, 0x2000) DEBUG(5): update_stats: blk_list[1]: 3553626-3556978 (3353, 0x2000) DEBUG(5): update_stats: blk_list[2]: 16777217-16780688 (3472, 0x2000) DEBUG(5): update_stats: blk_list[3]: 16780689-16792832 (12144, 0x2000) DEBUG(5): update_stats: blk_list[4]: 17301147-17304618 (3472, 0x2000) DEBUG(5): update_stats: blk_list[5]: 17304619-17316762 (12144, 0x2000) ... DEBUG(5): update_stats: blk_list[81]: 31178385-31190528 (12144, 0x2000) DEBUG(5): update_stats: blk_list[82]: 31191553-31195024 (3472, 0x2000) DEBUG(5): update_stats: blk_list[83]: 31195025-31207168 (12144, 0x2000) DEBUG(5): update_stats: blk_list[84]: 31210641-31222385 (11745, 0x2001) filesystem: 655360 blocks of size 16384 655360 (100%) blocks type 0x2000 (shared) And here's a terrible example (33837 extents): DEBUG(4): finalize_blockstats: blk_list[0]: 257778-257841 (64, 0x2000) DEBUG(4): finalize_blockstats: blk_list[1]: 257842-257905 (64, 0x2000) DEBUG(4): finalize_blockstats: blk_list[2]: 263503-263513 (11, 0x2000) DEBUG(4): finalize_blockstats: blk_list[3]: 263558-263558 (1, 0x2000) DEBUG(4): finalize_blockstats: blk_list[4]: 263559-263569 (11, 0x2000) DEBUG(4): finalize_blockstats: blk_list[5]: 263587-263587 (1, 0x2000) DEBUG(4): finalize_blockstats: blk_list[6]: 263597-263610 (14, 0x2000) DEBUG(4): finalize_blockstats: blk_list[7]: 270414-270415 (2, 0x2000) ... DEBUG(4): finalize_blockstats: blk_list[90]: 382214-382406 (193, 0x2000) DEBUG(4): finalize_blockstats: blk_list[91]: 382791-382918 (128, 0x2000) DEBUG(4): finalize_blockstats: blk_list[92]: 382983-382990 (8, 0x2000) DEBUG(4): finalize_blockstats: blk_list[93]: 383520-383522 (3, 0x2000) DEBUG(4): finalize_blockstats: blk_list[94]: 384672-384692 (21, 0x2000) DEBUG(4): finalize_blockstats: blk_list[95]: 384860-384918 (59, 0x2000) DEBUG(4): finalize_blockstats: blk_list[96]: 385088-385089 (2, 0x2000) DEBUG(4): finalize_blockstats: blk_list[97]: 385090-385091 (2, 0x2000) ... DEBUG(4): finalize_blockstats: blk_list[805]: 2769213-2769213 (1, 0x2000) DEBUG(4): finalize_blockstats: blk_list[806]: 2769214-2769214 (1, 0x2000) DEBUG(4): finalize_blockstats: blk_list[807]: 2769259-2769259 (1, 0x2000) DEBUG(4): finalize_blockstats: blk_list[808]: 2769261-2769261 (1, 0x2000) DEBUG(4): finalize_blockstats: blk_list[809]: 2769314-2769314 (1, 0x2000) DEBUG(4): finalize_blockstats: blk_list[810]: 2772041-2772042 (2, 0x2000) DEBUG(4): finalize_blockstats: blk_list[811]: 2772076-2772076 (1, 0x2000) DEBUG(4): finalize_blockstats: blk_list[812]: 2772078-2772078 (1, 0x2000) DEBUG(4): finalize_blockstats: blk_list[813]: 2772079-2772080 (2, 0x2000) DEBUG(4): finalize_blockstats: blk_list[814]: 2772096-2772096 (1, 0x2000) DEBUG(4): finalize_blockstats: blk_list[815]: 2772099-2772099 (1, 0x2000) ... DEBUG(4): finalize_blockstats: blk_list[33829]: 39317682-39317704 (23, 0x2000) DEBUG(4): finalize_blockstats: blk_list[33830]: 39317770-39317775 (6, 0x2000) DEBUG(4): finalize_blockstats: blk_list[33831]: 39318022-39318045 (24, 0x2000) DEBUG(4): finalize_blockstats: blk_list[33832]: 39318274-39318284 (11, 0x2000) DEBUG(4): finalize_blockstats: blk_list[33833]: 39318327-39318344 (18, 0x2000) DEBUG(4): finalize_blockstats: blk_list[33834]: 39319157-39319166 (10, 0x2000) DEBUG(4): finalize_blockstats: blk_list[33835]: 39319172-39319184 (13, 0x2000) DEBUG(4): finalize_blockstats: blk_list[33836]: 39319896-39319936 (41, 0x2000) filesystem: 1966076 blocks of size 16384 mapped=1121733 (57%) 1007658 (51%) blocks type 0x2000 (shared) 114075 (6%) blocks type 0x2800 (unwritten|shared) So I wonder (while understanding the principle of copy-on-write for reflink snapshots): Is there a way to avoid or undo the fragmentation? Regards, Ulrich ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your
Re: [ClusterLabs] DRBD + VDO HowTo?
Have you tried to set VDO in async mode ?
Best Regards,Strahil Nikolov
On Mon, May 17, 2021 at 8:57, Klaus Wenninger wrote:
Did you try VDO in sync-mode for the case the flush-fua stuff isn't working
through the layers? Did you check that VDO-service is disabled and solely under
pacemaker-control and that the dependencies are set correctly?
Klaus
On 5/17/21 6:17 AM, Eric Robinson wrote:
_filtered {} _filtered {} _filtered {}#yiv7380066700 p.yiv7380066700MsoNormal,
#yiv7380066700 li.yiv7380066700MsoNormal, #yiv7380066700
div.yiv7380066700MsoNormal
{margin:0in;font-size:11.0pt;font-family:sans-serif;}#yiv7380066700 a:link,
#yiv7380066700 span.yiv7380066700MsoHyperlink
{color:blue;text-decoration:underline;}#yiv7380066700 p.yiv7380066700msonormal,
#yiv7380066700 li.yiv7380066700msonormal, #yiv7380066700
div.yiv7380066700msonormal
{margin-right:0in;margin-left:0in;font-size:11.0pt;font-family:sans-serif;}#yiv7380066700
p.yiv7380066700msonormal4, #yiv7380066700 li.yiv7380066700msonormal4,
#yiv7380066700 div.yiv7380066700msonormal4
{margin-right:0in;margin-left:0in;font-size:11.0pt;font-family:sans-serif;}#yiv7380066700
p.yiv7380066700msonormal31, #yiv7380066700 li.yiv7380066700msonormal31,
#yiv7380066700 div.yiv7380066700msonormal31
{margin-right:0in;margin-left:0in;font-size:11.0pt;font-family:sans-serif;}#yiv7380066700
span.yiv7380066700EmailStyle32
{font-family:sans-serif;color:windowtext;}#yiv7380066700
.yiv7380066700MsoChpDefault {font-family:sans-serif;}#yiv7380066700
div.yiv7380066700WordSection1 {}
Yes, DRBD is working fine.
From: Strahil Nikolov
Sent: Sunday, May 16, 2021 6:06 PM
To: Eric Robinson ; Cluster Labs - All topics related
to open-source clustering welcomed
Subject: RE: [ClusterLabs] DRBD + VDO HowTo?
Are you sure that the DRBD is working properly ?
Best Regards,
Strahil Nikolov
On Mon, May 17, 2021 at 0:32, Eric Robinson
wrote:
Okay, it turns out I was wrong. I thought I had it working, but I keep running
into problems. Sometimes when I demote a DRBD resource on Node A and promote it
on Node B, and I try to mount the filesystem, the system complains that it
cannot read the superblock. But when I move the DRBD primary back to Node A,
the file system is mountable again. Also, I have problems with filesystems not
mounting because the vdo devices are not present. All kinds of issues.
From: Users On Behalf Of Eric Robinson
Sent: Friday, May 14, 2021 3:55 PM
To: Strahil Nikolov ; Cluster Labs - All topics related
to open-source clustering welcomed
Subject: Re: [ClusterLabs] DRBD + VDO HowTo?
Okay, I have it working now. The default systemd service definitions did not
work, so I created my own.
From: Strahil Nikolov
Sent: Friday, May 14, 2021 3:41 AM
To: Eric Robinson ; Cluster Labs - All topics related
to open-source clustering welcomed
Subject: RE: [ClusterLabs] DRBD + VDO HowTo?
There is no VDO RA according to my knowledge, but you can use systemd service
as a resource.
Yet, the VDO service that comes with thr OS is a generic one and controlls all
VDOs - so you need to create your own vdo service.
Best Regards,
Strahil Nikolov
On Fri, May 14, 2021 at 6:55, Eric Robinson
wrote:
I created the VDO volumes fine on the drbd devices, formatted them as xfs
filesystems, created cluster filesystem resources, and the cluster us using
them. But the cluster won’t fail over. Is there a VDO cluster RA out there
somewhere already?
From: Strahil Nikolov
Sent: Thursday, May 13, 2021 10:07 PM
To: Cluster Labs - All topics related to open-source clustering welcomed
; Eric Robinson
Subject: Re: [ClusterLabs] DRBD + VDO HowTo?
For DRBD there is enough info, so let's focus on VDO.
There is a systemd service that starts all VDOs on the system. You can create
the VDO once drbs is open for writes and then you can create your own systemd
'.service' file which can be used as a cluster resource.
Best Regards,
Strahil Nikolov
On Fri, May 14, 2021 at 2:33, Eric Robinson
wrote:
Can anyone point to a document on how to use VDO de-duplication with DRBD?
Linbit has a blog page about it, but it was last updated 6 years ago and the
embedded links are dead.
https://linbit.com/blog/albireo-virtual-data-optimizer-vdo-on-drbd/
-Eric
Disclaimer : This email and any files transmitted with it are confidential and
intended solely for intended recipients. If you are not the named addressee you
should not disseminate, distribute, copy or alter this email. Any views or
opinions presented in this email are solely those of the author and might not
represent those of Physician Select Management. Warning: Although Physician
Select Management has taken reasonable precautions to ensure no viruses are
present in this email
Re: [ClusterLabs] Problem with the cluster becoming mostly unresponsive
>So a monitor failure on the fence agent rendered the cluster effectively unresponsive? How would I normally recover from this? Actually it will ban the resource (stonith) from the node when it reaches the maximum fail count. When the stonith is banned from all nodes, no node will be able to use that stonith. You can use 'failure-timeout' meta attribute to reset the fail count. I'm using it for the ipmi fencing mechanisms. Of course the best approach is to make that stonith more reliable but usually this is out of our control. Another approach is to define a second stonith method and use stonith topology. Best Regards,Strahil Nikolov ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] bit of wizardry bit of trickery needed.
Here is the example I had promised:
pcs node attribute server1 city=LApcs node attribute server2 city=NY
# Don't run on any node that is not in LApcs constraint location DummyRes1 rule
score=-INFINITY city ne LA
#Don't run on any node that is not in NYpcs constraint location DummyRes2 rule
score=-INFINITY city ne NY
The idea is that if you add a node and you forget to specify the attribute with
the name 'city' , DummyRes1 & DummyRes2 won't be started on it.
For resources that do not have a constraint based on the city -> they will run
everywhere unless you specify a colocation constraint between the resources.
Best Regards,Strahil Nikolov
On Tue, May 11, 2021 at 9:15, Klaus Wenninger wrote:
On 5/10/21 7:16 PM, lejeczek wrote:
>
>
> On 10/05/2021 17:04, Andrei Borzenkov wrote:
>> On 10.05.2021 16:48, lejeczek wrote:
>>> Hi guys
>>>
>>> Before I begin my adventure with this I though I would ask experts if
>>> something like below is possible.
>>>
>>> resourceA if started on nodeA, then nodes B & C start resourceB (or
>>> recourceC)
>>>
>> Configure colocation with negative score between resourceB and
>> resourceA, so resourceB will be started on different node.
>>
>>> whether to start resource B or C on two nodes (might thing of it as
>>> master node + two slaves) depend on which node resourceA got started.
>>>
>>> eg.
>>> if nodeA runs rMaster -> nodeB, nodeC run rToA
>>> if nodeB runs rMaster -> nodeA, nodeC run rToB
>>> if nodeC runs rMaster -> nodeA, nodeB run rToC
>>>
>>> any light shedding or idea sharing are much welcomed.
>>> many thanks, L.
>>> ___
>>> Manage your subscription:
>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>
>>> ClusterLabs home: https://www.clusterlabs.org/
>> ___
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users
>>
>> ClusterLabs home: https://www.clusterlabs.org/
> perhaps I did not do best explaining, I'll try again
>
> if nodeA runs rMaster -> nodeB, nodeC run rToA (meaning..
> a) no r{ToA,ToB,ToC} allowed on the node if "rMaster" runs on that
> node, this case nodeA,
> b) if it's nodeA cluster chose to run "rMaster" on, then only "rToA"
> is allowed to run on nodeB & nodeC
> c) a applies across to nodeB,C respectively
>
> I'm starting with "rMaster" and other three resources as clones but I
> fail to see how to make it work.
Not sure if there is an easy way to get that working directly.
An anti-colocation as already suggested is probably a good idea.
A resource that sets an attribute to select which clone to start
could do the rest - with location-constraints using that attribute.
Without knowing more about your resources it is hard to
tell if there would be a more elegant way to solve your
problem.
If it is mainly about IP communication of the slaves with the
master (we btw. removed that wording from pacemaker as
considered offensive) you could have a floating IP-address that is
moved with the master (or more precisely rather the other
way round) and your slaves would connect with that IP without
really having to know who the master is.
If the logic is more complex and you anyway already have
a custom resource agent it might be worth thinking of a
promotable clone that runs the master
in it's promoted state and the slave in it's demoted state
with the logic moved into a resource agent.
Klaus
>
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
>
___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users
ClusterLabs home: https://www.clusterlabs.org/
___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users
ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Sub-clusters / super-clusters?
Here is the example I had promised: pcs node attribute server1 city=LApcs node attribute server2 city=NY # Don't run on any node that is not in LApcs constraint location DummyRes1 rule score=-INFINITY city ne LA #Don't run on any node that is not in NYpcs constraint location DummyRes2 rule score=-INFINITY city ne NY The idea is that if you add a node and you forget to specify the attribute with the name 'city' , DummyRes1 & DummyRes2 won't be started on it. For resources that do not have a constraint based on the city -> they will run everywhere unless you specify a colocation constraint between the resources. Best Regards,Strahil Nikolov On Mon, May 10, 2021 at 17:53, Antony Stone wrote: On Monday 10 May 2021 at 16:49:07, Strahil Nikolov wrote: > You can use node attributes to define in which city is each host and then > use a location constraint to control in which city to run/not run the > resources. I will try to provide an example tomorrow. Thank you - that would be helpful. I did think that a location constraint could be a way to do this, but I wasn't sure how to label three machines in one cluster as a "single location". Any pointers most welcome :) > On Mon, May 10, 2021 at 15:52, Antony Stone wrote: > > On Monday 10 May 2021 at 14:41:37, Klaus Wenninger wrote: > > On 5/10/21 2:32 PM, Antony Stone wrote: > > > Hi. > > > > > > I'm using corosync 3.0.1 and pacemaker 2.0.1, currently in the > > > following way: > > > > > > I have two separate clusters of three machines each, one in a data > > > centre in city A, and one in a data centre in city B. > > > > > > Several of the resources being managed by these clusters are based on > > > floating IP addresses, which are tied to the data centre, therefore the > > > resources in city A can run on any of the three machines there (alfa, > > > bravo and charlie), but cannot run on any machine in city B (delta, > > > echo and foxtrot). > > > > > > I now have a need to create a couple of additional resources which can > > > operate from anywhere, so I'm wondering if there is a way to configure > > > corosync / pacemaker so that: > > > > > > Machines alfa, bravo, charlie live in city A and manage resources X, Y > > > and Z between them. > > > > > > Machines delta, echo and foxtrot live in city B and manage resources U, > > > V and W between them. > > > > > > All of alpha to foxtrot are also in a "super-cluster" managing > > > resources P and Q, so these two can be running on any of the 6 > > > machines. > > > > > > > > > I hope the question is clear. Is there an answer :) ? > > > > Sounds like a use-case for https://github.com/ClusterLabs/booth > > Interesting - hadn't come across that feature before. > > Thanks - I'll look into further documentation. > > If anyone else has any other suggestions I'm happy to see whether something > else might work better for my setup. > > > Antony. -- 90% of networking problems are routing problems. 9 of the remaining 10% are routing problems in the other direction. The remaining 1% might be something else, but check the routing anyway. Please reply to the list; please *don't* CC me. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] bit of wizardry bit of trickery needed.
Oh wrong thread, just ignore .
Best Regards
On Tue, May 11, 2021 at 13:54, Strahil Nikolov wrote:
Here is the example I had promised:
pcs node attribute server1 city=LApcs node attribute server2 city=NY
# Don't run on any node that is not in LApcs constraint location DummyRes1 rule
score=-INFINITY city ne LA
#Don't run on any node that is not in NYpcs constraint location DummyRes2 rule
score=-INFINITY city ne NY
The idea is that if you add a node and you forget to specify the attribute with
the name 'city' , DummyRes1 & DummyRes2 won't be started on it.
For resources that do not have a constraint based on the city -> they will run
everywhere unless you specify a colocation constraint between the resources.
Best Regards,Strahil Nikolov
On Tue, May 11, 2021 at 9:15, Klaus Wenninger wrote:
On 5/10/21 7:16 PM, lejeczek wrote:
>
>
> On 10/05/2021 17:04, Andrei Borzenkov wrote:
>> On 10.05.2021 16:48, lejeczek wrote:
>>> Hi guys
>>>
>>> Before I begin my adventure with this I though I would ask experts if
>>> something like below is possible.
>>>
>>> resourceA if started on nodeA, then nodes B & C start resourceB (or
>>> recourceC)
>>>
>> Configure colocation with negative score between resourceB and
>> resourceA, so resourceB will be started on different node.
>>
>>> whether to start resource B or C on two nodes (might thing of it as
>>> master node + two slaves) depend on which node resourceA got started.
>>>
>>> eg.
>>> if nodeA runs rMaster -> nodeB, nodeC run rToA
>>> if nodeB runs rMaster -> nodeA, nodeC run rToB
>>> if nodeC runs rMaster -> nodeA, nodeB run rToC
>>>
>>> any light shedding or idea sharing are much welcomed.
>>> many thanks, L.
>>> ___
>>> Manage your subscription:
>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>
>>> ClusterLabs home: https://www.clusterlabs.org/
>> ___
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users
>>
>> ClusterLabs home: https://www.clusterlabs.org/
> perhaps I did not do best explaining, I'll try again
>
> if nodeA runs rMaster -> nodeB, nodeC run rToA (meaning..
> a) no r{ToA,ToB,ToC} allowed on the node if "rMaster" runs on that
> node, this case nodeA,
> b) if it's nodeA cluster chose to run "rMaster" on, then only "rToA"
> is allowed to run on nodeB & nodeC
> c) a applies across to nodeB,C respectively
>
> I'm starting with "rMaster" and other three resources as clones but I
> fail to see how to make it work.
Not sure if there is an easy way to get that working directly.
An anti-colocation as already suggested is probably a good idea.
A resource that sets an attribute to select which clone to start
could do the rest - with location-constraints using that attribute.
Without knowing more about your resources it is hard to
tell if there would be a more elegant way to solve your
problem.
If it is mainly about IP communication of the slaves with the
master (we btw. removed that wording from pacemaker as
considered offensive) you could have a floating IP-address that is
moved with the master (or more precisely rather the other
way round) and your slaves would connect with that IP without
really having to know who the master is.
If the logic is more complex and you anyway already have
a custom resource agent it might be worth thinking of a
promotable clone that runs the master
in it's promoted state and the slave in it's demoted state
with the logic moved into a resource agent.
Klaus
>
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
>
___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users
ClusterLabs home: https://www.clusterlabs.org/
___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users
ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] fencing
If you have SAN & Hardware Watchdog device, you can also use SBD.If SAN is lost and nodes cannot communicate - they will suicide. Best Regards,Strahil Nikolov___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Sub-clusters / super-clusters?
You can use node attributes to define in which city is each host and then use a location constraint to control in which city to run/not run the resources. I will try to provide an example tomorrow. Best Regards,Strahil Nikolov On Mon, May 10, 2021 at 15:52, Antony Stone wrote: On Monday 10 May 2021 at 14:41:37, Klaus Wenninger wrote: > On 5/10/21 2:32 PM, Antony Stone wrote: > > Hi. > > > > I'm using corosync 3.0.1 and pacemaker 2.0.1, currently in the following > > way: > > > > I have two separate clusters of three machines each, one in a data centre > > in city A, and one in a data centre in city B. > > > > Several of the resources being managed by these clusters are based on > > floating IP addresses, which are tied to the data centre, therefore the > > resources in city A can run on any of the three machines there (alfa, > > bravo and charlie), but cannot run on any machine in city B (delta, echo > > and foxtrot). > > > > I now have a need to create a couple of additional resources which can > > operate from anywhere, so I'm wondering if there is a way to configure > > corosync / pacemaker so that: > > > > Machines alfa, bravo, charlie live in city A and manage resources X, Y > > and Z between them. > > > > Machines delta, echo and foxtrot live in city B and manage resources U, V > > and W between them. > > > > All of alpha to foxtrot are also in a "super-cluster" managing resources > > P and Q, so these two can be running on any of the 6 machines. > > > > > > I hope the question is clear. Is there an answer :) ? > > Sounds like a use-case for https://github.com/ClusterLabs/booth Interesting - hadn't come across that feature before. Thanks - I'll look into further documentation. If anyone else has any other suggestions I'm happy to see whether something else might work better for my setup. Antony. -- What do you get when you cross a joke with a rhetorical question? Please reply to the list; please *don't* CC me. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Cluster Stopped, No Messages?
what is your fencing agent ? Best Regards,Strahil Nikolov On Thu, May 27, 2021 at 20:52, Eric Robinson wrote: We found one of our cluster nodes down this morning. The server was up but cluster services were not running. Upon examination of the logs, we found that the cluster just stopped around 9:40:31 and then I started it up manually (pcs cluster start) at 11:49:48. I can’t imagine that Pacemaker just randomly terminates. Any thoughts why it would behave this way? May 27 09:25:31 [92170] 001store01a pengine: notice: process_pe_message: Calculated transition 91482, saving inputs in /var/lib/pacemaker/pengine/pe-input-756.bz2 May 27 09:25:31 [92171] 001store01a crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response May 27 09:25:31 [92171] 001store01a crmd: info: do_te_invoke: Processing graph 91482 (ref=pe_calc-dc-1622121931-124396) derived from /var/lib/pacemaker/pengine/pe-input-756.bz2 May 27 09:25:31 [92171] 001store01a crmd: notice: run_graph: Transition 91482 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-756.bz2): Complete May 27 09:25:31 [92171] 001store01a crmd: info: do_log: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd May 27 09:25:31 [92171] 001store01a crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd May 27 09:40:31 [92171] 001store01a crmd: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped (90ms) May 27 09:40:31 [92171] 001store01a crmd: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped May 27 09:40:31 [92171] 001store01a crmd: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED May 27 09:40:31 [92170] 001store01a pengine: info: process_pe_message: Input has not changed since last time, not saving to disk May 27 09:40:31 [92170] 001store01a pengine: info: determine_online_status: Node 001store01a is online May 27 09:40:31 [92170] 001store01a pengine: info: determine_op_status: Operation monitor found resource p_pure-ftpd-itls active on 001store01a May 27 09:40:31 [92170] 001store01a pengine: warning: unpack_rsc_op_failure: Processing failed op monitor for p_vip_ftpclust01 on 001store01a: unknown error (1) May 27 09:40:31 [92170] 001store01a pengine: info: determine_op_status: Operation monitor found resource p_pure-ftpd-etls active on 001store01a May 27 09:40:31 [92170] 001store01a pengine: info: unpack_node_loop: Node 1 is already processed May 27 09:40:31 [92170] 001store01a pengine: info: unpack_node_loop: Node 1 is already processed May 27 09:40:31 [92170] 001store01a pengine: info: common_print: p_vip_ftpclust01 (ocf::heartbeat:IPaddr2): Started 001store01a May 27 09:40:31 [92170] 001store01a pengine: info: common_print: p_replicator (systemd:pure-replicator): Started 001store01a May 27 09:40:31 [92170] 001store01a pengine: info: common_print: p_pure-ftpd-etls (systemd:pure-ftpd-etls): Started 001store01a May 27 09:40:31 [92170] 001store01a pengine: info: common_print: p_pure-ftpd-itls (systemd:pure-ftpd-itls): Started 001store01a May 27 09:40:31 [92170] 001store01a pengine: info: LogActions: Leave p_vip_ftpclust01 (Started 001store01a) May 27 09:40:31 [92170] 001store01a pengine: info: LogActions: Leave p_replicator (Started 001store01a) May 27 09:40:31 [92170] 001store01a pengine: info: LogActions: Leave p_pure-ftpd-etls (Started 001store01a) May 27 09:40:31 [92170] 001store01a pengine: info: LogActions: Leave p_pure-ftpd-itls (Started 001store01a) May 27 09:40:31 [92170] 001store01a pengine: notice: process_pe_message: Calculated transition 91483, saving inputs in /var/lib/pacemaker/pengine/pe-input-756.bz2 May 27 09:40:31 [92171] 001store01a crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response May 27 09:40:31 [92171] 001store01a crmd: info: do_te_invoke: Processing graph 91483 (ref=pe_calc-dc-1622122831-124397) derived from /var/lib/pacemaker/pengine/pe-input-756.bz2 May 27 09:40:31 [92171] 001store01a crmd: notice: run_graph: Transition 91483 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-756.bz2): Complete May 27 09:40:31 [92171] 001store01
Re: [ClusterLabs] Cluster Stopped, No Messages?
I agree -> fencing is mandatory. You can enable the debug logs by editing corosync.conf or /etc/sysconfig/pacemaker. In case simple reload doesn't work, you can set the cluster in global maintenance, stop and then start the stack. Best Regards,Strahil Nikolov On Fri, May 28, 2021 at 22:13, Digimer wrote: On 2021-05-28 3:08 p.m., Eric Robinson wrote: > >> -Original Message- >> From: Digimer >> Sent: Friday, May 28, 2021 12:43 PM >> To: Cluster Labs - All topics related to open-source clustering welcomed >> ; Eric Robinson ; Strahil >> Nikolov >> Subject: Re: [ClusterLabs] Cluster Stopped, No Messages? >> >> Shared storage is not what triggers the need for fencing. Coordinating >> actions >> is what triggers the need. Specifically; If you can run resource on both/all >> nodes at the same time, you don't need HA. If you can't, you need fencing. >> >> Digimer > > Thanks. That said, there is no fencing, so any thoughts on why the node > behaved the way it did? Without fencing, when a communication or membership issues arises, it's hard to predict what will happen. I don't see anything in the short log snippet to indicate what happened. What's on the other node during the event? When did the node disappear and when was it rejoined, to help find relevant log entries? Going forward, if you want predictable and reliable operation, implement fencing asap. Fencing is required. -- Digimer Papers and Projects: https://alteeve.com/w/ "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops." - Stephen Jay Gould ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Pacemaker not issuing start command intermittently
Most RA scripts are writen in bash.Usually you can change the shebang to '!#/usr/bin/bash -x' or you can set trace_ra=1 via 'pcs resource update RESOURCE trace_ra=1 trace_file=/somepath'. If you don't define trace_file, it should create them in /var/lib/heartbeat/trace_ra (based on memory -> so use find/locate). Best Regards,Strahil Nikolov On Fri, May 28, 2021 at 22:10, Abithan Kumarasamy wrote: Hello Team, We have been recently running some tests on our Pacemaker clusters that involve two Pacemaker resources on two nodes respectively. The test case in which we are experiencing intermittent problems is one in which we bring down the Pacemaker resources on both nodes simultaneously. Now our expected behaviour is that our monitor function in our resource agent script detects the downtime, and then should issue a start command. This happens on most successful iterations of our test case. However, on some iterations (approximately 1 out of 30 simulations) we notice that Pacemaker is issuing the start command on only one of the hosts. On the troubled host the monitor function is logging that the resource is down as expected and is exiting with OCF_ERR_GENERIC return code (1) . According to the documentation, this should perform a soft disaster recovery, but when scanning the Pacemaker logs, there is no indication of the start command being issued or invoked. However, it works as expected on the other host. To summarize the issue: - The resource’s monitor is running and returning OCF_ERR_GENERIC - The constraints we have for the resources are satisfied. - There are no visible differences in the Pacemaker logs between the test iteration that failed, and the multiple successful iterations, other than the fact that Pacemaker does not start the resource after the monitor returns OCF_ERR_GENERIC Could you provide some more insight into why this may be happening and how we can further debug this issue? We are currently relying on Pacemaker logs, but are there additional diagnostics to further debug? Thanks,Abithan ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Cluster Stopped, No Messages?
Did you configure pacemaker blackbox ? If not, it could be valuable in such cases. Also consider updating as soon as possible. Most probably nobody can count the bug fixes that were introduced between 7.5 and 7.9, nor anyone will be able to help as you are running a pretty outdated version (even by RH standards). Best Regards,Strahil Nikolov___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] One Failed Resource = Failover the Cluster?
It shouldn't relocate or affect any other resource,as long as the stop succeeds.If the stop operation times out or fails -> fencing kicks in. Best Regards,Strahil Nikolov___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] One Failed Resource = Failover the Cluster?
Based on the constraint rules you have mentioned , failure of mysql should not cause a failover to another node. For better insight, you have to be able to reproduce the issue and share the logs with the community. Best Regards,Strahil Nikolov On Sat, Jun 5, 2021 at 23:33, Eric Robinson wrote: > -Original Message- > From: Users On Behalf Of > kgail...@redhat.com > Sent: Friday, June 4, 2021 4:49 PM > To: Cluster Labs - All topics related to open-source clustering welcomed > > Subject: Re: [ClusterLabs] One Failed Resource = Failover the Cluster? > > On Fri, 2021-06-04 at 19:10 +, Eric Robinson wrote: > > Sometimes it seems like Pacemaker fails over an entire cluster when > > only one resource has failed, even though no other resources are > > dependent on it. Is that expected behavior? > > > > For example, suppose I have the following colocation constraints… > > > > filesystem with drbd master > > vip with filesystem > > mysql_01 with filesystem > > mysql_02 with filesystem > > mysql_03 with filesystem > > By default, a resource that is colocated with another resource will influence > that resource's location. This ensures that as many resources are active as > possible. > > So, if any one of the above resources fails and meets its migration- > threshold, > all of the resources will move to another node so a recovery attempt can be > made for the failed resource. > > No resource will be *stopped* due to the failed resource unless it depends > on it. > Thanks, but I'm confused by your previous two paragraphs. On one hand, "if any one of the above resources fails and meets its migration- threshold, all of the resources will move to another node." Obviously moving resources requires stopping them. But then, "No resource will be *stopped* due to the failed resource unless it depends on it." Those two statements seem contradictory to me. Not trying to be argumentative. Just trying to understand. > As of the forthcoming 2.1.0 release, the new "influence" option for > colocation constraints (and "critical" resource meta-attribute) controls > whether this effect occurs. If influence is turned off (or the resource made > non-critical), then the failed resource will just stop, and the other > resources > won't move to try to save it. > That sounds like the feature I'm waiting for. In the example configuration I provided, I would not want the failure of any mysql instance to cause cluster failover. I would only want the cluster to failover if the filesystem or drbd resources failed. Basically, if a resource breaks or fails to stop, I don't want the whole cluster to failover if nothing depends on that resource. Just let it stay down until someone can manually intervene. But if an underlying resource fails that everything else is dependent on (drbd or filesystem) then go ahead and failover the cluster. > > > > …and the following order constraints… > > > > promote drbd, then start filesystem > > start filesystem, then start vip > > start filesystem, then start mysql_01 > > start filesystem, then start mysql_02 > > start filesystem, then start mysql_03 > > > > Now, if something goes wrong with mysql_02, will Pacemaker try to fail > > over the whole cluster? And if mysql_02 can’t be run on either > > cluster, then does Pacemaker refuse to run any resources? > > > > I’m asking because I’ve seen some odd behavior like that over the > > years. Could be my own configuration mistakes, of course. > > > > -Eric > -- > Ken Gaillot > > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ Disclaimer : This email and any files transmitted with it are confidential and intended solely for intended recipients. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management. Warning: Although Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] VIP monitor Timed Out
I would try to add 'trace_ra=1' or 'trace_ra=1 trace_file=' to debug it further. In the first option (without trace_file) , the file will be at /var/lib/heartbeat/trace_ra//*timestamp Are you sure that the system is not overloaded and can't respond in time ? Best Regards, Strahil Nikolov В петък, 2 юли 2021 г., 17:53:06 ч. Гринуич+3, PASERO Florent написа: Hi, Once or twice a week, we have a 'Timed out' on our VIP: ~$ pcs status Cluster name: zbx_pprod_Web_Core Cluster Summary: * Stack: corosync * Current DC: #(version 2.0.5-9.el8_4.1-ba59be7122) - partition with quorum * Last updated: Mon Jun 28 16:32:09 2021 * Last change: Mon Jun 14 12:42:57 2021 by root via cibadmin on ## * 2 nodes configured * 2 resource instances configured Node List: * Online: [ # #] Full List of Resources: * Resource Group: zbx_pprod_Web_Core: * VIP (ocf::heartbeat:IPaddr2): Started # * ZabbixServer (systemd:zabbix-server): Started ## Failed Resource Actions: * VIP_monitor_5000 on # 'error' (1): call=69, status='Timed Out', exitreason='', last-rc-change='2021-06-24 14:41:57 +02:00', queued=0ms, exec=0ms * VIP_monitor_5000 on # 'error' (1): call=11, status='Timed Out', exitreason='', last-rc-change='2021-06-17 14:18:20 +02:00', queued=0ms, exec=0ms We have the same issue on two completely different clusters. We can see in the log : Jun 24 14:41:29 # pacemaker-execd [1442069] (child_timeout_callback) warning: VIP_monitor_5000 process (PID 2752333) timed out Jun 24 14:41:34 #pacemaker-execd [1442069] (child_timeout_callback) crit: VIP_monitor_5000 process (PID 2752333) will not die! Jun 24 14:41:57 # pacemaker-execd [1442069] (operation_finished) warning: VIP_monitor_5000[2752333] timed out after 2ms Jun 24 14:41:57 # pacemaker-controld [1442072] (process_lrm_event) error: Result of monitor operation for VIP on #: Timed Out | call=69 key=VIP_monitor_5000 timeout=2ms Jun 24 14:41:57 # pacemaker-based [1442067] (cib_process_request) info: Forwarding cib_modify operation for section status to all (origin=local/crmd/722) Jun 24 14:41:57 # pacemaker-based [1442067] (cib_perform_op) info: Diff: --- 0.54.443 2 Jun 24 14:41:57 # pacemaker-based [1442067] (cib_perform_op) info: Diff: +++ 0.54.444 (null) Jun 24 14:41:57 # pacemaker-based [1442067] (cib_perform_op) info: + /cib: @num_updates=444 Thanks for help Classification : Internal This message and any attachments (the "message") is intended solely for the intended addressees and is confidential. If you receive this message in error,or are not the intended recipient(s), please delete it and any copies from your systems and immediately notify the sender. Any unauthorized view, use that does not comply with its purpose, dissemination or disclosure, either whole or partial, is prohibited. Since the internet cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS (and its subsidiaries) shall not be liable for the message if modified, changed or falsified. Do not print this message unless it is necessary, consider the environment. -- Ce message et toutes les pieces jointes (ci-apres le "message") sont etablis a l'intention exclusive de ses destinataires et sont confidentiels. Si vous recevez ce message par erreur ou s'il ne vous est pas destine, merci de le detruire ainsi que toute copie de votre systeme et d'en avertir immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de ce message qui n'est pas conforme a sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer l'integrite de ce message electronique susceptible d'alteration, BNP Paribas (et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese ou il aurait ete modifie, deforme ou falsifie. N'imprimez ce message que si necessaire, pensez a l'environnement. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] @ maillist Admins - DMARC (yahoo)
Actually, I don't mind but it will be nice if I don't get kicked from time to time due to too many bounces. :) Best Regards,Strahil Nikolov On Sat, 2021-07-10 at 12:34 +0100, lejeczek wrote: > Hi Admins(of this mailing list) > > Could you please fix in DMARC(s) so those of us who are on > Yahoo would be able to receive own emails/thread. > > many thanks, L. I suppose we should do something, since this is likely to be more of an issue as time goes on. Unfortunately, it's not as simple as flipping a switch. These are the two reasonable choices: (1) Change the "From" on list messages so that they appear to be from the list, rather than the poster. For example, your posts would show up as "From: lejeczek via ClusterLabs Users " rather than "From: lejeczek ". This is less intrusive but makes it more difficult to reply directly to the sender, add the sender to an address book, etc. (2) Stop adding [ClusterLabs] to subject lines, setting ReplyTo: to the list instead of original author, and adding the list signature. This is more standards-compliant, since the List-* headers can still be used for filtering, unsubscribing, and replying to the list, but not all mail clients make those easy to use. Anyone have preferences for one over the other? (Less reasonable options include wrapping every post in MIME, and disallowing users from DMARC domains to post to the list.) -- Ken Gaillot ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] ZFS Opinions?
DRBD is a good option but if you use it with iSCSI - don't forget to add the iSCSI block devices into the lvm filter (client will most probably useit as LVM PV). Another option that I like is GlusterFS. It's easy to deploy and performance scales-out the number of servers. Of course, performance tuning on both is not a trivial task - as any other performance tuning. Best Regards,Strahil Nikolov On Sat, Jul 10, 2021 at 0:12, Eric Robinson wrote: ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Autostart/Enabling of Pacemaker and corosync
I prefer that the stack is auto enabled. Imagine that you got a DB that is replicated and primary DB node is fenced. You would like that node to join the cluster and if possible to sync with the new primary instead of staying down. One such example is the SAP HANA DB. Imagine that the current primary node looses storage and it failed to commit all transactions to disk. Without replication you will endure data loss for the last 1-2 minutes (depends on your monitoring interval) unless you got a replication. Yet, there are cases where the previously fenced node joins the cluster and due to hardware issues causes performance degradation on the current master. Both cases have their benefits and drawbacks and you have to weight them all before taking that decision. Best Regards,Strahil Nikolov On Mon, Apr 26, 2021 at 20:04, Moneta, Howard wrote: Hello community. I have read that it is not recommended to set Pacemaker and corosync to enabled/auto start on the nodes. Is this how people have it configured? If a computer restarts unexpectedly, is it better to manually investigate first or allow the node to come back online and rejoin the cluster automaticly in order to minimize downtime? If the auto start is not enabled, how do you handle patching? I’m using Pacemaker with PAF, PostgreSQL Automatic Failover. I had thought to follow the published guidance and not set those processes to enabled but other coworkers are resisting and saying that the systems should be configured to recover by themselves around patching or even a temporary unplanned network/virtualization glitch. Thanks, Howard This message may contain information, including personally identifiable information that is confidential, privileged, or otherwise legally protected. If you are not the intended recipient, please immediately notify the sender and delete this message without copying, disclosing, or distributing it. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Resolving cart before the horse with mounted filesystems.
Ken ment yo use 'Filesystem' resourse for mounting that NFS server and then clone that resource. Best Regards,Strahil Nikolov On Fri, Apr 30, 2021 at 18:44, Matthew Schumacher wrote: On 4/30/21 8:11 AM, Ken Gaillot wrote >> 2. Make the nfs mount itself a resource and make VirtualDomain >> resources depend on it. In order for this to work each node would >> need >> it's own nfs mount resource, and VirtualDomain resources that can run >> on >> any node would need to depend on the nfs mount resource of whatever >> node >> they decide to run on but not the nfs mount resource of any other >> node. >> I'm not sure how to make this work because the dependency changes >> with >> what node the VirtualDomain resource is started on. > If each VM needs a particular mount, you can clone the NFS server, > create separate groups where groupN = (mountN, vmN), and colocate/order > the groups relative to the clone. > Thanks for the reply, but I'm not sure I follow. Why would I clone the NFS server? I only need one server, and I only need the singular NFS mount on each node. Do you have documentation you could point to so I can catch up in my understanding? Matt ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] VirtualDomain & "deeper" monitors - what/how?
Hey Ken, does this feature work for other Nagios stuff ? Best Regards,Strahil Nikolov On Fri, Apr 30, 2021 at 17:57, Ken Gaillot wrote: On Fri, 2021-04-30 at 11:00 +0100, lejeczek wrote: > Hi guys > > I'd like to ask around for thoughts & suggestions on any > semi/official ways to monitor VirtualDomain. > Something beyond what included RA does - such as actual > health testing of and communication with VM's OS. > > many thanks, L. This use case led to a Pacemaker feature many moons ago ... Pacemaker supports nagios plug-ins as a resource type (e.g. nagios:check_apache_status). These are service checks usually used with monitoring software such as nagios, icinga, etc. If the service being monitored is inside a VirtualDomain, named vm1 for example, you can configure the nagios resource with the resource meta- attribute container="vm1". If the nagios check fails, Pacemaker will restart vm1. (This feature was added before OS containers became popular, so the naming is a bit unfortunate.) -- Ken Gaillot ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Preventing multiple resources from moving at the same time.
In order iSCSI to be transparent to the relevant clients, you need to use a special resource that blocks the iSCSI port during the failover. TCP will retransmit during the failover and will never receive an error due to the fact that the VIP is missing. The name is ocf:heartbeat:portblock that should be defined in the beginning and end of the resource group. It doesn't work for EL8 systems (or at least it didn't work for me on Gluster) as they use nftables and iptables translator is quite limited. Also, clients that use multipathed iSCSI also need some tuning as the default 120s is not suitable. Best Regards,Strahil Nikolov___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] CCIB migration from Pacemaker 1.x to 2.x
> How to handle it? You need to : - Setup and TEST stonith - Add a 3rd node (even if it doesn't host any resources) or setup a node for kronosnet Best Regards, Strahil Nikolov ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
[ClusterLabs] Order set troubles
Hello All, I have a trouble creating an order set . The end goal is to create a 2 node cluster where nodeA will mount nfsA , while nodeB will mount nfsB.On top of that a depended cloned resource should start on the node only if nfsA or nfsB has started locally. A prototype code would be something like: pcs constraint order start (nfsA or nfsB) then start resource-clone I tried to create a set like this, but it works only on nodeB: pcs constraint order set nfsA nfsB resource-clone Any idea how to implement that order constraint ? Thanks in advance. Best Regards, Strahil Nikolov ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Antw: Re: Antw: [EXT] Re: Order set troubles
Thanks everyone! I really appreciate your help. Actually , I found a RH solution (#5423971) that gave me enough ideas /it is missing some steps/ to setup the cluster prooperly. So far , I have never used node attributes, order sets and location constraints based on 'ocf:pacemaker: attribute's active/inactive values . I can say that I have learned alot today. Best Regards,Strahil Nikolov___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Antw: Re: Antw: [EXT] Re: Order set troubles
Just a clarification. I'm using separate NFS shares for each HANA, so even if someone wipes the NFS for DC1, the cluster will failover to DC2 (separate NFS) and survive. Best Regards,Strahil Nikolov___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Antw: [EXT] Re: Order set troubles
Use Case: nfsA is shared filesystem for HANA running in site AnfsB is shared filesystem for HANA running in site B clusterized resource of type SAPHanaTopology must run on all systems if the FS for the HANA is running Yet, if siteA dies for some reason, I want to make SAPHanaTopology to still start on the nodes in site B. I think that it's a valid use case. Best Regards,Strahil Nikolov On Thu, Mar 25, 2021 at 8:59, Ulrich Windl wrote: >>> Ken Gaillot schrieb am 24.03.2021 um 18:56 in Nachricht <5bffded9c6e614919981dcc7d0b2903220bae19d.ca...@redhat.com>: > On Wed, 2021‑03‑24 at 09:27 +0000, Strahil Nikolov wrote: >> Hello All, >> >> I have a trouble creating an order set . >> The end goal is to create a 2 node cluster where nodeA will mount >> nfsA , while nodeB will mount nfsB.On top of that a depended cloned >> resource should start on the node only if nfsA or nfsB has started >> locally. This looks like ad odd design to me, and I wonder: What is the use case? (We are using "NFS loop-mounts" for many years, where the cluster needs the NFS service it provides, but that's a different design) Regards, Ulrich >> >> A prototype code would be something like: >> pcs constraint order start (nfsA or nfsB) then start resource‑clone >> >> I tried to create a set like this, but it works only on nodeB: >> pcs constraint order set nfsA nfsB resource‑clone >> >> Any idea how to implement that order constraint ? >> Thanks in advance. >> >> Best Regards, >> Strahil Nikolov > > Basically you want two sets, one with nfsA and nfsB with no ordering > between them, and a second set with just resource‑clone, ordered after > the first set. > > I believe the pcs syntax is: > > pcs constraint order set nfsA nfsB sequential=false require‑all=false > set resource‑clone > > sequential=false says nfsA and nfsB have no ordering between them, and > require‑all=false says that resource‑clone only needs one of them. > > (I don't remember for sure the order of the sets in the command, i.e. > whether it's the primary set first or the dependent set first, but I > think that's right.) > ‑‑ > Ken Gaillot > > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Antw: [EXT] Re: Order set troubles
OCF_CHECK_LEVEL 20NFS sometimes fails to start (systemd racing condition with dnsmasq) Best Regards,Strahil Nikolov On Thu, Mar 25, 2021 at 12:18, Andrei Borzenkov wrote: On Thu, Mar 25, 2021 at 10:31 AM Strahil Nikolov wrote: > > Use Case: > > nfsA is shared filesystem for HANA running in site A > nfsB is shared filesystem for HANA running in site B > > clusterized resource of type SAPHanaTopology must run on all systems if the > FS for the HANA is running > And the reason you put NFS under pacemaker control in the first place? It is not going to switch over, just put it in /etc/fstab. > Yet, if siteA dies for some reason, I want to make SAPHanaTopology to still > start on the nodes in site B. > > I think that it's a valid use case. > > Best Regards, > Strahil Nikolov > > On Thu, Mar 25, 2021 at 8:59, Ulrich Windl > wrote: > >>> Ken Gaillot schrieb am 24.03.2021 um 18:56 in > Nachricht > <5bffded9c6e614919981dcc7d0b2903220bae19d.ca...@redhat.com>: > > On Wed, 2021‑03‑24 at 09:27 +, Strahil Nikolov wrote: > >> Hello All, > >> > >> I have a trouble creating an order set . > >> The end goal is to create a 2 node cluster where nodeA will mount > >> nfsA , while nodeB will mount nfsB.On top of that a depended cloned > >> resource should start on the node only if nfsA or nfsB has started > >> locally. > > This looks like ad odd design to me, and I wonder: What is the use case? > (We are using "NFS loop-mounts" for many years, where the cluster needs the > NFS service it provides, but that's a different design) > > Regards, > Ulrich > > > > >> > >> A prototype code would be something like: > >> pcs constraint order start (nfsA or nfsB) then start resource‑clone > >> > >> I tried to create a set like this, but it works only on nodeB: > >> pcs constraint order set nfsA nfsB resource‑clone > >> > >> Any idea how to implement that order constraint ? > >> Thanks in advance. > >> > >> Best Regards, > >> Strahil Nikolov > > > > Basically you want two sets, one with nfsA and nfsB with no ordering > > between them, and a second set with just resource‑clone, ordered after > > the first set. > > > > I believe the pcs syntax is: > > > > pcs constraint order set nfsA nfsB sequential=false require‑all=false > > set resource‑clone > > > > sequential=false says nfsA and nfsB have no ordering between them, and > > require‑all=false says that resource‑clone only needs one of them. > > > > (I don't remember for sure the order of the sets in the command, i.e. > > whether it's the primary set first or the dependent set first, but I > > think that's right.) > > ‑‑ > > Ken Gaillot > > > > > ___ > > Manage your subscription: > > https://lists.clusterlabs.org/mailman/listinfo/users > > > > ClusterLabs home: https://www.clusterlabs.org/ > > > > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ > > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Antw: [EXT] Colocation per site ?
Hi Ken, can you provide a prototype code example.
Currently,I'm making a script that will be used in a systemd service managed by
the cluster.Yet, I would like to avoid non-pacemaker solutions.
Best Regards,Strahil Nikolov
On Mon, Mar 29, 2021 at 20:12, Ken Gaillot wrote: On
Sun, 2021-03-28 at 09:20 +0300, Andrei Borzenkov wrote:
> On 28.03.2021 07:16, Strahil Nikolov wrote:
> > I didn't mean DC as a designated coordinator, but as a physical
> > Datecenter location.
> > Last time I checked, the node attributes for all nodes seemed the
> > same.I will verify that tomorrow (Monday).
> >
>
> Yes, I was probably mistaken. It is different with scale-out, agent
> puts
> information in global property section of CIB.
>
> Ideally we'd need expression that says "on node where site attribute
> is
> the same as on node where clone master is active" but I guess there
> is
> no way to express it in pacemaker.
Yep, colocation by node attribute (combined with colocation with
promoted role)
https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/2.0/html-single/Pacemaker_Explained/index.html#_colocation_properties
>
> I do not see any easy way to implement it without essentially
> duplicating SAPHanaTopology. There are some attributes that are
> defined
> but never set so far, you may try to open service request to
> implement
> consistent attribute for all nodes on current primary site.
>
> ...
>
> Hmm ... agent sets (at least, should set) hana_${SID}_vhost attribute
> for each node and this attribute must be unique and different between
> two sites. May be worth to look into it.
>
>
> > Best Regards,Strahil Nikolov
> >
> >
> > On Fri, Feb 19, 2021 at 16:51, Andrei Borzenkov<
> > arvidj...@gmail.com> wrote: On Fri, Feb 19, 2021 at 2:44 PM
> > Strahil Nikolov wrote:
> > >
> > >
> > > > Do you have a fixed relation between node >pairs and VIPs? I.e.
> > > > must
> > > > A/D always get VIP1, B/E - VIP2 etc?
> > >
> > > I have to verify it again, but generally speaking - yes , VIP1 is
> > > always on nodeA/D (master), VIP2 on nodeB/E (worker1) , etc.
> > >
> > > I guess I can set negative constraints (-inf) -> VIP1 on node B/E
> > > + nodeC/F, but the stuff with the 'same DC as master' is the
> > > tricky part.
> > >
> >
> > I am not sure I understand what DC has to do with it. You have two
> > scale-out SAP HANA instances, one is primary, another is secondary.
> > If
> > I understand correctly your requirements, your backup application
> > needs to contact the primary instance which may failover to another
> > site. You must be using some resource agent for it, to manage
> > failover. The only one I am aware of is SAPHanaSR-ScaleOut. It
> > already
> > sets different node properties for primary and secondary sites.
> > Just
> > use them. If you use something else, just look at what attributes
> > your
> > RA sets. Otherwise you will be essentially duplicating your RA
> > functionality because you will somehow need to find out which site
> > is
> > currently primary.
> >
> > There is no guarantee that pacemaker DC wil be on the same site as
> > SAP
> > HANA primary system.
> >
> >
>
> ___
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
>
--
Ken Gaillot
___
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users
ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Antw: [EXT] Colocation per site ?
Hi Reid,
in order the colocation to work , it should (logically) look like
pcs constraint colocation add BKP_IP1 with Master
rsc_SAPHana__HDB-clone INFINITY node-attribute=$(attribute
SITE on Master rsc_SAPHana__HDB-clone)
So far , I have created a bash script to:- detect global maintenance and sleep-
detect status without primary hana- set node attributes (all nodes) indicating
primary and secondary- on 'cleanup' , removesall attributes
That script is called via systemd service managed by pacemaker (and running
with the master).
There is a delay in the bash script and also a delay resource is defined.
All 3 IPs have a location constraints:- Work only on specific pair- Don't land
on othernodes- ensure -INFINITY on nodes with attribute not equal to 'primary'
Order set combines the systemd service, the delay resource and all IPs
(require-all=true).
Ugly, but it looks promising. Yet, I'm not happy to have my own scripts in the
cluster's logic.
Best Regards,Strahil Nikolov
On Tue, Mar 30, 2021 at 10:06, Reid Wahl wrote: You can
try the following and see if it works, replacing the items in angle brackets
(<>).
# pcs constraint colocation add with Master
INFINITY node-attribute=hana__site
However, `pcs constraint colocation add --help` gives no information about
what options it accepts. It just says "[options]".
Usage: pcs constraint [constraints]...
colocation add [] with []
[score] [options] [id=constraint-id]
Request to run on the same node where pacemaker has
determined should run. Positive values of score
mean the resources should be run on the same node, negative values
mean the resources should not be run on the same node. Specifying
'INFINITY' (or '-INFINITY') for the score forces to
run (or not run) with (score defaults to "INFINITY").
A role can be: 'Master', 'Slave', 'Started', 'Stopped' (if no role is
specified, it defaults to 'Started').
So it's entirely possible that pcs doesn't support creating colocation
constraints with node attributes. If not, then you could edit the CIB manually
and add a constraint like this:
On Mon, Mar 29, 2021 at 9:07 PM Strahil Nikolov wrote:
Hi Ken, can you provide a prototype code example.
Currently,I'm making a script that will be used in a systemd service managed by
the cluster.Yet, I would like to avoid non-pacemaker solutions.
Best Regards,Strahil Nikolov
On Mon, Mar 29, 2021 at 20:12, Ken Gaillot wrote: On
Sun, 2021-03-28 at 09:20 +0300, Andrei Borzenkov wrote:
> On 28.03.2021 07:16, Strahil Nikolov wrote:
> > I didn't mean DC as a designated coordinator, but as a physical
> > Datecenter location.
> > Last time I checked, the node attributes for all nodes seemed the
> > same.I will verify that tomorrow (Monday).
> >
>
> Yes, I was probably mistaken. It is different with scale-out, agent
> puts
> information in global property section of CIB.
>
> Ideally we'd need expression that says "on node where site attribute
> is
> the same as on node where clone master is active" but I guess there
> is
> no way to express it in pacemaker.
Yep, colocation by node attribute (combined with colocation with
promoted role)
https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/2.0/html-single/Pacemaker_Explained/index.html#_colocation_properties
>
> I do not see any easy way to implement it without essentially
> duplicating SAPHanaTopology. There are some attributes that are
> defined
> but never set so far, you may try to open service request to
> implement
> consistent attribute for all nodes on current primary site.
>
> ...
>
> Hmm ... agent sets (at least, should set) hana_${SID}_vhost attribute
> for each node and this attribute must be unique and different between
> two sites. May be worth to look into it.
>
>
> > Best Regards,Strahil Nikolov
> >
> >
> > On Fri, Feb 19, 2021 at 16:51, Andrei Borzenkov<
> > arvidj...@gmail.com> wrote: On Fri, Feb 19, 2021 at 2:44 PM
> > Strahil Nikolov wrote:
> > >
> > >
> > > > Do you have a fixed relation between node >pairs and VIPs? I.e.
> > > > must
> > > > A/D always get VIP1, B/E - VIP2 etc?
> > >
> > > I have to verify it again, but generally speaking - yes , VIP1 is
> > > always on nodeA/D (master), VIP2 on nodeB/E (worker1) , etc.
> > >
> > > I guess I can set negative constraints (-inf) -> VIP1 on node B/E
> > > + nodeC/F, but the stuff with the 'same DC as master' is the
> > > tricky part.
> > >
> >
> > I am not sure I understand what DC has to do with it. You have two
> > scale-out SAP HANA instances, one is primary, another is sec
Re: [ClusterLabs] Antw: [EXT] Colocation per site ?
I guess that feature was added in a later version (still on RHEL 8.2). pcs constraint colocation bkp2 with Master rsc_SAPHana__HDB score=INFINITY node-attribute=hana__site (or using any other attribute) brings 'pcs constraint help' output. It seems that it doesn't like the syntax. Maybe someone can share the 'pcs cluster edit' xml section, so I can try to push it directly into the cib ? Best Regards,Strahil Nikolov On Tue, Mar 30, 2021 at 19:45, Andrei Borzenkov wrote: On 30.03.2021 17:42, Ken Gaillot wrote: >> >> Colocation does not work, this will force everything on the same node >> where master is active and that is not what we want. > > Nope, you can colocate by node attribute instead of node. > > Colocating by node attribute says "put this resource on a node that has > the same value of this node attribute as the node running this other > resource". So if site is the node attribute, colocating A with B by > site means that A has to run on a node that has the same value for site > as wherever B is. > > Simple colocation by node is actually implemented internally as > colocation by node attribute where the attribute is the node name. > Oh! Thank you! Indeed, this is exactly what OP needs. No custom scripting is needed. Thank you for your patience. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Antw: [EXT] Colocation per site ?
Disregard the previous one... it needs 'pcs constraint colocation add' to work. Best Regards,Strahil Nikolov On Wed, Mar 31, 2021 at 8:08, Strahil Nikolov wrote: I guess that feature was added in a later version (still on RHEL 8.2). pcs constraint colocation bkp2 with Master rsc_SAPHana__HDB score=INFINITY node-attribute=hana__site (or using any other attribute) brings 'pcs constraint help' output. It seems that it doesn't like the syntax. Maybe someone can share the 'pcs cluster edit' xml section, so I can try to push it directly into the cib ? Best Regards,Strahil Nikolov On Tue, Mar 30, 2021 at 19:45, Andrei Borzenkov wrote: On 30.03.2021 17:42, Ken Gaillot wrote: >> >> Colocation does not work, this will force everything on the same node >> where master is active and that is not what we want. > > Nope, you can colocate by node attribute instead of node. > > Colocating by node attribute says "put this resource on a node that has > the same value of this node attribute as the node running this other > resource". So if site is the node attribute, colocating A with B by > site means that A has to run on a node that has the same value for site > as wherever B is. > > Simple colocation by node is actually implemented internally as > colocation by node attribute where the attribute is the node name. > Oh! Thank you! Indeed, this is exactly what OP needs. No custom scripting is needed. Thank you for your patience. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Antw: [EXT] Colocation per site ?
Damn... I am too hasty. It seems that the 2 resources I have already configured are also running on the master. The colocation constraint is like: rsc_bkpip3_SAPHana_SID_HDBinst_num with rsc_SAPHana_SID_HDBinst_num-clone (score: INFINITY) (node-attribute:hana_sid_site) (rsc-role:Started) (with-rsc-role:Master) Setting location constraints to prevent from running on the masters causes the resource to stop. Best Regards,Strahil Nikolov___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Antw: [EXT] Colocation per site ?
Actually , I found the error with the help of the RH support. I was colocating with the Master SAPHanaController resource and thus the cluster did what I have told it to do ;) Now, I just colocated (node-attribute=hana_sid_site) with the first backup ip (which is colocated with the master) and added a location constraint for bkpip2 & bkpip3 and works like a charm. I am still considering wether to keep the delay primitive that prevents the backup IPs to be brought up immediately. The problem is that the current backup solution will be triggered from each HANA node , and as we got a lot of custom stuff - I want to make it fool-proof as much as possible. I've already organised a discussion about those backup IPs. Best Regards,Strahil Nikolov On Wed, Mar 31, 2021 at 10:54, Andrei Borzenkov wrote: On Wed, Mar 31, 2021 at 8:34 AM Strahil Nikolov wrote: > > Damn... I am too hasty. > > It seems that the 2 resources I have already configured are also running on > the master. > > The colocation constraint is like: > > rsc_bkpip3_SAPHana_SID_HDBinst_num with rsc_SAPHana_SID_HDBinst_num-clone > (score: INFINITY) (node-attribute:hana_sid_site) (rsc-role:Started) > (with-rsc-role:Master) > > > Setting location constraints to prevent from running on the masters causes > the resource to stop. > Well, this constraint does not prevent two resources from running on the same node. You would need to either use clone and restrict the number of total and per-node clone instances or use negative colocation between each pair of your resources (so they are not started on the same node). But I suspect you really overengineer it. At the end SAP HANA backup is always initiated by SAP HANA itself and you only need the single IP address to do it. This single IP address is already available as part of SAP HANA replication setup (and should be active on the master HANA node). Backup software usually wants to know which *physical* servers a given virtual client may be allowed to run on, but in this case it should really be enough to simply add all nodes to this list. You may want to discuss it with your backup team again. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
[ClusterLabs] SAPHanaController & SAPHanaTopology question
Hello All, I am testing the newly built HANA (Scale-out) cluster and it seems that:Neither SAPHanaController, nor SAPHanaTopology are stopping the HANA when I put the nodes (same DC = same HANA) in standby. This of course leads to a situation where the NFS cannot be umounted and despite the stop timeout - leads to fencing(on-fail=fence). I thought that the Controller resource agent is stopping the HANA and the slave role should not be 'stopped' before that . Maybe my expectations are wrong ? Best Regards,Strahil Nikolov ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] SAPHanaController & SAPHanaTopology question
I just checked and it seems that I got 2 regular rules for stopping topology before nfs_active: stop Topology-clone then stop nfs_active_siteA (kind: Mandatory)stop Topology-clone then stop nfs_active_siteB (kind Mandatory) Also I got:start Topology-clone then start Controller-clone (kind: Mandatory) And also resource sets that take care that all FS start and then the relevant nfs_active resources. Also, It seems that regular order rules cannot be removed via ID , maybe a Feature request is needed. Best Regards,Strahil Nikolov If you mean a whole constraint set, then yes -- run `pcs constraint --full` to get a list of all constraints with their constraint IDs. Then run `pcs constraint remove ` to remove a particular constraint. This can include set constraints. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] "iscsi.service: Unit cannot be reloaded because it is inactive."
Unless you clusterize your iSCSI.You can see https://forums.centos.org/viewtopic.php?t=65539 Keep in mind that you will need to setup another resource to block the iscsi port before and after the iscsi resources or your clients will get a I/O error. Sadly I didn't have the time to refine it and it never ended into centos's wiki. Best Regards,Strahil Nikolov On Sat, Apr 3, 2021 at 17:52, Andrei Borzenkov wrote: On 03.04.2021 17:35, Jason Long wrote: > Hello, > I configure my clustering labs with three nodes. You have two node cluster. What is running on nodes outside of cluster is out of scope of pacemaker. > One of my nodes is iSCSI Shared Storage. Everything was OK until I restarted > my iSCSI Shared Storage. On node1 I checked the status of my cluster: > > # pcs status > Cluster name: mycluster > Cluster Summary: > * Stack: corosync > * Current DC: node1 (version 2.0.5-10.fc33-ba59be7122) - partition with > quorum > * Last updated: Sat Apr 3 18:45:49 2021 > * Last change: Mon Mar 29 19:36:35 2021 by root via cibadmin on node1 > * 2 nodes configured > * 3 resource instances configured > > > Node List: > * Online: [ node1 node2 ] > > > Full List of Resources: > * Resource Group: apache: > * httpd_fs(ocf::heartbeat:Filesystem): Stopped > * httpd_vip(ocf::heartbeat:IPaddr2): Stopped > * httpd_ser(ocf::heartbeat:apache): Stopped > > > Failed Resource Actions: > * httpd_fs_start_0 on node1 'not installed' (5): call=14, > status='complete', exitreason='Couldn't find device > [/dev/mapper/vg_apache-lv_apache]. Expected /dev/??? to exist', > last-rc-change='2021-04-03 18:37:04 +04:30', queued=0ms, exec=502ms > * httpd_fs_start_0 on node2 'not installed' (5): call=14, > status='complete', exitreason='Couldn't find device > [/dev/mapper/vg_apache-lv_apache]. Expected /dev/??? to exist', > last-rc-change='2021-04-03 18:37:05 +04:30', queued=0ms, exec=540ms > > > Daemon Status: > corosync: active/enabled > pacemaker: active/enabled > pcsd: active/enabled > > > And checked the iSCSI Shared Storage. It showed me below error: > > [root@node3 ~]# systemctl status iscsi.service > ● iscsi.service - Login and scanning of iSCSI devices > Loaded: loaded (/usr/lib/systemd/system/iscsi.service; enabled; vendor > preset: enabled) > Active: inactive (dead) > Condition: start condition failed at Sat 2021-04-03 18:49:08 +0430; 2s ago > └─ ConditionDirectoryNotEmpty=/var/lib/iscsi/nodes was not met It seems to tell you why it was not started. > Docs: man:iscsiadm(8) > man:iscsid(8) > > > Apr 03 18:39:17 node3.localhost.localdomain systemd[1]: Condition check > resulted in Login and scanning of iSCSI devices being skipped. > Apr 03 18:39:17 node3.localhost.localdomain systemd[1]: iscsi.service: Unit > cannot be reloaded because it is inactive. > Apr 03 18:39:17 node3.localhost.localdomain systemd[1]: iscsi.service: Unit > cannot be reloaded because it is inactive. > Apr 03 18:49:08 node3.localhost.localdomain systemd[1]: Condition check > resulted in Login and scanning of iSCSI devices being skipped. > > > Why "iscsi.service" is inactive? I tried to restart it, but it couldn't start! > How to solve it? > Did you try asking on support list/forum dedicated to Linux iSCSI? ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] SAPHanaController & SAPHanaTopology question
Hi Reid, I will check it out in Monday, but I'm pretty sure I created an order set that first stops the topology and only then it stops the nfs-active. Yet, I made the stupid decision to prevent ocf:heartbeat:Filesystem (and setting a huge timeout for the stop operation) from killing those 2 SAP processes which led to 'I can't umount, giving up'-like notification and of course fenced the entire cluster :D . Note taken, stonith has now different delays , and Filesystem can kill the processes. As per the SAP note from Andrei, these could really be 'fast restart' mechanisms in HANA 2.0 and it looks safe to be killed (will check with SAP about that). P.S: Is there a way to remove a whole set in pcs , cause it's really irritating when the stupid command wipes the resource from multiple order constraints? Best Regards,Strahil Nikolov On Fri, Apr 2, 2021 at 23:44, Reid Wahl wrote: Hi, Strahil. Based on the constraints documented in the article you're following (RH KB solution 5423971), I think I see what's happening. The SAPHanaTopology resource requires the appropriate nfs-active attribute in order to run. That means that if the nfs-active attribute is set to false, the SAPHanaTopology resource must stop. However, there's no rule saying SAPHanaTopology must finish stopping before the nfs-active attribute resource stops. In fact, it's quite the opposite: the SAPHanaTopology resource stops only after the nfs-active resource stops. At the same time, the NFS resources are allowed to stop after the nfs-active attribute resource has stopped. So the NFS resources are stopping while the SAPHana* resources are likely still active. Try something like this: # pcs constraint order hana_nfs1_active-clone then SAPHanaTopology__-clone kind=Optional # pcs constraint order hana_nfs2_active-clone then SAPHanaTopology__-clone kind=Optional This says "if both hana_nfs1_active and SAPHanaTopology are scheduled to start, then make hana_nfs1_active start first. If both are scheduled to stop, then make SAPHanaTopology stop first." "kind=Optional" means there's no order dependency unless both resources are already going to be scheduled for the action. I'm using kind=Optional here even though kind=Mandatory (the default) would make sense, because IIRC there were some unexpected interactions with ordering constraints for clones, where events on one node had unwanted effects on other nodes. I'm not able to test right now since setting up an environment for this even with dummy resources is non-trivial -- but you're welcome to try this both with and without kind=Optional if you'd like. Please let us know how this goes. On Fri, Apr 2, 2021 at 2:20 AM Strahil Nikolov wrote: Hello All, I am testing the newly built HANA (Scale-out) cluster and it seems that:Neither SAPHanaController, nor SAPHanaTopology are stopping the HANA when I put the nodes (same DC = same HANA) in standby. This of course leads to a situation where the NFS cannot be umounted and despite the stop timeout - leads to fencing(on-fail=fence). I thought that the Controller resource agent is stopping the HANA and the slave role should not be 'stopped' before that . Maybe my expectations are wrong ? Best Regards,Strahil Nikolov ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ -- Regards, Reid Wahl, RHCA Senior Software Maintenance Engineer, Red Hat CEE - Platform Support Delivery - ClusterHA ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Antw: Re: Antw: [EXT] Re: Order set troubles
>I also remember something about racing with dnsmasq, at which point I'dsay >that making cluster depend on availability of DNS is e-h-h-h unwise Not my choice... Or at least I would deploy bind/unbound caching servers in the same VLAN instead of dnsmasq.Also, Filesystem resource agent's read + write check is quite usefull. We got a crazy clusterized environment where on several occasions read-only FS did not cause a failover (it's not pacemaker) and I prefer to not be awaken by the oncall when this happens with the Scale-out I'm building (currently it's just the QA cluster, but soon coming the prod). The bad thing is this one is the first pacemaker in the environment and I need to make it completely 'killer' or I will be forced to use the old cluster solution which is crap (due to the implementation , not due to the product) double fencing mechanisms, fencing when FS is dead or HANA is having troubles, etc. Anyway, I am hoping that such kind of constraints will be more easier to implement in the future, as this one is quite complex and it will give me a hard time to explain it to the colleagues. Best Regards,Strahil Nikolov___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Which fence agent is needed for an Apache web server cluster?
My notes:- ilo ssh fence mechanism is crappy due to ilo itself, try to avoid if possible- fence_ipmi requires some tunings (-c flag) and also to disable power button from the system - triple 'sbd' is quite reliable.My previous company was using 'softdog' kernel module for a watchdog device and it never failed us. Yet, it's just a kernel module (no hardware required) and thus RH do not support such setup. If you decide to use 'sbd', disable vendor's system recovery solution (like HPE's ASR) as it will also tinker with the watchdog. Best Regards,Strahil Nikolov On Sat, Mar 27, 2021 at 22:15, Reid Wahl wrote: ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Antw: [EXT] Colocation per site ?
I didn't mean DC as a designated coordinator, but as a physical Datecenter location. Last time I checked, the node attributes for all nodes seemed the same.I will verify that tomorrow (Monday). Best Regards,Strahil Nikolov On Fri, Feb 19, 2021 at 16:51, Andrei Borzenkov wrote: On Fri, Feb 19, 2021 at 2:44 PM Strahil Nikolov wrote: > > > >Do you have a fixed relation between node >pairs and VIPs? I.e. must > >A/D always get VIP1, B/E - VIP2 etc? > > I have to verify it again, but generally speaking - yes , VIP1 is always on > nodeA/D (master), VIP2 on nodeB/E (worker1) , etc. > > I guess I can set negative constraints (-inf) -> VIP1 on node B/E + nodeC/F, > but the stuff with the 'same DC as master' is the tricky part. > I am not sure I understand what DC has to do with it. You have two scale-out SAP HANA instances, one is primary, another is secondary. If I understand correctly your requirements, your backup application needs to contact the primary instance which may failover to another site. You must be using some resource agent for it, to manage failover. The only one I am aware of is SAPHanaSR-ScaleOut. It already sets different node properties for primary and secondary sites. Just use them. If you use something else, just look at what attributes your RA sets. Otherwise you will be essentially duplicating your RA functionality because you will somehow need to find out which site is currently primary. There is no guarantee that pacemaker DC wil be on the same site as SAP HANA primary system. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Which fence agent is needed for an Apache web server cluster?
I had to tune the fence_ipmi recently on some older HPE blades. The default settings were working, but also returning some output about problems negotiating the cypher.As that output could make future version of the fence agent go wild, I tested several options untill no errors are reported. Maybe the cypher flag was different, but I think it was '-c'. If I'm wrong , the author of this thread can check the man page . Yes -> 'HandlePowerKey=ignore' . I have never expected ipmi to try graceful shutdown when I tell it to 'press and hold' or 'cold boot', yet I never checked the code of fence_ipmi. With triple sbd , I mean sbd with 3 block devices. Best Regards,Strahil Nikolov On Sat, Mar 27, 2021 at 23:15, Reid Wahl wrote: On Saturday, March 27, 2021, Strahil Nikolov wrote: > My notes: > - ilo ssh fence mechanism is crappy due to ilo itself, try to avoid if > possible It has been unreliable in my experience. > - fence_ipmi requires some tunings (-c flag) and also to disable power button > from the system I've rarely, perhaps never, seen a customer have to tune the -c flag. By disabling the power button, do you mean setting HandlePowerKey=ignore in logind.conf? That's not specific to fence_ipmilan, to be clear. > - triple 'sbd' is quite reliable.My previous company was using 'softdog' > kernel module for a watchdog device and it never failed us. Yet, it's just a > kernel module (no hardware required) and thus RH do not support such setup. What do you mean by triple sbd? Correct that RH doesn't support using softdog as an sbd watchdog device. It was determined that it's not reliable in all situations. It's probably fine much of the time, and I'm glad you had a smooth experience with it. > > On Sat, Mar 27, 2021 at 22:15, Reid Wahl > wrote: > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ > -- Regards, Reid Wahl, RHCA Senior Software Maintenance Engineer, Red Hat CEE - Platform Support Delivery - ClusterHA ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Live migration possible with KSM ?
Keep in mind that KSM is highly cpu intensive and is most suitable for same type of VMs,so similar memory pages will be merged until a change happen (and that change is allocated elsewhere). In oVirt migration is possible with KSM actively working, so it should work with pacemaker. I doubt that KSM would be a problem... most probably performance would not be optimal. Best Regards,Strahil Nikolov On Tue, Mar 30, 2021 at 19:47, Andrei Borzenkov wrote: On 30.03.2021 18:16, Lentes, Bernd wrote: > Hi, > > currently i'm reading "Mastering KVM Virtualization", published by Packt > Publishing, a book i can really recommend. > There are some proposals for tuning guests. One is KSM (kernel samepage > merging), which sounds quite interesting. > Especially in a system with lots of virtual machines with the same OS this > could lead to significant merory saving. > I'd like to test, but i don't know if KSM maybe prevents live migration in a > pacemaker cluster. I do not think pacemaker cares or is aware about KSM. It just tells resource agent to perform migration; what happens is entirely up to resource agent. If you can migrate without pacemaker you can also migrate with pacemaker. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
[ClusterLabs] maximum token value (knet)
Hello all, I'm building a test cluster on RHEL8.2 and I have noticed that the cluster fails to assemble ( nodes stay inquorate as if the network is not working) if I set the token at 3 or more (30s+). What is the maximum token value with knet ?On SLES12 (I think it was corosync 1) , I used to set the token/consensus with far greater values on some of our clusters. Best Regards,Strahil Nikolov___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] maximum token value (knet)
Interesting... Yet, this doesn't explain why token of 3 causes the nodes to never assemble a cluster (waiting for half an hour, using wait_for_all=1) , while setting it to 29000 works like a charm. Thankfully we got RH subsciption, so RH devs will provide more detailed output on the issue. I was hoping that I missed in the documentation about the maximum token size... Best Regards, Strahil Nikolov В четвъртък, 11 март 2021 г., 19:12:58 ч. Гринуич+2, Jan Friesse написа: Strahil, > Hello all, > I'm building a test cluster on RHEL8.2 and I have noticed that the cluster > fails to assemble ( nodes stay inquorate as if the network is not working) if > I set the token at 3 or more (30s+). Knet waits for enough pong replies for other nodes before it marks them as alive and starts sending/receiving packets from them. By default it needs to receive 2 pongs and ping is sent 4 times in token timeout so it means 15 sec until node is considered up for 30 sec token timeout. > What is the maximum token value with knet ?On SLES12 (I think it was > corosync 1) , I used to set the token/consensus with far greater values on > some of our clusters. I'm really not aware about any arbitrary limits. > Best Regards,Strahil Nikolov > Regards, Honza > > > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ > ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Q: callback hooks for sbd?
What about creating a custom fencing agent that does what you need and always reporting that it failed ? Using fencing topology and a custom fencing script as the first stonith device (that always fails) , you will be able to execute some commands (or at least try to do that) before the next fencing mechanism kicks in. Best Regards, Strahil Nikolov В четвъртък, 11 март 2021 г., 19:16:04 ч. Гринуич+2, Klaus Wenninger написа: On 3/11/21 12:30 PM, Ulrich Windl wrote: > Hi! > > I wonder: Is it possible to register some callback to sbd that is called > whenever a fencing operation is to be executed? > I would like to run some command on the node that is going to be fenced. Don't know of anything that exists you could use for that purpose. Because of the base-principle it is anyway never gonna be reliable that the command is executed. When the watchdog is taking down the node you can't execute anything before of course. And we couldn't wait for your callback to return of course as we don't have a reliable channel back to the fencing party and thus have to execute with a strict timeout. What you can of course always do is put some pseudo-fence-agent on a topology level with the poison-pill agent that tries to contact the to be fenced party first to execute your command. What I could imagine is to trigger execution of a customizable command(s) in parallel with the sequence of reboot / sysrq or as a replacement of that. We would still have the watchdog to take care of a strict timeout. I suppose idea is trying to do something graceful before being taken down by the watchdog. Or were you thinking along those lines: https://bugzilla.redhat.com/show_bug.cgi?id=1869728 Regards, Klaus > > Regards, > Ulrich > > > > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ > ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] maximum token value (knet)
I will try to get into the details on monday, when I have access to the cluster again.I guess the /var/log/cluster/corosync.log and /etc/corosync/corosync.conf are the most interesting. So far, I have 6 node cluster with separate VLANs for HANA replication, prod and backup.Initially, I used pcs to create the corosync.conf with 2 IPs per node, token 4, consensus 48000 and wait_for_all=1.Later I have expanded the cluster to 3 links and added qnet to the setup (only after I made it run (token 29000) ), so I'm ruling it out.I updated the cluster nodes from RHEL 8.1 to 8.2 , removed the consensus and enabled debug. As knet is using udp by default, and because the problem is hitting me both in udp (default settings) and sctp - the problem is not in the protocol. I've also enabled pacemaker blackbox, although I doubt that has any effect on corosync. How can I enable trace logs for corosync only ? Best Regards,Strahil Nikolov On Fri, Mar 12, 2021 at 17:01, Jan Friesse wrote: Strahil, > Interesting... > Yet, this doesn't explain why token of 3 causes the nodes to never > assemble a cluster (waiting for half an hour, using wait_for_all=1) , while > setting it to 29000 works like a charm. Definitively. Could you please provide a bit more info about your setup (config/logs/how many nodes cluster has/...)? Because I've just briefly tested two nodes setup with 30 sec token timeout and it was working perfectly fine. > > Thankfully we got RH subsciption, so RH devs will provide more detailed > output on the issue. As Jehan correctly noted if it would really get to RH devs it would probably get to me ;) But before that GSS will take care of checking configs/hw/logs/... and they are really good in finding problems with setup/hw/... > > I was hoping that I missed in the documentation about the maximum token > size... Nope. No matter what, if you can send config/logs/... we may try to find out what is root of the problem here on ML or you can really try GSS, but as Jehan told, it would be nice if you can post result so other people (me included) knows what was the main problem. Thanks and regards, Honza > > Best Regards, > Strahil Nikolov > > > > > > > В четвъртък, 11 март 2021 г., 19:12:58 ч. Гринуич+2, Jan Friesse > написа: > > > > > > Strahil, >> Hello all, >> I'm building a test cluster on RHEL8.2 and I have noticed that the cluster >> fails to assemble ( nodes stay inquorate as if the network is not working) >> if I set the token at 3 or more (30s+). > > Knet waits for enough pong replies for other nodes before it marks them > as alive and starts sending/receiving packets from them. By default it > needs to receive 2 pongs and ping is sent 4 times in token timeout so it > means 15 sec until node is considered up for 30 sec token timeout. > >> What is the maximum token value with knet ?On SLES12 (I think it was >> corosync 1) , I used to set the token/consensus with far greater values on >> some of our clusters. > > I'm really not aware about any arbitrary limits. > > >> Best Regards,Strahil Nikolov >> > > Regards, > > Honza > >> >> >> ___ >> Manage your subscription: >> https://lists.clusterlabs.org/mailman/listinfo/users >> >> ClusterLabs home: https://www.clusterlabs.org/ > >> > ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Corosync - qdevice not voting
Is there any reason to use lms mode for the qdevice ? Best Regards, Strahil Nikolov ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Corosync - qdevice not voting
If firewalld is available, just try with 'firewall-cmd --panic-on' (or something like that). Best Regards,Strahil Nikolov On Fri, Mar 19, 2021 at 12:50, Marcelo Terres wrote: ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Our 2-Node Cluster with a Separate Qdevice Went Down Anyway?
When you change the token, you might consider adjusting the consensus timeout (see man corosync.conf). Best Regards,Strahil Nikolov___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Our 2-Node Cluster with a Separate Qdevice Went Down Anyway?
As this is in Asure and they support shared disks , I think that a simple SBD could solve the stonith case. Best Regards,Strahil Nikolov___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Antw: [EXT] Re: how to setup single node cluster
Maybe booth can take care when it dies and powers up the resource in the DR. Best Regards,Strahil Nikolov On Thu, Apr 8, 2021 at 10:28, Ulrich Windl wrote: >>> Reid Wahl schrieb am 08.04.2021 um 08:32 in Nachricht : > On Wed, Apr 7, 2021 at 11:27 PM d tbsky wrote: > >> Reid Wahl >> > I don't think we do require fencing for single-node clusters. (Anyone at >> Red Hat, feel free to comment.) I vaguely recall an internal mailing list >> or IRC conversation where we discussed this months ago, but I can't find it >> now. I've also checked our support policies documentation, and it's not >> mentioned in the "cluster size" doc or the "fencing" doc. >> >> since the cluster is 100% alive or 100% dead with single node, I >> think fencing/quorum is not required. I am just curious what is the >> usage case. since RedHat supports it, it must be useful in real >> scenario. >> > > Disaster recovery is the main use case we had in mind. See the RHEL 8.2 > release notes: > - > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/htm > l/8.2_release_notes/rhel-8-2-0-release#enhancement_high-availability-and-clus > ters I wonder: How does desaster recovery look like if a plane crashed into your single-node of the cluster? (Without a cluster you would have some off-site backup, and probably some spare hardware off-site, too) HA in that case is really interesting, as setting up the new hardware has to be done manually... > > I thought I also remembered some other use case involving MS SQL, but I > can't find anything about it so I might be remembering incorrectly. > > > -- > Regards, > > Reid Wahl, RHCA > Senior Software Maintenance Engineer, Red Hat > CEE - Platform Support Delivery - ClusterHA ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] how to setup single node cluster
I always though that the setup is the same, just the node count is only one. I guess you need pcs, corosync + pacemaker.If RH is going to support it, they will require fencing. Most probably sbd or ipmi are the best candidates. Best Regards,Strahil Nikolov On Thu, Apr 8, 2021 at 6:52, d tbsky wrote: Hi: I found RHEL 8.2 support single node cluster now. but I didn't find further document to explain the concept. RHEL 8.2 also support "disaster recovery cluster". so I think maybe a single node disaster recovery cluster is not bad. I think corosync is still necessary under single node cluster. or is there other new style of configuration? thanks for help! ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Node fenced for unknown reason
IPMI fencing on some vendors will first try graceful shutdown and only then it will use ungraceful. Disabling power button is also described in https://access.redhat.com/solutions/1578823 Best Regards,Strahil Nikolov___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Antw: [EXT] best practice for scripting
By the way , how do you monitor your pacemaker clusters ?We are using Nagios and I found only 'check_crm' but it looks like it was made for crmsh and most probably won't work with pcs without modifications. Best Regards,Strahil Nikolov On Tue, Apr 13, 2021 at 10:57, d tbsky wrote: Ulrich Windl > "Coming from SUSE" I'm using crm shell for most tasks, and I think the syntax > is quite good. good for SUSE! unfortunately RHEL didn't include the utility... ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Single-node automated startup question
What about a small form factor device to serve as a quorum maker ? Best Regards,Strahil Nikolov___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] PAF resource agent & stickiness - ?
Better check for a location constraint created via 'pcs resource move'!pcs constraint location --full | grep cli Best Regards,Strahil Nikolov On Sat, Apr 10, 2021 at 18:19, Jehan-Guillaume de Rorthais wrote: Le 10 avril 2021 14:22:34 GMT+02:00, lejeczek a écrit : >Hi guys. > >Any users perhaps experts on PAF agent if happen to read >this - a question - with pretty regular 3-node cluster when >node on which "master" runs goes down then cluster/agent >successfully moves 'master' to a next node. How did you take down the node ? And what are the scores at this time ? > but.. >When node which held master, which was off/down, comes back >up then master/agent reshuffles PAF resources again and >moves 'master' back that last 'master' node and... >slapping a 'stickiness' on the resource does not seem to >have any affect, cluster/agent still persists on moving >'master' to that same node. Could you share your cluster setup ? Regards, ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Single-node automated startup question
If it's a VM or container - it should be on a third location. Using a VM hosted on one of the nodes is like giving that node more votes in a two-node cluster. Cheap 3rd node for quorum makes more sense to me. Best Regards,Strahil Nikolov On Wed, Apr 14, 2021 at 21:19, Antony Stone wrote: On Wednesday 14 April 2021 at 19:33:39, Strahil Nikolov wrote: > What about a small form factor device to serve as a quorum maker ? > Best Regards,Strahil Nikolov If you're going to take that approach, why not a virtual machine or two, hosted inside the physical machine which is your single real node? I'm not necessarily advocating this method of achieving quorum, but it's probably an idea worth considering for specific situations. Antony. -- Someone has stolen all the toilets from New Scotland Yard. Police say they have absolutely nothing to go on. Please reply to the list; please *don't* CC me. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Antw: [EXT] weird xml snippet in "crm configure show"
WARNING: cib-bootstrap-options: unknown attribute 'no-quirum-policy' That looks like a typo. Best Regards,Strahil Nikolov On Fri, Feb 12, 2021 at 12:30, Lentes, Bernd wrote: - On Feb 12, 2021, at 11:18 AM, Ulrich Windl ulrich.wi...@rz.uni-regensburg.de wrote: > > What is the output of "crm configure verify"? ha-idg-1:~ # crm configure verify WARNING: cib-bootstrap-options: unknown attribute 'no-quirum-policy' WARNING: clvmd: specified timeout 20 for monitor is smaller than the advised 90s WARNING: dlm: specified timeout 80 for stop is smaller than the advised 100 WARNING: dlm: specified timeout 80 for start is smaller than the advised 90 WARNING: fs_ocfs2: specified timeout 20 for monitor is smaller than the advised 40s WARNING: fs_test_ocfs2: specified timeout 20 for monitor is smaller than the advised 40s WARNING: gfs2_share: specified timeout 20 for monitor is smaller than the advised 40s WARNING: gfs2_snap: specified timeout 20 for monitor is smaller than the advised 40s WARNING: vm_amok: specified timeout 25 for monitor is smaller than the advised 30s WARNING: vm_crispor: specified timeout 25 for monitor is smaller than the advised 30s WARNING: vm_crispor-server: specified timeout 25 for monitor is smaller than the advised 30s WARNING: vm_dietrich: specified timeout 25 for monitor is smaller than the advised 30s WARNING: vm_documents-oo: specified timeout 25 for monitor is smaller than the advised 30s WARNING: vm_geneious: specified timeout 25 for monitor is smaller than the advised 30s WARNING: vm_geneious-license: specified timeout 25 for monitor is smaller than the advised 30s WARNING: vm_geneious-license-mcd: specified timeout 25 for monitor is smaller than the advised 30s WARNING: vm_genetrap: specified timeout 25 for monitor is smaller than the advised 30s WARNING: vm_greensql: specified timeout 25 for monitor is smaller than the advised 30s WARNING: vm_idcc_devel: specified timeout 25 for monitor is smaller than the advised 30s WARNING: vm_mausdb: specified timeout 25 for monitor is smaller than the advised 30s WARNING: vm_mouseidgenes: specified timeout 25 for monitor is smaller than the advised 30s WARNING: vm_nextcloud: specified timeout 25 for monitor is smaller than the advised 30s WARNING: vm_pathway: specified timeout 25 for monitor is smaller than the advised 30s WARNING: vm_photoshop: specified timeout 25 for monitor is smaller than the advised 30s WARNING: vm_seneca: specified timeout 25 for monitor is smaller than the advised 30s WARNING: vm_severin: specified timeout 25 for monitor is smaller than the advised 30s WARNING: vm_sim: specified timeout 25 for monitor is smaller than the advised 30s WARNING: vm_snipanalysis: specified timeout 25 for monitor is smaller than the advised 30s WARNING: vm_ssh: specified timeout 25 for monitor is smaller than the advised 30s Bernd Helmholtz Zentrum München Helmholtz Zentrum Muenchen Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH) Ingolstaedter Landstr. 1 85764 Neuherberg www.helmholtz-muenchen.de Aufsichtsratsvorsitzende: MinDir.in Prof. Dr. Veronika von Messling Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Kerstin Guenther Registergericht: Amtsgericht Muenchen HRB 6466 USt-IdNr: DE 129521671 ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Feedback wanted: using the systemd message catalog
Hi Ken, I'm using a lot of 'journalctl -x' , especially when something fails and I really like the idea of having that.Yet, this will require some efforts and also a modification in the cluster's log saying something like 'The cluster is mis-configured - shutting down and staying down. For more details use journalctl -xe'. Isn't it easier to just provide more details in the logs than integrating that feature ? Best Regards,Strahil Nikolov On Tue, Feb 16, 2021 at 21:48, Ken Gaillot wrote: Hi all, The systemd journal has a feature called the message catalog, which allows "journalctl -x" to show extended troubleshooting hints along with log messages. Apparently a similar feature is familiar to many Windows administrators. So far, I've only found systemd itself that uses the feature. As an example, if the usual log message is a one-liner about httpd dumping core, with journalctl -x it would look like: -- fc2e22bc6ee647b6b90729ab34a250b1 Subject: Process 1234 (httpd) dumped core Defined-By: systemd Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel Documentation: man:core(5) Documentation: http://www.freedesktop.org/wiki/Software/systemd/catalog/fc2e22bc6ee647b6b90729ab34a250b1 Process 1234 (httpd) crashed and dumped core. This usually indicates a programming error in the crashing program and should be reported to its vendor as a bug. There is currently a proposed pull request ( https://github.com/ClusterLabs/libqb/pull/433 ) to add support for this to libqb's logging API, which the rest of the cluster software could use. As an example, if someone forcibly uploads a CIB with syntax errors, Pacemaker will currently log "The cluster is mis-configured - shutting down and staying down" and exit. We could make it so that the journalctl -x entry tells the user how to use crm_verify to figure out what's wrong, how to use cibadmin to correct any errors, and a link to Pacemaker Explained as a configuration reference. This would involve considerable work for developers, so I'm curious how many users would find this useful and would use it. Especially if most of the time you used journalctl -x, there was no extended information, but occasionally there was. -- Ken Gaillot ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
[ClusterLabs] Colocation per site ?
Hello All, I'm currently in a process of building SAP HANA Scale-out cluster and the HANA team has asked that all nodes on the active instance should have one IP for backup purposes. Yet, I'm not sure how to setup the constraints (if it is possible at all) so all IPs will follow the master resource and yet run on a separate host. I know that I can set location constraints between each pair of nodes , but the logic identifying the master's DC is eluding me. Best Regards,Strahil Nikolov___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Antw: [EXT] Feedback wanted: using the systemd message catalog
Hi Ulrich, actually you can suppress them. Best Regards,Strahil Nikolov On Wed, Feb 17, 2021 at 13:04, Ulrich Windl wrote: Hi Ken, personally I think systemd is already logging too much, and I don't think that adding instructions to many log messages is actually helpful (It could be done as separate log message (maybe at severity info) already). In Windows I see the problem that it's very hard to find real problems as the logs are flooded with all kinds of stuff. Example (being rather useless): Feb 17 12:01:01 h03 systemd[1]: Started Session 41667 of user root. -- Subject: Unit session-41667.scope has finished start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit session-41667.scope has finished starting up. -- -- The start-up result is done. So is there any useful info in those extra lines? Regards, Ulrich >>> Ken Gaillot schrieb am 16.02.2021 um 20:47 in Nachricht <8a2c91037be49816055a717583c6c505c996f5e7.ca...@redhat.com>: > Hi all, > > The systemd journal has a feature called the message catalog, which > allows "journalctl ‑x" to show extended troubleshooting hints along > with log messages. Apparently a similar feature is familiar to many > Windows administrators. > > So far, I've only found systemd itself that uses the feature. As an > example, if the usual log message is a one‑liner about httpd dumping > core, with journalctl ‑x it would look like: > > ‑‑ fc2e22bc6ee647b6b90729ab34a250b1 > Subject: Process 1234 (httpd) dumped core > Defined‑By: systemd > Support: http://lists.freedesktop.org/mailman/listinfo/systemd‑devel > Documentation: man:core(5) > Documentation: > http://www.freedesktop.org/wiki/Software/systemd/catalog/fc2e22bc6ee647b6b907 > 29ab34a250b1 > > Process 1234 (httpd) crashed and dumped core. > > This usually indicates a programming error in the crashing program and > should be reported to its vendor as a bug. > > There is currently a proposed pull request ( > https://github.com/ClusterLabs/libqb/pull/433 ) to add support for this > to libqb's logging API, which the rest of the cluster software could > use. > > As an example, if someone forcibly uploads a CIB with syntax errors, > Pacemaker will currently log "The cluster is mis‑configured ‑ shutting > down and staying down" and exit. We could make it so that the > journalctl ‑x entry tells the user how to use crm_verify to figure out > what's wrong, how to use cibadmin to correct any errors, and a link to > Pacemaker Explained as a configuration reference. > > This would involve considerable work for developers, so I'm curious how > many users would find this useful and would use it. Especially if most > of the time you used journalctl ‑x, there was no extended information, > but occasionally there was. > ‑‑ > Ken Gaillot > > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Antw: [EXT] Colocation per site ?
DC1:- nodeA- nodeB- nodeC DC2:- nodeD- nodeE- nodeF DC3:- majority maker I will have 3 VIPs:VIP1VIP2VIP3 I will have to setup the cluster to:1. Find where is the master HANA resource2. Place VIP1 on nodeA/nodeD3. Place VIP2 on nodeB/nodeE4. Place VIP3 on nodeC/nodeF End Goal:Master on nodeAVIP1 on nodeAVIP2 on nodeBVIP3 on nodeC Master on nodeDVIP1 on nodeDVIP2 on nodeEVIP3 on nodeF Master down:All VIPs down I think that Ken has mentioned a possible solution, bit I have to check it out. Best Regards,Strahil Nikolov On Thu, Feb 18, 2021 at 9:40, Ulrich Windl wrote: >>> Strahil Nikolov schrieb am 17.02.2021 um 17:46 in Nachricht <2134183555.2122291.1613580414...@mail.yahoo.com>: > Hello All, > I'm currently in a process of building SAP HANA Scale-out cluster and the > HANA team has asked that all nodes on the active instance should have one IP > for backup purposes. > Yet, I'm not sure how to setup the constraints (if it is possible at all) so > all IPs will follow the master resource and yet run on a separate host. I think you should explain the scenario a bit better to undertsand what you want. > I know that I can set location constraints between each pair of nodes , but > the logic identifying the master's DC is eluding me. > Best Regards,Strahil Nikolov ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Antw: [EXT] Colocation per site ?
>Do you have a fixed relation between node >pairs and VIPs? I.e. must >A/D always get VIP1, B/E - VIP2 etc? I have to verify it again, but generally speaking - yes , VIP1 is always on nodeA/D (master), VIP2 on nodeB/E (worker1) , etc. I guess I can set negative constraints (-inf) -> VIP1 on node B/E + nodeC/F, but the stuff with the 'same DC as master' is the tricky part. Best Regards,Strahil Nikolov___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] unexpected fenced node and promotion of the new master PAF - postgres
If you experience multiple outages, you should consider enabling the kdump feature of sbd. It will increase the takeover time, but might provide valuable info. Best Regards,Strahil Nikolov On Wed, Jul 14, 2021 at 15:12, Klaus Wenninger wrote: ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] QDevice vs 3rd host for majority node quorum
In some cases the third location has a single IP and it makes sense to use it as QDevice. If it has multiple network connections to that location - use a full blown node . Best Regards,Strahil Nikolov On Tue, Jul 13, 2021 at 20:44, Andrei Borzenkov wrote: On 13.07.2021 19:52, Gerry R Sommerville wrote: > Hello everyone, > I am currently comparing using QDevice vs adding a 3rd host to my > even-number-node cluster and I am wondering about the details concerning > network > communication. > For example, say my cluster is utilizing multiple heartbeat rings. Would the > QDevice take into account and use the IPs specified in the different rings? > Or No. > does it only use the one specified under the quorum directive for QDevice? Yes. Remote device is unrelated to corosync rings. Qdevice receives information of current cluster membership from all nodes (point of view), computes partitions and selects partition that will remain quorate. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Question about automating cluster unfencing.
You can setup the system in such case that on fabric fence, the node is rebooted which will allow it to 'unfence' itself afterwards. For details check https://access.redhat.com/solutions/3367151 or https://access.redhat.com/node/65187 (You may use RH developer subscription in order to acess it). It seems that fence_mpath has watchdog integration after a certain version, while you can still use /usr/share/cluster/fence_mpath_check (via watchdog service and supported watchdog device). Even if you don't have a proper watchdog device, you can use the 'softdog' module as the system is fenced via SAN and even if not rebooted , there is no risk . Best Regards,Strahil Nikolov Sent from Yahoo Mail on Android On Sat, Aug 28, 2021 at 10:14, Andrei Borzenkov wrote: On Fri, Aug 27, 2021 at 8:11 PM Gerry R Sommerville wrote: > > Hey all, > > From what I see in the documentation for fabric fencing, Pacemaker requires > an administrator to login to the node to manually start and unfence the node > after some failure. > >https://clusterlabs.org/pacemaker/doc/deprecated/en-US/Pacemaker/2.0/html/Pacemaker_Explained/s-unfencing.html > This is about fabric (or resource) fencing. In this case node is cut off from some vital resources but remains up and running. In this case someone indeed needs to intervene manually. > The concern I have is if there is an intermittent network issues, a node may > get fenced and we have to wait for someone to log into the cluster and bring > the node back online. Meanwhile the network issue may have resolved itself > shortly after the node was fenced. > > I wonder if there are any configurations or popular solutions that people use > to automatically unfence nodes and have them rejoin the cluster? > Most people use stonith (or node fencing) and affected node is rebooted. As long as pacemaker is configured to start automatically and network connectivity is restored after reboot node will join custer automatically. I think that in case of fabric fencing node is undefnced automatically when it reboots and attempts to join cluster (hopefully someone may chime in here). I am not sure what happens if node is not rebooted but connectivity is restored. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Qemu VM resources - cannot acquire state change lock
Are gou using sharding for glusterfs ? I would put libvirt service and glusterfs service in a systemd dependency as your libvirt relies on gluster being available. Also, check if you got 'backup-volfile-servers' mount option if using FUSE.With libgfapi, I got no clue how to configure that. Your setup looksfar close to the oVirt project ... (just mentioning). Best Regards,Strahil Nikolov Sent from Yahoo Mail on Android On Sat, Aug 28, 2021 at 13:33, lejeczek via Users wrote: On 26/08/2021 10:35, Klaus Wenninger wrote: > > > On Thu, Aug 26, 2021 at 11:13 AM lejeczek via Users > mailto:users@clusterlabs.org>> wrote: > > Hi guys. > > I sometimes - I think I know when in terms of any > pattern - > get resources stuck on one node (two-node cluster) with > these in libvirtd's logs: > ... > Cannot start job (query, none, none) for domain > c8kubermaster1; current job is (modify, none, none) > owned by > (192261 qemuProcessReconnect, 0 , 0 > (flags=0x0)) for (1093s, 0s, 0s) > Cannot start job (query, none, none) for domain > ubuntu-tor; > current job is (modify, none, none) owned by (192263 > qemuProcessReconnect, 0 , 0 (flags=0x0)) for > (1093s, 0s, 0s) > Timed out during operation: cannot acquire state > change lock > (held by monitor=qemuProcessReconnect) > Timed out during operation: cannot acquire state > change lock > (held by monitor=qemuProcessReconnect) > ... > > when this happens, and if the resourec is meant to be the > other node, I have to to disable the resource first, then > the node on which resources are stuck will shutdown > the VM > and then I have to re-enable that resource so it > would, only > then, start on that other, the second node. > > I think this problem occurs if I restart 'libvirtd' > via systemd. > > Any thoughts on this guys? > > > What are the logs on the pacemaker-side saying? > An issue with migration? > > Klaus I'll have to try to tidy up the "protocol" with my stuff so I could call it all reproducible, at the moment if only feels that way, as reproducible. I'm on CentOS Stream and have 2-node cluster, with KVM resources, with same glusterfs cluster 2-node. (all psychically is two machines) 1) I power down one node in orderly manner and the other node is last-man-standing. 2) after a while (not sure if time period is also a key here) I brought up that first node. 3) the last man-standing-node libvirtd becomes irresponsive (don't know yet, if that is only after the first node came back up) to virt cmd and to probably everything else, pacameker log says: ... pacemaker-controld[2730]: error: Result of probe operation for c8kubernode2 on dzien: Timed Out ... and libvirtd log does not say anything really (with default debug levels) 4) if glusterfs might play any role? Healing of the volume(s) is finished at this time, completed successfully. This the moment where I would manually 'systemd restart libvirtd' that irresponsive node(was last-man-standing) and got original error messages. There is plenty of room for anybody to make guesses, obvious. Is it 'libvirtd' going haywire because glusterfs volume is in an unhealthy state and needs healing? Is it pacemaker last-man-standing which makes 'libvirtd' go haywire? etc... I can add much concrete stuff at this moment but will appreciate any thoughts you want to share. thanks, L > many thanks, L. > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > <https://lists.clusterlabs.org/mailman/listinfo/users> > > ClusterLabs home: https://www.clusterlabs.org/ > <https://www.clusterlabs.org/> > ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
