Re: [ClusterLabs] Antw: [EXT] DRBD ms resource keeps getting demoted
I guess I missed the OS version, otherwise I would have powered up my 3 node CentOS 7 test cluster.I will check later the settings on my CentOS 7 cluster.Last time I checked it, the drbd was running fine.Best Regards,Strahil NikolovOn Jan 20, 2021 04:41, Stuart Massey wrote:Strahil,That is very kind of you, thanks. I see that in your (feature set 3.4.1) cib, drbd is in a with some meta_attributes and operations having to do with promotion, while in our (feature set 3.0.14) cib, drbd is in a which does not have those (maybe since promotion is implicit).Our cluster has been working quite well for some time, too. I wonder what would happen if you could hang the os in one of your nodes? If a VM, maybe the constrained secondary could be starved by setting disk IOPs to something really low. Of course, you are using different versions of just about everything, as we're on centos7.Regards,StuartOn Tue, Jan 19, 2021 at 6:20 PM Strahil Nikolov <hunter86_bg@yahoo.com> wrote:I have just built a test cluster (centOS 8.3) for testing DRBD and it works quite fine.Actually I followed my notes from https://forums.centos.org/viewtopic.php?t=65539 with the exception of point 8 due to the "promotable" stuff.I'm attaching the output of 'pcs cluster cib file' and I hope it helps you fix your issue.Best Regards,Strahil NikolovВ 09:32 -0500 на 19.01.2021 (вт), Stuart Massey написа:Ulrich,Thank you for that observation. We share that concern.We have 4 ea 1G nics active, bonded in pairs. One bonded pair serves the "public" (to the intranet) IPs, and the other bonded pair is private to the cluster, used for drbd replication. HA will, I hope, be using the "public" IP, since that is the route to the IP addresses resolved for the host names; that will certainly be the only route to the quorum device. I can say that this cluster has run reasonably well for quite some time with this configuration prior to the recently developed hardware issues on one of the nodes.Regards,StuartOn Tue, Jan 19, 2021 at 2:49 AM Ulrich Windl <Ulrich.Windl@rz.uni-regensburg.de> wrote:>>> Stuart Massey <djangoschef@gmail.com> schrieb am 19.01.2021 um 04:46 in Nachricht <CABQ68NQuTyYXcYgwcUpg5TxxaJjwhSp+c6GCOKfOwGyRQSAAjQ@mail.gmail.com>: > So, we have a 2-node cluster with a quorum device. One of the nodes (node1) > is having some trouble, so we have added constraints to prevent any > resources migrating to it, but have not put it in standby, so that drbd in > secondary on that node stays in sync. The problems it is having lead to OS > lockups that eventually resolve themselves - but that causes it to be > temporarily dropped from the cluster by the current master (node2). > Sometimes when node1 rejoins, then node2 will demote the drbd ms resource. > That causes all resources that depend on it to be stopped, leading to a > service outage. They are then restarted on node2, since they can't run on > node1 (due to constraints). > We are having a hard time understanding why this happens. It seems like > there may be some sort of DC contention happening. Does anyone have any > idea how we might prevent this from happening? I think if you are routing high-volume DRBD traffic throuch "the same pipe" as the cluster communication, cluster communication may fail if the pipe is satiated. I'm not happy with that, but it seems to be that way. Maybe running a combination of iftop and iotop could help you understand what's going on... Regards, Ulrich > Selected messages (de-identified) from pacemaker.log that illustrate > suspicion re DC confusion are below. The update_dc and > abort_transition_graph re deletion of lrm seem to always precede the > demotion, and a demotion seems to always follow (when not already demoted). > > Jan 18 16:52:17 [21938] node02.example.com crmd: info: > do_dc_takeover: Taking over DC status for this partition > Jan 18 16:52:17 [21938] node02.example.com crmd: info: update_dc: > Set DC to node02.example.com (3.0.14) > Jan 18 16:52:17 [21938] node02.example.com crmd: info: > abort_transition_graph: Transition aborted by deletion of > lrm[@id='1']: Resource state removal | cib=0.89.327 > source=abort_unless_down:357 > path=/cib/status/node_state[@id='1']/lrm[@id='1'] complete=true > Jan 18 16:52:19 [21937] node02.example.com pengine: info: > master_color: ms_drbd_ourApp: Promoted 0 instances of a possible 1 to > master > Jan 18 16:52:19 [21937] node02.example.com pengine: notice: LogAction: > * Demote drbd_ourApp:1 ( Master -> Slave > node02.example.com ) ___ Manage your subscription: https://lists.clusterlabs.org/mailm
Re: [ClusterLabs] How to unfence without reboot (fence_mpath)
Hello all, I think I found the problem. On the fenced node after a restart of the cluster stack , I saw the following: controld(dlm)[13025]: ERROR: Uncontrolled lockspace exists, system must reboot. Executing suicide fencing I was so focused on the DC logs, so I missed it. I guess with HALVM , there will be no need to reboot - yet when dlm/clvmd were interrupted , the only path will be to reboot. Best Regards, Strahil Nikolov В понеделник, 17 февруари 2020 г., 15:36:39 ч. Гринуич+2, Ondrej написа: Hello Strahil, On 2/17/20 3:39 PM, Strahil Nikolov wrote: > Hello Ondrej, > > thanks for your reply. I really appreciate that. > > I have picked fence_multipath as I'm preparing for my EX436 and I can't know > what agent will be useful on the exam. > Also ,according to https://access.redhat.com/solutions/3201072 , there could > be a race condition with fence_scsi. I believe that exam is about testing knowledge in configuration and not testing knowledge in knowing which race condition bugs are present and how to handle them :) If you have access to learning materials for EX436 exam I would recommend trying those ones out - they have labs and comprehensive review exercises that are useful in preparation for exam. > So, I've checked the cluster when fencing and the node immediately goes > offline. > Last messages from pacemaker are: > > Feb 17 08:21:57 node1.localdomain stonith-ng[23808]: notice: Client > stonith_admin.controld.23888.b57ceee7 wants to fence (reboot) > 'node1.localdomain' with device '(any)' > Feb 17 08:21:57 node1.localdomain stonith-ng[23808]: notice: Requesting > peer fencing (reboot) of node1.localdomain > Feb 17 08:21:57 node1.localdomain stonith-ng[23808]: notice: FENCING can > fence (reboot) node1.localdomain (aka. '1'): static-list > Feb 17 08:21:58 node1.localdomain stonith-ng[23808]: notice: Operation > reboot of node1.localdomain by node2.localdomain for stonith_admin.controld.23888@node1.localdomain.ede38ffb: OK - This part looks OK - meaning the fencing looks like a success. > Feb 17 08:21:58 node1.localdomain crmd[23812]: crit: We were allegedly > just fenced by node2.localdomain for node1.localdomai - this is also normal as node just announces that it was fenced by other node > > > Which for me means - node1 just got fenced again. Actually fencing works ,as > I/O is immediately blocked and the reservation is removed. > > I've used https://access.redhat.com/solutions/2766611 to setup the > fence_mpath , but I could have messed up something. - note related to exam: you will not have Internet on exam, so I would expect that you would have to configure something that would not require access to this (and as Dan Swartzendruber pointed out in other email - we cannot* even see RH links without account) * you can get free developers account to read them, but ideally that should be not needed and is certainly inconvenient for wide public audience > > Cluster config is: > [root@node3 ~]# pcs config show > Cluster Name: HACLUSTER2 > Corosync Nodes: > node1.localdomain node2.localdomain node3.localdomain > Pacemaker Nodes: > node1.localdomain node2.localdomain node3.localdomain > > Resources: > Clone: dlm-clone > Meta Attrs: interleave=true ordered=true > Resource: dlm (class=ocf provider=pacemaker type=controld) > Operations: monitor interval=30s on-fail=fence (dlm-monitor-interval-30s) > start interval=0s timeout=90 (dlm-start-interval-0s) > stop interval=0s timeout=100 (dlm-stop-interval-0s) > Clone: clvmd-clone > Meta Attrs: interleave=true ordered=true > Resource: clvmd (class=ocf provider=heartbeat type=clvm) > Operations: monitor interval=30s on-fail=fence >(clvmd-monitor-interval-30s) > start interval=0s timeout=90s (clvmd-start-interval-0s) > stop interval=0s timeout=90s (clvmd-stop-interval-0s) > Clone: TESTGFS2-clone > Meta Attrs: interleave=true > Resource: TESTGFS2 (class=ocf provider=heartbeat type=Filesystem) > Attributes: device=/dev/TEST/gfs2 directory=/GFS2 fstype=gfs2 >options=noatime run_fsck=no > Operations: monitor interval=15s on-fail=fence OCF_CHECK_LEVEL=20 >(TESTGFS2-monitor-interval-15s) > notify interval=0s timeout=60s (TESTGFS2-notify-interval-0s) > start interval=0s timeout=60s (TESTGFS2-start-interval-0s) > stop interval=0s timeout=60s (TESTGFS2-stop-interval-0s) > > Stonith Devices: > Resource: FENCING (class=stonith type=fence_mpath) > Attributes: devices=/dev/mapper/36001405cb123d000 >pcmk_host_argument=key >pcmk_host_map=node1.localdomain:1;node2.localdomain:2;node3.localdomain:3 >pcmk_monitor_action=metadata pcmk_reboot_action=off > Meta Attrs: provides=unfencing > Operations: monitor interval=60s (FENCING-monitor-interval-60s) > Fencing Levels: > > Location Constraints: > Ordering Constraints: > start
Re: [ClusterLabs] How to make Pacemaker less trigger-happy
Your e-mail ended up in my spam mailbox. Yes, you can do the following: 1. Increase token + consensus timeouts (check the man for the proper ratio) 2. Always set the node/cluster in maintenance and stop the cluster stack before patching. Best Regards, Strahil Nikolov В понеделник, 28 октомври 2019 г., 19:26:39 ч. Гринуич+2, Casey Allen Shobe написа: I'm seeing a couple different situations where Pacemaker (using PostgreSQL Automated Failover resource) ends up thinking that the master node is not responding, and fences it when in fact the node was up and running fine. We are using a VMWare ESXi infrastructure, which is fairly overcommitted especially in our lower environments, and many times this correlates exactly with when a VMWare vMotion happens, which seems to cause some delay in the response to one of Pacemaker's health checks. In other cases, I have seen logind get restarted by an apt update, and that seems to trigger a failover even though PostgreSQL never went down. Looking for potential solutions to these - is there a way to increase the tolerance on # of failures or timeout length to avoid unnecessary failovers? Thank you for any advice! -- Casey ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
