Re: [ClusterLabs] Which fence agent is needed for an Apache web server cluster?
On Sat, Mar 27, 2021 at 10:23 PM Andrei Borzenkov wrote: > On 28.03.2021 02:42, Reid Wahl wrote: > > On Sat, Mar 27, 2021 at 4:28 PM Strahil Nikolov > > wrote: > > > >> I had to tune the fence_ipmi recently on some older HPE blades. The > >> default settings were working, but also returning some output about > >> problems negotiating the cypher. > >> As that output could make future version of the fence agent go wild, I > >> tested several options untill no errors are reported. Maybe the cypher > flag > >> was different, but I think it was '-c'. If I'm wrong , the author of > this > >> thread can check the man page . > >> > > For the sake of archives - this is upper case 'C', not lower case, right? > Yes, thanks >> Yes -> 'HandlePowerKey=ignore' . I have never expected ipmi to try > >> graceful shutdown when I tell it to 'press and hold' or 'cold boot', > yet I > >> never checked the code of fence_ipmi. > >> > > > > fence_ipmilan uses ipmitool to send a poweroff signal. The iLO then > sends a > > virtual power button press, which IIRC goes through ACPI. By default on > > RHEL 7 and above, if the system is responsive, systemd-logind handles a > > power key press by initiating a graceful shutdown. You have to disable it > > from handling the power key press so if you want hard-power-off behavior. > > > > This is really a bug in iLO then. "ipmitool power off" is expected to > initiate hard power off; soft power off goes via "ipmitoo power soft". > Either that, or I'm wrong about how it's implemented :) Regardless, somehow systemd-login is able to handle (intercept) the button press. > Although I have seen with other vendor that "ipmitool power off" *also* > sent signal to OS and OS started shutdown which was then interrupted by > hard power off. Superficially (if you do not pay attention to console) > it looked like normal shutdown. > > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ > > -- Regards, Reid Wahl, RHCA Senior Software Maintenance Engineer, Red Hat CEE - Platform Support Delivery - ClusterHA ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Which fence agent is needed for an Apache web server cluster?
On 28.03.2021 02:42, Reid Wahl wrote: > On Sat, Mar 27, 2021 at 4:28 PM Strahil Nikolov > wrote: > >> I had to tune the fence_ipmi recently on some older HPE blades. The >> default settings were working, but also returning some output about >> problems negotiating the cypher. >> As that output could make future version of the fence agent go wild, I >> tested several options untill no errors are reported. Maybe the cypher flag >> was different, but I think it was '-c'. If I'm wrong , the author of this >> thread can check the man page . >> For the sake of archives - this is upper case 'C', not lower case, right? >> Yes -> 'HandlePowerKey=ignore' . I have never expected ipmi to try >> graceful shutdown when I tell it to 'press and hold' or 'cold boot', yet I >> never checked the code of fence_ipmi. >> > > fence_ipmilan uses ipmitool to send a poweroff signal. The iLO then sends a > virtual power button press, which IIRC goes through ACPI. By default on > RHEL 7 and above, if the system is responsive, systemd-logind handles a > power key press by initiating a graceful shutdown. You have to disable it > from handling the power key press so if you want hard-power-off behavior. > This is really a bug in iLO then. "ipmitool power off" is expected to initiate hard power off; soft power off goes via "ipmitoo power soft". Although I have seen with other vendor that "ipmitool power off" *also* sent signal to OS and OS started shutdown which was then interrupted by hard power off. Superficially (if you do not pay attention to console) it looked like normal shutdown. ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Which fence agent is needed for an Apache web server cluster?
I had to tune the fence_ipmi recently on some older HPE blades. The default settings were working, but also returning some output about problems negotiating the cypher.As that output could make future version of the fence agent go wild, I tested several options untill no errors are reported. Maybe the cypher flag was different, but I think it was '-c'. If I'm wrong , the author of this thread can check the man page . Yes -> 'HandlePowerKey=ignore' . I have never expected ipmi to try graceful shutdown when I tell it to 'press and hold' or 'cold boot', yet I never checked the code of fence_ipmi. With triple sbd , I mean sbd with 3 block devices. Best Regards,Strahil Nikolov On Sat, Mar 27, 2021 at 23:15, Reid Wahl wrote: On Saturday, March 27, 2021, Strahil Nikolov wrote: > My notes: > - ilo ssh fence mechanism is crappy due to ilo itself, try to avoid if > possible It has been unreliable in my experience. > - fence_ipmi requires some tunings (-c flag) and also to disable power button > from the system I've rarely, perhaps never, seen a customer have to tune the -c flag. By disabling the power button, do you mean setting HandlePowerKey=ignore in logind.conf? That's not specific to fence_ipmilan, to be clear. > - triple 'sbd' is quite reliable.My previous company was using 'softdog' > kernel module for a watchdog device and it never failed us. Yet, it's just a > kernel module (no hardware required) and thus RH do not support such setup. What do you mean by triple sbd? Correct that RH doesn't support using softdog as an sbd watchdog device. It was determined that it's not reliable in all situations. It's probably fine much of the time, and I'm glad you had a smooth experience with it. > > On Sat, Mar 27, 2021 at 22:15, Reid Wahl > wrote: > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ > -- Regards, Reid Wahl, RHCA Senior Software Maintenance Engineer, Red Hat CEE - Platform Support Delivery - ClusterHA ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Which fence agent is needed for an Apache web server cluster?
My notes:- ilo ssh fence mechanism is crappy due to ilo itself, try to avoid if possible- fence_ipmi requires some tunings (-c flag) and also to disable power button from the system - triple 'sbd' is quite reliable.My previous company was using 'softdog' kernel module for a watchdog device and it never failed us. Yet, it's just a kernel module (no hardware required) and thus RH do not support such setup. If you decide to use 'sbd', disable vendor's system recovery solution (like HPE's ASR) as it will also tinker with the watchdog. Best Regards,Strahil Nikolov On Sat, Mar 27, 2021 at 22:15, Reid Wahl wrote: ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/ ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Which fence agent is needed for an Apache web server cluster?
On Sat, Mar 27, 2021 at 4:28 PM Strahil Nikolov wrote: > I had to tune the fence_ipmi recently on some older HPE blades. The > default settings were working, but also returning some output about > problems negotiating the cypher. > As that output could make future version of the fence agent go wild, I > tested several options untill no errors are reported. Maybe the cypher flag > was different, but I think it was '-c'. If I'm wrong , the author of this > thread can check the man page . > > Yes -> 'HandlePowerKey=ignore' . I have never expected ipmi to try > graceful shutdown when I tell it to 'press and hold' or 'cold boot', yet I > never checked the code of fence_ipmi. > fence_ipmilan uses ipmitool to send a poweroff signal. The iLO then sends a virtual power button press, which IIRC goes through ACPI. By default on RHEL 7 and above, if the system is responsive, systemd-logind handles a power key press by initiating a graceful shutdown. You have to disable it from handling the power key press so if you want hard-power-off behavior. > With triple sbd , I mean sbd with 3 block devices. > > Best Regards, > Strahil Nikolov > > On Sat, Mar 27, 2021 at 23:15, Reid Wahl > wrote: > > > On Saturday, March 27, 2021, Strahil Nikolov > wrote: > > My notes: > > - ilo ssh fence mechanism is crappy due to ilo itself, try to avoid if > possible > > It has been unreliable in my experience. > > > - fence_ipmi requires some tunings (-c flag) and also to disable power > button from the system > > I've rarely, perhaps never, seen a customer have to tune the -c flag. > > By disabling the power button, do you mean setting HandlePowerKey=ignore > in logind.conf? That's not specific to fence_ipmilan, to be clear. > > > - triple 'sbd' is quite reliable.My previous company was using 'softdog' > kernel module for a watchdog device and it never failed us. Yet, it's just > a kernel module (no hardware required) and thus RH do not support such > setup. > > What do you mean by triple sbd? Correct that RH doesn't support using > softdog as an sbd watchdog device. It was determined that it's not reliable > in all situations. It's probably fine much of the time, and I'm glad you > had a smooth experience with it. > > > > On Sat, Mar 27, 2021 at 22:15, Reid Wahl > > wrote: > > ___ > > Manage your subscription: > > https://lists.clusterlabs.org/mailman/listinfo/users > > > > ClusterLabs home: https://www.clusterlabs.org/ > > > > > -- > Regards, > > > Reid Wahl, RHCA > Senior Software Maintenance Engineer, Red Hat > CEE - Platform Support Delivery - ClusterHA > > -- Regards, Reid Wahl, RHCA Senior Software Maintenance Engineer, Red Hat CEE - Platform Support Delivery - ClusterHA ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Which fence agent is needed for an Apache web server cluster?
On Saturday, March 27, 2021, Strahil Nikolov wrote: > My notes: > - ilo ssh fence mechanism is crappy due to ilo itself, try to avoid if possible It has been unreliable in my experience. > - fence_ipmi requires some tunings (-c flag) and also to disable power button from the system I've rarely, perhaps never, seen a customer have to tune the -c flag. By disabling the power button, do you mean setting HandlePowerKey=ignore in logind.conf? That's not specific to fence_ipmilan, to be clear. > - triple 'sbd' is quite reliable.My previous company was using 'softdog' kernel module for a watchdog device and it never failed us. Yet, it's just a kernel module (no hardware required) and thus RH do not support such setup. What do you mean by triple sbd? Correct that RH doesn't support using softdog as an sbd watchdog device. It was determined that it's not reliable in all situations. It's probably fine much of the time, and I'm glad you had a smooth experience with it. > > On Sat, Mar 27, 2021 at 22:15, Reid Wahl > wrote: > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ > -- Regards, Reid Wahl, RHCA Senior Software Maintenance Engineer, Red Hat CEE - Platform Support Delivery - ClusterHA ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Which fence agent is needed for an Apache web server cluster?
On Sat, Mar 27, 2021 at 1:13 PM Reid Wahl wrote: > Your hardware or virtualization platform largely determines which fence > agents are available to you. > By "available" here I mean "which fence agents will work to fence a node in your cluster." You can install any fence agents you want, but only a small subset are viable options to fence a node. That depends on your platform. For example, fence_ipmilan works with HP ProLiant, Dell PowerEdge, and any > other platform that offers power management via IPMI over LAN. > > sbd (optionally combined with fence_sbd) is a great option if you're on a > platform that offers a hardware watchdog device. Notably, VMware and the > major public cloud providers don't currently offer hardware watchdog > devices. > > The fence-agents-all package installs most of te available fence agents as > dependencies. IIRC, a few agents (e.g., fence-agents-aws) aren't included > when you install fence-agents-all. > By "available" here I mean "available to install." > On Sat, Mar 27, 2021 at 7:53 AM Jason Long wrote: > >> Hello, >> How To Configure High-Availability Cluster on CentOS 7 / RHEL 7 tutorial >> In " >> https://www.itzgeek.com/how-tos/linux/centos-how-tos/configure-high-avaliablity-cluster-on-centos-7-rhel-7.html; >> address, used below command: >> >> # yum install pcs fence-agents-all -y >> >> Is "fence-agents-all" package needed? >> >> Thank you. >> ___ >> Manage your subscription: >> https://lists.clusterlabs.org/mailman/listinfo/users >> >> ClusterLabs home: https://www.clusterlabs.org/ >> >> > > -- > Regards, > > Reid Wahl, RHCA > Senior Software Maintenance Engineer, Red Hat > CEE - Platform Support Delivery - ClusterHA > -- Regards, Reid Wahl, RHCA Senior Software Maintenance Engineer, Red Hat CEE - Platform Support Delivery - ClusterHA ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Which fence agent is needed for an Apache web server cluster?
Your hardware or virtualization platform largely determines which fence agents are available to you. For example, fence_ipmilan works with HP ProLiant, Dell PowerEdge, and any other platform that offers power management via IPMI over LAN. sbd (optionally combined with fence_sbd) is a great option if you're on a platform that offers a hardware watchdog device. Notably, VMware and the major public cloud providers don't currently offer hardware watchdog devices. The fence-agents-all package installs most of te available fence agents as dependencies. IIRC, a few agents (e.g., fence-agents-aws) aren't included when you install fence-agents-all. On Sat, Mar 27, 2021 at 7:53 AM Jason Long wrote: > Hello, > How To Configure High-Availability Cluster on CentOS 7 / RHEL 7 tutorial > In " > https://www.itzgeek.com/how-tos/linux/centos-how-tos/configure-high-avaliablity-cluster-on-centos-7-rhel-7.html; > address, used below command: > > # yum install pcs fence-agents-all -y > > Is "fence-agents-all" package needed? > > Thank you. > ___ > Manage your subscription: > https://lists.clusterlabs.org/mailman/listinfo/users > > ClusterLabs home: https://www.clusterlabs.org/ > > -- Regards, Reid Wahl, RHCA Senior Software Maintenance Engineer, Red Hat CEE - Platform Support Delivery - ClusterHA ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
