Re: [Linux-HA] OCF_RESKEY_interval
Hello Lars, Lars Marowsky-Bree wrote: I'm missing the point here. But when you return the proper status - running, failed, not running -, heartbeat should do the "right thing" automatically when it finds the resource active prior to heartbeat being (re-)started? The point is, that we misuse heartbeat to monitor a process, that maybe already running before heartbeat starts, like cron. And we don't want heartbeat to stop it when it's already running. Therefore on probe we return not running. We only need a way to distinguish between a probe action and a monitor action, which we have done by checking the OCF_RESKEY_interval. The question now is, how can we find out if "probe" instead of "monitor" is what heartbeat executes? And what mechanism can we use that survives more than two releases? Cheers, Peter ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
[Linux-HA] BadThingsHappen with v2.0.5.
Hello, thanks for reading this, as it's with ancient v2.0.5., please tell me that this problem can not happen with recent version of heartbeat. Problem description: yesterday in one of our 2node HA-Clusters a successful takeover happened, where the failed node was resetted, so far so good. After I started heartbeat again on the failed node, it tried to takeover the resources, although they were running on the other node (BAD!). Ok, I detected an error in the setup, /var/lib/heartbeat/pengine was not writable by hacluster, causing this error message: pengine: [5580]: ERROR: Cannot write to /var/lib/heartbeat/pengine/pe-input-0.bz2: Permission denied Now my question: Is this error responsible for the faulty behavior of heartbeat? Will this error also trigger the faulty behavior in a recent version of heartbeat? (Please tell me that it won't). You may argue that a wrong configuration can cause all sorts of error behavior but I don't think that heartbeat should have ignored this error and continue to start the resource. Thanks for reading this far, Peter syslog attached. heartbeatlog.gz Description: GNU Zip compressed data ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] BadThingsHappen with v2.0.5.
Hi Andrew! Andrew Beekhof wrote: beosrv-c-2 is the failed node right? it was beosrv-c-1 that failed, beosrv-c-2 took over. do you have logs from there too? attached (messages about Gmain_timeout removed, there were too many of them) The problem now is that cibadmin -m reports: CIB on localhost _is_ the master instance on both nodes. Thanks for your time, Peter heartbeatlog2.gz Description: GNU Zip compressed data ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] BadThingsHappen with v2.0.5.
Andrew Beekhof wrote: then i'm afraid your use of the "dont fence nodes on startup" option has come back to haunt you beosrv-c-1 came up but was not able to find beosrv-c-2 (even though it _was_ running) and because of that option beosrv-c-1 just pretended beosrv-c-2 wasn't running and happily started activating resources. remember how we said that option wasn't a good idea :-) Hm, I don't understand, beosrv-c-2 fenced beosrv-c-1 in order to take over. Now you say, that as soon as beosrv-c-1 came back up again, it should fence beosrv-c-2, because it "thought" it was not there, but it was there? How can this happen? Peter ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Distinguish probe and monitor
Hello, Andrew Beekhof wrote: On 4/19/07, Keisuke MORI <[EMAIL PROTECTED]> wrote: Alan Robertson <[EMAIL PROTECTED]> writes: > Answering Andrew's question above would help clarify which: > "Why do you think you need to know?" The point is: the _meaning_ of probe is different to monitor: on monitor, if the resource is running then it's ok on probe, it is generally not ok in the sense that it's generally not what you want. That means, maybe you want your RA to log a message when called as "probe" to write something like: "Resource is running although it shouldn't" but when called as "monitor" you would probably log nothing on success. Convinced? Peter ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Distinguish probe and monitor
Hi, Andrew Beekhof wrote: On 4/19/07, Peter Kruse <[EMAIL PROTECTED]> wrote: The point is: the _meaning_ of probe is different to monitor: no, its not. trust me :-) what do you mean? of course it's different! both ask the same question: "Is this resource running?" yes, but the consequence to this question is different, depending on the answer: on the one hand, "running" can mean heartbeat has to stop it, because it shouldn't be running on the other hand, "running" has no consequence because that's how it's supposed to be. on monitor, if the resource is running then it's ok on probe, it is generally not ok in the sense that it's generally not what you want. That means, maybe you want your RA to log a message when called as "probe" to write something like: "Resource is running although it shouldn't" probes don't only happen at startup so this assumption does not hold That's exactly the problem, the RA should find out when to log the correct message. Example: On startup, when the probe is called, all (our) RAs log error messages like "ERROR! ERROR! Apache is not running!" which is wrong and confusing, because it's not an error at this stage. It should log a different message that don't alert an admin. I hope you can agree to that. but when called as "monitor" you would probably log nothing on success. Convinced? nope :-) still not? Peter ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Distinguish probe and monitor
Hello, thanks for this discussion. Andrew Beekhof wrote: On 4/19/07, Peter Kruse <[EMAIL PROTECTED]> wrote: the PE makes zero distinction between them and since it's the one doing the asking i believe that it is its "meaning" that counts. yes, think so, too. > both ask the same question: "Is this resource running?" yes, but the consequence to this question is different, depending on the answer: sometimes, but thats not the RA's business i don't agree and right now we're talking about logging which is not the only reason why the RAs should know more about what's going on. on the one hand, "running" can mean heartbeat has to stop it, because it shouldn't be running on the other hand, "running" has no consequence because that's how it's supposed to be. you're making an artificial distinction it's not artificial but a distinction that's important in practice. we can also probe in situations where we expect the resource to be running repeat after me - I can not and should not infer anything from the fact that an operation is a "probe" no, i won't repeat it :-P That's exactly the problem, the RA should find out when to log the correct message. no, it shouldn't. you're putting cluster policy into the RA and it doesn't belong there. i don't agree. only the top-level cluster-aware pieces of the cluster know enough to know if the result of an action was good or bad. this is precisely why the crmd process doesn't log ERRORs just because they didn't exit with OCF_SUCCESS (and instead relies on the TE and PE to detect the error conditions) which is wrong and confusing, because it's not an error at this stage. my point exactly - the RA does not (and can not) ever have enough information to log those ALERT!! ALERT!! messages accurately for monitor actions. maybe in the context but they know nothing (and should not know) about the resources, only the RAs know. They know the reason why a reason is failing, and there can be several reasons for one resource not running. for example, apache can be failing because the config file is incorrect, the ip address is not up, too many requests, oh! whatever, it's the RAs job to find it out. let the PE and TE do that for you - they see everything in context. you're not saying that PE or TE should do the logging? or that the RAs should not log messages themselves? Only the RAs can log the correct message, PE and TE only know "success" or "failure" that's simply not enough. >> Convinced? > > nope :-) still not? afraid not still trying... ;) ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] setting is_managed to true triggers restart
Hi, thanks for your replies. Andrew Beekhof wrote: On 5/8/07, Peter Kruse <[EMAIL PROTECTED]> wrote: for this reason, and to avoid polluting the parameter namespace with CRM options, we created meta attributes at some point. you can operate on these by simply adding the --meta option to your current command line. my crm_resource doesn't seem to have this option. however, there is a slight problem in that if is_managed was present as a regular attribute, then instead of creating/modifying a "meta attribute" when --meta was supplied, it would find and modify the "regular attribute" and still cause the resource to be restarted. > we found that out yesterday and i'm working on getting that fixed... so until then I have to avoid to have is_managed as a regular attribute in my cib? Peter ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] setting is_managed to true triggers restart
Hi, Andrew Beekhof wrote: the blocks look the same, just use meta_attributes instead of instance_attributes and put the options in there (and use cibadmin to update them). That works, thanks! Peter ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] ERROR: write_last_sequence: /var/lib/heartbeat/pengine/pe-input.last does not exist
Hi, Andrew Beekhof wrote: later versions do not seem to contain that message anymore That means it's nothing to worry about? Even in the latest (released) version (which me thinks is 2.0.8)? I can continue to ignore it then? Peter ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
[Linux-HA] Help understand an incident
Hello list! today in one of our clusters a failover occured. Good news: it succeeded. But... while looking through the logs we found that messages are missing on one node so we can not say exactly what happened. Attached is the syslog from node-2 from the time where there are no messages on node-1. Is it possible to say from that log what happened on node-1? Especially there are messages like this: Jul 3 11:22:59 beosrv-c-2 cibmon: [16501]: info: mask(cib_apply_diff): + crm-debug-origin="do_update_resource" transition_key="6:ad6f57b8-295b-4c20-8e0f-e01494577dfb" transition_magic="2:152;6:ad6f57b8-295b-4c20-8e0f-e01494577dfb" call_id="45" rc_code="152" op_status="2" interval="0" __crm_diff_marker__="added:top"/> Does that mean the action maillastnfs_stop_0 was run but returned the status 2? Or is it possible that the action never was run on node 1? Thanks in advance. Peter ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Stale NFS File Handles, even with fsid=1234
Hi, Stefan Lasiewski wrote: OS: RHEL4 u4 , x86_64 Heartbeat version: heartbeat-2.0.8-2.el4.centos Two servers: fs1 is 'primary'. fs2 is 'standby'. Client name is app1 , running RHEL4 u4 i386 What kernel version is that, make sure it is not vulnerable to this: http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2006-3468 - Each server exports /export1 . I use BakBone::Replicator to keep the fs1:/export1 & fs2:/export2 in sync. I don't know this Replicator, does it keep the inodes in sync? If it does not, then it's not the right product for you. Why are you not using DRBD? - On both fs1 & fs2, /var/lib/nfs is a symlink to /export1/nfs . fs1:/export1/nfs is copied to fs2:/export1/nfs . - fs1:/etc/exports is identical to fs2:/etc/exports . Each server is should be exporting similar filesystems with identical values. The problem is, it should be identical filesystems as well. [snip] When I shut down fs1 (primary NFS server), the following happens: - fs1 removes bond0 - fs1 shuts down nfslock - fs1 shuts down nfs - fs2 brings up bond0 - fs2 starts up nfslock - fs2 starts up nfs This doesn't look right to me, shouldn't it first bring up nfs, and finally start bond0? Otherwise, as soon as the address is available again, the clients will continue to request the nfs service but the server hasn't started nfs yet so the clients will receive an error. However, the NFS clients get the following errors: - I get a 'Stale NFS File Handle' on all NFS files and directories - /var/log/messages prints errors like the following: Jul 9 18:51:34 app902 kernel: nfs_update_inode: inode number mismatch Jul 9 18:51:34 app902 kernel: expected (0:13/0x2b30003), got (0:13/0x2ec003) Jul 9 18:51:35 app902 kernel: nfs_update_inode: inode number mismatch Jul 9 18:51:35 app902 kernel: expected (0:18/0x3970001), got (0:18/0x3ee4001) Exactly that could result in the error mentioned in above CVE. These errors go away if I simply remount the filesystems with a 'umount -a -t nfs && mount -a -t nfs'. Possibly, but not what you want, you want the application using the nfs share to continue. Cheers, Peter ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
[Linux-HA] How to validate meta-data of OCF RAs
Hello all, OCF Resource agents are required to "provide information about this resource as an XML snippet" (http://www.linux-ha.org/OCFResourceAgent) How can I validate that XML output? Thanks for any hint, Peter ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Re: Stale NFS File Handles, even with fsid=1234
Stefan, Stefan Lasiewski wrote: I have the following haresources (It's copied to cib.xml) on each host. Shouldn't this start up and shut down services in the correct order? The correct start order is: 1. rpc.lockd 2. rpc.statd 3. export filesystems with exportfs 4. rpc.nfsd 5. rpc.mountd 6. bond0 And for stop the other way around. # cat /etc/ha.d/haresources fs1.example.com \ 10.0.1.147/23/bond0 \ nfslock nfs I don't completely understand the order of events. However, I assumed this order would be properly handled by heartbeat. From /var/log/ha-debug , I think the startup order is bond0, nfslock, nfs. heartbeat does, what you tell it to do. Cheers, Peter Here is the output from /var/log/ha-debug : Jul 10 13:50:11 fs2 crmd: [5388]: info: do_lrm_rsc_op: Performing op=IPaddr_10_0_1_147_start_0 key=8:48:6b2b6e53-764f-4f44-99b7-2882ef78d270) Jul 10 13:50:11 fs2 crmd: [5388]: info: process_lrm_event: LRM operation IPaddr_10_0_1_147_start_0 (call=115, rc=0) complete Jul 10 13:50:11 fs2 crmd: [5388]: info: do_lrm_rsc_op: Performing op=nfslock_2_start_0 key=11:48:6b3b6e53-764f-4f44-99b7-2882ef78d270) Jul 10 13:50:12 fs2 crmd: [5388]: info: process_lrm_event: LRM operation nfslock_2_start_0 (call=117, rc=0) complete Jul 10 13:50:12 fs2 crmd: [5388]: info: do_lrm_rsc_op: Performing op=nfs_3_start_0 key=14:48:6b2b6e53-764f-4f44-99b7-2882ef78d270) Jul 10 13:50:12 fs2 crmd: [5388]: info: process_lrm_event: LRM operation nfs_3_start_0 (call=119, rc=0) complete ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Re: Stale NFS File Handles, even with fsid=1234
Hi Stefan, Stefan Lasiewski wrote: BakBone support tells me that Replicator will not keep the inodes in sync unfortunately. Does DRBD keep inodes in sync? I didn't know that was possible to transfer inodes from one filesystem to another filesystem. I thought the inodes were explicitly set on each filesystem. I don't see any documentation at DRBD regarding inodes (e.g. at http://www.google.com/search?hl=en&q=inode+site%3Ahttp%3A%2F%2Fdrbd.org+-svn). That's probably because DRBD does not know anything about inodes. It's not replicating the filesystem, but the block device below it. One detail I failed to mention earlier: fs1:/export1 and fs2:/export1 are on different storage units. They are different physical LUNs, and we were hoping to find some sync software which would keep the file data and file meta-data in sync. very good, DRBD is the way to go then. Thank you Peter and Yan, I appreciate your help. You are welcome, why don't you start reading here: http://www.linux-ha.org/DRBD/FAQ cheers, Peter ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] How to validate meta-data of OCF RAs
Hello, Max Hofer wrote: 1. Use the right DTD 2. use xmlstarlet to valdate the XML output of your script example: ./your_ocf_script meta-data | xmlstarlet val ra-api-1.dtd ah! very good, I guess that's what I was looking for. The hard part is the "right" DTD. As Andrew pointed out the HA Linux Project uses some non offical extension which were never updated on the OCF side. urgs, not so good... Attached a DTD which fits fine for me (make a diff against the offcial DTD to see what i changed). Yes, to summarize: 1. use "monitor" instead of "status" 2. use "validate-all" instad of "verify-all" But I don't understand this change: (version,longdesc+,shortdesc+,parameters,actions,special?) > does that mean it should be "longdesc+" instead of "longdesc" and "shortdesc+" instead of "shortdesc"? Peter ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
[Linux-HA] Confusion about MailTo RA and monitoring
Hello list, According to http://www.linux-ha.org/OCFResourceAgent a Resource Agent is required to support the monitor action. But in the MailTo agent I find: ocf_log warn "Don't stat/monitor me! MailTo is a pseudo resource agent, so the status reported may be incorrect" That indicates in my cib I should not define a monitor action for the MailTo agent. So my first question is: Why is it legal to not define a monitor action while it is required for any resource agent to support monitoring? Thanks, Peter ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Confusion about MailTo RA and monitoring
Hello, thanks for your fast replies. matilda matilda wrote: Peter Kruse <[EMAIL PROTECTED]> 16.07.2007 10:58 >>> 1) MailTo RA does have the monitor call. So it can be called and the required API is fullfilled. 2) In the case of MailTo the output of 'monitor' is a warning to the log. You can monitor (requirement) but you need not. 3) 'monitor' doesn't make sense to MailTo as this is a 'oneshot' RA, because there is no ongoing process after calling start. Does this help? This was indeed exactly the answers I was anticipating. So, if I'm not forced to specify a monitor action in my cib, then I might also not define a monitor action for my filesystem resource. Because once it's mounted, it's mounted, isn't it? And if another resource like apache is running on top of it, it cannot not be unmounted anyways. So it really does not make any sense to check for the filesystem being mounted every monitor interval. Makes sense? Even if you don't agree you should accept that it would be perfectly legal to do this. My second question is: If I haven't defined a monitor action in my cib for a resource, will heartbeat still probe it? Peter ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Confusion about MailTo RA and monitoring
Hi Lars, Lars Marowsky-Bree wrote: _If_ the fs goes haywire or is forcibly unmounted somehow, _and_ you're not monitoring it, heartbeat will never detect that error, but instead restart the application on top. That will fail though (because the fs is gone), and the node be blacklisted for that resource (start failure) and then migrated somewhere else. ie, no local restart will be attempted. Ok, but I could be living with that, if I were to do that. My second question is: If I haven't defined a monitor action in my cib for a resource, will heartbeat still probe it? Yes. Probes are mandatory and cannot be bypassed. Are you sure? I haven't seen the probes being called for MailTo RAs, or has that behaviour been changed since 2.0.8? Peter ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Confusion about MailTo RA and monitoring
Hi, David Lang wrote: there is a second issue with MailTo part of the OCF specs are that it is considered 'safe' to call start or stop multiple times on a RA, with MailTo this will generate multiple e-mails. this isn't a fatal problem, but it is an annoyance (I've had the shutting down on the inactive box in a pair generate MailTo messages from both boxes, causing management to freak out) are there enough 'oneshot' type things that it is worth adding the concept to the cib directly rather then trying to fake it out in the scripts? First, I think heartbeat should follow the (slightly modified) Law of Software Envelopment (http://www.jwz.org/hacks/). Every program attempts to expand until it can send mail. Those programs which cannot so expand are replaced by ones which can. I'm not sure how it should be done, but receiving mails when something in the cluster happens is very important. So important that it shouldn't be done in a Pseudo Resource, that uses a file to check if the resource is online or not. This will cause problems and false alarms. If those tracking files are leftover on startup the chances are too high you end up in multiple active recovery mode. Which is very annoying. If crm_mon indicates that a resource (MailTo) is split accross nodes, this should be alarming, shouldn't it? Peter ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
[Linux-HA] 2.1.1 change in behaviour
Hello, while testing version 2.1.1. I found a change in behaviour when a resource in a group failed. If there are resources a b c d e f in the group G, and e failed this happens: 2.0.8: stop f, stop e, start e, start f 2.1.1: stop f, stop e, start a, start b, ..., start f What is the reasoning behind this? Regards, Peter ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] 2.1.1 change in behaviour
Hi, Lars Marowsky-Bree wrote: On 2007-07-17T12:25:48, Peter Kruse <[EMAIL PROTECTED]> wrote: Good question. I assume Andrew has a good explanation when you have the testcase (pe inputs). ;-) done, bug #1648 But, this is safe by definition. true. What problem is this causing for you? It just takes longer until the resource is started again. Peter ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] APC SNMP STONITH
Hello, Philip Gwyn wrote: As discussed earlier, I'm writing a new SNMP STONITH plugin. The goal is for it to seamlessly work with the new and old MIBs (AP9606 vs AP7900). Ok, the old apcmastersnmp needed work, right. Instead of fixing the current apcmastersnmp.c, I started over from stratch, very roughly basing it on the net-snmp tutorial. one thing that bothered me in the old apcmastersnmp was that one could not configure the oids, they were hardcoded as #defines, would it be possible to change that? (I know, configuration files end in .c) So far, I have a small library that will - query the PDU - detect which MIB to use - find necessary outlet - turn the outlet on (or off) - query the PDU until the outlet goes to that state (or timeout) http://www.awale.qc.ca/ha-linux/apc-snmp/ Tomorrow I'm going to go over apcmastersnmp.c again to see if there are some gotchas that I might have missed. However, it does a reset (not turn off) so I don't know how useful that is. why don't you use the reset as well? That is a feature of the pdu that allows to configure the delay between off and on. I really think you should stick to that. It worked like this for some years now, and there was no problem at all with it. Your argument that if the server needs to be resetted because there is a problem with it, and therefore should not start automatically I can not follow. If the server boots after a reset what harm can it do? And you can change the behaviour in the BIOS. If the heartbeat projects thinks about replacing the current apcmastersnmp with yours it should be a compatible as possible. (Is Heartbeat going to replace the plugin with this one?) What firmware did you test your plugin with? Please have a look at this thread: http://lists.community.tummy.com/pipermail/linux-ha-dev/2007-April/014240.html make sure your oids are valid for versions 2.x and 3.x. Regards, Peter ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] [Pacemaker] new doc about stonith/fencing
Hi Dejan, Dejan Muhamedagic wrote: > As usual, constructive criticism/suggestions/etc are welcome. Thanks for sharing. Allow me to bring up a topic that to my point of view is important. You have written: > The lights-out devices (IBM RSA, HP iLO, Dell DRAC) are becoming increasingly > popular > and in future they may even become standard equipment of of-the-shelf > computers. > They are, however, inferior to UPS devices, because they share a power supply > with their > host (a cluster node). If a node stays without power, the device supposed to > control it > would be just as useless. Even though this is obvious to us, the cluster > manager is not > in the know and will try to fence the node in vain. This will continue > forever because all > other resource operations would wait for the fencing/stonith operation to > succeed. This is the same problem with PDUs as they share the same power supply with the host as well. Is there any intention to deal with this issue? I'm thinking of the powerfail algorithm: If the PDUs becomes unavailable and shortly after the host is unavailable as well, then assume the host is down and fenced successfully. This would be true if the PDU (and with it the host) loses power. At the moment it looks that stonith without such an algorithm is a SPoF by design, because after a single failure (powerloss), the cluster is not able to bring up the resources again. Looking forward to your comments, Peter ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] [Pacemaker] new doc about stonith/fencing
Hello, thanks for your replies, Andreas Mock wrote: >> If the PDUs becomes unavailable and shortly after the host is unavailable as >> well, then assume the host is down and fenced successfully. > > 'assume' is the bad word here. Stonith is there so that the cluster does NOT > have > to assume anything, but be SURE that there is a predictible state of the > cluster. You are saying that it is okay that a single failure can bring the cluster in a unsolvable situation? I thought "SPoF" would be the bad word. Because that's what it is. > IMHO you answered your question for yourself. ;-) I don't think so, the powerfail algorithm is of course a bit more complicated. Regards, Peter ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] [Pacemaker] new doc about stonith/fencing
Hi Andrew, Andrew Beekhof wrote: > I'd have to agree here. > However, I can imagine a (non-default) option for the agent called > cause-probable-data-corruption that let the agent behave in the way > Peter describes. > > Just so that there is no way the user can claim they didn't understand > the implications :) You probably guessed it but I'm talking about this: http://oss.sgi.com/cgi-bin/cvsweb.cgi/failsafe/FailSafe/cluster_services/cmd/crs/crsd_misc.c?annotate=1.1 (read from line 139) yes it's 8 years old, but the problem it tries to solve remains and i think there is no alternative. Regards, Peter ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] [Pacemaker] new doc about stonith/fencing
Hi Andrew, Andrew Beekhof wrote: > On Wed, May 6, 2009 at 10:13 AM, Peter Kruse wrote: >> You are saying that it is okay that a single failure can bring the cluster >> in a unsolvable situation? I thought "SPoF" would be the bad word. >> Because that's what it is. > > Its a very bad word, but the SPoF is very clearly the hardware here. > > I understand that there are many reasons to want these integrated > power switches to work in a clustered environment, but they don't. > We all know they don't, but we come up with complex algorithms so that > we can pretend that they do. great, I thought they were the recommended stonith devices? Are they not? Obviously not, so please do not mention them in the documentation (coming back to the topic ...). Or if you put it in the documentation then also say that it's not recommended because they do not work. Peter ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] [Pacemaker] new doc about stonith/fencing
Hi Andrew, Andrew Beekhof wrote: > Any switch that shares power with the host(s) it controls clearly has a SPoF. > You don't need me to tell you that. But that does not have to be a SPoF for the entire system! The problem here is that a single failure (power loss) causes not only one node to go down (and the pdu itself, yes), but the whole system stops working properly. Now you now have to say that one has to equip the pdus with redundant power supplies. Unfortunately I know of no such device. Which brings me to the conclusion that nobody has yet developed a device that works as a fully supported and recommended stonith device. Which is kind of a dilemma. > The scenario they don't work in might be acceptably unlikely for most > people, but the risk is there. > However, as I keep saying, I've no objection an option that implements > the failsafe algorithm (and documents the reason it exists) I would certainly vote for this. > Which particular documentation are you referring to? > I've not personally written any on stonith. Sorry for the misunderstanding, it's Dejan's documentation, who started this thread. Regards, Peter ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] [Pacemaker] new doc about stonith/fencing
Hello, Dejan Muhamedagic wrote: > I tried to list all devices which manage host's power and may be > used for fencing. I also tried to describe their deficiencies. > None of the devices are "recommended" really as that depends on > particular circumstances. I mean "recommended" in the sense that it really works. The only device mentioned here by Karl Katzke is the battery backed up RAC device by Dell, that probably fulfills the requirements. Regards, Peter ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] [Pacemaker] new doc about stonith/fencing
Hi Karl, Karl Katzke wrote: > > Actually, I believe that the different vendor implementations of "lights > out" systems (DRAC, HP/Compaq ILO, various others) *do* support that in > various ways and fashions. Dell's RAC has a battery that lasts for up to 30 > minutes last time I read it's specs. Regardless, with a "lights out" card > watching the server, you have two paths to positively query the status of a > node at the node itself, which is enough to be 90% sure it's dead. if the RAC is on battery, then you could be 100% sure. > The switched PDU devices in question, generally made by APC, have some > instabilities and, well, 'difficulties' in their implementations that are > not well-documented or intuitive. Some models don't inter-operate well with > other models in a mixed environment. And there's no positive feedback from > the node itself; you still don't know if the server's dead or just but you receive the feedback from the PDU itself, which is reliable. > unreachable due to a NIC failure. Checking that the ports you THINK the no, no, if you switch off the appropriate outlets, then you are 100% sure the node is down. > power is on isn't bad, but if the PDU is dead or your well-meaning coworker > changed the placement of the plugs, well... I would say you have a double failure here. > A decent design with DRAC is to have two switches. With the nodes that are > on Switch A, put the DRAC interfaces on Switch B, and vice versa. Switch A > and B should have separate battery backups; APC does make 'dumb' hot-fail > power switches that work reliably. I'm not sure what devices you are referring to here. Are these UPSs? Regards, Peter ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
