Re: [ClusterLabs] Antw: Re: Antw: Delayed first monitoring
On 13/08/15 04:38 AM, Ulrich Windl wrote: Miloš Kozák milos.ko...@lejmr.com schrieb am 13.08.2015 um 09:56 in Nachricht 55cc4daa.4020...@lejmr.com: Dne 13.8.2015 v 09:26 Andrei Borzenkov napsal(a): On Thu, Aug 13, 2015 at 10:01 AM, Miloš Kozák milos.ko...@lejmr.com wrote: However, this does not make sense at all. Presumably, the pacemaker should get along with lsb scripts which comes from system repository, right? Let's forget about pacemaker for a moment. You have system startup where service B needs service A. initscript for service A completes and script for service B is started but service A is not yet ready to be used. This is a bug in startup script. Irrespectively of whether you use it with pacemaker or not. I am sorry, but I didnt get the point.. If service A is not ready then service B should not be started. As you seem to be ignorant for advice: Ok, I'm starting to get annoyed now. You need to be more polite and respectful on this list. Yes, you are right: Service B should check whether service A is up before starzing itself. The easy change for the start script of B is to find aout what command was run before it to check whether the command before did everything OK by checking again itself. [...] ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] implementation of fence and stonith agents for pacemaker
On 13/08/15 07:54 AM, Kostiantyn Ponomarenko wrote: Digimer, Thank you. I will try this out. One more question. What about directories for those agents, what rules are here? Thank you, Kostya I'm not entirely sure I understand the question, sorry. What do you mean by directories for those agents? If you're asking about implementation details like language to use, etc, there are no rules. Python and bash are the most common languages, I think, but I write my fence agents in perl just fine. I think a couple are even in C. I suspect that python is the language upstream maintainer are happier with, but as beekhof said in the RA script; the person doing the work gets to make the decisions. :) If I didn't answer your question, please clarify. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] upgrade from 1.1.9 to 1.1.12 fails to start
On 18/08/15 10:15 AM, Streeter, Michelle N wrote: I created a whole new virtual and installed everything with the new version and pacemaker wouldn’t start. I have not yet learned how to use the logs yet to see what they have to say. No, I did not upgrade corosync. I am running the latest which will work with rhel6. When I tried later versions, they failed and I was told it was because we are not running rhel7. I am getting the feeling this version of Pacemaker does not work on rhel6 either. Do you believe this is the case? Or is there some configuration that needs to be done between 1.1.9 and 1.1.12? Michelle Streeter You need to upgrade all of the cluster components please. Ideally, upgrade the whole OS... -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] [Slightly OT] OCFS2 over LVM
On 23/08/15 04:40 PM, Jorge Fábregas wrote: On 08/23/2015 02:16 PM, Digimer wrote: One, this is on-topic, so don't worry. :) Thanks. Two, I've never used ocfs2 (allergic to all things Oracle), but clvmd makes LVM cluster-aware, as you know. So I have no idea why they'd say that. I just restarted everything and then it worked. I was doing a bunch of stuff so I don't know what really happend. It is working perfectly fine now over LVM. I even extended the logical volume and then used tunefs.ocfs2 -S /dev/vg1/lvol1 and it did the online-resize perfectly fine. Thanks Digimer sorry for the noise! No worries at all, glad you sorted it out. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] [Slightly OT] OCFS2 over LVM
On 24/08/15 07:55 AM, Jorge Fábregas wrote: On 08/24/2015 06:52 AM, Kai Dupke wrote: Not sure what you want to run on top of your 2-node cluster, but OCFS2 is only needed when you need a shared file system. This is for an application that manages the high-availability by itself (in an active/active fashion) and the only thing that's needed from the OS is a shared filesystem. I quickly thought about NFS but then the reliability of the NFS server was questioned etc. I could create an NFS cluster for that but that will be two more servers. You get the idea. I'm still googling NFSv4 vs OCFS2 If anyone here have experience (going from one to the other) I'd like to hear it. For plain failover with volumes managed by cLVM you don't need OCFS2 (and can save one level of complexity). This is my first time using a cluster filesystem and indeed I get it: there's lots of things to be taken care of many possible ways to break it. Thanks, Jorge Speaking from a gfs2 background, but assuming it's similar in concept to ocfs2... Cluster locking comes at a performance cost. All locks need to be coordinated between the nodes, and that will always be slower that local locking only. They are also far less commonly used than options like nfs. Using a pair of nodes with a traditional file system exported by NFS and made accessible by a floating (virtual) IP address gives you redundancy without incurring the complexity and performance overhead of cluster locking. Also, you won't need clvmd either. The trade-off through is that if/when the primary fails, the nfs daemon will appear to restart to the users and that may require a reconnection (not sure, I use nfs sparingly). Generally speaking, I recommend always avoiding cluster FSes unless they're really required. I say this as a person who uses gfs2 in every cluster I build, but I do so carefully and in limited uses. In my case, gfs2 backs ISOs and XML definition files for VMs, things that change rarely so cluster locking overhead is all but a non-issue, and I have to have DLM for clustered LVM anyway, so I've already incurred the complexity costs so hey, why not. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Cluster.conf
The cluster.conf is needed by cman, and in RHEL 6, pacemaker needs to use cman as the quorum provider. So you need a skeleton cluster.conf and it is different from cib.xml. If you use pcs/pcsd to setup pacemaker on RHEL 6.7, it should configure everything for you, so you should be able to go straight to setting up pacemaker and not worry about cman/corosync directly. digimer On 24/08/15 01:52 PM, Streeter, Michelle N wrote: If I have a cluster.conf file in /etc/cluster, my cluster will not start. Pacemaker 1.1.11, Corosync 1.4.7, cman 3.0.12, But if I do not have a cluster.conf file then my cluster does start with my current configuration. However, when I try to stop the cluster, it wont stop unless I have my cluster.conf file in place. How can I dump my cib to my cluster.conf file so my cluster will start with the conf file in place? Michelle Streeter ASC2 MCS – SDE/ACL/SDL/EDL OKC Software Engineer The Boeing Company ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] upgrade from 1.1.9 to 1.1.12 fails to start
On 17/08/15 05:13 PM, Streeter, Michelle N wrote: I was recommended to upgrade from 1.1.9 to 1.1.12. I had to uninstall the 1.1.9 version to install the 1.1.12 version I am not allowed to connect to a repo and so I have to download the rpms and install them individually. After I installed pacemaker-lib, cli, cluster-lib, and pacemaker itself, when I rebooted, the cluster failed to start When I tried to manually start it, I got Starting Pacemaker Cluster Manager/etc/init.d/pacemaker: line 94: 8219 Segmentation fault (core dumped) $prog /dev/null 21 I deleted the Cluster.conf file and the cib.xml and all the back up versions and tried again and got the same error. I googled this error and really got nothing. Any ideas? As a test, can you create a fresh, new cluster? -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Node lost early in HA startup -- no STONITH
On 02/08/15 11:10 PM, Chris Walker wrote: Hello, We recently had an unfortunate sequence on our two-node cluster (nodes n02 and n03) that can be summarized as: 1. n03 became pathologically busy and was STONITHed by n02 2. The heavy load migrated to n02, which also became pathologically busy 3. n03 was rebooted 4. During the startup of HA on n03, n02 was initially seen by n03: Jul 26 15:23:43 n03 crmd: [143569]: info: crm_update_peer_proc: n02.ais is now online 5. But later during the startup sequence (after DC election and CIB sync) we see n02 die (n02 is really wrapped around the axle, many stuck threads, etc) Jul 26 15:27:44 n03 heartbeat: [143544]: WARN: node n02: is dead ... Jul 26 15:27:45 n03 crmd: [143569]: info: ais_status_callback: status: n02 is now lost (was member) our deadtime is 240 seconds, so n02 became unresponsive almost immediately after n03 reported it up at 15:23:43 6. The troubling aspect of this incident is that even though there are multiple STONITH resources configured for n03, none of them was engaged and n03 then mounted filesystems that were also active on n02. I'm wondering whether the fact that no STONITH resources were started by this time explains why n02 was not STONITHed. Shortly after n02 is declared dead we see STONITH resources begin starting, e.g., Jul 26 15:27:47 n03 pengine: [152499]: notice: LogActions: Start n03-3-ipmi-stonith (n03) Does the fact that since there were no active STONITH resources when n02 was declared dead, no STONITH action was taken against this node? Is there a fix/workaround for this scenario (we're using heartbeat 3.0.5 and pacemaker 3.1.6 (RHEL6.2))? Thanks very much! Chris Please share your full config and the logs from both nodes through the duration of the events. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] implementation of fence and stonith agents for pacemaker
On 11/08/15 11:17 AM, Kostiantyn Ponomarenko wrote: Hi guys, Is there any documentation which describes implementation of fence and STONITH agents like those ones for Resource Agents?: http://www.linux-ha.org/wiki/OCF_Resource_Agents http://www.linux-ha.org/doc/dev-guides/ra-dev-guide.html I am particular interested in the arguments which are passed to a stonith resource by stonithd. Is there any guidelines what arguments it has to handle and where it must be put (which directories are allowed)? So far I found this http://linux.die.net/man/7/stonithd . But for example, it is not clear for me how pcmk_host_check=dynamic-list which is (query the device) works. Do I need to handle some action in my stonith agent for that parameter? Thank you, Kostya This is the API; https://fedorahosted.org/cluster/wiki/FenceAgentAPI It needs to be updated to reflect the need for agents to output the XML metadata. For now, you should be able to see the format needed by looking at the metadata output of existing FAs. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] [ClusterLabs Developers] Resource Agent language discussion
On 07/08/15 09:36 AM, Jan Pokorný wrote: Hello, [adding users list as I think there's an overlap] On 07/08/15 12:09 +0200, Jehan-Guillaume de Rorthais wrote: Now, I would like to discuss about the language used to write a RA in Pacemaker. I never seen discussion or page about this so far. it wasn't in such a heretic :) tone, but I tried to show that it is extremely hard (if not impossible in some instances) to write bullet-proof code in bash (or POSIX shell, for that matter) because it's so cumbersome to move from whitespace-delimited words as a single argument and words as standalone arguments back and forth, connected with quotation-desired/-counterproductive madness (what if one wants to indeed pass quotation marks as legitimate characters within the passed value, etc.) few months back: http://clusterlabs.org/pipermail/users/2015-May/000403.html (even on developers list, but with fewer replies and broken threading: http://clusterlabs.org/pipermail/developers/2015-May/23.html). HINT: I don't want to discuss (neither troll about) what is the best language. I would like to know why **ALL** the RA are written in bash I would expect the original influence were the init scripts (as RAs are mostly just enriched variants to support more flexible configuration and better diagnostics back to the cluster stack), which in turn were born having simplicity and ease of debugging (maintainability) in mind. and if there's traps (hidden far in ocf-shellfuncs as instance) to avoid if using a different language. And is it acceptable to include new libs for other languages? https://github.com/ClusterLabs/resource-agents/blob/v3.9.6/doc/dev-guides/ra-dev-guide.txt#L33 doesn't make any assumption about the target language beside stating what's a common one. We rewrote the RA in perl, mostly because of me. I was bored with bash/sh limitations AND syntax AND useless code complexity for some easy tasks AND traps (return code etc). In my opinion, bash/sh are fine if you RA code is short and simple. Which was mostly the case back in the time of heartbeat which was stateless only. But it became a nightmare with multi-state agents struggling with complexe code to fit with Pacemaker behavior. Have a look to the mysql or pgsql agents. Moreover, with bash, I had some weird behaviors (timeouts) from the RA between runuser/su/sudo and systemd/pamd some months ago. The three of them have system implications or side effects deep in the system you need to take care off. Using a language able to seteuid/setuid after forking is much more natural and clean to drop root privileges and start the daemon (PostgreSQL refuses to start as root and is not able to drop its privileges to another system user itself). Other disadvantage of shell scripts is that frequently many processes are spawned for simple changes within the filesystem and for string parsing/reformatting, which in turn creates a dependency on plenty of external executables. Now, we are far to have a enterprise class certified code, our RA had its very first tests passed successfully yesterday, but here is a quick feedback. The downside of picking another language than bash/sh is that there is no OCF module/library available for them. This is quite inconvenient when you need to get system specifics variables or logging shortcut only defined in ocf-shellfuncs (and I would guess patched by packagers ?). As instance, I had to capture values of $HA_SBIN_DIR and $HA_RSCTMP from my perl code. There could be a shell wrapper that would put these values into the environment and then executed the target itself for its disposal (generic solution for arbitrary executable). That's not applicable for procedural knowledge (logging, etc.), though, as you mention below. Another exemple, our perl RA is only logging to syslog presently. We will probably have to rewrite the ocf_log/ha_log/ha_debug in perl before publishing the final code. Any tip about this ? At some point, to have a clean, portable and OS agnostic code, I wonder how much code we will have to port from ocf-shellfuncs to perl... Allow me to add, uselessly; perl! 3 -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] nfsServer Filesystem Failover average 76s
It may or may not help, but I know some performance improvements were made recently. Can you update to pacemaker 1.1.12? It should be in the stock repos. 1.1.9 was an odd release. If you're still on RHEL 6.4 (I am guessing, but the pacemaker version), it would be a good idea to update in general. *Lots* changed in HA from 6.4 - 6.6. digimer On 14/08/15 01:17 PM, Streeter, Michelle N wrote: I am getting an average failover for nfs of 76s. I have set all the start and stop settings to 10s but no change. The Web page is instant but not nfs. I am running two node cluster on rhel6 with pacemaker 1.1.9 Surely these times are not right? Any suggestions? Resources: Group: nfsgroup Resource: nfsshare (class=ocf provider=heartbeat type=Filesystem) Attributes: device=/dev/sdb1 directory=/data fstype=ext4 Operations: start interval=0s (nfsshare-start-interval-0s) stop interval=0s (nfsshare-stop-interval-0s) monitor interval=10s (nfsshare-monitor-interval-10s) Resource: nfsServer (class=ocf provider=heartbeat type=nfsserver) Attributes: nfs_shared_infodir=/data/nfsinfo nfs_no_notify=true Operations: start interval=0s timeout=10s (nfsServer-start-timeout-10s) stop interval=0s timeout=10s (nfsServer-stop-timeout-10s) monitor interval=10 timeout=20s (nfsServer-monitor-interval-10) Resource: NAS (class=ocf provider=heartbeat type=IPaddr2) Attributes: ip=192.168.56.110 cidr_netmask=24 Operations: start interval=0s timeout=20s (NAS-start-timeout-20s) stop interval=0s timeout=20s (NAS-stop-timeout-20s) monitor interval=10s timeout=20s (NAS-monitor-interval-10s) Michelle Streeter ASC2 MCS – SDE/ACL/SDL/EDL OKC Software Engineer The Boeing Company ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] 2 Nodes Pacemaker for Nginx can only failover 1 time
On 08/08/15 11:02 PM, jun huang wrote: Hello Everyone, I setup a cluster with two nodes with pacemaker 1.1.10 on CentOS 7. Then I downloaded aresource agent for nginx from github https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/nginx I tested my setup like this: 1. Node 1 is started with the nginx and vip, everyting is ok 2. Kill Node1 nginx, wait for a few seconds 3. See the ngnix and vip are moved to node2, failover succeeded, and Node1 doesn't have any resources active 4. I kill nginx on node2, but nginx and vip don't come back to Node1 I set |no-quorum-policy=ignore| and |stonith-enabled=false|. Without fencing (stonith), the cluster will enter an undefined state after a node fails and debugging becomes very difficult. Please configure (and test!) stonith, then see if your problem remains. Why won't pacemaker let the resource come back to Node1? What did I miss here? I guess the node1 is still in some failure status, how can I recovery the node? Does anyone can shed some light on my questions? Thank you in advance. Thanks, Jacob ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] fence-virtd reach remote server serial/VM channel/TCP
On 05/08/15 03:09 PM, Jan Pokorný wrote: On 02/08/15 16:30 +0200, Noel Kuntze wrote: I would like to know if it is possible for fence-virtd to realy a request from a client, which it received via serial, VM channel or TCP connection from an agent to another daemon, if the VM that should be fenced does not run on the same host as the contacted daemon. First, it doesn't sound like very commendable or at least common setup to have virtualized cluster nodes spread around multiple hosts. When increasing the complexity of a deployment, new points of failure can be introduced, defeating the purpose of HA. Could you please share details of your use case? To interject; It is not something I would do, but I've heard of cases where a separate department handles hardware and the devops types are restricted to VMs only. In such a case, you would want to span hosts to protect against a host failure. Not sure if this is Noel's use-case, of course. To your question, it might (hypothetically) be doable if you manage to put the guests on the first host together with the other host into the same multicast-friendly network or will rely multicast packets between those remote sides by other means. Alternatively, you might implement such relying directly as fence_virtd module (backend), possibly reusing some code from the client side (fence_virt). -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Pacemaker failover failure
As said before, fencing. On 01/07/15 06:54 AM, alex austin wrote: so did another test: two nodes: node1 and node2 Case: node1 is the active node node2: is pasive if I killall -9 pacemakerd corosync on node 1 the services do not fail over to node2, but if I start corosync and pacemaker on node1 then it fails over to node 2. Where am I mistaking? Alex On Wed, Jul 1, 2015 at 12:42 PM, alex austin alexixa...@gmail.com mailto:alexixa...@gmail.com wrote: Hi all, I have configured a virtual ip and redis in master-slave with corosync pacemaker. If redis fails, then the failover is successful, and redis gets promoted on the other node. However if pacemaker itself fails on the active node, the failover is not performed. Is there anything I missed in the configuration? Here's my configuration (i have hashed the ip address out): node host1.com http://host1.com nodehost2.com http://host2.com primitiveClusterIP IPaddr2 \ paramsip=xxx.xxx.xxx.xxx cidr_netmask=23\ opmonitor interval=1stimeout=20s\ opstart interval=0timeout=20s\ opstop interval=0timeout=20s\ metais-managed=truetarget-role=Startedresource-stickiness=500 primitiveredis redis \ metatarget-role=Masteris-managed=true\ opmonitor interval=1srole=Mastertimeout=5son-fail=restart msredis_clone redis\ metanotify=trueis-managed=trueordered=falseinterleave=falseglobally-unique=falsetarget-role=Mastermigration-threshold=1 colocationClusterIP-on-redis inf: ClusterIPredis_clone:Master colocationip-on-redis inf: ClusterIPredis_clone:Master propertycib-bootstrap-options: \ dc-version=1.1.11-97629de\ cluster-infrastructure=classic openais (with plugin)\ expected-quorum-votes=2\ stonith-enabled=false propertyredis_replication: \ redis_REPL_INFO=host.com http://host.com thank you in advance Kind regards, Alex ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Antw: Re: [Slightly OT] OCFS2 over LVM
On 25/08/15 04:45 AM, Ulrich Windl wrote: Digimer li...@alteeve.ca schrieb am 24.08.2015 um 18:20 in Nachricht 55db4453.10...@alteeve.ca: [...] Using a pair of nodes with a traditional file system exported by NFS and made accessible by a floating (virtual) IP address gives you redundancy without incurring the complexity and performance overhead of cluster locking. Also, you won't need clvmd either. The trade-off through is that if/when the primary fails, the nfs daemon will appear to restart to the users and that may require a reconnection (not sure, I use nfs sparingly). But that's a cheap trick: You say don't provide HA-storage (CFS), but use existing one (NFS). How do you build a HA-NFS server? You need another cluster. Not everybody has that many nodes available. DRBD in single-primary will do the job just fine. Recovery is simply a matter of; fence - promote to primary - mount - start nfs - take virtual IP, done. Only 2-nodes needed. This is a common setup. Generally speaking, I recommend always avoiding cluster FSes unless they're really required. I say this as a person who uses gfs2 in every cluster I build, but I do so carefully and in limited uses. In my case, gfs2 backs ISOs and XML definition files for VMs, things that change rarely so cluster locking overhead is all but a non-issue, and I have to have DLM for clustered LVM anyway, so I've already incurred the complexity costs so hey, why not. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] how to fence in a two node cluster.If haretbeat network is down between the two nodes, which node will fence the other node?
On 05/11/15 02:09 AM, Shilu wrote: > cid:image002.png@01D1091A.AEFC8740 > > In HA, if a node is declared dead, it needs to be fenced/stonith'ed > before its services are recovered. Not doing this can lead to a split-brain. > > If haretbeat network is down between the two nodes,which node will fence > the other node? > > If two nodes were fenced respectively,the service will be not available. > > I want to know if IPMI fence is appropriate to two nodes cluster. Is > there a better fence way? You will choose a node to be the survivor in this case, and the set 'delay="15"' in the IPMI fence attribute list. If you set the delay against node 1, for example, then in a comm break like you describe, both will declare the other dead and start fencing at the same time. Node 1 looks up how to fence node 2, sees no delay and immediately starts to fence. Meanwhile, node 2 looks up how to fence node 1, sees a delay and pauses for 15 seconds. Node 1 will kill node 2 long before the delay expires. Also, disable acpid. You want the node to react to the power button being pressed by shutting down immediately. Also, this doesn't solve all problems. Pull all power to a node. This will cause IPMI to fail, so the fence call will fail. This is why I use IPMI *and* a pair of switched PDUs (IPMI on one switch, PDUs on another). -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] large cluster - failure recovery
9] ip-10-178-149-131 stonith-ng: info: > apply_xml_diff: Digest mis-match: expected > 01192e5118739b7c33c23f7645da3f45, calculated > f8028c0c98526179ea5df0a2ba0d09de > Nov 04 18:21:15 [17749] ip-10-178-149-131 stonith-ng: warning: > cib_process_diff: Diff 1.15046.2 -> 1.15046.3 from local not > applied to 1.15046.2: Failed application of an update diff > Nov 04 18:21:15 [17749] ip-10-178-149-131 stonith-ng: notice: > update_cib_cache_cb:[cib_diff_notify] Patch aborted: Application of > an update diff failed (-1006) > Nov 04 18:21:15 [17749] ip-10-178-149-131 stonith-ng: notice: > cib_process_diff: Diff 1.15046.2 -> 1.15046.3 from local not > applied to 1.15046.3: current "num_updates" is greater than required > [...] > > > ps. Sorry if should posted on corosync newsgroup, just the CIB > synchronization fails, so this group seemed to me the right place. All of the HA mailing lists are merging into Cluster labs. This is the right place to ask. > -- > Best Regards, > > Radoslaw Garbacz -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Fencing Two-Node Cluster on Two ESXi Hosts
On 05/11/15 11:04 AM, Ken Gaillot wrote:
> On 11/05/2015 02:43 AM, Gonçalo Lourenço wrote:
>> Greetings, everyone!
>>
>>
>> I'm having some trouble understanding how to properly setup fencing in my
>> two-node cluster (Pacemaker + Corosync). I apologize beforehand if this
>> exact question has been answered in the past, but I think the intricacies of
>> my situation might be interesting enough to warrant yet another thread on
>> this matter!
>
> No apology needed! Fencing is both important and challenging for a
> beginner, and I'd much rather see a question about fencing every week
> than see someone disable it.
I love you.
>> My setup consists of two virtual machines (the nodes of the cluster) running
>> on two separate VMWare ESXi servers: VM 1 (CentOS 7) is running on ESXi 1;
>> VM 2 (another CentOS 7) is on ESXi 2. I have all resources except for
>> fencing running as intended (DRBD, a virtual IP address, and a DHCP server).
>> I have no access to any additional computing resources, both physical and
>> virtual. Both nodes use one NIC for DRBD and Corosync (since it's a virtual
>> environment, I thought this would be enough) and another one used
>> exclusively for the DHCP server.
>
> One NIC for DRBD+corosync is fine. You can configure DRBD not to use the
> full bandwidth, so corosync always has some breathing room.
>
>> My idea for fencing this two-node cluster is the following:
>> . Setup one VMWare SOAP fencing agent on VM 1 that talks to ESXi 1. This
>> agent would run exclusively on VM 1 and would only serve to fence VM 2;
>> . Another VMWare SOAP fencing agent on VM 2 that'll talk to ESXi 2. Yet
>> again, this agent would run solely on VM 2 and would only fence VM 1.
>>
>> Basically, the idea is to have them fence one another through the ESXi host
>> they're running on.
>> Is this the right way to go? If so, how should I configure the fencing
>> resource? If not, what should I change?
>>
>> Thank you for your time.
>>
>>
>> Kind regards,
>> Gonçalo Lourenço
>
> I'm not familiar enough with VMWare to address the specifics, but that
> general design is a common approach in a two-node cluster. It's a great
> first pass for fencing: if there's a problem in one VM, the other will
> fence it.
>
> However what if the underlying host is not responsive? The other node
> will attempt to fence but get a timeout, and so the cluster will refuse
> to run any resources ("better safe than sorry").
>
> The usual solution is to have two levels of fencing: the first as you
> suggested, then another for the underlying host in case that fails.
>
> The underlying hosts probably have IPMI, so you could use that as a
> second level without needing any new hardware. If the underlying host OS
> is having trouble, the other node can contact the IPMI and power-kill it.
>
> However if IPMI shares power with the host (i.e. on-board as opposed to
> a separate unit on a blade chassis), then you still have no recovery if
> power fails. The most common solution is to use an intelligent power
> switch, whether as the second level, or as a third level after IPMI. If
> that's not an option, VM+IPMI fencing will still cover most of your
> bases (especially if the physical hosts have redundant power supplies).
>
> Be sure to use "two_node: 1" in corosync.conf (assuming you're using
> corosync 2). That will allow one node to keep quorum if the other is
> shut down.
>
> ___
> Users mailing list: Users@clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Cluster node loss detection.
On 16/10/15 12:37 PM, Vallevand, Mark K wrote: > Fencing, yes. I have pcmk-redirect for each node in cluster.conf. Do you have stonith configured (and tested!) in Pacemaker as well? > I run with default cman settings for corosync. No totem clause. That gives > the 20s detection. Not sure what the defaults really are. > I added to > cluster.conf and get about a 5s detection. > > The corosync man page says: >token This timeout specifies in milliseconds until a token loss is > declared after not receiving a token. This is the time spent detecting a > failure of a processor in the current configuration. Reforming > a new configuration takes about 50 milliseconds in addition to this > timeout. > > The default is 1000 milliseconds. > >token_retransmit > This timeout specifies in milliseconds after how long before > receiving a token the token is retransmitted. This will be automatically > calculated if token is modified. It is not recommended to > alter this value without guidance from the corosync community. > > The default is 238 milliseconds. > >hold This timeout specifies in milliseconds how long the token > should be held by the representative when the protocol is under low utiliza‐ > tion. It is not recommended to alter this value without > guidance from the corosync community. > > The default is 180 milliseconds. > >token_retransmits_before_loss_const > This value identifies how many token retransmits should > be attempted before forming a new configuration. If this value is set, > retransmit and hold will be automatically calculated from > retransmits_before_loss and token. > > The default is 4 retransmissions. > > But, I don't know what cman sets these to. But, they aren't these values. > And, they aren't the values in the cman man page, which says this: Maybe it's changed by the ubuntu packagers? I don't know, I don't use debian or ubuntu. > Cman uses different defaults for some of the corosync > parameters listed in corosync.conf(5). If you wish to use a non-default set‐ > ting, they can be configured in cluster.conf as shown above. > Cman uses the following default values: > >vsftype="none" > token="1" > token_retransmits_before_loss_const="20" > join="60" > consensus="4800" > rrp_mode="none" >From: Digimer [mailto:li...@alteeve.ca] > Sent: Friday, October 16, 2015 11:18 AM > To: Cluster Labs - All topics related to open-source clustering welcomed > Subject: Re: [ClusterLabs] Cluster node loss detection. > > On 16/10/15 11:40 AM, Vallevand, Mark K wrote: >> Thanks. I wasn't completely aware of corosync's role in this. I see new >> things in the docs every time I read them. >> >> I looked up the corosync settings at one time and did it again: >> token loss 3000ms >> retransmits 10 >> So 30s. Redid my simple testing and got detection times of 22s, 26s, and >> 25s using very crude methods. >> Any warnings about setting these values to something else? >> We require our customers to use an isolated, private network for cluster >> communications. All taken care of in our instructions and cluster >> configuration scripts. Network traffic will not be a factor. So, I'm >> thinking 1000ms and 5 retransmits as an experiment. > > That is very high. I think the default is something like 236ms x 4 losses. > > You do have fencing, right? > >> I was pretty sure that DLM was just being informed by clustering, but I >> needed to ask. >> >> Again, thanks. >> >> >> Regards. >> Mark K Vallevand mark.vallev...@unisys.com >> <mailto:mark.vallev...@unisys.com> >> Never try and teach a pig to sing: it's a waste of time, and it annoys the >> pig. > > -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Fencing questions.
On 19/10/15 06:53 AM, Arjun Pandey wrote: > Hi > > I am running a 2 node cluster with this config on centos 6.5/6.6 where It's important to keep both nodes on the same minor version, particularly in this case. Please either upgrade centos 6.5 to 6.6 or both to 6.7. > i have a multi-state resource foo being run in master/slave mode and a > bunch of floating IP addresses configured. Additionally i have > a collocation constraint for the IP addr to be collocated with the master. > > Please find the following files attached > cluster.conf > CIB It's preferable on a mailing list to copy the text into the body of the message. Easier to read. > Issues that i have :- > 1. Daemons required for fencing > Earlier we were invoking cman start quorum from pacemaker script which > ensured that fenced / gfs and other daemons are not started. This was ok > since fencing wasn't being handled earlier. The cman fencing is simply a pass-through to pacemaker. When pacemaker tells cman that fencing succeeded, it inform DLM and begins cleanup. > For fencing purpose do we only need the fenced to be started ? We don't > have any gfs partitions that we want to monitor via pacemaker. My > concern here is that if i use the unmodified script then pacemaker start > time increases significantly. I see a difference of 60 sec from the > earlier startup before service pacemaker status shows up as started. Don't start fenced manually, just start pacemaker and let it handle everything. Ideally, use the pcs command (and pcsd daemon on the nodes) to start/stop the cluster, but you'll need to upgrade to 6.7. > 2. Fencing test cases. > Based on the internet queries i could find , apart from plugging out > the dedicated cable. The only other case suggested is killing corosync > process on one of the nodes. > Are there any other basic cases that i should look at ? > What about bring up interface down manually ? I understand that this is > an unlikely scenario but i am just looking for more ways to test this out. echo c > /proc/sysrq-trigger == kernel panic. It's my preferred test. Also, killing the power to the node will cause IPMI to fail and will test your backup fence method, if you have it, or ensure the cluster locks up if you don't (better to hang than to risk corruption). > 3. Testing whether fencing is working or not. > Previously i have been using fence_ilo4 from the shell to test whether > the command is working. I was assuming that similar invocation would be > done by stonith when actual fencing needs to be done. > > However based on other threads i could find people also use fence_tool > to try this out. According to me this tests out whether > fencing when invoked by fenced for a particular node succeeds or not. Is > that valid ? Fence tool is just a command to control the cluster's fencing. The fence_X agents do the actual work. > Since we are configuring fence_pcmk as the fence device the flow of > things is > fenced -> fence_pcmk -> stonith -> fence agent. Basically correct. > 4. Fencing agent to be used (fence_ipmilan vs fence_ilo4) > Also for ILO fencing i see fence_ilo4 and fence_ipmilan both available. > I had been using fence_ilo4 till now. Which ever works is fine. I believe a lot of the fence_X out-of-band agents are actually just links to fence_ipmilan, but I might be wrong. > I think this mail has multiple questions and i will probably send out > another mail for a few issues i see after fencing takes place. > > Thanks in advance > Arjun > > > ___ > Users mailing list: Users@clusterlabs.org > http://clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Cluster node loss detection.
On 16/10/15 11:40 AM, Vallevand, Mark K wrote: > Thanks. I wasn't completely aware of corosync's role in this. I see new > things in the docs every time I read them. > > I looked up the corosync settings at one time and did it again: > token loss 3000ms > retransmits 10 > So 30s. Redid my simple testing and got detection times of 22s, 26s, and 25s > using very crude methods. > Any warnings about setting these values to something else? > We require our customers to use an isolated, private network for cluster > communications. All taken care of in our instructions and cluster > configuration scripts. Network traffic will not be a factor. So, I'm > thinking 1000ms and 5 retransmits as an experiment. That is very high. I think the default is something like 236ms x 4 losses. You do have fencing, right? > I was pretty sure that DLM was just being informed by clustering, but I > needed to ask. > > Again, thanks. > > > Regards. > Mark K Vallevand mark.vallev...@unisys.com > <mailto:mark.vallev...@unisys.com> > Never try and teach a pig to sing: it's a waste of time, and it annoys the > pig. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Cluster node loss detection.
On 16/10/15 01:14 PM, Vallevand, Mark K wrote: > No stonith configured. Not explicitly anyway. > Does that factor into this somehow? Yes, you will eventually have a split-brain. All fencing in cman does with 'fence_pcmk' is say "hey, if you need to fence, ask pacemaker to do it". That's useless if pacemaker can't fence. > I've tested stonith, but we aren't doing it for customers. Maybe in the > future if someone cries or pays us money. > Our solution is deployed onto too many different machines. A couple of bare > metal. A couple of VMs. We don't want customers to need to figure out > stonith and we can't test all possible configurations and write instructions. > So, they get one-size-fits-all. https://alteeve.ca/w/AN!Cluster_Tutorial_2#Concept.3B_Fencing You are doing a disservice to your customers. Without fencing, you *will* have a bad day, it's just a question of when. I can't tell you how many times I've heard "but it worked fine for over a year!". Stonith is worth the hassle. > Regards. > Mark K Vallevand mark.vallev...@unisys.com > <mailto:mark.vallev...@unisys.com> > Never try and teach a pig to sing: it's a waste of time, and it annoys the > pig. > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is thus for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all computers. > > > -Original Message- > From: Digimer [mailto:li...@alteeve.ca] > Sent: Friday, October 16, 2015 11:51 AM > To: Cluster Labs - All topics related to open-source clustering welcomed > Subject: Re: [ClusterLabs] Cluster node loss detection. > > On 16/10/15 12:37 PM, Vallevand, Mark K wrote: >> Fencing, yes. I have pcmk-redirect for each node in cluster.conf. > > Do you have stonith configured (and tested!) in Pacemaker as well? > >> I run with default cman settings for corosync. No totem clause. That gives >> the 20s detection. Not sure what the defaults really are. >> I added to >> cluster.conf and get about a 5s detection. >> >> The corosync man page says: >>token This timeout specifies in milliseconds until a token loss is >> declared after not receiving a token. This is the time spent detecting a >> failure of a processor in the current configuration. >> Reforming a new configuration takes about 50 milliseconds in addition to >> this >> timeout. >> >> The default is 1000 milliseconds. >> >>token_retransmit >> This timeout specifies in milliseconds after how long before >> receiving a token the token is retransmitted. This will be automatically >> calculated if token is modified. It is not recommended to >> alter this value without guidance from the corosync community. >> >> The default is 238 milliseconds. >> >>hold This timeout specifies in milliseconds how long the token >> should be held by the representative when the protocol is under low utiliza‐ >> tion. It is not recommended to alter this value without >> guidance from the corosync community. >> >> The default is 180 milliseconds. >> >>token_retransmits_before_loss_const >> This value identifies how many token retransmits should >> be attempted before forming a new configuration. If this value is set, >> retransmit and hold will be automatically calculated from >> retransmits_before_loss and token. >> >> The default is 4 retransmissions. >> >> But, I don't know what cman sets these to. But, they aren't these values. >> And, they aren't the values in the cman man page, which says this: > > Maybe it's changed by the ubuntu packagers? I don't know, I don't use > debian or ubuntu. > >> Cman uses different defaults for some of the corosync >> parameters listed in corosync.conf(5). If you wish to use a non-default set‐ >> ting, they can be configured in cluster.conf as shown above. >> Cman uses the following default values: >> >> > vsftype="none" >> token="1" >> token_retransmits_before_loss_const="20" >> join="60" >> consensus="4800" >> rrp_mode="none" >> > From: Digimer [mailto:li...@alteeve.ca] &g
Re: [ClusterLabs] (no subject)
On 08/10/15 09:03 PM, TaMen说我挑食 wrote: > Corosync+Pacemaker error during failover You need to ask a question if you want us to be able to help you. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] multiple drives looks like balancing but why and causing troubles
-timeout-40) stop interval=0s timeout=20s (oomtrserver-stop-timeout-20s) monitor interval=10 timeout=20s (oomtrserver-monitor-interval-10) Group: oomblg Resource: oomblshare (class=ocf provider=heartbeat type=Filesystem) Attributes: device=/dev/sde1 directory=/oombl fstype=ext4 Operations: start interval=0s timeout=60 (oomblshare-start-timeout-60) stop interval=0s timeout=60 (oomblshare-stop-timeout-60) monitor interval=20 timeout=40 (oomblshare-monitor-interval-20) Resource: oomblserver (class=ocf provider=heartbeat type=nfsserver) Attributes: nfs_shared_infodir=/oombl/nfsinfo nfs_no_notify=true Operations: start interval=0s timeout=40 (oomblserver-start-timeout-40) stop interval=0s timeout=20s (oomblserver-stop-timeout-20s) monitor interval=10 timeout=20s (oomblserver-monitor-interval-10) Group: oombrg Resource: oombrshare (class=ocf provider=heartbeat type=Filesystem) Attributes: device=/dev/sdf1 directory=/oombr fstype=ext4 Operations: start interval=0s timeout=60 (oombrshare-start-timeout-60) stop interval=0s timeout=60 (oombrshare-stop-timeout-60) monitor interval=20 timeout=40 (oombrshare-monitor-interval-20) Resource: oombrserver (class=ocf provider=heartbeat type=nfsserver) Attributes: nfs_shared_infodir=/oombr/nfsinfo nfs_no_notify=true Operations: start interval=0s timeout=40 (oombrserver-start-timeout-40) stop interval=0s timeout=20s (oombrserver-stop-timeout-20s) monitor interval=10 timeout=20s (oombrserver-monitor-interval-10) Stonith Devices: Fencing Levels: Location Constraints: Ordering Constraints: Colocation Constraints: Cluster Properties: cluster-infrastructure: classic openais (with plugin) dc-version: 1.1.11-97629de expected-quorum-votes: 2 no-quorum-policy: ignore stonith-enabled: false Michelle Streeter ASC2 MCS – SDE/ACL/SDL/EDL OKC Software Engineer The Boeing Company ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] VG activation on Active/Passive
On 29/08/15 02:30 PM, Jorge Fábregas wrote: Hi, For an active/passive cluster, using a non-cluster filesystem like ext3 over LVM (using cLVM DLM), would you: No need for clustered LVM, only the active node should see the PV. When the passive takes over, after connecting to the PV, it should do a pvscan - vgscan - lvscan before mounting the FS on the LV. Keep you cluster simple; remove everything not needed. - include the VG activation in the same cloned group that hosts cLVM DLM? (top of screenshot) so that in both nodes the VG is activated (even though this is not a cluster-filesystem so there's no need to have the VG activated on both sides simultaneously)? or - do you create a separate group with the VG activation (LVM resource) and tie it with the Filesystem resource? (bottom of screenshot). This makes more sense to me since the VG only needs to be activated where the filesystem is going to be mounted. Here's my screenshot: http://oi60.tinypic.com/10pn9n6.jpg I belive this last one is THE way to go for such setup (active/passive) but I'm wondering why I have seen examples online with the first case. Maybe I'm missing something. Thanks! Jorge ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] VG activation on Active/Passive
On 29/08/15 02:51 PM, Jorge Fábregas wrote: On 08/29/2015 02:37 PM, Digimer wrote: No need for clustered LVM, only the active node should see the PV. When the passive takes over, after connecting to the PV, it should do a pvscan - vgscan - lvscan before mounting the FS on the LV. Keep you cluster simple; remove everything not needed. Hi Digimer, It would be great if my cluster could communicate with the storage-array to make it present or unpresent LUNs based on the cluster conditions but I'm afraid that's not possible :) So yes, these are LUNs that are visible on both nodes. I'm the one who asked you something regarding cLVM on IRC the other day (one of the many times you have helped me there) and I decided to go the cLVM/DLM route (because the LUNs are seen on both nodes) but now I have my issues with this VG activation. Thanks. Regards, Jorge Ah, OK then. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] SBD & Failed Peer
On 06/09/15 09:28 PM, Jorge Fábregas wrote: > On 09/06/2015 04:23 PM, Jorge Fábregas wrote: >> Assume an active/active cluster using OCFS2 and SBD with shared storage. >> Then one node explodes (the hardware watchdog is gone as well >> obviously). > > Ok I did two tests with this setup on my KVM lab (one with SBD with > shared-storage and the other with hypervisor-based STONITH > (external/libvirt) while actively writing to an ocfs2 filesystem. > > > ## SBD with shared-storage > > Shut off one node abruptly (VM power-off) . Result: DLM/OCFS2 blocked > for about 30 to 40 seconds and then it resumed. That's nice! I think > at this moment (when resuming) the assumptions were: And this is why I am nervous; It is always ideal to have a primary fence method that has a method of confirming the 'off' state. IPMI fencing can do this, as can hypervisor-based fence methods like fence_virsh and fence_xvm. Now I say this as a someone who uses PDU-based fencing for backup, which has the same problem... It can't verify the 'off' and a human could potentially change the plugs around (I personally deal with this by mechanically strapping the cables in place). > -if the peer were alive it would have swallowed the poison pill we just > placed > - if the peer is freezed the watchdog would have taken care of him > - we just wait a little extra bit before continuing... > > (I really don't know if checking when was the last update of your > partner - on the SBD disk- is part of the role of the SBD daemon) > > > ## External/Libvirt > > I shut off one node but then disabled SSH on KVM host (so that fencing > via qemu+ssh couldn't work). Result: it blocked FOREVER. Right; Fencing is not allowed to make assumptions. If the fence action can't be confirmed to have succeeded, it's better to lock up than to risk corruption. When fence_virsh works, you *know* it worked, so it's ideal as a primary fence method. > Am I right in thinking that SBD is the way to go when using OCFS2 > filesystems? (compared to hypervisor-based fencing or management-boards > like iLO, DRAC etc)? I would use IPMI (iLO, DRAC, etc) as the primary fence method and something else as a secondary, backup method. You can use SBD + watchdog as the backup method, or as I do, a pair of switched PDUs (I find APC brand to be very fast in fencing). > Now, the only thing I don't like about SBD is that when it loses contact > with the shared disk, both nodes commit suicide. I found out there's > the "-P" option to SBD (that's supposed to prevent that as long as > there's cluster communication) but it doesn't work in my SLES 11 SP4 > setup. Maybe it's on SLES 12. > > > Thanks, > Jorge > > ___ > Users mailing list: Users@clusterlabs.org > http://clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] Problem with fence_virsh in RHEL 6 - selinux denial
Hi all,
I've been using KVM-based VMs as a testbed for clusters for ages,
always using fence_virsh.
I noticed today though that fence_virsh is now being blocked by
selinux (rhel 6.7, fully updated as of today):
type=AVC msg=audit(1441752343.878:3269): avc: denied { execute } for
pid=8848 comm="fence_virsh" name="ssh" dev=vda2 ino=2103935
scontext=unconfined_u:system_r:fenced_t:s0
tcontext=system_u:object_r:ssh_exec_t:s0 tclass=file
type=SYSCALL msg=audit(1441752343.878:3269): arch=c03e syscall=21
success=no exit=-13 a0=1a363a0 a1=1 a2=7f02aa7f89e8 a3=7ffdff0dc7c0
items=0 ppid=7759 pid=8848 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0
egid=0 sgid=0 fsgid=0 tty=(none) ses=27 comm="fence_virsh"
exe="/usr/bin/python" subj=unconfined_u:system_r:fenced_t:s0 key=(null)
t
[root@node1 ~]# rpm -q fence-agents cman corosync
fence-agents-4.0.15-8.el6.x86_64
cman-3.0.12.1-73.el6.1.x86_64
corosync-1.4.7-2.el6.x86_64
[root@node1 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.7 (Santiago)
I'll post a follow-up if I can sort out how to fix it. My selinux-fu is
weak...
--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] [ClusterLabs Developers] Problem with fence_virsh in RHEL 6 - selinux denial
id=2625 pid=23581 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0
sgid=0 fsgid=0 tty=pts1 ses=2 comm="setenforce"
exe="/usr/sbin/setenforce"
subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
type=AVC msg=audit(1441767229.702:9367): avc: denied { execute } for
pid=23606 comm="fence_virsh" name="ssh" dev=vda2 ino=2103935
scontext=unconfined_u:system_r:fenced_t:s0
tcontext=system_u:object_r:ssh_exec_t:s0 tclass=file
type=SYSCALL msg=audit(1441767229.702:9367): arch=c03e syscall=21
success=yes exit=0 a0=16a11a0 a1=1 a2=7f81b57009e8 a3=7ffc2776dc10
items=0 ppid=2879 pid=23606 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0
egid=0 sgid=0 fsgid=0 tty=(none) ses=2 comm="fence_virsh"
exe="/usr/bin/python" subj=unconfined_u:system_r:fenced_t:s0 key=(null)
type=AVC msg=audit(1441767229.705:9368): avc: denied { read open } for
pid=23611 comm="fence_virsh" name="ssh" dev=vda2 ino=2103935
scontext=unconfined_u:system_r:fenced_t:s0
tcontext=system_u:object_r:ssh_exec_t:s0 tclass=file
type=AVC msg=audit(1441767229.705:9368): avc: denied {
execute_no_trans } for pid=23611 comm="fence_virsh" path="/usr/bin/ssh"
dev=vda2 ino=2103935 scontext=unconfined_u:system_r:fenced_t:s0
tcontext=system_u:object_r:ssh_exec_t:s0 tclass=file
type=SYSCALL msg=audit(1441767229.705:9368): arch=c03e syscall=59
success=yes exit=0 a0=169f4a0 a1=164ac60 a2=168b620 a3=7ffc2776dd50
items=0 ppid=23606 pid=23611 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0
egid=0 sgid=0 fsgid=0 tty=pts2 ses=2 comm="ssh" exe="/usr/bin/ssh"
subj=unconfined_u:system_r:fenced_t:s0 key=(null)
type=AVC msg=audit(1441767229.707:9369): avc: denied { setuid } for
pid=23611 comm="ssh" capability=7
scontext=unconfined_u:system_r:fenced_t:s0
tcontext=unconfined_u:system_r:fenced_t:s0 tclass=capability
type=SYSCALL msg=audit(1441767229.707:9369): arch=c03e syscall=117
success=yes exit=0 a0= a1=0 a2= a3=3
items=0 ppid=23606 pid=23611 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0
egid=0 sgid=0 fsgid=0 tty=pts2 ses=2 comm="ssh" exe="/usr/bin/ssh"
subj=unconfined_u:system_r:fenced_t:s0 key=(null)
type=AVC msg=audit(1441767229.708:9370): avc: denied { search } for
pid=23611 comm="ssh" name=".ssh" dev=vda2 ino=1966197
scontext=unconfined_u:system_r:fenced_t:s0
tcontext=system_u:object_r:ssh_home_t:s0 tclass=dir
type=SYSCALL msg=audit(1441767229.708:9370): arch=c03e syscall=2
success=no exit=-2 a0=7ffed853ecd0 a1=0 a2=1b6 a3=0 items=0 ppid=23606
pid=23611 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0
tty=pts2 ses=2 comm="ssh" exe="/usr/bin/ssh"
subj=unconfined_u:system_r:fenced_t:s0 key=(null)
type=AVC msg=audit(1441767229.709:9371): avc: denied { name_connect }
for pid=23611 comm="ssh" dest=22
scontext=unconfined_u:system_r:fenced_t:s0
tcontext=system_u:object_r:ssh_port_t:s0 tclass=tcp_socket
type=SYSCALL msg=audit(1441767229.709:9371): arch=c03e syscall=42
success=yes exit=0 a0=3 a1=7fa79084eb30 a2=10 a3=fee0
items=0 ppid=23606 pid=23611 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0
egid=0 sgid=0 fsgid=0 tty=pts2 ses=2 comm="ssh" exe="/usr/bin/ssh"
subj=unconfined_u:system_r:fenced_t:s0 key=(null)
type=AVC msg=audit(1441767229.710:9372): avc: denied { setgid } for
pid=23611 comm="ssh" capability=6
scontext=unconfined_u:system_r:fenced_t:s0
tcontext=unconfined_u:system_r:fenced_t:s0 tclass=capability
type=SYSCALL msg=audit(1441767229.710:9372): arch=c03e syscall=119
success=yes exit=0 a0=0 a1=0 a2=0 a3=e items=0 ppid=23606 pid=23611
auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts2
ses=2 comm="ssh" exe="/usr/bin/ssh"
subj=unconfined_u:system_r:fenced_t:s0 key=(null)
type=AVC msg=audit(1441767229.710:9373): avc: denied { getattr } for
pid=23611 comm="ssh" path="/root/.ssh" dev=vda2 ino=1966197
scontext=unconfined_u:system_r:fenced_t:s0
tcontext=system_u:object_r:ssh_home_t:s0 tclass=dir
type=SYSCALL msg=audit(1441767229.710:9373): arch=c03e syscall=4
success=yes exit=0 a0=7ffed853ecd0 a1=7ffed853ec40 a2=7ffed853ec40 a3=0
items=0 ppid=23606 pid=23611 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0
egid=0 sgid=0 fsgid=0 tty=pts2 ses=2 comm="ssh" exe="/usr/bin/ssh"
subj=unconfined_u:system_r:fenced_t:s0 key=(null)
type=AVC msg=audit(1441767229.710:9374): avc: denied { read } for
pid=23611 comm="ssh" name="id_rsa" dev=vda2 ino=1966200
scontext=unconfined_u:system_r:fenced_t:s0
tcontext=unconfined_u:object_r:ssh_home_t:s0 tclass=file
type=AVC msg=audit(1441767229.710:9374): avc: denied { open } for
pid=23611 comm="ssh" name="id_rsa" dev=vda2 ino=1966200
scontext=unconfined_u:system_r:fenced_t:s0
tcontext=unconfined_u:object_r:ssh_home_
Re: [ClusterLabs] Coming in 1.1.14: Fencing topology based on node attribute
On 08/09/15 11:23 AM, Ken Gaillot wrote: > Pacemaker's upstream master branch has a new feature that will be part > of the eventual 1.1.14 release. > > Fencing topology is used when a node requires multiple fencing devices > (in combination or as fallbacks). Currently, topologies must be > specified by node name (or a regular expression matching node names). > > The new feature allows topologies to specified by node attribute. > > For example, imagine a data center where all devices in rack #1 use > fence devices apc01 and apc02, while all devices in rack #2 use fence > devices apc03 and apc04. > > Previously, if node1 was in rack #1, you'd have to register a fencing > topology by its name, which at the XML level would look like: > > > devices="apc01,apc02"/> > > > > With the new feature, you could instead register a topology for all > hosts that have a node attribute "rack" whose value is "1": > > > devices="apc01,apc02"/> > > > > You would assign that attribute to all nodes in that rack, e.g.: > >crm_attribute --type nodes --node node1 --name rack --update 1 > > > If a server is moved to a different rack, simply update the value of its > attribute and it will use that rack's fencing configuration. Or if a > rack gets a new fencing device, you only have to update the fencing > configuration once rather than for every node in the rack. > > The syntax accepts either '=' or ':' as the separator for the name/value > pair, so target="rack:1" would work in the XML as well. Holy crap that is awesome! :D -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] Clustered LVM with iptables issue
Hi all, I've hit another recent, odd issue. Since adding RRP, I can't start clvmd anymore if the iptables rules are in place. Starting clvmd sits there and eventually times out with rc=5. If I drop iptables, it works perfectly. From what I understand, clvmd uses dlm and corosync, so it shouldn't need its own ports. Obviously I am wrong though... What ports/protocols are needed for clvmd to work right? It's a RHEL 6.7 box, in case it matters. Here's my 'iptables-save' (10.20.0.0/16 is the back-channel that corosync used to use exclusively. 10.10.0.0/16 is the storage network that corosync's backup ring uses now. 10.255.0.0/16 is the internet-facing network and is not used by anything cluster related): [root@node1 ~]# iptables-save # Generated by iptables-save v1.4.7 on Thu Sep 10 22:12:38 2015 *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [50:5318] -A INPUT -s 192.168.122.0/24 -d 192.168.122.0/24 -p tcp -m state --state NEW -m tcp --dport 5900:6000 -j ACCEPT -A INPUT -s 10.20.0.0/16 -d 10.20.0.0/16 -p tcp -m state --state NEW -m tcp --dport 5900:6000 -j ACCEPT -A INPUT -s 10.20.0.0/16 -d 10.20.0.0/16 -p tcp -m tcp --dport 49152:49216 -j ACCEPT -A INPUT -s 10.10.0.0/16 -d 10.10.0.0/16 -p tcp -m tcp --dport 49152:49216 -j ACCEPT -A INPUT -s 10.10.0.0/16 -d 10.10.0.0/16 -p tcp -m state --state NEW -m tcp --dport 7789 -j ACCEPT -A INPUT -s 10.10.0.0/16 -d 10.10.0.0/16 -p tcp -m state --state NEW -m tcp --dport 7788 -j ACCEPT -A INPUT -p igmp -j ACCEPT -A INPUT -s 10.20.0.0/16 -d 10.20.0.0/16 -p tcp -m state --state NEW -m tcp --dport 16851 -j ACCEPT -A INPUT -s 10.10.0.0/16 -d 10.10.0.0/16 -p tcp -m state --state NEW -m tcp --dport 16851 -j ACCEPT -A INPUT -s 10.20.0.0/16 -d 10.20.0.0/16 -p tcp -m state --state NEW -m tcp --dport 1 -j ACCEPT -A INPUT -s 10.10.0.0/16 -d 10.10.0.0/16 -p tcp -m state --state NEW -m tcp --dport 1 -j ACCEPT -A INPUT -s 10.20.0.0/16 -d 10.20.0.0/16 -p tcp -m state --state NEW -m tcp --dport 21064 -j ACCEPT -A INPUT -s 10.10.0.0/16 -d 10.10.0.0/16 -p tcp -m state --state NEW -m tcp --dport 21064 -j ACCEPT -A INPUT -s 10.20.0.0/16 -p udp -m addrtype --dst-type MULTICAST -m state --state NEW -m multiport --dports 5404,5405 -j ACCEPT -A INPUT -s 10.20.0.0/16 -d 10.20.0.0/16 -p udp -m state --state NEW -m multiport --dports 5404,5405 -j ACCEPT -A INPUT -s 10.10.0.0/16 -p udp -m addrtype --dst-type MULTICAST -m state --state NEW -m multiport --dports 5404,5405 -j ACCEPT -A INPUT -s 10.10.0.0/16 -d 10.10.0.0/16 -p udp -m state --state NEW -m multiport --dports 5404,5405 -j ACCEPT -A INPUT -s 10.20.0.0/16 -d 10.20.0.0/16 -p udp -m state --state NEW -m udp --dport 123 -j ACCEPT -A INPUT -s 192.168.122.0/24 -d 192.168.122.0/24 -p udp -m state --state NEW -m udp --dport 123 -j ACCEPT -A INPUT -s 10.20.0.0/16 -d 10.20.0.0/16 -p tcp -m state --state NEW -m tcp --dport 5900 -j ACCEPT -A INPUT -s 10.20.0.0/16 -d 10.20.0.0/16 -p tcp -m state --state NEW -m tcp --dport 5800 -j ACCEPT -A INPUT -s 192.168.122.0/24 -d 192.168.122.0/24 -p tcp -m state --state NEW -m tcp --dport 5900 -j ACCEPT -A INPUT -s 192.168.122.0/24 -d 192.168.122.0/24 -p tcp -m state --state NEW -m tcp --dport 5800 -j ACCEPT -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited -A FORWARD -j REJECT --reject-with icmp-host-prohibited COMMIT # Completed on Thu Sep 10 22:12:38 2015 Any help is appreciated! -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Clustered LVM with iptables issue
On 10/09/15 06:31 PM, Noel Kuntze wrote: > > Hello Digimer, > > Pro tip: look at the 'multiport' module. You can substantially reduce the > number of rules with it. > Right now, I'm scratching my eyes out. > You can use `ss` or `netstat` to find out where clmvd wants to phone to. That > might be > an additional lead. Or use tcpdump. > But please, tidy up your rules. The rules are as terse as I thought I could make them. ss shows no difference: [root@node1 ~]# /etc/init.d/clvmd start Starting clvmd: Activating VG(s): [ OK ] [root@node1 ~]# ss State Recv-Q Send-Q Local Address:Port Peer Address:Port ESTAB 0 0 192.168.122.10:ssh 192.168.122.1:53935 ESTAB 0 0 192.168.122.10:ssh 192.168.122.1:53934 ESTAB 0 0 10.10.10.1:48985 10.10.10.2:7788 ESTAB 0 0 10.10.10.1:7788 10.10.10.2:51681 ESTAB 0 0 :::10.20.10.1:16851 :::10.20.10.2:43553 [root@node1 ~]# /etc/init.d/clvmd stop Signaling clvmd to exit[ OK ] clvmd terminated [ OK ] [root@node1 ~]# ss State Recv-Q Send-Q Local Address:Port Peer Address:Port ESTAB 0 0 192.168.122.10:ssh 192.168.122.1:53935 ESTAB 0 0 192.168.122.10:ssh 192.168.122.1:53934 ESTAB 0 0 10.10.10.1:48985 10.10.10.2:7788 ESTAB 0 0 10.10.10.1:7788 10.10.10.2:51681 ESTAB 0 0 :::10.20.10.1:16851 :::10.20.10.2:43553 [root@node1 ~]# netcat netstat had a lot more output, so I pushed the output to files and diff'ed them: [root@node1 ~]# netstat > 1 [root@node1 ~]# /etc/init.d/clvmd start Starting clvmd: Activating VG(s): [ OK ] [root@node1 ~]# netstat > 2 [root@node1 ~]# diff -U0 1 2 --- 1 2015-09-10 22:46:31.27503 + +++ 2 2015-09-10 22:46:51.04411 + @@ -7,0 +8,2 @@ +sctp 0 0 node1.bcn:21064 node2.bcn:21064 ESTABLISHED +node1.snnode2.sn @@ -12 +14,6 @@ -unix 15 [ ] DGRAM12986 /dev/log +unix 16 [ ] DGRAM12986 /dev/log +unix 2 [ ] DGRAM23743 +unix 3 [ ] STREAM CONNECTED 23689 @corosync.ipc +unix 3 [ ] STREAM CONNECTED 23688 +unix 3 [ ] STREAM CONNECTED 23685 /var/run/cman_client +unix 3 [ ] STREAM CONNECTED 23684 I'm not familiar with netstat, so I'll need to read up to understand the differences and how to translate them to iptables rules. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Clustered LVM with iptables issue
On 10/09/15 06:54 PM, Noel Kuntze wrote:
>
> Hello Digimer,
>
> I initially assumed you were familiar with ss or netstat and simply
> forgot about them.
> Seems I was wrong.
>
> Check the output of this: `ss -tpn` and `ss -upn`.
> Those commands give you the current open TCP and UDP connections,
> as well as the program that opened the connection.
> Check listening sockets with `ss -tpnl` and `ss -upnl`
I'm not so strong on the network side of things, so I am not very
familiar with ss or netstat.
I have clvmd running:
[root@node1 ~]# /etc/init.d/clvmd status
clvmd (pid 3495) is running...
Clustered Volume Groups: (none)
Active clustered Logical Volumes: (none)
Thought I don't seem to see anything:
[root@node1 ~]# ss -tpnl
State Recv-Q Send-Q Local Address:Port
Peer Address:Port
LISTEN 0 5 :::1
:::* users:(("ricci",2482,3))
LISTEN 0 128 127.0.0.1:199
*:* users:(("snmpd",2020,8))
LISTEN 0 128 :::111
:::* users:(("rpcbind",1763,11))
LISTEN 0 128 *:111
*:* users:(("rpcbind",1763,8))
LISTEN 0 128 *:48976
*:* users:(("rpc.statd",1785,8))
LISTEN 0 5 :::16851
:::* users:(("modclusterd",2371,5))
LISTEN 0 128 :::55476
:::* users:(("rpc.statd",1785,10))
LISTEN 0 128 :::22
:::* users:(("sshd",2037,4))
LISTEN 0 128 *:22
*:* users:(("sshd",2037,3))
LISTEN 0 100::1:25
:::* users:(("master",2142,13))
LISTEN 0 100 127.0.0.1:25
*:* users:(("master",2142,12))
[root@node1 ~]# ss -tpn
State Recv-Q Send-Q Local Address:Port
Peer Address:Port
ESTAB 0 0 192.168.122.10:22
192.168.122.1:53935 users:(("sshd",2636,3))
ESTAB 0 0 192.168.122.10:22
192.168.122.1:53934 users:(("sshd",2613,3))
ESTAB 0 0 10.10.10.1:48985
10.10.10.2:7788
ESTAB 0 0 10.10.10.1:7788
10.10.10.2:51681
ESTAB 0 0:::10.20.10.1:16851
:::10.20.10.2:43553 users:(("modclusterd",2371,6))
[root@node1 ~]# ss -upn
State Recv-Q Send-Q Local Address:Port
Peer Address:Port
I ran all three again and routed output to a file, stopped clvmd and
re-ran the three calls to a different file. I diff'ed the resulting
files and saw nothing of interest:
[root@node1 ~]# /etc/init.d/clvmd status
clvmd (pid 3495) is running...
Clustered Volume Groups: (none)
Active clustered Logical Volumes: (none)
[root@node1 ~]# ss -tpnl > tpnl.on
[root@node1 ~]# ss -tpn > tpn.on
[root@node1 ~]# ss -upn > upn.on
[root@node1 ~]# /etc/init.d/clvmd stop
Signaling clvmd to exit[ OK ]
clvmd terminated [ OK ]
[root@node1 ~]# ss -tpnl > tpnl.off
[root@node1 ~]# ss -tpn > tpn.off
[root@node1 ~]# ss -upn > upn.off
[root@node1 ~]# diff -U0 tpnl.on tpnl.off
[root@node1 ~]# diff -U0 tpn.on tpn.off
[root@node1 ~]# diff -U0 upn.on upn.off
I'm reading up on 'multiport' now and will adjust my iptables. It does
look a lot cleaner.
--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Clustered LVM with iptables issue
For the record;
Noel helped me on IRC. The problem was that sctp was now allowed in
the firewall. The clue was:
[root@node1 ~]# /etc/init.d/clvmd start
Starting clvmd:
Activating VG(s): [ OK ]
] syslog
Sep 10 23:30:47 node1 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Sep 10 23:30:47 node1 kernel: nf_conntrack version 0.5.0 (16384 buckets,
65536 max)
*** Sep 10 23:31:02 node1 kernel: dlm: Using SCTP for communications
Sep 10 23:31:03 node1 clvmd: Cluster LVM daemon started - connected to CMAN
[root@node2 ~]# /etc/init.d/clvmd start
Starting clvmd: clvmd startup timed out
] syslog
Sep 10 23:31:03 node2 kernel: dlm: Using SCTP for communications
Sep 10 23:31:05 node2 corosync[3001]: [TOTEM ] Incrementing problem
counter for seqid 5644 iface 10.20.10.2 to [1 of 3]
Sep 10 23:31:07 node2 corosync[3001]: [TOTEM ] ring 0 active with no
faults
Adding;
iptables -I INPUT -p sctp -j ACCEPT
Got it working. Obviously, that needs to be tightened up.
digimer
On 10/09/15 07:01 PM, Digimer wrote:
> On 10/09/15 06:54 PM, Noel Kuntze wrote:
>>
>> Hello Digimer,
>>
>> I initially assumed you were familiar with ss or netstat and simply
>> forgot about them.
>> Seems I was wrong.
>>
>> Check the output of this: `ss -tpn` and `ss -upn`.
>> Those commands give you the current open TCP and UDP connections,
>> as well as the program that opened the connection.
>> Check listening sockets with `ss -tpnl` and `ss -upnl`
>
> I'm not so strong on the network side of things, so I am not very
> familiar with ss or netstat.
>
> I have clvmd running:
>
>
> [root@node1 ~]# /etc/init.d/clvmd status
> clvmd (pid 3495) is running...
> Clustered Volume Groups: (none)
> Active clustered Logical Volumes: (none)
>
>
> Thought I don't seem to see anything:
>
>
> [root@node1 ~]# ss -tpnl
> State Recv-Q Send-Q Local Address:Port
> Peer Address:Port
> LISTEN 0 5 :::1
> :::* users:(("ricci",2482,3))
> LISTEN 0 128 127.0.0.1:199
> *:* users:(("snmpd",2020,8))
> LISTEN 0 128 :::111
> :::* users:(("rpcbind",1763,11))
> LISTEN 0 128 *:111
> *:* users:(("rpcbind",1763,8))
> LISTEN 0 128 *:48976
> *:* users:(("rpc.statd",1785,8))
> LISTEN 0 5 :::16851
> :::* users:(("modclusterd",2371,5))
> LISTEN 0 128 :::55476
> :::* users:(("rpc.statd",1785,10))
> LISTEN 0 128 :::22
> :::* users:(("sshd",2037,4))
> LISTEN 0 128 *:22
> *:* users:(("sshd",2037,3))
> LISTEN 0 100::1:25
> :::* users:(("master",2142,13))
> LISTEN 0 100 127.0.0.1:25
> *:* users:(("master",2142,12))
>
>
>
> [root@node1 ~]# ss -tpn
> State Recv-Q Send-Q Local Address:Port
> Peer Address:Port
> ESTAB 0 0 192.168.122.10:22
> 192.168.122.1:53935 users:(("sshd",2636,3))
> ESTAB 0 0 192.168.122.10:22
> 192.168.122.1:53934 users:(("sshd",2613,3))
> ESTAB 0 0 10.10.10.1:48985
> 10.10.10.2:7788
> ESTAB 0 0 10.10.10.1:7788
> 10.10.10.2:51681
> ESTAB 0 0:::10.20.10.1:16851
> :::10.20.10.2:43553 users:(("modclusterd",2371,6))
>
>
>
> [root@node1 ~]# ss -upn
> State Recv-Q Send-Q Local Address:Port
> Peer Address:Port
>
>
> I ran all three again and routed output to a file, stopped clvmd and
> re-ran the three calls to a different file. I diff'ed the resulting
> files and saw nothing of intere
Re: [ClusterLabs] Adding 'virsh migrate migrate-setspeed' support to the vm RA
On 14/09/15 07:19 AM, Dejan Muhamedagic wrote: > Hi Digimer, > > On Fri, Sep 04, 2015 at 03:36:09AM -0400, Digimer wrote: >> Hi all, >> >> I hit an issue a little while ago where live-migrating a VM (on the >> same management network normally used for corosync and a few other >> monitoring tasks) ate up all the bandwidth, causing corosync to declare >> a failure and triggering a fence. >> >> I've dealt with this, in part, by adding redundant ring support on a >> different network. However, I'd like to also limit the migration >> bandwidth so that I don't need to fail over the ring in the first place. > > As you certainly know, heavy traffic networks are not well suited > for the latency sensitive cluster communication. Best to have the > two separated. Oh for sure, but I've already hit 6 NICs per node (back-channel, normally just corosync, storage dedicated to DRBD and internet-facing dedicated to the user app). Adding two more NICs just for live migration, which happens very rarely, is hard to justify, despite this issue. If I can't reliably solve it with this (and/or tc and/or rrp...) then I will revisit this option. >> Is it reasonable/feasible to add 'virsh migrate-setspeed' support to >> the vm.sh RA? Something like; 'setspeed="75MiB"'? > > Yes, sounds like a good idea. Care to open an issue in github? > > Thanks, > > Dejan Done. https://github.com/ClusterLabs/resource-agents/issues/679 -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] EL6, cman, rrp, unicast and iptables
On 14/09/15 10:46 AM, Noel Kuntze wrote: > > Hello Christine, > > I googled a bit and some doc[1] says that TC_PRIO_INTERACTIVE maps to value > 6, whatever that is. > Assuming that value of 6 is the same as the "priority value", Corosync > traffic should go into band 0, because > TOS values of 0x10 and 0x14 have "priority value" 6, too. The page[2] on > lartc.org says that, too. > > That means that at least when pfifo_fast is used, there's no need for > iptables rules or tc filters to prioritize Corosync traffic manually. > > [1] http://linux-tc-notes.sourceforge.net/tc/doc/priority.txt > [2] http://lartc.org/howto/lartc.qdisc.classless.html#AEN658 So what's the final verdict on this? I followed your back and forth, and it sounds like corosync uses 0, so nothing else is to be done? I'm also fully willing to admit that something else triggered the fault detection. It happened during a long live migration (actually, several servers back to back), so I *assumed* that was the cause. Given it was a cut-over weekend though, I made a mental note and went back to work. Bad choice... I should have snagged the logs for later investigation. =/ digimer -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] EL6, cman, rrp, unicast and iptables
On 14/09/15 04:20 AM, Jan Friesse wrote:
> Digimer napsal(a):
>> Hi all,
>>
>>Starting a new thread from the "Clustered LVM with iptables issue"
>> thread...
>>
>>I've decided to review how I do networking entirely in my cluster. I
>> make zero claims to being great at networks, so I would love some
>> feedback.
>>
>>I've got three active/passive bonded interfaces; Back-Channel, Storage
>> and Internet-Facing networks. The IFN is "off limits" to the cluster as
>> it is dedicated to hosted server traffic only.
>>
>>So before, I uses only the BCN for cluster traffic for cman/corosync
>> multicast traffic, no rrp. A couple months ago, I had a cluster
>> partition when VM live migration (also on the BCN) congested the
>> network. So I decided to enable RRP using the SN as backup, which has
>> been marginally successful.
>>
>>Now, I want to switch to unicast (> the SN as the backup and BCN as the primary ring and do a proper
>> IPTables firewall. Is this sane?
>>
>>When I stopped iptables entirely and started cman with unicast + RRP,
>> I saw this:
>>
>> ] Node 1
>> Sep 11 17:31:24 node1 kernel: DLM (built Aug 10 2015 09:45:36) installed
>> Sep 11 17:31:24 node1 corosync[2523]: [MAIN ] Corosync Cluster Engine
>> ('1.4.7'): started and ready to provide service.
>> Sep 11 17:31:24 node1 corosync[2523]: [MAIN ] Corosync built-in
>> features: nss dbus rdma snmp
>> Sep 11 17:31:24 node1 corosync[2523]: [MAIN ] Successfully read
>> config from /etc/cluster/cluster.conf
>> Sep 11 17:31:24 node1 corosync[2523]: [MAIN ] Successfully parsed
>> cman config
>> Sep 11 17:31:24 node1 corosync[2523]: [TOTEM ] Initializing transport
>> (UDP/IP Unicast).
>> Sep 11 17:31:24 node1 corosync[2523]: [TOTEM ] Initializing
>> transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
>> Sep 11 17:31:24 node1 corosync[2523]: [TOTEM ] Initializing transport
>> (UDP/IP Unicast).
>> Sep 11 17:31:24 node1 corosync[2523]: [TOTEM ] Initializing
>> transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
>> Sep 11 17:31:24 node1 corosync[2523]: [TOTEM ] The network interface
>> [10.20.10.1] is now up.
>> Sep 11 17:31:24 node1 corosync[2523]: [QUORUM] Using quorum provider
>> quorum_cman
>> Sep 11 17:31:24 node1 corosync[2523]: [SERV ] Service engine loaded:
>> corosync cluster quorum service v0.1
>> Sep 11 17:31:24 node1 corosync[2523]: [CMAN ] CMAN 3.0.12.1 (built
>> Jul 6 2015 05:30:35) started
>> Sep 11 17:31:24 node1 corosync[2523]: [SERV ] Service engine loaded:
>> corosync CMAN membership service 2.90
>> Sep 11 17:31:24 node1 corosync[2523]: [SERV ] Service engine loaded:
>> openais checkpoint service B.01.01
>> Sep 11 17:31:24 node1 corosync[2523]: [SERV ] Service engine loaded:
>> corosync extended virtual synchrony service
>> Sep 11 17:31:24 node1 corosync[2523]: [SERV ] Service engine loaded:
>> corosync configuration service
>> Sep 11 17:31:24 node1 corosync[2523]: [SERV ] Service engine loaded:
>> corosync cluster closed process group service v1.01
>> Sep 11 17:31:24 node1 corosync[2523]: [SERV ] Service engine loaded:
>> corosync cluster config database access v1.01
>> Sep 11 17:31:24 node1 corosync[2523]: [SERV ] Service engine loaded:
>> corosync profile loading service
>> Sep 11 17:31:24 node1 corosync[2523]: [QUORUM] Using quorum provider
>> quorum_cman
>> Sep 11 17:31:24 node1 corosync[2523]: [SERV ] Service engine loaded:
>> corosync cluster quorum service v0.1
>> Sep 11 17:31:24 node1 corosync[2523]: [MAIN ] Compatibility mode set
>> to whitetank. Using V1 and V2 of the synchronization engine.
>> Sep 11 17:31:24 node1 corosync[2523]: [TOTEM ] adding new UDPU member
>> {10.20.10.1}
>> Sep 11 17:31:24 node1 corosync[2523]: [TOTEM ] adding new UDPU member
>> {10.20.10.2}
>> Sep 11 17:31:24 node1 corosync[2523]: [TOTEM ] The network interface
>> [10.10.10.1] is now up.
>> Sep 11 17:31:24 node1 corosync[2523]: [TOTEM ] adding new UDPU member
>> {10.10.10.1}
>> Sep 11 17:31:24 node1 corosync[2523]: [TOTEM ] adding new UDPU member
>> {10.10.10.2}
>> Sep 11 17:31:27 node1 corosync[2523]: [TOTEM ] Incrementing problem
>> counter for seqid 1 iface 10.10.10.1 to [1 of 3]
>> Sep 11 17:31:27 node1 corosync[2523]: [TOTEM ] A processor joined or
>> left the membership and a new membership was formed.
>> Sep 11 17:31:27 node1 corosync[2523]: [CMAN ] quorum regained,
>> resuming
Re: [ClusterLabs] EL6, cman, rrp, unicast and iptables
On 15/09/15 12:10 PM, Noel Kuntze wrote: > > Hello Digimer, > >> So what's the final verdict on this? I followed your back and forth, and >> it sounds like corosync uses 0, so nothing else is to be done? > > Missing prioritization itself cannot be the cause of the problem. > Either some other application puts its packets into band 0 or the cause > of the problem is not to be found in this domain. > > The problem should be reproduced and diagnostics should be done. I'm off to trade show then client site for the next week or so, will test when I return. For now, I'll hold as it seems that RRP is doing the job. If not, I'll up the token retransmit timer. Thanks very much for all this info. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] [ClusterLabs Developers] Problem with fence_virsh in RHEL 6 - selinux denial
I've created an rhbz: https://bugzilla.redhat.com/show_bug.cgi?id=1261711 digimer On 08/09/15 11:04 PM, Digimer wrote: > ere is my cluster.conf, in case it matters: > > > [root@node1 ~]# cat /etc/cluster/cluster.conf > > > > > > > > >port="an-a02n01" delay="15" action="reboot" /> > > > > > > > >port="an-a02n02" action="reboot" /> > > > > > >ipaddr="192.168.122.1" login="root" passwd="it's a secret" /> > > > > > > >name="wait-for-drbd"/> > <script file="/etc/init.d/clvmd" name="clvmd"/> > <clusterfs device="/dev/node1_vg0/shared" > force_unmount="1" > fstype="gfs2" mountpoint="/shared" name="sharedfs" /> > <script file="/etc/init.d/libvirtd" name="libvirtd"/> > </resources> > <failoverdomains> > <failoverdomain name="only_n01" nofailback="1" > ordered="0" > restricted="1"> > <failoverdomainnode name="node1.ccrs.bcn"/> > </failoverdomain> > <failoverdomain name="only_n02" nofailback="1" > ordered="0" > restricted="1"> > <failoverdomainnode name="node2.ccrs.bcn"/> > </failoverdomain> > <failoverdomain name="primary_n01" nofailback="1" > ordered="1" > restricted="1"> > <failoverdomainnode name="node1.ccrs.bcn" > priority="1"/> > <failoverdomainnode name="node2.ccrs.bcn" > priority="2"/> > </failoverdomain> > <failoverdomain name="primary_n02" nofailback="1" > ordered="1" > restricted="1"> > <failoverdomainnode name="node1.ccrs.bcn" > priority="2"/> > <failoverdomainnode name="node2.ccrs.bcn" > priority="1"/> > </failoverdomain> > </failoverdomains> > <service name="storage_n01" autostart="1" domain="only_n01" > exclusive="0" recovery="restart"> > <script ref="drbd"> > <script ref="wait-for-drbd"> > <script ref="clvmd"> > <clusterfs ref="sharedfs"/> > > > > >exclusive="0" recovery="restart"> > > <script ref="wait-for-drbd"> > <script ref="clvmd"> > <clusterfs ref="sharedfs"/> > > > > >exclusive="0" recovery="restart"> > > >exclusive="0" recovery="restart"> > > > > > -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] EL6, cman, rrp, unicast and iptables
On 15/09/15 03:20 AM, Jan Friesse wrote:
> Digimer napsal(a):
>> On 14/09/15 04:20 AM, Jan Friesse wrote:
>>> Digimer napsal(a):
>>>> Hi all,
>>>>
>>>> Starting a new thread from the "Clustered LVM with iptables issue"
>>>> thread...
>>>>
>>>> I've decided to review how I do networking entirely in my
>>>> cluster. I
>>>> make zero claims to being great at networks, so I would love some
>>>> feedback.
>>>>
>>>> I've got three active/passive bonded interfaces; Back-Channel,
>>>> Storage
>>>> and Internet-Facing networks. The IFN is "off limits" to the cluster as
>>>> it is dedicated to hosted server traffic only.
>>>>
>>>> So before, I uses only the BCN for cluster traffic for
>>>> cman/corosync
>>>> multicast traffic, no rrp. A couple months ago, I had a cluster
>>>> partition when VM live migration (also on the BCN) congested the
>>>> network. So I decided to enable RRP using the SN as backup, which has
>>>> been marginally successful.
>>>>
>>>> Now, I want to switch to unicast (>>> the SN as the backup and BCN as the primary ring and do a proper
>>>> IPTables firewall. Is this sane?
>>>>
>>>> When I stopped iptables entirely and started cman with unicast +
>>>> RRP,
>>>> I saw this:
>>>>
>>>> ] Node 1
>>>> Sep 11 17:31:24 node1 kernel: DLM (built Aug 10 2015 09:45:36)
>>>> installed
>>>> Sep 11 17:31:24 node1 corosync[2523]: [MAIN ] Corosync Cluster
>>>> Engine
>>>> ('1.4.7'): started and ready to provide service.
>>>> Sep 11 17:31:24 node1 corosync[2523]: [MAIN ] Corosync built-in
>>>> features: nss dbus rdma snmp
>>>> Sep 11 17:31:24 node1 corosync[2523]: [MAIN ] Successfully read
>>>> config from /etc/cluster/cluster.conf
>>>> Sep 11 17:31:24 node1 corosync[2523]: [MAIN ] Successfully parsed
>>>> cman config
>>>> Sep 11 17:31:24 node1 corosync[2523]: [TOTEM ] Initializing transport
>>>> (UDP/IP Unicast).
>>>> Sep 11 17:31:24 node1 corosync[2523]: [TOTEM ] Initializing
>>>> transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
>>>> Sep 11 17:31:24 node1 corosync[2523]: [TOTEM ] Initializing transport
>>>> (UDP/IP Unicast).
>>>> Sep 11 17:31:24 node1 corosync[2523]: [TOTEM ] Initializing
>>>> transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
>>>> Sep 11 17:31:24 node1 corosync[2523]: [TOTEM ] The network interface
>>>> [10.20.10.1] is now up.
>>>> Sep 11 17:31:24 node1 corosync[2523]: [QUORUM] Using quorum provider
>>>> quorum_cman
>>>> Sep 11 17:31:24 node1 corosync[2523]: [SERV ] Service engine loaded:
>>>> corosync cluster quorum service v0.1
>>>> Sep 11 17:31:24 node1 corosync[2523]: [CMAN ] CMAN 3.0.12.1 (built
>>>> Jul 6 2015 05:30:35) started
>>>> Sep 11 17:31:24 node1 corosync[2523]: [SERV ] Service engine loaded:
>>>> corosync CMAN membership service 2.90
>>>> Sep 11 17:31:24 node1 corosync[2523]: [SERV ] Service engine loaded:
>>>> openais checkpoint service B.01.01
>>>> Sep 11 17:31:24 node1 corosync[2523]: [SERV ] Service engine loaded:
>>>> corosync extended virtual synchrony service
>>>> Sep 11 17:31:24 node1 corosync[2523]: [SERV ] Service engine loaded:
>>>> corosync configuration service
>>>> Sep 11 17:31:24 node1 corosync[2523]: [SERV ] Service engine loaded:
>>>> corosync cluster closed process group service v1.01
>>>> Sep 11 17:31:24 node1 corosync[2523]: [SERV ] Service engine loaded:
>>>> corosync cluster config database access v1.01
>>>> Sep 11 17:31:24 node1 corosync[2523]: [SERV ] Service engine loaded:
>>>> corosync profile loading service
>>>> Sep 11 17:31:24 node1 corosync[2523]: [QUORUM] Using quorum provider
>>>> quorum_cman
>>>> Sep 11 17:31:24 node1 corosync[2523]: [SERV ] Service engine loaded:
>>>> corosync cluster quorum service v0.1
>>>> Sep 11 17:31:24 node1 corosync[2523]: [MAIN ] Compatibility mode set
>>>> to whitetank. Using V1 and V2 of the synchronization engine.
>>>> Sep 11 17:31:24 node1 corosync[2523]: [TOTEM ] adding new UDPU member
>>>> {10.20.10.1}
>
Re: [ClusterLabs] Major problem with iSCSITarget resource on top of DRBD M/S resource.
On 27/09/15 11:02 AM, Alex Crow wrote: > > > On 27/09/15 15:54, Digimer wrote: >> On 27/09/15 10:40 AM, Alex Crow wrote: >>> Hi List, >>> >>> I'm trying to set up a failover iSCSI storage system for oVirt using a >>> self-hosted engine. I've set up DRBD in Master-Slave for two iSCSI >>> targets, one for the self-hosted engine and one for the VMs. I has this >>> all working perfectly, then after trying to move the engine's LUN to the >>> opposite host, all hell broke loose. The VMS LUN is still fine, starts >> I'm guessing no fencing? > > Hi Digimer, > > No, but I've tried turning off one machine and still no success as a > single node :-( You *must* have working fencing, anyway. So now strikes me as a fantastic time to add it. Turning off the node alone doesn't help. >>> and migrates as it should. However the engine LUN always seems to try to >>> launch the target on the host that is *NOT* the master of the DRBD >>> resource. My constraints look fine, and should be self-explanatory about >>> which is which: >>> >>> [root@granby ~]# pcs constraint --full >>> Location Constraints: >>> Ordering Constraints: >>>promote drbd-vms-iscsi then start iscsi-vms-ip (kind:Mandatory) >>> (id:vm_iscsi_ip_after_drbd) >>>start iscsi-vms-target then start iscsi-vms-lun (kind:Mandatory) >>> (id:vms_lun_after_target) >>>promote drbd-vms-iscsi then start iscsi-vms-target (kind:Mandatory) >>> (id:vms_target_after_drbd) >>>promote drbd-engine-iscsi then start iscsi-engine-ip (kind:Mandatory) >>> (id:ip_after_drbd) >>>start iscsi-engine-target then start iscsi-engine-lun >>> (kind:Mandatory) >>> (id:lun_after_target) >>>promote drbd-engine-iscsi then start iscsi-engine-target >>> (kind:Mandatory) (id:target_after_drbd) >>> Colocation Constraints: >>>iscsi-vms-ip with drbd-vms-iscsi (score:INFINITY) (rsc-role:Started) >>> (with-rsc-role:Master) (id:vms_ip-with-drbd) >>>iscsi-vms-lun with drbd-vms-iscsi (score:INFINITY) (rsc-role:Started) >>> (with-rsc-role:Master) (id:vms_lun-with-drbd) >>>iscsi-vms-target with drbd-vms-iscsi (score:INFINITY) >>> (rsc-role:Started) (with-rsc-role:Master) (id:vms_target-with-drbd) >>>iscsi-engine-ip with drbd-engine-iscsi (score:INFINITY) >>> (rsc-role:Started) (with-rsc-role:Master) (id:ip-with-drbd) >>>iscsi-engine-lun with drbd-engine-iscsi (score:INFINITY) >>> (rsc-role:Started) (with-rsc-role:Master) (id:lun-with-drbd) >>>iscsi-engine-target with drbd-engine-iscsi (score:INFINITY) >>> (rsc-role:Started) (with-rsc-role:Master) (id:target-with-drbd) >>> >>> But see this from pcs status, the iSCSI target has FAILED on glenrock, >>> but the DRBD master is on granby!: >>> >>> [root@granby ~]# pcs status >>> Cluster name: storage >>> Last updated: Sun Sep 27 15:30:08 2015 >>> Last change: Sun Sep 27 15:20:58 2015 >>> Stack: cman >>> Current DC: glenrock - partition with quorum >>> Version: 1.1.11-97629de >>> 2 Nodes configured >>> 10 Resources configured >>> >>> >>> Online: [ glenrock granby ] >>> >>> Full list of resources: >>> >>> Master/Slave Set: drbd-vms-iscsi [drbd-vms] >>> Masters: [ glenrock ] >>> Slaves: [ granby ] >>> iscsi-vms-target(ocf::heartbeat:iSCSITarget): Started glenrock >>> iscsi-vms-lun(ocf::heartbeat:iSCSILogicalUnit): Started glenrock >>> iscsi-vms-ip(ocf::heartbeat:IPaddr2):Started glenrock >>> Master/Slave Set: drbd-engine-iscsi [drbd-engine] >>> Masters: [ granby ] >>> Slaves: [ glenrock ] >>> iscsi-engine-target(ocf::heartbeat:iSCSITarget): FAILED glenrock >>> (unmanaged) >>> iscsi-engine-ip(ocf::heartbeat:IPaddr2):Stopped >>> iscsi-engine-lun(ocf::heartbeat:iSCSILogicalUnit): Stopped >>> >>> Failed actions: >>> iscsi-engine-target_stop_0 on glenrock 'unknown error' (1): >>> call=177, status=Timed Out, last-rc-change='Sun Sep 27 15:20:59 2015', >>> queued=0ms, exec=10003ms >>> iscsi-engine-target_stop_0 on glenrock 'unknown error' (1): >>> call=177, status=Timed Out, last-rc-change='Sun Sep 27 15:20:59 2015', >>> queued=0ms, exec=10003ms >>> >>> I have tried various combinations of pcs resource clear and cleanup, but >>> that all result in the same outcome
Re: [ClusterLabs] Major problem with iSCSITarget resource on top of DRBD M/S resource.
]: notice: operation_finished: > iscsi-engine-target_stop_0:1079:stderr [ tgtadm: can't find the target ] > Sep 27 15:35:59 glenrock lrmd[3363]: notice: operation_finished: > iscsi-engine-target_stop_0:1079:stderr [ tgtadm: can't find the target ] > Sep 27 15:35:59 glenrock lrmd[3363]: notice: operation_finished: > iscsi-engine-target_stop_0:1079:stderr [ tgtadm: can't find the target ] > Sep 27 15:35:59 glenrock lrmd[3363]: notice: operation_finished: > iscsi-engine-target_stop_0:1079:stderr [ tgtadm: can't find the target ] > Sep 27 15:35:59 glenrock lrmd[3363]: notice: operation_finished: > iscsi-engine-target_stop_0:1079:stderr [ tgtadm: can't find the target ] > Sep 27 15:35:59 glenrock crmd[3366]:error: process_lrm_event: > Operation iscsi-engine-target_stop_0: Timed Out (node=glenrock, > call=336, timeout=1ms) > Sep 27 15:35:59 glenrock crmd[3366]: notice: process_lrm_event: > glenrock-iscsi-engine-target_stop_0:336 [ tgtadm: can't find the > target\ntgtadm: can't find the target\ntgtadm: can't find the > target\ntgtadm: can't find the target\ntgtadm: can't find the > target\ntgtadm: can't find the target\ntgtadm: can't find the > target\ntgtadm: can't find the target\ntgtadm: can't find the > target\ntgtadm: can't find the target\n ] > Sep 27 15:35:59 glenrock crmd[3366]: warning: status_from_rc: Action 78 > (iscsi-engine-target_stop_0) on glenrock failed (target: 0 vs. rc: 1): > Error > Sep 27 15:35:59 glenrock crmd[3366]: warning: update_failcount: > Updating failcount for iscsi-engine-target on glenrock after failed > stop: rc=1 (update=INFINITY, time=1443364559) > Sep 27 15:35:59 glenrock crmd[3366]: notice: abort_transition_graph: > Transition aborted by iscsi-engine-target_stop_0 'modify' on (null): > Event failed (magic=2:1;78:53:0:4ac61a75-532d-45c4-b07c-eee753699adc, > cib=0.374.146, source=match_graph_event:344, 0) > Sep 27 15:35:59 glenrock attrd[3364]: notice: attrd_trigger_update: > Sending flush op to all hosts for: last-failure-iscsi-engine-target > (1443364559) > Sep 27 15:35:59 glenrock crmd[3366]: warning: update_failcount: > Updating failcount for iscsi-engine-target on glenrock after failed > stop: rc=1 (update=INFINITY, time=1443364559) > Sep 27 15:35:59 glenrock crmd[3366]: warning: status_from_rc: Action 78 > (iscsi-engine-target_stop_0) on glenrock failed (target: 0 vs. rc: 1): > Error > Sep 27 15:35:59 glenrock crmd[3366]: warning: update_failcount: > Updating failcount for iscsi-engine-target on glenrock after failed > stop: rc=1 (update=INFINITY, time=1443364559) > Sep 27 15:35:59 glenrock attrd[3364]: notice: attrd_perform_update: > Sent update 203: last-failure-iscsi-engine-target=1443364559 > Sep 27 15:35:59 glenrock crmd[3366]: warning: update_failcount: > Updating failcount for iscsi-engine-target on glenrock after failed > stop: rc=1 (update=INFINITY, time=1443364559) > Sep 27 15:35:59 glenrock attrd[3364]: notice: attrd_trigger_update: > Sending flush op to all hosts for: last-failure-iscsi-engine-target > (1443364559) > Sep 27 15:35:59 glenrock crmd[3366]: notice: run_graph: Transition 53 > (Complete=8, Pending=0, Fired=0, Skipped=3, Incomplete=0, > Source=/var/lib/pacemaker/pengine/pe-input-536.bz2): Stopped > Sep 27 15:35:59 glenrock attrd[3364]: notice: attrd_perform_update: > Sent update 205: last-failure-iscsi-engine-target=1443364559 > Sep 27 15:35:59 glenrock attrd[3364]: notice: attrd_trigger_update: > Sending flush op to all hosts for: last-failure-iscsi-engine-target > (1443364559) > Sep 27 15:35:59 glenrock attrd[3364]: notice: attrd_perform_update: > Sent update 207: last-failure-iscsi-engine-target=1443364559 > Sep 27 15:35:59 glenrock pengine[3365]: notice: unpack_config: On loss > of CCM Quorum: Ignore > Sep 27 15:35:59 glenrock pengine[3365]: warning: unpack_rsc_op_failure: > Processing failed op stop for iscsi-engine-target on glenrock: unknown > error (1) > Sep 27 15:35:59 glenrock pengine[3365]: warning: unpack_rsc_op_failure: > Processing failed op stop for iscsi-engine-target on glenrock: unknown > error (1) > Sep 27 15:35:59 glenrock pengine[3365]: warning: > common_apply_stickiness: Forcing iscsi-engine-target away from glenrock > after 100 failures (max=100) > Sep 27 15:35:59 glenrock pengine[3365]: notice: process_pe_message: > Calculated Transition 54: /var/lib/pacemaker/pengine/pe-input-537.bz2 > Sep 27 15:35:59 glenrock crmd[3366]: notice: run_graph: Transition 54 > (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, > Source=/var/lib/pacemaker/pengine/pe-input-537.bz2): Complete > Sep 27 15:35:59 glenrock crmd[3366]: notice: do_state_transition: > State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS > cause=C_FSA_INTER
[ClusterLabs] will rgmanager/ccs support the vm RA's new migrate-setspeed?
Re: https://github.com/ClusterLabs/resource-agents/pull/629 I'd love to support this in the current gen Anvil!. Would it be hard to add support to ccs for this? -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] [Linux-HA] Cluster for HA VM's serving our local network
Hi Juergen, First; This list is deprecated and you should use the Cluster Labs - Users list (which I've cc'ed here). Second; That tutorial is quite old and was replaced a while ago with this one: https://alteeve.ca/w/AN!Cluster_Tutorial_2. It has a lot of improvements we made after having many systems out in the field, so it is well worth re-doing your setup to match it. It's mostly the same, so it shouldn't be a big job. I'll address your comments in-line: On 23/09/15 08:38 AM, J. Echter wrote: > Hi, > > i was using this guide > https://alteeve.ca/w/2-Node_Red_Hat_KVM_Cluster_Tutorial_-_Archive to > set up my cluster for some services, all works pretty good. > > I decided to use this cluster as a HA vm provider for my network. > > I have a little, maybe silly, question. > > The guide tells me to disable qemu default network, like this: > >> >> Disable the 'qemu' Bridge >> >> By default, libvirtd <https://alteeve.ca/w/Libvirtd> creates a bridge >> called virbr0 designed to connect virtual machines to the first eth0 >> interface. Our system will not need this, so we will remove it now. >> >> If libvirtd has started, skip to the next step. If you haven't started >> libvirtd yet, you can manually disable the bridge by blanking out the >> config file. >> >> cat /dev/null>/etc/libvirt/qemu/networks/default.xml > i skipped the step to create the bridge device, as it was not needed for > my belongings. OK. >> vim /etc/sysconfig/network-scripts/ifcfg-vbr2 >> # Internet-Facing Network - Bridge >> DEVICE="vbr2" >> TYPE="Bridge" >> BOOTPROTO="static" >> IPADDR="10.255.0.1" >> NETMASK="255.255.0.0" >> GATEWAY="10.255.255.254" >> DNS1="8.8.8.8" >> DNS2="8.8.4.4" >> DEFROUTE="yes" > > > Now i want to know how to proceed? > > i have bond0 - connected to my network (both nodes got different ip's > from my dhcp) > bond1 & bond2 are used for corosync and drbd. > > what would be the best decision to have some vm's served from this > 2-node cluster too? >From a bridging perspective, the quoted example config above is good. The default libvirtd bridge is a NAT'ed bridge, so your VMs would get IPs in the 192.168.122.0/24 subnet, and the libvirtd bridge would route them to the outside world. Using the bridge type in the tutorial though, your VMs would appear to be directly on your network and would get (or you would assign) IPs just the same as the rest of your system. > thanks, and please tell me what infos i may have forgotten to provide > for you. :) > > cheers > > juergen -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] [Linux-HA] Cluster for HA VM's serving our local network
On 23/09/15 10:23 AM, J. Echter wrote: > Hi Digimer, > > Am 23.09.2015 um 15:38 schrieb Digimer: >> Hi Juergen, >> >>First; This list is deprecated and you should use the Cluster Labs - >> Users list (which I've cc'ed here). > > i already got that reminder as i sent my message, and i subscribed :) I'm switching the thread to there then. >>Second; That tutorial is quite old and was replaced a while ago with >> this one: https://alteeve.ca/w/AN!Cluster_Tutorial_2. It has a lot of >> improvements we made after having many systems out in the field, so it >> is well worth re-doing your setup to match it. It's mostly the same, so >> it shouldn't be a big job. > > i'll have a look over the new one. The main change, relative to this discussion, is more descriptive interface names. >>I'll address your comments in-line: >> >> On 23/09/15 08:38 AM, J. Echter wrote: >>> Hi, >>> >>> i was using this guide >>> https://alteeve.ca/w/2-Node_Red_Hat_KVM_Cluster_Tutorial_-_Archive to >>> set up my cluster for some services, all works pretty good. >>> >>> I decided to use this cluster as a HA vm provider for my network. >>> >>> I have a little, maybe silly, question. >>> >>> The guide tells me to disable qemu default network, like this: >>> >>>>Disable the 'qemu' Bridge >>>> >>>> By default, libvirtd <https://alteeve.ca/w/Libvirtd> creates a bridge >>>> called virbr0 designed to connect virtual machines to the first eth0 >>>> interface. Our system will not need this, so we will remove it now. >>>> >>>> If libvirtd has started, skip to the next step. If you haven't started >>>> libvirtd yet, you can manually disable the bridge by blanking out the >>>> config file. >>>> >>>> cat /dev/null>/etc/libvirt/qemu/networks/default.xml >>> i skipped the step to create the bridge device, as it was not needed for >>> my belongings. >> OK. >> >>>> vim /etc/sysconfig/network-scripts/ifcfg-vbr2 >>>> # Internet-Facing Network - Bridge >>>> DEVICE="vbr2" >>>> TYPE="Bridge" >>>> BOOTPROTO="static" >>>> IPADDR="10.255.0.1" >>>> NETMASK="255.255.0.0" >>>> GATEWAY="10.255.255.254" >>>> DNS1="8.8.8.8" >>>> DNS2="8.8.4.4" >>>> DEFROUTE="yes" >>> >>> Now i want to know how to proceed? >>> >>> i have bond0 - connected to my network (both nodes got different ip's >>> from my dhcp) >>> bond1 & bond2 are used for corosync and drbd. >>> >>> what would be the best decision to have some vm's served from this >>> 2-node cluster too? >> From a bridging perspective, the quoted example config above is good. >> The default libvirtd bridge is a NAT'ed bridge, so your VMs would get >> IPs in the 192.168.122.0/24 subnet, and the libvirtd bridge would route >> them to the outside world. Using the bridge type in the tutorial though, >> your VMs would appear to be directly on your network and would get (or >> you would assign) IPs just the same as the rest of your system. > > so i can just use this example on my setup? > > bond0 = LAN = 192.168.0.0/24 This is the BCN, and is usually on 10.20.0.0/16 > bridge = 10.255.0.1 The bridge is on the IFN, which in the tutorial is on 10.255.0.0/16, so yes. Note that the IP assigned to the bridge has no bearing at all on the IPs set in the VMs. > can i use my own dns server, working on the lan? > > like this: > > DEVICE="vbr2" > TYPE="Bridge" > BOOTPROTO="static" > IPADDR="10.255.0.1" > NETMASK="255.255.0.0" > GATEWAY="10.255.255.254" > DNS1="192.168.0.1" > DEFROUTE="yes" Sure. With this style of bridging, it's like you VMs are plugged directly into the physical switch. What you do on the node has no bearing. The only thing is that you move the IP assignment for the node out of the bond and into the bridge. In fact, you can assign no IP to the bridge and traffic from the VMs will route fine. So think of this bridge as being like a regular hardware switch that the VMs plug into and that the node itself plugs into, and the bond as the "cable" linking the vritual switch to the hardware switch. When you think of it like that, you can see how the setup of the node has no bearing on anything else. >>> thanks, and please tell me w
[ClusterLabs] Odd clvmd error - clvmd: Unable to create DLM lockspace for CLVM: Address already in use
<clusterfs ref="sharedfs"/> <script ref="wait-for-drbd"> <script ref="clvmd"> <clusterfs ref="sharedfs"/> Nothing special there at all. While writing this email though, I saw this on the other node: Sep 24 23:03:39 node1 corosync[4770]: [TOTEM ] Retransmit List: 14e Sep 24 23:03:39 node1 corosync[4770]: [TOTEM ] Retransmit List: 14e Sep 24 23:03:49 node1 corosync[4770]: [TOTEM ] Retransmit List: 158 Sep 24 23:03:49 node1 corosync[4770]: [TOTEM ] Retransmit List: 15a Sep 24 23:03:49 node1 corosync[4770]: [TOTEM ] Retransmit List: 15a Sep 24 23:03:59 node1 corosync[4770]: [TOTEM ] Retransmit List: 161 Sep 24 23:03:59 node1 corosync[4770]: [TOTEM ] Retransmit List: 161 Sep 24 23:03:59 node1 corosync[4770]: [TOTEM ] Retransmit List: 161 Sep 24 23:03:59 node1 corosync[4770]: [TOTEM ] Retransmit List: 161 163 Sep 24 23:03:59 node1 corosync[4770]: [TOTEM ] Retransmit List: 163 Sep 24 23:04:19 node1 corosync[4770]: [TOTEM ] Retransmit List: 177 Sep 24 23:04:19 node1 corosync[4770]: [TOTEM ] Retransmit List: 177 Sep 24 23:04:19 node1 corosync[4770]: [TOTEM ] Retransmit List: 179 Sep 24 23:04:19 node1 corosync[4770]: [TOTEM ] Retransmit List: 179 Sep 24 23:04:29 node1 corosync[4770]: [TOTEM ] Retransmit List: 181 Sep 24 23:04:29 node1 corosync[4770]: [TOTEM ] Retransmit List: 181 Sep 24 23:04:29 node1 corosync[4770]: [TOTEM ] Retransmit List: 181 Sep 24 23:04:29 node1 corosync[4770]: [TOTEM ] Retransmit List: 183 Sep 24 23:04:29 node1 corosync[4770]: [TOTEM ] Retransmit List: 183 Sep 24 23:04:39 node1 corosync[4770]: [TOTEM ] Retransmit List: 18c Sep 24 23:04:39 node1 corosync[4770]: [TOTEM ] Retransmit List: 18c Sep 24 23:04:39 node1 corosync[4770]: [TOTEM ] Retransmit List: 18c 18e Sep 24 23:04:39 node1 corosync[4770]: [TOTEM ] Retransmit List: 18e Sep 24 23:07:20 node1 corosync[4770]: [TOTEM ] Retransmit List: 23c Sep 24 23:07:20 node1 corosync[4770]: [TOTEM ] Retransmit List: 23c Sep 24 23:07:20 node1 corosync[4770]: [TOTEM ] Retransmit List: 23c Sep 24 23:07:20 node1 corosync[4770]: [TOTEM ] Retransmit List: 23e Sep 24 23:07:20 node1 corosync[4770]: [TOTEM ] Retransmit List: 23e Sep 24 23:07:30 node1 corosync[4770]: [TOTEM ] Retransmit List: 247 Sep 24 23:07:30 node1 corosync[4770]: [TOTEM ] Retransmit List: 247 Sep 24 23:07:30 node1 corosync[4770]: [TOTEM ] Retransmit List: 249 Sep 24 23:07:30 node1 corosync[4770]: [TOTEM ] Retransmit List: 24b Sep 24 23:07:30 node1 corosync[4770]: [TOTEM ] Retransmit List: 24b Sep 24 23:07:40 node1 corosync[4770]: [TOTEM ] Retransmit List: 252 Sep 24 23:07:40 node1 corosync[4770]: [TOTEM ] Retransmit List: 252 Sep 24 23:07:40 node1 corosync[4770]: [TOTEM ] Retransmit List: 254 Sep 24 23:07:40 node1 corosync[4770]: [TOTEM ] Retransmit List: 254 Certainly *looks* like a network problem, but I can't see what's wrong... Any ideas? Thanks! -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Quorum With Two Nodes And an "observer" Questions
On 18/09/15 03:32 AM, Ariel S wrote: > Right now we are building a(nother) cluster, that basically built from two > physical servers with VMs running resources managed by Pacemaker. Since > this is > two-node setup, and I'm constrained only to that two beefy servers, I > realize > that this would mean quorum=1 which also means no majority vote. > > The VMs have multiple cluster, eg. gateway cluster, infrastructure > cluster and > postgresql cluster. > > I'm thinking I could (probably) ask for another (weak) server which would > function as an observer, a quorum-maker. Quorum should be possible with 2 > against 1, right? I think this is possible, by setting resources priority > towards -infinity from going to that quorum-maker server which means > resources > should run on those two beefy servers. > > I probably will have to run multiple instance of corosync and pacemaker, > with > different configuration and authkey files (for each cluster in the beefy > servers) in a small-ish baremetal centos box. > > My question, is this actually feasible? Anyone have run into this > situation and > come up with such idea too? Got anything to share about this? > > > Thank you Feasible, sure. Needed? No. Quorum is nice to have, but if you use a fence delay on a node and tell corosync to use 'wait_for_all', then you're fine. All the clusters I've built in the last 5~6 years have been 2-node only. You just need to be sure you've got the fencing/stonith sorted out well. digimer -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Pacemaker/pcs & DRBD not demoting secondary node to Slave (always Stopped)
On 21/09/15 11:23 AM, Jason Gress wrote: > Yeah, I had problems, which I am thinking might be firewall related. In a > previous place of employment I had ipmi working great (but with > heartbeat), so I do have some experience with IPMI STONITH. If you can query the IPMI sensor data, you should have no trouble fencing. It's basically a wrapper for ipmitool. Can you use ipmitool to query anything over the network? > Example: > > [root@fx201-1a ~]# fence_ipmilan -a 10.XX.XX.XX -l root -p calvin -o status > Failed: Unable to obtain correct plug status or plug is not available > > (Don't worry, default dell pw left in for illustrative purposes only!) That doesn't make sense... No plug is needed for IPMI fencing. What OS and what hardware? > When I tried to watch it with tcpdump, it seems to use random ports, so I > couldn't really advise the firewall team on how to make this work. SSH > into the drac works fine, and IPMI over IP is enabled. If anyone has > ideas on this, they would be greatly appreciated. You're initiating the connection, so no firewall edits should be needed (anything returning should be ESTABLISHED/RELATED). -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Pacemaker/pcs & DRBD not demoting secondary node to Slave (always Stopped)
IPMI fencing is a very common type, and it shouldn't be so hard to get working. Easiest is to test it out first on the command line, outside pacemaker. Run: fence_ipmilan -a -l -p -o status If that doesn't work, you may need to use lanplus or similar. See 'man fence_ipmilan'. Once you can use the above command to query the power status of the nodes, you're 95% of the way there. Fencing can not be put on the back burner, as you've now seen. Without it, things can and will go very wrong. On 21/09/15 09:34 AM, Jason Gress wrote: > Thank you for comment. I attempted to use iDRAC/IPMI STONITH, and after > spending over a day, I had to put it on the backburner for timeline > reasons. For whatever reason, I could not get IPMI to talk, and the > iDRAC5 plugin was not working either for reasons I don't understand. > > Is that what you had in mind, or is there another method/configuration for > fencing DRBD? > > Thank you for your advice, > > Jason > > On 9/20/15, 9:40 PM, "Digimer" <li...@alteeve.ca> wrote: > >> On 20/09/15 09:18 PM, Jason Gress wrote: >>> I had seemed to cause a split brain attempting to repair this. But that >> >> Use fencing! Voila, no more split-brains. >> >>> wasn't the issue. You can't have any colocation requirements for DRBD >>> resources; that's what killed me. This line did it: >>> >>> ms_drbd_vmfs with ClusterIP (score:INFINITY) >>> (id:colocation-ms_drbd_vmfs-ClusterIP-INFINITY) >>> >>> Do NOT do this! >>> >>> Jason >>> >>> From: Jason Gress <jgr...@accertify.com <mailto:jgr...@accertify.com>> >>> Reply-To: Cluster Labs - All topics related to open-source clustering >>> welcomed <users@clusterlabs.org <mailto:users@clusterlabs.org>> >>> Date: Friday, September 18, 2015 at 3:03 PM >>> To: Cluster Labs - All topics related to open-source clustering welcomed >>> <users@clusterlabs.org <mailto:users@clusterlabs.org>> >>> Subject: Re: [ClusterLabs] Pacemaker/pcs & DRBD not demoting secondary >>> node to Slave (always Stopped) >>> >>> Well, it almost worked. I was able to modify the existing cluster per >>> your command, and it worked great. >>> >>> Today, I made two more clusters via the exact same process (I >>> used/modified my notes as I was building and fixing the first one >>> yesterday) and now it's doing the same thing, despite having your >>> improved master slave rule. Here's the config: >>> >>> [root@fx201-1a ~]# pcs config --full >>> Cluster Name: fx201-vmcl >>> Corosync Nodes: >>> fx201-1a.zwo fx201-1b.zwo >>> Pacemaker Nodes: >>> fx201-1a.zwo fx201-1b.zwo >>> >>> Resources: >>> Resource: ClusterIP (class=ocf provider=heartbeat type=IPaddr2) >>> Attributes: ip=10.XX.XX.XX cidr_netmask=24 >>> Operations: start interval=0s timeout=20s >>> (ClusterIP-start-timeout-20s) >>> stop interval=0s timeout=20s (ClusterIP-stop-timeout-20s) >>> monitor interval=15s (ClusterIP-monitor-interval-15s) >>> Master: ms_drbd_vmfs >>> Meta Attrs: master-max=1 master-node-max=1 clone-max=2 >>> clone-node-max=1 notify=true >>> Resource: drbd_vmfs (class=ocf provider=linbit type=drbd) >>>Attributes: drbd_resource=vmfs >>>Operations: start interval=0s timeout=240 >>> (drbd_vmfs-start-timeout-240) >>>promote interval=0s timeout=90 >>> (drbd_vmfs-promote-timeout-90) >>>demote interval=0s timeout=90 >>> (drbd_vmfs-demote-timeout-90) >>>stop interval=0s timeout=100 (drbd_vmfs-stop-timeout-100) >>>monitor interval=29s role=Master >>> (drbd_vmfs-monitor-interval-29s-role-Master) >>>monitor interval=31s role=Slave >>> (drbd_vmfs-monitor-interval-31s-role-Slave) >>> Resource: vmfsFS (class=ocf provider=heartbeat type=Filesystem) >>> Attributes: device=/dev/drbd0 directory=/exports/vmfs fstype=xfs >>> Operations: start interval=0s timeout=60 (vmfsFS-start-timeout-60) >>> stop interval=0s timeout=60 (vmfsFS-stop-timeout-60) >>> monitor interval=20 timeout=40 >>> (vmfsFS-monitor-interval-20) >>> Resource: nfs-server (class=systemd type=nfs-server) >>> Operations: monitor interval=60s (nfs-server-monitor-interval-60s) >>> >>> Stonith Devices: >>> Fencing Levels: >>> >>> Location Constrai
Re: [ClusterLabs] Odd clvmd error - clvmd: Unable to create DLM lockspace for CLVM: Address already in use
On 25/09/15 03:44 AM, Christine Caulfield wrote: > On 25/09/15 00:09, Digimer wrote: >> I had a RHEL 6.7, cman + rgmanager cluster that I've built many times >> before. Oddly, I just hit this error: >> >> >> [root@node2 ~]# /etc/init.d/clvmd start >> Starting clvmd: clvmd could not connect to cluster manager >> Consult syslog for more information >> >> >> syslog: >> >> Sep 24 23:00:30 node2 kernel: dlm: Using SCTP for communications >> Sep 24 23:00:30 node2 clvmd: Unable to create DLM lockspace for CLVM: >> Address already in use >> Sep 24 23:00:30 node2 kernel: dlm: Can't bind to port 21064 addr number 1 > > This seems to be the key to it. I can't imagine what else would be using > port 21064 (apart from DLM using TCP as well as SCTP but I don' think > that's possible!) > > Have a look in netstat and see what else is using that port. > > It could be that the socket was in use and is taking a while to shut > down so it might go away on its own too. > > Chrissie netstat and lsof showed nothing using it. Looking at the logs (of our installer), it had manually started drbd + clvmd + gfs2 just fine. Then it asked rgmanager to start which tried to tear down the already running storage services before (re)starting them. It looks like the (drbd) UpToDate node got called to stop before the Inconsistent node, which failed because the Inconsistent node needed the UpToDate node (gfs2 was still mounted so clvmd held open the drbd device). I'm not clear on what happened next, but things went sideways. So I manually stopped everything, including clvmd/cman, and it still threw that error. Eventually I rebooted both nodes and it went back to working. Odd. >> Sep 24 23:00:30 node2 kernel: dlm: cannot start dlm lowcomms -98 >> >> >> There are no iptables rules: >> >> >> [root@node2 ~]# iptables-save >> >> >> And there are no DLM lockspaces, either: >> >> >> [root@node2 ~]# dlm_tool ls >> [root@node2 ~]# >> >> >> I tried withdrawing the node from the cluster entirely, the started cman >> alone and tried to start clvmd, same issue. >> >> Pinging between the two nodes seems OK: >> >> >> [root@node1 ~]# uname -n >> node1.ccrs.bcn >> [root@node1 ~]# ping -c 2 node1.ccrs.bcn >> PING node1.bcn (10.20.10.1) 56(84) bytes of data. >> 64 bytes from node1.bcn (10.20.10.1): icmp_seq=1 ttl=64 time=0.015 ms >> 64 bytes from node1.bcn (10.20.10.1): icmp_seq=2 ttl=64 time=0.017 ms >> >> --- node1.bcn ping statistics --- >> 2 packets transmitted, 2 received, 0% packet loss, time 1000ms >> rtt min/avg/max/mdev = 0.015/0.016/0.017/0.001 ms >> >> [root@node2 ~]# uname -n >> node2.ccrs.bcn >> [root@node2 ~]# ping -c 2 node1.ccrs.bcn >> PING node1.bcn (10.20.10.1) 56(84) bytes of data. >> 64 bytes from node1.bcn (10.20.10.1): icmp_seq=1 ttl=64 time=0.079 ms >> 64 bytes from node1.bcn (10.20.10.1): icmp_seq=2 ttl=64 time=0.076 ms >> >> --- node1.bcn ping statistics --- >> 2 packets transmitted, 2 received, 0% packet loss, time 999ms >> rtt min/avg/max/mdev = 0.076/0.077/0.079/0.008 ms >> >> >> I have RRP configured and pings work on the second network, too: >> >> >> [root@node1 ~]# corosync-objctl |grep ring -A 5 >> totem.interface.ringnumber=0 >> totem.interface.bindnetaddr=10.20.10.1 >> totem.interface.mcastaddr=239.192.100.163 >> totem.interface.mcastport=5405 >> totem.interface.member.memberaddr=node1.ccrs.bcn >> totem.interface.member.memberaddr=node2.ccrs.bcn >> totem.interface.ringnumber=1 >> totem.interface.bindnetaddr=10.10.10.1 >> totem.interface.mcastaddr=239.192.100.164 >> totem.interface.mcastport=5405 >> totem.interface.member.memberaddr=node1.sn >> totem.interface.member.memberaddr=node2.sn >> >> [root@node1 ~]# ping -c 2 node2.sn >> PING node2.sn (10.10.10.2) 56(84) bytes of data. >> 64 bytes from node2.sn (10.10.10.2): icmp_seq=1 ttl=64 time=0.111 ms >> 64 bytes from node2.sn (10.10.10.2): icmp_seq=2 ttl=64 time=0.120 ms >> >> --- node2.sn ping statistics --- >> 2 packets transmitted, 2 received, 0% packet loss, time 999ms >> rtt min/avg/max/mdev = 0.111/0.115/0.120/0.011 ms >> >> [root@node2 ~]# ping -c 2 node1.sn >> PING node1.sn (10.10.10.1) 56(84) bytes of data. >> 64 bytes from node1.sn (10.10.10.1): icmp_seq=1 ttl=64 time=0.079 ms >> 64 bytes from node1.sn (10.10.10.1): icmp_seq=2 ttl=64 time=0.171 ms >> >> --- node1.sn ping statistics --- >>
Re: [ClusterLabs] design of a two-node cluster
On 08/12/15 03:13 AM, Lentes, Bernd wrote: > Digimer wrote: > >>>>> Should I install all vm's in one partition or every vm in a seperate >>>>> partition ? The advantage of one vm per partition is that I don't >>>>> need a cluster fs, right ? >>>> >>>> I would put each VM on a dedicated LV and not have an FS between >> the >>>> VM and the host. The question then becomes; What is the PV? I use >>>> clustered LVM to make sure all nodes are in sync, LVM-wise. >>> >>> Is this the setup you are running (without fs) ? >> >> Yes, we use DRBD to replicate the storage and use the /dev/drbdX >> device as the clustered LVM PV. We have one VG for the space (could >> add a new DRBD resource later if needed...) and then create a dedicated >> LV per VM. >> We have, as I mentioned, one small LV formatted with gfs2 where we >> store the VM's XML files (so that any change made to a VM is >> immediately available to all nodes. >> > > How can i migrate my current vm's ? They are stored in raw files (one or > two). > How do I transfer them to a naked lv ? > > Bernd If your VM images are 'raw', then you should be able to just dd the file to the LV directly. If they're qcow2 (or something else), convert it to raw first. digimer -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Antw: Re: Antw: Re: design of a two-node cluster
On 08/12/15 08:35 AM, Ulrich Windl wrote: >>>> "Lentes, Bernd" <bernd.len...@helmholtz-muenchen.de> schrieb am 08.12.2015 >>>> um > 13:10 in Nachricht <012101d131b1$5ec1b2e0$1c4518a0$@helmholtz-muenchen.de>: >> Ulrich wrote: >> >>> >>>>>> "Lentes, Bernd" <bernd.len...@helmholtz-muenchen.de> schrieb >>> am >>>>>> 08.12.2015 um >>> 09:13 in Nachricht <00a901d13190$5c6db3c0$15491b40$@helmholtz- >>> muenchen.de>: >>>> Digimer wrote: >>>> >>>>>>>> Should I install all vm's in one partition or every vm in a >>>>>>>> seperate partition ? The advantage of one vm per partition is >>>>>>>> that I don't need a cluster fs, right ? >>>>>>> >>>>>>> I would put each VM on a dedicated LV and not have an FS >>> between >>>>> the >>>>>>> VM and the host. The question then becomes; What is the PV? I >>> use >>>>>>> clustered LVM to make sure all nodes are in sync, LVM-wise. >>>>>> >>>>>> Is this the setup you are running (without fs) ? >>>>> >>>>> Yes, we use DRBD to replicate the storage and use the /dev/drbdX >>>>> device as the clustered LVM PV. We have one VG for the space (could >>>>> add a new DRBD resource later if needed...) and then create a >>>>> dedicated LV per VM. >>>>> We have, as I mentioned, one small LV formatted with gfs2 where we >>>>> store the VM's XML files (so that any change made to a VM is >>>>> immediately available to all nodes. >>>>> >>>> >>>> How can i migrate my current vm's ? They are stored in raw files (one >>>> or two). >>>> How do I transfer them to a naked lv ? >>> >>> For migration the image must be available on both nodes (thus gfs2). >>> >>>> >> >> Hi Ulrich, >> >> the migration i meant is the transition from my current setup (virtual >> machines in raw files in a partition with filesystem) to the one >> anticipated in the cluster (virtual machines in blank logical volumes >> without fs). How can I do that ? And can I expand my disks in the vm >> afterwards if necessary ? > > You can copy the images with rsync or similar while the VMs are down. Then > you'll have the same filesystem layout. If you want to change the partition > sizes, I'd suggest to create new disks and partitions, the mount the > partitions on old and new system, and then rsync (or similar) the _files_ > from OLD to NEW. Some boot loaders may need some extra magic. If you use LVM, > you might just add another disk to the VM, then make that disk a PV and add > it to the VG. Then you can expand your LVs inside the VM. > >> But the "other" migration (live-migration of vm's) is of course also >> interesting. Digimer wrote if I have my vm in a blank logical volume >> without fs, which is placed on a SAN, I can live-migrate because the >> process of live-migration takes care about the access to the lv and I >> don't need a cluster fs, just cLVM. > > If logical volume means LUN (disk), I'd agree. If you mean LVM LV, I'd be > very careful, especially when changing the LVM configuration. If you never > plan to change LVM configuration, you could consider partitioning your disk > with GPT with one partition for each VM. > > Regards, > Ulrich This approach is to do it from the VM's level, and it is not needed. The VM does need to be stopped, yes, but you can simple go from a raw format to the LV using dd, in my experience. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Antw: Re: design of a two-node cluster
On 08/12/15 02:44 AM, Ulrich Windl wrote: >>>> Digimer <li...@alteeve.ca> schrieb am 07.12.2015 um 22:40 in Nachricht > <5665fcdc.1030...@alteeve.ca>: > [...] >> Node 1 looks up how to fence node 2, sees no delay and fences >> immediately. Node 2 looks up how to fence node 1, sees a delay and >> pauses. Node 2 will be dead long before the delay expires, ensuring that >> node 2 always loses in such a case. If you have VMs on both nodes, then >> no matter which node the delay is on, some servers will be interrupted. > > AFAIK, the cluster will try to migrate resources if a fencing is pending, but > not yet complete. Is that true? > > [...] > > Regards, > Ulrich A cluster can't (and shouldn't!) do anything about resources until fencing has completed. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] design of a two-node cluster
On 07/12/15 03:27 PM, Lentes, Bernd wrote: > Digimer wrote: >> >> On 07/12/15 12:35 PM, Lentes, Bernd wrote: >>> Hi, >>> >>> i've been asking all around here a while ago. Unfortunately I couldn't >>> continue to work on my cluster, so I'm still thinking about the > design. >>> I hope you will help me again with some recommendations, because >> when >>> the cluster is running changing of the design is not possible anymore. >>> >>> These are my requirements: >>> >>> - all services are running inside virtual machines (KVM), mostly >>> databases and static/dynamic webpages >> >> This is fine, it's what we do with our 2-node clusters. >> >>> - I have two nodes and would like to have some vm's running on node >> A >>> and some on node B during normal operation as a kind of loadbalancing >> >> I used to do this, but I've since stopped. The reasons are: >> >> 1. You need to know that one node can host all servers and still perform >> properly. By always running on one node, you know that this is the case. >> Further, if one node ever stops being powerful enough, you will find out >> early and can address the issue immediately. >> >> 2. If there is a problem, you can always be sure which node to terminate >> (ie: the node hosting all servers gets the fence delay, so the node >> without servers will always get fenced). If you lose input power, you > can >> quickly power down the backup node to shed load, etc. > > Hi Digimer, > thanks for your reply. > I don't understand what you want to say in (2). To prevent a dual fence, where both nodes fence each other when communication between the nodes fail but the nodes are otherwise healthy, you need to set a fence delay against one of the nodes. So when this happens, if the delay is on node 1, this will happen; Node 1 looks up how to fence node 2, sees no delay and fences immediately. Node 2 looks up how to fence node 1, sees a delay and pauses. Node 2 will be dead long before the delay expires, ensuring that node 2 always loses in such a case. If you have VMs on both nodes, then no matter which node the delay is on, some servers will be interrupted. This is just one example. The other, as I mentioned, would be a lost power condition. Your UPSes can hold up both nodes for a period of time. If you can shut down one node, you can extend how long the UPSes can run. So if the power goes out for a period of time, you can immediately power down one node (the one hosting no servers) without first live-migrating VMs, which will make things simpler and save time. Another similar example would be a loss of cooling, where you would want to shut down nodes to minimize how much heat is being created. There are other examples, but I think this clarifies what I meant. >>> - I'd like to keep the setup simple (if possible) >> >> There is a minimum complexity in HA, but you can get as close as >> possible. We've spent years trying to simplify our VM hosting clusters > as >> much as possible. >> >>> - availability is important, performance not so much (webpages some >>> hundred requests per day, databases some hundred inserts/selects >> per >>> day) >> >> All the more reason to consolidate all VMs on one host. >> >>> - I'd like to have snapshots of the vm's >> >> This is never a good idea, as you catch the state of the disk at the > point of >> the snapshot, but not RAM. Anything in buffers will be missed so you can >> not rely on the snapshot images to always be consistent or even >> functional. >> >>> - live migration of the vm's should be possible >> >> Easy enough. >> >>> - nodes are SLES 11 SP4, vm's are Windows 7 and severable linux >>> distributions (Ubuntu, SLES, OpenSuSE) >> >> The OS installed on the guest VMs should not factor. As for the node OS, >> SUSE invests in making sure that HA works well so you should be fine. >> >>> - setup should be extensible (add further vm's) >> >> That is entirely a question of available hardware resources. >> >>> - I have a shared storage (FC SAN) >> >> Personally, I prefer DRBD (truly replicated storage), but SAN is fine. >> >>> My ideas/questions: >>> >>> Should I install all vm's in one partition or every vm in a seperate >>> partition ? The advantage of one vm per partition is that I don't need >>> a cluster fs, right ? >> >> I would put each VM on a dedicated LV and not have an FS between the >> VM and the host. The question then becomes; What is the PV? I use >
Re: [ClusterLabs] design of a two-node cluster
On 07/12/15 12:35 PM, Lentes, Bernd wrote: > Hi, > > i've been asking all around here a while ago. Unfortunately I couldn't > continue to work on my cluster, so I'm still thinking about the design. > I hope you will help me again with some recommendations, because when the > cluster is running changing of the design is not possible anymore. > > These are my requirements: > > - all services are running inside virtual machines (KVM), mostly databases > and static/dynamic webpages This is fine, it's what we do with our 2-node clusters. > - I have two nodes and would like to have some vm's running on node A and > some on node B during normal operation as a kind of loadbalancing I used to do this, but I've since stopped. The reasons are: 1. You need to know that one node can host all servers and still perform properly. By always running on one node, you know that this is the case. Further, if one node ever stops being powerful enough, you will find out early and can address the issue immediately. 2. If there is a problem, you can always be sure which node to terminate (ie: the node hosting all servers gets the fence delay, so the node without servers will always get fenced). If you lose input power, you can quickly power down the backup node to shed load, etc. > - I'd like to keep the setup simple (if possible) There is a minimum complexity in HA, but you can get as close as possible. We've spent years trying to simplify our VM hosting clusters as much as possible. > - availability is important, performance not so much (webpages some > hundred requests per day, databases some hundred inserts/selects per day) All the more reason to consolidate all VMs on one host. > - I'd like to have snapshots of the vm's This is never a good idea, as you catch the state of the disk at the point of the snapshot, but not RAM. Anything in buffers will be missed so you can not rely on the snapshot images to always be consistent or even functional. > - live migration of the vm's should be possible Easy enough. > - nodes are SLES 11 SP4, vm's are Windows 7 and severable linux > distributions (Ubuntu, SLES, OpenSuSE) The OS installed on the guest VMs should not factor. As for the node OS, SUSE invests in making sure that HA works well so you should be fine. > - setup should be extensible (add further vm's) That is entirely a question of available hardware resources. > - I have a shared storage (FC SAN) Personally, I prefer DRBD (truly replicated storage), but SAN is fine. > My ideas/questions: > > Should I install all vm's in one partition or every vm in a seperate > partition ? The advantage of one vm per partition is that I don't need a > cluster fs, right ? I would put each VM on a dedicated LV and not have an FS between the VM and the host. The question then becomes; What is the PV? I use clustered LVM to make sure all nodes are in sync, LVM-wise. > I read to avoid a cluster fs if possible because it adds further > complexity. Below the fs I'd like to have logical volumes because they are > easy to expand. Avoiding clustered FS is always preferable, yes. I use a small gfs2 partition, but this is just for storing VM XML data, install media, etc. Things that change rarely. Some advocate for having independent FSes on each node and keeping the data in sync using things like rsync or what have you. > Do I need cLVM (I think so) ? Is it an advantage to install the vm's in > plain partitions, without a fs ? I advise it, yes. > It would reduce the complexity further because I don't need a fs. Would > live migration still be possible ? Live migration is possible provided both nodes can see the same physical storage at the same time. For example, DRBD dual-primary works. If you use clustered LVM, you can be sure that the backing LVs are the same across the nodes. > snapshots: > I was playing around with virsh (libvirt) to create snapshots of the vm's. > In the end I gave up. virsh explains commands in its help, but when you > want to use them you get messages > like "not supported yet", although I use libvirt 1.2.11. This is > ridiculous. I think I will create my snapshots inside the vm's using lvm. > We have a network based backup solution (Legato/EMC) which saves the disks > every night. > Supplying a snapshot for that I have a consistent backup. The databases > are dumped with their respective tools. > > Thanks in advance. I don't recommend snapshots, as I mentioned. Focus on your backup application and create DR VMs if you want to minimize the time to recovery after a total VM loss is what I recommend. > Bernd -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@
Re: [ClusterLabs] cluster node config - DNS or IP addresses?
On 22/12/15 05:31 PM, Ilia Sokolinski wrote: > Hi, > > What are the best practices with respect to using DNS names vs IP addresses > in pacemaker/corosync configuration? > > Thanks > > Ilia Sokolinski I use `uname -n` and have the hostname resolve via /etc/hosts. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] mail server (postfix)
On 04/06/16 01:27 PM, Dmitri Maziuk wrote: > On 2016-06-04 01:10, Digimer wrote: > >> We're running postfix/dovecot/postgres for our mail on an HA cluster, >> but we put it all in a set of VMs and made the VMs HA on DRBD. > > Hmm. I deliver to ~/Maildir and /home is NFS-mounted all over the place, > so my primary goal is HA NFS server. I'd hesitate to add a VM layer > there. NFS-mounted /home on the mail gateway is what I have now and that > works fine... having that in HA setup is more or less icing on the cake. > > Thanks, > Dimitri Avoiding complexity is a very valid goal in HA, so I won't fault your logic at all. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] how to "switch on" cLVM ?
On 07/06/16 07:23 AM, Lentes, Bernd wrote: >>> Ok. Does DLM takes care that a LV just can be used on one host ? >>> cLVM just takes care that the naming is the same on all nodes, right ? >> >> AFAIK DLM takes care about the LVM Locking cluster wide. > > What does that mean concretely, "LVM Locking cluster wide" ? > I read it always, but no explaination. > Does that mean that a LV just can be accesses from one node ? > That DLM "locks" (in the sense of "blocking the other node") the access ? > Or does the LV has to be a cluster ressource to prevent concurrent access ? DLM is just a distributed lock manager, that's it. LVM uses it to coordinate actions within the cluster, so what LVM does is still up to LVM. I'm not a dev, so I might get this a little wrong, but basically it works like this... You want to create an LV from a clustered VG. The dlm and clvmd daemons are running on all nodes. You type 'lvcreate ...' on node 1. Node 1 asks for a lock from DLM, DLM checks to make sure the lock your asking for doesn't conflict with any locks held elsewhere in the cluster and then it is granted. Now the local machine creates the LV (same that no one else is going to work on the same bits because of the DLM lock), releases the lock and informs the other nodes. The other nodes update their view of the VG and see the new LV. >From the user's perspective, the LV they created on node 1 is instantly seen on the other nodes. >>>>> Later on it's possible that some vm's run on host 1 and some on host 2. >>>>> Does >>>>> clvm needs to be a ressource managed by the cluster manager ? >>>> >>>> Yes, you can live-migrate as well. I do this all the time, except I use >>>> DRBD instead of a SAN and RHEL instead of SUSE, but those are trivial >>>> differences in this case. >>>> >>>>> If i use a fs inside the lv, a "normal" fs like ext3 is sufficient, i >>>>> think. But >>>>> it has to be a cluster ressource, right ? >>>> >>>> You can format a clustered LV with a cluster unaware filesystem just >>>> fine. However, the FS is not made magically cluster aware... If you >>>> mount it on two nodes, you will almost certainly corrupt the FS quickly. >>>> If you want to mount an LV on two+ nodes at once, you need a >>>> cluster-aware file system, life GFS2. >>> >>> No. Pacemaker takes care that the FS is just mounted on one node. >>> So it should not be a problem ? >> >> If you want to be sure to mount an LV on just one Node, you have to activate >> the >> VG exclusively on one node. You have to configure the ressource for the VG >> accordingly. Otherwise it's possible to activate and mount an LV on several >> nodes at the same time, even with a non Cluster FS, e.g. ext4, which would >> end >> up in corrupted FS, most likely. (as mentioned above allready). > > But maybe i want to have some vm's running on host A and others on host B. > Remember: one vm per LV. > So i need access to the VG concurrently from both nodes, right ? > But if the FS from the LV is a cluster ressource, pacemaker takes care the > the FS is mounted just from one node. > I can rely it on it, right ? That's what i read often. > But what if i don't have a FS ? It's possible to have vm's in plain > partitions, which should be a performance advantage. Clustered LVM doesn't care how an LV is used. It only cares that changes won't conflict (thanks to DLM) and that all nodes have the same view of the LVM. So deactivate, activate, grow, create, delete of PVs, VGs and LVs are always seen on all nodes right away. What you do on the LVs is up to you. If boot a VM on node 1 using an LV as backing storage, nothing in LVM stopping you from accessing that LV on another node and destroying your data. For that, you need pacemaker or something else smart enough and cluster aware to prevent uncoordinated access. So if you create a standard FS, you configure pacemaker to ensure it's only mounted on one node at a time (and use fencing!). If you do use a cluster-aware FS, like GFS2, then it's perfectly fine to mount the LV on multiple drives (which is why LVM doesn't get in your way). -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] how to "switch on" cLVM ?
On 06/06/16 01:13 PM, Lentes, Bernd wrote: > Hi, > > i'm currently establishing a two node cluster. I have a FC SAN and two hosts. > My services are integrated into virtual machines (kvm). The vm's should > reside on the SAN. > The hosts are connected to the SAN via FC HBA. Inside the hosts i see already > the volume from the SAN. I'd like to store each vm in a dedicated logical > volume (with or without filesystem in the LV). > Hosts are SLES 11 SP4. LVM2 and LVM2-clvm is installed. > How do i have to "switch on" clvm ? Is locking_type=3 in /etc/lvm/lvm.conf > all which is necessary ? That tells LVM to use cluster locking, you still need to actually provide the cluster locking, called DLM. With the cluster formed (and fencing working!), you should be able to start the clvmd daemon. > In both nodes ? Yes > Restart of the init-scripts afterwards sufficient ? No, DLM needs to be added to the cluster and be running. > And how do i have to proceed afterwards ? > My idea is: > 1. Create a PV > 2. Create a VG > 3. Create several LV's If the VG is created while dlm is running, it should automatically flag the VG as clustered. If not, you will need to tell LVM that the VG is clustered (-cy, iirc). > And because of clvm i only have to do that on one host and the other host > sees all automatically ? Once clvmd is running, any changes made (lvcreate, delete, resize, etc) will immediately appear on the other nodes. > Later on it's possible that some vm's run on host 1 and some on host 2. Does > clvm needs to be a ressource managed by the cluster manager ? Yes, you can live-migrate as well. I do this all the time, except I use DRBD instead of a SAN and RHEL instead of SUSE, but those are trivial differences in this case. > If i use a fs inside the lv, a "normal" fs like ext3 is sufficient, i think. > But it has to be a cluster ressource, right ? You can format a clustered LV with a cluster unaware filesystem just fine. However, the FS is not made magically cluster aware... If you mount it on two nodes, you will almost certainly corrupt the FS quickly. If you want to mount an LV on two+ nodes at once, you need a cluster-aware file system, life GFS2. > Thanks in advance. > > > Bernd > -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] mail server (postfix)
On 03/06/16 01:33 PM, Dimitri Maziuk wrote: > Hi all, > > quick question: is anyone running an MTA on an active-passive cluster? > Specifically, I need to do a stop - wait for drbd fs to move over - > update symlinks - then start again -- on both nodes. So that on the > active node the MTA runs with "mail gateway" postfix config in > /drbd/etc/postfix and on the passive: with "send-only" config in > /etc/postfix. > > Off the top of my head it looks like defining two postfix resources that > both start/stop the same postfix only at different times/on different > nodes should do the trick. Any gotchas I'm not seeing? Better ways to > accomplish it? > > (I know running an MTA that way is not the Approved Way(tm), I have my > reasons for wanting to it like this.) > > TIA We're running postfix/dovecot/postgres for our mail on an HA cluster, but we put it all in a set of VMs and made the VMs HA on DRBD. We go this route because one setup can be adapted to just about any application. This also allows migrations without interruptions. Didn't answer you question, but maybe an alternate approach to consider. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] newbie questions
On 31/05/16 10:41 PM, Jay Scott wrote: > hooray for me, but, how? > > I got about 3/4 of Digimer's list done and got stuck. > I did a pcs cluster status, and, behold, the cluster was up. > I pinged the ClusterIP and it answered. I didn't know what > to do with the 'delay="x"' part, that's the thing I couldn't figure > out. (I've been assuming the delay part is a big deal.) Delay works like this; Both nodes are up, but comms break (switch loop/broadcast storm, STP/stack renegotiation, iptables oops, whatever)... Both nodes declare their peer lost. Node 1's stonith config includes 'delay="15"'. Node 1 looks up how to fence node 2, calls the fence. Node 2 looks up how to fence node 1, calls fence (passing to the agent the delay). The fence agent running on node 1 executes without delay. The fence agent running on node 2 sees a delay of 15 seconds, and sleeps. Node 1 kills node 2 before the sleep exits, thus ensuring that node 1 lived and node 2 died. Assuming you have your services on node 1, then that means no recovery is needed. Now assume that node 1 truly died. Node 2's fence agent would exit the sleep after 15 seconds and proceed to shoot node 1 and then recover any resources that had been on node 1. digimer > However, there are more things for me to read and more experiments > for me to try so I'm good for now. > > Thanks to everyone for the prompt help. > > j. > > On Tue, May 31, 2016 at 5:22 PM, Ken Gaillot <kgail...@redhat.com > <mailto:kgail...@redhat.com>> wrote: > > On 05/31/2016 03:59 PM, Jay Scott wrote: > > Greetings, > > > > Cluster newbie > > Centos 7 > > trying to follow the "Clusters from Scratch" intro. > > 2 nodes (yeah, I know, but I'm just learning) > > > > [root@smoking ~]# pcs status > > Cluster name: > > Last updated: Tue May 31 15:32:18 2016Last change: Tue May 31 > > 15:02:21 > > 2016 by root via cibadmin on smoking > > Stack: unknown > > "Stack: unknown" is a big problem. The cluster isn't aware of the > corosync configuration. Did you do the "pcs cluster setup" step? > > > Current DC: NONE > > 2 nodes and 1 resource configured > > > > OFFLINE: [ mars smoking ] > > > > Full list of resources: > > > > ClusterIP(ocf::heartbeat:IPaddr2):Stopped > > > > PCSD Status: > > smoking: Online > > mars: Online > > > > Daemon Status: > > corosync: active/enabled > > pacemaker: active/enabled > > pcsd: active/enabled > > > > > > What concerns me at the moment: > > I did > > pcs resource enable ClusterIP > > while simultaneously doing > > tail -f /var/log/cluster/corosync.log > > (the only log in there) > > The system log (/var/log/messages or whatever your system has > configured) is usually the best place to start. The cluster software > sends messages of interest to end users there, and it includes messages > from all components (corosync, pacemaker, resource agents, etc.). > > /var/log/cluster/corosync.log (and in some configurations, > /var/log/pacemaker.log) have more detailed log information for > debugging. > > > and nothing happens in the log, but the ClusterIP > > stays "Stopped". Should I be able to ping that addr? > > I can't. > > It also says OFFLINE: and both of my machines are offline, > > though the PCSD says they're online. Which do I trust? > > The first online/offline output is most important, and refers to the > node's status in the actual cluster; the "PSCD" online/offline output > simply tells whether the pcs daemon is running. Typically, the pcs > daemon is enabled at boot and is always running. The pcs daemon is not > part of the clustering itself; it's a front end to configuring and > administering the cluster. > > > [root@smoking ~]# pcs property show stonith-enabled > > Cluster Properties: > > stonith-enabled: false > > > > yet I see entries in the corosync.log referring to stonith. > > I'm guessing that's normal. > > Yes, you can enable stonith at any time, so the stonith daemon will > still run, to stay aware of the cluster status. > > > My corosync.conf file says the quorum is off. > > > > I also don't know what to include in this for any of you to > > help me debug. > > > > Ahh, also,
Re: [ClusterLabs] design question to DRBD
On 22/06/16 08:30 AM, Lentes, Bernd wrote: > Hi, > > we have a two node cluster. It is running a Web-Application. Web-Application > needs a MySQL Database, has static and dynamic (perlscripts) webpages. > I will make the DB HA with MySQL replication. > From time to time it's likely that something in the webapp is changed, so we > have to edit some scripts or install a perl module. > I would like have the changes automatically synchronized to the other side, > without any manual intervention. And also without knowing which node is the > active one. > I'm thinking about putting /srv/www and /usr/lib/perl5 on a DRBD device in an > active/active webapp. For that i need a cluster FS and DLM, right ? > This should synchronize automatically in both directions ? > Do the services have to a ressource for this setup of DRBD or is it possible > to have them as "normal" services, started by init. > Or is it better to have them as ressources because other services will also > run in this HA-system (likely some virtual machines) ? > > Thanks. > > > Bernd Our approach to this problem is a 2-node cluster using DRBD to back virtual machines, and then we make he VMs the HA service. This way, to you and whomever works on the system, you can ignore the HA stuff and treat it like a regular server. If anything happens to the current host, the server reboots on the good node. Being a VM, reboots are generally quite quick. We've detailed how we build our system in detail here: https://alteeve.ca/w/AN!Cluster_Tutorial_2 However, you may want to use pacemaker instead of cman+rgmanager. In that case, the tutorial above is still useful as it explains the approach. You can just mentally switch "cman+rgmanager" for "pacemaker", adjust the actual commands and the rest of the guide works fine. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Recovering after split-brain
On 22/06/16 01:09 AM, Nikhil Utane wrote: > We are not using virtual IP. There is a separate discovery mechanism > between the server and client. The client will reach out to new server > only if it is incommunicado with the old one. That's fine, but it really doesn't change anything. Whether you're using a shared IP, shared storage or something else, it's all the same to pacemaker in the end. > On Tue, Jun 21, 2016 at 8:27 PM, Dmitri Maziuk <dmitri.maz...@gmail.com > <mailto:dmitri.maz...@gmail.com>> wrote: > > On 2016-06-20 17:19, Digimer wrote: > > Nikhil indicated that they could switch where traffic went up-stream > without issue, if I understood properly. > > > They have some interesting setup, but that notwithstanding: if split > brain happens some clients will connect to "old master" and some: to > "new master", dep. on arp update. If there's a shared resource > unavailable on one node, clients going there will error out. The > other ones will not. It will work for some clients. > > Cf. both nodes going into stonith deathmatch and killing each other: > the service now is not available for all clients. What I don't get > is the blanket assertion that this "more highly" available that > option #1. > > Dimitri > > > ___ > Users mailing list: Users@clusterlabs.org <mailto:Users@clusterlabs.org> > http://clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > > > > > ___ > Users mailing list: Users@clusterlabs.org > http://clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Node is silently unfenced if transition is very long
On 21/06/16 12:19 PM, Ken Gaillot wrote: > On 06/17/2016 07:05 AM, Vladislav Bogdanov wrote: >> 03.05.2016 01:14, Ken Gaillot wrote: >>> On 04/19/2016 10:47 AM, Vladislav Bogdanov wrote: >>>> Hi, >>>> >>>> Just found an issue with node is silently unfenced. >>>> >>>> That is quite large setup (2 cluster nodes and 8 remote ones) with >>>> a plenty of slowly starting resources (lustre filesystem). >>>> >>>> Fencing was initiated due to resource stop failure. >>>> lustre often starts very slowly due to internal recovery, and some such >>>> resources were starting in that transition where another resource >>>> failed to stop. >>>> And, as transition did not finish in time specified by the >>>> "failure-timeout" (set to 9 min), and was not aborted, that stop >>>> failure was successfully cleaned. >>>> There were transition aborts due to attribute changes, after that >>>> stop failure happened, but fencing >>>> was not initiated for some reason. >>> >>> Unfortunately, that makes sense with the current code. Failure timeout >>> changes the node attribute, which aborts the transition, which causes a >>> recalculation based on the new state, and the fencing is no longer >> >> Ken, could this one be considered to be fixed before 1.1.15 is released? > > I'm planning to release 1.1.15 later today, and this won't make it in. > > We do have several important open issues, including this one, but I > don't want them to delay the release of the many fixes that are ready to > go. I would only hold for a significant issue introduced this cycle, and > none of the known issues appear to qualify. I wonder if it would be worth appending a "known bugs/TODO" list to the release announcements? Partly as a "heads-up" and partly as a way to show folks what might be coming in .x+1. >> I was just hit by the same in the completely different setup. >> Two-node cluster, one node fails to stop a resource, and is fenced. >> Right after that second node fails to activate clvm volume (different >> story, need to investigate) and then fails to stop it. Node is scheduled >> to be fenced, but it cannot be because first node didn't come up yet. >> Any cleanup (automatic or manual) of a resource failed to stop clears >> node state, removing "unclean" state from a node. That is probably not >> what I could expect (resource cleanup is a node unfence)... >> Honestly, this potentially leads to a data corruption... >> >> Also (probably not related) there was one more resource stop failure (in >> that case - timeout) prior to failed stop mentioned above. And that stop >> timeout did not lead to fencing by itself. >> >> I have logs (but not pe-inputs/traces/blackboxes) from both nodes, so >> any additional information from them can be easily provided. >> >> Best regards, >> Vladislav -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Recovering after split-brain
On 21/06/16 10:57 AM, Dmitri Maziuk wrote: > On 2016-06-20 17:19, Digimer wrote: > >> Nikhil indicated that they could switch where traffic went up-stream >> without issue, if I understood properly. > > They have some interesting setup, but that notwithstanding: if split > brain happens some clients will connect to "old master" and some: to > "new master", dep. on arp update. If there's a shared resource > unavailable on one node, clients going there will error out. The other > ones will not. It will work for some clients. > > Cf. both nodes going into stonith deathmatch and killing each other: the > service now is not available for all clients. What I don't get is the > blanket assertion that this "more highly" available that option #1. > > Dimitri As I've explained many times (here and on IRC); If you don't need to coordinate services/access, you don't need HA. If you do need to coordinate services/access, you need fencing. So if Nikhil really believes s/he doesn't need fencing and that split-brains are OK, then drop HA. If that is not the case, then s/he needs to implement fencing in pacemaker. It's pretty much that simple. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Recovering after split-brain
On 21/06/16 01:27 PM, Dimitri Maziuk wrote: > On 06/21/2016 12:13 PM, Andrei Borzenkov wrote: > >> You should not run pacemaker without some sort of fencing. This need not >> be network-controlled power socket (and tiebreaker is not directly >> related to fencing). > > Yes it can be sysadmin-controlled power socket. It has to be a power > socket, if you don't trust me, read Dejan's list of fencing devices. You can now use redundant and complex fencing configurations in pacemaker. Our company always has this setup; IPMI is the primary fence method (when it works, we can trust 'off' 100%, but it draws power from the host and is thus vulnerable) Pair of switched PDUs as backup fencing (when it works, you are confident that the outlets are opened, but you have to make sure the cables are in the right place. However, it is entirely external to the target). > Tiebreaking is directly related to figuring out which of the two nodes > is to be fenced. because neither of them can tell on its own. See my comment on 'delay="15"'. You do NOT need a 3 node cluter/tie breaker. We've run nothing but 2-node clusters for years all over north america and we've heard of people running our system globally. With the above fence setup and proper delay, it has never once been a problem. >> I fail to see how heartbeat makes any difference here, sorry. > > Third node and remote-controlled PDU were not a requirement for > haresources mode. If I wanted to run it so that when it breaks I get to > keep the pieces, I could. You technically can in pacemaker, too, but it's dumb in any HA environment. As soon as you make assumptions, you open up the chance of being wrong. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] restarting pacemakerd
On 19/06/16 01:59 AM, Andrei Borzenkov wrote: > 18.06.2016 22:04, Dmitri Maziuk пишет: >> On 2016-06-18 05:15, Ferenc Wágner wrote: >> ... >>> On the other hand, one could argue that restarting failed services >>> should be the default behavior of systemd (or any init system). Still, >>> it is not. >> >> As an off-topic snide comment, I never understood the thinking behind >> that: restarting without removing the cause of the failure will just >> make it fail again. If at first you don't succeed, then try, try, try >> again? >> > > Some problems are transient and restarting may succeed (most obvious > example is program crash which includes OS kernel crash). What is needed > here is rate limiting so restart is not attempted indefinitely. Rgmanager offers this via "max_restarts". I'd be shocked if there wasn't a version of this in pacemaker already, given that it has for more flexibility than rgmanager. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Resource ocf:heartbeat:asterisk fails to start
On 17/06/16 03:05 PM, FreeSoftwareServers wrote: > Just wanted to share! > > This misinformation got me started down the wrong path, which was > running user/group root/root. Good old internet! > > http://www.klaverstyn.com.au/david/wiki/index.php?title=Asterisk_Cluster They disable stonith, so ya, not a great resource. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Recovering after split-brain
On 20/06/16 09:30 AM, Nikhil Utane wrote: > Hi, > > For our solution we are making a conscious choice to not use > quorum/fencing as for us service availability is more important than > having 2 nodes take up the same active role. Split-brain is not an issue > for us (at least i think that way) since we have a second line of Then wouldn't it be a lot better to just run your services on both nodes all the time and take HA out of the picture? Availability is predicated on building the simplest system possible. If you have no concerns about uncoordinated access, then make like simpler and remove pacemaker entirely. > defense. We have clients who can connect to only one of the two active > nodes. So in that sense, even if we end up with 2 nodes becoming active, > since the clients can connect to only 1 of the active node, we should > not have any issue. > > Now my question is what happens after recovering from split-brain since > the resource will be active on both the nodes. From application point of > view we want to be able to find out which node is servicing the clients > and keep that operational and make the other one as standby. > > Does Pacemaker make it easy to do this kind of thing through some means? > Are there any issues that I am completely unaware due to letting > split-brain occur? > > -Thanks > Nikhil -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] DLM standalone without crm ?
Yes, technically. I've not played with it stand-alone, and I believe it will still need corosync for internode communication and membership. Also, if a node fails and can't be fenced, I believe it will block. Others here might be able to speak more authoritatively than I. madi On 24/06/16 11:34 AM, Lentes, Bernd wrote: > Hi, > > is it possible to have a DLM running without CRM ? Just for playing around a > bit and get used to some stuff. > > > Bernd > -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] restarting pacemakerd
On 18/06/16 03:04 PM, Dmitri Maziuk wrote: > On 2016-06-18 05:15, Ferenc Wágner wrote: > ... >> On the other hand, one could argue that restarting failed services >> should be the default behavior of systemd (or any init system). Still, >> it is not. > > As an off-topic snide comment, I never understood the thinking behind > that: restarting without removing the cause of the failure will just > make it fail again. If at first you don't succeed, then try, try, try > again? > > Dimitri When your focus is availability, restarting makes sense. What you want to do is alert an admin that a restart was needed, so that he or she can investigate the cause. Pacemaker 1.1.15 allows for this alerting now. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] DLM hanging when corosync is OK causes cluster to hang
Hi all, We hit a strange problem where a RAID controller on a node failed, causing DLM (gfs2/clvmd) to hang, but the node was never fenced. I assume this was because corosync was still working. Is there a way in rhel6/cman/rgmanager to have a node suicide or get fenced in a condition like this? -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Pacemaker 1.1.14 released
Congrats!! On 14/01/16 04:49 PM, Ken Gaillot wrote: > ClusterLabs is proud to announce the latest release of the Pacemaker > cluster resource manager, version 1.1.14. The source code is available at: > > https://github.com/ClusterLabs/pacemaker/releases/tag/Pacemaker-1.1.14 > > This version introduces some valuable new features: > > * Resources will now start as soon as their state has been confirmed on > all nodes and all dependencies have been satisfied, rather than waiting > for the state of all resources to be confirmed. This allows for faster > startup of some services, and more even startup load. > > * Fencing topology levels can now be applied to all nodes whose name > matches a configurable pattern, or that have a configurable node attribute. > > * When a fencing topology level has multiple devices, reboots are now > automatically mapped to all-off-then-all-on, allowing much simplified > configuration of redundant power supplies. > > * Guest nodes can now be included in groups, which simplifies the common > Pacemaker Remote use case of a grouping a storage device, filesystem and VM. > > * Clone resources have a new clone-min metadata option, specifying that > a certain number of instances must be running before any dependent > resources can run. This is particularly useful for services behind a > virtual IP and haproxy, as is often done with OpenStack. > > As usual, the release includes many bugfixes and minor enhancements. For > a more detailed list of changes, see the change log: > > https://github.com/ClusterLabs/pacemaker/blob/1.1/ChangeLog > > Feedback is invited and welcome. > -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Antw: Re: DLM fencing
On 10/02/16 02:40 AM, Ulrich Windl wrote: >>>> Digimer <li...@alteeve.ca> schrieb am 08.02.2016 um 20:03 in Nachricht > <56b8e68a.1060...@alteeve.ca>: >> On 08/02/16 01:56 PM, Ferenc Wágner wrote: >>> Ken Gaillot <kgail...@redhat.com> writes: >>> >>>> On 02/07/2016 12:21 AM, G Spot wrote: >>>> >>>>> Thanks for your response, am using ocf:pacemaker:controld resource >>>>> agent and stonith-enabled=false do I need to configure stonith device >>>>> to make this work? >>>> >>>> Correct. DLM requires access to fencing. >>> >>> I've ment to explore this connection for long, but never found much >>> useful material on the subject. How does DLM fencing fit into the >>> modern Pacemaker architecture? Fencing is a confusing topic in itself >>> already (fence_legacy, fence_pcmk, stonith, stonithd, stonith_admin), >>> then dlm_controld can use dlm_stonith to proxy fencing requests to >>> Pacemaker, and it becomes hopeless... :) >>> >>> I'd be grateful for a pointer to a good overview document, or a quick >>> sketch if you can spare the time. To invoke some concrete questions: >>> When does DLM fence a node? Is it necessary only when there's no >>> resource manager running on the cluster? Does it matter whether >>> dlm_controld is run as a standalone daemon or as a controld resource? >>> Wouldn't Pacemaker fence a failing node itself all the same? Or is >>> dlm_stonith for the case when only the stonithd component of Pacemaker >>> is active somehow? >> >> DLM is a thing onto itself, and some tools like gfs2 and clustered-lvm >> use it to coordinate locking across the cluster. If a node drops out, >> the cluster informs dlm and it blocks until the lost node is confirmed >> fenced. Then it reaps the lost locks and recovery can begin. >> >> If fencing fails or is not configured, DLM never unblocks and anything >> using it is left hung (by design, better to hang than risk corruption). >> >> One of many reasons why fencing is critical. > > I'm not deeply in DLM, but it seems to me DLM can run standalone, or in the > cluster infrastructure (we only use it inside the cluster). When running > standalone, it makes sense that DLM has ist own fencing, but when running > inside the cluster infrastructure, I'd expect tha tthe cluster's fencing > mechanisms are used (maybe just because if the better logging of reasons). To be clear; DLM does NOT have it's own fencing. It relies on the cluster's fencing. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] OT - Someone mailed me something, was it someone here?
OK, so this is pretty off topic, but I am racking my brains... Someone sent me a pretty neat glassware thing with no return address. It shipped from China, and I can't think of anyone I know there. I didn't order anything from there, so I doubt it was an accidental shipment. This got me wondering; Was it someone from here maybe? It's definitely something meant for me so I was wondering if someone I've helped found my address... A mystery! If it was, thanks to whoever sent it. It it wasn't anyone here, then sorry for the line noise. :) -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] DLM fencing
On 08/02/16 03:55 PM, G Spot wrote: > Hi Ken, > > Am trying to create shared storage with clvm/gfs2 and when I try to > fence I only see scsi option but my storage is conencted through FC is > there any otherways can I fence my 1G stonith device other than scsi? fencing of a lost node with clvmd/gfs2 is no different than normal cluster fencing. To be clear, DLM does NOT fence, it simply waits for the cluster to fence. So you can use IPMI, switched PDUs or whatever else is available in your environment. > On Mon, Feb 8, 2016 at 2:03 PM, Digimer <li...@alteeve.ca > <mailto:li...@alteeve.ca>> wrote: > > On 08/02/16 01:56 PM, Ferenc Wágner wrote: > > Ken Gaillot <kgail...@redhat.com <mailto:kgail...@redhat.com>> writes: > > > >> On 02/07/2016 12:21 AM, G Spot wrote: > >> > >>> Thanks for your response, am using ocf:pacemaker:controld resource > >>> agent and stonith-enabled=false do I need to configure stonith device > >>> to make this work? > >> > >> Correct. DLM requires access to fencing. > > > > I've ment to explore this connection for long, but never found much > > useful material on the subject. How does DLM fencing fit into the > > modern Pacemaker architecture? Fencing is a confusing topic in itself > > already (fence_legacy, fence_pcmk, stonith, stonithd, stonith_admin), > > then dlm_controld can use dlm_stonith to proxy fencing requests to > > Pacemaker, and it becomes hopeless... :) > > > > I'd be grateful for a pointer to a good overview document, or a quick > > sketch if you can spare the time. To invoke some concrete questions: > > When does DLM fence a node? Is it necessary only when there's no > > resource manager running on the cluster? Does it matter whether > > dlm_controld is run as a standalone daemon or as a controld resource? > > Wouldn't Pacemaker fence a failing node itself all the same? Or is > > dlm_stonith for the case when only the stonithd component of Pacemaker > > is active somehow? > > DLM is a thing onto itself, and some tools like gfs2 and clustered-lvm > use it to coordinate locking across the cluster. If a node drops out, > the cluster informs dlm and it blocks until the lost node is confirmed > fenced. Then it reaps the lost locks and recovery can begin. > > If fencing fails or is not configured, DLM never unblocks and anything > using it is left hung (by design, better to hang than risk corruption). > > One of many reasons why fencing is critical. > > -- > Digimer > Papers and Projects: https://alteeve.ca/w/ > What if the cure for cancer is trapped in the mind of a person without > access to education? > > ___ > Users mailing list: Users@clusterlabs.org <mailto:Users@clusterlabs.org> > http://clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > > > > > ___ > Users mailing list: Users@clusterlabs.org > http://clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Antw: Re: Antw: Re: DLM fencing
On 11/02/16 02:37 AM, Ulrich Windl wrote: >>>> Digimer <li...@alteeve.ca> schrieb am 10.02.2016 um 17:32 in Nachricht > <56bb6637.6090...@alteeve.ca>: >> On 10/02/16 02:40 AM, Ulrich Windl wrote: > > [...] >>>> If fencing fails or is not configured, DLM never unblocks and anything >>>> using it is left hung (by design, better to hang than risk corruption). >>>> >>>> One of many reasons why fencing is critical. >>> >>> I'm not deeply in DLM, but it seems to me DLM can run standalone, or in the >>> cluster infrastructure (we only use it inside the cluster). When running >>> standalone, it makes sense that DLM has ist own fencing, but when running >>> inside the cluster infrastructure, I'd expect tha tthe cluster's fencing >>> mechanisms are used (maybe just because if the better logging of reasons). >> >> To be clear; DLM does NOT have it's own fencing. It relies on the >> cluster's fencing. > > OK, is this true for cLVM and O2CB as well? I always felt some of those is > doing a fencing themselves as soon as they fail to communicate with DLM. So > the first guess was it's DLM... I can't speak to o2cb, never used it. However, clustered LVM, gfs2 and rgmanager use DLM, and in all cases, DLM does nothing but block until it is told that the fence was successful. It plays no active role in fencing. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Antw: Re: DLM fencing
On 11/02/16 04:42 AM, Vladislav Bogdanov wrote: > 10.02.2016 19:32, Digimer wrote: > [snip] >> >> To be clear; DLM does NOT have it's own fencing. It relies on the >> cluster's fencing. >> > > Actually, dlm4 can use fence-agents directly (device keyword in > dlm.conf). Default is to use dlm_stonith though. Really?! Neat! -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] The cluster stack in Debian
On 29/01/16 12:01 PM, Richard B Winters wrote: > Hello, > > I thought it might be nice (with a good recommendation from the ClusterLabs' > IRC community), to send an email to the users ML which describes the current > state of the cluster stack as it exists in Debian. > > As some may already know, the Debian-HA team had missed the 'Freeze' date for > the Jessie release of Debian. The result of this was that any packages which > are maintained by the Debian-HA-Maintainers team - and which were not ready > for migration to testing; were not included in the release of Debian Jessie. > This is closely followed by a policy which excludes the aforementioned > packages from the Jessie (main) repository permanently. > > Our only option at this point is to prepare the cluster stack and provide an > alternative method for Debian users to utilize said packages - until the next > Debian release. > > With that said, the team has become active and is full of new members and > sponsors. We have been working quite diligently since October of 2014, in > order to prepare the cluster stack (the latest stack) for Debian. So far we > have uploaded (into unstable/sid) : > > Libqb > Pacemaker > Corosync > DLM > DRBD/Utils > CRMSH > Fence-Agents > Ruby-Rpam-Ruby19 > Ruby-Monkey-Lib > PCS > > Some of the above packages have already migrated to testing, others are still > in the NEW queue. > > We also have prepared, and are reviewing prior to upload: > > Cluster-Glue > Resource-Agents > > > Any who are interested in getting involved and helping out, we have much left > to do - even once all the packages are in Debian sid/stretch. For more > information, one could visit our wiki [1], browse our public repository [2], > or check our homepage [3] from time to time for updates (once we start > regularly posting there). > > I have also been personally hosting a PPA [4] for Debian users, until we are > able to provide a more official solution. Please be advised that the > packages on my PPA are not actually 100% up-to-date at this time, but are > still usable (I'll be updating soon!). > > I'll do my best to post updates regularly, if for no other reason; to keep > interested parties informed of our progress! There is also the clusterlabs developers list (http://clusterlabs.org/mailman/listinfo/developers), if you want to CC development progress reports there. As I said in IRC, I am looking forward to being able to add Debian to the list of well supported distros. Please keep up the good work! > References: > > [1] https://wiki.debian.org/Debian-HA/ > [2] https://anonscm.debian.org/cgit/debian-ha/ > [3] https://debian-ha.alioth.debian.org/ > [4] https://ppa.mmogp.com/ > > > Best regards, > > > Richard B Winters (Rik/devrikx) > > ___ > Users mailing list: Users@clusterlabs.org > http://clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[ClusterLabs] Moving Anvil! and Striker development to this list
Hi all, The Anvil! and Striker (and now ScanCore) projects used to have their own low-volume mailing list. Recently it started getting some use and, being the one who advocated the strongest for the merger of lists, I decided to close it down and migrate over to here. To give a brief overview of what Anvil!, Striker and ScanCore is; The "Anvil!" is the name of our 2-node HA cluster based on this[1] tutorial. It's also the term we generally use for the full HA stack we've built. Striker[2] is a web-based front-end for Managing Anvil! clusters and the servers that run on them. It has been in very heavy development the last year and change and we're getting close to the version 2 release "real soon now(tm)". ScanCore[3] is a new component that runs on both Anvil! nodes and Striker dashboards. It was initially announced at the HA Summit in Brno in 2015 is it's release will coincide with Striker v2's release. It is an alert, predictive failure and mitigation program that is technically stand-alone but has been built into the heart of the Anvil! platform. It is inherently redundant does things like watch for faults, try to recover from known problems autonomously and alert users to these issues. I've been somewhat reluctant to move our list over because Alteeve, our company and the company that builds and maintains the Anvil!, Striker and ScanCore is for-profit. However, _everything_ we do is open source, so I hope that won't be held against us. :) If anyone has any comments or concerns about us moving our project discussion to this list, please let me know and I'll do what I can to make sure we address those concerns. Cheers! digimer 1. https://alteeve.ca/w/AN!Cluster_Tutorial_2 2. https://github.com/digimer/striker 3. https://github.com/digimer/striker/tree/master/ScanCore -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] booth release v1.0
On 18/03/16 09:25 AM, Dejan Muhamedagic wrote: > Hello everybody, > > I'm happy to announce that the booth repository was yesterday > tagged as v1.0: > > https://github.com/ClusterLabs/booth/releases/tag/v1.0 > > There were very few patches since the v1.0 rc1. The complete > list of changes is available in the ChangeLog: > > https://github.com/ClusterLabs/booth/blob/v1.0/ChangeLog > > The binaries are provided for some Linux distributions. Currently, > there are packages for CentOS7 and various openSUSE versions: > > http://download.opensuse.org/repositories/network:/ha-clustering:/Stable/ > > If you don't know what booth is and what is it good for, please > check the README at the bottom of the git repository home page: > > https://github.com/ClusterLabs/booth > > Cheers, > > Dejan Hey hey, congratulations!! -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] GFS and cLVM fencing requirements with DLM
On 15/03/16 02:12 PM, Ferenc Wágner wrote: > Hi, > > I'm referring here to an ancient LKML thread introducing DLM. In > http://article.gmane.org/gmane.linux.kernel/299788 David Teigland > states: > > GFS requires that a failed node be fenced prior to gfs being told to > begin recovery for that node > > which sounds very plausible as according to that thread DLM itself does > not make sure fencing happens before DLM recovery, thus DLM locks could > be granted to others before the failed node is fenced (if at all). > > Now more than ten years passed and I wonder > > 1. if the above is still true (or maybe my interpretation was wrong to >start with) Yes, fencing is required. DLM will not begin recovery until it is told that the lost node is confirmed fenced. > 2. how it is arranged for in the GFS2 code (I failed to find it with >naive search phrases) > > 3. whether clvmd does the same Anything that uses DLM requires fencing, which includes clustered LVM. (Really, all clusters require fencing to avoid split-brains). > 4. what are the pros/cons of disabling DLM fencing (even the dlm_stonith >proxy) and leaving fencing fully to the resource manager (Pacemaker) Pacemaker's fencing will inform DLM when the node has been terminated. If EL6-based clusters, this is done via 'fence_pcmk' config in cman's cluster.conf (which simply asks pacemaker to do the fence and report back when successful). -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] reproducible split brain
On 16/03/16 04:04 PM, Christopher Harvey wrote: > On Wed, Mar 16, 2016, at 04:00 PM, Digimer wrote: >> On 16/03/16 03:59 PM, Christopher Harvey wrote: >>> I am able to create a split brain situation in corosync 1.1.13 using >>> iptables in a 3 node cluster. >>> >>> I have 3 nodes, vmr-132-3, vmr-132-4, and vmr-132-5 >>> >>> All nodes are operational and form a 3 node cluster with all nodes are >>> members of that ring. >>> vmr-132-3 ---> Online: [ vmr-132-3 vmr-132-4 vmr-132-5 ] >>> vmr-132-4 ---> Online: [ vmr-132-3 vmr-132-4 vmr-132-5 ] >>> vmr-132-5 ---> Online: [ vmr-132-3 vmr-132-4 vmr-132-5 ] >>> so far so good. >>> >>> running the following on vmr-132-4 drops all incoming (but not outgoing) >>> packets from vmr-132-3: >>> # iptables -I INPUT -s 192.168.132.3 -j DROP >>> # iptables -L >>> Chain INPUT (policy ACCEPT) >>> target prot opt source destination >>> DROP all -- 192.168.132.3anywhere >>> >>> Chain FORWARD (policy ACCEPT) >>> target prot opt source destination >>> >>> Chain OUTPUT (policy ACCEPT) >>> target prot opt source destination >>> >>> vmr-132-3 ---> Online: [ vmr-132-3 vmr-132-4 vmr-132-5 ] >>> vmr-132-4 ---> Online: [ vmr-132-4 vmr-132-5 ] >>> vmr-132-5 ---> Online: [ vmr-132-4 vmr-132-5 ] >>> >>> vmr-132-3 thinks everything is normal and continues to provide service, >>> vmr-132-4 and 5 form a new ring, achieve quorum and provide the same >>> service. Splitting the link between 3 and 4 in both directions isolates >>> vmr 3 from the rest of the cluster and everything fails over normally, >>> so only a unidirectional failure causes problems. >>> >>> I don't have stonith enabled right now, and looking over the >>> pacemaker.log file closely to see if 4 and 5 would normally have fenced >>> 3, but I didn't see any fencing or stonith logs. >>> >>> Would stonith solve this problem, or does this look like a bug? >> >> It should, that is its job. > > is there some log I can enable that would say > "ERROR: hey, I would use stonith here, but you have it disabled! your > warranty is void past this point! do not pass go, do not file a bug"? If I had it my way, that would be printed to STDOUT when you start pacemaker without stonith... -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Antw: Re: reproducible split brain
On 17/03/16 01:57 PM, vija ar wrote: > root file system is fine ... > > but fencing is not a necessity a cluster shld function without it .. i > see the issue with corosync which has all been .. a inherent way of not > working neatly or smoothly .. Absolutely wrong. If you have a service that can run on both/all nodes at the same time without coordination, you don't need a cluster, just run your services everywhere. If that's not the case, then you need fencing so that the (surviving) node(s) can be sure that they know where services are and are not running. > for e.g. take an issue where the live node is hung in db cluster .. now > db perspective transactions r not happening and tht is fine as the node > is having some issue .. now there is no need to fence this hung node but > just to switch over to passive one .. but tht doesnt happens and fencing > takes place either by reboot or shut .. which further makes the DB dirty > or far more than tht in non-recoverable state which wouldnt have happen > if a normal switch to other node as in cluster would have happened ... > > i see fencing is not a solution its only required to forcefully take > control which is not the case always > > On Thu, Mar 17, 2016 at 12:49 PM, Ulrich Windl > <ulrich.wi...@rz.uni-regensburg.de > <mailto:ulrich.wi...@rz.uni-regensburg.de>> wrote: > > >>> Christopher Harvey <c...@eml.cc> schrieb am 16.03.2016 um 21:04 > in Nachricht > <1458158684.122207.551267810.11f73...@webmail.messagingengine.com > > <mailto:1458158684.122207.551267810.11f73...@webmail.messagingengine.com>>: > [...] > >> > Would stonith solve this problem, or does this look like a bug? > >> > >> It should, that is its job. > > > > is there some log I can enable that would say > > "ERROR: hey, I would use stonith here, but you have it disabled! your > > warranty is void past this point! do not pass go, do not file a bug"? > > What should the kernel say during boot if the user has not defined a > root file system? > > Maybe the "stonith-enabled=false" setting should be called either > "data-corruption-mode=true" or "hang-forever-on-error=true" ;-) > > Regards, > Ulrich > > > > ___ > Users mailing list: Users@clusterlabs.org <mailto:Users@clusterlabs.org> > http://clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > > > > > _______ > Users mailing list: Users@clusterlabs.org > http://clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] [DRBD-user] DRBD fencing issue on failover causes resource failure
On 16/03/16 01:51 PM, Tim Walberg wrote: > Is there a way to make this work properly without STONITH? I forgot to mention > that both nodes are virtual machines (QEMU/KVM), which makes STONITH a minor > challenge. Also, since these symptoms occur even under "pcs cluster standby", > where STONITH *shouldn't* be invoked, I'm not sure if that's the entire > answer. Not sanely, no. All clusters need HA (if you don't need to coordinate services, you don't need HA). In shared storage though, it becomes extra critical. As for fencing KVM; fence_xvm or fence_virsh (the former being ideal for production, the later being easier to setup but more fragile, so only good for testing). -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] IPMI working but evacuations don't work
On 31/03/16 02:26 AM, Moiz Arif wrote: > Hi, > > I am working on VM evacuations and i have noticed that when my compute > node's network is disconnected there is call from STONITH to fence the > node and my node gets rebooted. But the VMs are not evacuated. I have > checked the logs and do not see the NovaEvacuate call. > > What can be the issue here? > > Thanks. > Moiz I assume "evacuations" means "recovered on the healthy node"? Please share more information about your setup (program versions, OS) and your configuration. Also, please share the log files from the surviving node starting just before you crashed the node until a few minutes after. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Totem is unable to form a cluster because of an operating system or network fault
gt; > listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes > > 10:08:24.484892 IP 10.1.0.6.44299 > 10.1.0.7.5405: UDP, length 26 > > > > > > Tested also sending UDP packets from node 2 – all ok. > > > > So connectivity seems to be ok. > > > > Port scanner also shows the port as Open > > > > > > root@node-1:/home/dinor# nmap -sUV 10.1.0.7 -p 5402-5405 > > > > Starting Nmap 5.21 ( http://nmap.org ) at 2016-04-12 10:31 UTC > > Nmap scan report for node-2 (10.1.0.7) > > Host is up (0.00060s latency). > > PORT STATE SERVICE VERSION > > 5402/udp closedunknown > > 5403/udp closedunknown > > 5404/udp closedunknown > > *5405/udp open|filtered unknown* > > MAC Address: 12:34:56:78:9A:BC (Unknown) > > > > Service detection performed. Please report any incorrect results at > http://nmap.org/submit/ . > > Nmap done: 1 IP address (1 host up) scanned in 79.07 seconds > > > > > > There is no FW and no selinux enabled > > > help appreciated. They disable stonith, so I question the entire tutorial... Suggesting using ssh for fencing shows a real flawed understanding. If you install CentOS 7 instead of Ubuntu, you will be able to follow the Cluster from Scratch tutorial on Clusterlabs.org website. It is written and maintained by the Pacemaker author and includes 'pcs' that will configure the base cluster for you. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] reproducible split brain
On 16/03/16 03:59 PM, Christopher Harvey wrote: > I am able to create a split brain situation in corosync 1.1.13 using > iptables in a 3 node cluster. > > I have 3 nodes, vmr-132-3, vmr-132-4, and vmr-132-5 > > All nodes are operational and form a 3 node cluster with all nodes are > members of that ring. > vmr-132-3 ---> Online: [ vmr-132-3 vmr-132-4 vmr-132-5 ] > vmr-132-4 ---> Online: [ vmr-132-3 vmr-132-4 vmr-132-5 ] > vmr-132-5 ---> Online: [ vmr-132-3 vmr-132-4 vmr-132-5 ] > so far so good. > > running the following on vmr-132-4 drops all incoming (but not outgoing) > packets from vmr-132-3: > # iptables -I INPUT -s 192.168.132.3 -j DROP > # iptables -L > Chain INPUT (policy ACCEPT) > target prot opt source destination > DROP all -- 192.168.132.3anywhere > > Chain FORWARD (policy ACCEPT) > target prot opt source destination > > Chain OUTPUT (policy ACCEPT) > target prot opt source destination > > vmr-132-3 ---> Online: [ vmr-132-3 vmr-132-4 vmr-132-5 ] > vmr-132-4 ---> Online: [ vmr-132-4 vmr-132-5 ] > vmr-132-5 ---> Online: [ vmr-132-4 vmr-132-5 ] > > vmr-132-3 thinks everything is normal and continues to provide service, > vmr-132-4 and 5 form a new ring, achieve quorum and provide the same > service. Splitting the link between 3 and 4 in both directions isolates > vmr 3 from the rest of the cluster and everything fails over normally, > so only a unidirectional failure causes problems. > > I don't have stonith enabled right now, and looking over the > pacemaker.log file closely to see if 4 and 5 would normally have fenced > 3, but I didn't see any fencing or stonith logs. > > Would stonith solve this problem, or does this look like a bug? It should, that is its job. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Antw: Re: reproducible split brain
On 19/03/16 10:10 AM, Dennis Jacobfeuerborn wrote: > On 18.03.2016 00:50, Digimer wrote: >> On 17/03/16 07:30 PM, Christopher Harvey wrote: >>> On Thu, Mar 17, 2016, at 06:24 PM, Ken Gaillot wrote: >>>> On 03/17/2016 05:10 PM, Christopher Harvey wrote: >>>>> If I ignore pacemaker's existence, and just run corosync, corosync >>>>> disagrees about node membership in the situation presented in the first >>>>> email. While it's true that stonith just happens to quickly correct the >>>>> situation after it occurs it still smells like a bug in the case where >>>>> corosync in used in isolation. Corosync is after all a membership and >>>>> total ordering protocol, and the nodes in the cluster are unable to >>>>> agree on membership. >>>>> >>>>> The Totem protocol specifies a ring_id in the token passed in a ring. >>>>> Since all of the 3 nodes but one have formed a new ring with a new id >>>>> how is it that the single node can survive in a ring with no other >>>>> members passing a token with the old ring_id? >>>>> >>>>> Are there network failure situations that can fool the Totem membership >>>>> protocol or is this an implementation problem? I don't see how it could >>>>> not be one or the other, and it's bad either way. >>>> >>>> Neither, really. In a split brain situation, there simply is not enough >>>> information for any protocol or implementation to reliably decide what >>>> to do. That's what fencing is meant to solve -- it provides the >>>> information that certain nodes are definitely not active. >>>> >>>> There's no way for either side of the split to know whether the opposite >>>> side is down, or merely unable to communicate properly. If the latter, >>>> it's possible that they are still accessing shared resources, which >>>> without proper communication, can lead to serious problems (e.g. data >>>> corruption of a shared volume). >>> >>> The totem protocol is silent on the topic of fencing and resources, much >>> the way TCP is. >>> >>> Please explain to me what needs to be fenced in a cluster without >>> resources where membership and total message ordering are the only >>> concern. If fencing were a requirement for membership and ordering, >>> wouldn't stonith be part of corosync and not pacemaker? >> >> Corosync is a membership and communication layer (and in v2+, a quorum >> provider). It doesn't care about or manage anything higher up. So it >> doesn't care about fencing itself. >> >> It simply cares about; >> >> * Who is in the cluster? >> * How do the members communicate? >> * (v2+) Is there enough members for quorum? >> * Notify resource managers of membership changes (join or loss). >> >> The resource manager, pacemaker or rgmanager, care about resources, so >> it is what cares about making smart decisions. As Ken pointed out, >> without fencing, it can never tell the difference between no access and >> dead peer. >> >> This is (again) why fencing is critical. > > I think the key issue here is that people think about corosync they > believe there can only be two state for membership (true or false) when > in reality there are three possible states: true, false and unknown. > > The problem then is that corosync apparently has no built-in way to deal > with the "unknown" situation and requires guidance from an external > entity for that (in this case pacemakers fencing). > > This means that corosync alone simply cannot give you reliable > membership guarantees. I strictly requires external help to be able to > provide that. > > Regards, > Dennis I'm not sure that is accurate. If corosync declares a node lost (failed to receive X tokens in Y time), the node is declared lost and it reforms a new cluster, without the lost member. So from corosync's perspective, the lost node is no longer a member (it won't receive messages). It is possible that the lost node might itself be alive, in which case it's corosync will do the same thing (reform a new cluster, possibly with itself as the sole member). If you're trying to have corosync *do* something, then that is missing the point of corosync, I think. In all cases I've ever seen, you need a separate resource manager to actually react to the membership changes. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Antw: Re: reproducible split brain
On 17/03/16 07:30 PM, Christopher Harvey wrote: > On Thu, Mar 17, 2016, at 06:24 PM, Ken Gaillot wrote: >> On 03/17/2016 05:10 PM, Christopher Harvey wrote: >>> If I ignore pacemaker's existence, and just run corosync, corosync >>> disagrees about node membership in the situation presented in the first >>> email. While it's true that stonith just happens to quickly correct the >>> situation after it occurs it still smells like a bug in the case where >>> corosync in used in isolation. Corosync is after all a membership and >>> total ordering protocol, and the nodes in the cluster are unable to >>> agree on membership. >>> >>> The Totem protocol specifies a ring_id in the token passed in a ring. >>> Since all of the 3 nodes but one have formed a new ring with a new id >>> how is it that the single node can survive in a ring with no other >>> members passing a token with the old ring_id? >>> >>> Are there network failure situations that can fool the Totem membership >>> protocol or is this an implementation problem? I don't see how it could >>> not be one or the other, and it's bad either way. >> >> Neither, really. In a split brain situation, there simply is not enough >> information for any protocol or implementation to reliably decide what >> to do. That's what fencing is meant to solve -- it provides the >> information that certain nodes are definitely not active. >> >> There's no way for either side of the split to know whether the opposite >> side is down, or merely unable to communicate properly. If the latter, >> it's possible that they are still accessing shared resources, which >> without proper communication, can lead to serious problems (e.g. data >> corruption of a shared volume). > > The totem protocol is silent on the topic of fencing and resources, much > the way TCP is. > > Please explain to me what needs to be fenced in a cluster without > resources where membership and total message ordering are the only > concern. If fencing were a requirement for membership and ordering, > wouldn't stonith be part of corosync and not pacemaker? Corosync is a membership and communication layer (and in v2+, a quorum provider). It doesn't care about or manage anything higher up. So it doesn't care about fencing itself. It simply cares about; * Who is in the cluster? * How do the members communicate? * (v2+) Is there enough members for quorum? * Notify resource managers of membership changes (join or loss). The resource manager, pacemaker or rgmanager, care about resources, so it is what cares about making smart decisions. As Ken pointed out, without fencing, it can never tell the difference between no access and dead peer. This is (again) why fencing is critical. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] DRBD fencing issue on failover causes resource failure
provider=linbit type=drbd) >Attributes: drbd_resource=drbd0 >Operations: start interval=0s timeout=240 (drbd_dev-start-interval-0s) >promote interval=0s timeout=90 (drbd_dev-promote-interval-0s) >demote interval=0s timeout=90 (drbd_dev-demote-interval-0s) >stop interval=0s timeout=100 (drbd_dev-stop-interval-0s) >monitor interval=29s role=Master > (drbd_dev-monitor-interval-29s) >monitor interval=31s role=Slave > (drbd_dev-monitor-interval-31s) > Resource: drbd_fs (class=ocf provider=heartbeat type=Filesystem) > Attributes: device=/dev/drbd0 directory=/exports/drbd0 fstype=xfs > Operations: start interval=0s timeout=60 (drbd_fs-start-interval-0s) > stop interval=0s timeout=60 (drbd_fs-stop-interval-0s) > monitor interval=20 timeout=40 (drbd_fs-monitor-interval-20) > > Stonith Devices: > Fencing Levels: > > Location Constraints: > Ordering Constraints: > start nfsVIP then start nfs-server (kind:Mandatory) > (id:order-nfsVIP-nfs-server-mandatory) > start drbd_fs then start nfs-server (kind:Mandatory) > (id:order-drbd_fs-nfs-server-mandatory) > promote drbd_master then start drbd_fs (kind:Mandatory) > (id:order-drbd_master-drbd_fs-mandatory) > Colocation Constraints: > nfs-server with nfsVIP (score:INFINITY) > (id:colocation-nfs-server-nfsVIP-INFINITY) > nfs-server with drbd_fs (score:INFINITY) > (id:colocation-nfs-server-drbd_fs-INFINITY) > drbd_fs with drbd_master (score:INFINITY) (with-rsc-role:Master) > (id:colocation-drbd_fs-drbd_master-INFINITY) > > Resources Defaults: > resource-stickiness: 100 > failure-timeout: 60 > Operations Defaults: > No defaults set > > Cluster Properties: > cluster-infrastructure: corosync > cluster-name: nfscluster > dc-version: 1.1.13-10.el7_2.2-44eb2dd > have-watchdog: false > maintenance-mode: false > stonith-enabled: false Configure *and test* stonith in pacemaker first, then DRBD will hook into it and use it properly. DRBD simply asks pacemaker to do the fence, but you currently don't have it setup. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] [Announce] libqb 1.0 release
On 01/04/16 08:59 AM, Christine Caulfield wrote: > I am very pleased to announce the 1.0 release of libqb > > This release is identical to 1.0rc4 but with the doxygen generation fixed. > > Huge thanks you to all of the people who have contributed to this release. > > Chrissie > > The current release tarball is here: > https://github.com/ClusterLabs/libqb/releases/download/v1.0/libqb-1.0.tar.gz > > The github repository is here: > https://github.com/ClusterLabs/libqb > > Please report bugs and issues in bugzilla: > https://bugzilla.redhat.com Congratulations! An auspicious day to release, if any. ;) -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] DLM hanging when corosync is OK causes cluster to hang
On 19/01/16 08:04 PM, Jan Pokorný wrote: > On 11/01/16 11:59 -0500, Digimer wrote: >> We hit a strange problem where a RAID controller on a node failed, >> causing DLM (gfs2/clvmd) to hang, but the node was never fenced. I >> assume this was because corosync was still working. >> >> Is there a way in rhel6/cman/rgmanager to have a node suicide or get >> fenced in a condition like this? > > something like this in the crontab (beside cron and other components > are now the SPOF and I/O spike or DoS will finish the apocalypse)? > > */1 * * * * timeout 30s touch || fence_node > > Sophistications at the components you mentioned might be preferred, > though. Very very late reply, but I finally implemented this. :D https://github.com/ClusterLabs/striker/commit/8c11cf1edd9278c4fe5256096748fb62c330a948 The way I've done it is that in ScanCore, certain post-scan actions happen depending on whether the machine is a node in the cluster or a dashboard. If the user enables the feature, at the end of a dashboard scan, ScanCore checks for access to each node (echo 1) and if so, calls 'timeout X ls /shared || echo timeout' (being the gfs2 mount point when a node is in the cluster). If 'X' seconds elapse and it returns 'timeout', the dashboard reboots the node using the node's primary fence method (which we cache on the dashboards). Initial testing is working great! Thank you again for the 'timeout' pointer. I never knew it existed and I can imagine this helping me elsewhere, too. :D digi -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Resource failure-timeout does not reset when resource fails to connect to both nodes
On 28/03/16 12:44 PM, Sam Gardner wrote: > I have a simple resource defined: > > [root@ha-d1 ~]# pcs resource show dmz1 > Resource: dmz1 (class=ocf provider=internal type=ip-address) > Attributes: address=172.16.10.192 monitor_link=true > Meta Attrs: migration-threshold=3 failure-timeout=30s > Operations: monitor interval=7s (dmz1-monitor-interval-7s) > > This is a custom resource which provides an ethernet alias to one of the > interfaces on our system. > > I can unplug the cable on either node and failover occurs as expected, > and 30s after re-plugging it I can repeat the exercise on the opposite > node and failover will happen as expected. > > However, if I unplug the cable from both nodes, the failcount goes up, > and the 30s failure-timeout does not reset the failcounts, meaning that > pacemaker never tries to start the failed resource again. > > Full list of resources: > > Resource Group: network > inif (off::internal:ip.sh): Started ha-d1.dev.com > outif (off::internal:ip.sh): Started ha-d2.dev.com > dmz1 (off::internal:ip.sh): Stopped > Master/Slave Set: DRBDMaster [DRBDSlave] > Masters: [ ha-d1.dev.com ] > Slaves: [ ha-d2.dev.com ] > Resource Group: filesystem > DRBDFS (ocf::heartbeat:Filesystem):Stopped > Resource Group: application > service_failover (off::internal:service_failover):Stopped > > Failcounts for dmz1 > ha-d1.dev.com: 4 > ha-d2.dev.com: 4 > > Is there any way to automatically recover from this scenario, other than > setting an obnoxiously high migration-threshold? > > -- > > *Sam Gardner * > > Software Engineer > > *Trustwave** *| SMART SECURITY ON DEMAND Stonith? -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] HA meetup at OpenStack Summit
On 13/04/16 10:16 AM, Ken Gaillot wrote: > On 04/12/2016 06:39 PM, Digimer wrote: >> On 12/04/16 07:09 PM, Ken Gaillot wrote: >>> Hi everybody, >>> >>> The upcoming OpenStack Summit is April 25-29 in Austin, Texas (US). Some >>> regular ClusterLabs contributors are going, so I was wondering if anyone >>> would like to do an informal meetup sometime during the summit. >>> >>> It looks like the best time would be that Wednesday, either lunch (at >>> the venue) or dinner (offsite). It might also be possible to reserve a >>> small (10-person) meeting room, or just meet informally in the expo hall. >>> >>> Anyone interested? Preferences/conflicts? >> >> Informal meet-up, or to try and get work done? > > Informal, though of course HA will be the likely topic of conversation :) OK, would be expensive to come down for $drinks, but I think we're still working on something semi-official for late summer/early spring. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Simple Clarification's regarding pacemaker
On 26/04/16 08:15 AM, RaSca wrote: > Il 26/04/2016 13:39, K Aravind ha scritto: >> Hi all >> Have some doubts regarding pacemaker >> It would be of great help if you could help me out >> >> 1.Pacemaker handles split brains using quorum and stonith only right .? >> It is not a case where a split brain is resolved wothout the use of >> stonith and qourm am I right ? > > Not by default. But if you set no-quorum-policy according to what you > want to achieve you may obtain something similar to a split brain > resolution. Note that quorum has nothing to do with split-brain protection, only stonith does. Quorum is used to prevent a sole node (that is operating properly, say after a reboot) from trying to run cluster services when it is inquorate. Said another way; Stonith/Fencing is a tool to protect your cluster when things go wrong. Quorum is a tool to help your nodes make decisions when things are working properly. >> 2.Resource stickiness /location cannot be used to solve split brain right ? > > Again, it depends. For example if you set a location based upon the > network connectivity, then you may obtain that a disconnected node will > not run any resource. > That said if you want to be 100% sure you must use stonith. > >> 3.let's say I have qourum enabled but not stonith for a two node cluster >> ..can it be used for solving split brain ..if so can the user >> application can add rules for quorum to elect appropriate / preffered >> the master ..similar to resource stickiness ? > > Again, see no-quorum-policy. To again clarify; No. Quorum does not help prevent split-brains and quorum can not be used on 2-node clusters at any rate. Quorum only works on 3+ node clusters because quorum is calculated as: 50% of total votes, plus 1, rounded down. 2 Node == 2 / 1 = 1 + 1 = 2 rounded down to 2. 3 Nodes == 3 / 2 = 1.5 + 1 = 2.5 rounded down to 2. 4 Nodes == 4 / 2 = 2 + 1 = 3 rounded down to 3. 5 Nodes == 5 / 2 = 2.5 + 1 = 3.5 rounded down to 3. So, quorum in a 2-node cluster is '2', meaning the cluster will shut down if one node fails. Not very HA :). 3 and 4 node clusters can each lose 1 node, 5 nodes can lose 2 nodes, and so on. >> 4.can I configure the default role for a node ? >> Meaning ...let's say I have a two node master slave setup >> ,master node goes down and the slave is elected as a master so far so >> good now let's say the older master which was down comes up. So is there >> a way I can configure it be slave by default meaning whenever a new node >> comes up it be a slave and not a master > > You can work on default resource stickiness to INFINITY in this case. > >> 5.Can I have node roles other than master or slave in a master slave >> cluster say unknown ? > > No, AFAIK. > >> 6.Is there notification from pacemaker when a split brain happens , >> When an election beings , >> When an election is done >> Thanks >> Aravind > > Just in the logs, but you can setup something like email notification > for each action. I don't know how things are now, but in the past those > were a lot of mail (and when I say a lot I mean a LOT). > > Bye, With stonith enabled, a failed fence will leave the cluster hung, by design. The logic is that, as bad as a hung cluster is, it is better than risking a split-brain (which can lead to data loss / corruption, confused switches, etc). digimer -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Using pacemaker for manual failover only?
On 23/05/16 03:03 PM, Stephano-Shachter, Dylan wrote: > Hello, > > I am using pacemaker 1.1.14 with pcs 0.9.149. I have successfully > configured pacemaker for highly available nfs with drbd. Pacemaker > allows me to easily failover without interrupting nfs connections. I, > however, am only interested in failing over manually (currently I use > "pcs resource move --master"). I would like for > the cluster to do nothing when a node fails unexpectedly. > > Right now the solution I am going with is to run > "pcs property set is-managed-default=no" > until I need to failover, at which point I set is-managed-default=yes, > then failover, then set it back to no. > > While this method works for me, it can be unpredictable if people run > move commands at the wrong time. > > Is there a way to disable automatic failover permanently while still > allowing manual failover (with "pcs resource move" or with something else)? Setting aside the use-case for this... Ditch the HA stack, it's an avoidable complexity. Instead, just write a small shell script that drops the IP, stops nfs, unmounts the FS, demotes DRBD and the does the reverse on the peer (over ssh). -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] install software in centos5
On 11/05/16 02:31 AM, Liu, Mark wrote: > Hello all, > > I want to install the pacemaker & corosync in the Centos5.x > server,but can't find the repo.who can tell me how to do this?Thank you. That is a very ill-advised course of action. If you must use HA on CentOS 5, use cman + rgmanager. If you want to use pacemaker, use CentOS 7 (or at the very least, the lastest 6 with the cman plugin). -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] availibility
On 08/05/16 01:29 PM, Thierry Boibary wrote: > Hi, > > is cluster lab still available and up to date? > > How can i download and install ? > > Thanks > T. Clusterlabs is an umbrella for open source clustering in general. The website is up and running as of a minute ago, so it is available. If I am assuming that you're asking about the Pacemaker project, yes it is also up to date and very actively supported and developed. The projects under the Clusterlabs umbrella are all available here: https://github.com/ClusterLabs -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] dropping ssh connection on failover
On 14/04/16 05:38 PM, Dimitri Maziuk wrote: > Hi all, > > I've an up to date centos 7 w/ pcs + corosync + pacemaker active-passive > cluster running drbd (8.4) and apache. It's all working fine except when > I trigger a fail-over, my ssh connection to active (being demoted) node > gets cut. > > Am I missing something or is this an expected behaviour of the new and > improved ip RA and/or ip command? > > TIA, I don't believe it is possible to maintain TCP connections in a failover. You could create a VM and ssh into that, and live-migrate the VM back and forth but that is someone more complicated. -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
