William But i would like to know if you have a lvm resource in your pacemaker configuration
Remember clvmd it's not for active di vg or lv it's for propagate the lvm meta data on all node of the cluster Il giorno 26 marzo 2012 23:17, William Seligman <selig...@nevis.columbia.edu > ha scritto: > On 3/26/12 4:28 PM, emmanuel segura wrote: > > Sorry Willian i can't post my config now because i'm at home now not in > my > > job > > > > I think it's no a problem if clvm start before drbd, because clvm not > > needed and devices to start > > > > This it's the point, i hope to be clear > > > > The introduction of pacemaker in redhat cluster was thinked for replace > > rgmanager not whole cluster stack > > > > and i suggest you to start clvmd at boot time > > > > chkconfig clvmd on > > I'm afraid this doesn't work. It's as I predicted; when gfs2 starts I get: > > Mounting GFS2 filesystem (/usr/nevis): invalid device path > "/dev/mapper/ADMIN-usr" > [FAILED] > > ... and so on, because the ADMIN volume group was never loaded by clvmd. > Without > a "vgscan" in there somewhere, the system can't see the volume groups on > the > drbd resource. > > > Sorry for my bad english :-) i can from a spanish country and all days i > > speak Italian > > I'm sorry that I don't speak more languages! You're the one who's helping > me; > it's my task to learn and understand. Certainly your English is better > than my > French or Russian. > > > Il giorno 26 marzo 2012 22:04, William Seligman < > selig...@nevis.columbia.edu > >> ha scritto: > > > >> On 3/26/12 3:48 PM, emmanuel segura wrote: > >>> I know it's normal fence_node doesn't work because the request of fence > >>> must be redirect to pacemaker stonith > >>> > >>> I think call the cluster agents with rgmanager it's really ugly thing, > i > >>> never seen a cluster like this > >>> ============================================================== > >>> If I understand "Pacemaker Explained" <http://bit.ly/GR5WEY> and how > I'd > >>> invoke > >>> clvmd from cman <http://bit.ly/H6ZbKg>, the clvmd script that would be > >>> invoked > >>> by either HA resource manager is exactly the same: /etc/init.d/clvmd. > >>> ============================================================== > >>> > >>> clvm doesn't need to be called from rgmanger in the cluster > configuration > >>> > >>> this the boot sequence of redhat daemons > >>> > >>> 1:cman, 2:clvm, 3:rgmanager > >>> > >>> and if you don't wanna use rgmanager you just replace rgmanager > >> > >> I'm sorry, but I don't think I understand what you're suggesting. Do you > >> suggest > >> that I start clvmd at boot? That won't work; clvmd won't see the volume > >> groups > >> on drbd until drbd is started and promoted to primary. > >> > >> May I ask you to post your own cluster.conf on pastebin.com so I can > see > >> how you > >> do it? Along with "crm configure show" if that's relevant for your > cluster? > >> > >>> Il giorno 26 marzo 2012 19:21, William Seligman < > >> selig...@nevis.columbia.edu > >>>> ha scritto: > >>> > >>>> On 3/24/12 5:40 PM, emmanuel segura wrote: > >>>>> I think it's better you use clvmd with cman > >>>>> > >>>>> I don't now why you use the lsb script of clvm > >>>>> > >>>>> On Redhat clvmd need of cman and you try to running with pacemaker, i > >> not > >>>>> sure this is the problem but this type of configuration it's so > strange > >>>>> > >>>>> I made it a virtual cluster with kvm and i not foud a problems > >>>> > >>>> While I appreciate the advice, it's not immediately clear that trying > to > >>>> eliminate pacemaker would do me any good. Perhaps someone can > >> demonstrate > >>>> the > >>>> error in my reasoning: > >>>> > >>>> If I understand "Pacemaker Explained" <http://bit.ly/GR5WEY> and how > >> I'd > >>>> invoke > >>>> clvmd from cman <http://bit.ly/H6ZbKg>, the clvmd script that would > be > >>>> invoked > >>>> by either HA resource manager is exactly the same: /etc/init.d/clvmd. > >>>> > >>>> If I tried to use cman instead of pacemaker, I'd be cutting myself off > >>>> from the > >>>> pacemaker features that cman/rgmanager does not yet have available, > >> such as > >>>> pacemaker's symlink, exportfs, and clonable IPaddr2 resources. > >>>> > >>>> I recognize I've got a strange problem. Given that fence_node doesn't > >> work > >>>> but > >>>> stonith_admin does, I strongly suspect that the problem is caused by > the > >>>> behavior of my fencing agent, not the use of pacemaker versus > rgmanager, > >>>> nor by > >>>> how clvmd is being started. > >>>> > >>>>> Il giorno 24 marzo 2012 13:09, William Seligman < > >>>> selig...@nevis.columbia.edu > >>>>>> ha scritto: > >>>>> > >>>>>> On 3/24/12 4:47 AM, emmanuel segura wrote: > >>>>>>> How do you configure clvmd? > >>>>>>> > >>>>>>> with cman or with pacemaker? > >>>>>> > >>>>>> Pacemaker. Here's the output of 'crm configure show': > >>>>>> <http://pastebin.com/426CdVwN> > >>>>>> > >>>>>>> Il giorno 23 marzo 2012 22:14, William Seligman < > >>>>>> selig...@nevis.columbia.edu > >>>>>>>> ha scritto: > >>>>>>> > >>>>>>>> On 3/23/12 5:03 PM, emmanuel segura wrote: > >>>>>>>> > >>>>>>>>> Sorry but i would to know if can show me your > >>>> /etc/cluster/cluster.conf > >>>>>>>> > >>>>>>>> Here it is: <http://pastebin.com/GUr0CEgZ> > >>>>>>>> > >>>>>>>>> Il giorno 23 marzo 2012 21:50, William Seligman < > >>>>>>>> selig...@nevis.columbia.edu > >>>>>>>>>> ha scritto: > >>>>>>>>> > >>>>>>>>>> On 3/22/12 2:43 PM, William Seligman wrote: > >>>>>>>>>>> On 3/20/12 4:55 PM, Lars Ellenberg wrote: > >>>>>>>>>>>> On Fri, Mar 16, 2012 at 05:06:04PM -0400, William Seligman > >> wrote: > >>>>>>>>>>>>> On 3/16/12 12:12 PM, William Seligman wrote: > >>>>>>>>>>>>>> On 3/16/12 7:02 AM, Andreas Kurz wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> s----- ... DRBD suspended io, most likely because of it's > >>>>>>>>>>>>>>> fencing-policy. For valid dual-primary setups you have to > use > >>>>>>>>>>>>>>> "resource-and-stonith" policy and a working "fence-peer" > >>>> handler. > >>>>>>>> In > >>>>>>>>>>>>>>> this mode I/O is suspended until fencing of peer was > >> succesful. > >>>>>>>>>> Question > >>>>>>>>>>>>>>> is, why the peer does _not_ also suspend its I/O because > >>>>>> obviously > >>>>>>>>>>>>>>> fencing was not successful ..... > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> So with a correct DRBD configuration one of your nodes > should > >>>>>>>> already > >>>>>>>>>>>>>>> have been fenced because of connection loss between nodes > (on > >>>>>> drbd > >>>>>>>>>>>>>>> replication link). > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> You can use e.g. that nice fencing script: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> http://goo.gl/O4N8f > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> This is the output of "drbdadm dump admin": < > >>>>>>>>>> http://pastebin.com/kTxvHCtx> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> So I've got resource-and-stonith. I gather from an earlier > >>>> thread > >>>>>>>> that > >>>>>>>>>>>>>> obliterate-peer.sh is more-or-less equivalent in > functionality > >>>>>> with > >>>>>>>>>>>>>> stonith_admin_fence_peer.sh: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> < > >>>> http://www.gossamer-threads.com/lists/linuxha/users/78504#78504> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> At the moment I'm pursuing the possibility that I'm > returning > >>>> the > >>>>>>>>>> wrong return > >>>>>>>>>>>>>> codes from my fencing agent: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> <http://www.gossamer-threads.com/lists/linuxha/users/78572> > >>>>>>>>>>>>> > >>>>>>>>>>>>> I cleaned up my fencing agent, making sure its return code > >>>> matched > >>>>>>>>>> those > >>>>>>>>>>>>> returned by other agents in /usr/sbin/fence_, and allowing > for > >>>> some > >>>>>>>>>> delay issues > >>>>>>>>>>>>> in reading the UPS status. But... > >>>>>>>>>>>>> > >>>>>>>>>>>>>> After that, I'll look at another suggestion with lvm.conf: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> < > >>>> http://www.gossamer-threads.com/lists/linuxha/users/78796#78796> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Then I'll try DRBD 8.4.1. Hopefully one of these is the > source > >>>> of > >>>>>>>> the > >>>>>>>>>> issue. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Failure on all three counts. > >>>>>>>>>>>> > >>>>>>>>>>>> May I suggest you double check the permissions on your fence > >> peer > >>>>>>>>>> script? > >>>>>>>>>>>> I suspect you may simply have forgotten the "chmod +x" . > >>>>>>>>>>>> > >>>>>>>>>>>> Test with "drbdadm fence-peer minor-0" from the command line. > >>>>>>>>>>> > >>>>>>>>>>> I still haven't solved the problem, but this advice has gotten > me > >>>>>>>>>> further than > >>>>>>>>>>> before. > >>>>>>>>>>> > >>>>>>>>>>> First, Lars was correct: I did not have execute permissions set > >> on > >>>> my > >>>>>>>>>> fence peer > >>>>>>>>>>> scripts. (D'oh!) I turned them on, but that did not change > >>>> anything: > >>>>>>>>>> cman+clvmd > >>>>>>>>>>> still hung on the vgdisplay command if I crashed the peer node. > >>>>>>>>>>> > >>>>>>>>>>> I started up both nodes again (cman+pacemaker+drbd+clvmd) and > >> tried > >>>>>>>> Lars' > >>>>>>>>>>> suggested command. I didn't save the response for this message > >>>> (d'oh > >>>>>>>>>> again!) but > >>>>>>>>>>> it said that the fence-peer script had failed. > >>>>>>>>>>> > >>>>>>>>>>> Hmm. The peer was definitely shutting down, so my fencing > script > >> is > >>>>>>>>>> working. I > >>>>>>>>>>> went over it, comparing the return codes to those of the > existing > >>>>>>>>>> scripts, and > >>>>>>>>>>> made some changes. Here's my current script: < > >>>>>>>>>> http://pastebin.com/nUnYVcBK>. > >>>>>>>>>>> > >>>>>>>>>>> Up until now my fence-peer scripts had either been Lon > >> Hohberger's > >>>>>>>>>>> obliterate-peer.sh or Digimer's rhcs_fence. I decided to try > >>>>>>>>>>> stonith_admin-fence-peer.sh that Andreas Kurz recommended; > unlike > >>>> the > >>>>>>>>>> first two > >>>>>>>>>>> scripts, which fence using fence_node, the latter script just > >> calls > >>>>>>>>>> stonith_admin. > >>>>>>>>>>> > >>>>>>>>>>> When I tried the stonith_admin-fence-peer.sh script, it worked: > >>>>>>>>>>> > >>>>>>>>>>> # drbdadm fence-peer minor-0 > >>>>>>>>>>> stonith_admin-fence-peer.sh[10886]: stonith_admin successfully > >>>> fenced > >>>>>>>>>> peer > >>>>>>>>>>> orestes-corosync.nevis.columbia.edu. > >>>>>>>>>>> > >>>>>>>>>>> Power was cut on the peer, the remaining node stayed up. Then I > >>>>>> brought > >>>>>>>>>> up the > >>>>>>>>>>> peer with: > >>>>>>>>>>> > >>>>>>>>>>> stonith_admin -U orestes-corosync.nevis.columbia.edu > >>>>>>>>>>> > >>>>>>>>>>> BUT: When the restored peer came up and started to run cman, > the > >>>>>> clvmd > >>>>>>>>>> hung on > >>>>>>>>>>> the main node again. > >>>>>>>>>>> > >>>>>>>>>>> After cycling through some more tests, I found that if I > brought > >>>> down > >>>>>>>>>> the peer > >>>>>>>>>>> with drbdadm, then brought up with the peer with no HA > services, > >>>> then > >>>>>>>>>> started > >>>>>>>>>>> drbd and then cman, the cluster remained intact. > >>>>>>>>>>> > >>>>>>>>>>> If I crashed the peer, the scheme in the previous paragraph > >> didn't > >>>>>>>> work. > >>>>>>>>>> I bring > >>>>>>>>>>> up drbd, check that the disks are both UpToDate, then bring up > >>>> cman. > >>>>>> At > >>>>>>>>>> that > >>>>>>>>>>> point the vgdisplay on the main node takes so long to run that > >>>> clvmd > >>>>>>>>>> will time out: > >>>>>>>>>>> > >>>>>>>>>>> vgdisplay Error locking on node > >>>> orestes-corosync.nevis.columbia.edu > >>>>>> : > >>>>>>>>>> Command > >>>>>>>>>>> timed out > >>>>>>>>>>> > >>>>>>>>>>> I timed how long it took vgdisplay to run. I might be able to > >> work > >>>>>>>>>> around this > >>>>>>>>>>> by setting the timeout on my clvmd resource to 300s, but that > >> seems > >>>>>> to > >>>>>>>>>> be a > >>>>>>>>>>> band-aid for an underlying problem. Any suggestions on what > else > >> I > >>>> could > >>>>>>>>>> check? > >>>>>>>>>> > >>>>>>>>>> I've done some more tests. Still no solution, just an > observation: > >>>>>>>>>> The "death mode" appears to be: > >>>>>>>>>> > >>>>>>>>>> - Two nodes running cman+pacemaker+drbd+clvmd > >>>>>>>>>> - Take one node down = one remaining node > >>>> w/cman+pacemaker+drbd+clvmd > >>>>>>>>>> - Start up dead node. If it ever gets into a state in which > >>>>>>>>>> it's running cman but not clvmd, clvmd on the uncrashed node > >> hangs. > >>>>>>>>>> - Conversely, if I bring up drbd, make it primary, start > >>>>>>>>>> cman+clvmd, there's no problem on the uncrashed node. > >>>>>>>>>> > >>>>>>>>>> My guess is that clvmd is getting the number of nodes it expects > >>>>>>>>>> from cman. When the formally-dead node starts running cman, the > >>>>>>>>>> number of cluster nodes goes to 2 (I checked with 'cman_tool > >>>>>>>>>> status') but the number of nodes running clvmd is still 1, hence > >>>>>>>>>> the crash. > >>>>>>>>>> > >>>>>>>>>> Does this guess make sense? > > > -- > Bill Seligman | Phone: (914) 591-2823 > Nevis Labs, Columbia Univ | mailto://selig...@nevis.columbia.edu > PO Box 137 | > Irvington NY 10533 USA | http://www.nevis.columbia.edu/~seligman/ > > > _______________________________________________ > Linux-HA mailing list > Linux-HA@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems > -- esta es mi vida e me la vivo hasta que dios quiera _______________________________________________ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
