Re: [Linux-HA] DRBD and pacemaker interaction
On Fri, Apr 1, 2011 at 10:56 AM, Lars Ellenberg wrote: > But possibly you should rather have some monitoring (nagios, ...) notice this, > page/email/alert with your favorite method the relevant people, and have > them take appropriate actions? > Ok, lets see how this might work. You would need a separate monitor for the cluster and since this monitor also can potentially crash, you would need another monitor to observer the first one, then we would want the first one to monitor second one, so we would need a cluster of monitors. Wait, don't we have already cluster in place? It seems logical to have monitor to be part of the cluster. I was expecting "monitor" operation to handle that, but it seems for DRBD this is not the case. Maybe we should have another primitive running? drbd_status or something? When drbd subsystem is in degraded state, have drbd_status in "stopped" state? Vadym ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] DRBD and pacemaker interaction
On Apr 2, 2011, at 12:30 AM, Vadym Chepkov wrote: >> >> Of course you can patch pacemaker to detect that the RAID1 is degraded, >> and trigger faxing a PO to your supplier for a replacement drive. >> > > Pacemaker doesn't have RAID1 RA, does it? > > It's was a jest, of cause, but still, I guess it's a fundamental concept of "Master"/"Slave" relationship. If "slave" doesn't do it's job, is it really "running". Lets transfer the same situation to the mysql agent. If mysql replica doesn't replicate anything, is it "running" ? mysqld process will be running, but it's useless. It's like an answer to "Do you know what time it is?" question. "I do" would be an insult, wouldn't it? :) Vadym ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] DRBD and pacemaker interaction
On Apr 1, 2011, at 10:56 AM, Lars Ellenberg wrote: > On Fri, Apr 01, 2011 at 08:42:04AM -0600, Serge Dubrouski wrote: >> On Fri, Apr 1, 2011 at 8:38 AM, Lars Ellenberg >> wrote: >>> On Fri, Apr 01, 2011 at 11:35:19AM +0200, Christoph Bartoschek wrote: Am 01.04.2011 11:27, schrieb Florian Haas: > On 2011-04-01 10:49, Christoph Bartoschek wrote: >> Am 01.04.2011 10:27, schrieb Andrew Beekhof: >>> On Sat, Mar 26, 2011 at 12:10 AM, Lars Ellenberg >>>wrote: On Fri, Mar 25, 2011 at 06:18:07PM +0100, Christoph Bartoschek wrote: > I am missing the state: running degraded or suboptimal. Yep, "degraded" is not a state available for pacemaker. Pacemaker cannot do much about "suboptimal". >>> >>> I wonder what it would take to change that. I suspect either a >>> crystal ball or way too much knowledge of drbd internals. >> >> The RA would be responsible to check this. For drbd any diskstate >> different from UpToDate/UpToDate is suboptimal. > > Have you actually looked at the resource agent? It does already evaluate > the disk state and adjusts the master preference accordingly. What else > is there to do? Maybe I misunderstood Andrew's comment. I read it this way: If we introduce a new state "suboptimal", would it be hard to detect it? I just wanted to express that detecting suboptimality seems not to be that hard. >>> >>> But that state is useless for pacemaker, >>> since it cannot do anything about it. >> >> Looks like a lot of people, including myself, are still confused with >> this statement. Basically this state of DRBD resource is unstable and >> resource is unusable, why do you think that this is normal for >> Pacemaker to report a such state as Ok state? > > It is usable. It is being used. > It is at least as usable as a degraded RAID1. > Pacemaker cannot do anything about that missing disk, either. > > Of course you can patch pacemaker to detect that the RAID1 is degraded, > and trigger faxing a PO to your supplier for a replacement drive. > Pacemaker doesn't have RAID1 RA, does it? > But possibly you should rather have some monitoring (nagios, ...) notice this, > page/email/alert with your favorite method the relevant people, and have > them take appropriate actions? > > -- > : Lars Ellenberg > : LINBIT | Your Way to High Availability > : DRBD/HA support and consulting http://www.linbit.com > ___ > Linux-HA mailing list > Linux-HA@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] DRBD problems are not reported
On Fri, Apr 1, 2011 at 8:50 AM, Lars Ellenberg
wrote:
> On Fri, Apr 01, 2011 at 09:37:09AM -0400, Vadym Chepkov wrote:
>>
>> On Apr 1, 2011, at 3:22 AM, Tim Serong wrote:
>>
>> > On 4/1/2011 at 11:37 AM, Vadym Chepkov wrote:
>> >> On Mar 31, 2011, at 2:30 PM, Christoph Bartoschek wrote:
>> >>
>> >>> Am 29.03.2011 15:31, schrieb Dejan Muhamedagic:
>> On Tue, Mar 29, 2011 at 08:13:49AM +0200, Christoph Bartoschek wrote:
>> > Am 29.03.2011 02:35, schrieb Vadym Chepkov:
>> >>
>> >> On Mar 28, 2011, at 10:55 AM, Christoph Bartoschek wrote:
>> >>
>> >>> Am 28.03.2011 16:30, schrieb Dejan Muhamedagic:
>> Hi,
>>
>> On Mon, Mar 21, 2011 at 11:33:49PM +0100, Christoph Bartoschek
>> wrote:
>> > Hi,
>> >
>> > I am testing a NFS failover setup. During the tests I created a
>> > split-brain situation and now node A thinks it is primary and
>> > uptodate
>> > while node B thinks that it is Outdated.
>> >
>> > crm_mon however does not indicate any error to me. Why is this the
>> > case?
>> > I expect to see anything that shows me the degraded status. How
>> > can this
>> > be fixed?
>>
>> The cluster relies on the RA (in this case drbd) to report any
>> problems. Do you have a monitor operation defined for that
>> resource?
>> >>>
>> >>> I have the resource defined as:
>> >>>
>> >>> primitive p_drbd ocf:linbit:drbd \
>> >>> params drbd_resource="home-data"
>> >>> op monitor interval="15" role="Master" \
>> >>> op monitor interval="30" role="Slave"
>> >>>
>> >>> Is this a correct monitor operation?
>>
>> Yes, though you should also add timeout specs.
>>
>> >> Just out of curiosity, you do have ms resource defined?
>> >>
>> >> ms ms_p_drbd p_drbd \
>> >> meta master-max="1" master-node-max="1" clone-max="2"
>> >> clone-node-max="1"
>> >> notify="true"
>> >>
>> >> Because if you do and cluster is not aware of the split-brain, drbd
>> >> RA has a
>> >> serious flaw.
>> >>
>> >
>> > I'm sorry. Yes, the ms resource is also defined.
>>
>> Well, I'm really confused. You basically say that the drbd disk
>> gets into a degraded mode (i.e. it detects split brain), but the
>> cluster (pacemaker) never learns about that. Perhaps you should
>> open a bugzilla for this and supply hb_report. Though it's
>> really hard to believe. It's like basic functionality failing.
>> >>>
>> >>>
>> >>> What would you expect to see?
>> >>>
>> >>> Currently I see the following in crm_mon:
>> >>>
>> >>> Master/Slave Set: ms_drbd_nfs [p_drbd_nfs]
>> >>> Masters: [ ries ]
>> >>> Slaves: [ laplace ]
>> >>>
>> >>>
>> >>> At the same time "cat /proc/drbd" on ries says:
>> >>>
>> >>> ries:~ # cat /proc/drbd
>> >>> version: 8.3.9 (api:88/proto:86-95)
>> >>> srcversion: A67EB2D25C5AFBFF3D8B788
>> >>> 0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r-
>> >>> ns:0 nr:0 dw:4 dr:1761 al:1 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:4
>> >>>
>> >>>
>> >>> And on node laplace it says:
>> >>>
>> >>> laplace:~ # cat /proc/drbd
>> >>> version: 8.3.9 (api:88/proto:86-95)
>> >>> srcversion: A67EB2D25C5AFBFF3D8B788
>> >>> 0: cs:StandAlone ro:Secondary/Unknown ds:Outdated/DUnknown r-
>> >>> ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:4
>> >>>
>> >>>
>> >>
>> >>
>> >>
>> >> yes, and according to the RA script everything is perfect:
>> >>
>> >> drbd_status() {
>> >> local rc
>> >> rc=$OCF_NOT_RUNNING
>> >>
>> >> if ! is_drbd_enabled || ! [ -b "$DRBD_DEVICE" ]; then
>> >> return $rc
>> >> fi
>> >>
>> >> # ok, module is loaded, block device node exists.
>> >> # lets see its status
>> >> drbd_set_status_variables
>> >> case "${DRBD_ROLE_LOCAL}" in
>> >> Primary)
>> >> rc=$OCF_RUNNING_MASTER
>> >> ;;
>> >> Secondary)
>> >> rc=$OCF_SUCCESS
>> >> ;;
>> >> Unconfigured)
>> >> rc=$OCF_NOT_RUNNING
>> >> ;;
>> >> *)
>> >> ocf_log err "Unexpected role ${DRBD_ROLE_LOCAL}"
>> >> rc=$OCF_ERR_GENERIC
>> >> esac
>> >>
>> >> return $rc
>> >> }
>> >>
>> >> Staggering.
>> >>
>> >> drbd_set_status_variable subroutine does set DRBD_CSTATE
>> >>
>> >> I think the RA needs to be modified to something like this:
>> >>
>> >> Secondary)
>> >> if [[ $DRBD_CSTATE == Connected ]]; then
>> >> rc=$OCF_SUCCESS
>> >> else
>> >> rc=$OCF_NOT_RUNNING
>> >> fi
>> >
>> > That wouldn't strictly be correct - DRBD *is* currently running on
>> > both nodes, Primary (master) on one and Secondary (slave) on the
>> > other. This state is correctly reported in crm_mon. The thin
Re: [Linux-HA] DRBD and pacemaker interaction
Am 01.04.2011 16:38, schrieb Lars Ellenberg: > On Fri, Apr 01, 2011 at 11:35:19AM +0200, Christoph Bartoschek wrote: >> Am 01.04.2011 11:27, schrieb Florian Haas: >>> On 2011-04-01 10:49, Christoph Bartoschek wrote: Am 01.04.2011 10:27, schrieb Andrew Beekhof: > On Sat, Mar 26, 2011 at 12:10 AM, Lars Ellenberg > wrote: >> On Fri, Mar 25, 2011 at 06:18:07PM +0100, Christoph Bartoschek wrote: >>> I am missing the state: running degraded or suboptimal. >> >> Yep, "degraded" is not a state available for pacemaker. >> Pacemaker cannot do much about "suboptimal". > > I wonder what it would take to change that. I suspect either a > crystal ball or way too much knowledge of drbd internals. The RA would be responsible to check this. For drbd any diskstate different from UpToDate/UpToDate is suboptimal. >>> >>> Have you actually looked at the resource agent? It does already evaluate >>> the disk state and adjusts the master preference accordingly. What else >>> is there to do? >> >> Maybe I misunderstood Andrew's comment. I read it this way: If we >> introduce a new state "suboptimal", would it be hard to detect it? >> >> I just wanted to express that detecting suboptimality seems not to be >> that hard. > > But that state is useless for pacemaker, > since it cannot do anything about it. > > I thought I made that clear. > You made clear that pacemaker cannot do anything about it. However crm_mon could report it. One may think that is can be neglected. But the current output of crm_mon is unexpected for me. I have to check whether ocf:pacemaker:ClusterMon reports the score changes done by the DRBD RA in such a situation. This would help. Christoph ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] DRBD and pacemaker interaction
Though the previous discussion was beneficial to all the readers, my conclusion is that if there were a document or preferably a picture (pictures worth a 1000 words) of the boundaries of each of the components (All of them, not just some. In this case DRDB and pacemaker. But in other threads it has been the interaction of 2 or 3 of the components.) then it could be understood where to look; i.e., what component is responsible. It appears that many understand some of the components but no one appears to understand all of them and how they interplay. If they did, then the boundaries document would be readily available and this confusion would not be happening. The explanation by Lars makes perfect sense. It just needs to be documented with all the rest of the components so that everyone can better understand the interplay between all of the components making up a Linux-HA environment It also appears that most of us wish to one-stop shop (look at one component and get all the information). If we had descriptions of the roles & responsibilities of each component (for example, pacemaker) then we might not be expecting it to be solving something outside of its role. As an example to try to explain my intent, most of us would not take our car to a oncologist to get the oil changed. If the intent is to be efficient, we should not be expecting the components to be doing more that it role/responsibility else very quickly, we will be generating something that cannot be upgraded because it is no longer modular. Thanks, Clyde ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] DRBD and pacemaker interaction
Serge Dubrouski wrote: > The primary point so far is that current situation is unacceptable for > most of customers. I'd just like to point out that current situation is unacceptable to the *3 users* *who complained so far*. If you don't get it, google for "faulty generalization". And look up representative sample while you're at it. Dima -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] update question
Helmut Wollmersdorfer wrote: > Am 01.04.2011 um 09:44 schrieb Andrew Beekhof: > > >> On Mon, Mar 28, 2011 at 9:08 PM, Miles Fidelman >> wrote: >> >>> Hi Folks, >>> >>> I'm getting ready to upgrade a 2-node HA cluster from Debian Etch to >>> Squeeze. I'd very much appreciate any suggestions regarding >>> "gotchas" to avoid, and so forth. >>> > Etch -> Lenny -> Squeeze > > The Lenny -> Squeeze upgrade works fine, if you have a clean Lenny- > installation. I.e. you first must upgrade to Lenny. > Ooops, my mistake here. I'm already running Lenny (old stable). Not enough coffee when I wrote this, I guess - got the names confused. >>> Basic current configuration: >>> - 2 vanilla Intel-based servers, 4 SATA drives each >>> > If you have Intel e100 NICs, then you will experience problems with > the non-free firmware drivers. The e1000 works fine. > Actually seems to be one of these: NetXtreme BCM5721 Gigabit Ethernet PCI Express > >>> - each machine: disks set up as 4-drive md10, LVM >>> - xen 3.2 hypervisor, Debian Etch Dom0 >>> - DRBD 8.2, Pacemaker linking the two nodes >>> - several Debian Etch DomUs >>> > Same problem here (many Dom0 and DomU with different Debian/Ubuntu > versions - Etch, Lenny, Hardy). > I do not want to risk the problems Etch > Squeeze, even not on the > development (used for programming and test) machines. > > Lenny to Squeeze was nearly without problems (Dom0 and DomU). > Again, my mistake. I'll be going directly from Lenny to Squeeze. > DRBD 8.2 (since 8.3.2 api:88/proto:86-88) should work with the other > node having DRBD 8.3.7 (api:88/proto:86-91) -- theoretically. > This is the one I'm most concerned with. There's going to be a point at which I'm running production VMs on the older code-base, and have to sync and migrate with the upgraded node - so I can migrate the VMs and then upgrade the 2nd node. That's the point at which I'm going to have to have to sync volumes between DRBD 8.2 and 8.3, and then migrate the volumes between Xen 3.2 and 4.1. > BTW: If your hardware is as old as the software, then think about new > hardware. > > Helmut Wollmersdorfer > > ___ > Linux-HA mailing list > Linux-HA@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems > -- In theory, there is no difference between theory and practice. In practice, there is. Yogi Berra ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] DRBD and pacemaker interaction
On Fri, Apr 1, 2011 at 8:56 AM, Lars Ellenberg wrote: > On Fri, Apr 01, 2011 at 08:42:04AM -0600, Serge Dubrouski wrote: >> On Fri, Apr 1, 2011 at 8:38 AM, Lars Ellenberg >> wrote: >> > On Fri, Apr 01, 2011 at 11:35:19AM +0200, Christoph Bartoschek wrote: >> >> Am 01.04.2011 11:27, schrieb Florian Haas: >> >> > On 2011-04-01 10:49, Christoph Bartoschek wrote: >> >> >> Am 01.04.2011 10:27, schrieb Andrew Beekhof: >> >> >>> On Sat, Mar 26, 2011 at 12:10 AM, Lars Ellenberg >> >> >>> wrote: >> >> On Fri, Mar 25, 2011 at 06:18:07PM +0100, Christoph Bartoschek wrote: >> >> > I am missing the state: running degraded or suboptimal. >> >> >> >> Yep, "degraded" is not a state available for pacemaker. >> >> Pacemaker cannot do much about "suboptimal". >> >> >>> >> >> >>> I wonder what it would take to change that. I suspect either a >> >> >>> crystal ball or way too much knowledge of drbd internals. >> >> >> >> >> >> The RA would be responsible to check this. For drbd any diskstate >> >> >> different from UpToDate/UpToDate is suboptimal. >> >> > >> >> > Have you actually looked at the resource agent? It does already evaluate >> >> > the disk state and adjusts the master preference accordingly. What else >> >> > is there to do? >> >> >> >> Maybe I misunderstood Andrew's comment. I read it this way: If we >> >> introduce a new state "suboptimal", would it be hard to detect it? >> >> >> >> I just wanted to express that detecting suboptimality seems not to be >> >> that hard. >> > >> > But that state is useless for pacemaker, >> > since it cannot do anything about it. >> >> Looks like a lot of people, including myself, are still confused with >> this statement. Basically this state of DRBD resource is unstable and >> resource is unusable, why do you think that this is normal for >> Pacemaker to report a such state as Ok state? > > It is usable. It is being used. > It is at least as usable as a degraded RAID1. > Pacemaker cannot do anything about that missing disk, either. I see your point. Would it be possible to provide some options through RA? I mean add a parameter that would control a degraded state. Something like OCF_RESKEY_on_degraded with possible values FAIL or CONTINUE. > > Of course you can patch pacemaker to detect that the RAID1 is degraded, > and trigger faxing a PO to your supplier for a replacement drive. > > But possibly you should rather have some monitoring (nagios, ...) notice this, > page/email/alert with your favorite method the relevant people, and have > them take appropriate actions? > > -- > : Lars Ellenberg > : LINBIT | Your Way to High Availability > : DRBD/HA support and consulting http://www.linbit.com > ___ > Linux-HA mailing list > Linux-HA@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems > -- Serge Dubrouski. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] DRBD and pacemaker interaction
On Fri, Apr 1, 2011 at 8:50 AM, Andrew Beekhof wrote: > On Fri, Apr 1, 2011 at 4:46 PM, Serge Dubrouski wrote: >> On Fri, Apr 1, 2011 at 8:43 AM, Andrew Beekhof wrote: >>> On Fri, Apr 1, 2011 at 4:38 PM, Lars Ellenberg >>> wrote: On Fri, Apr 01, 2011 at 11:35:19AM +0200, Christoph Bartoschek wrote: > Am 01.04.2011 11:27, schrieb Florian Haas: > > On 2011-04-01 10:49, Christoph Bartoschek wrote: > >> Am 01.04.2011 10:27, schrieb Andrew Beekhof: > >>> On Sat, Mar 26, 2011 at 12:10 AM, Lars Ellenberg > >>> wrote: > On Fri, Mar 25, 2011 at 06:18:07PM +0100, Christoph Bartoschek wrote: > > I am missing the state: running degraded or suboptimal. > > Yep, "degraded" is not a state available for pacemaker. > Pacemaker cannot do much about "suboptimal". > >>> > >>> I wonder what it would take to change that. I suspect either a > >>> crystal ball or way too much knowledge of drbd internals. > >> > >> The RA would be responsible to check this. For drbd any diskstate > >> different from UpToDate/UpToDate is suboptimal. > > > > Have you actually looked at the resource agent? It does already evaluate > > the disk state and adjusts the master preference accordingly. What else > > is there to do? > > Maybe I misunderstood Andrew's comment. I read it this way: If we > introduce a new state "suboptimal", would it be hard to detect it? >>> >>> No, detecting is the easy part. >>> > I just wanted to express that detecting suboptimality seems not to be > that hard. But that state is useless for pacemaker, since it cannot do anything about it. >>> >>> This was the part I was wondering about - if pacemaker _could_ do >>> something intelligent. >> >> Isn't a simple reporting that resource is broken rather than healthy >> intelligent enough? > > But is it completely broken? Is it broken in a way that pacemaker can > do something to repair it? The primary point so far is that current situation is unacceptable for most of customers. Even if it's half broken or broken to 1/3 it shouldn't be reported as healthy resource. And as far as I understand Pacemaker cannot do anything more than that. Fixing this situation, if possible, would be responsibility of DRBD RA. > ___ > Linux-HA mailing list > Linux-HA@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems > -- Serge Dubrouski. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] DRBD and pacemaker interaction
On Fri, Apr 01, 2011 at 08:42:04AM -0600, Serge Dubrouski wrote: > On Fri, Apr 1, 2011 at 8:38 AM, Lars Ellenberg > wrote: > > On Fri, Apr 01, 2011 at 11:35:19AM +0200, Christoph Bartoschek wrote: > >> Am 01.04.2011 11:27, schrieb Florian Haas: > >> > On 2011-04-01 10:49, Christoph Bartoschek wrote: > >> >> Am 01.04.2011 10:27, schrieb Andrew Beekhof: > >> >>> On Sat, Mar 26, 2011 at 12:10 AM, Lars Ellenberg > >> >>> wrote: > >> On Fri, Mar 25, 2011 at 06:18:07PM +0100, Christoph Bartoschek wrote: > >> > I am missing the state: running degraded or suboptimal. > >> > >> Yep, "degraded" is not a state available for pacemaker. > >> Pacemaker cannot do much about "suboptimal". > >> >>> > >> >>> I wonder what it would take to change that. I suspect either a > >> >>> crystal ball or way too much knowledge of drbd internals. > >> >> > >> >> The RA would be responsible to check this. For drbd any diskstate > >> >> different from UpToDate/UpToDate is suboptimal. > >> > > >> > Have you actually looked at the resource agent? It does already evaluate > >> > the disk state and adjusts the master preference accordingly. What else > >> > is there to do? > >> > >> Maybe I misunderstood Andrew's comment. I read it this way: If we > >> introduce a new state "suboptimal", would it be hard to detect it? > >> > >> I just wanted to express that detecting suboptimality seems not to be > >> that hard. > > > > But that state is useless for pacemaker, > > since it cannot do anything about it. > > Looks like a lot of people, including myself, are still confused with > this statement. Basically this state of DRBD resource is unstable and > resource is unusable, why do you think that this is normal for > Pacemaker to report a such state as Ok state? It is usable. It is being used. It is at least as usable as a degraded RAID1. Pacemaker cannot do anything about that missing disk, either. Of course you can patch pacemaker to detect that the RAID1 is degraded, and trigger faxing a PO to your supplier for a replacement drive. But possibly you should rather have some monitoring (nagios, ...) notice this, page/email/alert with your favorite method the relevant people, and have them take appropriate actions? -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] DRBD and pacemaker interaction
On Fri, Apr 1, 2011 at 4:46 PM, Serge Dubrouski wrote: > On Fri, Apr 1, 2011 at 8:43 AM, Andrew Beekhof wrote: >> On Fri, Apr 1, 2011 at 4:38 PM, Lars Ellenberg >> wrote: >>> On Fri, Apr 01, 2011 at 11:35:19AM +0200, Christoph Bartoschek wrote: Am 01.04.2011 11:27, schrieb Florian Haas: > On 2011-04-01 10:49, Christoph Bartoschek wrote: >> Am 01.04.2011 10:27, schrieb Andrew Beekhof: >>> On Sat, Mar 26, 2011 at 12:10 AM, Lars Ellenberg >>> wrote: On Fri, Mar 25, 2011 at 06:18:07PM +0100, Christoph Bartoschek wrote: > I am missing the state: running degraded or suboptimal. Yep, "degraded" is not a state available for pacemaker. Pacemaker cannot do much about "suboptimal". >>> >>> I wonder what it would take to change that. I suspect either a >>> crystal ball or way too much knowledge of drbd internals. >> >> The RA would be responsible to check this. For drbd any diskstate >> different from UpToDate/UpToDate is suboptimal. > > Have you actually looked at the resource agent? It does already evaluate > the disk state and adjusts the master preference accordingly. What else > is there to do? Maybe I misunderstood Andrew's comment. I read it this way: If we introduce a new state "suboptimal", would it be hard to detect it? >> >> No, detecting is the easy part. >> I just wanted to express that detecting suboptimality seems not to be that hard. >>> >>> But that state is useless for pacemaker, >>> since it cannot do anything about it. >> >> This was the part I was wondering about - if pacemaker _could_ do >> something intelligent. > > Isn't a simple reporting that resource is broken rather than healthy > intelligent enough? But is it completely broken? Is it broken in a way that pacemaker can do something to repair it? ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] DRBD problems are not reported
On Fri, Apr 01, 2011 at 09:37:09AM -0400, Vadym Chepkov wrote:
>
> On Apr 1, 2011, at 3:22 AM, Tim Serong wrote:
>
> > On 4/1/2011 at 11:37 AM, Vadym Chepkov wrote:
> >> On Mar 31, 2011, at 2:30 PM, Christoph Bartoschek wrote:
> >>
> >>> Am 29.03.2011 15:31, schrieb Dejan Muhamedagic:
> On Tue, Mar 29, 2011 at 08:13:49AM +0200, Christoph Bartoschek wrote:
> > Am 29.03.2011 02:35, schrieb Vadym Chepkov:
> >>
> >> On Mar 28, 2011, at 10:55 AM, Christoph Bartoschek wrote:
> >>
> >>> Am 28.03.2011 16:30, schrieb Dejan Muhamedagic:
> Hi,
>
> On Mon, Mar 21, 2011 at 11:33:49PM +0100, Christoph Bartoschek
> wrote:
> > Hi,
> >
> > I am testing a NFS failover setup. During the tests I created a
> > split-brain situation and now node A thinks it is primary and
> > uptodate
> > while node B thinks that it is Outdated.
> >
> > crm_mon however does not indicate any error to me. Why is this the
> > case?
> > I expect to see anything that shows me the degraded status. How can
> > this
> > be fixed?
>
> The cluster relies on the RA (in this case drbd) to report any
> problems. Do you have a monitor operation defined for that
> resource?
> >>>
> >>> I have the resource defined as:
> >>>
> >>> primitive p_drbd ocf:linbit:drbd \
> >>>params drbd_resource="home-data"
> >>>op monitor interval="15" role="Master" \
> >>>op monitor interval="30" role="Slave"
> >>>
> >>> Is this a correct monitor operation?
>
> Yes, though you should also add timeout specs.
>
> >> Just out of curiosity, you do have ms resource defined?
> >>
> >> ms ms_p_drbd p_drbd \
> >> meta master-max="1" master-node-max="1" clone-max="2"
> >> clone-node-max="1"
> >> notify="true"
> >>
> >> Because if you do and cluster is not aware of the split-brain, drbd RA
> >> has a
> >> serious flaw.
> >>
> >
> > I'm sorry. Yes, the ms resource is also defined.
>
> Well, I'm really confused. You basically say that the drbd disk
> gets into a degraded mode (i.e. it detects split brain), but the
> cluster (pacemaker) never learns about that. Perhaps you should
> open a bugzilla for this and supply hb_report. Though it's
> really hard to believe. It's like basic functionality failing.
> >>>
> >>>
> >>> What would you expect to see?
> >>>
> >>> Currently I see the following in crm_mon:
> >>>
> >>> Master/Slave Set: ms_drbd_nfs [p_drbd_nfs]
> >>>Masters: [ ries ]
> >>>Slaves: [ laplace ]
> >>>
> >>>
> >>> At the same time "cat /proc/drbd" on ries says:
> >>>
> >>> ries:~ # cat /proc/drbd
> >>> version: 8.3.9 (api:88/proto:86-95)
> >>> srcversion: A67EB2D25C5AFBFF3D8B788
> >>> 0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r-
> >>>ns:0 nr:0 dw:4 dr:1761 al:1 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:4
> >>>
> >>>
> >>> And on node laplace it says:
> >>>
> >>> laplace:~ # cat /proc/drbd
> >>> version: 8.3.9 (api:88/proto:86-95)
> >>> srcversion: A67EB2D25C5AFBFF3D8B788
> >>> 0: cs:StandAlone ro:Secondary/Unknown ds:Outdated/DUnknown r-
> >>>ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:4
> >>>
> >>>
> >>
> >>
> >>
> >> yes, and according to the RA script everything is perfect:
> >>
> >> drbd_status() {
> >>local rc
> >>rc=$OCF_NOT_RUNNING
> >>
> >>if ! is_drbd_enabled || ! [ -b "$DRBD_DEVICE" ]; then
> >>return $rc
> >>fi
> >>
> >># ok, module is loaded, block device node exists.
> >># lets see its status
> >>drbd_set_status_variables
> >>case "${DRBD_ROLE_LOCAL}" in
> >>Primary)
> >>rc=$OCF_RUNNING_MASTER
> >>;;
> >>Secondary)
> >>rc=$OCF_SUCCESS
> >>;;
> >>Unconfigured)
> >>rc=$OCF_NOT_RUNNING
> >>;;
> >>*)
> >>ocf_log err "Unexpected role ${DRBD_ROLE_LOCAL}"
> >>rc=$OCF_ERR_GENERIC
> >>esac
> >>
> >>return $rc
> >> }
> >>
> >> Staggering.
> >>
> >> drbd_set_status_variable subroutine does set DRBD_CSTATE
> >>
> >> I think the RA needs to be modified to something like this:
> >>
> >> Secondary)
> >>if [[ $DRBD_CSTATE == Connected ]]; then
> >>rc=$OCF_SUCCESS
> >>else
> >>rc=$OCF_NOT_RUNNING
> >>fi
> >
> > That wouldn't strictly be correct - DRBD *is* currently running on
> > both nodes, Primary (master) on one and Secondary (slave) on the
> > other. This state is correctly reported in crm_mon. The thing
> > that crm_mon can't tell you is that *third* piece of inform
Re: [Linux-HA] DRBD and pacemaker interaction
On Fri, Apr 1, 2011 at 8:43 AM, Andrew Beekhof wrote: > On Fri, Apr 1, 2011 at 4:38 PM, Lars Ellenberg > wrote: >> On Fri, Apr 01, 2011 at 11:35:19AM +0200, Christoph Bartoschek wrote: >>> Am 01.04.2011 11:27, schrieb Florian Haas: >>> > On 2011-04-01 10:49, Christoph Bartoschek wrote: >>> >> Am 01.04.2011 10:27, schrieb Andrew Beekhof: >>> >>> On Sat, Mar 26, 2011 at 12:10 AM, Lars Ellenberg >>> >>> wrote: >>> On Fri, Mar 25, 2011 at 06:18:07PM +0100, Christoph Bartoschek wrote: >>> > I am missing the state: running degraded or suboptimal. >>> >>> Yep, "degraded" is not a state available for pacemaker. >>> Pacemaker cannot do much about "suboptimal". >>> >>> >>> >>> I wonder what it would take to change that. I suspect either a >>> >>> crystal ball or way too much knowledge of drbd internals. >>> >> >>> >> The RA would be responsible to check this. For drbd any diskstate >>> >> different from UpToDate/UpToDate is suboptimal. >>> > >>> > Have you actually looked at the resource agent? It does already evaluate >>> > the disk state and adjusts the master preference accordingly. What else >>> > is there to do? >>> >>> Maybe I misunderstood Andrew's comment. I read it this way: If we >>> introduce a new state "suboptimal", would it be hard to detect it? > > No, detecting is the easy part. > >>> I just wanted to express that detecting suboptimality seems not to be >>> that hard. >> >> But that state is useless for pacemaker, >> since it cannot do anything about it. > > This was the part I was wondering about - if pacemaker _could_ do > something intelligent. Isn't a simple reporting that resource is broken rather than healthy intelligent enough? > ___ > Linux-HA mailing list > Linux-HA@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems > -- Serge Dubrouski. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] DRBD and pacemaker interaction
On Fri, Apr 1, 2011 at 4:38 PM, Lars Ellenberg wrote: > On Fri, Apr 01, 2011 at 11:35:19AM +0200, Christoph Bartoschek wrote: >> Am 01.04.2011 11:27, schrieb Florian Haas: >> > On 2011-04-01 10:49, Christoph Bartoschek wrote: >> >> Am 01.04.2011 10:27, schrieb Andrew Beekhof: >> >>> On Sat, Mar 26, 2011 at 12:10 AM, Lars Ellenberg >> >>> wrote: >> On Fri, Mar 25, 2011 at 06:18:07PM +0100, Christoph Bartoschek wrote: >> > I am missing the state: running degraded or suboptimal. >> >> Yep, "degraded" is not a state available for pacemaker. >> Pacemaker cannot do much about "suboptimal". >> >>> >> >>> I wonder what it would take to change that. I suspect either a >> >>> crystal ball or way too much knowledge of drbd internals. >> >> >> >> The RA would be responsible to check this. For drbd any diskstate >> >> different from UpToDate/UpToDate is suboptimal. >> > >> > Have you actually looked at the resource agent? It does already evaluate >> > the disk state and adjusts the master preference accordingly. What else >> > is there to do? >> >> Maybe I misunderstood Andrew's comment. I read it this way: If we >> introduce a new state "suboptimal", would it be hard to detect it? No, detecting is the easy part. >> I just wanted to express that detecting suboptimality seems not to be >> that hard. > > But that state is useless for pacemaker, > since it cannot do anything about it. This was the part I was wondering about - if pacemaker _could_ do something intelligent. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] DRBD and pacemaker interaction
On Fri, Apr 1, 2011 at 8:38 AM, Lars Ellenberg wrote: > On Fri, Apr 01, 2011 at 11:35:19AM +0200, Christoph Bartoschek wrote: >> Am 01.04.2011 11:27, schrieb Florian Haas: >> > On 2011-04-01 10:49, Christoph Bartoschek wrote: >> >> Am 01.04.2011 10:27, schrieb Andrew Beekhof: >> >>> On Sat, Mar 26, 2011 at 12:10 AM, Lars Ellenberg >> >>> wrote: >> On Fri, Mar 25, 2011 at 06:18:07PM +0100, Christoph Bartoschek wrote: >> > I am missing the state: running degraded or suboptimal. >> >> Yep, "degraded" is not a state available for pacemaker. >> Pacemaker cannot do much about "suboptimal". >> >>> >> >>> I wonder what it would take to change that. I suspect either a >> >>> crystal ball or way too much knowledge of drbd internals. >> >> >> >> The RA would be responsible to check this. For drbd any diskstate >> >> different from UpToDate/UpToDate is suboptimal. >> > >> > Have you actually looked at the resource agent? It does already evaluate >> > the disk state and adjusts the master preference accordingly. What else >> > is there to do? >> >> Maybe I misunderstood Andrew's comment. I read it this way: If we >> introduce a new state "suboptimal", would it be hard to detect it? >> >> I just wanted to express that detecting suboptimality seems not to be >> that hard. > > But that state is useless for pacemaker, > since it cannot do anything about it. Looks like a lot of people, including myself, are still confused with this statement. Basically this state of DRBD resource is unstable and resource is unusable, why do you think that this is normal for Pacemaker to report a such state as Ok state? > > I thought I made that clear. > > -- > : Lars Ellenberg > : LINBIT | Your Way to High Availability > : DRBD/HA support and consulting http://www.linbit.com > ___ > Linux-HA mailing list > Linux-HA@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems > -- Serge Dubrouski. ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] DRBD and pacemaker interaction
On Fri, Apr 01, 2011 at 11:35:19AM +0200, Christoph Bartoschek wrote: > Am 01.04.2011 11:27, schrieb Florian Haas: > > On 2011-04-01 10:49, Christoph Bartoschek wrote: > >> Am 01.04.2011 10:27, schrieb Andrew Beekhof: > >>> On Sat, Mar 26, 2011 at 12:10 AM, Lars Ellenberg > >>>wrote: > On Fri, Mar 25, 2011 at 06:18:07PM +0100, Christoph Bartoschek wrote: > > I am missing the state: running degraded or suboptimal. > > Yep, "degraded" is not a state available for pacemaker. > Pacemaker cannot do much about "suboptimal". > >>> > >>> I wonder what it would take to change that. I suspect either a > >>> crystal ball or way too much knowledge of drbd internals. > >> > >> The RA would be responsible to check this. For drbd any diskstate > >> different from UpToDate/UpToDate is suboptimal. > > > > Have you actually looked at the resource agent? It does already evaluate > > the disk state and adjusts the master preference accordingly. What else > > is there to do? > > Maybe I misunderstood Andrew's comment. I read it this way: If we > introduce a new state "suboptimal", would it be hard to detect it? > > I just wanted to express that detecting suboptimality seems not to be > that hard. But that state is useless for pacemaker, since it cannot do anything about it. I thought I made that clear. -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Fw: [Pacemaker] WARN: msg_to_op(1324): failed to get the value of field lrm_opstatus from a ha_msg
On Tue, Mar 29, 2011 at 05:14:39PM -0700, Bob Schatz wrote: > I am forwarding this to the LinuxHA mailing list in case I emailed it to the > wrong list. Sometimes it takes time ... > A few more thoughts that occurred after I hit > > 1. This problem sees to only occur when "/etc/init.d/heartbeat start" is > executed on two nodes at the same time. If I only do one at a time it does > not > seem to occur. (this may be related to the creation of master/slave > resources > in /etc/ha.d/resource.d/startstop when heartbeat starts) > 2. This problem seemed to occur most frequently when I went from 4 > master/slave > > resources to 6 master/slave resources. OK. Interesting. > Thanks, > > Bob > > > - Original Message > From: Bob Schatz > To: The Pacemaker cluster resource manager > Sent: Fri, March 25, 2011 4:22:39 PM > Subject: Re: [Pacemaker] WARN: msg_to_op(1324): failed to get the value of > field > > lrm_opstatus from a ha_msg > > After reading more threads, I noticed that I needed to include the PE outputs. > > Therefore, I have rerun the tests and included the PE outputs, the > configuration > > > file and the logs for both nodes. > > The test was rerun with max-children of 20. > > Thanks, > > Bob > > > - Original Message > From: Bob Schatz > To: pacema...@oss.clusterlabs.org > Sent: Thu, March 24, 2011 7:35:54 PM > Subject: [Pacemaker] WARN: msg_to_op(1324): failed to get the value of field > lrm_opstatus from a ha_msg > > I am getting these messages in the log: > >2011-03-24 18:53:12| warning |crmd: [27913]: WARN: msg_to_op(1324): failed > to > > > > get the value of field lrm_opstatus from a ha_msg >2011-03-24 18:53:12| info |crmd: [27913]: info: msg_to_op: Message follows: >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG: Dumping message with > 16 > fields >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[0] : [lrm_t=op] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[1] : > [lrm_rid=SSJE02A2:0] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[2] : [lrm_op=start] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[3] : > [lrm_timeout=30] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[4] : [lrm_interval=0] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[5] : [lrm_delay=0] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[6] : [lrm_copyparams=1] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[7] : [lrm_t_run=0] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[8] : [lrm_t_rcchange=0] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[9] : [lrm_exec_time=0] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[10] : > [lrm_queue_time=0] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[11] : [lrm_targetrc=-1] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[12] : [lrm_app=crmd] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[13] : > [lrm_userdata=91:3:0:dc9ad1c7-1d74-4418-a002-34426b34b576] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[14] : > [(2)lrm_param=0x64c230(938 1098)] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG: Dumping message with > 27 > fields >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[0] : [CRM_meta_clone=0] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[1] : > [CRM_meta_notify_slave_resource= ] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[2] : > [CRM_meta_notify_active_resource= ] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[3] : > [CRM_meta_notify_demote_uname= ] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[4] : > [CRM_meta_notify_inactive_resource=SSJE02A2:0 SSJE02A2:1 ] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[5] : > [ssconf=/var/omneon/config/config.JE02A2] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[6] : > [CRM_meta_master_node_max=1] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[7] : > [CRM_meta_notify_stop_resource= ] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[8] : > [CRM_meta_notify_master_resource= ] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[9] : > [CRM_meta_clone_node_max=1] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[10] : > [CRM_meta_clone_max=2] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[11] : > [CRM_meta_notify=true] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[12] : > [CRM_meta_notify_start_resource=SSJE02A2:0 SSJE02A2:1 ] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[13] : > [CRM_meta_notify_stop_uname= ] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[14] : > [crm_feature_set=3.0.1] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[15] : > [CRM_meta_notify_master_uname= ] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[16] : > [CRM_meta_master_max=1] >2011-03-24 18:53:12| info |crmd: [27913]: info: MSG[17] : > [CRM
Re: [Linux-HA] DRBD problems are not reported
On Apr 1, 2011, at 3:22 AM, Tim Serong wrote:
> On 4/1/2011 at 11:37 AM, Vadym Chepkov wrote:
>> On Mar 31, 2011, at 2:30 PM, Christoph Bartoschek wrote:
>>
>>> Am 29.03.2011 15:31, schrieb Dejan Muhamedagic:
On Tue, Mar 29, 2011 at 08:13:49AM +0200, Christoph Bartoschek wrote:
> Am 29.03.2011 02:35, schrieb Vadym Chepkov:
>>
>> On Mar 28, 2011, at 10:55 AM, Christoph Bartoschek wrote:
>>
>>> Am 28.03.2011 16:30, schrieb Dejan Muhamedagic:
Hi,
On Mon, Mar 21, 2011 at 11:33:49PM +0100, Christoph Bartoschek wrote:
> Hi,
>
> I am testing a NFS failover setup. During the tests I created a
> split-brain situation and now node A thinks it is primary and
> uptodate
> while node B thinks that it is Outdated.
>
> crm_mon however does not indicate any error to me. Why is this the
> case?
> I expect to see anything that shows me the degraded status. How can
> this
> be fixed?
The cluster relies on the RA (in this case drbd) to report any
problems. Do you have a monitor operation defined for that
resource?
>>>
>>> I have the resource defined as:
>>>
>>> primitive p_drbd ocf:linbit:drbd \
>>>params drbd_resource="home-data"
>>>op monitor interval="15" role="Master" \
>>>op monitor interval="30" role="Slave"
>>>
>>> Is this a correct monitor operation?
Yes, though you should also add timeout specs.
>> Just out of curiosity, you do have ms resource defined?
>>
>> ms ms_p_drbd p_drbd \
>> meta master-max="1" master-node-max="1" clone-max="2"
>> clone-node-max="1"
>> notify="true"
>>
>> Because if you do and cluster is not aware of the split-brain, drbd RA
>> has a
>> serious flaw.
>>
>
> I'm sorry. Yes, the ms resource is also defined.
Well, I'm really confused. You basically say that the drbd disk
gets into a degraded mode (i.e. it detects split brain), but the
cluster (pacemaker) never learns about that. Perhaps you should
open a bugzilla for this and supply hb_report. Though it's
really hard to believe. It's like basic functionality failing.
>>>
>>>
>>> What would you expect to see?
>>>
>>> Currently I see the following in crm_mon:
>>>
>>> Master/Slave Set: ms_drbd_nfs [p_drbd_nfs]
>>>Masters: [ ries ]
>>>Slaves: [ laplace ]
>>>
>>>
>>> At the same time "cat /proc/drbd" on ries says:
>>>
>>> ries:~ # cat /proc/drbd
>>> version: 8.3.9 (api:88/proto:86-95)
>>> srcversion: A67EB2D25C5AFBFF3D8B788
>>> 0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r-
>>>ns:0 nr:0 dw:4 dr:1761 al:1 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:4
>>>
>>>
>>> And on node laplace it says:
>>>
>>> laplace:~ # cat /proc/drbd
>>> version: 8.3.9 (api:88/proto:86-95)
>>> srcversion: A67EB2D25C5AFBFF3D8B788
>>> 0: cs:StandAlone ro:Secondary/Unknown ds:Outdated/DUnknown r-
>>>ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:4
>>>
>>>
>>
>>
>>
>> yes, and according to the RA script everything is perfect:
>>
>> drbd_status() {
>>local rc
>>rc=$OCF_NOT_RUNNING
>>
>>if ! is_drbd_enabled || ! [ -b "$DRBD_DEVICE" ]; then
>>return $rc
>>fi
>>
>># ok, module is loaded, block device node exists.
>># lets see its status
>>drbd_set_status_variables
>>case "${DRBD_ROLE_LOCAL}" in
>>Primary)
>>rc=$OCF_RUNNING_MASTER
>>;;
>>Secondary)
>>rc=$OCF_SUCCESS
>>;;
>>Unconfigured)
>>rc=$OCF_NOT_RUNNING
>>;;
>>*)
>>ocf_log err "Unexpected role ${DRBD_ROLE_LOCAL}"
>>rc=$OCF_ERR_GENERIC
>>esac
>>
>>return $rc
>> }
>>
>> Staggering.
>>
>> drbd_set_status_variable subroutine does set DRBD_CSTATE
>>
>> I think the RA needs to be modified to something like this:
>>
>> Secondary)
>> if [[ $DRBD_CSTATE == Connected ]]; then
>> rc=$OCF_SUCCESS
>> else
>> rc=$OCF_NOT_RUNNING
>> fi
>
> That wouldn't strictly be correct - DRBD *is* currently running on
> both nodes, Primary (master) on one and Secondary (slave) on the
> other. This state is correctly reported in crm_mon. The thing
> that crm_mon can't tell you is that *third* piece of information,
> i.e. that there's some sort of communication breakdown between
> the two instances.
>
Well, it is definitely not doing it's "Slave" job when it is not connected.
> That being said, I'll defer to the DRBD crew as to whether or not
> returning $OCF_NOT_RUNNING in this case is technically safe and/or
> desirable.
>
> (I know its administr
Re: [Linux-HA] DRBD and pacemaker interaction
Am 01.04.2011 11:27, schrieb Florian Haas: > On 2011-04-01 10:49, Christoph Bartoschek wrote: >> Am 01.04.2011 10:27, schrieb Andrew Beekhof: >>> On Sat, Mar 26, 2011 at 12:10 AM, Lars Ellenberg >>>wrote: On Fri, Mar 25, 2011 at 06:18:07PM +0100, Christoph Bartoschek wrote: > I am missing the state: running degraded or suboptimal. Yep, "degraded" is not a state available for pacemaker. Pacemaker cannot do much about "suboptimal". >>> >>> I wonder what it would take to change that. I suspect either a >>> crystal ball or way too much knowledge of drbd internals. >> >> The RA would be responsible to check this. For drbd any diskstate >> different from UpToDate/UpToDate is suboptimal. > > Have you actually looked at the resource agent? It does already evaluate > the disk state and adjusts the master preference accordingly. What else > is there to do? Maybe I misunderstood Andrew's comment. I read it this way: If we introduce a new state "suboptimal", would it be hard to detect it? I just wanted to express that detecting suboptimality seems not to be that hard. Christoph ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Problem with an active/active NFS setup with exportfs RA
Il giorno Gio 31 Mar 2011 18:42:17 CET, Alessandro Iurlano ha scritto: > Hello. [...] > I have tried to put /var/lib/nfs directory on an OCFS2 filesystem > shared by both nodes but I had a lot of stability problems with the > nfs server processes. In particular they often seems to hang while > starting or even stopping. I think this could be because they may be > locking some files on the shared filesystems. As the file are kept > locked by the daemons, further lock operation may be blocked > indefinitely. Having a shared /var/lib/nfs make sense with active-standby configurations because the nfs servers does not talk with each other, so I think it's normal to have instability. > Then I tried to find a way to keep just the rmtab file synchronized on > both nodes. I cannot find a way to have pacemaker do this for me. Is > there one? As far as I know, all those operations are handled by the exportfs RA. > Also, I have found that the exportfs RA originally had a mechanism to > keep rmtab synchronized but it has been removed in this commit: > https://github.com/ClusterLabs/resource-agents/commit/0edb009a87f0d47b310998f2cb3809d2775e2de8 > Is there another way to accomplish this active/active setup? I've configured a lot of A/A setup with exportfs without having problems. What's the boot order of your resources? Are you sure you're removing first of all the IP and then exportfs? I'm asking this because one of my problem was about the connections keep opened by the clients (i was removing first exportfs). -- RaSca Mia Mamma Usa Linux: Niente è impossibile da capire, se lo spieghi bene! ra...@miamammausalinux.org http://www.miamammausalinux.org ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] DRBD and pacemaker interaction
On 2011-04-01 10:49, Christoph Bartoschek wrote: > Am 01.04.2011 10:27, schrieb Andrew Beekhof: >> On Sat, Mar 26, 2011 at 12:10 AM, Lars Ellenberg >> wrote: >>> On Fri, Mar 25, 2011 at 06:18:07PM +0100, Christoph Bartoschek wrote: I am missing the state: running degraded or suboptimal. >>> >>> Yep, "degraded" is not a state available for pacemaker. >>> Pacemaker cannot do much about "suboptimal". >> >> I wonder what it would take to change that. I suspect either a >> crystal ball or way too much knowledge of drbd internals. > > The RA would be responsible to check this. For drbd any diskstate > different from UpToDate/UpToDate is suboptimal. Have you actually looked at the resource agent? It does already evaluate the disk state and adjusts the master preference accordingly. What else is there to do? Cheers, Florian signature.asc Description: OpenPGP digital signature ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
[Linux-HA] Linux-HA Over WAN Advisable?
Greetings! We have a few Corosync+PaceMaker+DRBD clusters and a couple older Heartbeat+DRBD clusters. Our infrastructure is currently located in a single facility. We have the opportunity to establish a DR site in another data center over a high bandwidth connection (2-4Gbps). I am thinking of moving the standby cluster nodes to the secondary data center. One thing that concerns me is the greater possibility of split-brain scenarios. Currently our clusters communicate heartbeat through multiple paths: 1. The primary LAN link that users communicate through; 2. The DRBD link; 3. A dedicated cluster communication LAN. If we move the standby cluster nodes to the secondary facility, there will no longer be redundancy on the heartbeat links. Everything will go through the inter-data-center connection. This concept is a little worrisome. Should I be very concerned? Is there a way to minimize the risk of split-brain over WAN? --Eric Disclaimer - March 31, 2011 This email and any files transmitted with it are confidential and intended solely for linux-ha@lists.linux-ha.org. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physicians' Managed Care or Physician Select Management. Warning: Although Physicians' Managed Care or Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. This disclaimer was added by Policy Patrol: http://www.policypatrol.com/ ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
[Linux-HA] Problem with an active/active NFS setup with exportfs RA
Hello. I am trying to setup an highly available active/active NFS server with two Debian boxes and pacemaker, corosync, ocfs2 and drbd. The pacemaker cluster manages the drbd dual primary configuration, DLM and O2CB daemons and the mount / umount of the shared filesystems. I have the nfs server process starting at boot with the debian init.d scripts with an empty /etc/exports file. The exports are added/removed with the recent exportfs RA. Active/active configuration is obtained by having each server share a different set of filesystems, and thus the nfs server processes are always active on both nodes. The cluster is able to correctly migrate the exports from a node to the other but the nfs state is not kept in synch between the two nodes. This cause any already connected client to hang when there is a failover of the resource. The client node correctly resume the operations when the exportfs is migrated back to the original node. Manually synchronizing the /var/lib/nfs/rmtab file on both servers seems to solve this problem and now the client node works fine both when the exportfs is migrated to the other node and when it is taken back to the original one. It hangs just for a few seconds during the migration. I have tried to put /var/lib/nfs directory on an OCFS2 filesystem shared by both nodes but I had a lot of stability problems with the nfs server processes. In particular they often seems to hang while starting or even stopping. I think this could be because they may be locking some files on the shared filesystems. As the file are kept locked by the daemons, further lock operation may be blocked indefinitely. Then I tried to find a way to keep just the rmtab file synchronized on both nodes. I cannot find a way to have pacemaker do this for me. Is there one? Also, I have found that the exportfs RA originally had a mechanism to keep rmtab synchronized but it has been removed in this commit: https://github.com/ClusterLabs/resource-agents/commit/0edb009a87f0d47b310998f2cb3809d2775e2de8 Is there another way to accomplish this active/active setup? Thanks! Alessandro ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] DRBD and pacemaker interaction
Am 01.04.2011 10:27, schrieb Andrew Beekhof: > On Sat, Mar 26, 2011 at 12:10 AM, Lars Ellenberg > wrote: >> On Fri, Mar 25, 2011 at 06:18:07PM +0100, Christoph Bartoschek wrote: >>> Hi, >>> >>> we experiment with DRBD and pacemaker and see several times that the >>> DRBD part is degraded (One node is outdated or diskless or something >>> similar) but crm_mon just reports that the DRBD resource runs as master >>> and slave on the nodes. >>> >>> There is no indication that the resource is not in its optimal mode of >>> operation. >>> >>> For me it seems as if pacemaker knows only the states: running, stopped, >>> failed. >>> >>> I am missing the state: running degraded or suboptimal. >> >> Yep, "degraded" is not a state available for pacemaker. >> Pacemaker cannot do much about "suboptimal". > > I wonder what it would take to change that. I suspect either a > crystal ball or way too much knowledge of drbd internals. The RA would be responsible to check this. For drbd any diskstate different from UpToDate/UpToDate is suboptimal. I now check for the diskstate with a separate Zabbix installation. Christoph ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] update question
Am 01.04.2011 um 09:44 schrieb Andrew Beekhof: > On Mon, Mar 28, 2011 at 9:08 PM, Miles Fidelman > wrote: >> Hi Folks, >> >> I'm getting ready to upgrade a 2-node HA cluster from Debian Etch to >> Squeeze. I'd very much appreciate any suggestions regarding >> "gotchas" >> to avoid, and so forth. Etch -> Lenny -> Squeeze The Lenny -> Squeeze upgrade works fine, if you have a clean Lenny- installation. I.e. you first must upgrade to Lenny. >> >> Basic current configuration: >> - 2 vanilla Intel-based servers, 4 SATA drives each If you have Intel e100 NICs, then you will experience problems with the non-free firmware drivers. The e1000 works fine. >> - each machine: disks set up as 4-drive md10, LVM >> - xen 3.2 hypervisor, Debian Etch Dom0 >> - DRBD 8.2, Pacemaker linking the two nodes >> - several Debian Etch DomUs Same problem here (many Dom0 and DomU with different Debian/Ubuntu versions - Etch, Lenny, Hardy). I do not want to risk the problems Etch > Squeeze, even not on the development (used for programming and test) machines. Lenny to Squeeze was nearly without problems (Dom0 and DomU). DRBD 8.2 (since 8.3.2 api:88/proto:86-88) should work with the other node having DRBD 8.3.7 (api:88/proto:86-91) -- theoretically. BTW: If your hardware is as old as the software, then think about new hardware. Helmut Wollmersdorfer ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Need help on the femce_vmware configuration ... facing errors ... !!
Hi, This is not the right ML, I think you may want to try with the linux-cluster list. Thanks, Dejan On Fri, Apr 01, 2011 at 05:30:53AM +, Amit Jathar wrote: > Hi, > > I am facing following issues:- > > I have installed VMware-VIPerl-1.6.0-104313.x86_64.tar.gz & > VMware-vSphere-Perl-SDK-4.1.0-254719.x86_64.tar.gz on the RHEL6 image. > > My ESXi server hostname is "esx5" > > 1) If I use wrong password, I get error message as expected :- > [root@OEL6_VIP_1 ~]# /usr/sbin/fence_vmware -o reboot -a x.x.x.x -l 'root' > -p 'wrong_pass' -n esx5 > fence_vmware_helper returned Cannot connect to server! > VMware error:Cannot complete login due to an incorrect user name or password. > > Please use '-h' for usage > > 2) If I use right password, then I get error message like:- > [root@OEL6_VIP_1 ~]# /usr/sbin/fence_vmware -o reboot -a 172.16.150.5 -l > 'root' -p 'right_pass' -n esx5 > fence_vmware_helper returned Cannot find vm esx5! > > I can ping that node "esx5" :- > PING esx5.localdomain (x.x.x.x) 56(84) bytes of data. > 64 bytes from esx5.localdomain (x.x.x.x): icmp_seq=1 ttl=64 time=0.064 ms > > What might be going wrong, so that my fence_vmware script is not able to find > "esx5"? > > Thanks, > Amit > > -Original Message- > From: linux-ha-boun...@lists.linux-ha.org > [mailto:linux-ha-boun...@lists.linux-ha.org] On Behalf Of Amit Jathar > Sent: Wednesday, March 30, 2011 10:08 PM > To: General Linux-HA mailing list > Subject: [Linux-HA] Need help on the femce_vmware configuration > > Hi, > > I am having two RHEL6 vmware images running on Windows machines. > I have configured STONITH on those RHEL6 images. > > I have installed the VI API on those machines. > I get error as "cound not connect http://x.x.x.x/SDK/webService"; when I > manually try "/usr/sbin/fence_vmware -o reboot -a x.x.x.x -l xxx -p xxx -n > xxx" > > My question is :- can I use this fence_vmware script on the RHEL6 vmware > images running on Windows machines ? > or I must use the RHEL6 vmware images running on > ESXi server ? > > Thanks, > Amit > > > > This email (message and any attachment) is confidential and may be > privileged. If you are not certain that you are the intended recipient, > please notify the sender immediately by replying to this message, and delete > all copies of this message and attachments. Any other use of this email by > you is prohibited. > > > > ___ > Linux-HA mailing list > Linux-HA@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems > > > > This email (message and any attachment) is confidential and may be > privileged. If you are not certain that you are the intended recipient, > please notify the sender immediately by replying to this message, and delete > all copies of this message and attachments. Any other use of this email by > you is prohibited. > > > > ___ > Linux-HA mailing list > Linux-HA@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] DRBD and pacemaker interaction
On Sat, Mar 26, 2011 at 12:10 AM, Lars Ellenberg wrote: > On Fri, Mar 25, 2011 at 06:18:07PM +0100, Christoph Bartoschek wrote: >> Hi, >> >> we experiment with DRBD and pacemaker and see several times that the >> DRBD part is degraded (One node is outdated or diskless or something >> similar) but crm_mon just reports that the DRBD resource runs as master >> and slave on the nodes. >> >> There is no indication that the resource is not in its optimal mode of >> operation. >> >> For me it seems as if pacemaker knows only the states: running, stopped, >> failed. >> >> I am missing the state: running degraded or suboptimal. > > Yep, "degraded" is not a state available for pacemaker. > Pacemaker cannot do much about "suboptimal". I wonder what it would take to change that. I suspect either a crystal ball or way too much knowledge of drbd internals. > > Pacemaker can stop, start, and promote/demote resources. > No more, no less. > > If your resources are running "suboptimal" (but working), > stopping/restarting things, in the hope that would make them > run better, likely won't add to your availability. > > Pacemaker is not a substitute for proper monitoring (nagios, whatever). > > Monitoring can page your engineer on duty (or yourself) > for things that require immediate admin intervention. > Monitoring can provide you with nice graphs, so you can detect early > which things may require strategic admin intervention. > > It is not pacemaker's job to do either. > >> Is it already there and I have made an configuration error? Or what is >> the recommended way to check the sanity of the resources controlled by >> pacemaker? > > Do you expect the cluster manager to sound the alarm beep as well, > if a disk falls out of the raid, or the battery of the BBWC on the > controler is depleted? > Or if the response time of your home page goes bad (but the status > page comes still back within the timeout)? > > What is Pacemaker expected to do? Stop everything? > > If you are Primary on DRBD, and the lower level disk has some IO error, > DRBD detaches from the local disk. The RA will notice this on the next > monitoring intervall, and adjust the master score accordingly. > Depending on overall configuration, pacemaker may then decide to migrate > resource over to the other node, or not. > > But many other resource internal problems, > replication link damage or something like that, > pacemaker has no way to magically heal things. > > > But ok, for strictly "informational purposes", conceivably, > we could add a monitoring result code to the RA spec saying > "working [slave/master], but degraded". > > That could then be presented in some obvious way in crm_mon, or even > trigger certain action scripts (which again could then page you). > > Currently, a similar effect could be achieved > by adding some sort of "supervisor resource", > which would need to be made dependent of the supervised resource, > and would "fail" if the supervised resource is not running "optimal". > > My feeling is, don't try to do everything with the same tool. > Use the best tool for the job. > Use a monitoring tool for system monitoring. > Use a cluster manager for cluster management. > > -- > : Lars Ellenberg > : LINBIT | Your Way to High Availability > : DRBD/HA support and consulting http://www.linbit.com > ___ > Linux-HA mailing list > Linux-HA@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems > ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] update question
On Mon, Mar 28, 2011 at 9:08 PM, Miles Fidelman wrote: > Hi Folks, > > I'm getting ready to upgrade a 2-node HA cluster from Debian Etch to > Squeeze. I'd very much appreciate any suggestions regarding "gotchas" > to avoid, and so forth. > > Basic current configuration: > - 2 vanilla Intel-based servers, 4 SATA drives each > - each machine: disks set up as 4-drive md10, LVM > - xen 3.2 hypervisor, Debian Etch Dom0 > - DRBD 8.2, Pacemaker linking the two nodes > - several Debian Etch DomUs > > Target configuration: > - update to xen 4.1, Debian Squeeze Dom0 > - update to latest DRBD, Pacemaker > - update DomUs on a case-by-case basis > > The most obvious question: Are later versions of Xen, DRBD, Pacemaker > compatible with the older ones? I.e., can I take the simple approach: > - migrate all DomUs to one machine > - take the other machine off-line > - upgrade Debian, Xen, DRBD, Pacemaker on the off-line node > - bring that node back on-line (WILL THE NEW VERSIONS OF XEN, DRBD, > PACEMAKER SYNC WITH THE PREVIOUS RELEASES ON THE OTHER NODE???) I can only comment on Pacemaker: I think so, but you haven't indicated which versions > - migrate stuff to the updated node > - take the 2nd node off-line, update everything, bring it back up, resync > > 1. Will that work? (If not: Alternative suggestions?) > 2. Anything to watch out for? > 3. As an alternative, does it make sense to install Ganeti and use it to > manage the cluster? If so, any suggestions on a migration path? > > (Yes, it would be easier if I had one or two additional servers to use > for intermediate staging, but such is life.) > > Thanks Very Much, > > Miles Fidelman > > -- > In theory, there is no difference between theory and practice. > In practice, there is. Yogi Berra > > > ___ > Linux-HA mailing list > Linux-HA@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems > ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] Which STONITH plugin for DELL 510 iDRAC6?
Am 01.04.2011 um 03:10 schrieb Vadym Chepkov: > I use external/ipmi quite successfully. > Enable IPMI over LAN: [...] Thx for the hint and config example. Helmut Wollmersdorfer ___ Linux-HA mailing list Linux-HA@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
Re: [Linux-HA] DRBD problems are not reported
On 4/1/2011 at 11:37 AM, Vadym Chepkov wrote:
> On Mar 31, 2011, at 2:30 PM, Christoph Bartoschek wrote:
>
> > Am 29.03.2011 15:31, schrieb Dejan Muhamedagic:
> >> On Tue, Mar 29, 2011 at 08:13:49AM +0200, Christoph Bartoschek wrote:
> >>> Am 29.03.2011 02:35, schrieb Vadym Chepkov:
>
> On Mar 28, 2011, at 10:55 AM, Christoph Bartoschek wrote:
>
> > Am 28.03.2011 16:30, schrieb Dejan Muhamedagic:
> >> Hi,
> >>
> >> On Mon, Mar 21, 2011 at 11:33:49PM +0100, Christoph Bartoschek wrote:
> >>> Hi,
> >>>
> >>> I am testing a NFS failover setup. During the tests I created a
> >>> split-brain situation and now node A thinks it is primary and
> >>> uptodate
> >>> while node B thinks that it is Outdated.
> >>>
> >>> crm_mon however does not indicate any error to me. Why is this the
> >>> case?
> >>> I expect to see anything that shows me the degraded status. How can
> >>> this
> >>> be fixed?
> >>
> >> The cluster relies on the RA (in this case drbd) to report any
> >> problems. Do you have a monitor operation defined for that
> >> resource?
> >
> > I have the resource defined as:
> >
> > primitive p_drbd ocf:linbit:drbd \
> > params drbd_resource="home-data"
> > op monitor interval="15" role="Master" \
> > op monitor interval="30" role="Slave"
> >
> > Is this a correct monitor operation?
> >>
> >> Yes, though you should also add timeout specs.
> >>
> Just out of curiosity, you do have ms resource defined?
>
> ms ms_p_drbd p_drbd \
> meta master-max="1" master-node-max="1" clone-max="2"
> clone-node-max="1"
> notify="true"
>
> Because if you do and cluster is not aware of the split-brain, drbd RA
> has a
> serious flaw.
>
> >>>
> >>> I'm sorry. Yes, the ms resource is also defined.
> >>
> >> Well, I'm really confused. You basically say that the drbd disk
> >> gets into a degraded mode (i.e. it detects split brain), but the
> >> cluster (pacemaker) never learns about that. Perhaps you should
> >> open a bugzilla for this and supply hb_report. Though it's
> >> really hard to believe. It's like basic functionality failing.
> >
> >
> > What would you expect to see?
> >
> > Currently I see the following in crm_mon:
> >
> > Master/Slave Set: ms_drbd_nfs [p_drbd_nfs]
> > Masters: [ ries ]
> > Slaves: [ laplace ]
> >
> >
> > At the same time "cat /proc/drbd" on ries says:
> >
> > ries:~ # cat /proc/drbd
> > version: 8.3.9 (api:88/proto:86-95)
> > srcversion: A67EB2D25C5AFBFF3D8B788
> > 0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r-
> > ns:0 nr:0 dw:4 dr:1761 al:1 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:4
> >
> >
> > And on node laplace it says:
> >
> > laplace:~ # cat /proc/drbd
> > version: 8.3.9 (api:88/proto:86-95)
> > srcversion: A67EB2D25C5AFBFF3D8B788
> > 0: cs:StandAlone ro:Secondary/Unknown ds:Outdated/DUnknown r-
> > ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:4
> >
> >
>
>
>
> yes, and according to the RA script everything is perfect:
>
> drbd_status() {
> local rc
> rc=$OCF_NOT_RUNNING
>
> if ! is_drbd_enabled || ! [ -b "$DRBD_DEVICE" ]; then
> return $rc
> fi
>
> # ok, module is loaded, block device node exists.
> # lets see its status
> drbd_set_status_variables
> case "${DRBD_ROLE_LOCAL}" in
> Primary)
> rc=$OCF_RUNNING_MASTER
> ;;
> Secondary)
> rc=$OCF_SUCCESS
> ;;
> Unconfigured)
> rc=$OCF_NOT_RUNNING
> ;;
> *)
> ocf_log err "Unexpected role ${DRBD_ROLE_LOCAL}"
> rc=$OCF_ERR_GENERIC
> esac
>
> return $rc
> }
>
> Staggering.
>
> drbd_set_status_variable subroutine does set DRBD_CSTATE
>
> I think the RA needs to be modified to something like this:
>
> Secondary)
> if [[ $DRBD_CSTATE == Connected ]]; then
> rc=$OCF_SUCCESS
> else
> rc=$OCF_NOT_RUNNING
> fi
That wouldn't strictly be correct - DRBD *is* currently running on
both nodes, Primary (master) on one and Secondary (slave) on the
other. This state is correctly reported in crm_mon. The thing
that crm_mon can't tell you is that *third* piece of information,
i.e. that there's some sort of communication breakdown between
the two instances.
That being said, I'll defer to the DRBD crew as to whether or not
returning $OCF_NOT_RUNNING in this case is technically safe and/or
desirable.
(I know its administratively highly desirable to see these failures,
of course, I'm just not clear on how best to expose them).
Regards,
Tim
--
