(Big Cheers and celebrations from this end!!!)

Finally figured out what the problem was, it seems that the kernel oops were being caused by the 8.3 version of DRBD, once downgraded to 8.2.7 everything started to work as it should. primary / secondary automatic fail over is in place and resources are now following the DRBD master!

Thanks a mill for all the help.

Jason

On Feb 12, 2009 8:48am, Jason Fitzpatrick <jayfitzpatr...@gmail.com> wrote:
Hi Dominik

thanks again for the feedback,

I had noticed some kernel opps's since the last kernel update that i and
they seem to be pointing to DRBD, i will downgrade the kernel again and see if this improves things,


re Stonith I Uninstalled as part of the move from heartbeat v2.1 to 2.9
but must have missed this bit.

user land and kernel module all report the same version.

I am on my way into the office now and I will apply the changes once there


thanks again

Jason

2009/2/12 Dominik Klein d...@in-telegence.net>

Right, this one looks better.



I'll refer to nodes as 1001 and 1002.



1002 is your DC.

You have stonith enabled, but no stonith devices. Disable stonith or get

and configure a stonith device (_please_ dont use ssh).



1002 ha-log lines 926:939, node 1002 wants to shoot 1001, but cannot (l

978). Retries in l 1018 and fails again in l 1035.



Then, the cluster tries to start drbd on 1001 in l 1079, followed by a

bunch of kernel messages I don't understand (pretty sure _this_ is the

first problem you should address!), ending up in the drbd RA not able to

see the secondary state (1449) and considering the start failed.



The RA code for this is

if do_drbdadm up $RESOURCE ; then

drbd_get_status

if [ "$DRBD_STATE_LOCAL" != "Secondary" ]; then

ocf_log err "$RESOURCE start: not in Secondary mode after start."

return $OCF_ERR_GENERIC

fi

ocf_log debug "$RESOURCE start: succeeded."

return $OCF_SUCCESS

else

ocf_log err "$RESOURCE: Failed to start up."

return $OCF_ERR_GENERIC

fi



The cluster then successfully stops drbd again (l 1508-1511) and tries

to start the other clone instance (l 1523).



Log says

RA output: (Storage1:1:start:stdout) /dev/drbd0: Failure: (124) Device

is attached to a disk (use detach first) Command 'drbdsetup /dev/drbd0

disk /dev/sdb /dev/sdb internal --set-defaults --create-device

--on-io-error=pass_on' terminated with exit code 10


Feb 11 15:39:05 lpissan1002 drbd[3473]: ERROR: Storage1 start: not in

Secondary mode after start.



So this is interesting. Although "stop" (basically drbdadm down)

succeeded, the drbd device is still attached!



Please try:

stop the cluster

drbdadm up $resource

drbdadm up $resource #again

echo $?

drbdadm down $resource

echo $?

cat /proc/drbd



Btw: Does your userland match your kernel module version?



To bring this to an end: The start of the second clone instance also

failed, so both instances are unrunnable on the node and no further

start is tried on 1002.



Interestingly, then (could not see any attempt before), the cluster

wants to start drbd on node 1001, but it also fails and also gives those

kernel messages. In l 2001, each instance has a failed start on each node.



So: Find out about those kernel messages. Can't help much on that

unfortunately, but there were some threads about things like that on

drbd-user recently. Maybe you can find answers to that problem there.



And also: please verify returncodes of drbdadm in your case. Maybe

that's a drbd tools bug? (can't say for sure, for me, up on an alreay up

resource gives 1, which is ok).



Regards

Dominik



Jason Fitzpatrick wrote:


> it seems that I had the incorrect version of openais installed (from the

> fedora repo vs the HA one)

>

> I have corrected and the hb_report ran correctly using the following

>

> hb_report -u root -f 3pm /tmp/report

>

> Please see attached

>

> Thanks again

>

> Jason





_______________________________________________

Linux-HA mailing list

Linux-HA@lists.linux-ha.org

http://lists.linux-ha.org/mailman/listinfo/linux-ha

See also: http://linux-ha.org/ReportingProblems






_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

Reply via email to