So...
1.) If a system "panic"'s, then filesystem corruption found when the
Service Group onlines on the other node was likely caused by the same
problem that caused the panic.
and
2.) If a system "hangs" (to use a word) -- but does not panic -- and
filesystem corruption is found when the Service Group onlines on the other
node, it is possible the first node was still writing to the filesystem
resulting in a split brain scenario. And if so, it's likely that caused the
filesystem corruption.
And one of the ways to prevent the split brain scenario is I/O Fencing...
Jon
"Jim Senicka"
<[EMAIL PROTECTED]
mantec.com> To
"Jon E Price/SYS/NYTIMES"
10/27/2008 08:23 <[EMAIL PROTECTED]>, "Andrey
PM Dmitriev" <[EMAIL PROTECTED]>,
"Joshua Fielden"
<[EMAIL PROTECTED]>,
<veritas-ha@mailman.eng.auburn.edu>
cc
Subject
RE: [Veritas-ha] Question about HA
and disks
In the original message
" We had an issue where a serverA failed and serverB took over.
However, serverB took over when serverA was still 'crashing' (it took a
good 10-15mins to crash),"
I can assume crash = panic, as "crashing" has to refer to dumping core
to disk.
If this is the case, there will be no logs on server A, as it is mid
panic.
In this case (the node is in the middle of a crash dump), it will not be
writing to data disks. What ever was written happened before the kernel
call to panic. Fencing will protect that data once the new node imports,
but in the case described here, the corruption had to happen before the
panic, so fence would not have helped.
Bottom line is the node ceased writing as soon as the non maskable
interrupt was called for panic (unless Linux somehow violates every Unix
kernel rule, which I seriously doubt). When VCS took over the service
group on Server B, Server A was down and could not have been writing
-----Original Message-----
From: Jon E Price/SYS/NYTIMES [mailto:[EMAIL PROTECTED]
Sent: Monday, October 27, 2008 8:14 PM
To: Jim Senicka; Andrey Dmitriev; Joshua Fielden;
veritas-ha@mailman.eng.auburn.edu
Subject: Re: [Veritas-ha] Question about HA and disks
Hi,
A few questions..
Andrey: Could you post the logs (or even portions of them) which show
what
ServerA was doing during the takeover?
Joshua: You're saying that IO Fencing can prevent split brain situations
in
which one server is still writing to a filesystem while a 2nd server has
taken over that same service group and begun writing to the same fs,
thus
possibly causing corruption?
http://sfdoccentral.symantec.com/sf/5.0/linux/html/vcs_install/ch_vcs_in
stall_iofence.html#190559
Jim: What's the evidence that the server panic'd?
And is 16 seconds the default for the heartbeat failure?
Jon
"Jim Senicka"
<[EMAIL PROTECTED]
mantec.com>
To
Sent by: "Andrey Dmitriev"
veritas-ha-bounce <[EMAIL PROTECTED]>,
[EMAIL PROTECTED]
<veritas-ha@mailman.eng.auburn.edu>
urn.edu
cc
Subject
10/27/2008 07:19 Re: [Veritas-ha] Question about
HA
PM and disks
When a server panics, it stops writing to anything but the dump device.
VCS did exactly as designed. 16 seconds after heartbeat failure it
started takeover. Whatever was damaged on your file system was already
damaged at that point, regardless how long it took to dump core to the
dump device. I would look at the cause of the panic, and it is likely it
was something to do with what garbaged your FS
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Andrey
Dmitriev
Sent: Monday, October 27, 2008 2:01 PM
To: veritas-ha@mailman.eng.auburn.edu
Subject: [Veritas-ha] Question about HA and disks
We had an issue where a serverA failed and serverB took over.
However, serverB took over when serverA was still 'crashing' (it took a
good 10-15mins to crash), and apparently still had a hold of file
systems (system logs confirm that takeover occurred while serverA was
still 'puking').
The file systems on ServerB came up corrupt, and we lost some data b/c
of that.
HA is setup via heartbeats. File system is vxfs, OS is RedHat 4.0.
Is there are any way to avoid that?
Thanks,
Andrey
_______________________________________________
Veritas-ha maillist - Veritas-ha@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-ha
_______________________________________________
Veritas-ha maillist - Veritas-ha@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-ha
_______________________________________________
Veritas-ha maillist - Veritas-ha@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-ha
