Dana H. Myers wrote:
Torrey McMahon wrote:
Dana H. Myers wrote:
Ed Gould wrote:
On Jan 26, 2007, at 12:13, Richard Elling wrote:
On Fri, Jan 26, 2007 at 11:05:17AM -0800, Ed Gould wrote:
A number that I've been quoting, albeit without a good reference,
comes from Jim Gray, who has been around the data-management industry
for longer than I have (and I've been in this business since 1970);
he's currently at Microsoft. Jim says that the controller/drive
subsystem writes data to the wrong sector of the drive without notice
about once per drive per year. In a 400-drive array, that's once a
day. ZFS will detect this error when the file is read (one of the
blocks' checksum will not match). But it can only correct the error
if it manages the redundancy.
Actually, Jim was referring to everything but the trunk. He didn't
specify where from the HBA to the drive the error actually occurs. I
don't think it really matters. I saw him give a talk a few years ago at
the Usenix FAST conference; that's where I got this information.
So this leaves me wondering how often the controller/drive subsystem
reads data from the wrong sector of the drive without notice; is it
symmetrical with respect to writing, and thus about once a drive/year,
or are there factors which change this?
It's not symmetrical. Often times its a fw bug. Others a spurious event
causes one block to be read/written instead of an other one. (Alpha
particles anyone?)
I would tend to expect these spurious events to impact read and write
equally; more specifically, the chance of any one read or write being
mis-addressed is about the same. Since, AFAIK, there are many more reads
from a disk typically than writes, this would seem to suggest that there
would be more mis-addressed reads in a drive/year than mis-addressed
writes. Is this the reason for the asymmetry?
(I'm sure waving my hands here)
For the spurious events, yes, I would expect things to be impacted
symmetrically depending when it comes to errors during reads and errors
during writes. That is if you could figure out what spurious event
occurred. In most cases the spurious errors are caught only at read time
and you're left wondering. Was it an incorrect read? Was the data
written incorrectly? You end up throwing your hands up and saying, "Lets
hope that doesn't happen again." It's much easier to unearth a fw bug in
a particular disk drive operating in certain conditions and fix it.
Now that we're checksumming things I'd expect to find more errors, and
hopefully be in a condition to fix them, then we have in the past. We
will also start getting customer complaints like, "We moved to ZFS and
now we are seeing media errors more often. Why is ZFS broken?" This is
similar to the StorADE issues we had in NWS - Ahhh, the good old days -
when we started doing a much better job discovering issues and reporting
them when in the past we were blissfully silent. We used to have some
data on that with nice graphs but I can't find them lying about.
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
