Re: raid5:md3: read error corrected , followed by , Machine Check Exception: .
On Sat, 14 Jul 2007 17:08:27 -0700 (PDT) "Mr. James W. Laferriere" <[EMAIL PROTECTED]> wrote: > Hello All , I was under the impression that a 'machine check' would be > caused by some near to the CPU hardware failure , Not a bad disk ? It indicates a hardware failure > Jul 14 23:00:26 filesrv2 kernel: sd 2:0:1:0: SCSI error: return code = > 0x0802 > Jul 14 23:00:26 filesrv2 kernel: sdd: Current: sense key: Medium Error > Jul 14 23:00:26 filesrv2 kernel: Additional sense: Read retries exhausted So your disk throws a fit > Jul 14 23:17:06 filesrv2 kernel: raid5:md3: read error corrected (8 sectors > at 72269184 on sdd) > Jul 14 23:17:06 filesrv2 kernel: raid5:md3: read error corrected (8 sectors > at 72269192 on sdd) Raid happily recovers the mess > Jul 14 23:20:28 filesrv2 kernel: raid5:md3: read error corrected (8 sectors > at 76388336 on sdd) > Jul 14 23:20:28 filesrv2 kernel: raid5:md3: read error corrected (8 sectors > at 76388344 on sdd) > Jul 14 23:38:48 filesrv2 -- MARK -- > CPU 5: Machine Check Exception: 0004 > CPU 4: Machine Check Exception: 0004 And at some point at least 18 minutes after the raid incident you log CPU problems. Are you sure the box or the room it is in didn't get excessively hot ? - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Linux Software RAID is really RAID?
> > A hard(ware) lockup, not software. > > That's why Intel says ICH5 doesn't do hotplug. > > OIC. I don't think there's much left to do from the driver side then. > Or is there any workaround? I'm not familiar with the ICH5 SATA side but on the PATA side we also need to run code to fix up chips on a hotplug (and *before* the plug in some cases) - eg the ICH2 requires we turn off IORDY. - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Linux-usb-users] Failed reads from RAID-0 array (from newbie who has read the FAQ)
On Mon, 19 Mar 2007, Michael Schwarz wrote: > Yeah; But here was where I lacked confidence. I used to know every inch of > my kernel and my hardware, but, as previously stated, that was back in the > 2.2.x days. I wasn't confident that I could run my hardware with a > plain-vanilla kernel or that I could successfully roll my own working > 2.6.x kernel in a timely manner. But, of course, I understand why this is > a good idea. It's not so hard to do, if you start from a known-good configuration. For instance, you could take the config your current distribution's kernel is built from and just use it, although it would take a long time to build because it includes so many drivers. Whittling it down to just the drivers you need would be tedious but not very difficult. Alan Stern - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Linux-usb-users] Failed reads from RAID-0 array (from newbie who has read the FAQ)
On Mon, 19 Mar 2007, Michael Schwarz wrote: > I'm going to hang on to the hardware. This is a pilot/demo that may lead > to development of a new device, and, if so, I'll be getting back into > device driver writing. Working this problem would be great practice for > that. So I will do it. The only problem is I don't know when! > > I believe I can replicate the problem, so I'll find time (perhaps next > weekend) to capture the data of interest. Michael, you don't seem to appreciate the basic principles for tracking down problems. First: Simplify. Get rid of everything that isn't relevant to the problem and could serve to distract you. In particular, don't run X. That will eliminate around half of your running processes and shrink the stack dump down so that it might fit in the kernel buffer without overflowing. Second: Simplify. Don't run kernels that have been modified by Fedora or anybody else. Use a plain vanilla kernel from kernel.org. Third: Simplify. Try not to collect the same data over and over again (take a look at the starts of all those dmesg files you compressed and emailed). You can clear the kernel's log buffer after dumping it by doing "dmesg -c >/dev/null". Fourth: Be prepared to make changes. This means making changes to the kernel configuration or source code, another reason for using a stock kernel. To get some really useful data, you need to build a kernel with CONFIG_USB_DEBUG turned on. Without that setting there won't be any helpful debugging information in the log. Then you should run a minimal system. Single-user mode would be best, but that can be _too_ bare-bones. No GUI will suffice. Then you should clear the kernel log before before starting the big file copy. Basically nothing that happens before then is important, because nothing has gone wrong. Then after the hang occurs, see what shows up in the dmesg log. And get a stack dump. > Mr. Stern: Where might I go for low level programming information on USB > devices? I'm interested in registers/DMA/packet formats, etc. Are you interested in USB devices (i.e., flash drives, webcams, and so on -- the things you plug in to a USB connection) or USB controllers (the hardware in your computer that manages the USB bus)? > I've found info on the USB protocol itself, but I haven't found info on > devices. Obviously I can dig through kernel source, but documents would be > nice! Again, if this is an unreasonable request for you to "do my > homework," just say so! I won't be offended. I'm sure I can find it myself > given time, but if you happen to have some URLs handy, they'd be > appreciated. There are three types of USB controllers used in personal computers: UHCI, OHCI, and EHCI. Links to their specifications are available here: http://www.usb.org/developers/resources/ Specifications for various classes of USB devices are available here: http://www.usb.org/developers/devclass_docs Alan Stern - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Linux-usb-users] Failed reads from RAID-0 array (from newbie who has read the FAQ)
On Sat, 17 Mar 2007, Michael Schwarz wrote: > Nasty big stack trace set follows: This format is kind of awkward. For one thing, a lot of lines were wrapped by your email program. For another, you copied the stack trace from the syslog log file. That is not a good way to do it; syslogd is liable to miss bits and pieces of the kernel log when a lot of information comes along all at once. You're much better off getting the stack trace data directly from dmesg. (And when you do, you don't end up with 30 columns of wasted data added to the beginning of each line.) Alan Stern - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Linux-usb-users] Failed reads from RAID-0 array (from newbie who has read the FAQ)
On Sat, 17 Mar 2007, Michael Schwarz wrote: > Comments/questions below... > > -- > Michael Schwarz > > > This isn't much help. The important processes here are khubd, > > usb-storage, and scsi_eh_*. Possibly some raid-related processes too, but > > I don't know which they would be. > > I have no copy khubd running. That in itself is a very bad sign. You need to look at the dmesg log. Alan Stern - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Linux-usb-users] Failed reads from RAID-0 array (from newbie who has read the FAQ)
On Sat, 17 Mar 2007, Michael Schwarz wrote: > Neil: > > Relevant stack trace follows. Any suggestions? blk_backing_dev_unplug... > Does that mean the raid subsystem thinks one of the usb drives has been > removed? I assure you that physically this is untrue, but that doesn't > mean that some sort logical disconnect hasn't happened... > > Makes me wonder if one of my USB hub connections is intermittent... > > I would also welcome any tips on any other developers group to follow up > with. I haven't hacked any kernel code since the 2.2.x kernel and things > have changed a bit! I don't mind digging into this, but I suspect I could > get things cleared up fast if I could find the right subject expert! > > > > === > cpD E2FBEDB0 1784 4271 4270 (NOTLB) >e2fbedb4 00200086 c15dc550 e2fbedb0 0001 00200082 1000 > > c15dc550 000a e94182b0 f3161430 26320f40 01c5 > >e94183bc c1c8c480 ecd7d300 c04e0bf2 c042e0e4 f7d767f8 > 003b6622 > Call Trace: > [] blk_backing_dev_unplug+0x73/0x7b > [] getnstimeofday+0x30/0xb6 > [] io_schedule+0x3a/0x5c > [] sync_page+0x0/0x3b > [] sync_page+0x38/0x3b > [] __wait_on_bit_lock+0x2a/0x52 > [] __lock_page+0x58/0x5e > [] wake_bit_function+0x0/0x3c > [] do_generic_mapping_read+0x1e0/0x459 > [] generic_file_aio_read+0x173/0x1a6 > [] file_read_actor+0x0/0xe0 > [] do_sync_read+0xc7/0x10a > [] autoremove_wake_function+0x0/0x35 > [] do_sync_read+0x0/0x10a > [] vfs_read+0xa6/0x152 > [] sys_read+0x41/0x67 > [] syscall_call+0x7/0xb > === This isn't much help. The important processes here are khubd, usb-storage, and scsi_eh_*. Possibly some raid-related processes too, but I don't know which they would be. It also would help a lot to see your dmesg log. Especially if you would build your kernel with CONFIG_USB_DEBUG turned on. > Update: > > (For those who've been waiting breathlessly). It hangs at a particular > point in a particular file. In other words, it doesn't depend on the total > number of bytes transfered. Rather, when it reaches a particular point in > a particular file (12267520 bytes into a file that is 1073709056 bytes > long) it hangs. > > I begin to suspect that I have a "dead spot" in my USB hub. But what gets > me if that is true is why does the write work? Do cp and dd not check to > see if writes succeed? Depends what you mean. They do check the return codes from the underlying device drivers, but they don't try to read the data back to make sure it really was written. > I know it isn't a particular flash drive because I've used two different > sets of 7 USB drives and it seems to fail consistently no matter which. But you haven't tried using different hubs, different USB cables, or different computers. > Nonetheless, I'm beginning to think I'm dealing with a hardware issue, not > a kernel issue, just because it is so consistent. People have reported problems in which the hardware fails when it encounters a certain pattern of bytes in the data stream. Maybe you're seeing the same sort of thing. Alan Stern - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: end to end error recovery musings
> Not sure you're up-to-date on the T10 data integrity feature. > Essentially it's an extension of the 520 byte sectors common in disk I saw the basics but not the detail. Thanks for the explanation it was most helpful and promises to fix a few things for some controllers.. but here's a minor bit of passing bad news - quite a few older ATA controllers can't issue DMA transfers that are not a multiple of 512 bytes without crapping themselves (eg READ_LONG). Guess we may need to add ap->i_do_not_suck or similar 8) On the bright side I believe the Intel ICH is the only one with this problem (and a workaround) which is SATA capable 8) Alan - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: end to end error recovery musings
> These features make the most sense in terms of WRITE. Disks already > have plenty of CRC on the data so if a READ fails on a regular drive > we already know about it. Don't bet on it. If you want to do this seriously you need an end to end (media to host ram) checksum. We do see bizarre and quite evil things happen to people occasionally because they rely on bus level protection - both faulty network cards and faulty disk or controller RAM can cause very bad things to happen in a critical environment and are very very hard to detect and test for. IDE has another hideously evil feature in this area. Command blocks are sent by PIO cycles, and are therefore unprotected from corruption. So while a data burst with corruption will error and retry and command which corrupts the block number although very very much less likely (less bits and much lower speed) will not be caught on a PATA system for read or for write and will hit the wrong block. With networking you can turn off hardware IP checksumming (and many cluster people do) with disks we don't yet have a proper end to end checksum to media system in the fs or block layers. > It would be great if the app tag was more than 16 bits. Ted mentioned > that ideally he'd like to store the inode number in the app tag. But > as it stands there isn't room. The lowest few bits are the most important with ext2/ext3 because you normally lose a sector of inodes which means you've got dangly bits associated with a sequence of inodes with the same upper bits. More problematic is losing indirect blocks, and being able to keep some kind of [inode low bits/block index] would help put stuff back together. Alan - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: end to end error recovery musings
> One interesting counter example is a smaller write than a full page - say 512 > bytes out of 4k. > > If we need to do a read-modify-write and it just so happens that 1 of the 7 > sectors we need to read is flaky, will this "look" like a write failure? The current core kernel code can't handle propogating sub-page sized errors up to the file system layers (there is nowhere in the page cache to store 'part of this page is missing'). This is a long standing (four year plus) problem with CD-RW support as well. For ATA we can at least retrieve the true media sector size now, which may be helpful at the physical layer but the page cache would need to grow some brains to do anything with it. - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: end to end error recovery musings
> I think that this is mostly true, but we also need to balance this against > the > need for higher levels to get a timely response. In a really large IO, a > naive > retry of a very large write could lead to a non-responsive system for a very > large time... And losing the I/O could result in a system that is non responsive until the tape restore completes two days later - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: end to end error recovery musings
> the new location. I believe this should be always true, so presumably > with all modern disk drives a write error should mean something very > serious has happend. Not quite that simple. If you write a block aligned size the same size as the physical media block size maybe this is true. If you write a sector on a device with physical sector size larger than logical block size (as allowed by say ATA7) then it's less clear what happens. I don't know if the drive firmware implements multiple "tails" in this case. On a read error it is worth trying the other parts of the I/O. Alan - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: sata badness in 2.6.20-rc1? [Was: Re: md patches in -mm]
On Fri, 15 Dec 2006 13:39:27 -0800 Andrew Morton <[EMAIL PROTECTED]> wrote: > On Fri, 15 Dec 2006 13:05:52 -0800 > Andrew Morton <[EMAIL PROTECTED]> wrote: > > > Jeff, I shall send all the sata patches which I have at you one single time > > and I shall then drop the lot. So please don't flub them. > > > > I'll then do a rc1-mm2 without them. > > hm, this is looking like a lot of work for not much gain. Rafael, are > you able to do a quick chop and tell us whether these: The md one and the long history of reports about parallel I/O causing problems sounds a lot more like the kmap stuff you were worried about Andrew. I'd be very intereste dto know if it happens on x86_32 built with a standard memory split and no highmem - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] dmaengine: clean up and abstract function types (was Re: [PATCH 08/19] dmaengine: enable multiple clients and operations)
Ar Llu, 2006-09-18 am 20:05 -0500, ysgrifennodd Olof Johansson:
> On Mon, 18 Sep 2006 15:56:37 -0700 "Dan Williams" <[EMAIL PROTECTED]> wrote:
> op.src_type = PG; op.src = pg;
> op.dest_type = BUF; op.dest = buf;
> op.len = len;
> dma_async_commit(chan, op);
At OLS Linus suggested it should distinguish between sync and async
events for locking reasons.
if(dma_async_commit(foo) == SYNC_COMPLETE) {
finalise_stuff();
}
else/* will call foo->callback(foo->dev_id) */
because otherwise you have locking complexities - the callback wants to
take locks to guard the object it works on but if it is called
synchronously - eg if hardware is busy and we fall back - it might
deadlock with the caller of dmaa_async_foo() who also needs to hold the
lock.
Alan
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
large copy to mdadm array fails, marks readonly
hello all, first message to this group. Ive recently created a 3 disk raid5 using mdadm. file copy to this array fails consistently after a few gigs. the array does not die completely, but rather it is marked readonly. I created the array on 3 disks of equal partition size with: sudo mdadm /dev/md0 --create -l5 --raid-devices=3 /dev/sda1 /dev/sdb1 /dev/sdc1 then formatted with: mkfs.ext3 /dev/md0 badblocks shows no error on any of the drives. [EMAIL PROTECTED]:~$ cat /proc/mdstat Personalities : [raid5] md0 : active raid5 sda1[0] sdb1[2] sdc1[1] 234436352 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU] unused devices: [EMAIL PROTECTED]:~$ dmesg [17180879.388000] EXT3 FS on md0, internal journal [17180879.392000] EXT3-fs: mounted filesystem with ordered data mode. [17181702.872000] EXT3-fs error (device md0): ext3_new_block: Allocating block in system zone - block = 38993924 [17181702.872000] Aborting journal on device md0. [17181702.952000] __journal_remove_journal_head: freeing b_committed_data [17181702.956000] EXT3-fs error (device md0) in ext3_prepare_write: Journal has aborted [17181702.96] ext3_abort called. [17181702.96] EXT3-fs error (device md0): ext3_journal_start_sb: Detected aborted journal [17181702.96] Remounting filesystem read-only im stumped, any ideas? thanks much, alan - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Linux: Why software RAID?
Ar Iau, 2006-08-24 am 07:31 -0700, ysgrifennodd Marc Perkel: > So - the bottom line answer to my question is that unless you are > running raid 5 and you have a high powered raid card with cache and > battery backup that there is no significant speed increase to use > hardware raid. For raid 0 there is no advantage. > If your raid is entirely on PCI plug in cards and you are doing RAID1 there is a speed up using hardware assisted raid because of the PCI bus contention. If your controllers are PCI express, on internal high speed busses (eg chipset controllers) or at least half of them are then generally there is no win with hardware raid 0/1 and it is often slower. - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Linux: Why software RAID?
Ar Iau, 2006-08-24 am 09:07 -0400, ysgrifennodd Adam Kropelin: > Jeff Garzik <[EMAIL PROTECTED]> wrote: > with sw RAID of course if the builder is careful to use multiple PCI > cards, etc. Sw RAID over your motherboard's onboard controllers leaves > you vulnerable. Generally speaking the channels on onboard ATA are independant with any vaguely modern card. And for newer systems well the motherboard tends to be festooned with random SATA controllers, all separate! - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 000 of 5] md: Introduction
On Mer, 2006-01-18 at 09:14 +0100, Sander wrote: > If the (harddisk internal) remap succeeded, the OS doesn't see the bad > sector at all I believe. True for ATA, in the SCSI case you may be told about the remap having occurred but its a "by the way" type message not an error proper. > If you (the OS) do see a bad sector, the disk couldn't remap, and goes > downhill from there, right? If a hot spare is configured it will be dropped into the configuration at that point. - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Where is the performance bottleneck?
* Holger Kiehl ([EMAIL PROTECTED]) wrote: > There is however one difference, here I had set > /sys/block/sd?/queue/nr_requests to 4096. Well from that it looks like none of the queues get about 255 (hmm that's a round number) > avg-cpu: %user %nice%sys %iowait %idle >0.100.00 21.85 58.55 19.50 Fair amount of system time. > Device:rrqm/s wrqm/s r/s w/s rsec/s wsec/srkB/swkB/s > avgrq-sz avgqu-sz await svctm %util > sdf11314.90 0.00 365.10 0.00 93440.000.00 46720.00 0.00 > 255.93 1.925.26 2.74 99.98 > sdg7973.20 0.00 257.20 0.00 65843.200.00 32921.60 0.00 > 256.00 1.947.53 3.89 100.01 There seems to be quite a spread of read performance accross the drives (pretty consistent accross the run); what makes sdg so much slower than sdf (which seems to be the slowest and fastest drives respectively). I guess if everyone was running at sdf's speed you would be pretty happy. If you physically swap f and g does the performance follow the drive or the letter? Dave -- -Open up your eyes, open up your mind, open up your code --- / Dr. David Alan Gilbert| Running GNU/Linux on Alpha,68K| Happy \ \ gro.gilbert @ treblig.org | MIPS,x86,ARM,SPARC,PPC & HPPA | In Hex / \ _|_ http://www.treblig.org |___/ - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Where is the performance bottleneck?
* Holger Kiehl ([EMAIL PROTECTED]) wrote: > On Wed, 31 Aug 2005, Jens Axboe wrote: > > Full vmstat session can be found under: Have you got iostat? iostat -x 10 might be interesting to see for a period while it is going. Dave -- -Open up your eyes, open up your mind, open up your code --- / Dr. David Alan Gilbert| Running GNU/Linux on Alpha,68K| Happy \ \ gro.gilbert @ treblig.org | MIPS,x86,ARM,SPARC,PPC & HPPA | In Hex / \ _|_ http://www.treblig.org |___/ - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] RAID5 NULL Checking Bug Fixt
> On Wednesday May 16, [EMAIL PROTECTED] wrote: > > > > (more patches to come. They will go to Linus, Alan, and linux-raid only). > > This is the next one, which actually addresses the "NULL Checking > Bug". Thanks. As Linus merges I'll switch over to match his tree. Less diff is good 8) - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED]
Re: Proposed RAID5 design changes.
> Umm. Isn't RAID implemented as the md device? That implies that it is > responsible for some kind of error management. Bluntly, the file systems > don't declare a file system kaput until they've retried the critical > I/O operations. Why should RAID5 be any less tolerant? File systems give up the moment they get an error. The actual device driver does the retries. This is done because only the driver or things like the scsi layer know about the right retry procedures for a given device - eg for a smart raid controller retrying is done by the card - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED]
Re: Proposed RAID5 design changes.
> any data, but under normal default drive setup the sector will not be > reallocated. If testing the failing sector is too much effort, a > simple overwrite with the corrected data, at worst, improves the > chances of the drive firmware being able to reallocate the sector. > This works just fine when you have complete faith in error detection > and correction. Agreed - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED]
Re: Proposed RAID5 design changes.
> > 1) Read and write errors should be retried at least once before kicking > >the drive out of the array. > > This doesn't seem unreasonable on the face of it. Device level retries are the job of the device level driver > > 2) On more persistent read errors, the failed block (or whatever unit is > > Generally, it is a good idea to try to rewrite (assuming the original data > can be recovered via another method) as sector. If we were really good, we > would run a short (changing random) pattern test on the sector before the > rewrite to determine whether sparing would be a good idea. Micro defects or Twenty years ago maybe. Nowdays your drive does all that for you and a rewrite will cause a block realloc from the spare list - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [EMAIL PROTECTED]
