Re: 2.6.23.1: mdadm/raid5 hung/d-state
Neil Brown wrote:
On Sunday November 4, [EMAIL PROTECTED] wrote:
# ps auxww | grep D
USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND
root 273 0.0 0.0 0 0 ?DOct21 14:40 [pdflush]
root 274 0.0 0.0 0 0 ?DOct21 13:00 [pdflush]
After several days/weeks, this is the second time this has happened, while
doing regular file I/O (decompressing a file), everything on the device
went into D-state.
At a guess (I haven't looked closely) I'd say it is the bug that was
meant to be fixed by
commit 4ae3f847e49e3787eca91bced31f8fd328d50496
except that patch applied badly and needed to be fixed with
the following patch (not in git yet).
These have been sent to stable@ and should be in the queue for 2.6.23.2
My linux-2.6.23/drivers/md/raid5.c contains your patch for a long time :
...
spin_lock(sh-lock);
clear_bit(STRIPE_HANDLE, sh-state);
clear_bit(STRIPE_DELAYED, sh-state);
s.syncing = test_bit(STRIPE_SYNCING, sh-state);
s.expanding = test_bit(STRIPE_EXPAND_SOURCE, sh-state);
s.expanded = test_bit(STRIPE_EXPAND_READY, sh-state);
/* Now to look around and see what can be done */
/* clean-up completed biofill operations */
if (test_bit(STRIPE_OP_BIOFILL, sh-ops.complete)) {
clear_bit(STRIPE_OP_BIOFILL, sh-ops.pending);
clear_bit(STRIPE_OP_BIOFILL, sh-ops.ack);
clear_bit(STRIPE_OP_BIOFILL, sh-ops.complete);
}
rcu_read_lock();
for (i=disks; i--; ) {
mdk_rdev_t *rdev;
struct r5dev *dev = sh-dev[i];
...
but it doesn't fix this bug.
Regards,
JKB
-
To unsubscribe from this list: send the line unsubscribe linux-raid in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
man mdadm - suggested correction.
Hello, I did read 'man mdadm' from top to bottom, but I totally forgot to look into /usr/share/doc/mdadm ! And there is much more - FAQs, recipes, etc! Can you please add to the manual under 'SEE ALSO' a reference to /usr/share/doc/mdadm ? thanks :-) -- Janek Kozicki | - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
man mdadm - suggested correction.
Hello, I did read 'man mdadm' from top to bottom, but I totally forgot to look into /usr/share/doc/mdadm ! And there is much more - FAQs, recipes, etc! Can you please add do the manual under 'SEE ALSO' a reference to /usr/share/doc/mdadm ? thanks :-) -- Janek Kozicki | - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: man mdadm - suggested correction.
Janek Kozicki said: (by the date of Mon, 5 Nov 2007 11:58:15 +0100) I did read 'man mdadm' from top to bottom, but I totally forgot to look into /usr/share/doc/mdadm ! PS: this why I asked so much questions on this list ;-) -- Janek Kozicki | - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: man mdadm - suggested correction.
Janek Kozicki wrote: [] Can you please add do the manual under 'SEE ALSO' a reference to /usr/share/doc/mdadm ? /usr/share/doc/mdadm is Debian-specific (well.. not sure it's really Debian (or something derived from it) -- some other distros may use the same naming scheme, too). Other distributions may place the files into a different directory, or not ship them at all, or ship them in alternative package. In any case, say, on Debian a user always knows that other misc. docs are in /usr/share/doc/$package - even if no other links are provided in the manpage. Users familiar with other distributions knows where/how to find other docs there. /mjt - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 2.6.23.1: mdadm/raid5 hung/d-state
On 11/4/07, Justin Piszcz [EMAIL PROTECTED] wrote: On Mon, 5 Nov 2007, Neil Brown wrote: On Sunday November 4, [EMAIL PROTECTED] wrote: # ps auxww | grep D USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND root 273 0.0 0.0 0 0 ?DOct21 14:40 [pdflush] root 274 0.0 0.0 0 0 ?DOct21 13:00 [pdflush] After several days/weeks, this is the second time this has happened, while doing regular file I/O (decompressing a file), everything on the device went into D-state. At a guess (I haven't looked closely) I'd say it is the bug that was meant to be fixed by commit 4ae3f847e49e3787eca91bced31f8fd328d50496 except that patch applied badly and needed to be fixed with the following patch (not in git yet). These have been sent to stable@ and should be in the queue for 2.6.23.2 Ah, thanks Neil, will be updating as soon as it is released, thanks. Are you seeing the same md thread takes 100% of the CPU that Joël is reporting? - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 2.6.23.1: mdadm/raid5 hung/d-state
On Mon, 5 Nov 2007, Dan Williams wrote: On 11/4/07, Justin Piszcz [EMAIL PROTECTED] wrote: On Mon, 5 Nov 2007, Neil Brown wrote: On Sunday November 4, [EMAIL PROTECTED] wrote: # ps auxww | grep D USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND root 273 0.0 0.0 0 0 ?DOct21 14:40 [pdflush] root 274 0.0 0.0 0 0 ?DOct21 13:00 [pdflush] After several days/weeks, this is the second time this has happened, while doing regular file I/O (decompressing a file), everything on the device went into D-state. At a guess (I haven't looked closely) I'd say it is the bug that was meant to be fixed by commit 4ae3f847e49e3787eca91bced31f8fd328d50496 except that patch applied badly and needed to be fixed with the following patch (not in git yet). These have been sent to stable@ and should be in the queue for 2.6.23.2 Ah, thanks Neil, will be updating as soon as it is released, thanks. Are you seeing the same md thread takes 100% of the CPU that Joël is reporting? Yes, in another e-mail I posted the top output with md3_raid5 at 100%. Justin.
Kernel Module - Raid
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 All, Hello. I don't know if this is the right place to post this issue but it does deal with RAID so I thought I would try. I successfully built a new kernel and am able to boot from it. However, I need to incorporate a specific RAID driver (adpahci.ko) so we can use the on-board RAID. The reason I am needing this is because we have it working with one kernel (the one it was compiled on) but we can't just copy the adpahci.ko and incorporate it into another kernel because the kernel complains. I have the adpahci.ko and am unable to get it to compile against any other kernel because I don't have the original kernel module (adpahci.c I assume is what I need). Is there any way I can view the adpahci.ko and copy the contents to make a adpahci.c? Is it possible to get the kernel object to compile with another kernel only using the adpahci.ko? Am I making sense? :) Thanks in advance. /paul -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHL3rtul+dQkHY548RAtIDAJoDZWME/Hku9QCESFW/yo/YG3WE5wCfeHPb fzs7FpKbhSr8PUbl2xrCfGA= =Lcdq -END PGP SIGNATURE- - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel Module - Raid
Paul VanGundy wrote: All, Hello. I don't know if this is the right place to post this issue but it does deal with RAID so I thought I would try. It deals primarily with linux *software* raid. But stick with it - you may end up doing that... What hardware/distro etc are you using? Is this an expensive (hundreds of £) card? Or an onboard/motherboard chipset? Once you answer this then it may be worth suggesting using sw-raid (in which case we can help out) or pointing you elsewhere... I successfully built a new kernel and am able to boot from it. However, I need to incorporate a specific RAID driver (adpahci.ko) so we can use the on-board RAID. I think this is the adaptec proprietary code - in which case you may need a very specific kernel to run it. You may find others on here who can help but you'll probably need an Adaptec forum/list. I have the adpahci.ko and am unable to get it to compile against any other kernel because I don't have the original kernel module (adpahci.c I assume is what I need). Is there any way I can view the adpahci.ko and copy the contents to make a adpahci.c? No Is it possible to get the kernel object to compile with another kernel only using the adpahci.ko? No Am I making sense? :) Yes That's one of the big reasons proprietary drivers suck on linux. David - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel Module - Raid
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Thanks for the prompt replay David. Below are the answers to your questions: Hello. I don't know if this is the right place to post this issue but it does deal with RAID so I thought I would try. It deals primarily with linux *software* raid. But stick with it - you may end up doing that... What hardware/distro etc are you using? Is this an expensive (hundreds of £) card? Or an onboard/motherboard chipset? The distro is Suse 10.1. It is an onboard chipset. Once you answer this then it may be worth suggesting using sw-raid (in which case we can help out) or pointing you elsewhere... I successfully built a new kernel and am able to boot from it. However, I need to incorporate a specific RAID driver (adpahci.ko) so we can use the on-board RAID. I think this is the adaptec proprietary code - in which case you may need a very specific kernel to run it. You may find others on here who can help but you'll probably need an Adaptec forum/list. Oi. This is a pain... ;) I have the adpahci.ko and am unable to get it to compile against any other kernel because I don't have the original kernel module (adpahci.c I assume is what I need). Is there any way I can view the adpahci.ko and copy the contents to make a adpahci.c? No That sucks. :) Is it possible to get the kernel object to compile with another kernel only using the adpahci.ko? No Am I making sense? :) Yes Good, because sometimes I don't make sense to myself. :) That's one of the big reasons proprietary drivers suck on linux. Ok. So this chipset has the ability to use an Intel based RAID. Would that be better? Thanks again. /paul -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHL4QOul+dQkHY548RAgF7AJ49tcLXtEp9DgZugK1ozhBNyP0tOgCeNygd HVtJmjZR8qYm7+7lCwH7gLI= =EQwg -END PGP SIGNATURE- - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Kernel Module - Raid
Paul VanGundy wrote: Thanks for the prompt replay David. Below are the answers to your questions: What hardware/distro etc are you using? Is this an expensive (hundreds of £) card? Or an onboard/motherboard chipset? The distro is Suse 10.1. As a bit of trivia, Neil (who wrote and maintains linux RAID) works for Suse. It is an onboard chipset. In which case it's not likely to be hardware RAID. See: http://linux-ata.org/faq-sata-raid.html Once you answer this then it may be worth suggesting using sw-raid (in which case we can help out) or pointing you elsewhere... You should probably configure the BIOS to use That's one of the big reasons proprietary drivers suck on linux. Ok. So this chipset has the ability to use an Intel based RAID. Would that be better? mmm, see the link above... In almost any case where you are considering 'onboard' raid, linux software raid (using md and mdadm) is a better choice. Start here: http://linux-raid.osdl.org/index.php/Main_Page (feel free to correct it or ask here for clarification) Also essential reading is the mdadm man page. David - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 2.6.23.1: mdadm/raid5 hung/d-state
On 11/5/07, Justin Piszcz [EMAIL PROTECTED] wrote: [..] Are you seeing the same md thread takes 100% of the CPU that Joël is reporting? Yes, in another e-mail I posted the top output with md3_raid5 at 100%. This seems too similar to Joël's situation for them not to be correlated, and it shows that iscsi is not a necessary component of the failure. The attached patch allows the debug statements in MD to be enabled via sysfs. Joël, since it is easier for you to reproduce can you capture the kernel log output after the raid thread goes into the spin? It will help if you have CONFIG_PRINTK_TIME=y set in your kernel configuration. After the failure run: echo 1 /sys/block/md_d0/md/debug_print_enable; sleep 5; echo 0 /sys/block/md_d0/md/debug_print_enable ...to enable the print messages for a few seconds. Please send the output in a private message if it proves too big for the mailing list. raid5-debug-print-enable.patch Description: Binary data
question about mdadm + grub interaction
Greetings. In order to insure that a Debian stock kernel (i.e. the kernel installed from the linux-image-2.6.22-2-686-bigmem package) boots correctly off of a mdadm RAID 1 set of two disks if one of the disks is dead, do we: 1) Have to manually install grub on the MBR on *both* drives, or is this done automagically by the grub package installer if the RAID 1 set is in place at the time of the grub install? 2) Need to have something that looks like this in /boot/grub/menu.lst: A) title Debian GNU/Linux, kernel 2.6.21-2-686-bigmem, RAID1 root(hd0,0) kernel /vmlinuz-2.6.21-2-686-bigmem root=/dev/md2 md=0,/dev/sda1,/dev/sdb1 ro initrd /initrd.img-2.6.21-2-686-bigmem savedefault title Debian GNU/Linux, kernel 2.6.21-2-686-bigmem, BROKEN RAID1 root(hd0,0) kernel /vmlinuz-2.6.21-2-686-bigmem root=/dev/md2 md=0,/dev/sdb1 ro initrd /initrd.img-2.6.21-2-686-bigmem savedefault i.e. with the 'md=...' configuration options? Or do we just need this: B) title Debian GNU/Linux, kernel 2.6.21-2-686-bigmem root(hd0,0) kernel /vmlinuz-2.6.21-2-686-bigmem root=/dev/md2 ro initrd /initrd.img-2.6.21-2-686-bigmem savedefault which is the sort of thing that is installed automagically by the kernel package installer? 3) Need to have the 'md=...' configuration option point at the *boot* partition, or the *root* partition, so that if we have: md0 : active raid1 sda1[0] sdb1[1] md2 : active raid1 sda3[0] sdb3[1] and /dev/md0 /boot /dev/md2 / do we then need: md=0,/dev/sda1,/dev/sdb1 md=0,/dev/sdb1 or: md=2,/dev/sda3,/dev/sdb3 md=2,/dev/sdb3 Any pointers to *detailed* documentation about how the 'md=...' option works would be much appreciated; I've already seen: http://www.debian-administration.org/articles/238 (which has a question at the very bottom of the page, a bit over two months old, asking very similar sorts of questions -- no answer on the page so far...); and http://alephnull.net/software/linux-raid/4%20-%20Setup.html (which is lilo-specific, but also informative); and http://www.epimetrics.com/topics/one-page?page_id=449topic=Bit-head%20Stuffpage_topic_id=120 AND: http://www.opensubscriber.com/message/linux-raid@vger.kernel.org/7889881.html (which is informative, but doesn't get at precise answers to my questions). Thanks much, -- Steve Lane System, Network and Security Administrator Doudna Lab Biomolecular Structure and Mechanism Group UC Berkeley - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Software raid - controller options
Hello, I just started looking into software raid with linux a few weeks ago. I am outgrowing the commercial NAS product that I bought a while back. I've been learning as much as I can, suscribing to this mailing list, reading man pages, experimenting with loopback devices setting up and expanding test arrays. I have a few questions now that I'm sure someone here will be able to enlighten me about. First, I want to run a 12 drive raid 6, honestly, would I be better of going with true hardware raid like the areca ARC-1231ML vs software raid? I would prefer software raid just for the sheer cost savings. But what kind of processing power would it take to match or exceed a mid to high-level hardware controller? I haven't seen much, if any, discussion of this, but how many drives are people putting into software arrays? And how are you going about it? Motherboards seem to max out around 6-8 SATA ports. Do you just add SATA controllers? Looking around on newegg (and some googling) 2-port SATA controllers are pretty easy to find, but once you get to 4 ports the cards all seem to include some sort of built in *raid* functionality. Are there any 4+ port PCI-e SATA controllers cards? Are there any specific chipsets/brands of motherboards or controller cards that you software raid veterans prefer? Thank you for your time and any info you are able to give me! Lyle - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Software raid - controller options
On Tue, 2007-11-06 at 15:51 +1300, Richard Scobie wrote: Lyle Schlueter wrote: Are there any 4+ port PCI-e SATA controllers cards? Hi Lyle, I've been doing a similar exercise here and have been looking at portmultiplier options using the Silicon Image 3124. Is a port multiplier a decent option? I looked at the 3124 after you mentioned it and a few of the other controllers offered by Silicon Image. I had been looking at the Adaptec 2240900-R PCI Express and HighPoint RocketRAID 2300 PCI Express. These are both *raid* cards. But if they can be used as a regular controller card, they both provide 4 SATA ports and are PCI-e. But sounds like the RocketRAID doesn't work with the 2.6.22+ kernel (according to newegg reviewers). It sounds like the Adaptec works quite nicely though. The other possibility is the Marvell based 8 port dumb SATA controller from Supermicro. http://www.supermicro.com/products/accessories/addon/AoC-SAT2-MV8.cfm It is PCI-X though, but there are plenty of boards around still with these slots. My only concern here is the opening comment in the driver: 1) Needs a full errata audit for all chipsets. I implemented most of the errata workarounds found in the Marvell vendor driver, but I distinctly remember a couple workarounds (one related to PCI-X) are still needed. Sounds pretty iffy there. That Adaptec card I mentioned is going for about 100 USD. Seems like a lot for 4 ports. But sounds like it works nicely with linux, and I would only need 1 or 2 of them (plus the 6 or 8 ports on the motherboard) to be able to use all 12 drives. Do you know of any concerns of using all the ports on a motherboard? Slowdowns or anything like that? Enquiries here previously have not found anyone using this card. Regards, Richard - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Thanks, Lyle - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Software raid - controller options
You've probably missed a discussion on issues I've been having with SATA, software RAID and bad drivers. A clear thing from the responses I got is that you really need to use a recent kernel, as they may have fixed those problems. I didn't get clear responses indicating specific cards that are known to work well when hardrives fail. But if you can deal with a server crashing and then rebooting manually then software RAID is the way to go. I've always been able to get the servers back online even with the problematic drivers. I am happy with the 3ware cards and do use their hardware RAID to avoid the problems that I've had. With those I've fully tested 16 drive systems with 2 arrays using 2 8-port cards. Others have recommended the Areca line. As for cheap dumb interfaces I am now using the RocketRAID 2220, which gives you 8 ports on a PCI-X. I believe the built in RAID on those is just firmware based so you may as well use them to show the drives in normal/legacy mode and use software RAID on top. Keep in mind I haven't fully tested this solution nor have tested for proper functioning when a drive fails. Another inexpensive card I've used with good results is the Q-stor PCI-X card, but I think this is now obsolete. Hope this helps, Alberto On Tue, 2007-11-06 at 05:20 +0300, Lyle Schlueter wrote: Hello, I just started looking into software raid with linux a few weeks ago. I am outgrowing the commercial NAS product that I bought a while back. I've been learning as much as I can, suscribing to this mailing list, reading man pages, experimenting with loopback devices setting up and expanding test arrays. I have a few questions now that I'm sure someone here will be able to enlighten me about. First, I want to run a 12 drive raid 6, honestly, would I be better of going with true hardware raid like the areca ARC-1231ML vs software raid? I would prefer software raid just for the sheer cost savings. But what kind of processing power would it take to match or exceed a mid to high-level hardware controller? I haven't seen much, if any, discussion of this, but how many drives are people putting into software arrays? And how are you going about it? Motherboards seem to max out around 6-8 SATA ports. Do you just add SATA controllers? Looking around on newegg (and some googling) 2-port SATA controllers are pretty easy to find, but once you get to 4 ports the cards all seem to include some sort of built in *raid* functionality. Are there any 4+ port PCI-e SATA controllers cards? Are there any specific chipsets/brands of motherboards or controller cards that you software raid veterans prefer? Thank you for your time and any info you are able to give me! Lyle - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html -- Alberto AlonsoGlobal Gate Systems LLC. (512) 351-7233http://www.ggsys.net Hardware, consulting, sysadmin, monitoring and remote backups - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
