Measuring RAID performance
Hi! I am currently experimenting with the Software RAID (patch for 2.2.11 applied to 2.2.12 with raidtools snapshot from about three weeks ago). The system I am doing these tests on is an older K5-100 with 64 MB RAM. For RAID, I have three Quantum Atlas I on a single Fast SCSI Host Adapter with NCR Chipset. The system itself is installed on a fourth disk that isn't included in the RAID. The system disk is also used as a source disk for the copy tests. Main test is bonnie with a 300 MB file. For the copy tests, I use the file and directory structure of a Windows NT 4.0 installation CD because it has many small files. "Copy 1" is the time taken for "cp -R 1 /mnt/raid/1/". The source directory is on the system disk, the target directory on the RAID. This test is meant to check write rate of the RAID but will measure source disk performance instead because the source disk is likely to be the bottle neck here. "Copy 2" is the time taken for "cp -R 1 2" , with both source and target directory on the RAID array. I hope to simulate a realistic usage profile with many small files, read and write operations. "Copy 3" is the time taken for "tar cf - 1/* 2/* | cat /dev/null". I needed to use tar|cat because directories can't be copied to /dev/null and tar notices that it is writing to /dev/null. I don't believe it to copy 700 MB in 20 seconds. Here are my results: | Bonnie | |Chunk Block Stride Out CharOut Block Out Rewri In Char | In BlockRandom Copy 1 Copy 2 Copy 3 |Size SizeKB CPU KB CPU KB CPU KB CPU | KB CPU KB CPU mm:ss mm:ss mm:ss |32 4 8 176781,7437535,7265340,2224794,3 | 738640,9166,7 11,606:48 07:20 07:04 |4 4 8 175080,5462936,7254443,4207591,3 | 579636,6118,8 8,2 |4 4 1 170177,6436435,5236140,1177179,2 | 582236,6119,1 7,8 06:52 07:40 07:51 |2564 64 175080,8474736,8243734,1191779,5 | 691835,5173,9 10,907:01 08:02 07:45 |64 4 16 174880,7458436,4275739,9224493,8 | 752640,2171,6 11,706:52 07:50 07:28 |16 4 4 176181,6459137,1257939,193,7 | 697737,1160,3 11,307:01 07:39 07:09 | | |a single disc w/o RAID 208796 707957,8306742,8193681,8 | 670734,8105,5 7 05:15 06:15 06:00 I am quite astonished that the chunk size doesn't seem to have a large impact on array performance and suspect that I inadvertendly measured something else than RAID performance. Second source of astonishment is that the RAID array is quite slower than a single disk. Under these circumstances, I'd better refrain from putting a news spool on the RAID array, right? With this performance, the RAID array can only be used as a safety measure to be able to recover from a drive failure for system and home directories. Out of academic interest only, I put a second identical SCSI host adapter into the system and moved two of the RAID disk to the second SCSI bus. The third RAID disk and the system disk remained on the first SCSI bus. This is what I clocked with that setup. | Bonnie | |Chunk Block Stride Rich- Out CharOut Block Out Rewri In |Char In BlockRandom Copy 1 Copy 2 Copy 3 |Size Sizetig?KB CPU KB CPU KB CPU KB | CPU KB CPU KB CPU mm:ss mm:ss mm:ss |32 4 8 ja 171278,7437535,5324948,22266 | 95,1947856,7170 10,605:43 06:14 06:48 Even these values are mainly below the performance of a single Atlas disk. Having taken care of the bottleneck of the single Fast SCSI bus, I don't know what still is the problem here. Is it possible that my old 100 MHz CPU is the bottleneck in this system? Any hints and comments will be appreciated. Greetings Marc -- -- !! No courtesy copies, please !! - Marc Haber | " Questions are the | Mailadresse im Header Karlsruhe, Germany | Beginning of Wisdom " | Fon: *49 721 966 32 15 Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fax: *49 721 966 31 29
moving /lib - off topic advice wanted
I suppose this is an off topic question, other than it's related to
disks...:) I find that my / partition is more full than I would like. I
like to keep extra space available in case something unexpected happens.
Here is the current setup:
Filesystem 1k-blocks Used Available Use% Mounted on
/dev/hda862187 48208 10768 82% /
/dev/hda1 7746 3955 3391 54% /boot
/dev/hdc5 495714 61545408568 13% /home
/dev/hda10 1274632690633518135 57% /usr
/dev/hda746632 18515 25709 42% /var
Turns out /lib is taking about 30 megs, so my first idea is to make a new
partition for /lib. I have done this:
Filesystem 1k-blocks Used Available Use% Mounted on
/dev/hda862187 48208 10768 82% /
/dev/hda1 7746 3955 3391 54% /boot
/dev/hdc5 495714 61545408568 13% /home
/dev/hda10 1274632690633518135 57% /usr
/dev/hda746632 18515 25709 42% /var
/dev/hda6 116630 30565 80043 28% /lib
I figured this was a good first step, because if anything went wrong, I
could just reboot and the old /lib directory would still be there:)
(I have not put the /lib mount into /etc/fstab yet.)
So here are my questions, is it safe for me to do the following steps:
1. reboot to get /dev/hda6 unmounted (it's busy now, so umount does not
work)
2. mv /lib /lib.old
3. mkdir /lib
4. chmod 755 /lib
5. chown 0.0 /lib
6. mount /dev/hda6 /lib
7. rm -rf /lib.old {after testing}
Or am I going to run into trouble because /lib's files will be unavailable
for a bit while I enter these commands? Is there a better way to enlarge
/? In general how to you recommend changing partition sizes? Is this an
argument for not seperating directories into different partitions, since
it's harder to keep the free space evenly distributed?
Best Regards,
Robert Laughlin
Re: moving /lib - off topic advice wanted
Or am I going to run into trouble because /lib's files will be unavailable for a bit while I enter these commands? Is there a better way to enlarge /? In general how to you recommend changing partition sizes? Is this an argument for not seperating directories into different partitions, since it's harder to keep the free space evenly distributed? I think unless your drive mounting and init binaries are statically linked you're going to hit trouble at boot time. However have you sifted through /lib to find out if you have an libraries with debugging codes left in or binaries that have been upgraded and/or ones you can recompile to put in /usr/lib ? -sv
Re: moving /lib - off topic advice wanted
On Wed, 20 Oct 1999, Seth Vidal wrote: I think unless your drive mounting and init binaries are statically linked you're going to hit trouble at boot time. Perhaps I am wrong, I expected that a reboot would make the original /lib available again at boot time. The data is still there, just hidden by the mount, right? Best Regards, Robert Laughlin
Re: moving /lib - off topic advice wanted
Perhaps I am wrong, I expected that a reboot would make the original /lib available again at boot time. The data is still there, just hidden by the mount, right? mounting only occurs after fstab is processed. you can't process fstab with the mount command if there are no libraries for mount to rely on. see the problem? -sv
Re: Measuring RAID performance
I am currently experimenting with the Software RAID (patch for 2.2.11 applied to 2.2.12 with raidtools snapshot from about three weeks ago). Same as I am running here. The system I am doing these tests on is an older K5-100 - slow CPU For RAID5 you need quite some CPU power (esp. if writing to the array). For RAID0 or RAID1, this is no problem. For RAID, I have three Quantum Atlas I Don't know these HDDs from practical experience ... on a single Fast SCSI Host Adapter with NCR Chipset. - slow hostadapter, only one channel for 3 HDDs - theoretical limit is 10MB/s. This test is meant to check write rate of the RAID but will measure source disk performance instead because the source disk is likely to be the bottle neck here. Yes ;-) "Copy 2" is the time taken for "cp -R 1 2" , with both source and target directory on the RAID array. I hope to simulate a realistic usage profile with many small files, read and write operations. More or less a seek time test, too. Second source of astonishment is that the RAID array is quite slower than a single disk. It can't get much faster because of your SCSI controller (= 10MB/s) and CPU (maybe limiting, too !?). If you use RAID5, you also must consider, that MORE data is stored on disks than size of files you copied to the disks (well: REDUNDANT data). So if you have 3 disks with RAID5 and store a 10MB file on them, 15MB data will have to be written to disk. If your SCSI bus could make full 10MB/s you would only see 6.6 MB/s of them (in theory). the RAID array can only be used as a safety measure to be able to recover from a drive failure for system and home directories. In your case, yes. Maybe get 2 more SCSI controllers of same type (or U(2)W ctlr + disks), put 1 disk on each channel and try RAID0 for better performance. Just some performance values I got with my RAID arrays: 3 disks IBM DNES 9GB UW on single UW channel, Dual Celeron 333MHz, SW RAID5: this gives about 17MB/s bonnie (block read and write). 4 disks IBM DNES 9GB UW on 2 UW channels, Dual Celeron 500 MHz, 100MHz FSB ;-), SW RAID5: this gives about 25MB/s bonnie (block read and write) 3 disks IBM DDRS (or DNES !?) 9GB UW on 2 channels, Dual Celeron ???MHz, SW RAID0: this gave me about 40MB/s bonnie (block) ! Thomas -- Thomas Waldmann (com_ma, Computer nach Masz) email: [EMAIL PROTECTED] www: www.com-ma.de Please be patient if sending me email, response may be slow. Bitte Geduld, wenn Sie mir email senden, Antwort kann dauern.
Re: moving /lib - off topic advice wanted
Nope, sure don't. My new /lib *partition* mount is not even in /etc/fstab. What will happen at boot time is the /lib *directory* on the / partition will be available. Right? I never deleted the data in that partition. I just mounted my new /lib *partition* on top of it for test. Best Regards, Robert Laughlin On Wed, 20 Oct 1999, Seth Vidal wrote: Perhaps I am wrong, I expected that a reboot would make the original /lib available again at boot time. The data is still there, just hidden by the mount, right? mounting only occurs after fstab is processed. you can't process fstab with the mount command if there are no libraries for mount to rely on. see the problem? -sv
Re: moving /lib - off topic advice wanted
if your init and mount are not statically linked, how on earth are they going to work before /lib is mounted? or wait, suppose you could just keep in the real /lib only libs needed for init, mount and whatever other basic commands on boot (for instance if you have to boot "linux single" one day or something). on my system: # ldd /sbin/init libc.so.6 = /lib/libc.so.6 (0x40018000) /lib/ld-linux.so.2 = /lib/ld-linux.so.2 (0x4000) # ldd /bin/mount libc.so.6 = /lib/libc.so.6 (0x40018000) /lib/ld-linux.so.2 = /lib/ld-linux.so.2 (0x4000) so if you could get away w/ just having those two libs you'd be SET. however take a look at login here, things may not be quite so cozy for "linux single" after all.. # ldd /bin/login libcrypt.so.1 = /lib/libcrypt.so.1 (0x40018000) libpam.so.0 = /lib/libpam.so.0 (0x40046000) libpam_misc.so.0 = /lib/libpam_misc.so.0 (0x4004e000) libdl.so.2 = /lib/libdl.so.2 (0x40051000) libc.so.6 = /lib/libc.so.6 (0x40055000) /lib/ld-linux.so.2 = /lib/ld-linux.so.2 (0x4000) a quick ls shows that this might not be so bad... # ls -sh1 /lib/libcrypt-2.1.2.so /lib/libdl-2.1.2.so /lib/libpam.so.0.70 /lib/libpam_misc.so.0.70 /lib/libpamc.so.0.70 /lib/libc-2.1.2.so /lib/ld-2.1.2.so 81k /lib/ld-2.1.2.so 875k /lib/libc-2.1.2.so 20k /lib/libcrypt-2.1.2.so 10k /lib/libdl-2.1.2.so 28k /lib/libpam.so.0.70 6.0k /lib/libpam_misc.so.0.70 9.0k /lib/libpamc.so.0.70 another thing to take into consideration is which modules you'll need before mount. initrd is your friend in that case i guess. just some stuff to consider. my system is debian potato so YMMV, but the numbers shouldn't be wildly different. -- stephen waters amicus, inc. Seth Vidal wrote: Nope, sure don't. My new /lib *partition* mount is not even in /etc/fstab. What will happen at boot time is the /lib *directory* on the / partition will be available. Right? I never deleted the data in that partition. I just mounted my new /lib *partition* on top of it for test. oh. I see now. that should work. Might be weird in the moments of the mount. hmm. good luck. -sv
Booting on RAID-1
Hi. I guess this keeps on coming up, but I'll ask anyhow. I want to boot on a RAID 1 array. I have 3 disks (sda, sdb, sdc), all of which work fine. They are all partitioned identically: sd?1 = 8GB, sd?2=128MB swap. I build the system on sda1. All the swap files are used fine. I build a RAID from sdb1 and sdc1, and also compiled the kernel (2.2.12) with all the best versions of stuff, so that it recognises RAID 1 during boot. All this worked fine. I copied the entire / to /dev/md0, and edited /etc/lilo.conf so that it loads the new kernel and mounts /dev/md0 as /, and also edited /etc/fstab (and the copy on md0 as well) to mount /dev/md0 as / Still all fine. I can boot as often as I like, and the system boots using /dev/sda, recognises /dev/md0 and then comes up fine, using /dev/md0 as / So here's the problem: I don't want it to use sda other than the MBR. I tried changing /etc/lilo/conf so that it has boot=/dev/md0, and the machine then boots up and ignores md0 entirely, not even mounting it as / after the boot. I heard that lilo now supports RAID devices (lilo v 0.21-10). So can anyone tell me how to make it **boot** off the raid device? the Software-RAID HOWTO is not very clear (at least, not to me) on this. Thanks. P. Paul Unix techy
kernel 2.2.13
anyone tried the 2.2.11-patch on 2.2.13 yet? same hunks fail as on 2.2.12 or...? -- stephen waters amicus, inc.
Re: Measuring RAID performance
btw, I forgot the important information that my RAID array is a RAID 5 array. On Wed, 20 Oct 1999 15:09:06 +0200, you wrote: I am currently experimenting with the Software RAID (patch for 2.2.11 applied to 2.2.12 with raidtools snapshot from about three weeks ago). Same as I am running here. The system I am doing these tests on is an older K5-100 - slow CPU actually, _yes_ ;-) For RAID, I have three Quantum Atlas I Don't know these HDDs from practical experience ... It's an about four year old Fast SCSI 2 GB hard disk that maxes out at about 7 MB/s transfer rate. on a single Fast SCSI Host Adapter with NCR Chipset. - slow hostadapter, only one channel for 3 HDDs - theoretical limit is 10MB/s. I am well aware of that. However, my tests showed that the system only uses about 60 % of that theoretical limit - a single disk alone does more. Greetings Marc -- -- !! No courtesy copies, please !! - Marc Haber | " Questions are the | Mailadresse im Header Karlsruhe, Germany | Beginning of Wisdom " | Fon: *49 721 966 32 15 Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fax: *49 721 966 31 29
Re: stripes of raid5s - crash
On Tue, Oct 19, 1999 at 11:16:47AM -0800, Christopher E. Brown wrote: Can you duplicate this using only one of the raid5 sets? I tried to cause the same behvior with a single raid5 set and it worked fine... but I did not layer raid on raid, perhaps this is where the issue is? When working with a 5 x 18G RAID5 (0 spare) using 2.2.12SMP + raid 2.2.11 (compiled in, not modules) I would get a endless stream about buffers when trying to mount the device, mke2fs and e2fsck worked fine. Seemed to happen when the array was in the beginning of the reconstruct. With 2.2.13pre15SMP + raid 2.2.11 I managed to get this a couple times, but only if I mount it right after reconstruct starts on a just mkraided array. If I wait till the reconstruct hits 2 - 3 % it mounts just fine. I have not seen this on arrays smaller than 50G (but this is not hard data, it could just be the faster reconstruct). I am still having these problems - 2 RAID5s with 3 drives each (10 MB per drive for fast rebuild) i get an instant "Got MD request, not good" kernel hang when doing an mke2fs on the stripe containing the both raid5s ... Very bad - This should work from the manpages but doesn (Reproducable with large and small stripe/raid sets) BTW: I am UP 2.2.12plain + raid patches ... BTW: The Hang does occur within rebuild and afterwards - Doesnt matter .. Flo -- Florian Lohoff [EMAIL PROTECTED] +49-5241-470566 ... The failure can be random; however, when it does occur, it is catastrophic and is repeatable ... Cisco Field Notice
Raid troubles
Hi All, I had a raid fail do to an external cable falling off the box last night (came loose andyway) :-/ I have 2 drives that got marked as failed...It there anyway to mark them good so I can try to save some of the data ?? It is a 8 disk software raid 5 array on dual scsi controllers. Randy Winch
RAID0
Can anyone of you give me a description how to setup up a Software RAID0 partition ?? Thank you
Re: 71% full raid - no space left on device
I was under the impression that the reiserfs was more of just an experiemnt in making a BTree sorted filesystem. "Stephen C. Tweedie" wrote: Hi, On Thu, 14 Oct 1999 09:22:25 -0700, Thomas Davis [EMAIL PROTECTED]> said: > I don't know of any Unix FS with dynamic inode allocation.. Is there > one? Reiserfs does, doesn't it? --Stephen -- Mark Ferrell : [EMAIL PROTECTED] (972) 685-7868 : Desk (972) 685-4210 : Lab (972) 879-4326 : Pager
Re: (reiserfs) Re: 71% full raid - no space left on device
We don't have inodes in our FS, but we do have stat data, and that is dynamically allocated (dynamic per FS, not per file yet, soon but not yet each field will be optional and inheritable per file). Does XFS dynamically allocate? It might. Best, Hans "Stephen C. Tweedie" wrote: Hi, On Thu, 14 Oct 1999 09:22:25 -0700, Thomas Davis [EMAIL PROTECTED] said: I don't know of any Unix FS with dynamic inode allocation.. Is there one? Reiserfs does, doesn't it? --Stephen -- Get Linux (http://www.kernel.org) plus ReiserFS (http://devlinux.org/namesys). If you sell an OS or internet appliance, buy a port of ReiserFS! If you need customizations and industrial grade support, we sell them.
Re: (reiserfs) Re: 71% full raid - no space left on device
Experiments sometimes succeed, the code works, try it. Hans Mark Ferrell wrote: I was under the impression that the reiserfs was more of just an experiemnt in making a BTree sorted filesystem. -- Get Linux (http://www.kernel.org) plus ReiserFS (http://devlinux.org/namesys). If you sell an OS or internet appliance, buy a port of ReiserFS! If you need customizations and industrial grade support, we sell them.
boot from RAID 1
Hello! I have still problems using lilo 21 with raid 1 to boot. When i use root=/dev/md0 OR root=900 lilo reports everything unable to handle device 0x0900 ... What can I do? Greetings, Joerg
Re: boot from RAID 1
Jörg Hinz wrote: I have still problems using lilo 21 with raid 1 to boot. When i use root=/dev/md0 OR root=900 lilo reports everything unable to handle device 0x0900 ... What can I do? Read this list. Look at this URL: http://linuxwww.db.erau.edu/mail_archives/linux-raid/Aug_99/0273.html Dewa, CB -- // CB aka Christian Balzer, Tannenstr. 23c, D-64342 Seeheim, Germany \X/ [EMAIL PROTECTED] | Voice: +49 6257 83036, Fax/Data: +49 6257 83037 SWB - The Software Brewery - | http://www.swb.de/ | Anime no Otaku
Re: Measuring RAID performance
It's an about four year old Fast SCSI 2 GB hard disk that maxes out at about 7 MB/s transfer rate. on a single Fast SCSI Host Adapter with NCR Chipset. - slow hostadapter, only one channel for 3 HDDs - theoretical limit is 10MB/s. I am well aware of that. However, my tests showed that the system only uses about 60 % of that theoretical limit - a single disk alone does more. Did you forget to add 50% RAID5 redundancy data, so your 60% are 90%! Thomas -- Thomas Waldmann (com_ma, Computer nach Masz) email: [EMAIL PROTECTED] www: www.com-ma.de Please be patient if sending me email, response may be slow. Bitte Geduld, wenn Sie mir email senden, Antwort kann dauern.
RE: Raid troubles
Randy Winch wrote: I had a raid fail do to an external cable falling off the box last night (came loose andyway) :-/ I have 2 drives that got marked as failed...It there anyway to mark them good so I can try to save some of the data ?? http://ostenfeld.dk/~jakob/Software-RAID.HOWTO/Software-RAID.HOWTO-6.html should be just what you're looking for. tom
