Re: RAID0: Fast writes, Slow reads...
When running Bonnie, you should always set the file size to 3-4 times the size of your RAM, else you get the 200Mb /sec speeds (which are very pleasant, but not realistic). I think the 100% CPU is in great part Bonnie generating the test files. I've tried copying files, this takes almost no CPU power. The crashes are very annoying, no messages in the logs or anything. I've run extensive memory tests and today I'm installing a new 3c905 card i case that's the problem. After that I don't know... Rearranging all PCI cards? I'll tell if I discover anything useful. Kent Funny (or not), this sounds a lot like my current problems. I have a HP Netserver LH4 with integrated HP Netraid controller. When doing heavy I/O on the Netraid controller the entire system more or less freezes. I am using newet firmware version (have even tried a few older versions), and have tried it on 2.2.14 and 2.2.15pre14. Doing bonnie on a single 9GB HP disk shows the following: ---Sequential Output ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- MachineMB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU 100 7771 99.5 6753 3.8 4513 9.2 7830 99.6 245756 100.8 22572.5 95.9 That is approx. 4 times faster on block read than block write, but 100% cpu load on block read!! I was coming to the conclusion that I had to look deep into the AMI Megaraid driver to try and locate the problem, but could the problem by chance be more generic ? Can anyone offer some qualified guidance about where to start look for this problem ? /bart
Re: product testimonials
Well, we've been using assorted versions of the 0.90 raid code for over a year in a couple of servers. We've had mostly good success with both the raid1 and raid5 code. I don't have any raid5 disk failure stories (yet ;-), but we are using EIDE drives so I expect one before TOO long ;-) Raid5 has given us good performance and reliability so far. Now, we do have a raid1 array that did something interesting (and bad). One of the drives was failing intermittently (and fairly silently) and had been removed from the array. Unfortunately, backups were also failing (again, fairly silently). On a normal power down / reboot, it appears that the wrong drive was marked as master and on the reboot it re-synced to the drive that had been out of the array for a couple of months. (yeah, yeah, we need a sys-admin ;-) Anyway, 2 months of data went down the tubes. No level of raid is a replacement for good backups. Keith P.S. That system was 2.0.0-pre3, I think, with the matching raidtools. On Sun, 19 Mar 2000, Seth Vidal wrote: Hi folks, I've got a user in my dept who is thinking about using software raid5 (after I explained the advantages to them) - but they want "testimonials" ie: - people who have used software raid5 under linux and have had it save their ass or have had it work correctly and keep them from a costly backup restore. IE: success stories. Also I would like to hear some failure stories too - sort of horror stories - now the obscure situations I don't care about - if you got an axe stuck in your drive by accident and it killed the entire array then I feel sorry for you but I don't consider that average use. Can anyone give some testimonials on the 0.90 raid? thanks -sv
Re: product testimonials
Well, we've been using assorted versions of the 0.90 raid code for over a year in a couple of servers. We've had mostly good success with both the raid1 and raid5 code. I don't have any raid5 disk failure stories (yet ;-), but we are using EIDE drives so I expect one before TOO long ;-) Raid5 has given us good performance and reliability so far. Now, we do have a raid1 array that did something interesting (and bad). One of the drives was failing intermittently (and fairly silently) and had been removed from the array. Unfortunately, backups were also failing (again, fairly silently). On a normal power down / reboot, it appears that the wrong drive was marked as master and on the reboot it re-synced to the drive that had been out of the array for a couple of months. (yeah, yeah, we need a sys-admin ;-) Anyway, 2 months of data went down the tubes. No level of raid is a replacement for good backups. On the subject of semi-silent failures: Has anyone written a script to monitor the [U]'s in the /proc/mdstat location? It would be fairly trivial (start beeping the system speaker loudly and emailing repetitively) Has this already been or should I work on it? Thanks -sv
RE: Benchmark 1 [Mylex DAC960PG / 2.2.12-20 / P3]
On Thu, 2 Mar 2000, Gregory Leblanc wrote: -Original Message- I think the bonnie test at least tells me what max throughput of the drives and controller's ability to do RAID-5 are. I'll be happy to run other benchmarks though. Where can I find tiotest? Searches on google/altavista/freshmeat turned up nothing. It's here: http://www.icon.fi/~mak/tiotest/ James has done a fair bit of work on it, but nobody replied to me when I asked a couple of questions about how the makefile should be modified. Maybe they just don't want any more help. This last guess it not true at all. Any and all help is appreciated. I have been busy with work and so i have been blocked out more or less from tiotest things. But James have been doing great work and also thanks to all who have sent patches! Just see TODO and BUGS file in latest 0.27 and see that help is truely needed :) -- Mika [EMAIL PROTECTED]
Re: Benchmarking.. how can I get more out of my box?
On Thu, 9 Mar 2000, Christian Robottom Reis wrote: On Thu, 9 Mar 2000, Jakob Østergaard wrote: On Wed, 08 Mar 2000, Brian Pomerantz wrote: On Thu, Mar 09, 2000 at 12:44:32AM +0100, Jakob Østergaard wrote: If there isn't hot-swap RAID 5 with auto rebuild, it will never happen. It would be nice if a program such as ASCI could put the resources needed into Linux to actually get decent hot swap capability... Let's see what happens. We're talking about something more automatic than "echo scsi-add-single-device [...]", I surmise, from this? It works fine with me and RAID1, and I don't think it would be hard to put up a framework for making it better. Of course, you need to prepare the other disk for the RAID, but I usually keep some pre-mkraided around handy. For what i have experienced, the usefulness of "echo scsi-add-single-device [...]" is heavily dependant on the scsi driver beneath the common scsi stuff. With aic7xxx i was able to succeed partially with failed/nonexisting SCSI negotiations and thus very hard performance degration. What controller did you use ? -- Mika [EMAIL PROTECTED]
Re: Mylex RAID oddity?
On Fri, 10 Mar 2000, Paul Jakma wrote: On Thu, 9 Mar 2000, Kent Nilsen wrote: Just installed a new server (PIII 600, 392Mb RAM, Mylex ExtremeRAID (DAC1164P, 64Mb Cache), Mandrake 7.0, 2.2.14, 2.2.5 patch for DAC960) some of those figures are a bit off, esp the first few cause the filesize used is much less than RAM so you'd be testing RAM more than disk. Also, should I test with the -W option, will Linux write like this when using it as a NFS/Samba only? Here are the Tiotest (latest) results: With -W option in tiotest/tiobench the writes will be sequential. Second writing thread will wait first one to finish first and so on. So in effect this is a useless way to test parallel IO performance on write side. But what it is for is that this way when only one thread is allowed to write in time, the test files will not be fragmented across disk area. This makes a difference atleast in raid1 READ case when you want to measure concurrent potential read performance. So the -W could be translated as, '-Write avoiding fragmentation' The default behaviour of tiobench/tiotest should be that it is off. (after all its name says it should test concurrent io :) I have to check if this is the case. -- Mika [EMAIL PROTECTED]
Re: product testimonials
On Mon, Mar 20, 2000 at 10:29:28AM -0500, Seth Vidal wrote:
[...]
On the subject of semi-silent failures: Has anyone written a script to
monitor the [U]'s in the /proc/mdstat location? It would be fairly
trivial (start beeping the system speaker loudly and emailing
repetitively) Has this already been or should I work on it?
OK, as usual I've taken somebody else's idea and ran with it. Here's
a little script daemon which monitors the raids for failures:
---SOF---
#!/bin/bash
PATH="/bin:/usr/bin:/usr/local/bin:${PATH}"
ADMIN_EMAIL="root@localhost"
RAID_STAT="/proc/mdstat"
while true
do
sleep 3
if [ -n "`cat $RAID_STAT | perl -ne 'if (/(.*\[U*[^\]\[U]+U*\])$/) { print \"Failure!
$1\n\"; }'`" ]
then
cat $RAID_STAT | mail -s " Raid Failure Warning " $ADMIN_EMAIL
sleep 600
fi
done
---EOF---
(Watch for wrapped lines. There are a couple long ones.)
If you want to test it, then do this:
cat /proc/mdstat testfile
then, change RAID_STAT to ="testfile"
Then, start the daemon and let it run. After a while, simulate the
failure by editing 'testfile'. If it works, kill the daemon, and
point RAID_STAT to the real file (/proc/mdstat).
To use it in rc.local (or where ever), just start it like this:
su nobody raid-monitor.sh
(Or change 'nobody' to some other under priledged user of your
choice.)
Phil
--
Philip Edelbrock -- IS Manager -- Edge Design, Corvallis, OR
[EMAIL PROTECTED] -- http://www.netroedge.com/~phil
PGP F16: 01 D2 FD 01 B5 46 F4 F0 3A 8B 9D 7E 14 7F FB 7A
Re: Are Ultra 160 adapters compatible with UW SCSI?
On 20 Mar 2000 [EMAIL PROTECTED] wrote: I just read the Mylex eXtremeRAID 2000 blurb, and it says it is an "Ultra 160 SCSI to PCI RAID Controller." I'm not familiar with Ultra Also known as Ultra3 SCSI, I believe. 160... is that hardware compatible with my UW SCSI drives? [snip] It should be. I don't know for sure how backwards-compatable the different controllers are, but I am able to get U2 drives working fine on my U3 Dell PERC, no problems. -- Jeremy Stanley, Network Engineer, IT Consultant /*= http://www.k4d4th.org/~rcarter -=*\ | Mathematic Algorithm Design (MAD) Scientist | | CTHULHU FHTAGN Project, Crawling Chaos Labs | \*=- http://www.k4d4th.org -=*/ Kibo ran off with my spleen, sporks AND Cesium.
RE: Are Ultra 160 adapters compatible with UW SCSI?
I just read the Mylex eXtremeRAID 2000 blurb, and it says it is an "Ultra 160 SCSI to PCI RAID Controller." I'm not familiar with Ultra 160... is that hardware compatible with my UW SCSI drives? SCSI is SCSI, unless it's differential (LVD, VHVD, etc). So yeah, they should work just fine. Actually, LVD and Single Ended are 'compatible' in the sense that an LVD controller can usually fall back to Single Ended mode. Not so for HVD.
Re: Are Ultra 160 adapters compatible with UW SCSI?
Jeremy Stanley wrote: On 20 Mar 2000 [EMAIL PROTECTED] wrote: I just read the Mylex eXtremeRAID 2000 blurb, and it says it is an "Ultra 160 SCSI to PCI RAID Controller." I'm not familiar with Ultra Also known as Ultra3 SCSI, I believe. That is true. This probably isn't worth mentioning, but as a veteran of the SCSI naming wars, I'll just add a comment that I stole from an IBM page: "Ultra160 refers more specifically to an implementation of Ultra3 SCSI, which contains CRC, domain validation, and double transition clocking." http://ssdweb01.storage.ibm.com/hardsoft/diskdrdl/library/ultra3.htm According to time honored SCSI tradition, a drive can call itself Ultra3 SCSI if it implements any one of the new features of Ultra3. -- Dan Jones, Storage Engineer VA Linux Systems V:(408)542-5737 F:(408)745-9130 1382 Bordeaux Drive [EMAIL PROTECTED]Sunnyvale, CA 94089
Re: kernel 2.3.4X raid0 performance problems
I wanted to follow-up that with kernel 2.3.51 I get raid0 performance problems similar to those reported below, but at larger file sizes. I created files of approx. size 256 MB, 512 MB, and 1 GB. I did simple tests 'time cat file ... /dev/null' on different file sizes/counts, always totalling 1 GB of data. the real time for one 512 MB file is about 15 seconds, and for two 512 MB files is about 30 seconds. however, the real time for one 1 GB file is about 65 seconds. this is on a dual 550 MHz PIII with 256MB RAM, two SCSI disks with a 48 GB partition on each disk, raid0 w/ 256 KB chunk size, and an ext2 filesystem. the SCSI adapter is Adaptec AIC-7896/7 Ultra2. karl Date: Fri, 3 Mar 2000 19:58:26 -0500 From: James Manning [EMAIL PROTECTED] Subject: Re: kernel 2.3.4X raid0 performance problems [ Friday, March 3, 2000 ] Karl Czajkowski wrote: I upgraded the kernel to 2.3.47, 48, and 49 and got a performance problem where "time cat file ... /dev/null" for a 300 MB file shows some scaling, but for a 600 MB file the throughput is almost identical to a single disk. how much memory in the machine? is there a known scheduling problem with the 2.3.4X kernel raid vs. the 2.2.12-20 patches distributed by redhat? I need the new kernel for ethernet patches... 2.3.4x raid merge isn't finished yet, but I'm surprised raid0 not working as well as it sounds like it should. I also noticed that the "boot with raid" option in the kernel won't compile properly in the 2.3.4X series. should once merge is finished. James
Are Ultra 160 adapters compatible with UW SCSI?
I just read the Mylex eXtremeRAID 2000 blurb, and it says it is an "Ultra 160 SCSI to PCI RAID Controller." I'm not familiar with Ultra 160... is that hardware compatible with my UW SCSI drives? Dave
Re: product testimonials
On Mon, Mar 20, 2000 at 10:36:40AM -0800, [EMAIL PROTECTED] wrote:
while true
do
sleep 3
if [ -n "`cat $RAID_STAT | perl -ne 'if (/(.*\[U*[^\]\[U]+U*\])$/) { print
\"Failure! $1\n\"; }'`" ]
then
cat $RAID_STAT | mail -s " Raid Failure Warning " $ADMIN_EMAIL
sleep 600
fi
done
Ugh. Why would you want to check every _3 seconds_ for a disk failure?
That's a lot of wasted cycles. Instead of forking a few different programs
like that, how about:
a cronjob that runs every 5 minutes and just diffs mdstat and a copy of a
good mdstat:
*/5 * * * * /usr/bin/diff /mdstat.good /proc/mdstat
Just do "cp /proc/mdstat /mdstat.good", and that's it. You'll get a mail if
there's a difference (including failure), and no mail if they're the same.
You still get mailed every 5 minutes, but you don't use quite so many cycles
and processes to do it. (worst case, you have a 5-10 minute window with a
failed disk. If you're worried about it, change the */5 to * ...) You could
also throw this into a monitor using "mon"
(http://www.kernel.org/software/mon) which you can set to mail you once an
hour/etc.
--
Randomly Generated Tagline:
I'd love to, but I want to spend more time with my blender.
Results
I just finished up some benchmarks on a RAID1 system here at work. The machine is an HP E30, Pentium 166, 128MB of ram, HP PCI Adaptec SCSI card (I think it's a 2920 with a boot rom of some sort). 2x2.1GB HP drives, running mirrored across the board, except for swap. Right now, the machine isn't do anything, but that gives me plenty of time to do re-compiles, which aren't very fast. All tests have been done with tiotest-0.24, and with settings --numruns 6 and --size 384. This first test is 2.2.12 from a stock RH6.1 install. Size is MB, BlkSz is Bytes, Read and Write are MB/sec, Seeks are Seeks/sec Dir Size BlkSz Thr# Read (CPU%) Write (CPU%) Seeks (CPU%) - -- --- - -- -- .38440961 3.32616 15.8% 1.99082 9.02% 78.1085 2.58% .38440962 3.17759 15.2% 1.95808 9.06% 78.0405 2.43% .38440964 3.17624 15.5% 1.97681 9.45% 78.9123 2.51% .38440968 3.25837 16.3% 1.99858 9.77% 80.1691 2.53% Not very impressive... Second test, stock 2.2.14. Size is MB, BlkSz is Bytes, Read and Write are MB/sec, Seeks are Seeks/sec Dir Size BlkSz Thr# Read (CPU%) Write (CPU%) Seeks (CPU%) - -- --- - -- -- .38440961 2.86103 14.4% 2.11047 9.43% 79.8300 2.52% .38440962 2.89427 14.2% 2.07059 9.59% 80.5877 2.55% .38440964 2.97460 14.6% 2.05492 10.3% 82.0260 2.54% .38440968 3.12418 15.8% 2.03119 10.9% 83.2683 2.59% Well, shoot, those writes are a little better, but the reads are sucking... At least it doesn't use much CPU. This last test is with 2.2.14 and Miko's RAID1 read balancing test. Size is MB, BlkSz is Bytes, Read and Write are MB/sec, Seeks are Seeks/sec Dir Size BlkSz Thr# Read (CPU%) Write (CPU%) Seeks (CPU%) - -- --- - -- -- .38440961 3.86832 18.2% 2.05358 9.99% 88.9377 2.69% .38440962 4.68355 23.0% 2.03622 9.78% 106.298 3.18% .38440964 5.04067 25.2% 2.02812 10.3% 118.177 3.56% .38440968 5.26915 26.8% 2.00952 10.9% 128.202 3.85% Wow, nice stuff of those reads look purty nice, reads are OK. The whole machine seems to have pretty slow disk performance in general, perhaps they're just slow disks. Cool stuff, thanks everybody! Greg
RE: Are Ultra 160 adapters compatible with UW SCSI?
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] Sent: Monday, March 20, 2000 11:05 AM To: [EMAIL PROTECTED] Subject: Are Ultra 160 adapters compatible with UW SCSI? I just read the Mylex eXtremeRAID 2000 blurb, and it says it is an "Ultra 160 SCSI to PCI RAID Controller." I'm not familiar with Ultra 160... is that hardware compatible with my UW SCSI drives? SCSI is SCSI, unless it's differential (LVD, VHVD, etc). So yeah, they should work just fine. Greg
