Re: AMI MegaRAID datapoint... or is it NFS

[Matthew Dillon]

:Another thing I've found with the MegaRAID (or maybe this is an nfs
:thing?) is that large scale (100Mb, full duplex) hits on the NFS
:server tend to lock up the nfs server (which has the megaraid in it).
:Typically, this includes not being able to access the non-raid root
:var and usr partitions.
:
:Any ideas?  I can reproduce the problem, but it doesn't cause a panic.
:
:Dave.
:|David Gilbert, Velocet Communications.   | Two things can only be |

You need to figure out what the processes are locked up in.  If 'ps axl'
doesn't work then you need to ctl-alt-esc into DDB (assuming the kernel
is configured for DDB) and do a 'ps' there.  

It should be possible to narrow the problem down from that output.

-Matt
Matthew Dillon 
<[EMAIL PROTECTED]>


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: AMI MegaRAID datapoint... or is it NFS

[David Gilbert]

Another thing I've found with the MegaRAID (or maybe this is an nfs
thing?) is that large scale (100Mb, full duplex) hits on the NFS
server tend to lock up the nfs server (which has the megaraid in it).
Typically, this includes not being able to access the non-raid root
var and usr partitions.

Any ideas?  I can reproduce the problem, but it doesn't cause a panic.

Dave.

-- 

|David Gilbert, Velocet Communications.   | Two things can only be |
|Mail:   [EMAIL PROTECTED] |  equal if and only if they |
|http://www.velocet.net/~dgilbert |   are precisely opposite.  |
=GLO


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: AMI MegaRAID datapoint.

[Mike Smith]

> Mike> The Mylex controllers seem to have a small edge in performance,
> Mike> which may be due to them doing cache-line-sized I/Os (usually
> Mike> only 8k) in that case.
> 
> Maybe so, but they also don't seem to support the LVD-enabled versions 
> of the Mylex cards.

Who is "they" here?  We currently support the AcceleRAID 250 (single 
channel LVD card), and I just committed code to -current that adds 
support for the AcceleRAID 352 (Ultra160) and eXtremeRAID 1164P (three 
channel LVD).

As I said earlier, we support all of Mylex' currently-shipping RAID 
controllers (and I expect we'll support the 2000 and 3000 when they 
finally arrive).

-- 
\\ Give a man a fish, and you feed him for a day. \\  Mike Smith
\\ Tell him he should learn how to fish himself,  \\  [EMAIL PROTECTED]
\\ and he'll hate you for a lifetime. \\  [EMAIL PROTECTED]




To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: AMI MegaRAID datapoint.

[David Gilbert]

> "Mike" == Mike Smith <[EMAIL PROTECTED]> writes:

Mike> Try enabling DirectIO and WriteBack if you haven't already.
Mike> AMI's RAID5 implementation seems to suffer from rewriting the
Mike> entire stripe when you do sub-stripe-sized writes, but I'm not
Mike> sure about that yet.

Already done.  This would explain why 128K stripes are bad.

Mike> The Mylex controllers seem to have a small edge in performance,
Mike> which may be due to them doing cache-line-sized I/Os (usually
Mike> only 8k) in that case.

Maybe so, but they also don't seem to support the LVD-enabled versions 
of the Mylex cards.

Dave.

-- 

|David Gilbert, Velocet Communications.   | Two things can only be |
|Mail:   [EMAIL PROTECTED] |  equal if and only if they |
|http://www.velocet.net/~dgilbert |   are precisely opposite.  |
=GLO


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: AMI MegaRAID datapoint.

[Mike Smith]

> > "Mike" == Mike Smith <[EMAIL PROTECTED]> writes:
> 
> >> The AMI MegaRAID 1400 delivers between 16.5 and 19 M/s (the 19M/s
> >> value is somewhat contrived --- using 8 bonnies in parrallel and
> >> then summing their results --- which is not 100% valid)... but the
> >> MegaRAID appears to be stable.
> 
> Mike> Hmm.  Those numbers aren't so great though.  I'd be interested
> Mike> to know how busy the controller is during your test (use systat
> Mike> -vmstat 1 and look at the amrd0 device), as well as how you've
> Mike> configured it.  AMI's default configurations for those
> Mike> controllers is wildly inconsistent between one BIOS version and
> Mike> the next.
> 
> Well... it's RAID-5 across the same 8 drives with all 8 drives on one
> LVD chain (same configuration as the other test).  I have tried the
> 128k stripe, but it was slower than the default 64k stripe.

Try enabling DirectIO and WriteBack if you haven't already.  AMI's RAID5
implementation seems to suffer from rewriting the entire stripe when you 
do sub-stripe-sized writes, but I'm not sure about that yet.

The Mylex controllers seem to have a small edge in performance, which may 
be due to them doing cache-line-sized I/Os (usually only 8k) in that case.

-- 
\\ Give a man a fish, and you feed him for a day. \\  Mike Smith
\\ Tell him he should learn how to fish himself,  \\  [EMAIL PROTECTED]
\\ and he'll hate you for a lifetime. \\  [EMAIL PROTECTED]




To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: AMI MegaRAID datapoint.

[David Gilbert]

> "Brad" == Brad Knowles <[EMAIL PROTECTED]> writes:

Brad> At 10:02 AM -0500 1999/12/17, David Gilbert wrote:
>> Well... it's RAID-5 across the same 8 drives with all 8 drives on
>> one LVD chain (same configuration as the other test).  I have tried
>> the 128k stripe, but it was slower than the default 64k stripe.

Brad>   One of the lessons I learned from Greg was that, when dealing
Brad> with RAID implementations in hardware, you should usually take
Brad> their "native" stripe size, since that's the one that will
Brad> usually perform best.

Brad>   It's only when you do software RAID (e.g., vinum) that you can
Brad> choose larger or smaller stripe sizes with a reasonable
Brad> expectation that you will get the particular performance
Brad> enhancement that you're hoping for.

Another thing that I find very different betweent he vinum software
RAID-5 and the hardware I've tried (I've used both the DPT and the
MegaRAID controllers) that the latency of a command can be incredible
on the hardware raid.  If you type (for instance) 'du -ks foo' where
foo is a big directory on the hardware raid, and then press CTRL-C, it 
can take 10 to 15 seconds to come back (I'm assuming that this is
queued I/O requests).

Dave.

-- 

|David Gilbert, Velocet Communications.   | Two things can only be |
|Mail:   [EMAIL PROTECTED] |  equal if and only if they |
|http://www.velocet.net/~dgilbert |   are precisely opposite.  |
=GLO


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: AMI MegaRAID datapoint.

[Brad Knowles]

At 10:02 AM -0500 1999/12/17, David Gilbert wrote:

>  Well... it's RAID-5 across the same 8 drives with all 8 drives on one
>  LVD chain (same configuration as the other test).  I have tried the
>  128k stripe, but it was slower than the default 64k stripe.

One of the lessons I learned from Greg was that, when dealing 
with RAID implementations in hardware, you should usually take their 
"native" stripe size, since that's the one that will usually perform 
best.

It's only when you do software RAID (e.g., vinum) that you can 
choose larger or smaller stripe sizes with a reasonable expectation 
that you will get the particular performance enhancement that you're 
hoping for.

-- 
   These are my opinions -- not to be taken as official Skynet policy
  
|o| Brad Knowles, <[EMAIL PROTECTED]>Belgacom Skynet NV/SA |o|
|o| Systems Architect, News & FTP Admin  Rue Col. Bourg, 124   |o|
|o| Phone/Fax: +32-2-706.11.11/12.49 B-1140 Brussels   |o|
|o| http://www.skynet.be Belgium   |o|
\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
  Unix is like a wigwam -- no Gates, no Windows, and an Apache inside.
   Unix is very user-friendly.  It's just picky who its friends are.


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: AMI MegaRAID datapoint.

[David Gilbert]

> "Mike" == Mike Smith <[EMAIL PROTECTED]> writes:

>> The AMI MegaRAID 1400 delivers between 16.5 and 19 M/s (the 19M/s
>> value is somewhat contrived --- using 8 bonnies in parrallel and
>> then summing their results --- which is not 100% valid)... but the
>> MegaRAID appears to be stable.

Mike> Hmm.  Those numbers aren't so great though.  I'd be interested
Mike> to know how busy the controller is during your test (use systat
Mike> -vmstat 1 and look at the amrd0 device), as well as how you've
Mike> configured it.  AMI's default configurations for those
Mike> controllers is wildly inconsistent between one BIOS version and
Mike> the next.

Well... it's RAID-5 across the same 8 drives with all 8 drives on one
LVD chain (same configuration as the other test).  I have tried the
128k stripe, but it was slower than the default 64k stripe.

>> I would like to know, however, if there are any planned or
>> in-the-works utility programs for the amr device.  In particular, a
>> program to print the state of the array would be useful.

Mike> I'm currently waiting on AMI for more documentation, at which
Mike> point there will indeed be more monitoring and control
Mike> facilities added.

Dave.

-- 

|David Gilbert, Velocet Communications.   | Two things can only be |
|Mail:   [EMAIL PROTECTED] |  equal if and only if they |
|http://www.velocet.net/~dgilbert |   are precisely opposite.  |
=GLO


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: AMI MegaRAID datapoint.

[Mike Smith]

> I have the Enterprise 1400 Megaraid adapter with (currently 16M) on
> it.  I have tested the various modes of operation (different raid
> levels and striping) and find it to be working well.  My LVD array
> consists of 8 18G Quamtum IV's.
> 
> Now... using vinum and either the 2940U2W (Adaptec LVD) or the TekRAM
> (NCR) LVD (using the sym0 device) gives 30 to 35 M/s under RAID-5.
> This is impressive and subject to the bug that I mentioned in -STABLE
> which still hasn't been found.

It's pretty good, that's for sure. 8)

> The AMI MegaRAID 1400 delivers between 16.5 and 19 M/s (the 19M/s
> value is somewhat contrived --- using 8 bonnies in parrallel and then
> summing their results --- which is not 100% valid)... but the MegaRAID 
> appears to be stable.

Hmm.  Those numbers aren't so great though.  I'd be interested to know 
how busy the controller is during your test (use systat -vmstat 1 and 
look at the amrd0 device), as well as how you've configured it.  AMI's 
default configurations for those controllers is wildly inconsistent 
between one BIOS version and the next.

> I would like to know, however, if there are any planned or
> in-the-works utility programs for the amr device.  In particular, a
> program to print the state of the array would be useful.

I'm currently waiting on AMI for more documentation, at which point there 
will indeed be more monitoring and control facilities added.

-- 
\\ Give a man a fish, and you feed him for a day. \\  Mike Smith
\\ Tell him he should learn how to fish himself,  \\  [EMAIL PROTECTED]
\\ and he'll hate you for a lifetime. \\  [EMAIL PROTECTED]




To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: AMI MegaRAID datapoint.

[David Gilbert]

> "Brad" == Brad Knowles <[EMAIL PROTECTED]> writes:

Brad>   It sounds like the second RAID-5 bug listed on the page I
Brad> mentioned:

>>> 28 September 1999: We have seen hangs when perform heavy I/O to
>>> RAID-5 plexes. The symptoms are that processes hang waiting on
>>> vrlock and flswai. Use ps lax to display this information.
>>> 
>>> Technical explanation: A deadlock arose between code locking
>>> stripes on a RAID-5 plex (vrlock) and code waiting for buffers to
>>> be freed (flswai).
>>> 
>>> Status: Being fixed.

Maybe, maybe not.  My crashes have a different signature than your
according to Greg.

Brad>   I believe that I have seen this bug myself, but in my only
Brad>   I have never been impressed with the benchmarking that bonnie
Brad> is capable of.  In my experience, rawio is a much better tool,
Brad> because it handles coordinating large numbers of child
Brad> processes, doesn't lose information in communications between
Brad> the parent and the child processes, by-passes all the filesystem
Brad> overhead, etc

Brad>   If you want to do filesystem level benchmarking, I've been
Brad> more impressed by what I've seen out of Postmark.

... well... one of the pros that I have heard for bonnie is that you
get a number that is close to what you can expect with
applications... vs. something like rawio that is isolating one system
for intensive testing.  I often use bonnie as a good weathervane of
how fast a "whole system" is with it's disk.

Dave.

-- 

|David Gilbert, Velocet Communications.   | Two things can only be |
|Mail:   [EMAIL PROTECTED] |  equal if and only if they |
|http://www.velocet.net/~dgilbert |   are precisely opposite.  |
=GLO


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: AMI MegaRAID datapoint.

[Brad Knowles]

At 11:48 AM -0500 1999/12/16, David Gilbert wrote:

>  It's a really long thread. I'm not going to repeat it here.
>  Basically, under "enough" load, vinum trashes the kernel stack in such
>  a way that debugging is very tough.

It sounds like the second RAID-5 bug listed on the page I mentioned:

>>  28 September 1999: We have seen hangs when perform heavy I/O to
>>  RAID-5 plexes. The symptoms are that processes hang waiting on
>>  vrlock and flswai. Use ps lax to display this information.
>>
>>  Technical explanation: A deadlock arose between code locking stripes
>>  on a RAID-5 plex (vrlock) and code waiting for buffers to be freed
>>  (flswai).
>>
>>  Status: Being fixed.

I believe that I have seen this bug myself, but in my only 
serious attempt to replicate it, I managed to create what appears to 
be a new third bug which he had never seen before.  Yes, I've already 
given all debugging information to Greg.

>  I got the MegaRAID 1400 because the DPT V drivers weren't available.

Understandable.  I didn't know that the AMI MegaRAID controller 
was even an option, otherwise I would have looked at it.

>  The MegaRAID should be roughly equivlanet to the DPT V.

I'd really like to see these two benchmarked head-to-head, or at 
least under sufficiently similar circumstances that we can be 
reasonably comfortable with how well one performs relative to the 
other.

>   Do go with
>  LVD if you can.

The drives are using SCA attachment mechanisms, but I believe 
that electronically they are LVD.

>  I have done benchmarking with bonnie instead of rawIO.  The output is
>  as follows:

I have never been impressed with the benchmarking that bonnie is 
capable of.  In my experience, rawio is a much better tool, because 
it handles coordinating large numbers of child processes, doesn't 
lose information in communications between the parent and the child 
processes, by-passes all the filesystem overhead, etc

If you want to do filesystem level benchmarking, I've been more 
impressed by what I've seen out of Postmark.

-- 
   These are my opinions -- not to be taken as official Skynet policy
  
|o| Brad Knowles, <[EMAIL PROTECTED]>Belgacom Skynet NV/SA |o|
|o| Systems Architect, News & FTP Admin  Rue Col. Bourg, 124   |o|
|o| Phone/Fax: +32-2-706.11.11/12.49 B-1140 Brussels   |o|
|o| http://www.skynet.be Belgium   |o|
\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
  Unix is like a wigwam -- no Gates, no Windows, and an Apache inside.
   Unix is very user-friendly.  It's just picky who its friends are.


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: AMI MegaRAID datapoint.

[David Gilbert]

> "Brad" == Brad Knowles <[EMAIL PROTECTED]> writes:

>> This is impressive and subject to the bug that I mentioned in
>> -STABLE which still hasn't been found.

Brad>   Which one is this?

It's a really long thread. I'm not going to repeat it here.
Basically, under "enough" load, vinum trashes the kernel stack in such 
a way that debugging is very tough.  I have been talking to a number
of people about it in -STABLE.  I will continue to debug this on the
new news server I'm building (before it gets commissioned), but work
on this particular server had to halt with the instalation of the
MegaRAID.

Brad>   With luck, in about a month or so, I should be getting a new
Brad> server in with 1GB RAM, 450Mhz Pentium III w/ 1MB L2 cache, a
Brad> DPT SmartRAID V controller with 256MB ECC cache, and eight
Brad> Fujitsu MAE3182LC 7200RPM 18GB drives for RAID-5 configuration
Brad> on a single SCSI bus, for our new anonymous ftp server.

I got the MegaRAID 1400 because the DPT V drivers weren't available.
The MegaRAID should be roughly equivlanet to the DPT V.  Do go with
LVD if you can.

I have done benchmarking with bonnie instead of rawIO.  The output is
as follows:

1 process, for vinum:

  ---Sequential Output ---Sequential Input-- --Random--
  -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
MachineMB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
raid-1512  4554 20.0  4583  6.3  4495  8.3 18596 80.1 30001 25.9 446.0  5.3

1 process, for MegaRAID:

  ---Sequential Output ---Sequential Input-- --Random--
  -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
MachineMB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
mega  512  7419 32.4  7529 10.2  4653  9.6 14616 63.4 14942 18.8 345.4  5.0

... now the MegaRAID was at 16.5M/s block read with a different stripe 
size, but I don't have that output in front of me.

Dave.

-- 

|David Gilbert, Velocet Communications.   | Two things can only be |
|Mail:   [EMAIL PROTECTED] |  equal if and only if they |
|http://www.velocet.net/~dgilbert |   are precisely opposite.  |
=GLO


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: AMI MegaRAID datapoint.

[Brad Knowles]

At 10:52 AM -0500 1999/12/16, David Gilbert wrote:

>  Now... using vinum and either the 2940U2W (Adaptec LVD) or the TekRAM
>  (NCR) LVD (using the sym0 device) gives 30 to 35 M/s under RAID-5.

That's really interesting, because there are at least two or 
three outstanding bugs in the vinum RAID-5 implementation that have 
prevented me from successfully benchmarking it (see 
).

Have you been using rawio?  Have you made any attempt to try 
duplicating some of the sorts of benchmarks I've done, available at 
?

>  This is impressive and subject to the bug that I mentioned in -STABLE
>  which still hasn't been found.

Which one is this?

>  The AMI MegaRAID 1400 delivers between 16.5 and 19 M/s (the 19M/s
>  value is somewhat contrived --- using 8 bonnies in parrallel and then
>  summing their results --- which is not 100% valid)... but the MegaRAID
>  appears to be stable.

If you look at the page I mentioned, you'll see that my best 
sustained speed with a vinum 4-way stripe on IBM 10kRPM 9LZX drives 
attached to an Adaptec 2940UW controller was about 19MB/s sequential 
read (anywhere from four to 128 processes), 16.5MB/s random read 
(64-256 processes), 14MB/s sequential write (16 processes), and 
15MB/s random write (64-256 processes).

My best performance with a DPT SmartRAID IV in a 4-way stripe 
with the same disks was 16.5MB/s sequential read (4 processes), 
7.5MB/s random read (pretty much regardless of how many processes), 
17MB/s sequential write (256 processes), and 6MB/s (independent of 
the number of processes).  I did not attempt to benchmark DPT 
SmartRAID IV performance under RAID-5.


I'd be very interested to see more extensive benchmarking of this 
configuration.  In addition to my page above, I'd recommend you look 
at  and 
, and see which of 
these you can throw at your system.


With luck, in about a month or so, I should be getting a new 
server in with 1GB RAM, 450Mhz Pentium III w/ 1MB L2 cache, a DPT 
SmartRAID V controller with 256MB ECC cache, and eight Fujitsu 
MAE3182LC 7200RPM 18GB drives for RAID-5 configuration on a single 
SCSI bus, for our new anonymous ftp server.

I know this isn't an ideal configuration (I shouldn't have more 
than four devices on a SCSI bus, and I should be using faster drives 
with lower latency), but it will be very interesting to test out this 
configuration with 3.4-RELEASE and what should hopefully be stable 
DPT SmartRAID V/VI drivers by then.  I plan on beating the crap out 
of this machine before it goes online.  ;-)

-- 
Brad Knowles <[EMAIL PROTECTED]> 
 

Your mouse has moved.   Windows NT must be restarted for the change to
take effect.   Reboot now?  [ OK ]


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

AMI MegaRAID datapoint.

[David Gilbert]

I have the Enterprise 1400 Megaraid adapter with (currently 16M) on
it.  I have tested the various modes of operation (different raid
levels and striping) and find it to be working well.  My LVD array
consists of 8 18G Quamtum IV's.

Now... using vinum and either the 2940U2W (Adaptec LVD) or the TekRAM
(NCR) LVD (using the sym0 device) gives 30 to 35 M/s under RAID-5.
This is impressive and subject to the bug that I mentioned in -STABLE
which still hasn't been found.

The AMI MegaRAID 1400 delivers between 16.5 and 19 M/s (the 19M/s
value is somewhat contrived --- using 8 bonnies in parrallel and then
summing their results --- which is not 100% valid)... but the MegaRAID 
appears to be stable.

I would like to know, however, if there are any planned or
in-the-works utility programs for the amr device.  In particular, a
program to print the state of the array would be useful.

Dave.

-- 

|David Gilbert, Velocet Communications.   | Two things can only be |
|Mail:   [EMAIL PROTECTED] |  equal if and only if they |
|http://www.velocet.net/~dgilbert |   are precisely opposite.  |
=GLO


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message