francis picabia wrote the following on 1/25/2013 7:55 AM:
On Thu, Jan 24, 2013 at 12:22 PM, Blake Hudson <bl...@ispn.net
<mailto:bl...@ispn.net>> wrote:
There are a couple suggestions I'd like to put forth. First, improper
partition alignment is generally masked by the controller cache. I
strongly encourage you to check that your RAID array is making use of
this cache by enabling the WriteBack caching option on this array,
especially if your PERC card has a BBU (I think this was optional on
perc 5). You can install the MegaCLI tool from LSI to verify this (can
also be checked from OpenManage or reboot into the controller BIOS).
Thanks for this tip. It put me on to what is wrong.
Jan 18 07:25:39 myserv Server Administrator: Storage Service EventID:
2335 Controller event log: BBU disabled; changing WB virtual disks to
WT: Controller 0 (PERC 5/i Integrated)
Bingo! We had write back all along, and the performance tanked when it
fell back to write through. I was wondering why my policy change attempts
were flipping back when I tried testing WB this morning!
This explains everything we've been seeing. Wow. Gotta call Dell.
Thanks everyone for the assistance. I didn't think a battery which
shows OK
status in omreport could wound us!
The PERC cards will disable write-back caching while the BBU is
charging/exercising. However, within a few hours the BBU should return
to normal status. In rare instances, people on the Dell mailing list
have reported that their caching status never returns to write-back -
even after attempting to force write-back caching on the array. Attempts
and power cycling or firmware flashing are tried, but seem to be futile
in most cases. Often, replacement of the card is necessary. I'm unsure
if it's the battery, the card, or some software setting, but I would
definitely follow up with Dell.
On the next server (or array) you configure, I would attempt to align
your partitions as you've investigated. Sector 2048 seems to be a good
starting position for most RAID levels. I have no conclusive evidence
that a different file system or alignment improves my performance,
because I've never done a fair side by side test with controlled inputs.
However, we use ext4 and do align our partitions using RAID10 on 15k SAS
drives for all our Cyrus installs. I have found some issues with the
newer systems that I attribute to the move from ext3 to ext4 which can
result in MySQL replication problems on power loss/freeze, but these
issues are vary rare and usually easy to recover from in our
environment. I also notice that new systems always perform better than
the old systems, even with identical hardware - I've often attributed
this to fragmentation.
--Blake
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus