Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

On 06/13/2016 04:57 PM, cpol...@surewest.net wrote: you might want to try the very most recent kernel in git to see if your problem is fixed. Is there a specific change that is of interest? ___ CentOS mailing list CentOS@centos.org

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

On 2016-06-01 20:07, Kelly Lesperance wrote: > Software RAID 10. Servers are HP DL380 Gen 8s, with 12x4 TB 7200 RPM drives. > > On 2016-06-01, 3:52 PM, "centos-boun...@centos.org on behalf of > m.r...@5-cent.us" > wrote: > > >Kelly

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

Software RAID 10. Servers are HP DL380 Gen 8s, with 12x4 TB 7200 RPM drives. On 2016-06-01, 3:52 PM, "centos-boun...@centos.org on behalf of m.r...@5-cent.us" wrote: >Kelly Lesperance wrote: >> I did some additional testing - I stopped

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

Kelly Lesperance wrote: > I did some additional testing - I stopped Kafka on the host, and kicked > off a disk check, and it ran at the expected speed overnight. I started > kafka this morning, and the raid check's speed immediately dropped down to > ~2000K/Sec. > > I then enabled the write-back

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

I did some additional testing - I stopped Kafka on the host, and kicked off a disk check, and it ran at the expected speed overnight. I started kafka this morning, and the raid check's speed immediately dropped down to ~2000K/Sec. I then enabled the write-back cache on the drives (hdparm -W1

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

All of our Kafka clusters are fairly write-heavy. The cluster in question is our second-heaviest – we haven’t yet upgraded the heaviest, due to the issues we’ve been experiencing in this one. Here is an iostat example from a host within the same cluster, but without the RAID check running:

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

On 05/25/2016 09:54 AM, Kelly Lesperance wrote: What we're seeing is that when the weekly raid-check script executes, performance nose dives, and I/O wait skyrockets. The raid check starts out fairly fast (2K/sec - the limit that's been set), but then quickly drops down to about

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

Hi Charles, Looks to me like all of the drives are performing roughly the same – there’s certainly not 1 that sticks out (also note this is happening on all 23 nodes in the cluster). Thanks! Kelly [root@r1k1.kafka.log10.blackberry sys] # cat /proc/mdstat Personalities : [raid10]

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

On 5/25/2016 12:20 PM, m.r...@5-cent.us wrote: Also, you could either, on boot, go into the card's firmware interface, and that'll tell you, somewhere, what the firmware version is. Not sure if MegaRAID will work with this card - if it does, you really want it..even though it has an actively

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

On 2016-05-25 19:13, Kelly Lesperance wrote: > Hdparm didn’t get far: > > [root@r1k1 ~] # hdparm -tT /dev/sda > > /dev/sda: > Timing cached reads: Alarm clock > [root@r1k1 ~] # Hi Kelly, Try running 'iostat -xdmc 1'. Look for a single drive that has substantially greater await than ~10msec.

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

Kelly Lesperance wrote: > Already done – they’re not being very helpful, as we don’t have a support > contract, just standard warranty. > Right. We get support for five years, but then we keep things well past that, we don't get rid of them till they're dying. (Don't talk to me about "wasting tax

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

I should rephrase that – some parts of HP are helping us, but the team I opened the case with isn’t being very helpful. On 2016-05-25, 4:29 PM, "Kelly Lesperance" wrote: >Already done – they’re not being very helpful, as we don’t have a support >contract, just

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

Already done – they’re not being very helpful, as we don’t have a support contract, just standard warranty. On 2016-05-25, 4:27 PM, "centos-boun...@centos.org on behalf of m.r...@5-cent.us" wrote: >Kelly Lesperance wrote: >>

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

Kelly Lesperance wrote: > LSI/Avago’s web pages don’t have any downloads for the SAS2308, so I think > I’m out of luck wrt MegaRAID. > > Bounced the node, confirmed MPT Firmware 15.10.09.00-IT. > HP Driver is v 15.10.04.00. > > Both are the latest from HP. > > Unsure why, but the module itself

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

LSI/Avago’s web pages don’t have any downloads for the SAS2308, so I think I’m out of luck wrt MegaRAID. Bounced the node, confirmed MPT Firmware 15.10.09.00-IT. HP Driver is v 15.10.04.00. Both are the latest from HP. Unsure why, but the module itself reports version 20.100.00.00: [root@r1k1

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

I installed the latest firmware and driver (mpt2sas) from HP on one system. The driver is v20, it appears the firmware may be 15, though: [ 11.128979] mpt2sas version 20.100.00.00 loaded [ 11.513836] mpt2sas0: LSISAS2308: FWVersion(15.10.09.00), ChipRevision(0x05), BiosVersion(07.39.00.00)

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

John R Pierce wrote: > On 5/25/2016 11:44 AM, Kelly Lesperance wrote: >> The HBA is an HP H220. > > OH.its a very good idea to verify the driver is at the same revision > level as the firmware.not 100% sure how you do this under CentOS, my > H220 system is running FreeBSD, and is at

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

Hdparm didn’t get far: [root@r1k1 ~] # hdparm -tT /dev/sda /dev/sda: Timing cached reads: Alarm clock [root@r1k1 ~] # On 2016-05-25, 2:44 PM, "Kelly Lesperance" wrote: >The HBA is an HP H220. > >We haven’t really benchmarked individual drives – all 12 drives are

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

On 5/25/2016 11:44 AM, Kelly Lesperance wrote: The HBA is an HP H220. OH.its a very good idea to verify the driver is at the same revision level as the firmware.not 100% sure how you do this under CentOS, my H220 system is running FreeBSD, and is at revision P20, both firmware and

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

On 5/25/2016 11:44 AM, Kelly Lesperance wrote: The HBA is an HP H220. for the uninitated, thats a LSI SAS2308, in IT (initiator-terminator) mode. -- john r pierce, recycling bits in santa cruz ___ CentOS mailing list CentOS@centos.org

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

The HBA is an HP H220. We haven’t really benchmarked individual drives – all 12 drives are utilized in one RAID-10 array, I’m unsure how we would test individual drives without breaking the array. Trying ‘hdparm -tT /dev/sda’ now – it’s been running for 25 minutes so far… Kelly On

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

What is the HBA the drives are attached to? Have you done a quick benchmark on a single disk to check if this is a raid problem or further down the stack? Regards, Dennis On 25.05.2016 19:26, Kelly Lesperance wrote: > [merging] > > The HBA the drives are attached to has no configuration that

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

Kelly Lesperance wrote: > I’ve posted this on the forums at > https://www.centos.org/forums/viewtopic.php?f=47=57926=244614#p244614 > - posting to the list in the hopes of getting more eyeballs on it. > > We have a cluster of 23 HP DL380p Gen8 hosts running Kafka. Basic specs: > > 2x E5-2650 > 128

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

[merging] The HBA the drives are attached to has no configuration that I’m aware of. We would have had to accidentally change 23 of them ☺ Thanks, Kelly On 2016-05-25, 1:25 PM, "Kelly Lesperance" wrote: >They are: > >[root@r1k1 ~] # hdparm -I /dev/sda >

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

They are: [root@r1k1 ~] # hdparm -I /dev/sda /dev/sda: ATA device, with non-removable media Model Number: MB4000GCWDC Serial Number: S1Z06RW9 Firmware Revision: HPGD Transport: Serial, SATA Rev 3.0

Re: [CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

Kelly Lesperance wrote: > I’ve posted this on the forums at > https://www.centos.org/forums/viewtopic.php?f=47=57926=244614#p244614 > - posting to the list in the hopes of getting more eyeballs on it. > > We have a cluster of 23 HP DL380p Gen8 hosts running Kafka. Basic specs: > > 2x E5-2650 > 128

[CentOS] Slow RAID Check/high %iowait during check after updgrade from CentOS 6.5 -> CentOS 7.2

I’ve posted this on the forums at https://www.centos.org/forums/viewtopic.php?f=47=57926=244614#p244614 - posting to the list in the hopes of getting more eyeballs on it. We have a cluster of 23 HP DL380p Gen8 hosts running Kafka. Basic specs: 2x E5-2650 128 GB RAM 12 x 4 TB 7200 RPM SATA