Re: Still having problems with sata_promise on all late kernels
Brad Campbell wrote: > G'day all, > > This has been happening on a lightly loaded box for a while now. I have > posted about it before. > The issue is now I've upgraded 3 drives from 250GB Maxtors to 750GB > WD's, it's just getting worse, particularly when the drives are loaded. > I've jumpered the WD's to 1.5GB PHY speed (I had to do that or the > on-board via sata controller would not recognise one of the drives. > The box is a bit of a mish-mash of controllers. I have 3 Promise > SATA150TX4, one SIL3112 (all on PCI cards) and the on-board 2 port VIA > driver. > > I'm at a bit of a loss as to what to do really, short of replacing the > Promise cards with something else. ATA bus error is a strong indicator for hardware problem. Please get a separate power supply (doesn't have to be an expensive one), power it up [1], and connect some of the drives to the power supply and see whether errors persist on those drives. -- tejun [1] http://modtown.co.uk/mt/article2.php?id=psumod - To unsubscribe from this list: send the line "unsubscribe linux-ide" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Still having problems with sata_promise on all late kernels
Tejun Heo wrote: Brad Campbell wrote: G'day all, This has been happening on a lightly loaded box for a while now. I have posted about it before. The issue is now I've upgraded 3 drives from 250GB Maxtors to 750GB WD's, it's just getting worse, particularly when the drives are loaded. I've jumpered the WD's to 1.5GB PHY speed (I had to do that or the on-board via sata controller would not recognise one of the drives. The box is a bit of a mish-mash of controllers. I have 3 Promise SATA150TX4, one SIL3112 (all on PCI cards) and the on-board 2 port VIA driver. I'm at a bit of a loss as to what to do really, short of replacing the Promise cards with something else. ATA bus error is a strong indicator for hardware problem. Please get a separate power supply (doesn't have to be an expensive one), power it up [1], and connect some of the drives to the power supply and see whether errors persist on those drives. Hrm, I did try that a while back. I'll give that a bash again tonight. This is only happening on the Promise cards. The drives on the SIL and VIA controllers will work as hard as I care to work them and never fault. I'll try and do a GIT bisection this evening, If I go back to 2.6.16 I get no errors and the system works perfectly well. Same can be said for the failure to shut down actually. 2.6.16 worked flawlessly. Brad -- "Human beings, who are almost unique in having the ability to learn from the experience of others, are also remarkable for their apparent disinclination to do so." -- Douglas Adams - To unsubscribe from this list: send the line "unsubscribe linux-ide" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Still having problems with sata_promise on all late kernels
Tejun Heo wrote: ATA bus error is a strong indicator for hardware problem. Please get a separate power supply (doesn't have to be an expensive one), power it up [1], and connect some of the drives to the power supply and see whether errors persist on those drives. 2 separate tests with the 2 troublesome drives, each time with both drives on a different power supply unit. Also tried new cables from a different manufacturer. It ain't the cables and it ain't the power. Regards, Brad -- "Human beings, who are almost unique in having the ability to learn from the experience of others, are also remarkable for their apparent disinclination to do so." -- Douglas Adams - To unsubscribe from this list: send the line "unsubscribe linux-ide" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Still having problems with sata_promise on all late kernels
[cc'ing Mikael Pettersson] Brad Campbell wrote: > Tejun Heo wrote: > >> ATA bus error is a strong indicator for hardware problem. Please get a >> separate power supply (doesn't have to be an expensive one), power it up >> [1], and connect some of the drives to the power supply and see whether >> errors persist on those drives. >> > > 2 separate tests with the 2 troublesome drives, each time with both > drives on a different power supply unit. Also tried new cables from a > different manufacturer. > > It ain't the cables and it ain't the power. I see. Thanks. I'm pretty much out of ideas at this point. Mikael? -- tejun - To unsubscribe from this list: send the line "unsubscribe linux-ide" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Still having problems with sata_promise on all late kernels
On Wed, 15 Aug 2007 16:54:52 +0900, Tejun Heo wrote: >Brad Campbell wrote: >> Tejun Heo wrote: >> >>> ATA bus error is a strong indicator for hardware problem. Please get a >>> separate power supply (doesn't have to be an expensive one), power it up >>> [1], and connect some of the drives to the power supply and see whether >>> errors persist on those drives. >>> >> >> 2 separate tests with the 2 troublesome drives, each time with both >> drives on a different power supply unit. Also tried new cables from a >> different manufacturer. >> >> It ain't the cables and it ain't the power. > >I see. Thanks. I'm pretty much out of ideas at this point. Mikael? The box was described as a KT600 chipset era machine with 3 Promise SATA150TX4 cards, a SIL3112 card, and an on-board VIA controller. According to the `lspci' posted the box probably also has an AGP card and at least one gigaether card (there's two, I guess one might be built-in). All in all, a rather heavily loaded (in terms of power, cooling, and the PCI bus) box (for a desktop chipset). I've never seen the SATA150-generation cards misbehave, so at this point I have to assume that it's a system limit issue, be it PCI, power, or cooling. /Mikael - To unsubscribe from this list: send the line "unsubscribe linux-ide" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Still having problems with sata_promise on all late kernels
Mikael Pettersson wrote: The box was described as a KT600 chipset era machine with 3 Promise SATA150TX4 cards, a SIL3112 card, and an on-board VIA controller. According to the `lspci' posted the box probably also has an AGP card and at least one gigaether card (there's two, I guess one might be built-in). One PCI and one on-board. Otherwise quite an accurate assessment. All in all, a rather heavily loaded (in terms of power, cooling, and the PCI bus) box (for a desktop chipset). Yes, cooling is not an issue. The motherboard reports 28-29C, the processor is 32-34C and the drives all report between 32-38C. The machine has an excessive number of fans in it and the drives live in Supermicro hotswap bays with 90mm high speed noise makers strapped to the back. The machines live in a separate air conditioned and UPS'd environment which is regulated at 20C. I've put a thermocouple in the machine and I never see temperatures on any of the hardware exceeding 40C (Graphics card heatsink) The PSU is a 4 x 12v rail Thermaltake 600W jobbie with the 12v rails re-wired to feed mainly the drives (as one 18A rail would not do 15 drives all spinning up together). The PSU is now identical to the one fitted to its sister box. I've measured the voltages all within tolerance and the fully loaded ripple is less than 50mv on +5 and all +12 rails. That looks pretty good to me. This is the second PSU the box has seen as I just upgraded it from the old 420W that was in there. I have tried splitting the drives over 2 PSU's (several different combinations of drives and PSU's) and the problem still exists. I've never seen the SATA150-generation cards misbehave, so at this point I have to assume that it's a system limit issue, be it PCI, power, or cooling. It may well be a PCI bus issue however it's sister box which is exactly the same mainboard has 15 drives on 4 SATA150 cards and it has never had such a problem. It just locks up on shutdown :) I will note that this machine has run (except for the lower wattage PSU) in this configuration since about 2.6.9 and sat perfectly happy with 2.6.16 until I needed to upgrade for some reason around 2.6.21-rc (which is when the bus errors started). I'll try some more conservative PCI settings in the BIOS and see what happens. Thanks for the pointers anyway. Brad -- "Human beings, who are almost unique in having the ability to learn from the experience of others, are also remarkable for their apparent disinclination to do so." -- Douglas Adams - To unsubscribe from this list: send the line "unsubscribe linux-ide" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Still having problems with sata_promise on all late kernels - shutdown too
Brad Campbell wrote: G'day all, This has been happening on a lightly loaded box for a while now. I have posted about it before. The issue is now I've upgraded 3 drives from 250GB Maxtors to 750GB WD's, it's just getting worse, particularly when the drives are loaded. Oh, I forgot, I still have the problem where I can't shut the box down. The SCSI layer tries to sync cache on the drives, the 1st one errors out, and when the controller tries a hard reset on the port the box locks hard requiring a hard poweroff or a hard reset. (No serial console, sysrq, NMI oopser or anything else works) I have 2 boxes with sata_promise controllers and they both do this. Not surprising given they are both running late kernels (> 2.6.20) and both have almost exactly the same hardware in them. http://www.fnarfbargle.com/CIMG1029.JPG (I have posted this screen shot previously. It's still doing exactly the same thing on both boxes). I've played with options to halt and all the other things I can think of. I just can't get it to shut down cleanly and all 15 drives go *clunk* when I have to hard power off the box. Normally this is not an issue as it happens once or twice a year at most, but recently I've been doing some server room reconfiguration and have had to do a number of power cycles. Nothing like 30 drives across 2 machines going *clunk* to make your stomach turn. Brad -- "Human beings, who are almost unique in having the ability to learn from the experience of others, are also remarkable for their apparent disinclination to do so." -- Douglas Adams - To unsubscribe from this list: send the line "unsubscribe linux-ide" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Still having problems with sata_promise on all late kernels - shutdown too - BISECTED
Brad Campbell wrote: http://www.fnarfbargle.com/CIMG1029.JPG (I have posted this screen shot previously. It's still doing exactly the same thing on both boxes). git-bisect shows up this commit as the guilty party. I can't just revert it on a recent tree though. Will investigate further. [EMAIL PROTECTED]:~/git/linux-2.6$ git-bisect good 13b8d09f5de0aaa3153bbccc98baf247387823dc is first bad commit commit 13b8d09f5de0aaa3153bbccc98baf247387823dc Author: Tejun Heo <[EMAIL PROTECTED]> Date: Tue May 15 12:29:22 2007 +0200 libata: track spindown status and skip spindown_compat if possible Our assumption that most distros issue STANDBYNOW seems wrong. The upstream sysvinit and thus many distros including gentoo and opensuse don't take any action for libata disks on spindown. We can skip compat handling for these distros so that they don't need to update anything to take advantage of kernel-side shutdown. Signed-off-by: Tejun Heo <[EMAIL PROTECTED]> Signed-off-by: Jeff Garzik <[EMAIL PROTECTED]> :04 04 6084e11e2d53daf2506d16896d0aa5635f95a9af cd30a7aad2a6a6af63779797889d4d9eed52e473 M drivers :04 04 75e3a561488e2902eed8f197da960e55fbd88f27 a7f586a83645bedbce4b4286c2e9dbcfd69f8f86 M include Brad -- "Human beings, who are almost unique in having the ability to learn from the experience of others, are also remarkable for their apparent disinclination to do so." -- Douglas Adams - To unsubscribe from this list: send the line "unsubscribe linux-ide" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Still having problems with sata_promise on all late kernels - shutdown too - BISECTED
Brad Campbell wrote: > Brad Campbell wrote: > >> http://www.fnarfbargle.com/CIMG1029.JPG (I have posted this screen >> shot previously. It's still doing exactly the same thing on both boxes). >> > > git-bisect shows up this commit as the guilty party. I can't just revert > it on a recent tree though. > Will investigate further. Weird. Does 'hdparm -y /dev/sdX' work? -- tejun - To unsubscribe from this list: send the line "unsubscribe linux-ide" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Still having problems with sata_promise on all late kernels - shutdown too - BISECTED
Tejun Heo wrote: Brad Campbell wrote: Brad Campbell wrote: http://www.fnarfbargle.com/CIMG1029.JPG (I have posted this screen shot previously. It's still doing exactly the same thing on both boxes). git-bisect shows up this commit as the guilty party. I can't just revert it on a recent tree though. Will investigate further. Weird. Does 'hdparm -y /dev/sdX' work? Very interesting indeed.. for i in /dev/sd* ; do echo $i ; hdparm -y $i ; done sda-sdh worked fine.. sdi locked the box. So it's dying on the 3rd or 4th controller. Now that I think about it, on the other side of the git bisect I probably only heard 8 clicks as the kernel shut down the 1st 8 drives, then the remainder powered off when the machine powered off. So even prior to this it was probably only shutting down the 1st 8 drives properly. [EMAIL PROTECTED]:~$ lspci 00:00.0 Host bridge: VIA Technologies, Inc. VT8377 [KT400/KT600 AGP] Host Bridge (rev 80) 00:01.0 PCI bridge: VIA Technologies, Inc. VT8237 PCI Bridge 00:09.0 Ethernet controller: 3Com Corporation 3c940 10/100/1000Base-T [Marvell] (rev 12) 00:0b.0 Mass storage controller: Promise Technology, Inc. PDC20318 (SATA150 TX4) (rev 02) 00:0d.0 Mass storage controller: Promise Technology, Inc. PDC20318 (SATA150 TX4) (rev 02) 00:0e.0 Mass storage controller: Promise Technology, Inc. PDC20318 (SATA150 TX4) (rev 02) 00:0f.0 RAID bus controller: VIA Technologies, Inc. VIA VT6420 SATA RAID Controller (rev 80) 00:0f.1 IDE interface: VIA Technologies, Inc. VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus Master IDE (rev 06) 00:10.0 USB Controller: VIA Technologies, Inc. VT82x UHCI USB 1.1 Controller (rev 81) 00:10.1 USB Controller: VIA Technologies, Inc. VT82x UHCI USB 1.1 Controller (rev 81) 00:10.2 USB Controller: VIA Technologies, Inc. VT82x UHCI USB 1.1 Controller (rev 81) 00:10.3 USB Controller: VIA Technologies, Inc. VT82x UHCI USB 1.1 Controller (rev 81) 00:10.4 USB Controller: VIA Technologies, Inc. USB 2.0 (rev 86) 00:11.0 ISA bridge: VIA Technologies, Inc. VT8237 ISA bridge [KT600/K8T800/K8T890 South] 00:11.5 Multimedia audio controller: VIA Technologies, Inc. VT8233/A/8235/8237 AC97 Audio Controller (rev 60) 00:11.6 Communication controller: VIA Technologies, Inc. AC'97 Modem Controller (rev 80) 00:13.0 Mass storage controller: Promise Technology, Inc. PDC20318 (SATA150 TX4) (rev 02) 01:00.0 VGA compatible controller: Silicon Integrated Systems [SiS] 315PRO PCI/AGP VGA Display Adapter [ 46.624922] sata_promise :00:0b.0: version 2.07 [ 46.624937] ACPI: PCI Interrupt :00:0b.0[A] -> GSI 19 (level, low) -> IRQ 17 [ 46.641073] scsi0 : sata_promise [ 46.641301] scsi1 : sata_promise [ 46.641471] scsi2 : sata_promise [ 46.648281] scsi3 : sata_promise [ 46.654924] ata1: SATA max UDMA/133 cmd 0xf8816200 ctl 0xf8816238 bmdma 0x irq 0 [ 46.661784] ata2: SATA max UDMA/133 cmd 0xf8816280 ctl 0xf88162b8 bmdma 0x irq 0 [ 46.668432] ata3: SATA max UDMA/133 cmd 0xf8816300 ctl 0xf8816338 bmdma 0x irq 0 [ 46.674895] ata4: SATA max UDMA/133 cmd 0xf8816380 ctl 0xf88163b8 bmdma 0x irq 0 [ 47.170019] ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300) [ 47.190471] ata1.00: ata_hpa_resize 1: sectors = 490234752, hpa_sectors = 490234752 [ 47.197090] ata1.00: ATA-7: Maxtor 7Y250M0, YAR511W0, max UDMA/133 [ 47.203704] ata1.00: 490234752 sectors, multi 0: LBA48 [ 47.230410] ata1.00: ata_hpa_resize 1: sectors = 490234752, hpa_sectors = 490234752 [ 47.237100] ata1.00: configured for UDMA/133 [ 47.729427] ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 300) [ 47.749866] ata2.00: ata_hpa_resize 1: sectors = 490234752, hpa_sectors = 490234752 [ 47.756760] ata2.00: ATA-7: Maxtor 7Y250M0, YAR511W0, max UDMA/133 [ 47.763727] ata2.00: 490234752 sectors, multi 0: LBA48 [ 47.789820] ata2.00: ata_hpa_resize 1: sectors = 490234752, hpa_sectors = 490234752 [ 47.796966] ata2.00: configured for UDMA/133 [ 48.288835] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 300) [ 48.309277] ata3.00: ata_hpa_resize 1: sectors = 490234752, hpa_sectors = 490234752 [ 48.316381] ata3.00: ATA-7: Maxtor 7Y250M0, YAR511W0, max UDMA/133 [ 48.323494] ata3.00: 490234752 sectors, multi 0: LBA48 [ 48.349233] ata3.00: ata_hpa_resize 1: sectors = 490234752, hpa_sectors = 490234752 [ 48.356411] ata3.00: configured for UDMA/133 [ 48.848245] ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 300) [ 48.868684] ata4.00: ata_hpa_resize 1: sectors = 490234752, hpa_sectors = 490234752 [ 48.875960] ata4.00: ATA-7: Maxtor 7Y250M0, YAR511W0, max UDMA/133 [ 48.883230] ata4.00: 490234752 sectors, multi 0: LBA48 [ 48.908634] ata4.00: ata_hpa_resize 1: sectors = 490234752, hpa_sectors = 490234752 [ 48.916000] ata4.00: configured for UDMA/133 [ 48.923402] scsi 0:0:0:0: Direct-Access ATA Maxtor 7Y250M0 YAR5 PQ: 0 ANSI: 5 [ 48.931040] sd 0:0:0:0: [sda] 490234752 512-byte hardware secto
Re: Still having problems with sata_promise on all late kernels - shutdown too - BISECTED
Brad Campbell wrote: > Tejun Heo wrote: >> Brad Campbell wrote: >>> Brad Campbell wrote: >>> http://www.fnarfbargle.com/CIMG1029.JPG (I have posted this screen shot previously. It's still doing exactly the same thing on both boxes). >>> git-bisect shows up this commit as the guilty party. I can't just revert >>> it on a recent tree though. >>> Will investigate further. >> >> Weird. Does 'hdparm -y /dev/sdX' work? > > Very interesting indeed.. > > for i in /dev/sd* ; do echo $i ; hdparm -y $i ; done > > sda-sdh worked fine.. sdi locked the box. > > So it's dying on the 3rd or 4th controller. That's weird. You seem to have the same controllers and drives. What happens if you spin down from hdi upwards? -- tejun - To unsubscribe from this list: send the line "unsubscribe linux-ide" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Still having problems with sata_promise on all late kernels - shutdown too - BISECTED
Tejun Heo wrote: Brad Campbell wrote: Tejun Heo wrote: Brad Campbell wrote: Brad Campbell wrote: http://www.fnarfbargle.com/CIMG1029.JPG (I have posted this screen shot previously. It's still doing exactly the same thing on both boxes). git-bisect shows up this commit as the guilty party. I can't just revert it on a recent tree though. Will investigate further. Weird. Does 'hdparm -y /dev/sdX' work? Very interesting indeed.. for i in /dev/sd* ; do echo $i ; hdparm -y $i ; done sda-sdh worked fine.. sdi locked the box. So it's dying on the 3rd or 4th controller. That's weird. You seem to have the same controllers and drives. What happens if you spin down from hdi upwards? Any attempt to spin down sdi-sdo in any order locks hard. Any attempt to spin down sda-sdh works in any combination/permutation. Yes, all controllers and drives are identical. Brad -- "Human beings, who are almost unique in having the ability to learn from the experience of others, are also remarkable for their apparent disinclination to do so." -- Douglas Adams - To unsubscribe from this list: send the line "unsubscribe linux-ide" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Still having problems with sata_promise on all late kernels - shutdown too - BISECTED
Brad Campbell wrote: > Tejun Heo wrote: >> Brad Campbell wrote: >>> Tejun Heo wrote: Brad Campbell wrote: > Brad Campbell wrote: > >> http://www.fnarfbargle.com/CIMG1029.JPG (I have posted this screen >> shot previously. It's still doing exactly the same thing on both >> boxes). >> > git-bisect shows up this commit as the guilty party. I can't just > revert > it on a recent tree though. > Will investigate further. Weird. Does 'hdparm -y /dev/sdX' work? >>> Very interesting indeed.. >>> >>> for i in /dev/sd* ; do echo $i ; hdparm -y $i ; done >>> >>> sda-sdh worked fine.. sdi locked the box. >>> >>> So it's dying on the 3rd or 4th controller. >> >> That's weird. You seem to have the same controllers and drives. What >> happens if you spin down from hdi upwards? >> > > Any attempt to spin down sdi-sdo in any order locks hard. Any attempt to > spin down sda-sdh works in any combination/permutation. Hmmm... I'm out of ideas. It definitely sounds like some sort of hardware problem to me. I think the best way to diagnose the problem is playing the tedious but often rewarding add-remove-swap game with the hardware and see to which piece or combination of pieces the problem is attached. Thanks. -- tejun - To unsubscribe from this list: send the line "unsubscribe linux-ide" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Still having problems with sata_promise on all late kernels - shutdown too - BISECTED
Tejun Heo wrote: Any attempt to spin down sdi-sdo in any order locks hard. Any attempt to spin down sda-sdh works in any combination/permutation. Hmmm... I'm out of ideas. It definitely sounds like some sort of hardware problem to me. I think the best way to diagnose the problem is playing the tedious but often rewarding add-remove-swap game with the hardware and see to which piece or combination of pieces the problem is attached. The problem appears to be having more than 2 Promise cards in the system. Both machines have >2 cards (one has 3 and the other has 4) and they both lock up on shutdown in the same fashion (first 8 drives are ok, then it locks). Between this and the intermittent BUS Errors under load, perhaps I toss the promise cards and upgrade to some 4 port SIL cards instead. Brad -- "Human beings, who are almost unique in having the ability to learn from the experience of others, are also remarkable for their apparent disinclination to do so." -- Douglas Adams - To unsubscribe from this list: send the line "unsubscribe linux-ide" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
