Re: 157k interrupts per second causing 60% CPU load on idle system
On Apr 16, 2012 5:42 AM, "Ronald Klop" wrote: > > On Sun, 15 Apr 2012 15:09:34 +0200, Matt Thyer wrote: > >> On Apr 15, 2012 6:27 PM, "Ronald Klop" wrote: >>> >>> The problem does not occur with 9-STABLE. Who knows what the problem was ? USB maybe ? >>> >>> >>> >>> Do you still have the same hardware on the same interrupts on 9-STABLE? >>> Are there changes in the use of MSI(-X)? >>> >> I made no hardware or BIOS changes and I'm running a GENERIC kernel in all >> testing. > > > That does not mean FreeBSD 9 can't put devices on other interrupts than 8 did. > 'dmesg | grep irq' like you did before might show a difference with your previous output. > I'm just guessing here for a clue on the result you are seeing, but without any data I cannot answer you (and I guess nobody can). > Ronald, The irqs seem to be the same: $ grep irq\ 16 /var/run/dmesg.boot pcib1: irq 16 at device 1.0 on pci0 mps0: port 0xee00-0xeeff mem 0xfbdfc000-0xfbdf,0xfbd8-0xfbdb irq 16 at device 0.0 on pci1 vgapci0: port 0xff00-0xff07 mem 0xfb40-0xfb7f,0xe000-0xefff irq 16 at device 2.0 on pci0 uhci0: port 0xfe00-0xfe1f irq 16 at device 26.0 on pci0 pcib2: irq 16 at device 28.0 on pci0 pcib3: irq 16 at device 28.4 on pci0 atapci0: port 0xdf00-0xdf07,0xde00-0xde03,0xd00-0xdd07,0xdc00-0xdc03,0xdb00-0xdb0f irq 16 at device 0.0 on pci3 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 157k interrupts per second causing 60% CPU load on idle system
On Sun, 15 Apr 2012 15:09:34 +0200, Matt Thyer wrote: On Apr 15, 2012 6:27 PM, "Ronald Klop" wrote: The problem does not occur with 9-STABLE. Who knows what the problem was ? USB maybe ? Do you still have the same hardware on the same interrupts on 9-STABLE? Are there changes in the use of MSI(-X)? I made no hardware or BIOS changes and I'm running a GENERIC kernel in all testing. That does not mean FreeBSD 9 can't put devices on other interrupts than 8 did. 'dmesg | grep irq' like you did before might show a difference with your previous output. I'm just guessing here for a clue on the result you are seeing, but without any data I cannot answer you (and I guess nobody can). Ronald. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 157k interrupts per second causing 60% CPU load on idle system
On Apr 15, 2012 6:27 PM, "Ronald Klop" wrote: > >> The problem does not occur with 9-STABLE. >> >> Who knows what the problem was ? USB maybe ? > > > Do you still have the same hardware on the same interrupts on 9-STABLE? > Are there changes in the use of MSI(-X)? > I made no hardware or BIOS changes and I'm running a GENERIC kernel in all testing. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 157k interrupts per second causing 60% CPU load on idle system
On Sat, 14 Apr 2012 01:13:30 +0200, Matt Thyer wrote: On Apr 7, 2012 2:38 PM, "Matt Thyer" wrote: On 7 April 2012 14:31, Matt Thyer wrote: Since moving the SATA 3 disk to the onboard Intel SATA 2 controller I'm no longer having that disk evicted from the raidz2 pool with write errors and I thought that the high interrupt rate issue had also been solved but it's back again. This is on 8-STABLE at revision 230921 (before the new driver hit 8-STABLE). So now I need to go back to trying to determine what the cause is. vmstat -i has shown that the issue was on irq 16. Unfortunately there seems to be a lot of things on irq 16: $ dmesg | grep "irq 16" pcib1: irq 16 at device 1.0 on pci0 mps0: port 0xee00-0xeeff mem 0xfbdfc000-0xfbdf,0xfbd8-0xfbdb irq 16 at device 0.0 on pci1 vgapci0: port 0xff00-0xff07 mem 0xfb40-0xfb7f,0xe000-0xefff irq 16 at device 2.0 on pci0 uhci0: port 0xfe00-0xfe1f irq 16 at device 26.0 on pci0 pcib2: irq 16 at device 28.0 on pci0 pcib3: irq 16 at device 28.4 on pci0 atapci0: port 0xdf00-0xdf07,0xde00-0xde03,0xdd00-0xdd07,0xdc00-0xdc03,0xdb00-0xdb0f irq 16 at device 0.0 on pci3 Any idea how to isolate which bit of hardware could be triggering the interrupts ? Unfortunately the only device I could remove would be the SuperMicro AOC-USAS2-L8i (so yes I could eliminate that). My biggest problem right now is not knowing how to trigger the issue. At this stage I'm going to upgrade to 9-STABLE and see if it returns. The problem does not occur with 9-STABLE. Who knows what the problem was ? USB maybe ? Do you still have the same hardware on the same interrupts on 9-STABLE? Are there changes in the use of MSI(-X)? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 157k interrupts per second causing 60% CPU load on idle system
On Apr 7, 2012 2:38 PM, "Matt Thyer" wrote: > > On 7 April 2012 14:31, Matt Thyer wrote: >> Since moving the SATA 3 disk to the onboard Intel SATA 2 controller I'm no longer having that disk evicted from the raidz2 pool with write errors and I thought that the high interrupt rate issue had also been solved but it's back again. >> >> This is on 8-STABLE at revision 230921 (before the new driver hit 8-STABLE). >> >> So now I need to go back to trying to determine what the cause is. >> > vmstat -i has shown that the issue was on irq 16. > > Unfortunately there seems to be a lot of things on irq 16: > > $ dmesg | grep "irq 16" > > pcib1: irq 16 at device 1.0 on pci0 > mps0: port 0xee00-0xeeff mem 0xfbdfc000-0xfbdf,0xfbd8-0xfbdb irq 16 at device 0.0 on pci1 > vgapci0: port 0xff00-0xff07 mem 0xfb40-0xfb7f,0xe000-0xefff irq 16 at device 2.0 on pci0 > uhci0: port 0xfe00-0xfe1f irq 16 at device 26.0 on pci0 > pcib2: irq 16 at device 28.0 on pci0 > pcib3: irq 16 at device 28.4 on pci0 > atapci0: port 0xdf00-0xdf07,0xde00-0xde03,0xdd00-0xdd07,0xdc00-0xdc03,0xdb00-0xdb0f irq 16 at device 0.0 on pci3 > > Any idea how to isolate which bit of hardware could be triggering the interrupts ? > > Unfortunately the only device I could remove would be the SuperMicro AOC-USAS2-L8i (so yes I could eliminate that). > > My biggest problem right now is not knowing how to trigger the issue. > > At this stage I'm going to upgrade to 9-STABLE and see if it returns. The problem does not occur with 9-STABLE. Who knows what the problem was ? USB maybe ? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 157k interrupts per second causing 60% CPU load on idle system
On 7 April 2012 14:31, Matt Thyer wrote: > On 5 April 2012 01:18, Freddie Cash wrote: > >> On Wed, Apr 4, 2012 at 5:19 AM, Matt Thyer wrote: >> > So it seems that both the old and new mps driver have a problem with the >> > Western Digital WD20EARX SATA 3 drive on a SuperMicro AOC-USAS2-L8i (SAS >> > 6G) controller (flashed with -IT firmware). >> >> I wouldn't say the driver has a problem with that specific drive. >> More that it might have a problem with a mixed SATA2/SATA3 setup. >> >> Sorry, that's what I meant to say but it now seems that the 157K > interrupts per second is probably not due to the SuperMicro AOC-USAS2-L8i. > > Since moving the SATA 3 disk to the onboard Intel SATA 2 controller I'm no > longer having that disk evicted from the raidz2 pool with write errors and > I thought that the high interrupt rate issue had also been solved but it's > back again. > > This is on 8-STABLE at revision 230921 (before the new driver hit > 8-STABLE). > > So now I need to go back to trying to determine what the cause is. > > I'll stop posting in this thread as I don't think it's anything to do with > either the old or new version of this driver. > Oops... wrong thread I thought I was replying in -CURRENT. So on to the root cause. vmstat -i has shown that the issue was on irq 16. Unfortunately there seems to be a lot of things on irq 16: $ dmesg | grep "irq 16" pcib1: irq 16 at device 1.0 on pci0 mps0: port 0xee00-0xeeff mem 0xfbdfc000-0xfbdf,0xfbd8-0xfbdb irq 16 at device 0.0 on pci1 vgapci0: port 0xff00-0xff07 mem 0xfb40-0xfb7f,0xe000-0xefff irq 16 at device 2.0 on pci0 uhci0: port 0xfe00-0xfe1f irq 16 at device 26.0 on pci0 pcib2: irq 16 at device 28.0 on pci0 pcib3: irq 16 at device 28.4 on pci0 atapci0: port 0xdf00-0xdf07,0xde00-0xde03,0xdd00-0xdd07,0xdc00-0xdc03,0xdb00-0xdb0f irq 16 at device 0.0 on pci3 pcib1: irq 16 at device 1.0 on pci0 mps0: port 0xee00-0xeeff mem 0xfbdfc000-0xfbdf,0xfbd8-0xfbdb irq 16 at device 0.0 on pci1 vgapci0: port 0xff00-0xff07 mem 0xfb40-0xfb7f,0xe000-0xefff irq 16 at device 2.0 on pci0 uhci0: port 0xfe00-0xfe1f irq 16 at device 26.0 on pci0 pcib2: irq 16 at device 28.0 on pci0 pcib3: irq 16 at device 28.4 on pci0 atapci0: port 0xdf00-0xdf07,0xde00-0xde03,0xdd00-0xdd07,0xdc00-0xdc03,0xdb00-0xdb0f irq 16 at device 0.0 on pci3 Any idea how to isolate which bit of hardware could be triggering the interrupts ? Unfortunately the only device I could remove would be the SuperMicro AOC-USAS2-L8i (so yes I could eliminate that). My biggest problem right now is not knowing how to trigger the issue. At this stage I'm going to upgrade to 9-STABLE and see if it returns. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 157k interrupts per second causing 60% CPU load on idle system
On 5 April 2012 01:18, Freddie Cash wrote: > On Wed, Apr 4, 2012 at 5:19 AM, Matt Thyer wrote: > > So it seems that both the old and new mps driver have a problem with the > > Western Digital WD20EARX SATA 3 drive on a SuperMicro AOC-USAS2-L8i (SAS > > 6G) controller (flashed with -IT firmware). > > I wouldn't say the driver has a problem with that specific drive. > More that it might have a problem with a mixed SATA2/SATA3 setup. > > Sorry, that's what I meant to say but it now seems that the 157K interrupts per second is probably not due to the SuperMicro AOC-USAS2-L8i. Since moving the SATA 3 disk to the onboard Intel SATA 2 controller I'm no longer having that disk evicted from the raidz2 pool with write errors and I thought that the high interrupt rate issue had also been solved but it's back again. This is on 8-STABLE at revision 230921 (before the new driver hit 8-STABLE). So now I need to go back to trying to determine what the cause is. I'll stop posting in this thread as I don't think it's anything to do with either the old or new version of this driver. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 157k interrupts per second causing 60% CPU load on idle system
On Wed, Apr 4, 2012 at 5:19 AM, Matt Thyer wrote: > So it seems that both the old and new mps driver have a problem with the > Western Digital WD20EARX SATA 3 drive on a SuperMicro AOC-USAS2-L8i (SAS > 6G) controller (flashed with -IT firmware). I wouldn't say the driver has a problem with that specific drive. More that it might have a problem with a mixed SATA2/SATA3 setup. I have a 9-STABLE box with 24x WDC WD2002FAEX SATA3 (6 Gbps) drives attached to 3 SuperMicro AOC-USAS2-8Li controllers, using the new mps(4) driver without any issues. Was actually amazed yesterday when I say it doing writes just shy of 500 MBps to the ZFS pool, via zfs send/recv from another box. No issues with excessive interrupts. Using 10.0 firmware on the controllers. -- Freddie Cash fjwc...@gmail.com ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
RE: 157k interrupts per second causing 60% CPU load on idle system
Matt, Sorry for addressing to wrong person.! I am working in Device Driver group and it difficult to track request coming through emails. We can solve those request through emails which are minor/medium in complexity. There are some group in LSI which has expertise to root cause issue and provide better support than developers. Sometimes issue can be non-driver issue (e.a Can be hw issue/fw issues etc.) and those issue need customer interaction and may be sometimes reproduction. So I request to contact LSI support channel ( at least for complex issues) to get better tracking and speed up resolution. ` Kashyap From: Matt Thyer [mailto:matt.th...@gmail.com] Sent: Wednesday, April 04, 2012 6:08 PM To: Desai, Kashyap Cc: Mike Tancsa; freebsd-stable@freebsd.org; McConnell, Stephen Subject: Re: 157k interrupts per second causing 60% CPU load on idle system On 4 April 2012 21:55, Desai, Kashyap mailto:kashyap.de...@lsi.com>> wrote: Mike, Have your purchase LSI controller through Channel or OEM ? It would be a difficult for developers to help you without any support channel invovoled ? If possible can you contact LSI support channel ? Kashyap, It's me, Matt, not Mike reporting this problem. I purchased the Super Micro card off eBay through a seller with good reputation. I've bought a couple of said cards through him now. I'm not sure why you are thinking the lack of middle men would be a problem. I hope that Super Micro & LSI would both be interested in a detailed report of a problem. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 157k interrupts per second causing 60% CPU load on idle system
On 4 April 2012 21:55, Desai, Kashyap wrote: > Mike, > Have your purchase LSI controller through Channel or OEM ? > It would be a difficult for developers to help you without any support > channel invovoled ? > If possible can you contact LSI support channel ? > Kashyap, It's me, Matt, not Mike reporting this problem. I purchased the Super Micro card off eBay through a seller with good reputation. I've bought a couple of said cards through him now. I'm not sure why you are thinking the lack of middle men would be a problem. I hope that Super Micro & LSI would both be interested in a detailed report of a problem. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
RE: 157k interrupts per second causing 60% CPU load on idle system
> -Original Message- > From: owner-freebsd-sta...@freebsd.org [mailto:owner-freebsd- > sta...@freebsd.org] On Behalf Of Matt Thyer > Sent: Wednesday, April 04, 2012 5:50 PM > To: Mike Tancsa > Cc: freebsd-stable@freebsd.org > Subject: Re: 157k interrupts per second causing 60% CPU load on idle > system > > On 25 March 2012 22:26, Matt Thyer wrote: > > > > > Does anyone know if and when this driver was merged from current to > > 8-STABLE ? > > > > If I can work out what revision that occurred in I'll go back to just > > before then to confirm if the problem exists. > > > > In the -CURRENT list I've been told that the new driver was introduced > into > 8-STABLE at revision 230921. > I reverted to that revision but the problem was still apparent. > > So I've now tried: > > - Previous BIOS > - Updating the controller firmware from phase 7 to phase 11 > - Going back to the old (pre LSI authored) mps driver > > But all to no avail. > > What has worked is to move the single SATA 3 (6 Gbps) drive (the other 7 > drives are SATA 2) to the onboard SATA 2 Intel controller. > > So it seems that both the old and new mps driver have a problem with the > Western Digital WD20EARX SATA 3 drive on a SuperMicro AOC-USAS2-L8i (SAS > 6G) controller (flashed with -IT firmware). > > I'll continue this in the -CURRENT list in the thread about the new > driver > as that's where the main discussion has been. Mike, Have your purchase LSI controller through Channel or OEM ? It would be a difficult for developers to help you without any support channel invovoled ? If possible can you contact LSI support channel ? ~ Kashyap > ___ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable- > unsubscr...@freebsd.org" ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 157k interrupts per second causing 60% CPU load on idle system
On 25 March 2012 22:26, Matt Thyer wrote: > > Does anyone know if and when this driver was merged from current to > 8-STABLE ? > > If I can work out what revision that occurred in I'll go back to just > before then to confirm if the problem exists. > In the -CURRENT list I've been told that the new driver was introduced into 8-STABLE at revision 230921. I reverted to that revision but the problem was still apparent. So I've now tried: - Previous BIOS - Updating the controller firmware from phase 7 to phase 11 - Going back to the old (pre LSI authored) mps driver But all to no avail. What has worked is to move the single SATA 3 (6 Gbps) drive (the other 7 drives are SATA 2) to the onboard SATA 2 Intel controller. So it seems that both the old and new mps driver have a problem with the Western Digital WD20EARX SATA 3 drive on a SuperMicro AOC-USAS2-L8i (SAS 6G) controller (flashed with -IT firmware). I'll continue this in the -CURRENT list in the thread about the new driver as that's where the main discussion has been. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 157k interrupts per second causing 60% CPU load on idle system
On 23 March 2012 01:16, Mike Tancsa wrote: > Sorry, what I was getting at was that a bad bios (eg latest could have > introduced a regression) can cause the symptoms you are seeing. The bios > change sure seemed to fix my problem. > I've updated the firmware of the SuperMicro AOC-USAS2-L8i to the latest -IT firmware on the Supermicro FTP site (this is version 11 and I was on 7 before) but this made no difference. I then tried downgrading the motherboard BIOS to the F3 release that I was running previously but again this made no difference. So it would seem that this is a problem is to do with a change in FreeBSD-STABLE between r225723 and r232477. I'm wondering whether this is due to the new LSI authored driver for chips such as the LSI SAS2008 that are used in the SuperMicro AOC-USAS2-L8i. I know this driver is in CURRENT but do not know if it's in 8-STABLE. Does anyone know if and when this driver was merged from current to 8-STABLE ? If I can work out what revision that occurred in I'll go back to just before then to confirm if the problem exists. Matt ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 157k interrupts per second causing 60% CPU load on idle system
On 3/22/2012 10:33 AM, Matt Thyer wrote: > > The original post tells you this. > I've already updated to the latest BIOS and this could have caused the > problem. Sorry, what I was getting at was that a bad bios (eg latest could have introduced a regression) can cause the symptoms you are seeing. The bios change sure seemed to fix my problem. In the mean time, try dmidecode to extract some of the versions of the BIOS and hardware you are using. Perhaps it might tweak someone's memory to a similar problem as yours. ---Mike -- --- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, m...@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 157k interrupts per second causing 60% CPU load on idle system
On Mar 22, 2012 10:14 AM, "Mike Tancsa" wrote: > > On 3/20/2012 1:26 AM, Matt Thyer wrote: > > I've upgraded my FreeBSD-STABLE NAS from r225723 (22nd Sept 2011) to > > r232477 (4th Mar 2012) and am finding that a system process called "intr" > > is now constantly using about 60% of 1 CPU starting a short time after > > reboot (possibly triggered by use of the samba server). > > > > When this starts, systat -vm 1 says that the system is 85% idle and 14% > > interrupt handling. > > It says that there's around 157k interrupts per second. > > > > Any idea what could be the cause ? > > This sounds like the problem I had with my intel board. BIOS update > fixed it. What chipset and MB vendor do you have ? The original post tells you this. I've already updated to the latest BIOS and this could have caused the problem. I think I should try updating the firmware of the SAS/SATA HBA but not until I've confirmed if a BIOS downgrade works. Unfortunately it's very hard to get downtime on this system so it may be some time before I can report any news. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 157k interrupts per second causing 60% CPU load on idle system
On 3/20/2012 1:26 AM, Matt Thyer wrote: > I've upgraded my FreeBSD-STABLE NAS from r225723 (22nd Sept 2011) to > r232477 (4th Mar 2012) and am finding that a system process called "intr" > is now constantly using about 60% of 1 CPU starting a short time after > reboot (possibly triggered by use of the samba server). > > When this starts, systat -vm 1 says that the system is 85% idle and 14% > interrupt handling. > It says that there's around 157k interrupts per second. > > Any idea what could be the cause ? This sounds like the problem I had with my intel board. BIOS update fixed it. What chipset and MB vendor do you have ? /usr/ports/sysutils/dmidecode is handy for digging up BIOS and chipset info http://lists.freebsd.org/pipermail/freebsd-questions/2012-March/239394.html ---Mike -- --- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, m...@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 157k interrupts per second causing 60% CPU load on idle system
On 21 March 2012 00:03, Gary Palmer wrote: > On Tue, Mar 20, 2012 at 11:10:10PM +1030, Matt Thyer wrote: > > On 20 March 2012 22:24, Ivan Voras wrote: > > > > > On 20 March 2012 12:52, Matt Thyer wrote: > > > > On 20 March 2012 21:12, Ivan Voras wrote: > > > >> > > > >> On 20/03/2012 06:26, Matt Thyer wrote: > > > >> > I've upgraded my FreeBSD-STABLE NAS from r225723 (22nd Sept 2011) > to > > > >> > r232477 (4th Mar 2012) and am finding that a system process called > > > >> > "intr" > > > >> > is now constantly using about 60% of 1 CPU starting a short time > after > > > >> > reboot (possibly triggered by use of the samba server). > > > >> > > > > >> > When this starts, systat -vm 1 says that the system is 85% idle > and > > > 14% > > > >> > interrupt handling. > > > >> > It says that there's around 157k interrupts per second. > > > >> > > > > >> > > > >> Ok, but *which* interrupt is getting triggered? Please send the > output > > > >> of "vmstat -i". > > > >> > > > >> > > > > interrupt total rate > > > > irq16: uhci0+ 3392184862 126692 > > > > > > Ok, something's probably wrong with USB. Can you disable it in BIOS? > > > > > > > > > > cpu0: timer 53549677 1999 > > > > irq256: mps0 2643187 98 > > > > irq257: re0 5508108205 > > > > irq258: ahci0 160717 6 > > > > cpu1: timer 53525300 1999 > > > > cpu2: timer 53525300 1999 > > > > cpu3: timer 53525296 1999 > > > > Total 3614622447 134999 > > > > > > > > > > > I did just update the BIOS at about the same time so the difference may > be > > due to that change. > > > > I'll try a few things such as: > > > > - Unplugging any USB things (I've only got a keyboard plugged in). > > - Downgrade BIOS. > > > > I'll get back to you all soon. > > It would be interesting to know if there are other devices on irq16 also. > > grep 'irq 16' /var/run/dmesg.boot > > I think the '+' on the irq16 line from vmstat means the interrupt is > shared, but the man page doesn't mention it so I'm not 100% sure > > Thanks, > > Gary > Good point... pcib1: irq 16 at device 1.0 on pci0 mps0: port 0xee00-0xeeff mem 0xfbdfc000-0xfbdf,0xfbd8-0xfbdb irq 16 at device 0.0 on pci1 vgapci0: port 0xff00-0xff07 mem 0xfb40-0xfb7f,0xe000-0xefff irq 16 at device 2.0 on pci0 uhci0: port 0xfe00-0xfe1f irq 16 at device 26.0 on pci0 pcib2: irq 16 at device 28.0 on pci0 pcib3: irq 16 at device 28.4 on pci0 atapci0: port 0xdf00-0xdf07,0xde00-0xde03,0xdd00-0xdd07,0xdc00-0xdc03,0xdb00-0xdb0f irq 16 at device 0.0 on pci3 I'd suspect the SAS/SATA HBA using the mps0 driver as that's where I have the raidz2 on 8 drives. Is this still the old driver or has the new LSI authored driver been added to -STABLE yet ? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 157k interrupts per second causing 60% CPU load on idle system
On Tue, Mar 20, 2012 at 11:10:10PM +1030, Matt Thyer wrote: > On 20 March 2012 22:24, Ivan Voras wrote: > > > On 20 March 2012 12:52, Matt Thyer wrote: > > > On 20 March 2012 21:12, Ivan Voras wrote: > > >> > > >> On 20/03/2012 06:26, Matt Thyer wrote: > > >> > I've upgraded my FreeBSD-STABLE NAS from r225723 (22nd Sept 2011) to > > >> > r232477 (4th Mar 2012) and am finding that a system process called > > >> > "intr" > > >> > is now constantly using about 60% of 1 CPU starting a short time after > > >> > reboot (possibly triggered by use of the samba server). > > >> > > > >> > When this starts, systat -vm 1 says that the system is 85% idle and > > 14% > > >> > interrupt handling. > > >> > It says that there's around 157k interrupts per second. > > >> > > > >> > > >> Ok, but *which* interrupt is getting triggered? Please send the output > > >> of "vmstat -i". > > >> > > >> > > > interrupt total rate > > > irq16: uhci0+ 3392184862 126692 > > > > Ok, something's probably wrong with USB. Can you disable it in BIOS? > > > > > > > cpu0: timer 53549677 1999 > > > irq256: mps0 2643187 98 > > > irq257: re0 5508108205 > > > irq258: ahci0 160717 6 > > > cpu1: timer 53525300 1999 > > > cpu2: timer 53525300 1999 > > > cpu3: timer 53525296 1999 > > > Total 3614622447 134999 > > > > > > > I did just update the BIOS at about the same time so the difference may be > due to that change. > > I'll try a few things such as: > > - Unplugging any USB things (I've only got a keyboard plugged in). > - Downgrade BIOS. > > I'll get back to you all soon. It would be interesting to know if there are other devices on irq16 also. grep 'irq 16' /var/run/dmesg.boot I think the '+' on the irq16 line from vmstat means the interrupt is shared, but the man page doesn't mention it so I'm not 100% sure Thanks, Gary ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 157k interrupts per second causing 60% CPU load on idle system
On 20 March 2012 22:24, Ivan Voras wrote: > On 20 March 2012 12:52, Matt Thyer wrote: > > On 20 March 2012 21:12, Ivan Voras wrote: > >> > >> On 20/03/2012 06:26, Matt Thyer wrote: > >> > I've upgraded my FreeBSD-STABLE NAS from r225723 (22nd Sept 2011) to > >> > r232477 (4th Mar 2012) and am finding that a system process called > >> > "intr" > >> > is now constantly using about 60% of 1 CPU starting a short time after > >> > reboot (possibly triggered by use of the samba server). > >> > > >> > When this starts, systat -vm 1 says that the system is 85% idle and > 14% > >> > interrupt handling. > >> > It says that there's around 157k interrupts per second. > >> > > >> > >> Ok, but *which* interrupt is getting triggered? Please send the output > >> of "vmstat -i". > >> > >> > > interrupt total rate > > irq16: uhci0+ 3392184862 126692 > > Ok, something's probably wrong with USB. Can you disable it in BIOS? > > > > cpu0: timer 53549677 1999 > > irq256: mps0 2643187 98 > > irq257: re0 5508108205 > > irq258: ahci0 160717 6 > > cpu1: timer 53525300 1999 > > cpu2: timer 53525300 1999 > > cpu3: timer 53525296 1999 > > Total 3614622447 134999 > > > I did just update the BIOS at about the same time so the difference may be due to that change. I'll try a few things such as: - Unplugging any USB things (I've only got a keyboard plugged in). - Downgrade BIOS. I'll get back to you all soon. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 157k interrupts per second causing 60% CPU load on idle system
On 20 March 2012 12:52, Matt Thyer wrote: > On 20 March 2012 21:12, Ivan Voras wrote: >> >> On 20/03/2012 06:26, Matt Thyer wrote: >> > I've upgraded my FreeBSD-STABLE NAS from r225723 (22nd Sept 2011) to >> > r232477 (4th Mar 2012) and am finding that a system process called >> > "intr" >> > is now constantly using about 60% of 1 CPU starting a short time after >> > reboot (possibly triggered by use of the samba server). >> > >> > When this starts, systat -vm 1 says that the system is 85% idle and 14% >> > interrupt handling. >> > It says that there's around 157k interrupts per second. >> > >> >> Ok, but *which* interrupt is getting triggered? Please send the output >> of "vmstat -i". >> >> > interrupt total rate > irq16: uhci0+ 3392184862 126692 Ok, something's probably wrong with USB. Can you disable it in BIOS? > cpu0: timer 53549677 1999 > irq256: mps0 2643187 98 > irq257: re0 5508108 205 > irq258: ahci0 160717 6 > cpu1: timer 53525300 1999 > cpu2: timer 53525300 1999 > cpu3: timer 53525296 1999 > Total 3614622447 134999 > ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 157k interrupts per second causing 60% CPU load on idle system
On 20 March 2012 21:12, Ivan Voras wrote: > On 20/03/2012 06:26, Matt Thyer wrote: > > I've upgraded my FreeBSD-STABLE NAS from r225723 (22nd Sept 2011) to > > r232477 (4th Mar 2012) and am finding that a system process called "intr" > > is now constantly using about 60% of 1 CPU starting a short time after > > reboot (possibly triggered by use of the samba server). > > > > When this starts, systat -vm 1 says that the system is 85% idle and 14% > > interrupt handling. > > It says that there's around 157k interrupts per second. > > > > Ok, but *which* interrupt is getting triggered? Please send the output > of "vmstat -i". > > > interrupt total rate irq16: uhci0+ 3392184862 126692 cpu0: timer 53549677 1999 irq256: mps0 2643187 98 irq257: re0 5508108205 irq258: ahci0 160717 6 cpu1: timer 53525300 1999 cpu2: timer 53525300 1999 cpu3: timer 53525296 1999 Total 3614622447 134999 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 157k interrupts per second causing 60% CPU load on idle system
On 20/03/2012 06:26, Matt Thyer wrote: > I've upgraded my FreeBSD-STABLE NAS from r225723 (22nd Sept 2011) to > r232477 (4th Mar 2012) and am finding that a system process called "intr" > is now constantly using about 60% of 1 CPU starting a short time after > reboot (possibly triggered by use of the samba server). > > When this starts, systat -vm 1 says that the system is 85% idle and 14% > interrupt handling. > It says that there's around 157k interrupts per second. > Ok, but *which* interrupt is getting triggered? Please send the output of "vmstat -i". signature.asc Description: OpenPGP digital signature
