Re[2]: Weird system cpu usage
Hello Mike, Wednesday, March 19, 2008, 9:32:13 PM, you wrote: >>Also, I believe there was a report from another user who saw similar >>issues with em(4), and found that disabling MSI fixed the storm in >>question. I believe you can disable MSI/MSIX by placing the following >>in /boot/loader.conf, then reboot: >> >>hw.pci.enable_msi="0" >>hw.pci.enable_msix="0" > When MSI is enabled, the irq will be a strangely high number. e.g. Interesting, I have disabled MSI and MSIX support, but they still share the same irq. However, I don't know yet if the interrupt storm is going to be resolved, it needs some time (It always used to take some time until it has showed up). > If anything, I found enabling MSI helped matters where I saw strange > IRQ issues. However, not sure if the original poster's hardware > supports it. One thing it does remind me of is some strange IRQ > issues I had on an AMD board where a USB setting for "legacy handoff" > (something like that) would really slow down the machine with an in > inordinate amount of IRQs firing. I forget if I had to enable it or > disable it to fix the problem. If anything, I would try disabling > USB all together if its not being used even though its not figuring > in the above really high rate of IRQs. The USB isn't indeed used, I will think about disabling it, and also about trying to move around the em(4) NICs to the other slots. Unfortunately I don't have a physical access to the given maschine. -- Best regards, Danielmailto:[EMAIL PROTECTED] ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Weird system cpu usage
At 02:06 PM 3/19/2008, Jeremy Chadwick wrote: On Wed, Mar 19, 2008 at 06:53:36PM +0100, Oliver Fromme wrote: > Charlie Root <[EMAIL PROTECTED]> wrote: > > [EMAIL PROTECTED] ~]# vmstat -i > > interrupt total rate > > irq1: atkbd0 12 0 > > irq16: ohci0 1 0 > > irq17: ohci1 ohci3 1 0 > > irq18: ohci2 ohci4 1 0 > > irq20: em0 86255835 1361 > > irq22: em1 atapci0 18611379049 293795 > > Now that looks unusual indeed. Do you get that rate > on irq22 right after boot, before the services have > started? It looks like either hardware or driver > problems. Do you have polling enabled on em1? Also, I believe there was a report from another user who saw similar issues with em(4), and found that disabling MSI fixed the storm in question. I believe you can disable MSI/MSIX by placing the following in /boot/loader.conf, then reboot: hw.pci.enable_msi="0" hw.pci.enable_msix="0" When MSI is enabled, the irq will be a strangely high number. e.g. % vmstat -i interrupt total rate irq4: sio076 0 irq17: em3 360 0 irq19: atapci1 2901 0 cpu0: timer 33719800 1999 irq257: em156571 3 irq258: em24 0 cpu1: timer 33717664 1999 Total 67497376 4003 If anything, I found enabling MSI helped matters where I saw strange IRQ issues. However, not sure if the original poster's hardware supports it. One thing it does remind me of is some strange IRQ issues I had on an AMD board where a USB setting for "legacy handoff" (something like that) would really slow down the machine with an in inordinate amount of IRQs firing. I forget if I had to enable it or disable it to fix the problem. If anything, I would try disabling USB all together if its not being used even though its not figuring in the above really high rate of IRQs. ---Mike ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re[2]: Weird system cpu usage
Hello Oliver, Wednesday, March 19, 2008, 6:53:36 PM, you wrote: > Charlie Root <[EMAIL PROTECTED]> wrote: >> [...] > It's preferable to send mail as a real user, not as root, > for various reasons. I know, I've just forgot to edit the headers, my apologies. >> I have to report, that I have a very strange cpu usage by system (as >> the `top' reports). > You haven't mentioned what exactly you think is strange > in your top(1) output. I think it looks pretty normal > under the given circumstances. What do you mean by "given" circumstances? I don't think that 50+% cpu usage by system is that normal. >> [EMAIL PROTECTED] ~]# vmstat -i >> interrupt total rate >> irq1: atkbd0 12 0 >> irq16: ohci0 1 0 >> irq17: ohci1 ohci3 1 0 >> irq18: ohci2 ohci4 1 0 >> irq20: em0 86255835 1361 >> irq22: em1 atapci0 18611379049 293795 > Now that looks unusual indeed. Do you get that rate > on irq22 right after boot, before the services have > started? It looks like either hardware or driver > problems. Do you have polling enabled on em1? No, the rate slowly increases after the system boots. Polling is disabled at the moment. -- Best regards, Danielmailto:[EMAIL PROTECTED] ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Weird system cpu usage
On Wed, Mar 19, 2008 at 06:53:36PM +0100, Oliver Fromme wrote: > Charlie Root <[EMAIL PROTECTED]> wrote: > > [EMAIL PROTECTED] ~]# vmstat -i > > interrupt total rate > > irq1: atkbd0 12 0 > > irq16: ohci0 1 0 > > irq17: ohci1 ohci3 1 0 > > irq18: ohci2 ohci4 1 0 > > irq20: em0 86255835 1361 > > irq22: em1 atapci0 18611379049 293795 > > Now that looks unusual indeed. Do you get that rate > on irq22 right after boot, before the services have > started? It looks like either hardware or driver > problems. Do you have polling enabled on em1? Also, I believe there was a report from another user who saw similar issues with em(4), and found that disabling MSI fixed the storm in question. I believe you can disable MSI/MSIX by placing the following in /boot/loader.conf, then reboot: hw.pci.enable_msi="0" hw.pci.enable_msix="0" -- | Jeremy Chadwickjdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Weird system cpu usage
Charlie Root <[EMAIL PROTECTED]> wrote: > [...] It's preferable to send mail as a real user, not as root, for various reasons. > I have to report, that I have a very strange cpu usage by system (as > the `top' reports). You haven't mentioned what exactly you think is strange in your top(1) output. I think it looks pretty normal under the given circumstances. > [EMAIL PROTECTED] ~]# vmstat -i > interrupt total rate > irq1: atkbd0 12 0 > irq16: ohci0 1 0 > irq17: ohci1 ohci3 1 0 > irq18: ohci2 ohci4 1 0 > irq20: em0 86255835 1361 > irq22: em1 atapci0 18611379049 293795 Now that looks unusual indeed. Do you get that rate on irq22 right after boot, before the services have started? It looks like either hardware or driver problems. Do you have polling enabled on em1? Best regards Oliver -- Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd "Being really good at C++ is like being really good at using rocks to sharpen sticks." -- Thant Tessman ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Weird system cpu usage
Hello, I have to report, that I have a very strange cpu usage by system (as the `top' reports). The given box does not currently run any threaded applications (only lighttpd and php-fcgi with 80 children, maybe 100 reqs/s), but I can see the same behavior on almost identical machine which is running mysqld (Threads: 100 Questions: 36641597 Slow queries: 620 Opens: 1139 Flush tables: 1 Open tables: 1112 Queries per second avg: 584.349). Attached, I am providing dmesg.boot of the given system and an output of `top -CSaIud 10'. I have to note, that the system does not have I/O problems, the only thing that might be related is an interrupt strom on irq22, which is being shared by em1 and atapci0 (but the top's interrupt column doesn't report too high interrupt cpu usage). Both of the machines have this interrupt problem. [EMAIL PROTECTED] ~]# vmstat -i interrupt total rate irq1: atkbd0 12 0 irq16: ohci0 1 0 irq17: ohci1 ohci3 1 0 irq18: ohci2 ohci4 1 0 irq20: em0 86255835 1361 irq22: em1 atapci0 18611379049 293795 cpu0: timer126702021 2000 cpu1: timer126701933 2000 Total18951038853 299157 Any hints on what might be going on are welcome. Thank you. -- Sincerely, Daniel Gerzolast pid: 15637; load averages: 5.04, 5.16, 6.19 up 0+17:16:4009:36:52 166 processes: 3 running, 148 sleeping, 15 waiting Mem: 317M Active, 740M Inact, 249M Wired, 48M Cache, 214M Buf, 2554M Free Swap: 8192M Total, 8K Used, 8192M Free PIDUIDTHR PRI NICE SIZERES STATE C TIMECPU COMMAND 10 0 1 171 ki31 0K16K RUN1 501:19 33.98% [idle: cpu1] 11 0 1 171 ki31 0K16K CPU0 0 518:21 30.96% [idle: cpu0] 1425 80 1 40 39968K 21568K kqread 0 20:48 2.49% /usr/local/sbin/lighttpd -f /usr/local/etc/lighttpd.conf 15425 80 1 40 379M 24952K accept 1 0:20 1.37% /usr/local/bin/php-cgi 15472 80 1 40 379M 24496K accept 1 0:18 1.37% /usr/local/bin/php-cgi 15502 80 1 40 379M 24420K accept 1 0:17 1.37% /usr/local/bin/php-cgi 15450 80 1 40 378M 24828K accept 1 0:19 1.27% /usr/local/bin/php-cgi 15474 80 1 40 379M 24684K accept 1 0:19 1.27% /usr/local/bin/php-cgi 15452 80 1 40 379M 25160K sbwait 0 0:18 1.27% /usr/local/bin/php-cgi 15471 80 1 40 378M 24300K accept 0 0:18 1.27% /usr/local/bin/php-cgi 15482 80 1 40 379M 25844K accept 1 0:18 1.27% /usr/local/bin/php-cgi 15480 80 1 40 378M 23488K accept 1 0:18 1.27% /usr/local/bin/php-cgi 15503 80 1 40 379M 24580K sbwait 1 0:17 1.27% /usr/local/bin/php-cgi 15506 80 1 40 378M 23048K accept 1 0:16 1.27% /usr/local/bin/php-cgi 15512 80 1 40 378M 23200K sbwait 0 0:16 1.27% /usr/local/bin/php-cgi 15423 80 1 40 379M 23368K accept 1 0:22 1.17% /usr/local/bin/php-cgi 15426 80 1 40 378M 22864K accept 1 0:21 1.17% /usr/local/bin/php-cgi 15442 80 1 40 378M 23204K accept 0 0:19 1.17% /usr/local/bin/php-cgi last pid: 15637; load averages: 5.04, 5.16, 6.19 up 0+17:16:4209:36:54 166 processes: 4 running, 147 sleeping, 15 waiting CPU states: 16.4% user, 0.0% nice, 44.2% system, 0.4% interrupt, 39.1% idle Mem: 317M Active, 740M Inact, 249M Wired, 48M Cache, 214M Buf, 2554M Free Swap: 8192M Total, 8K Used, 8192M Free PIDUIDTHR PRI NICE SIZERES STATE C TIMECPU COMMAND 10 0 1 171 ki31 0K16K RUN1 501:20 35.35% [idle: cpu1] 11 0 1 171 ki31 0K16K RUN0 518:22 32.18% [idle: cpu0] 1425 80 1 40 39968K 21568K kqread 0 20:48 2.59% /usr/local/sbin/lighttpd -f /usr/local/etc/lighttpd.conf 15502 80 1 40 379M 24420K accept 1 0:17 1.46% /usr/local/bin/php-cgi 15450 80 1 40 378M 24828K accept 1 0:19 1.27% /usr/local/bin/php-cgi 15474 80 1 40 379M 24684K accept 0 0:19 1.27% /usr/local/bin/php-cgi 15472 80 1 40 379M 24536K accept 0 0:18 1.27% /usr/local/bin/php-cgi 15471 80 1 40 378M 24300K accept 1 0:18 1.27% /usr/local/bin/php-cgi 15506 80 1 40 379M 23172K accept 1 0:17 1.27% /usr/local/bin/php-cgi 15425 80 1 40 379M 24952K sbwait 0 0:20 1.17% /usr/local/bin/php-cgi 15442 80 1 40 378M 23204K accept 1 0:19 1.17% /usr/local/bin/php-cgi 15458 80 1 40 378M 23680K accept 1 0:18 1.17% /usr/local/bin/php-cgi 15461 80 1 40 379M 24676K accept 0 0:18 1.17% /usr/
