Re: Why is intr taking up so much cpu?

2010-07-21 Thread Andriy Gapon
Doug, could you please show your timer configuration, part of devinfo -u that describes interrupts and top of the output of top -SPH (including the header) when high interrupt load strikes? P.S. I saw output of top -SH, but I have a reason to be curious about top -SPH. -- Andriy Gapon

Re: Why is intr taking up so much cpu?

2010-07-21 Thread Doug Barton
On Wed, 21 Jul 2010, Andriy Gapon wrote: Doug, could you please show your timer configuration, Nothing special in /boot/loader.conf, /etc/sysctl.conf, or my kernel. It's basically just GENERIC minus devices I don't have, plus the following: options DDB_CTF options VESA

Re: Why is intr taking up so much cpu?

2010-07-21 Thread Andriy Gapon
on 21/07/2010 21:50 Doug Barton said the following: On Wed, 21 Jul 2010, Andriy Gapon wrote: Doug, could you please show your timer configuration, Nothing special in /boot/loader.conf, /etc/sysctl.conf, or my kernel. It's basically just GENERIC minus devices I don't have, plus the

Re: Why is intr taking up so much cpu?

2010-07-21 Thread Doug Barton
On Wed, 21 Jul 2010, Andriy Gapon wrote: I didn't mean your manual tuning, I meant how the system is configured :-) E.g. the relevant sysctl tree. Duh. :) Sorry. sysctl -a | grep timer kern.eventtimer.choice: LAPIC(500) HPET(450) HPET1(440) HPET2(440) i8254(100) RTC(0)

Re: Why is intr taking up so much cpu?

2010-07-20 Thread Doug Barton
On Sun, 18 Jul 2010, Dan Nelson wrote: You can also use dtrace to get a count of callouts and their time spent. Run this for a few seconds then hit ^C: Okey dokey, here you go: http://people.freebsd.org/~dougb/normal-dtrace.txt http://people.freebsd.org/~dougb/bad-dtrace.txt Thanks again,

Re: Why is intr taking up so much cpu?

2010-07-20 Thread Doug Barton
On Sun, 18 Jul 2010, Kostik Belousov wrote: When intr time starts accumulating again, try to do procstat -kk intr process pid and correlate the clock thread tid with the backtrace. Might be, it helps to guess what callouts are eating the CPU. Ok, I thought I was going to be able to do this

Re: Why is intr taking up so much cpu?

2010-07-20 Thread Kostik Belousov
On Mon, Jul 19, 2010 at 11:05:26PM -0700, Doug Barton wrote: On Sun, 18 Jul 2010, Kostik Belousov wrote: When intr time starts accumulating again, try to do procstat -kk intr process pid and correlate the clock thread tid with the backtrace. Might be, it helps to guess what callouts are

Re: Why is intr taking up so much cpu?

2010-07-20 Thread Dan Nelson
In the last episode (Jul 19), Doug Barton said: On Sun, 18 Jul 2010, Dan Nelson wrote: You can also use dtrace to get a count of callouts and their time spent. Run this for a few seconds then hit ^C: Okey dokey, here you go: http://people.freebsd.org/~dougb/normal-dtrace.txt

Re: Why is intr taking up so much cpu?

2010-07-19 Thread Kostik Belousov
On Sun, Jul 18, 2010 at 10:06:06PM -0700, Doug Barton wrote: On 07/18/10 12:41, Kostik Belousov wrote: On Sun, Jul 18, 2010 at 12:21:00PM -0700, Doug Barton wrote: On 07/18/10 03:30, Kostik Belousov wrote: On Sun, Jul 18, 2010 at 01:14:41AM -0700, Doug Barton wrote: On Sat, 17 Jul 2010,

Re: Why is intr taking up so much cpu?

2010-07-19 Thread Doug Barton
I added options KDTRACE_HOOKS to my kernel config, built a new kernel, and rebooted. I decided to try your script before things went sideways so I'd have an idea of what to expect, and it didn't work: dtrace: failed to initialize dtrace: DTrace device not available on system Is there

Re: Why is intr taking up so much cpu?

2010-07-19 Thread Chris Ruiz
On Mon, Jul 19, 2010 at 8:03 PM, Doug Barton do...@freebsd.org wrote: I added options KDTRACE_HOOKS to my kernel config, built a new kernel, and rebooted. I decided to try your script before things went sideways so I'd have an idea of what to expect, and it didn't work: dtrace: failed to

Re: Why is intr taking up so much cpu?

2010-07-19 Thread Doug Barton
On Mon, 19 Jul 2010, Chris Ruiz wrote: On Mon, Jul 19, 2010 at 8:03 PM, Doug Barton do...@freebsd.org wrote: I added options KDTRACE_HOOKS to my kernel config, built a new kernel, and rebooted. I decided to try your script before things went sideways so I'd have an idea of what to expect, and

Re: Why is intr taking up so much cpu?

2010-07-19 Thread Navdeep Parhar
On Mon, Jul 19, 2010 at 07:33:01PM -0700, Doug Barton wrote: On Mon, 19 Jul 2010, Chris Ruiz wrote: On Mon, Jul 19, 2010 at 8:03 PM, Doug Barton do...@freebsd.org wrote: I added options KDTRACE_HOOKS to my kernel config, built a new kernel, and rebooted. I decided to try your script before

Re: Why is intr taking up so much cpu?

2010-07-19 Thread Max Laier
On Tuesday 20 July 2010 04:33:01 Doug Barton wrote: On Mon, 19 Jul 2010, Chris Ruiz wrote: On Mon, Jul 19, 2010 at 8:03 PM, Doug Barton do...@freebsd.org wrote: I added options KDTRACE_HOOKS to my kernel config, built a new kernel, and rebooted. I decided to try your script before things

Re: Why is intr taking up so much cpu?

2010-07-19 Thread Doug Barton
On Tue, 20 Jul 2010, Max Laier wrote: Just a stab in the dark, did you kldload dtraceall? KDTRACE_HOOKS just adds the needed linkage for the dtrace modules to work. No, I had not done that, in fact, I didn't even know I needed those modules. I use MODULES_OVERRIDE so I had to add dtrace,

Re: Why is intr taking up so much cpu?

2010-07-19 Thread Dan Nelson
In the last episode (Jul 19), Doug Barton said: I added options KDTRACE_HOOKS to my kernel config, built a new kernel, and rebooted. I decided to try your script before things went sideways so I'd have an idea of what to expect, and it didn't work: dtrace: failed to initialize dtrace:

Re: Why is intr taking up so much cpu?

2010-07-18 Thread Doug Barton
On Sat, 17 Jul 2010, Kostik Belousov wrote: Run top in the mode where all system threads are shown separately (e.g. top -HS seems to do it), then watch what thread eats the processor. And the winner is! 11 root -32- 0K 168K WAIT0 0:28 18.02% {swi4: clock} 11 root

Re: Why is intr taking up so much cpu?

2010-07-18 Thread Bernd Walter
On Sat, Jul 17, 2010 at 10:21:28PM +0300, Kostik Belousov wrote: On Sat, Jul 17, 2010 at 12:10:26PM -0700, Doug Barton wrote: On Sat, 17 Jul 2010, Rui Paulo wrote: This doesn't indicate any problem. I suggest you try to figure out what interrupt is causing this by adding printfs or

Re: Why is intr taking up so much cpu?

2010-07-18 Thread Kostik Belousov
On Sun, Jul 18, 2010 at 01:14:41AM -0700, Doug Barton wrote: On Sat, 17 Jul 2010, Kostik Belousov wrote: Run top in the mode where all system threads are shown separately (e.g. top -HS seems to do it), then watch what thread eats the processor. And the winner is! 11 root -32

Re: Why is intr taking up so much cpu?

2010-07-18 Thread Doug Barton
On 07/18/10 03:30, Kostik Belousov wrote: On Sun, Jul 18, 2010 at 01:14:41AM -0700, Doug Barton wrote: On Sat, 17 Jul 2010, Kostik Belousov wrote: Run top in the mode where all system threads are shown separately (e.g. top -HS seems to do it), then watch what thread eats the processor. And

Re: Why is intr taking up so much cpu?

2010-07-18 Thread Kostik Belousov
On Sun, Jul 18, 2010 at 12:21:00PM -0700, Doug Barton wrote: On 07/18/10 03:30, Kostik Belousov wrote: On Sun, Jul 18, 2010 at 01:14:41AM -0700, Doug Barton wrote: On Sat, 17 Jul 2010, Kostik Belousov wrote: Run top in the mode where all system threads are shown separately (e.g. top -HS

Re: Why is intr taking up so much cpu?

2010-07-18 Thread Doug Barton
On 07/18/10 12:41, Kostik Belousov wrote: When intr time starts accumulating again, try to do procstat -kk intr process pid and correlate the clock thread tid with the backtrace. Might be, it helps to guess what callouts are eating the CPU. Will do, thanks! Doug -- ... and that's

Re: Why is intr taking up so much cpu?

2010-07-18 Thread Dan Nelson
In the last episode (Jul 18), Doug Barton said: On 07/18/10 12:41, Kostik Belousov wrote: When intr time starts accumulating again, try to do procstat -kk intr process pid and correlate the clock thread tid with the backtrace. Might be, it helps to guess what callouts are eating the CPU.

Re: Why is intr taking up so much cpu?

2010-07-18 Thread Dan Nelson
In the last episode (Jul 18), Dan Nelson said: In the last episode (Jul 18), Doug Barton said: On 07/18/10 12:41, Kostik Belousov wrote: When intr time starts accumulating again, try to do procstat -kk intr process pid and correlate the clock thread tid with the backtrace. Might be, it

Re: Why is intr taking up so much cpu?

2010-07-18 Thread Doug Barton
On 07/18/10 12:41, Kostik Belousov wrote: On Sun, Jul 18, 2010 at 12:21:00PM -0700, Doug Barton wrote: On 07/18/10 03:30, Kostik Belousov wrote: On Sun, Jul 18, 2010 at 01:14:41AM -0700, Doug Barton wrote: On Sat, 17 Jul 2010, Kostik Belousov wrote: Run top in the mode where all system

Why is intr taking up so much cpu?

2010-07-17 Thread Doug Barton
This is happening after I open a flash video in firefox and watch it for 15 minutes: root 20 -80- 0K 160K WAIT0 3:38 14.08% intr After this happens, my system goes into a death spiral and I have to shut it down. vmstat -i interrupt total

Re: Why is intr taking up so much cpu?

2010-07-17 Thread Doug Barton
On Sat, 17 Jul 2010, Rui Paulo wrote: On 17 Jul 2010, at 08:17, Doug Barton wrote: This is happening after I open a flash video in firefox and watch it for 15 minutes: root 20 -80- 0K 160K WAIT0 3:38 14.08% intr After this happens, my system goes into a death spiral and

Re: Why is intr taking up so much cpu?

2010-07-17 Thread Rui Paulo
On 17 Jul 2010, at 19:04, Doug Barton wrote: On Sat, 17 Jul 2010, Rui Paulo wrote: On 17 Jul 2010, at 08:17, Doug Barton wrote: This is happening after I open a flash video in firefox and watch it for 15 minutes: root 20 -80- 0K 160K WAIT0 3:38 14.08% intr After

Re: Why is intr taking up so much cpu?

2010-07-17 Thread Rui Paulo
On 17 Jul 2010, at 08:17, Doug Barton wrote: This is happening after I open a flash video in firefox and watch it for 15 minutes: root 20 -80- 0K 160K WAIT0 3:38 14.08% intr After this happens, my system goes into a death spiral and I have to shut it down. vmstat

Re: Why is intr taking up so much cpu?

2010-07-17 Thread Doug Barton
On Sat, 17 Jul 2010, Rui Paulo wrote: This doesn't indicate any problem. I suggest you try to figure out what interrupt is causing this by adding printfs or disabling drivers one by one. I've no idea where to even begin on something like that. Given that there are other -current users who

Re: Why is intr taking up so much cpu?

2010-07-17 Thread Rui Paulo
On 17 Jul 2010, at 20:10, Doug Barton wrote: On Sat, 17 Jul 2010, Rui Paulo wrote: This doesn't indicate any problem. I suggest you try to figure out what interrupt is causing this by adding printfs or disabling drivers one by one. I've no idea where to even begin on something like

Re: Why is intr taking up so much cpu?

2010-07-17 Thread Doug Barton
On Sat, 17 Jul 2010, Rui Paulo wrote: You can try bisecting the faulty revision. The problem has been going on for months, the primary symptom for a long time was the nvidia driver, so I stopped using it for a while hoping that a solution would magically appear. As of the last 6 weeks or so

Re: Why is intr taking up so much cpu?

2010-07-17 Thread Kostik Belousov
On Sat, Jul 17, 2010 at 12:10:26PM -0700, Doug Barton wrote: On Sat, 17 Jul 2010, Rui Paulo wrote: This doesn't indicate any problem. I suggest you try to figure out what interrupt is causing this by adding printfs or disabling drivers one by one. I've no idea where to even begin on

Re: Why is intr taking up so much cpu?

2010-07-17 Thread Doug Barton
On Sat, 17 Jul 2010, Kostik Belousov wrote: Note that intr time most likely come from the interrupt threads chewing the CPU, not from the real interrupt handlers doing something, and definitely not due to the high interrupt rate, as your vmstat -i output already shown. Run top in the mode