Re: Processes spinning forever, apparently in lock_timer_base()?

2007-10-29 Thread Bruno Wolff III
On Fri, Sep 21, 2007 at 16:58:15 +0100, Hugh Dickins <[EMAIL PROTECTED]> wrote: > > But once I look harder at it, I wonder what would have kept > 2.6.18 to 2.6.23 safe from the same issue: per-cpu deltas from > the global vm stats too low to get synched back to global, yet > adding up to somethi

Re: Processes spinning forever, apparently in lock_timer_base()?

2007-09-22 Thread richard kennedy
On Fri, 2007-09-21 at 03:33 -0700, Andrew Morton wrote: > On Fri, 21 Sep 2007 11:25:41 +0100 richard kennedy <[EMAIL PROTECTED]> wrote: > > > > That's all a bit crappy if the wrong races happen and some other task is > > > somehow exceeding the dirty limits each time this task polls them. Seems >

Re: Processes spinning forever, apparently in lock_timer_base()?

2007-09-21 Thread Peter Zijlstra
On Fri, 21 Sep 2007 16:58:15 +0100 (BST) Hugh Dickins <[EMAIL PROTECTED]> wrote: > But once I look harder at it, I wonder what would have kept > 2.6.18 to 2.6.23 safe from the same issue: per-cpu deltas from > the global vm stats too low to get synched back to global, yet > adding up to something

Re: Processes spinning forever, apparently in lock_timer_base()?

2007-09-21 Thread Chuck Ebbert
On 09/21/2007 11:58 AM, Hugh Dickins wrote: > Looking at the 2.6.18-2.6.23 code, I'm uncertain what to try instead. > There is a refresh_vm_stats function which we could call (then retest > the break condition) just before resorting to congestion_wait. But > the big NUMA people might get very upse

Re: Processes spinning forever, apparently in lock_timer_base()?

2007-09-21 Thread Hugh Dickins
On Fri, 21 Sep 2007, Andy Whitcroft wrote: > This sounds an awful lot like the same problem I reported with fsck > hanging. I believe that Hugh had a candidate patch for that, which was > related to dirty tracking limits. It seems that that patch tested, and > acked by Peter. All on lkml under:

Re: Processes spinning forever, apparently in lock_timer_base()?

2007-09-21 Thread Chuck Ebbert
On 09/21/2007 05:39 AM, Andy Whitcroft wrote: > This sounds an awful lot like the same problem I reported with fsck > hanging. I believe that Hugh had a candidate patch for that, which was > related to dirty tracking limits. It seems that that patch tested, and > acked by Peter. All on lkml unde

Re: Processes spinning forever, apparently in lock_timer_base()?

2007-09-21 Thread richard kennedy
On Fri, 2007-09-21 at 03:33 -0700, Andrew Morton wrote: > On Fri, 21 Sep 2007 11:25:41 +0100 richard kennedy <[EMAIL PROTECTED]> wrote: > > > > That's all a bit crappy if the wrong races happen and some other task is > > > somehow exceeding the dirty limits each time this task polls them. Seems >

Re: Processes spinning forever, apparently in lock_timer_base()?

2007-09-21 Thread Andrew Morton
On Fri, 21 Sep 2007 11:25:41 +0100 richard kennedy <[EMAIL PROTECTED]> wrote: > > That's all a bit crappy if the wrong races happen and some other task is > > somehow exceeding the dirty limits each time this task polls them. Seems > > unlikely that such a condition would persist forever. > > >

Re: Processes spinning forever, apparently in lock_timer_base()?

2007-09-21 Thread richard kennedy
On Thu, 2007-09-20 at 15:36 -0700, Andrew Morton wrote: > On Thu, 20 Sep 2007 18:04:38 -0400 > Chuck Ebbert <[EMAIL PROTECTED]> wrote: > > > > > > >> Can we get some kind of band-aid, like making the endless 'for' loop in > > >> balance_dirty_pages() terminate after some number of iterations? Cle

Re: Processes spinning forever, apparently in lock_timer_base()?

2007-09-21 Thread Andy Whitcroft
This sounds an awful lot like the same problem I reported with fsck hanging. I believe that Hugh had a candidate patch for that, which was related to dirty tracking limits. It seems that that patch tested, and acked by Peter. All on lkml under: 2.6.23-rc6-mm1 -- mkfs stuck in 'D' -apw

Re: Processes spinning forever, apparently in lock_timer_base()?

2007-09-21 Thread Andrew Morton
On Fri, 21 Sep 2007 10:08:08 +0200 Matthias Hensler <[EMAIL PROTECTED]> wrote: > On Thu, Sep 20, 2007 at 03:36:54PM -0700, Andrew Morton wrote: > > That's all a bit crappy if the wrong races happen and some other task > > is somehow exceeding the dirty limits each time this task polls them. > > Se

Re: Processes spinning forever, apparently in lock_timer_base()?

2007-09-21 Thread Matthias Hensler
On Thu, Sep 20, 2007 at 03:36:54PM -0700, Andrew Morton wrote: > That's all a bit crappy if the wrong races happen and some other task > is somehow exceeding the dirty limits each time this task polls them. > Seems unlikely that such a condition would persist forever. How exactly do you define for

Re: Processes spinning forever, apparently in lock_timer_base()?

2007-09-20 Thread Chuck Ebbert
On 09/20/2007 06:36 PM, Andrew Morton wrote: > > So the question is, why do we have large amounts of dirty pages for one > disk which appear to be sitting there not getting written? > > Do we know if there's any writeout at all happening when the system is in > this state? > > I guess it's possi

Re: Processes spinning forever, apparently in lock_timer_base()?

2007-09-20 Thread Andrew Morton
On Thu, 20 Sep 2007 18:04:38 -0400 Chuck Ebbert <[EMAIL PROTECTED]> wrote: > > > >> Can we get some kind of band-aid, like making the endless 'for' loop in > >> balance_dirty_pages() terminate after some number of iterations? Clearly > >> if we haven't written "write_chunk" pages after a few trie

Re: Processes spinning forever, apparently in lock_timer_base()?

2007-09-20 Thread Chuck Ebbert
On 09/20/2007 05:29 PM, Andrew Morton wrote: > On Thu, 20 Sep 2007 17:07:15 -0400 > Chuck Ebbert <[EMAIL PROTECTED]> wrote: > >> On 08/09/2007 12:55 PM, Andrew Morton wrote: >>> On Thu, 9 Aug 2007 11:59:43 +0200 Matthias Hensler <[EMAIL PROTECTED]> >>> wrote: >>> On Sat, Aug 04, 2007 at 10:4

Re: Processes spinning forever, apparently in lock_timer_base()?

2007-09-20 Thread Andrew Morton
On Thu, 20 Sep 2007 17:07:15 -0400 Chuck Ebbert <[EMAIL PROTECTED]> wrote: > On 08/09/2007 12:55 PM, Andrew Morton wrote: > > On Thu, 9 Aug 2007 11:59:43 +0200 Matthias Hensler <[EMAIL PROTECTED]> > > wrote: > > > >> On Sat, Aug 04, 2007 at 10:44:26AM +0200, Matthias Hensler wrote: > >>> On Fri,

Re: Processes spinning forever, apparently in lock_timer_base()?

2007-09-20 Thread Chuck Ebbert
On 08/09/2007 12:55 PM, Andrew Morton wrote: > On Thu, 9 Aug 2007 11:59:43 +0200 Matthias Hensler <[EMAIL PROTECTED]> wrote: > >> On Sat, Aug 04, 2007 at 10:44:26AM +0200, Matthias Hensler wrote: >>> On Fri, Aug 03, 2007 at 11:34:07AM -0700, Andrew Morton wrote: >>> [...] >>> I am also willing to

Re: Processes spinning forever, apparently in lock_timer_base()?

2007-08-09 Thread Matthias Hensler
On Thu, Aug 09, 2007 at 09:55:34AM -0700, Andrew Morton wrote: > On Thu, 9 Aug 2007 11:59:43 +0200 Matthias Hensler <[EMAIL PROTECTED]> wrote: > > On Sat, Aug 04, 2007 at 10:44:26AM +0200, Matthias Hensler wrote: > > > On Fri, Aug 03, 2007 at 11:34:07AM -0700, Andrew Morton wrote: > > > [...] > > >

Re: Processes spinning forever, apparently in lock_timer_base()?

2007-08-09 Thread Andrew Morton
On Thu, 9 Aug 2007 11:59:43 +0200 Matthias Hensler <[EMAIL PROTECTED]> wrote: > On Sat, Aug 04, 2007 at 10:44:26AM +0200, Matthias Hensler wrote: > > On Fri, Aug 03, 2007 at 11:34:07AM -0700, Andrew Morton wrote: > > [...] > > I am also willing to try the patch posted by Richard. > > I want to gi

Re: Processes spinning forever, apparently in lock_timer_base()?

2007-08-09 Thread Matthias Hensler
On Sat, Aug 04, 2007 at 10:44:26AM +0200, Matthias Hensler wrote: > On Fri, Aug 03, 2007 at 11:34:07AM -0700, Andrew Morton wrote: > [...] > I am also willing to try the patch posted by Richard. I want to give some update here: 1. We finally hit the problem on a third system, with a total differe

Re: Processes spinning forever, apparently in lock_timer_base()?

2007-08-04 Thread Matthias Hensler
On Fri, Aug 03, 2007 at 11:34:07AM -0700, Andrew Morton wrote: > (attempting to cc Matthias. If I have the wrong one, please fix it up) You got the correct one. > > Looks like the same problem with spinlock unfairness we've seen > > elsewhere: it seems to be looping here? Or is everyone stuck >

Re: Processes spinning forever, apparently in lock_timer_base()?

2007-08-03 Thread Oleg Nesterov
Chuck Ebbert wrote: > > Looks like the same problem with spinlock unfairness we've seen > elsewhere: it seems to be looping here? Or is everyone stuck > just waiting for writeout? > > lock_timer_base(): > for (;;) { > tvec_base_t *prelock_base = timer->base; >

Re: Processes spinning forever, apparently in lock_timer_base()?

2007-08-03 Thread Andrew Morton
(attempting to cc Matthias. If I have the wrong one, please fix it up) (please generally cc reporters when forwarding their bug reports) On Wed, 01 Aug 2007 18:39:51 -0400 Chuck Ebbert <[EMAIL PROTECTED]> wrote: > Looks like the same problem with spinlock unfairness we've seen > elsewhere: it

Re: Processes spinning forever, apparently in lock_timer_base()?

2007-08-02 Thread richard kennedy
On Wed, 01 Aug 2007 18:39:51 -0400, Chuck Ebbert wrote: > Looks like the same problem with spinlock unfairness we've seen > elsewhere: it seems to be looping here? Or is everyone stuck just > waiting for writeout? > > lock_timer_base(): > for (;;) { > tvec_base_t *prelock_

Processes spinning forever, apparently in lock_timer_base()?

2007-08-01 Thread Chuck Ebbert
Looks like the same problem with spinlock unfairness we've seen elsewhere: it seems to be looping here? Or is everyone stuck just waiting for writeout? lock_timer_base(): for (;;) { tvec_base_t *prelock_base = timer->base; base = tbase_get_base(prelock_base)