Re: writeout stalls in current -git

2007-11-20 Thread David Chinner
On Tue, Nov 20, 2007 at 02:16:17PM +0100, Damien Wyart wrote: > Hello, > > > > > Ok, so it's not synchronous writes that we are doing - we're just > > > > submitting bio's tagged as WRITE_SYNC to get the I/O issued quickly. > > > > The "synchronous" nature appears to be coming from higher level >

Re: writeout stalls in current -git

2007-11-20 Thread Damien Wyart
Hello, > > > Ok, so it's not synchronous writes that we are doing - we're just > > > submitting bio's tagged as WRITE_SYNC to get the I/O issued > > > quickly. The "synchronous" nature appears to be coming from higher > > > level locking when reclaiming inodes (on the flush lock). It > > > appears

Re: Patch tags [was writeout stalls in current -git]

2007-11-09 Thread Adrian Bunk
On Fri, Nov 09, 2007 at 09:10:47AM -0700, Jonathan Corbet wrote: > Adrian Bunk <[EMAIL PROTECTED]> wrote: > > > What's missing is a definition which of them are formal tags that must > > be explicitely given (look at point 13 in SubmittingPatches). > > > > Signed-off-by: and Reviewed-by: are the

Re: Patch tags [was writeout stalls in current -git]

2007-11-09 Thread Jonathan Corbet
Adrian Bunk <[EMAIL PROTECTED]> wrote: > What's missing is a definition which of them are formal tags that must > be explicitely given (look at point 13 in SubmittingPatches). > > Signed-off-by: and Reviewed-by: are the formal tags someone must have > explicitely given and that correspond to so

Re: writeout stalls in current -git

2007-11-07 Thread David Chinner
On Wed, Nov 07, 2007 at 08:15:06AM +0100, Torsten Kaiser wrote: > On 11/7/07, David Chinner <[EMAIL PROTECTED]> wrote: > > Ok, so it's not synchronous writes that we are doing - we're just > > submitting bio's tagged as WRITE_SYNC to get the I/O issued quickly. > > The "synchronous" nature appears

Re: writeout stalls in current -git

2007-11-06 Thread Torsten Kaiser
On 11/7/07, David Chinner <[EMAIL PROTECTED]> wrote: > Ok, so it's not synchronous writes that we are doing - we're just > submitting bio's tagged as WRITE_SYNC to get the I/O issued quickly. > The "synchronous" nature appears to be coming from higher level > locking when reclaiming inodes (on the

Re: writeout stalls in current -git

2007-11-06 Thread David Chinner
On Wed, Nov 07, 2007 at 10:31:14AM +1100, David Chinner wrote: > On Tue, Nov 06, 2007 at 10:53:25PM +0100, Torsten Kaiser wrote: > > On 11/6/07, David Chinner <[EMAIL PROTECTED]> wrote: > > > Rather than vmstat, can you use something like iostat to show how busy > > > your > > > disks are? i.e. a

Re: writeout stalls in current -git

2007-11-06 Thread David Chinner
On Tue, Nov 06, 2007 at 10:53:25PM +0100, Torsten Kaiser wrote: > On 11/6/07, David Chinner <[EMAIL PROTECTED]> wrote: > > Rather than vmstat, can you use something like iostat to show how busy your > > disks are? i.e. are we seeing RMW cycles in the raid5 or some such issue. > > Both "vmstat 10"

Re: Patch tags [was writeout stalls in current -git]

2007-11-06 Thread Adrian Bunk
On Tue, Nov 06, 2007 at 09:25:12AM -0700, Jonathan Corbet wrote: > Andrew wrote: > > > > Reviewed-by: Fengguang Wu <[EMAIL PROTECTED]> > > > > I would prefer Tested-by: :( > > This seems like as good an opportunity as any to toss my patch tags > document out there one more time. I still think

Re: writeout stalls in current -git

2007-11-06 Thread Torsten Kaiser
On 11/6/07, Fengguang Wu <[EMAIL PROTECTED]> wrote: > -- > Subject: writeback: remove pages_skipped accounting in > __block_write_full_page() > From: Fengguang Wu <[EMAIL PROTECTED]> > > Miklos Szeredi <[EMAIL PROTECTED]> and me identified a writ

Re: writeout stalls in current -git

2007-11-06 Thread Torsten Kaiser
On 11/6/07, Peter Zijlstra <[EMAIL PROTECTED]> wrote: > On Tue, 2007-11-06 at 15:25 +1100, David Chinner wrote: > > > I'm struggling to understand what possible changed in XFS or writeback that > > would lead to stalls like this, esp. as you appear to be removing files when > > the stalls occur. >

Re: writeout stalls in current -git

2007-11-06 Thread Peter Zijlstra
On Tue, 2007-11-06 at 15:25 +1100, David Chinner wrote: > I'm struggling to understand what possible changed in XFS or writeback that > would lead to stalls like this, esp. as you appear to be removing files when > the stalls occur. Just a crazy idea,.. Could there be a set_page_dirty() that do

Re: Patch tags [was writeout stalls in current -git]

2007-11-06 Thread Balbir Singh
> This seems like as good an opportunity as any to toss my patch tags > document out there one more time. I still think it's a good idea to > codify some sort of consensus on what these tags mean... > > jon > [snip] > +By offering my Reviewed-by: tag, I state that: > + > + (a) I have carried out

Patch tags [was writeout stalls in current -git]

2007-11-06 Thread Jonathan Corbet
Andrew wrote: > > Reviewed-by: Fengguang Wu <[EMAIL PROTECTED]> > > I would prefer Tested-by: :( This seems like as good an opportunity as any to toss my patch tags document out there one more time. I still think it's a good idea to codify some sort of consensus on what these tags mean... jo

Re: writeout stalls in current -git

2007-11-06 Thread Peter Zijlstra
On Mon, 2007-11-05 at 15:57 -0800, Andrew Morton wrote: > > > Subject: mm: speed up writeback ramp-up on clean systems > > > > > > We allow violation of bdi limits if there is a lot of room on the > > > system. Once we hit half the total limit we start enforcing bdi limits > > > and bdi ramp-up s

Re: writeout stalls in current -git

2007-11-06 Thread Fengguang Wu
On Fri, Nov 02, 2007 at 08:22:10PM +0100, Torsten Kaiser wrote: > [ 547.20] mm/page-writeback.c 676 wb_kupdate: pdflush(285) 58858 > > global 12829 72 0 wc __ tw 0 sk 0 > [ 550.48] mm/page-writeback.c 676 wb_kupdate: pdflush(285) 57834 > > global 12017 62 0 wc __ tw 0 sk 0 > [ 552.7100

Re: writeout stalls in current -git

2007-11-05 Thread Torsten Kaiser
On 11/6/07, David Chinner <[EMAIL PROTECTED]> wrote: > On Mon, Nov 05, 2007 at 07:27:16PM +0100, Torsten Kaiser wrote: > > On 11/5/07, David Chinner <[EMAIL PROTECTED]> wrote: > > > Ok, so it's probably a side effect of the writeback changes. > > > > > > Attached are two patches (two because one wa

Re: writeout stalls in current -git

2007-11-05 Thread David Chinner
On Mon, Nov 05, 2007 at 07:27:16PM +0100, Torsten Kaiser wrote: > On 11/5/07, David Chinner <[EMAIL PROTECTED]> wrote: > > Ok, so it's probably a side effect of the writeback changes. > > > > Attached are two patches (two because one was in a separate patchset as > > a standalone change) that shoul

Re: writeout stalls in current -git

2007-11-05 Thread Andrew Morton
On Fri, 2 Nov 2007 18:33:29 +0800 Fengguang Wu <[EMAIL PROTECTED]> wrote: > On Fri, Nov 02, 2007 at 11:15:32AM +0100, Peter Zijlstra wrote: > > On Fri, 2007-11-02 at 10:21 +0800, Fengguang Wu wrote: > > > > > Interestingly, no background_writeout() appears, but only > > > balance_dirty_pages() an

Re: writeout stalls in current -git

2007-11-05 Thread Torsten Kaiser
On 11/5/07, David Chinner <[EMAIL PROTECTED]> wrote: > Ok, so it's probably a side effect of the writeback changes. > > Attached are two patches (two because one was in a separate patchset as > a standalone change) that should prevent async writeback from blocking > on locked inode cluster buffers.

Re: writeout stalls in current -git

2007-11-04 Thread Torsten Kaiser
On 11/5/07, David Chinner <[EMAIL PROTECTED]> wrote: > On Sun, Nov 04, 2007 at 12:19:19PM +0100, Torsten Kaiser wrote: > > I can now confirm, that I see this also with the current > > mainline-git-version > > I used 2.6.24-rc1-git-b4f555081fdd27d13e6ff39d455d5aefae9d2c0c > > plus the fix for the s

Re: writeout stalls in current -git

2007-11-04 Thread David Chinner
On Sun, Nov 04, 2007 at 12:19:19PM +0100, Torsten Kaiser wrote: > On 11/2/07, David Chinner <[EMAIL PROTECTED]> wrote: > > That's stalled waiting on the inode cluster buffer lock. That implies > > that the inode lcuser is already being written out and the inode has > > been redirtied during writeou

Re: writeout stalls in current -git

2007-11-04 Thread Torsten Kaiser
On 11/2/07, David Chinner <[EMAIL PROTECTED]> wrote: > That's stalled waiting on the inode cluster buffer lock. That implies > that the inode lcuser is already being written out and the inode has > been redirtied during writeout. > > Does the kernel you are testing have the "flush inodes in ascendi

Re: writeout stalls in current -git

2007-11-02 Thread Torsten Kaiser
On 11/2/07, David Chinner <[EMAIL PROTECTED]> wrote: > On Fri, Nov 02, 2007 at 08:22:10PM +0100, Torsten Kaiser wrote: > > [ 630.00] SysRq : Emergency Sync > > [ 630.12] Emergency Sync complete > > [ 632.85] SysRq : Show Blocked State > > [ 632.85] task

Re: writeout stalls in current -git

2007-11-02 Thread David Chinner
On Fri, Nov 02, 2007 at 08:22:10PM +0100, Torsten Kaiser wrote: > [ 630.00] SysRq : Emergency Sync > [ 630.12] Emergency Sync complete > [ 632.85] SysRq : Show Blocked State > [ 632.85] taskPC stack pid father > [ 632.85] pdflush D 8100

Re: writeout stalls in current -git

2007-11-02 Thread Torsten Kaiser
On 11/2/07, Peter Zijlstra <[EMAIL PROTECTED]> wrote: > On Fri, 2007-11-02 at 10:21 +0800, Fengguang Wu wrote: > > > Interestingly, no background_writeout() appears, but only > > balance_dirty_pages() and wb_kupdate. Obviously wb_kupdate won't > > block the process. > > Yeah, the background thresh

Re: writeout stalls in current -git

2007-11-02 Thread Torsten Kaiser
On 11/2/07, Fengguang Wu <[EMAIL PROTECTED]> wrote: > I guess the new debug printks will provide more hints on it. The "throttle_vm_writeout" did not trigger for my new workload. Except one (the first) "balance_dirty_pages" came from line 445, the newly added. But I found an other workload that l

Re: writeout stalls in current -git

2007-11-02 Thread Fengguang Wu
On Fri, Nov 02, 2007 at 11:15:32AM +0100, Peter Zijlstra wrote: > On Fri, 2007-11-02 at 10:21 +0800, Fengguang Wu wrote: > > > Interestingly, no background_writeout() appears, but only > > balance_dirty_pages() and wb_kupdate. Obviously wb_kupdate won't > > block the process. > > Yeah, the backg

Re: writeout stalls in current -git

2007-11-02 Thread Peter Zijlstra
On Fri, 2007-11-02 at 10:21 +0800, Fengguang Wu wrote: > Interestingly, no background_writeout() appears, but only > balance_dirty_pages() and wb_kupdate. Obviously wb_kupdate won't > block the process. Yeah, the background threshold is not (yet) scaled. So it can happen that the bdi_dirty limit

Re: writeout stalls in current -git

2007-11-01 Thread Fengguang Wu
On Fri, Nov 02, 2007 at 08:42:05AM +0100, Torsten Kaiser wrote: > The Subject is still missleading, I'm using 2.6.23-mm1. > > On 11/2/07, Fengguang Wu <[EMAIL PROTECTED]> wrote: > > On Thu, Nov 01, 2007 at 07:20:51PM +0100, Torsten Kaiser wrote: > > > On 11/1/07, Fengguang Wu <[EMAIL PROTECTED]> w

Re: writeout stalls in current -git

2007-11-01 Thread Torsten Kaiser
On 11/2/07, Fengguang Wu <[EMAIL PROTECTED]> wrote: > On Thu, Nov 01, 2007 at 08:00:10PM +0100, Torsten Kaiser wrote: > > On 11/1/07, Torsten Kaiser <[EMAIL PROTECTED]> wrote: > > > On 11/1/07, Fengguang Wu <[EMAIL PROTECTED]> wrote: > > > > Thank you. Maybe we can start by the applied debug patch

Re: writeout stalls in current -git

2007-11-01 Thread Torsten Kaiser
The Subject is still missleading, I'm using 2.6.23-mm1. On 11/2/07, Fengguang Wu <[EMAIL PROTECTED]> wrote: > On Thu, Nov 01, 2007 at 07:20:51PM +0100, Torsten Kaiser wrote: > > On 11/1/07, Fengguang Wu <[EMAIL PROTECTED]> wrote: > > > On Wed, Oct 31, 2007 at 04:22:10PM +0100, Torsten Kaiser wrote

Re: writeout stalls in current -git

2007-11-01 Thread Fengguang Wu
On Thu, Nov 01, 2007 at 08:00:10PM +0100, Torsten Kaiser wrote: > On 11/1/07, Torsten Kaiser <[EMAIL PROTECTED]> wrote: > > On 11/1/07, Fengguang Wu <[EMAIL PROTECTED]> wrote: > > > Thank you. Maybe we can start by the applied debug patch :-) > > > > Will applied it and try to recreate this. > > P

writeout stalls in current -git

2007-11-01 Thread Fengguang Wu
On Thu, Nov 01, 2007 at 07:20:51PM +0100, Torsten Kaiser wrote: > On 11/1/07, Fengguang Wu <[EMAIL PROTECTED]> wrote: > > On Wed, Oct 31, 2007 at 04:22:10PM +0100, Torsten Kaiser wrote: > > > Since 2.6.23-mm1 I also experience strange hangs during heavy writeouts. > > > Each time I noticed this I w