Re: [ext3][kernels = 2.6.20.7 at least] KDE going comatose when FS is under heavy write load (massive starvation)

2007-08-17 Thread Alex Tomas

Andrew Morton wrote:

Sort-of.  But the per-superpblock, per-inode writeback code is pretty
careful to avoid livelocks.  The per-inode writeback is a strict single
linear sweep across the file.  It'll basically write out anything which was
dirty when it was called.  The per-superblock inode walk isn't as accurate
as that, becuase of the difficulties of juggling list_heads.  But we're
slowly working on that, and I suspect it'll be ggod enough for ext3
purposes already.


I'd say that these are two different mechanism solving different problems:
1) VFS/MM does periodic updates and uses regular writeback
2) data=ordered is to avoid metadata pointing to not-written-yet data

we can't use regular writeback in commit thread as long as it can fall into
allocation. so, we'd have to add one more WB mode (btw, i have a patch which
skips non-allocated blocks in writeback if special WB mode is requested).

OTOH, the faster we go through data sync part of commit, the better. given
that lots of inodes can be dirty with no data to sync, it's going to take
long in some cases. it's especially bad because commit doesn't scale to many
CPUs.

also, why would we need to flush *everything* every 5s? just because ext3 does
this? sounds strange. if somebody really need this we could add this possibility
to regular writeback path (making it tunable). but I'd rather prefer to have
a separate (fast, lightweight, scalable) mechanism to support data=ordered.

thanks, Alex

-
To unsubscribe from this list: send the line unsubscribe linux-ext4 in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [ext3][kernels = 2.6.20.7 at least] KDE going comatose when FS is under heavy write load (massive starvation)

2007-08-17 Thread Andrew Morton
On Fri, 17 Aug 2007 12:36:32 +0400 Alex Tomas [EMAIL PROTECTED] wrote:

 Andrew Morton wrote:
  Sort-of.  But the per-superpblock, per-inode writeback code is pretty
  careful to avoid livelocks.  The per-inode writeback is a strict single
  linear sweep across the file.  It'll basically write out anything which was
  dirty when it was called.  The per-superblock inode walk isn't as accurate
  as that, becuase of the difficulties of juggling list_heads.  But we're
  slowly working on that, and I suspect it'll be ggod enough for ext3
  purposes already.
 
 I'd say that these are two different mechanism solving different problems:
 1) VFS/MM does periodic updates and uses regular writeback
 2) data=ordered is to avoid metadata pointing to not-written-yet data

VFS/MM can do _much_ more than that!  Look at struct writeback_control.

That code path has many different modes of operation: it is used for
regular pdflush writeback, sync, fsync, throttling, etc.  Probably one of
its modes will be sufficient.  If we want to change ext3's existing
semantics and add an only writeback uninitialised blocks mode then
that'll be pretty straightforward: add more control information to
writeback_control and go for it.

 we can't use regular writeback in commit thread as long as it can fall into
 allocation. so, we'd have to add one more WB mode (btw, i have a patch which
 skips non-allocated blocks in writeback if special WB mode is requested).

yup

 OTOH, the faster we go through data sync part of commit, the better. given
 that lots of inodes can be dirty with no data to sync, it's going to take
 long in some cases. it's especially bad because commit doesn't scale to many
 CPUs.

eh?

 also, why would we need to flush *everything* every 5s? just because ext3 does
 this? sounds strange. if somebody really need this we could add this 
 possibility
 to regular writeback path (making it tunable). but I'd rather prefer to have
 a separate (fast, lightweight, scalable) mechanism to support data=ordered.
 

Yeah, that would make sense, perhaps.

Or just speed the existing stuff up.  iirc the main problem in there is 
unrelated
to data writeback.  There are situations where the running transaction has to 
block
behind metadata writeout which the committing transaction is performing.  I
reluctantly put that in years ago to get us out of a tight spot and it
never got optimised.


-
To unsubscribe from this list: send the line unsubscribe linux-ext4 in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html