[ I Cc:ed fsdevel - trust that's OK.  There are a couple
  of things here which Al may like to comment on ]

Andreas Dilger wrote:
> 
> > > for ext3_writepage_trans_blocks() you seem to be overestimating the number
> > > of possible dirty blocks caused by a single page write.  I may be wrong,
> > > but aren't blocks attached to a page supposed to represent contiguous
> > > regions in the file?
> >
> > They're contiguous in the file, but the blocks for a single
> > page could in theory be sprayed all over the disk.  Which
> > means the worst case number of blocks which must be written
> > for a single writepage is huge.  With a 32k PAGE_CACHE_SIZE and
> > a 1k block size I think it's over 200 blocks!
> 
> I still don't agree.  If they are contiguous in the file (even if
> we have 32 filesystem blocks in a page, then we are still only
> dirtying a limited number of indirect blocks (5).  Yes, we can dirty
> up to 32 block bitmaps, 32 group descriptors, and 32 data blocks.  If
> you take the fact that we need to allocate indirect blocks into account,
> it would be 3(N+5) + 2 + 2*EXT3_SINGLEDATATRANS_BLOCKS, so 129 for a 1k
> block filesystem and 32k pages.  Only 33 for 4k blocks and 4k pages.

You are correct.  But I don't quite understand the math.

With N = #blocks-per-page, a single page write can
dirty:

N data blocks
2 indirect blocks
2 dindirect
1 tindirect
N+5 bitmap blocks (for the above)
N+5 group descriptor summary blocks
1 inode block
1 superblock.

Giving 3*N + 17 - 20 blocks for 4k/4k, 113 blocks for 32k/1k.

I'm assuming that the page write can straddle two indirect blocks,
because the direct blocks push these onto a 12 block boundary.

With ordered or writeback data it's the same, less the N data
blocks.

> > hmm..   I changed all that yesterday.  There is a situation
> > where I need to shut down a transaction and I don't know
> > how deeply it is nested - it may be one or two levels,
> > according to the call path.  Hence the new journal_nested_stop().
> 
> I haven't looked at it yet, but this would seem to violate layering.
> What's wrong with stoping the current action and returning an error
> to the caller (which will stop their transaction)?  If you do multiple
> journal_stop()s at the low level, then the caller will still do
> journal_stop() themselves, right?

That's been tossed out. ext3_file_write() and ext3_writepage()
and fdatasync/fdatawait have been redone and are now most sweet.
The issue of stopping and starting transactions from multiple
nesting levels is no more.   (Should have used journal_restart()
anyway).

> > See that little __mark_buffer_clean() in ll_rw_block()?
> > It puts buffers onto the LRU lists.  See the little
> > __put_unused_buffer_head() in commit.c?  It puts buffers
> > onto the free list.  If they're also on the LRU lists,
> > things explode nastily some time later.  This problem
> > was somehow hidden by the
> >
> >        if (bh->b_list == BUF_DIRTY)
> >
> > which Peter had in mark_buffer_clean().
> >
> > I couldn't find an API in 2.4 which allows us to manage the temporary
> > buffer_heads which journal_write_metadata_buffer() wants to create
> > for journal area writes.  Recall that the intent here is to take an
> > existing buffer_head and to write it somewhere else on disk
> > without copying the data.
> 
> I think this is why Stephen wanted to set up "journal_heads" instead
> of buffer_heads, so that ext3/jbd would manage these directly.

On the truncate path, truncate_inode_pages() is trying to unmap
buffers which in full data journalling mode are on a transaction
in basically any state.  Here I simply skip the unmap_buffer() if
it's on a transaction.  ext3_truncate() will perform the
appropriate revoke/forget later on.

An interesting problem with revokes:

- Dirty a metadata block
- Release the metadata block.  Revoke the dirtying.
- Reallocate the block as metadata. use journal_get_create_access()
- Dirty it
- Release it again.  Revoke the second dirtying.
- Bang.  We're trying to revoke a buffer which has already been revoked.

The fix was to call journal_cancel_revoke() in
journal_get_create_access().  I'm rather curious to
know why this hasn't bitten in the 2.2 JFS/ext3.  Either
it does happen and hasn't been fixed, or it doesn't
happen for mysterious 2.2/2.4 reasons, or I've broken
something.

> ...
> 
> Actually, I was just starting to ask on the reiserfs list about their
> BH_* flags that they use.  Kind of a bad situation - they define BH_*
> flags in reiserfs_fs_sb.h rather than in fs.h.  This means the bits
> could be re-used elsewhere and cause a lot of grief/bugs.  I've asked
> them to move all of the bits to fs.h:

Well...  I think it's sensible to define "core" BH_ bits and
leave the fs-specific ones up to the fs.  buffer_heads are
already being put back onto the freelist in various junky states,
but (some of) the fields are initialised as people start to
recycle them.  Given that getblk() and create_buffers() both
set ->b_state to a known, sane value I don't think ext3 will
be affected no matter what state b_state is in when reiserfs
frees it.

So I'm in fact inclined to move much of the ext3/jfs changes
*out* of fs.h and into ext3_fs.h and jfs.h.  There's a lot
of that sort of thing needs doing - splitting apart, tidying
up, honouring the config options, etc.  Straightforward stuff -
leave that till things are more stable.

> ...
> One issue I see (in the old code, not sure if it is fixed yet), is we don't
> appear to extend the transaction anywhere during truncate.  Am I just not
> following the code path correctly?

extend_transaction() does a journal_restart().  It ensures
that the inode has the correct file size (copied in from
the ext3-private i_disksize) and that all the journalled
indirect blocks are in a consistent state when this occurs.



Good progress lately with the sync and write functions.
The fs is now happy while being hammered with 20 minutes
of simultaneous dbench and multithreaded iozone runs.

I haven't tested recovery explicitly, although I've been
checking it whenever I crash the fs (often).  The only
problems observed so far after recovery are that occasionally
the group descriptor summaries claim that there are too few
free inodes and blocks.  Missing a group descriptor block
somewhere in truncate probably.  Also, some deleted inodes are
coming up with zero dtime.  Apart from that - solid.

Let me describe the `write' design.

The fundamental journallable unit of a write is a page,
and its associated metadata changes:

1: The metadata updates resulting from ext3_get_block()
2: The data in the page and
3: Any inode update (size extension due to write()).

As long as these three items hit journal together,
we're consistent.

We only write data in two places: ext3_writepage() and
ext3_file_write().

We open and close a transaction in ext3_writepage().  This
encapsulates all three of the above items and the number
of journallable blocks has a hard upper-bound.  Easy.
write() transactions never nest.  (Haven't looked at quotas
yet...)

We also open and a close a transaction in ext3_file_write().
This can be arbitrarily large, so we need to extend the
transaction midway.  This happens on entry to ext3_prepare_write().
This is the right place to do it.  The get_block(), data write
and inode update (in generic_commit_write()) will all come
immediately after this extend.

All the crap I had in ext3_fdatasync/ext3_fdatawait has gone.



The kjournald-submits-write code has been enabled.   We need this
to work - kicking bdflush to do it isn't nice.

This may have been a 2.2 artifact anyway: take a close look
at wakeup_bdflush(1) in 2.4   Note how it arranges for both
bdflush *and* the caller to simultaneously walk the same
dirty buffers list and to simultaneously kick off I/O.  Both
of them popping out at random times and unplugging the request
queues and generally getting under each other's feet.  Very,
very strange.  If *that* works, then kjournald will work too.

The kjournald-initiated version seems to be slightly faster
based on some light testing.



Serious things have been done to the sync code.

One hairy part of the ext3 code is ext3_write_inode().  This
*cannot* be journalled, because callers come in via prune_icache,
and they may have transactions open against entirely different
filesystems.  This is now fixed.

There are two main sync functions:

ext3_sync_file()
================

This guy is called by msync(), fsync() and fdatasync().
In all three cases the caller cannot have a transaction
open.  So all we do here is run fsync_inode_buffers()
to flush the i_dirty_buffers list and then synchronously
force a commit.  (i_dirty_buffers is *empty* if we're
doing full data journalling).

This is not super-efficient because we're syncing stuff
which doesn't belong to this inode. Then again, it will
be *more* efficient if the caller is syncing several files...

The only real chance for optimisation I see here is
handling the case where no updates pertaining to this inode
exist in either the running or committing transactions,
so no journal flush is needed.  That's fairly straightforward
to handle, by implementing transaction->t_ilist: a list
of inodes which are used by the current and committing
transactions.  Not sure if this is worth optimising for.

ext3_write_inode()
==================

This is the one which was writing direct into the filesystem
behind journalling's back.

It's called by the following:

generic_file_write() - for O_SYNC files.

  All I do is set the transaction to be synchronous, and
  return.  When generic_file_write() returns through
  ext3_file_write(), the journal_stop() will see the
  synchronous flag and will force a commit.  Easy.

sys_sync(), kupdate, etc

  Here, we see that the calling task doesn't have a
  transaction (it can't) and we simply return.  We
  know that the caller will later run ext3_write_super()
  which will flush the journal.  Easy.

  This required that the sync_supers() call in fsync_super(),
  fsync_dev(), sync_dev() and sync_old_buffers() be moved
  *after* the inode syncing calls.  I don't know what
  collateral damage this could cause.  If it's significant
  I'll have to revert it and put in a custom sync_journal()
  or whatever.

prune_icache()

  Not so easy.  Again, the caller doesn't have a transaction
  open and we simply return.

  On this path, we don't *have* to sync anything.  All we're
  trying to do is to sync some inodes so that they can
  be released.

  Instead, I have implemented self-cleaning inodes.  Whenever
  ext3 copies an inode into a journalled buffer, the in-core
  inode is set clean with:

        spin_lock(&inode_lock);
        if ((inode->i_state & I_LOCK) == 0)
                inode->i_state &= ~(I_DIRTY_SYNC|I_DIRTY_DATASYNC);
        spin_unlock(&inode_lock);

  This means that ext3 is continually cleaning inodes.  These
  inodes will be instantly reapable by prune_icache without
  any I/O required.

  The downside is that a few seconds worth of dirty inodes
  will not be cleaned when prune_icache wants them cleaned.
  The other downside is that with full data journalling,
  a truckload of data buffers which *could* be flushed
  into the journal and then checkpointed simply are not.

  This could be fixed by getting prune_icache to run a commit
  and a checkpoint.  This won't actually clean any inodes, but
  it'll release some memory, which is what we're really trying
  to do.

  Instead, I implemented a standalone shrink_journal_memory()
  which forces a commit and a checkpoint, waiting on it if required,
  which can be called from try_to_free_buffers().  It needs
  additional memory to start the I/O, but can potentially
  release a fair bit of memory.  For future study.

  Self-cleaning inodes seem to be a new thing.  It seems that
  fsync() and msync() *could* mark the in-core inode clean
  at the point where they copy the data into the inode's buffer
  but they don't.  (This is safe for other filesystems as long
  as they do the right thing with i_dirty_buffers)  Net result:
  redundant I/O in prune_icache().  Not very important, given
  that [mf]sync() is quite rare.  Possibly racy too, but not
  racy in ext3 because we have full control over and visibility
  of inode dirtyings.



  Last but not least, update_atime() has been changed to not
  dirty the inode if its atime is already current.  Dunno why
  this isn't already done - it'll save cycles as well as
  some I/O.



Patch of the day (complete with 1.5 kbyte buffer_heads!) is
at
        http://www.zip.com.au/~akpm/08May01-ext3-akpm.patch.gz

Question: It may be useful to prevent simultaneous commits
against different partitions on the same disk.  How does one
determine this, given an in-core superblock?  Presumably,
comparing majors isn't enough.
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [EMAIL PROTECTED]

Reply via email to