Re: [RFC 12/26] ext2 white-out support
On Wed, 1 August 2007 15:33:30 -0400, Josef Sipek wrote: This brings up an very interesting (but painful) question...which makes more sense? Allowing the modifications in only the top-most branch, or any branch (given the user allows it at mount-time)? This is really question to the community at large, not just you, Dave :) Only write to top-most layer. There are two reasons for this. First it allows users to create a union mount, test something (e.g. update the distribution) and remove every trace from the test by umounting the top-most layer. Such a thing can be quite valuable. The second reason is simplicity. I personally couldn't even start to describe the semantics. If the user does a rename, which layer will the change end up in? What if source or target exist in multiple layers? How to rename a directory in a lower layer containing a new file in an upper layer? Finding new and interesting corner cases for such a beast can be quite entertaining. And until someone has properly documented the semantics for _all_ the corner cases, my enthusiasm is below freezing point. Does such a documentation exist? Jörn -- A surrounded army must be given a way out. -- Sun Tzu - To unsubscribe from this list: send the line unsubscribe linux-fsdevel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 12/26] ext2 white-out support
On Tue, 2007-07-31 at 13:11 -0400, Josef Sipek wrote:
On Tue, Jul 31, 2007 at 07:00:12PM +0200, Jan Blunck wrote:
On Tue, Jul 31, Josef Sipek wrote:
On Mon, Jul 30, 2007 at 06:13:35PM +0200, Jan Blunck wrote:
Introduce white-out support to ext2.
I think storing whiteouts on the branches is wrong. It creates all sort of
nasty cases when people actually try to use unioning. Imagine a (no-so
unlikely) scenario where you have 2 unions, and they share a branch. If
you
create a whiteout in one union on that shared branch, the whiteout
magically
affects the other union as well! Whiteouts are a union-level construct,
and
therefore storing them at the branch level is wrong.
So you think that just because you mounted the filesystem somewhere else it
should look different? This is what sharing is all about. If you share a
filesystem you also share the removal of objects.
The removal happens at the union level, not the branch level. Say you have:
/a/
/b/foo
/c/foo
And you mount /u1 as a union of {a,b}, and /u2 as union of {a,c}.
Who does this? I'm assuming that a is the top layer. Aren't union
mounts typically about sharing lower layers and having a separate rw
layer for each union mount?
$ find /u*
/u1
/u1/foo
/u2
/u2/foo
$ rm /u1/foo # this creates whiteout for foo in /a
$ find /u*
/u1
/u2
Is that what you'd expect as a user? I don't think so.
That's exactly what I would expect.
If I were to:
$ echo this is new /u1/foo
I would expect:
$ cat /u2/foo
this is new
So why should rm behave differently?
I haven't really been tuned into union mounts, so maybe I'm missing out
on something basic here.
Thanks,
Shaggy
--
David Kleikamp
IBM Linux Technology Center
-
To unsubscribe from this list: send the line unsubscribe linux-fsdevel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 12/26] ext2 white-out support
On Jul 31 2007 12:36, Josef Sipek wrote: [2] http://www.filesystems.org/unionfs-odf.txt Instead, the new ODF code stores whiteouts as hardlinks to a special (regular) zero-length file in odf (/odf/whiteout), and it stores opaqueness information for directories in the inode GID bits in an ODF file system (e.g., ext2, XFS, etc.) on the local machine. This avoids the name-space pollution and avoids races with network file systems, while minimizing inode consummation in /odf. Inode GID bits - are you reducing my 32 bits of gid_t to 31 bits? That does not work out either. Jan -- - To unsubscribe from this list: send the line unsubscribe linux-fsdevel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 12/26] ext2 white-out support
On Aug 1 2007 12:00, Hans-Peter Jansen wrote: *) The amount of administration work of any (necessary, unfortunately) VMware XP instance running on top of those diskless clients excels that of all diskless clients by an order of magnitude. Hardly :) Install XP, snapshot it when done. Copy .vmdk to 'all' machines. On security upgrades, revert to snapshot (well - if the workflow allows it), install, snapshot again. Etc. Work: 1 1/2. Jan -- - To unsubscribe from this list: send the line unsubscribe linux-fsdevel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 12/26] ext2 white-out support
On Wed, Aug 01, 2007 at 10:23:29AM -0500, Dave Kleikamp wrote:
On Tue, 2007-07-31 at 13:11 -0400, Josef Sipek wrote:
On Tue, Jul 31, 2007 at 07:00:12PM +0200, Jan Blunck wrote:
On Tue, Jul 31, Josef Sipek wrote:
On Mon, Jul 30, 2007 at 06:13:35PM +0200, Jan Blunck wrote:
Introduce white-out support to ext2.
I think storing whiteouts on the branches is wrong. It creates all sort
of
nasty cases when people actually try to use unioning. Imagine a (no-so
unlikely) scenario where you have 2 unions, and they share a branch. If
you
create a whiteout in one union on that shared branch, the whiteout
magically
affects the other union as well! Whiteouts are a union-level construct,
and
therefore storing them at the branch level is wrong.
So you think that just because you mounted the filesystem somewhere else
it
should look different? This is what sharing is all about. If you share a
filesystem you also share the removal of objects.
The removal happens at the union level, not the branch level. Say you have:
/a/
/b/foo
/c/foo
And you mount /u1 as a union of {a,b}, and /u2 as union of {a,c}.
Who does this? I'm assuming that a is the top layer. Aren't union
mounts typically about sharing lower layers and having a separate rw
layer for each union mount?
Alright not the greatest of examples, there is something to be said about
symmetry, so...let me try again :)
/a/
/b/bar (whiteout for bar)
/c/foo/qwerty
Now, let's mount a union of {a,b,c}, and we'll see:
$ find /u
/u
/u/foo
/u/foo/qwerty
$ mv /u/foo /u/bar
Now what? How do you rename? Do you rename in the same branch (assuming it
is rw)? If you do, you'll get:
$ find /u
/u
Oops! There's a whiteout in /b that hides the directory in /c -- rename(2)
shouldn't make directory subtrees disappear.
There are two ways to solve this:
1) cp -r the entire subtree being renamed to highest-priority branch, and
rename there (you might have to recreate a series of directories to have a
place to cp to...so you got cp -r _AND_ mkdir -p-like code in the VFS!
1/2 a :) )
2) Don't store whiteouts within branches. This makes it really easy to
rename and remove the whiteout.
Sure, you could try to rename in-place and remove the whiteout, but what if
you have:
/a/
/b/bar (whiteout)
/c/bar/blah
/d/foo/qwerty
$ mv /u/foo /u/bar
You can't just remove the whiteout, because that'd uncover the whited-out
directory bar in /c.
Josef 'Jeff' Sipek.
--
Bad pun of the week: The formula 1 control computer suffered from a race
condition
-
To unsubscribe from this list: send the line unsubscribe linux-fsdevel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 12/26] ext2 white-out support
On Wed, 2007-08-01 at 14:44 -0400, Josef Sipek wrote:
Alright not the greatest of examples, there is something to be said about
symmetry, so...let me try again :)
/a/
/b/bar(whiteout for bar)
/c/foo/qwerty
Now, let's mount a union of {a,b,c}, and we'll see:
$ find /u
/u
/u/foo
/u/foo/qwerty
$ mv /u/foo /u/bar
Now what? How do you rename? Do you rename in the same branch (assuming it
is rw)?
Er, no. According to Documentation/filesystems/union-mounts.txt, only
the topmost layer of the mount stack can be altered.
If you do, you'll get:
$ find /u
/u
Oops! There's a whiteout in /b that hides the directory in /c -- rename(2)
shouldn't make directory subtrees disappear.
There are two ways to solve this:
1) cp -r the entire subtree being renamed to highest-priority branch, and
rename there (you might have to recreate a series of directories to have a
place to cp to...so you got cp -r _AND_ mkdir -p-like code in the VFS!
1/2 a :) )
I think this is the only alternative, given the design.
2) Don't store whiteouts within branches. This makes it really easy to
rename and remove the whiteout.
Sure, you could try to rename in-place and remove the whiteout, but what if
you have:
/a/
/b/bar(whiteout)
/c/bar/blah
/d/foo/qwerty
$ mv /u/foo /u/bar
You can't just remove the whiteout, because that'd uncover the whited-out
directory bar in /c.
Josef 'Jeff' Sipek.
--
David Kleikamp
IBM Linux Technology Center
-
To unsubscribe from this list: send the line unsubscribe linux-fsdevel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 12/26] ext2 white-out support
On Mittwoch, 1. August 2007, Josef Sipek wrote: Alright not the greatest of examples, there is something to be said about symmetry, so...let me try again :) ... Oops! There's a whiteout in /b that hides the directory in /c -- rename(2) shouldn't make directory subtrees disappear. There are two ways to solve this: 1) cp -r the entire subtree ... 2) Don't store whiteouts within branches ... Sorry for making uninformed guesses, but if there are already special nodes (whiteout), why not extending them to some more general format - specifying a (source, destination) pair at the topmost level? - A delete is a (source, NULL) pair - A rename is a (source, destination) pair, which causes lookups on source to use the string destination in the lower branches. Would that work? Regards, Phil - To unsubscribe from this list: send the line unsubscribe linux-fsdevel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC 12/26] ext2 white-out support
Introduce white-out support to ext2.
Known Bugs:
- Needs a reserved inode number for white-outs
- S_OPAQUE isn't persistently stored
Signed-off-by: Jan Blunck [EMAIL PROTECTED]
---
fs/ext2/dir.c |2 ++
fs/ext2/namei.c | 18 ++
fs/ext2/super.c |5 -
include/linux/ext2_fs.h |4
4 files changed, 28 insertions(+), 1 deletion(-)
--- a/fs/ext2/dir.c
+++ b/fs/ext2/dir.c
@@ -230,6 +230,7 @@ static unsigned char ext2_filetype_table
[EXT2_FT_FIFO] = DT_FIFO,
[EXT2_FT_SOCK] = DT_SOCK,
[EXT2_FT_SYMLINK] = DT_LNK,
+ [EXT2_FT_WHT] = DT_WHT,
};
#define S_SHIFT 12
@@ -241,6 +242,7 @@ static unsigned char ext2_type_by_mode[S
[S_IFIFO S_SHIFT]= EXT2_FT_FIFO,
[S_IFSOCK S_SHIFT] = EXT2_FT_SOCK,
[S_IFLNK S_SHIFT]= EXT2_FT_SYMLINK,
+ [S_IFWHT S_SHIFT]= EXT2_FT_WHT,
};
static inline void ext2_set_de_type(ext2_dirent *de, struct inode *inode)
--- a/fs/ext2/namei.c
+++ b/fs/ext2/namei.c
@@ -288,6 +288,23 @@ static int ext2_rmdir (struct inode * di
return err;
}
+static int ext2_whiteout(struct inode *dir, struct dentry *dentry)
+{
+ struct inode *inode;
+ int err;
+
+ inode = ext2_new_inode (dir, S_IFWHT | S_IRUGO);
+ err = PTR_ERR(inode);
+ if (IS_ERR(inode))
+ goto out;
+
+ init_special_inode(inode, inode-i_mode, 0);
+ mark_inode_dirty(inode);
+ err = ext2_add_nondir(dentry, inode);
+out:
+ return err;
+}
+
static int ext2_rename (struct inode * old_dir, struct dentry * old_dentry,
struct inode * new_dir, struct dentry * new_dentry )
{
@@ -382,6 +399,7 @@ const struct inode_operations ext2_dir_i
.mkdir = ext2_mkdir,
.rmdir = ext2_rmdir,
.mknod = ext2_mknod,
+ .whiteout = ext2_whiteout,
.rename = ext2_rename,
#ifdef CONFIG_EXT2_FS_XATTR
.setxattr = generic_setxattr,
--- a/fs/ext2/super.c
+++ b/fs/ext2/super.c
@@ -752,6 +752,9 @@ static int ext2_fill_super(struct super_
ext2_xip_verify_sb(sb); /* see if bdev supports xip, unset
EXT2_MOUNT_XIP if not */
+ if (EXT2_HAS_INCOMPAT_FEATURE(sb, EXT2_FEATURE_INCOMPAT_WHITEOUT))
+ sb-s_flags |= MS_WHITEOUT;
+
if (le32_to_cpu(es-s_rev_level) == EXT2_GOOD_OLD_REV
(EXT2_HAS_COMPAT_FEATURE(sb, ~0U) ||
EXT2_HAS_RO_COMPAT_FEATURE(sb, ~0U) ||
@@ -1299,7 +1302,7 @@ static struct file_system_type ext2_fs_t
.name = ext2,
.get_sb = ext2_get_sb,
.kill_sb= kill_block_super,
- .fs_flags = FS_REQUIRES_DEV,
+ .fs_flags = FS_REQUIRES_DEV | FS_WHT,
};
static int __init init_ext2_fs(void)
--- a/include/linux/ext2_fs.h
+++ b/include/linux/ext2_fs.h
@@ -61,6 +61,7 @@
#define EXT2_ROOT_INO 2 /* Root inode */
#define EXT2_BOOT_LOADER_INO5 /* Boot loader inode */
#define EXT2_UNDEL_DIR_INO 6 /* Undelete directory inode */
+#define EXT2_WHT_INO7 /* Whiteout inode */
/* First non-reserved inode for old ext2 filesystems */
#define EXT2_GOOD_OLD_FIRST_INO11
@@ -479,10 +480,12 @@ struct ext2_super_block {
#define EXT3_FEATURE_INCOMPAT_RECOVER 0x0004
#define EXT3_FEATURE_INCOMPAT_JOURNAL_DEV 0x0008
#define EXT2_FEATURE_INCOMPAT_META_BG 0x0010
+#define EXT2_FEATURE_INCOMPAT_WHITEOUT 0x0020
#define EXT2_FEATURE_INCOMPAT_ANY 0x
#define EXT2_FEATURE_COMPAT_SUPP EXT2_FEATURE_COMPAT_EXT_ATTR
#define EXT2_FEATURE_INCOMPAT_SUPP (EXT2_FEATURE_INCOMPAT_FILETYPE| \
+EXT2_FEATURE_INCOMPAT_WHITEOUT| \
EXT2_FEATURE_INCOMPAT_META_BG)
#define EXT2_FEATURE_RO_COMPAT_SUPP(EXT2_FEATURE_RO_COMPAT_SPARSE_SUPER| \
EXT2_FEATURE_RO_COMPAT_LARGE_FILE| \
@@ -549,6 +552,7 @@ enum {
EXT2_FT_FIFO,
EXT2_FT_SOCK,
EXT2_FT_SYMLINK,
+ EXT2_FT_WHT,
EXT2_FT_MAX
};
--
-
To unsubscribe from this list: send the line unsubscribe linux-fsdevel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC 12/26] ext2 white-out support
On Mon, Jul 30, 2007 at 06:13:35PM +0200, Jan Blunck wrote: Introduce white-out support to ext2. Known Bugs: - Needs a reserved inode number for white-outs You picked different reserved inodes for the ext2 and ext3 filesystems. That's good for a NACK right there. The codepoints (i.e., reserved inode numbers, feature bit masks, etc.) for ext2, ext3, and ext4 MUST not overlap. After all, someone might use tune2fs -j to convert an ext2 filesystem to ext3, and is it's REALLY BAD that you're using a reserved inode of 7 for ext2, and 9 for ext3. Also, I note that you have created a new INCOMPAT feature flag support for whiteouts. That's really unfortunate; we try to avoid introducing incompatible feature flags unless absolutely necessary; note that even adding a COMPAT feature flag means that you need a new version of e2fsprogs if you want e2fsck to be willing to touch that filesystem. So --- if you're looking for a way to add whiteout support to ext2/ext3 without needing a feature bit, here's how. We allocate a new inode flag in struct ext3_inode.i_flags: #define EXT2_WHTOUT_FL 0x0004 We also allocate a new field in the ext2 superblock to store the whiteout inode. (Please coordinate with me so it's a superblock field not in use by ext3/ext4, and so it's reserved so that no one else uses it.) The superblock field, call it s_whtout_ino, stores the inode number for the white out inode. When you create a new whiteout file, the code checks sb-s_whtout_ino, and if it is zero, it allocates a new inode, and creates it as a zero-length regular file (i_mode |= S_IFREG) with the EXT2_WHTOUT_FL flag set in the inode, and then store the inode number in sb-s_whtout_ino. If sb-s_whtout_ino is non-zero, you must read in the inode and make sure that the EXT2_WHTOUT_FL is set. If it is not, then allocate a new whiteout inode as described previously. Then link the inode into the directory as before. When reading an inode, if the EXT2_WHTOUT_FL flag is set, then set the in-memory mode of the inode to be S_IFWHT. That's pretty much about it. For cleanliness sake, it would be good if ext2_delete_inode clears sb-s_whtout_ino if the last whiteout link has been deleted, but it's strictly speaking not necessary. If you do it this way, the filesystem is completely backwards compatible; the whiteout files will just appear to links to a normal zero-lenth file. I wouldn't bother with setting the directory type field to be DT_WHT, given that they will never be returned to userspace anyway. Regards, - Ted - To unsubscribe from this list: send the line unsubscribe linux-fsdevel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
