Re: Virtual files
Hans Reiser wrote: I am just coming into the middle of this, so forgive me, but why are you implementing this above the filesystem layer? It seems like virtual files should be plugins that the filesystem calls while resolving its lookups and finding links to the plugin. This means that something should be stored in the FS to indicate what plugin to call. This means that at least part of it should be in the FS. If you want Hurd-style translators, then yes. But if you want the ability to go "cd tar-file.tar.gz:/" or "less http://host/index.html" then that really ought to be independent of the filesystem, so that it works with all of them. -- Jamie
Re: Virtual files
Jamie Lokier wrote: - The contents of the real file (if there is one) and its virtual derivative aren't synchronised. E.g. if you look inside a tar file or the decompressed version of a .gz file, then changing either the virtual or real files should affect the other. Agreed. Userspace avfs does this. Kernel avfs is read-only at the moment. - This is only a few small steps to making cd-into-archive work. :-) Many of the remaining checks for S_ISDIR in the generic VFS code should go -- they're not consistent with other parts of the VFS which checks f_ops anyway. I don't think that making a directory from a file is necessarily a good idea: - Some programs will probably break by this (i.e. will not let you cd into a non-directory - You cannot distinguish virtual and non-virtual files in this way (e.g. how do you represent the gunzipped version of xyz.gz?) The approach taken by most virtual file implementations is to make a separate name for the virtual file/directory: tarfile.tgz - is a file tarfile.tgz@ - is a directory (place your favorite magic char instead of @) In the directory listing only the first appears, but most tools do not depend on the directory to be existent (notable exception was the old userspace getcwd implementation which is gone since kernel version 2.2 and glibc-2.0). Miklos
Virtual files (RFC, patch)
Hi,
Here's a new version of the virtual file patch (against 2.4.0-test5),
and the kernel module for avfs/podfuk.
It now does:
- Clean up automatic mounts when parent is umounted
- Clear dcache upon module insert, so virtual files are always found
- Correctly handle virtual files in multiply mounted filesystems
Missing:
- SMP safeness in the nredir module
- Virtual file creation (root node)
The kernel patch makes the following changes:
1) namei.c: adds a 'lookup_virtual' hook function. This is invoked,
when the real_lookup returns a negative dentry.
The virtual file implementation can check if the name does belong
to a virtual file, and act apropriately (e.g. by mounting
something on top the dentry)
2) namei.c: real_lookup() is passed the parent nameidata instead of
dentry, so that the virtual lookup function can have the global
path.
3) mount.h: a new mount operations structure is added with the following
operations:
- release() -- called when the mount is removed
- umount() -- called when the parent mount is about to be umounted
4) super.c: new function try_umount_children(), which traverses the
child-list of the mount, and calls the umount method of the child.
This function is called in do_umount(), before the mnt_count is
checked.
In try_umount_children() the 'dcache_lock' is not held, since the
umount method might do complex thinks (like call do_umount), but
if I'm not mistaken, then mount_sem is enough to protect the child
list...
5) super.c: add_vfsmnt has an extra parameter -- the mount operations.
6) various places: The following symbols are exported:
- lookup_virtual
- add_vfsmnt
- do_umount
- mount_sem
Alexander, could you please tell me, what is acceptable from these
changes! (not necessary for 2.4.0)
Thanks,
Miklos
===File ~/avfs/src/src/nredir/virtual-2.4.0-test5.patch=
diff -ru /tmp/linux/linux-2.4.0-test5.tar.gz@/linux/fs/namei.c linux/fs/namei.c
--- /tmp/linux/linux-2.4.0-test5.tar.gz@/linux/fs/namei.c Fri Jul 28 08:14:10
2000
+++ linux/fs/namei.cFri Jul 28 08:18:53 2000
@@ -98,6 +98,9 @@
* XEmacs seems to be relying on it...
*/
+/* lookup function for "virtual" files */
+struct dentry *(*lookup_virtual)(struct nameidata *, struct dentry *);
+
/* In order to reduce some races, while at the same time doing additional
* checking and hopefully speeding things up, we copy filenames to the
* kernel data space before using them..
@@ -260,9 +263,10 @@
* make sure that nobody added the entry to the dcache in the meantime..
* SMP-safe
*/
-static struct dentry * real_lookup(struct dentry * parent, struct qstr * name, int
flags)
+static struct dentry * real_lookup(struct nameidata * nd, struct qstr * name, int
+flags)
{
struct dentry * result;
+ struct dentry *parent = nd-dentry;
struct inode *dir = parent-d_inode;
down(dir-i_sem);
@@ -281,12 +285,24 @@
lock_kernel();
result = dir-i_op-lookup(dir, dentry);
unlock_kernel();
- if (result)
+ if (result) {
dput(dentry);
- else
+ up(dir-i_sem);
+ }
+ else {
+ up(dir-i_sem);
result = dentry;
- }
- up(dir-i_sem);
+ /*
+* If the dentry is negative it might
+* refer to a 'virtual' file
+*/
+ if (lookup_virtual !dentry-d_inode)
+ lookup_virtual(nd, dentry);
+ }
+ }
+ else
+ up(dir-i_sem);
+
return result;
}
@@ -489,7 +505,7 @@
/* This does the actual lookups.. */
dentry = cached_lookup(nd-dentry, this, LOOKUP_CONTINUE);
if (!dentry) {
- dentry = real_lookup(nd-dentry, this, LOOKUP_CONTINUE);
+ dentry = real_lookup(nd, this, LOOKUP_CONTINUE);
err = PTR_ERR(dentry);
if (IS_ERR(dentry))
break;
@@ -552,7 +568,7 @@
}
dentry = cached_lookup(nd-dentry, this, 0);
if (!dentry) {
- dentry = real_lookup(nd-dentry, this, 0);
+ dentry = real_lookup(nd, this, 0);
err = PTR_ERR(dentry);
if (IS_ERR(dentry))
break;
diff -ru /tmp/linux/linux-2.4.0-test5.tar.gz@/linux/fs/super.c linux/fs/super.c
--- /tmp/linux/l
Re: Virtual files
Hi! It looks much cleaner than my previous approaches. There seems to be something strange with usage counts: pavel@bug:/tmp/cz1_mbrola.tgz#utar$ ls -al total 88 drwxr-xr-x 2 pavelusers3586 Mar 26 18:43 ./ drwxrwxrwt 38 root root81920 Jul 24 09:57 ../ drwxr-xr-x 42949 pavelusers 0 Mar 25 11:16 czech/ pavel@bug:/tmp/cz1_mbrola.tgz#utar$ It should be the same as '/overlay/tmp/cz1_mbrola.tgz#utar'. If it is, then its a coda problem. Yes. You seem to have damaged locking: [...] In case of !dentry, up(dir-i_sem) is not executed at all. Oops. Corrected. I have some cleanups to nredir.c module, they are attached (please apply). Ouch, and there's slight problem with caching: Thanks for the patch. I've applied it (except the @/# change :). You access vmlinux#gz, then load nredir module, then access vmlinux#gz again. But failure is already cached and you are out of luck :-(. Yes you are right. I was thinking of a solution where the kernel module initialization would kill the negative dentries which contain the magic char. Miklos
Re: Virtual files
Miklos Szeredi wrote: Problems: - The filesystem will be littered with these loopback mounts. This should be cleared upon unmount, and possibly when the dcache is shrunk. There was a similar requirement for new autofs IIRC. - Creation/removal of virtual files are not handled by this code. Podfuk already has this problem, but I thought I'd mention it since a fix seems rather desirable: - The contents of the real file (if there is one) and its virtual derivative aren't synchronised. E.g. if you look inside a tar file or the decompressed version of a .gz file, then changing either the virtual or real files should affect the other. And this is a pure feature request: - This is only a few small steps to making cd-into-archive work. :-) Many of the remaining checks for S_ISDIR in the generic VFS code should go -- they're not consistent with other parts of the VFS which checks f_ops anyway. enjoy, -- Jamie
Re: Virtual files
Hi!
I've been looking at the possibility of cleanly implementing virtual
files (podfuk/avfs style) with the multiple mount thing in 2.4
kernels. Here is how it would work:
1) There needs to be a global lookup hook in VFS. If a real lookup
fails, the global lookup is invoked. The cached lookup path is not
effected.
2) The global lookup checks whether the name refers to a virtual file
(contains the magic char exists in /overlay). If it does, then the
file/dir from overlay is mounted on top of the negative dentry. The
dentry is not filled in, so the virtual file remains invisible.
For 1) the VFS needs to be modified, but with infinitesimal
performance effect. For 2) a kernel module could be used.
Great!
I've prepared a kernel patch against 2.4.0-test4 (just for comments on
this, NOT for inclusion in the main kernel), and a kernel module. I've
tested it with avfs, but it should also work for podfuk.
It looks much cleaner than my previous approaches. There seems to be
something strange with usage counts:
pavel@bug:/tmp/cz1_mbrola.tgz#utar$ ls -al
total 88
drwxr-xr-x 2 pavelusers3586 Mar 26 18:43 ./
drwxrwxrwt 38 root root81920 Jul 24 09:57 ../
drwxr-xr-x 42949 pavelusers 0 Mar 25 11:16 czech/
pavel@bug:/tmp/cz1_mbrola.tgz#utar$
Problems:
- The filesystem will be littered with these loopback mounts. This
should be cleared upon unmount, and possibly when the dcache is
shrunk. There was a similar requirement for new autofs IIRC.
- Creation/removal of virtual files are not handled by this code.
Comments?
Yes. You seem to have damaged locking:
@@ -281,12 +285,18 @@
lock_kernel();
result = dir-i_op-lookup(dir, dentry);
unlock_kernel();
- if (result)
+ if (result) {
dput(dentry);
- else
+up(dir-i_sem);
+}
+ else {
+ up(dir-i_sem);
result = dentry;
+ /* for avfs */
+ if (!dentry-d_inode lookup_global)
+ lookup_global(nd, dentry);
+ }
}
- up(dir-i_sem);
In case of !dentry, up(dir-i_sem) is not executed at all.
I have some cleanups to nredir.c module, they are attached (please
apply). Ouch, and there's slight problem with caching:
You access vmlinux#gz, then load nredir module, then access vmlinux#gz
again. But failure is already cached and you are out of luck :-(.
Pavel
--
I'm [EMAIL PROTECTED] "In my country we have almost anarchy and I don't care."
Panos Katsaloulis describing me w.r.t. patents at [EMAIL PROTECTED]
--- /tmp/nredir.c Mon Jul 24 10:03:48 2000
+++ nredir.cMon Jul 24 09:53:05 2000
@@ -1,4 +1,5 @@
#include linux/module.h
+#include linux/init.h
#include linux/kernel.h
#include linux/fs.h
#include linux/slab.h
@@ -7,7 +8,7 @@
#define NREDIR_VERSION "0.1"
-#define AVFS_MAGIC_CHAR '@'
+#define AVFS_MAGIC_CHAR '#'
#define OVERLAY_DIR "/overlay"
#define OVERLAY_DIR_LEN 8
@@ -99,7 +100,7 @@
return dentry;
}
-int init_module(void)
+static int __init init_nredir(void)
{
printk(KERN_INFO "nredir init (version %s)\n", NREDIR_VERSION);
@@ -111,11 +112,14 @@
}
-void cleanup_module(void)
+static void __exit cleanup_nredir(void)
{
printk(KERN_INFO "nredir cleanup\n");
-lookup_global = 0;
+lookup_global = NULL;
printk("lookup_global: %x\n", (int) lookup_global);
}
+
+module_init(init_nredir);
+module_exit(cleanup_nredir);
Re: Virtual files
On Mon, 24 Jul 2000, Pavel Machek wrote: Hi! I've been looking at the possibility of cleanly implementing virtual files (podfuk/avfs style) with the multiple mount thing in 2.4 kernels. Here is how it would work: 1) There needs to be a global lookup hook in VFS. If a real lookup fails, the global lookup is invoked. The cached lookup path is not effected. 2) The global lookup checks whether the name refers to a virtual file (contains the magic char exists in /overlay). If it does, then the file/dir from overlay is mounted on top of the negative dentry. The dentry is not filled in, so the virtual file remains invisible. For 1) the VFS needs to be modified, but with infinitesimal performance effect. For 2) a kernel module could be used. Erm? Negative lookup happens every time when you are creating an object. Problems: - The filesystem will be littered with these loopback mounts. This should be cleared upon unmount, and possibly when the dcache is shrunk. There was a similar requirement for new autofs IIRC. Not with the dcache shrinking. - Creation/removal of virtual files are not handled by this code. Comments? How will it interact with modifying the mount-tree? Comment on autofs is pretty interesting, though. Sigh... Just let me find some time to do mount-traps - that may very well be the right tool here...
Virtual files
Hi!
I've been looking at the possibility of cleanly implementing virtual
files (podfuk/avfs style) with the multiple mount thing in 2.4
kernels. Here is how it would work:
1) There needs to be a global lookup hook in VFS. If a real lookup
fails, the global lookup is invoked. The cached lookup path is not
effected.
2) The global lookup checks whether the name refers to a virtual file
(contains the magic char exists in /overlay). If it does, then the
file/dir from overlay is mounted on top of the negative dentry. The
dentry is not filled in, so the virtual file remains invisible.
For 1) the VFS needs to be modified, but with infinitesimal
performance effect. For 2) a kernel module could be used.
I've prepared a kernel patch against 2.4.0-test4 (just for comments on
this, NOT for inclusion in the main kernel), and a kernel module. I've
tested it with avfs, but it should also work for podfuk.
Problems:
- The filesystem will be littered with these loopback mounts. This
should be cleared upon unmount, and possibly when the dcache is
shrunk. There was a similar requirement for new autofs IIRC.
- Creation/removal of virtual files are not handled by this code.
Comments?
Miklos
===File /tmp/d==
diff -ru linux/fs/namei.c /spare/src/linux/fs/namei.c
--- linux/fs/namei.cFri Jul 7 21:52:56 2000
+++ /spare/src/linux/fs/namei.c Tue Jul 18 14:52:28 2000
@@ -98,6 +98,9 @@
* XEmacs seems to be relying on it...
*/
+/* for avfs */
+lookup_global_t lookup_global = NULL;
+
/* In order to reduce some races, while at the same time doing additional
* checking and hopefully speeding things up, we copy filenames to the
* kernel data space before using them..
@@ -260,9 +263,10 @@
* make sure that nobody added the entry to the dcache in the meantime..
* SMP-safe
*/
-static struct dentry * real_lookup(struct dentry * parent, struct qstr * name, int
flags)
+static struct dentry * real_lookup(struct nameidata * nd, struct qstr * name, int
+flags)
{
struct dentry * result;
+ struct dentry *parent = nd-dentry;
struct inode *dir = parent-d_inode;
down(dir-i_sem);
@@ -281,12 +285,18 @@
lock_kernel();
result = dir-i_op-lookup(dir, dentry);
unlock_kernel();
- if (result)
+ if (result) {
dput(dentry);
- else
+up(dir-i_sem);
+}
+ else {
+ up(dir-i_sem);
result = dentry;
+ /* for avfs */
+ if (!dentry-d_inode lookup_global)
+ lookup_global(nd, dentry);
+ }
}
- up(dir-i_sem);
return result;
}
@@ -487,7 +497,7 @@
/* This does the actual lookups.. */
dentry = cached_lookup(nd-dentry, this, LOOKUP_CONTINUE);
if (!dentry) {
- dentry = real_lookup(nd-dentry, this, LOOKUP_CONTINUE);
+ dentry = real_lookup(nd, this, LOOKUP_CONTINUE);
err = PTR_ERR(dentry);
if (IS_ERR(dentry))
break;
@@ -550,7 +560,7 @@
}
dentry = cached_lookup(nd-dentry, this, 0);
if (!dentry) {
- dentry = real_lookup(nd-dentry, this, 0);
+ dentry = real_lookup(nd, this, 0);
err = PTR_ERR(dentry);
if (IS_ERR(dentry))
break;
diff -ru linux/fs/super.c /spare/src/linux/fs/super.c
--- linux/fs/super.cThu Jul 6 01:10:44 2000
+++ /spare/src/linux/fs/super.c Tue Jul 18 15:41:31 2000
@@ -297,9 +297,9 @@
* dentry (if any).
*/
-static struct vfsmount *add_vfsmnt(struct nameidata *nd,
- struct dentry *root,
- const char *dev_name)
+struct vfsmount *add_vfsmnt(struct nameidata *nd,
+struct dentry *root,
+const char *dev_name)
{
struct vfsmount *mnt;
struct super_block *sb = root-d_inode-i_sb;
diff -ru linux/include/linux/fs.h /spare/src/linux/include/linux/fs.h
--- linux/include/linux/fs.hThu Jul 13 07:01:34 2000
+++ /spare/src/linux/include/linux/fs.h Tue Jul 18 15:42:45 2000
@@ -819,6 +819,8 @@
extern void kern_umount(struct vfsmount *);
extern int may_umount(struct vfsmount *);
extern long do_mount(char *, char *, char *, unsigned long, void *);
+extern struct vfsmount *add_vfsmnt(struct nameidata *nd, struct dentry *root,
+ const char *dev_name);
extern int vfs_statfs
