Re: reiser4 plugins
Stefan Smietanowski wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ok, still haven't heard much discussion of metafs vs file-as-directory, but it seems like it'd be easier in metafs. Why not implement it inside the directory containg the file ? Ie the metadata for /home/stesmi/foo is in /home/stesmi/.meta/foo This should be suit both camps I'd think? You still need to figure out the parent of "foo", which isn't necessarily easy, especially considering that even if we store a link to the parent, it will be inside the metadata. Then you have a chicken-and-egg situation. Both camps seem to want to allow easy access to the metadata of a file, whether we're given that file as an inode or as a pathname. That's why I suggested /meta/vfs and /meta/inode -- sometimes I want to look up a file by name, and sometimes by inode, but either way, I should be able to get its metadata. I mean, editing something is easy and you don't have to "know" how to navigate /meta But then you have to "know" how to navigate .meta, and more importantly, how to find .meta and you don't have the clash of files vs metadata (is /meta/vfs/home/stesmi/foo a file or an attribute called foo of the dir stesmi ?). So, how do I get metadata of a directory? If the metadata for /home/me is in /home/.meta/me, and the metadata for /home is in /.meta/home, then where is the metadata for / ? /home/stesmi/foo <- dir /home/stesmi/.meta/foo <- "dir" containing all metadata /home/stesmi/.meta/foo/attrib <- some attribute called attrib /home/stesmi/foo/bar <- file /home/stesmi/foo/.meta/bar <- "dir" containg all metadata /home/stesmi/foo/.meta/bar/attrib <- some attribute called attrib [...] If this has already been taken up, I must've missed it. It looks a lot like how I suggested we resolve the ambiguity within /meta/vfs, only I called it '...' instead of '.meta'. Either way, the major challenge to this method is not so much whether it'd work, but how to choose a suitable delimiter? The delimiter must be standard if we're going to have apps develop for it, and it must not be used in the name of any existing file or directory. I had a long essay here on how '.meta' breaks every recursive operation on directories (rm -rf, tar, mkisofs), but I deleted it when I remembered that I played around with the alpha/beta reiser4 which had a 'metas' dir that did the same thing, but was hidden from the normal directory listing. 'cd metas' worked, 'ls metas' worked, but 'ls' showed no directory called 'metas'. I don't *think* there are any apps that will break if you tell them to open a path that doesn't exist in a directory listing. That is, typing 'vim /home/metas/acl' should always let me edit the acl on /home, and vim should never complain about how /home/metas doesn't show up in /home. A program would have to be pretty retarded to fail on something like that. But, we have to support some pretty retarded programs. That is why I like /meta -- the default view is a completely POSIX-compliant tree that works with tar, and even the /meta view is POSIX-compliant, even if it'd be a bad idea to tar it. Then we don't have to worry at all about stupid programs. How much should we be worrying about this particular brand of stupidity? And what's a good delimiter? -- No virus found in this outgoing message. Checked by AVG Anti-Virus. Version: 7.0.323 / Virus Database: 267.8.12/46 - Release Date: 7/11/2005 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: reiser4 plugins
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Neil Brown wrote: > On Monday July 11, [EMAIL PROTECTED] wrote: > >>Stefan Smietanowski wrote: >> >>>-BEGIN PGP SIGNED MESSAGE- >>>Hash: SHA1 >>> >>> >>> Ok, still haven't heard much discussion of metafs vs file-as-directory, but it seems like it'd be easier in metafs. >>> >>> >>>Why not implement it inside the directory containg the file ? >>> >>>Ie the metadata for /home/stesmi/foo is in /home/stesmi/.meta/foo >>> >>>This should be suit both camps I'd think? >> >>You still need to figure out the parent of "foo", which isn't >>necessarily easy, especially considering that even if we store a link to >>the parent, it will be inside the metadata. Then you have a >>chicken-and-egg situation. >> >>Both camps seem to want to allow easy access to the metadata of a file, >>whether we're given that file as an inode or as a pathname. That's why >>I suggested /meta/vfs and /meta/inode -- sometimes I want to look up a >>file by name, and sometimes by inode, but either way, I should be able >>to get its metadata. > > > Well, it's not really 'as an inode or as a pathname'. It is 'as an > open file descriptor or as a path name' which is an important > difference. > > Maybe it is worth repeating Al Viro's suggestion at this point. I > don't have a reference but the idea was basically that if you open > "/foo" and get filedescriptor N, then >/proc/self/fds/N-meta How am I supposed to get there with a shell script? -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQtROqXgHNmZLgCUhAQLRcg/+I9PWSmFXRwKtj7pnEeMXOCjiTo6GQE3O 61fjH3f6aL9Ydkip4eXum3S14cioiU9bzC11GA5kRIM+W1DKcYex1dIpivrtF9Ht Rvozn9x2TP5tacDmSfqRJXvAB+xTRtZOu+M/rDjXdLsriDJDA0AdyDH8Yo/8WMbU 6i1DWzLTO0vHS3kEb/8oqgBj7sQ63sS/4KVszBx6+bN0KOikXbORDu6efKjC9w21 3DZPnBG0B03smhdCygd0j0Zmh0JEaZHfuFgNK1ZmRzipbvvUBDtdKY5MJ6f4pHnA GBO8ybsXp9qxNQr6DNenF/wbAT6n3dMyE/AWuql+qx3iumSwx/Prh7xDAhBZBMXp Oin7hOa1i583cdju4ErSBPaciRzumGluY6gbFvVA8Yva+IjPxxjPtfLwalK11cH1 k4oQO5Par1W0TmMOpc0PQ/bEeEHHxUcn1ToeJa4NYJWIiBe+UHMb/AyI4hKjSIkt Kp0wrCPBjRfuBCHXXL89bWZoSeSFkN8fAyOxBV928naxxr8e+vCPUX1/H3ca7UsB Nxg0Vzt4V9tz4xCw4QAy810Uya8/HSm3aVhqEzjHKBoKboHrMVDJvxRQxfkqQcnC 4jIFYPBdHgGw7OONyhfbgTPLIm1OCNPpcRkS4aidHqg0DkDU50h6zFQkhG5Xwh5Z x5REgxbqD+A= =FGm4 -END PGP SIGNATURE- - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: reiser4 plugins
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hans Reiser wrote: > Horst von Brand wrote: > > >>Hans Reiser <[EMAIL PROTECTED]> wrote: >> >> >> >>>Stefan Smietanowski wrote: >>> >>> >>> I think "..." and ".meta" both serve as a logical delimiter. However some programs implement their own "..." which would make it clash with them. Naturally if some program created a directory called .meta we're equally screwed. >> >> >> >> >>>I chose '' (four dots) because it clashes with less, not three dots. >>> >>> >> >>Is this some kind of "My dots are more than yours" contest?! >> >>/None/ of them is safe. ".meta", "...", "", ".this.has.five.dots." are >>all perfectly legal file (or directory) names, POSIXly. If any one of them >>won't do, none will. And, conversely, if any one of them makes sense, they all do. Well, with the exception that some of them have never, ever been used before, and that's what we should be aiming for. No conflicts with existing programs, then new programs can be written to avoid conflicting with us. > There is a long history of encroaching namespaces by creating new > keywords in CS, and it is a survivable problem. Agreed, but not one to be taken too lightly. > Clearcase (the version control filesystem) is an excellent example. > They have special meanings for @ and some other common characters, and > you have to learn how to escape those characters if you use them in > Clearcase. I like using @, though. Without having to escape it. That's why we're trying to find something that people won't actually touch, especially since if we design it right, this will be the last delimiter introduced at the fs/vfs level. > It works, it makes users happy, in practice it is far less of a problem > than one might think. I think there were two or three times I had to > remember how to escape things in 2 years of using it. > > Better to spend one's mind looking for bugs instead of this issue. .if bugs were seen as such a big deal. I think it's far easier to get into the kernel with something ludicrously buggy than something which actually changes fundamental behavior. That is, you can put in an FS which actually corrupts data (such as the old NTFS write support), so long as it doesn't break POSIX, or cause other weird restrictions like "No files named 'metas'" Now, if we can decide that we don't care about being in the vanilla kernel, then we can just call it ".metas" or "lost+found" or whatever and get to work on bug fixes and other much-needed features such as a repacker. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQtRQvXgHNmZLgCUhAQJXWBAAlZggJQGNi2lEZh0MAqnz+rNVT1JYcY/b adHgvVOZZNiJw2GmVjGLIiG5cqSqML1//f0+4XOxjYED2rbfOwBJiDqcq/0IsKPp 5JJflJV2uWkEZukJ+oA5mspfSeKof2N5egBoD1Ije079VBKXdjoN5Kprkbe4ZYZ2 +Yu+w3FpdcaGvSqVlTRHPWsS0je4z8ieh0u6ql+HNR4ze/hKwMw4nrb2sARW9NOQ EZ8Ot5NDhVaxz9Rj72rLVuQrUZEF8b0ihkpmzTauVV/nysEGi33SPqYTF7nYGBnM 5mVY63uXG4zxmThMDpn+J/iofA/hS1fd1bY9tCgF1ZAPu9HrCTnVzKaTYoOq+ciD sY0m7HEc49JfaiyZ/SJGH02WUH3JqLQAVGQftEkqAQLyYdVRbzdBHUVUR2i8hX2M ofFLM6DGJgFK784PfO0sjNboQT5ay4B8js8NltdfytsOVwDzjMST7dZAWcnMgTZU jrMCJuMYeYh8468fy1C9pCMUPahVXvmF4O5Xazt7m9HvV4lxRJIUDZsaR0Volg2K OAQIO3rtT9heK/+/PaUuXX8RYwLQUlhdUmdHfdtjRiLtKRcbw/fYBOqjkgXZfOfy RzyBgD463sVCVAE8qMbAVnnHNGKZ25tyRTYITiTZ2kfnIeURJawj2SZVmn5ezGQG xRnI3+Ir5e0= =moA6 -END PGP SIGNATURE- - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: reiser4 plugins
Hans Reiser wrote: David Masover wrote: That's why we're trying to find something that people won't actually touch, especially since if we design it right, this will be the last delimiter introduced at the fs/vfs level. Uh, no, there needs to be about a dozen or so more. Where? From what I (vaguely) remember of the future-vision paper, having the meta delimiter lets us do everything else from inside the metas. We can certainly add delimiters to stuff in a meta-dir... -- No virus found in this outgoing message. Checked by AVG Anti-Virus. Version: 7.0.323 / Virus Database: 267.8.12/46 - Release Date: 7/11/2005 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: reiser4 plugins
Horst von Brand wrote: David Masover <[EMAIL PROTECTED]> wrote: David Weinehall wrote: On Fri, Jul 01, 2005 at 03:08:58AM -0500, David Masover wrote: David Weinehall wrote: GNOME and KDE run on operating systems that run other kernels than Linux, hence they have to implement their own userland VFS anyway. Adding this to the Linux kernel won't help them one bit, unless we can magically convince Sun to add it to Solaris, all different BSD:s to add it to their kernels, etc. Not going to happen. An effort to get GNOME and KDE to unify their VFS:s would be far more benificial, Than what? Creating a unified VFS which I can access from Bash, and which obsoletes both GNOME and KDE's VFSes except in their presentation? On one of the platforms that they support, yes. But only for kernels newer than 2.6.yy... So they'd still have to have their own VFS for 2.4.xx, 2.6.xx (xx < yy), FreeBSD, OpenBSD, Solaris, etc... Right. But, /proc started somewhere, didn't it? Sun. I have the feeling that other systems will duplicate it if it's good. Linux copied here. So what? Even if they don't, it would be more beneficial to me How, exactly? Go back and read. Besides, /your/ convenience isn't the only thing that matters... Nor yours. Just because you can't get your mind around a concept doesn't mean that it's a bad concept, and that no one else can use it. and probably most Linux users "Most Linux users" don't use experimental filesystems at all... Actually, they do -- ext3 was experimental once. ReiserFS was very experimental once. Please stop bashing it just because it's new/experimental. to have metafs supported in both GNOME and KDE, even if they still need an emulation layer to support other systems. So Gnome and KDE get larger (and thus slower) for everybody. Smaller (and thus faster) on supported systems, otherwise exactly the same, but maybe a little more modular, which is good. Besides, Gnome and KDE will have to agree on the formats involved first, which is /exactly/ what is supposed to be impossible unless this stuff is implemented in the kernel... I never said that, but for one thing, whether they do or not, it's nice if my shell and my editor and all the other things that I use don't have to agree on anything to manipulate the formats involved. This is not just about GNOME/KDE. It is about GNOME/KDE not developing an additional layer, that you wouldn't like anyway, that cannot be accessed from anything except GNOME/KDE. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: reiser4 plugins
Horst von Brand wrote: Kevin Bowen <[EMAIL PROTECTED]> wrote: [...] So, for instance, if I want to grab all mp3s with Artist "Paul Oakenfold" and change the genre to "techno" (can you do that?), I can use Beagle's search tool to find all mp3s by Oakenfold, but to change the genre, I have to use some separate tool to manipulate id3 tags, Yes, this is basically what I was getting at, although I was thinking more in terms of the reiser5/6/whatever set theoretic semantics, which, from my point of view at least, reiser4 is simply the first step towards building the enabling infrastructure of. But the fact that reiser4 semantics + trivial shell scripting enables, as illustrated by David, the rough equivalent of set-theoretic semantics, demonstrates how reiser4 is in fact a step in this direction. I don't see any "trivial shell scripting", just need for a plethora of magic extract-this-or-that-metadata programs. Which can very well work exactly the same on independent files, no need to shove junk /into/ the indexed files. moment the case of system-wide or network-wide shared data, I.e., something like 90% of the use of Linux here. OK. users needs. In fact, I believe there is currently a JSR in progress to develop a more sophisticated Java packaging model. Presumably based on ReiserFS 4, which then has to be ported to whatever platform you want to run Java on ASAP? Great for you! Wait a bit, and you'll get what you want then, even across the board! No, obviously that's not what I was saying. But the need for these kinds of domain-specific packaging/metadata formats, each requiring their own tools to work with, would be alleviated if there were simply a more powerful filesystem semantic. *Please explain HOW*!! The domain-specific formats /will/ stay (if nothing else, because the /domains/ have /specific/ needs), the special tools to work with them /will/ have to be written exactly the same (because of the above). All for use on /one/ non-standard filesystem. One filesystem that exists right now. /proc, as people have made clear, was originally implemented on *one* "non-standard" Unix. Others then copied it. I see no reason why if /meta is a good idea, and programs start using it, that other FSes won't implement it. Plus exactly the same stuff to work on sane filesystems. "sane" -- I assume you mean "traditional". Well, yeah. So? How big a program do you need to create a new interface to an existing system? Consider: BitTorrent has at least three interfaces that I can remember at the moment. btdownloadgui.py, btdownloadcurses.py, btdownloadheadless.py. Are you saying that btdownloadgui.py was so much work if you start from btdownloadcurses.py that there should be no GTK library, because it takes so much work to convert existing curses apps to GTK? Remember, even if I have GTK installed, I can still get to my curses apps, and they may not even know that GTK exists. Even if I have /meta enabled, I can still use my POSIX apps, and they may not even know that /meta exists. So-called 'database-filesystems' ARE coming, whether from Microsoft, Apple, or us. IBM did it, long ago (AS/400 and OS/400 ring a bell?). And is now slowly moving away from it (and other structured filesystems), AFAICS, towards (guess what!) POSIX and Linux... Linux over POSIX, I think. Because Linux is big and open-source, not beacuse POSIX is so great. Again, Linux is what it is today in large part because "We have to get $FEATURE, because if not, $COMPETITOR will win on us!" arguments have no traction. I agree with that, at least. But Linux is also what it is today in large part because "I don't need $FEATURE, so it should be kept out" isn't a particularly good argument either. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: reiser4 plugins
Hans Reiser wrote: > Hubert Chan wrote: > > >>On Fri, 01 Jul 2005 03:06:19 -0500, David Masover <[EMAIL PROTECTED]> said: >> >> >> >> >>>Hubert Chan wrote: >>> >>> >> >> >> >> >>>>The main thing blocking file-as-dir is that there are some >>>>locking(IIRC?) issues. And, of course, some people wouldn't want it >>>>to be merged into the mainline kernel. (Of course, the latter >>>>doesn't prevent Namesys from maintaining their own patches for people >>>>to play around with.) >>>> >>>> >> >> >> >> >>>What's the locking issue? I think that was more about transactions... >>> >>> >> >>It was whatever was Al Viro's (technical) complaint about file-as-dir. >>I don't remember exactly what it was. The technical people know what it >>is (and the Namesys guys are probably working on it), and the exact >>issue doesn't concern us non-technical people that much, so I don't feel >>like looking it up. But if you want to, just look for Al Viro's message >>in this thread. >> >> >> > > Cycle detection when hard links to directories are allowed. There is a > debate over whether cycle detection is feasible that can only be > resolved by working code or a formal proof that it is not > computationally feasible. Ah. But then, one solution was to avoid the issue at all, and have the directory inside a file act as a mountpoint. After all, mount --bind doesn't cause problems... Hey! This sounds like metafs (/meta) already! I wonder if we can do file-as-dir in /meta, and just not support user-created hardlinks there? (other than creating brand-new files, of course...) - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: reiser4 plugins
Horst von Brand wrote: > David Masover <[EMAIL PROTECTED]> wrote: > >>Horst von Brand wrote: >> >>>David Masover <[EMAIL PROTECTED]> wrote: >>> >>>>David Weinehall wrote: >>>> >>>>>On Fri, Jul 01, 2005 at 03:08:58AM -0500, David Masover wrote: >>>>> >>>>>>David Weinehall wrote: > > > [...] > > >>>>Even if they don't, it would be more beneficial to me > > >>>How, exactly? > > >>Go back and read. > > > I have. And have seen /no/ benefit to you. Except, of course, the benefit > accrued from some magic in Linus' kernel, by which all format differences > go in a puff of smoke if they are implemented inside it, and furthermore > all userland gets rebuilt to use the kernel's way overnight. Let's say cryptocompress gets implemented. Not all of userland rewritten, not even any of userland rewritten, just a cryptocompress plugin for the kernel. And instead of having to learn a new tool, I can just browse around in /meta. Yes, all of userland gets rebuilt, and the world is better for it. But *incrementally*. Not overnight. > Here, the "how should it be done, if at all" discussion has barely started. > And AFAICS there are plenty of ways of getting 95% of the advantages > /without/ touching the kernel, Which advantages can we get without touching the kernel? And why does it make sense to do them outside the kernel? "Because we can" is a little HURD-ish... > so that should be the path to follow. Nobody > has shown any real evidence at all for this not being so. Go back and read again. >> Just because you can't get your mind around a concept >>doesn't mean that it's a bad concept, and that no one else can use it. > > > I /do/ get my mind around the concept, it /is/ sometimes useful. It /can/ > be done without the kernel, No, it can't. Unless you patch *all* of userland. > and so /should/ be done that way. That is all. I'll refer you to my HURD comment: Just about anything done currently in the kernel can be done in userland. Witness HURD. Doesn't mean it should. Witness HURD. >>>> and probably >>>>most Linux users > > >>>"Most Linux users" don't use experimental filesystems at all... > > >>Actually, they do -- ext3 was experimental once. > > > Right. And ext2, and ext, and xiafs, and ... So? When they were > experimental, only a handful of utter fools commited their data to them. But, they had to start somewhere. You seem hell-bent on keeping anything "unproven" out of the kernel and away from users. How do you suggest it "prove" itself, then? >>>>to have metafs supported in both GNOME and KDE, even >>>>if they still need an emulation layer to support other systems. >>> >>>So Gnome and KDE get larger (and thus slower) for everybody. > > >>Smaller (and thus faster) on supported systems, > > > Sorry, but just because some $DISTRO gives ReiserFS4 as an /option/ doesn't > mean they will have everything duplicated as "For ReiserFS4" and "For other > filesystems". My assertion stands, until there are ReiserFS4-only > distributions. Why not? The "For Reiser4" version is just a thin layer on top of metafs. And after all, metafs can be implemented for other filesystems. And, as someone else pointed out, dealing with different systems doing things in different ways is what portability is all about. >>>Besides, Gnome >>>and KDE will have to agree on the formats involved first, which is /exactly/ >>>what is supposed to be impossible unless this stuff is implemented in the >>>kernel... > > >>I never said that, > > > You (and others) Others, then. > have told us time and again that each of them have their > own incompatible ways of handling metadata, and that only if this was > handled in the kernel they would make use of a common way of managing it... All I ever said was that it would be easier for them to get along, and certainly *much* easier to use existing tools (the commandline) to poke inside the metadata, whether it's compatible or not. Ultimately, maybe it leads to cooperation, maybe not. But there are immediate benefits even if it doesn't lead to cooperation. >> but for one thing, whether they do or not, it's >>nice if my shell and my editor and all the other things that I use >>don't have to agree on anything to manipulate the formats involved. > >
Re: reiser4 plugins
Jeremy Maitin-Shepard wrote: > David Masover <[EMAIL PROTECTED]> writes: > > [snip] > > >>>I have. And have seen /no/ benefit to you. Except, of course, the benefit >>>accrued from some magic in Linus' kernel, by which all format differences >>>go in a puff of smoke if they are implemented inside it, and furthermore >>>all userland gets rebuilt to use the kernel's way overnight. > > >>Let's say cryptocompress gets implemented. Not all of userland >>rewritten, not even any of userland rewritten, just a cryptocompress >>plugin for the kernel. And instead of having to learn a new tool, I can >>just browse around in /meta. > > > What is the relationship between file-as-dir or special meta-data and > transparent encryption+compression? I do not see why file-as-dir would > require such a special interface. I'm ignoring file-as-dir until/unless someone figures out a sane way of doing it. Under Reiser4, think of all files/directories as objects. Meta-data, as accessed through file-as-dir or through /meta, is just a collection of methods to that file/directory. Some methods are just accessor (setter/getter) methods, like foo.mp3/artist -- they are effectively attributes, although there may be a little more logic going on there (foo.mp3/artist probably is connected to foo.mp3's id3 tag). Some methods actually do something, like secret.txt/compression. The reason I'm bringing that up is that some people seem determined to keep the very concept out of the kernel because it's new, strange, and doesn't seem to have any real applications. I want it in because it has real, fairly immediate applications, and much bigger payoffs down the road, and I don't think it's that strange -- it's a nice, clean design. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: reiser4 plugins
Hans Reiser wrote: > David Masover wrote: > > >>Hans Reiser wrote: >> >> >> >>>Hubert Chan wrote: >>> >>> >>> >>> >>> >>>>On Fri, 01 Jul 2005 03:06:19 -0500, David Masover <[EMAIL PROTECTED]> said: >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>>>Hubert Chan wrote: >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> >>>> >>>>>>The main thing blocking file-as-dir is that there are some >>>>>>locking(IIRC?) issues. And, of course, some people wouldn't want it >>>>>>to be merged into the mainline kernel. (Of course, the latter >>>>>>doesn't prevent Namesys from maintaining their own patches for people >>>>>>to play around with.) >>>>>> >>>>>> >>>>>> >>>>>> >>>> >>>> >>>> >>>> >>>> >>>>>What's the locking issue? I think that was more about transactions... >>>>> >>>>> >>>>> >>>>> >>>> >>>>It was whatever was Al Viro's (technical) complaint about file-as-dir. >>>>I don't remember exactly what it was. The technical people know what it >>>>is (and the Namesys guys are probably working on it), and the exact >>>>issue doesn't concern us non-technical people that much, so I don't feel >>>>like looking it up. But if you want to, just look for Al Viro's message >>>>in this thread. >>>> >>>> >>>> >>>> >>>> >>> >>>Cycle detection when hard links to directories are allowed. There is a >>>debate over whether cycle detection is feasible that can only be >>>resolved by working code or a formal proof that it is not >>>computationally feasible. >>> >>> >> >>Ah. But then, one solution was to avoid the issue at all, and have the >>directory inside a file act as a mountpoint. After all, mount --bind >>doesn't cause problems... >> >> > > Can you explain this idea at greater length? Just don't allow user-created hardlinks inside either metafs (/meta) or the magical meta directory inside files. In order to do that, one way would be to have "file/..." appear as a mountpoint. I don't know if this is feasable, performance-wise, but I think it would work. >>Hey! This sounds like metafs (/meta) already! I wonder if we can do >>file-as-dir in /meta, and just not support user-created hardlinks there? >>(other than creating brand-new files, of course...) This is still my preferred solution, because it's not any harder or less efficient to develop new apps based on metafs than on file-as-dir, but it means that old apps will see something *entirely* POSIX-compliant, and the strangeness will be confined to /meta. Basically, you have to allow hardlinks in /meta, in case some plugin wants to do that, but if you have files that are also directories, you also have to make sure that users can't create hardlinks. I don't think this would be particularly hard, though. Pretend it's a read-only filesystem unless the user is doing something safe, like creating a brand-new file, deleting an old one, or modifying the contents of an existing one, all assuming that the plugin responsible for the file/directory you're doing this in allows it. But then, if we're doing metafs, I don't think you need file-as-directory, and certain existing tools that we'd like to be able to point at metafs might not like file-as-directory anyway. Now, can anyone think of a situation where we want user-created hardlinks inside metadata? More importantly, what do we do about it? - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: reiser4 plugins
Hans Reiser wrote: Hubert Chan wrote: On Tue, 05 Jul 2005 20:50:08 -0400 EDT, "Alexander G. M. Smith" <[EMAIL PROTECTED]> said: That sounds equivalent to no hard links (other than the usual parent directory one). If there's any directory with two links to it, then there will be a cycle somewhere! What we want is no directed cycles. That is A is the parent of B is the parent of C is the parent of A. We don't care about A is the parent of B is the parent of C; A is the parent of B is the parent of C. OK, here's a random idea that just popped into my head, and to which I've given little thought (read: none whatsoever), and may be the stupidest idea ever proposed on LKML, but thought I would just toss it out to see if it could stimulate someone to come up with something better (read: sane): Conceptually, foo/ is just a symlink to /meta/[filesystem]/[inode of foo]. Except that we want the metafiles to go away when the base file goes away. Only, /meta is a filesystem that already makes stuff go away for us, so all we have left is the issue of whether using /meta costs us performance, or whether breaking POSIX to add a symlink (such as foo/...) really gives us that much more usability. I don't know the first thing about whether it costs us performance, although it seems like it could be negligable considering the existance of mount --bind. I don't think file-as-dir gives us that much more usability, because we can always create a simple program or shell script that 'cd's us into metadata. It's still easier than having a simple program that manipulates the metadata directly, because this way we can do 'cd' and 'ls' and so on inside the metadata directory. And, once we start talking about applications, /meta will be more readily supported (as in, some apps will go through a pathname and stop when they get to a file, and then there's tar). On apps which don't have direct support for /meta, you'd be navigating to the file in question and then manually typing '...' into the dialog, so I don't see why typing '...' at the end is better than typing '/meta' or '/meta/vfs' at the beginning. That said, I'm still not entirely sure how we get /meta/vfs to work other than adding a '...' sort of delimiter anyway. And a question: is it feasible to store, for each inode, its parent(s), instead of just the hard link count? Ooh, now that is an interesting old idea I haven't considered in 20 years makes fsck more robust too Doesn't it make directory operations slower, too? And, will it require a format change? - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: reiser4 plugins
Horst von Brand wrote: David Masover <[EMAIL PROTECTED]> wrote: [...] Just don't allow user-created hardlinks inside either metafs (/meta) or the magical meta directory inside files. And what is it useful for, after its advantage was that it was /exactly/ like regular files &c, and now it is severely crippled? The advantage was never that meta files look exactly like regular files, but that they look enough like regular files that you can use existing tools on them. Of course, there are some times when you want meta files to look exactly like regular files, as in (I hate to bring it up again, but...) zip files. So, a zip file (/meta/vfs/some/zip/file/.../contents/) would allow hardlinks between files inside the zipfile, but not outside of it. This would be like creating a separate mountpoint for the zipfile's contents, and doing "mount --bind" when a hardlink to the zipfile is created. I'm not sure if we actually have to pretend it's a mountpoint, or if we can just take the checking that mountpoints already do and generalize it. Can you think of a way in which hardlinks are useful in /meta, in a *general* way, instead of within a specific directory controlled by a specific plugin? - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: reiser4 plugins
Horst von Brand wrote: Hubert Chan <[EMAIL PROTECTED]> wrote: On Fri, 01 Jul 2005 03:41:00 -0400, Chet Hosey <[EMAIL PROTECTED]> said: Horst von Brand wrote: And who says that a normal user isn't allowed to annotate each and every file with its purpose or something else? Explain how you currently allow users to annotate arbitrary files. By keeping annotations /outside/ the files. [...] The situation is even better with file-as-dir. If the administrator wants to allow users to edit the description metadata for the file foo, the administrator can set the appropriate permissions for foo/.../description, and keep foo read-only. So now root is responsible in exquisite detail for random other users being able to keep info about my files? If it's the general info that's associated with the file, and may even be stored inside the file, then yes, that's fair. Although I could certainly imagine foo/.../descriptions being a directory that's world-writable, allowing each user to maintain their own file inside of it. You can even set these per-user descriptions to be stored somewhere else, like the user's home directory, and that could work for CDs. Actually, you could use something like unionfs to allow users to keep their own annotations without affecting everyone else's. Again, root has to mount that stuff for each and every user? Why is that a problem? Put it in a script. Mount each user's unionfs at boot. And it's "something like unionfs" -- maybe it's a feature of metafs or reiserfs that we haven't thought of yet. It certainly can't be unionfs as it stands, as unionfs doesn't work on top of any reiser. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: reiser4 plugins
Hans Reiser wrote: If we also add to this the restriction that once you create the file part of a file-directory, you can never hardlink to a child of it, it should then all work, yes? So, /filename//owner should be able to avoid colliding with any common names by virtue of the '', and not letting any filedir (file-directory) have children with links to them should also work. The one thing that seems inelegant is that when you create the file part of a filedir, you must check all its children for hardlinks and fail if they exist, and you must flag all its directory children so that the plugins for them will disallow hardlinks to their children from that point onward. Still, seems workable Ok, still haven't heard much discussion of metafs vs file-as-directory, but it seems like it'd be easier in metafs. Basically, we are entirely POSIX compliant outside of metafs, so that /filename is a file and may be hardlinked, and /directory/ is a directory and may not. No problems yet. Inside /meta/inode, we have no problems yet because any hardlinks created outside /meta won't even show up as hardlinks here, so we don't get hardlinked directories. Inside /meta/vfs, where we can find the metadata of '/filename' under '/meta/vfs/filename/...', we have what looks like a problem -- we may have hardlinks created outside metafs, which will show up as two directories inside metafs, so those directories are hardlinks. I don't think that's such a problem, since we won't allow users to change anything in /meta/vfs except when it's inside metadata, such as '/meta/vfs/some/where/...'. Thus, it's impossible to create a hardlink to a directory unless we do it *outside* metafs, as a hardlink to a file. Now, the question I asked is, what if we want hardlinks *inside* metafs, *inside* the metadata for a given file, and would we ever want such a thing? Because we can avoid the whole problem if we just disallow any sys_link calls inside metafs. Of course, sometimes we want to have a chunk of metadata that appears *exactly* as if it were a normal, POSIX-compliant directory tree, such as the contents of a tarball or a zipfile. In that case, we might want to have the "zip" plugin allow hardlinks inside "/meta/file.zip/.../contents" -- the only restriction is that any hardlink made to a file inside 'contents' must be made to another file inside 'contents'. Hans Reiser wrote: David Masover wrote: Now, can anyone think of a situation where we want user-created hardlinks inside metadata? More importantly, what do we do about it? I think the equivalent of symlinks would be good enough to get by on for now for most linking of metafiles. Maybe some years from now somebody can fault me for saying this and write a patch to fix it to be better, at which point I will be happy to concede the point. So the basic principal here is, one can have hardlinks to directories without cycles provided that one does not allow any child of the directory to have a hardlink. The question is, how cleanly can that relaxed restriction be coded? Hans - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: reiser4 plugins
Adrian Ulrich wrote: so all we have left is the issue of whether using /meta costs us performance, or whether breaking POSIX to add a symlink (such as foo/...) really gives us that much more usability. IMHO '/meta' isn't such a good idea, because a chrooted application won't be able to use it. mount --bind. Is there a performance hit for having too many of those? mount --bind /meta/vfs/some/chroot /some/chroot/meta Also, maybe a separately-mounted /meta, maybe with something like '-o root=' I can't think of when you'd have a chrooted application that uses /meta, but wasn't written with /meta in mind, so as to have these mount commands in its init scripts. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: reiser4 plugins
Hans Reiser wrote: David Masover wrote: And, once we start talking about applications, /meta will be more readily supported (as in, some apps will go through a pathname and stop when they get to a file, and then there's tar). On apps which don't have direct support for /meta, you'd be navigating to the file in question and then manually typing '...' into the dialog, so I don't see why typing '...' at the end is better than typing '/meta' or '/meta/vfs' at the beginning. Performance. If you type it at the end, and you already have done the lookup of the filename, then you can go from the file to one of its methods, instead of a complete new traversal of another tree under /meta Only, it's a different view of the same tree. That may not matter performance-wise, though. Also, for performance-critical applications, the /meta tree is pretty easy -- it becomes more like /meta/inode/some_inode_number/. There's also the chroot issue, though. That said, I'm still not entirely sure how we get /meta/vfs to work other than adding a '...' sort of delimiter anyway. And a question: is it feasible to store, for each inode, its parent(s), instead of just the hard link count? Ooh, now that is an interesting old idea I haven't considered in 20 years makes fsck more robust too Doesn't it make directory operations slower, too? Not sure. It consumes space though. And, will it require a format change? Yes, but we have plugins now, so. So, will the format change happen at mount time? Will it need a special mount flag? Will I need to use debugfs or some other offline tool? - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: reiser4 plugins
Jonathan Briggs wrote: On Wed, 2005-07-06 at 15:51 -0400, Hubert Chan wrote: On Wed, 06 Jul 2005 12:52:23 -0600, Jonathan Briggs <[EMAIL PROTECTED]> said: [snip] It still has the performance and locking problem of having to update every child file when moving a directory tree to a new parent. On the other hand, maybe the benefit is worth the cost. Every node should store the inode number(s) for its parent(s). Not the path name. So you don't need to do any updates, since when you move a tree, inode numbers don't change. You do need the updates if you change what inode is the parent. /a/b/c, /a/b/d, /a/b/e, /b mv /a/b /b Now you have to change the stored grand-parent inodes for /a/b/c, /a/b/d and /a/b/e so that they reference /b/b instead of /a/b. no, c, d, and e all would reference /a/b's *inode*, not *pathname*. Then /a/b would reference /a. So with the above mv, you just change the reference in /a/b (now /b/b) to point to /b as parent. Only way you would have to touch c, d, and e is if you did: mkdir /b/b mv /a/b/* /b/b/ but, since you're also essentially unlinking them from /a/b and linking them into /b, it's not that much more of a performance hit to update one pointer in the file. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: reiser4 plugins
Hubert Chan wrote: On Wed, 06 Jul 2005 16:33:23 -0400, Horst von Brand <[EMAIL PROTECTED]> said: Hubert Chan <[EMAIL PROTECTED]> wrote: If you can store the parents, then finding cycles (relatively) quickly is pretty easy: before you try to make A the parent of B, walk up the parent pointers starting from A. If you ever reach B, you have a cycle. If not, you don't have a cycle. (Hmmm. Do I need a proof of correctness for this? It's just depth-first-search, which everybody learned in their first algorithms course.) Correct. And you need space for potentially a huge lot of up pointers for each file. People (that I know of) don't normally have a huge lot of hardlinks to a single file. And speaking of which, the only doomsday scenario (running out of RAM) that I can think of with this scheme is if we have a ton of hardlinks to the same file and we try to move one of them. But this scales linearly with the number of hardlinks, I think. Maybe not quite, but certainly not exponentially. The only other doomsday scenario is if we have a ludicrously deep tree. To make this work in real usage, not DOS testing, we really need both of those, and even then I'm not sure it can work. What's the maximum number of hardlinks supported to a single file? What's the maximum tree depth? Can these be limited to prevent actual DOS attempts? Now, if we were to ever actually *create* a cycle, then yes, we might end up traversing the whole tree to delete it, possibly running out of RAM. But we don't ever actually create cycles except if we were to have corruption, which could as easily create cycles in any other FS. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: reiser4 plugins
Markus Törnqvist wrote: On Fri, Jul 01, 2005 at 05:54:46PM +0200, David Weinehall wrote: Which would neither need VFS changes nor be dependent on Reiser4 in any way, so I don't see why this thread lives on. Just get down to business and implement this metafs =) I've been gone for a while and suddenly drowning in mail... Anyway, I don't really like the metafs thing. To access the data, you still need to refactor userspace, so that's not a real advantage. Doing lookups from /meta all the time, instead of in the file-as-dir-whatever... I don't really see the disadvantage. Also, metafs means much less of a fight to get people to adopt the whole meta concept, because it can be done in a POSIX-compliant way which doesn't break tar. File-as-dir is nice if you're using meta files, but it causes lots of unexpected weirdness. I don't think metafs costs us much in performance, and with one or two shell scripts, it wouldn't cost us that much efficiency on the commandline. But, I also like file-as-dir. I think it might be time for a vote. I vote metafs. Or, maybe Hans needs to tell us which way we're going... And the best thing to do would be to bring these "Reiser4-specific" enhancements to every FS. Which has nothing to do with whether it's done in "metafs" or not. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: reiser4 plugins
Hubert Chan wrote: On Wed, 06 Jul 2005 23:42:50 -0700, Hans Reiser <[EMAIL PROTECTED]> said: Oh no, don't store the whole path, store just the parent list. This will make fsck more robust in the event that the directory gets clobbered by hardware error. I don't think it affects the cost of detecting cycles though, I think it only makes fsck more robust. It doesn't affect the worst-case cost of detecting cycles; by necessity, any (static [1]) directed cycle detection algorithm must take O(n) time, n being the size of the tree (nodes + links). However, under "reasonable assumptions" (where the reasonableness of those assumptions may be questioned, but I think they're reasonable), I think that doing a depth-first-search through the parent links makes the algorithm not-too-terrible. Namely, the "reasonable assumptions" are that a node doesn't have "too many" parents, and the filesystem hierarchy is not "too deep". And, remember, there are other things affected by depth of the tree anyway. For that matter, most of the time, when you push a system past "reasonable assumptions", you hit performance issues, if not stability ones. Most everything but Reiser assumes that you won't have "too many" files in a directory, or "too many" small files in the FS as a whole. I think it's fair to assume people will keep things "reasonable", especially if we tell them what "reasonable" means. As in, "This is the kind of organization which Reiser4 handles really well, and this is the kind of organization which will absolutely kill your performance." - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
