Re: [RFC PATCH 0/5] Shadow directories

[Al Viro]

On Fri, Oct 19, 2007 at 12:27:16PM +0930, David Newall wrote:

> >Learn to read.  Linux has never allowed that.  Most of the Unix systems
> >do not allow that.
> 
> I did read the claim and it is ambiguous, in that it can reasonably be 
> read to mean that most UNIX systems never allowed such links, which is 
> wrong.  All UNIX systems allowed it until relatively recently.

FVO"relatively recently" exceeding a decade and half.  In any case,
it's _trivial_ to get fs corruption on any system with such links -
play with rename() races a bit and you'll get it.  And yes, it does
include 4.4BSD and quite a chunk of even later history.

Anyway, you are quite welcome to propose a sane locking scheme capable
of dealing with that mess.

As for the posted patch, AFAICS it's FUBAR in handling of .. in such
directories.  Moreover, how are you going to keep that shadow tree
in sync with the main one if somebody starts doing renames in the
latter?  Or mount --move, or...
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH 0/5] Shadow directories

[David Newall]

Al Viro wrote:

On Fri, Oct 19, 2007 at 06:07:45AM +0930, David Newall wrote:
  
considerations of this whole scheme. Linux, like most Unix systems, 
has never allowed hard links to directories for a number of reasons;
  
The claim is wrong.  UNIX systems have traditionally allowed the 
superuser to create hard links to directories.  See link(2) for 2.10BSD 
. 
Having got that wrong throws doubt on the argument; perhaps a path can 
simultaneously be a file and a directory.



Learn to read.  Linux has never allowed that.  Most of the Unix systems
do not allow that.


I did read the claim and it is ambiguous, in that it can reasonably be 
read to mean that most UNIX systems never allowed such links, which is 
wrong.  All UNIX systems allowed it until relatively recently.

-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH 0/5] Shadow directories

[Al Viro]

On Fri, Oct 19, 2007 at 06:07:45AM +0930, David Newall wrote:
> >considerations of this whole scheme. Linux, like most Unix systems, 
> >has never allowed hard links to directories for a number of reasons;
> 
> The claim is wrong.  UNIX systems have traditionally allowed the 
> superuser to create hard links to directories.  See link(2) for 2.10BSD 
> . 
> Having got that wrong throws doubt on the argument; perhaps a path can 
> simultaneously be a file and a directory.

Learn to read.  Linux has never allowed that.  Most of the Unix systems
do not allow that.  Original _did_ allow that, but at the cost of very
easily triggered fs corruption (and it didn't have things like rename(2) -
it _did_ have userland implementation, of course, in suid-root mv(1),
but that sucker had been extremely racy and could be easily used to
screw filesystem to hell and back; adding rename(2) to the set of primitives
combined with multiple links to directories leads to very nasty issues on
_any_ system).
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH 0/5] Shadow directories

[David Newall]

Jaroslav Sykora wrote:

If anybody can think of any other solution of the "redirector problem", possibly
even non-kernel based one, let me know and I'd be glad :-)


If I understand your problem, you wish to treat an archive file as if it 
was a directory.  Thus, in the ideal situation, you could do the following:


cat hello.zip/hello.c
gcc hello.zip/hello.c -o hello
etc..


Rather than complicate matters with a second tree, use FUSE with an 
explicit directory.  For example, ~/expand could be your shadow, thus to 
compile hello.c from ~/hello.zip:


gcc ~/expand/hello.zip^/hello.c -o hello


I think no kernel change would be required.

I'm not keen on the caret.  One of the early claims made in 
http://lwn.net/Articles/100148/ is:
Another branch, led by Al Viro, worries about the locking 
considerations of this whole scheme. Linux, like most Unix systems, 
has never allowed hard links to directories for a number of reasons;


The claim is wrong.  UNIX systems have traditionally allowed the 
superuser to create hard links to directories.  See link(2) for 2.10BSD 
.  
Having got that wrong throws doubt on the argument; perhaps a path can 
simultaneously be a file and a directory.

-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH 0/5] Shadow directories

[Jan Engelhardt]

On Oct 18 2007 22:10, Jaroslav Sykora wrote:
>
>A minor implementation problem with chrooted environment is that the
>FUSE VFS server must be run with root privileges to allow setuid
>programs on the mounted filesystems. But it's certainly doable.

You would not want user-supplied filesystems to carry SUID bits...
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH 0/5] Shadow directories

[Jaroslav Sykora]

On Thursday 18 of October 2007, Jan Engelhardt wrote:
> >> >
> >> >The '^' is an escape character and it tells the computer to treat the 
> >> >file as a directory.
> >> 
> >> But what you could do is: write a FUSE fs that mirrors the lower content
> >> (lofs/fuseloop/however it was named) and expands .zip files as
> >> directories are readdir'ed or the zip files stat'ed. That saves us
> >> from cluttering up the Linux VFS with such stuff.
> >
> >Yes, that's exactly what RheaVFS and AVFS do. Except that they both use an 
> >escape
> >character because:
> >1. without it some programs may break [ http://lwn.net/Articles/100148/ ]
> >2. it's very useful to pass additional parameters after the escape char to 
> >the server.
> >
> >We can start VFS servers (mentioned above) and chroot the whole user session 
> >into
> >the mount directory of the server. It works but it's very slow, practically 
> >unusable.
> 
> Sounds like a program bug, since NTFS-3G is proof of concept that FUSE
> can be fast.
> 

Good point, I'll look onto it.

A minor implementation problem with chrooted environment is that the FUSE VFS 
server
must be run with root privileges to allow setuid programs on the mounted 
filesystems.
But it's certainly doable.


-- 
"Elves and Dragons!" I says to him.  "Cabbages and potatoes are better
for you and me."  -- J. R. R. Tolkien
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH 0/5] Shadow directories

[Jan Engelhardt]

On Oct 19 2007 05:32, David Newall wrote:
>
> The claim is wrong.  UNIX systems have traditionally allowed the
> superuser to create hard links to directories.  See link(2) for
> 2.10BSD
> .
> Having got that wrong throws doubt on the argument; perhaps a path
> can simultaneously be a file and a directory.

But hell will break lose if you allow hardlinking directories.

mkdir /tmp/a
ln /tmp/a /tmp/a/b

And you would not be able to rmdir /tmp/a/b because the directory is
not empty (it contains "b" [full path: /tmp/a/b/b]).
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH 0/5] Shadow directories

[Jan Engelhardt]

On Oct 18 2007 19:07, Jaroslav Sykora wrote:
>> On Oct 18 2007 17:21, Jaroslav Sykora wrote:
>> >Hello,
>> >
>> >Let's say we have an archive file "hello.zip" with a hello world program 
>> >source
>> >code. We want to do this:
>> >cat hello.zip^/hello.c
>> >gcc hello.zip^/hello.c -o hello
>> >etc..
>> >
>> >The '^' is an escape character and it tells the computer to treat the file 
>> >as a directory.
>> 
>> But what you could do is: write a FUSE fs that mirrors the lower content
>> (lofs/fuseloop/however it was named) and expands .zip files as
>> directories are readdir'ed or the zip files stat'ed. That saves us
>> from cluttering up the Linux VFS with such stuff.
>
>Yes, that's exactly what RheaVFS and AVFS do. Except that they both use an 
>escape
>character because:
>1. without it some programs may break [ http://lwn.net/Articles/100148/ ]
>2. it's very useful to pass additional parameters after the escape char to the 
>server.
>
>We can start VFS servers (mentioned above) and chroot the whole user session 
>into
>the mount directory of the server. It works but it's very slow, practically 
>unusable.

Sounds like a program bug, since NTFS-3G is proof of concept that FUSE
can be fast.

>If anybody can think of any other solution of the "redirector
>problem", possibly even non-kernel based one, let me know and I'd be
>glad :-)
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH 0/5] Shadow directories

[Jaroslav Sykora]

On Thursday 18 of October 2007, Jan Engelhardt wrote:
> 
> On Oct 18 2007 17:21, Jaroslav Sykora wrote:
> >Hello,
> >
> >Let's say we have an archive file "hello.zip" with a hello world program 
> >source
> >code. We want to do this:
> > cat hello.zip^/hello.c
> > gcc hello.zip^/hello.c -o hello
> > etc..
> >
> >The '^' is an escape character and it tells the computer to treat the file 
> >as a directory.
> 
> Too bad, since ^ is a valid character in a *file*name. Everything is, with
> the exception of '\0' and '/'. At the end of the day, there are no control
> characters you could use.
> 
> But what you could do is: write a FUSE fs that mirrors the lower content
> (lofs/fuseloop/however it was named) and expands .zip files as
> directories are readdir'ed or the zip files stat'ed. That saves us
> from cluttering up the Linux VFS with such stuff.
> 

Yes, that's exactly what RheaVFS and AVFS do. Except that they both use an 
escape
character because:
1. without it some programs may break [ http://lwn.net/Articles/100148/ ]
2. it's very useful to pass additional parameters after the escape char to the 
server.

We can start VFS servers (mentioned above) and chroot the whole user session 
into
the mount directory of the server. It works but it's very slow, practically 
unusable.
So both servers need some kind of VFS redirector. In the past there were many
different approaches -- LD_PRELOAD hack, CodaFS hack, NFS hack (?), 
proof-of-concept
kernel hacks (project podfuk) etc.

If anybody can think of any other solution of the "redirector problem", possibly
even non-kernel based one, let me know and I'd be glad :-)

-- 
I find television very educating. Every time somebody turns on the set, 
I go into the other room and read a book.
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH 0/5] Shadow directories

[David Newall]

David Newall wrote:

David Newall wrote:

Jaroslav Sykora wrote:
Let's say we have an archive file "hello.zip" with a hello world 
program source

code. We want to do this:
cat hello.zip^/hello.c
gcc hello.zip^/hello.c -o hello
etc..
  


Wouldn't you do this as a user space filesystem?

Which is what you were saying.

*SMACK* I so stupid.


On third thoughts, what's the reason for this?
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH 0/5] Shadow directories

[David Newall]

David Newall wrote:

Jaroslav Sykora wrote:
Let's say we have an archive file "hello.zip" with a hello world 
program source

code. We want to do this:
cat hello.zip^/hello.c
gcc hello.zip^/hello.c -o hello
etc..
  


Wouldn't you do this as a user space filesystem?

Which is what you were saying.

*SMACK* I so stupid.
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH 0/5] Shadow directories

[David Newall]

Jaroslav Sykora wrote:

Let's say we have an archive file "hello.zip" with a hello world program source
code. We want to do this:
cat hello.zip^/hello.c
gcc hello.zip^/hello.c -o hello
etc..
  


Wouldn't you do this as a user space filesystem?
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH 0/5] Shadow directories

[Jan Engelhardt]

On Oct 18 2007 17:21, Jaroslav Sykora wrote:
>Hello,
>
>Let's say we have an archive file "hello.zip" with a hello world program source
>code. We want to do this:
>   cat hello.zip^/hello.c
>   gcc hello.zip^/hello.c -o hello
>   etc..
>
>The '^' is an escape character and it tells the computer to treat the file as 
>a directory.

Too bad, since ^ is a valid character in a *file*name. Everything is, with
the exception of '\0' and '/'. At the end of the day, there are no control
characters you could use.

But what you could do is: write a FUSE fs that mirrors the lower content
(lofs/fuseloop/however it was named) and expands .zip files as
directories are readdir'ed or the zip files stat'ed. That saves us
from cluttering up the Linux VFS with such stuff.
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[RFC PATCH 0/5] Shadow directories

[Jaroslav Sykora]

Hello,

Let's say we have an archive file "hello.zip" with a hello world program source
code. We want to do this:
cat hello.zip^/hello.c
gcc hello.zip^/hello.c -o hello
etc..

The '^' is an escape character and it tells the computer to treat the file as a 
directory.
[Note: We can't do "cat hello.zip/hello.c" because of 
http://lwn.net/Articles/100148/ ]
The kernel patch implements only a redirection of the request to another 
directory
("shadow directory") where a FUSE server must be mounted. The decompression of 
archives is entirely  handled in the user space. More info can be found in the 
documentation
patch in the series.

The shadow directories are used in RheaVFS project [ 
http://rheavfs.sourceforge.net/ ],
and it also can be used with the original AVFS [ 
http://www.inf.bme.hu/~mszeredi/avfs/ ].

The patches are against vanilla 2.6.23.
This is my first bigger contribution to the kernel so please be gentle ;-)

Jara

-- 
"Elves and Dragons!" I says to him.  "Cabbages and potatoes are better
for you and me."  -- J. R. R. Tolkien
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html