[bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[James Youngman]

Update of bug #17877 (project findutils):

  Status:None => Wont Fix   
 Assigned to:None => jay
 Open/Closed:Open => Closed 

___

Follow-up Comment #13:

Looks like no further change is needed in findutils, so closing.

___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

Re: [bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Miklos Szeredi]

> > I'm very reluctant to try opening every file in the
> > filesystem just in case it turns out to be a directory we need to
> > descend into.
> 
> Your reluctance is understandable, since it is incorrect to
> always open every directory entry.  The corresponding file
> might be a special file, and opening devices can have side
> effects (e.g., opening and closing a tape device might
> rewind it).  Similarly for other kinds of files (e.g.,
> fifos).

Right.  And since the lstat and the open can be very far apart, an
unprivileged user can _very_easily_ do DoS against find by changing a
directory into a fifo, while find is traversing some deep directory
tree.

There's O_DIRECTORY on Linux (I don't know about other systems) to
handle this kind of problem.

On systems not supporting O_DIRECTORY, moving the lstat() and the
open() close to each other still makes a lot of sence from the
security POV.

So fts seems to have a security bug, as well as not working for
non-POSIX filesystems.

And they have a common underlying reason.  If you assume too much
about the system, you will easily fall into these kind of security
traps.  If you assume as little as possible about the system, it will
not only be more secure, but will also work better on sub-standard
systems.

Miklos


___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

Re: [bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Paul Eggert]

"James Youngman" <[EMAIL PROTECTED]> writes:

> I'm very reluctant to try opening every file in the
> filesystem just in case it turns out to be a directory we need to
> descend into.

Your reluctance is understandable, since it is incorrect to
always open every directory entry.  The corresponding file
might be a special file, and opening devices can have side
effects (e.g., opening and closing a tape device might
rewind it).  Similarly for other kinds of files (e.g.,
fifos).  If the app knows the file cannot be a special file
or a fifo or etc. then it should be OK, but 'find' doesn't
know this in general.


___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

Re: [bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[James Youngman]

On 10/7/06, Miklos Szeredi <[EMAIL PROTECTED]> wrote:

> Yes, for systems with O_NOFOLLOW, that is a perfect (efficient, race
> free) solution.  For systems without O_NOFOLLOW, just moving the
> lstat() and the open() close to each other

Actually moving the lstat() _after_ the open() totally removes the
race for inode-less filesystems.  The following should be equivalent
to open(O_NOFOLLOW):

fd = open(path);
lstat(path, &st1);
fstat(fd, &st2);
if (st1.st_ino != st2.st_ino)
/* ELOOP */;


For find, the stat result (in the absence of type information in
struct dient) determines if we need to consider descending into a
directory.  I'm very reluctant to try opening every file in the
filesystem just in case it turns out to be a directory we need to
descend into.  But then, the underlying principle of your technique
will doubtless be useful even if it needs to be adapted to work
efficiently for find.

James


___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

Re: [bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Paul Eggert]

Miklos Szeredi <[EMAIL PROTECTED]> writes:

> From what I understand, the solution for the find
> problem will actually make fts more optimal in some cases even for
> POSIX filesystems, and will cause no regressions whatsoever.

If so, then that's good for fts.  The problem will continue
to remain for other applications, though.

> You are clearly just making this up,

This discussion is going nowhere fast.  Let's turn to other matters.


___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

Re: [bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Miklos Szeredi]

> Yes, for systems with O_NOFOLLOW, that is a perfect (efficient, race
> free) solution.  For systems without O_NOFOLLOW, just moving the
> lstat() and the open() close to each other

Actually moving the lstat() _after_ the open() totally removes the
race for inode-less filesystems.  The following should be equivalent
to open(O_NOFOLLOW):

fd = open(path);
lstat(path, &st1);
fstat(fd, &st2);
if (st1.st_ino != st2.st_ino)
/* ELOOP */;

The above should be no less efficient than what applications do now:

lstat(path, &st1)
/* ... */ 
fd = open(path)
fstat(fd, &st2)
if (st1.st_ino != st2.st_ino)
/* ELOOP */;

Miklos


___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

Re: [bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Miklos Szeredi]

> > > There must already be such problems with hard links anyway, so this
> > > extra problem isn't that much worse.
> >
> > Yes it is.  Neither sftp, nor smbfs, nor fat support hard links.  All
> > of them support rename.
> 
> Not entiirely accurate.   SSHFS supports whatever the underlying
> remote filesystem supports.  That might include hard links.  That is,
> directories A and B visible under SSHS could both be symbolic links to
> some other directory.  Files C and D could be hard links to the same
> inode.  However, the ls command of SFTP does not have any kind of "-i"
> option and so it is impossible to discover that this is the case.

Yes, under sshfs a hard linked file would look like two different
files, that just happen to be magically synchornized.  There's nothing
wrong with this from the POSIX POV.

> > What about embedded devices, where the table would not fit in memory?
> > My guess is that an entry in such a table would be minimum 32bytes.  A
> > million such entries would take 32Mbytes.
> 
> Not sure what you're suggesting here.   You clearly wouldn't be
> suggesting that everybody's interpretation of struct stat.st_ino
> should change just because of potential implementation limitation of a
> wristwatch-based Linux system.  So I'm not sure what you *are*
> suggesting here.

You don't even want sshfs to permanently (until unmount) use 32Mbytes
on your desktop system, just because you did a 'find' on a filesystem
with a million nodes.  OK, firefox is worse than that, but still ;)

And there are filesystems with not a million but hundreds of millions
of nodes.  The point is, just permanently storing the mapping is not
the solution.

I'm not against an option for fuse, that tells it the number of inodes
to cache.  Then paranoid people can set it to a million or whatever.
That's OK, but it still does not generally solve the problem.

> Clearly using a hash function to supply inode numbers from path names
> would not reqiure you to store an in-memory hash table.   The hash
> value is enough.

No, because if you rename a file, the inode number and the path name
hash will no longer match.

> > > Again, it is unreasonable to expect standard utilities to port around
> > > file systems that don't have reliable inode numbers.  It breaks too
> > > many other things: simplicity and security being the most important
> > > two.
> >
> > It breaks neither of those.  You are clearly just making this up,
> > without understanding the issues.  If you are actually interested in
> > what's involved in fixing an application please follow the discussion
> > on gnulib-bugs.
> 
> Let's move a little towards a position where we can make progress.  As
> to how findutils might change,
> 
> 0. Changes to make find functinally incorrect (i.e. give the wrong
> answer ans return qith exit status 0) will not be (deliberately) made
> 1. Changes to find which reduce its level of security will only be
> accepted if this is necessary to correct functional defects
> 2. I'm basically not willing to make changes to fix (even functional)
> problems on accessory filesystems if this reduces the level of
> security of the program on 'core' filesystems.   (The distinction here
> is between for example the filesystem that / or /home is on and other
> stuff like AFS).
> 3. Code changes must follow the GNU coding standards (including
> copyright assignments, as the FSF requires).
> 4. I prefer changes which include documentation, and, normally, prefer
> changes that are small rather than changes that are big.

All of these sound perfectly reasonable.  Jim Meyering said, he has a
patch under testing.  Hopefully that will resolve this issue with fts
and find.  As I've said I'm not worried about other utilities, which
only rely on stable inodes for stat() and open() calls close to each
other.  That doesn't mean those can't be improved by using O_NOFOLLOW
if available, but it's not high priority.

> > > People will just have to learn to not trust running standard
> > > utilities on file systems that lack reliable-enough inode numbers.
> 
> Well, for find one of the principal problems is that there's no way
> around using directory inode numbers for security checks.  On POSIX
> filesystems without O_NOFOLLOW (and perhaps some with, if the
> semantics are synthesised) there's no alternative.

Yes, for systems with O_NOFOLLOW, that is a perfect (efficient, race
free) solution.  For systems without O_NOFOLLOW, just moving the
lstat() and the open() close to each other is a reasonably good
solution, and obviously no less secure, than if the lstat() and the
open() are far apart (which is the problem with fts).

> Filesystems not having support for symbolic links don't need the
> checks.  But I can't think of a way to tell if we're looking at such a
>  filesystem.   I would expect that using readlink() won't work because
> I expect you would get EINVAL on non-supporting filesystems, just as
> you do on symlink-supporting filesystems for non-symlin

Re: [bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[James Youngman]

On 10/7/06, Miklos Szeredi <[EMAIL PROTECTED]> wrote:

[ JY: I assume that "> >" quotes statements by Paul Eggert, but the
attribution has been lost ]


> There must already be such problems with hard links anyway, so this
> extra problem isn't that much worse.

Yes it is.  Neither sftp, nor smbfs, nor fat support hard links.  All
of them support rename.


Not entiirely accurate.   SSHFS supports whatever the underlying
remote filesystem supports.  That might include hard links.  That is,
directories A and B visible under SSHS could both be symbolic links to
some other directory.  Files C and D could be hard links to the same
inode.  However, the ls command of SFTP does not have any kind of "-i"
option and so it is impossible to discover that this is the case.


What about embedded devices, where the table would not fit in memory?
My guess is that an entry in such a table would be minimum 32bytes.  A
million such entries would take 32Mbytes.


Not sure what you're suggesting here.   You clearly wouldn't be
suggesting that everybody's interpretation of struct stat.st_ino
should change just because of potential implementation limitation of a
wristwatch-based Linux system.  So I'm not sure what you *are*
suggesting here.

Clearly using a hash function to supply inode numbers from path names
would not reqiure you to store an in-memory hash table.   The hash
value is enough.  You could perhaps use a pair of Bloom filters to
account for collisions (though this would mean that two SSHFS-using
clients might disagree about the inode number of a given file).


> Again, it is unreasonable to expect standard utilities to port around
> file systems that don't have reliable inode numbers.  It breaks too
> many other things: simplicity and security being the most important
> two.

It breaks neither of those.  You are clearly just making this up,
without understanding the issues.  If you are actually interested in
what's involved in fixing an application please follow the discussion
on gnulib-bugs.


Let's move a little towards a position where we can make progress.  As
to how findutils might change,

0. Changes to make find functinally incorrect (i.e. give the wrong
answer ans return qith exit status 0) will not be (deliberately) made
1. Changes to find which reduce its level of security will only be
accepted if this is necessary to correct functional defects
2. I'm basically not willing to make changes to fix (even functional)
problems on accessory filesystems if this reduces the level of
security of the program on 'core' filesystems.   (The distinction here
is between for example the filesystem that / or /home is on and other
stuff like AFS).
3. Code changes must follow the GNU coding standards (including
copyright assignments, as the FSF requires).
4. I prefer changes which include documentation, and, normally, prefer
changes that are small rather than changes that are big.

Changes within these guidelines are welcome and encouraged.   The only
patch currently attached to the bug report is one which didn't fix the
problem.



> People will just have to learn to not trust running standard
> utilities on file systems that lack reliable-enough inode numbers.


Well, for find one of the principal problems is that there's no way
around using directory inode numbers for security checks.  On POSIX
filesystems without O_NOFOLLOW (and perhaps some with, if the
semantics are synthesised) there's no alternative.

Filesystems not having support for symbolic links don't need the
checks.  But I can't think of a way to tell if we're looking at such a
filesystem.   I would expect that using readlink() won't work because
I expect you would get EINVAL on non-supporting filesystems, just as
you do on symlink-supporting filesystems for non-symlinks.  Creating a
symlink to verify that you get EPERM is also something that find
certainly shouldn't do.

James.


___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

Re: [bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Miklos Szeredi]

> > If you were thinking of a soltuion, where the filesystem itself
> > supplies two different inode numbers based on the variant of the
> > userspace interface, then I'm sorry to inform you, that this sort of
> > interface is not likely to happen in the Linux kernel.
> 
> I don't see why not.

Because it would be a horrible hack, and generally those are frowned
upon by Linux developers.  If you don't beleive me, just talk to Al
Viro about it.

> (which has the side effect of dumbing the utilities down and making
> them less secure).

This is just FUD.  From what I understand, the solution for the find
problem will actually make fts more optimal in some cases even for
POSIX filesystems, and will cause no regressions whatsoever.

> > However, I realized that inode numbers generated by hashing the path
> > have another fatal problem.  Namly that if you rename a file or
> > directory, then the inode number will of course move with the name,
> 
> I was relying on your earlier statement that a file was uniquely
> identified by its path, and that therefore a rename results in a
> different file.  Those semantics are not great, but they're better
> than nothing.

And are blatantly POSIX violating.

> There must already be such problems with hard links anyway, so this
> extra problem isn't that much worse.

Yes it is.  Neither sftp, nor smbfs, nor fat support hard links.  All
of them support rename.

> >> Sure it is.  All it takes is sufficient resources.  You could maintain
> >> an auxiliary inode table outside the file system, for example.
> >
> > And this could be an ever growing table, as the filesystem may not be
> > aware of any removals on the server end.
> 
> That's not a problem.  If the table has garbage and gets too large
> then the file system can garbage-collect it.

Based on what?  The filesystem has _no_ information about which inodes
may be still valid, and which may not.

> I don't seriously expect you to rush out and implement the inode table
> right away.  All I'm trying to show is that it's indeed possible to
> implement POSIX semantics, so that the claim that "it is
> _NOT_POSSIBLE_ to make these filesystems conform to POSIX" is
> incorrect.  With an auxiliary table, one can even implement
> POSIX-style renames and hard-links.  The issue is not whether it is
> _possible_ to conform to POSIX; it clearly is.

What about embedded devices, where the table would not fit in memory?
My guess is that an entry in such a table would be minimum 32bytes.  A
million such entries would take 32Mbytes.

Yes with infinite resources the problem is clearly solvable.  But
that's not what I'm interested in.

> The issue is to find a reasonable design point where the most
> commonly-run applications work and where it doesn't cost too much to
> implement it.

Yup.

> Again, it is unreasonable to expect standard utilities to port around
> file systems that don't have reliable inode numbers.  It breaks too
> many other things: simplicity and security being the most important
> two.

It breaks neither of those.  You are clearly just making this up,
without understanding the issues.  If you are actually interested in
what's involved in fixing an application please follow the discussion
on gnulib-bugs.

> People will just have to learn to not trust running standard
> utilities on file systems that lack reliable-enough inode numbers.

We'll see.

Miklos


___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

Re: [bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Paul Eggert]

Miklos Szeredi <[EMAIL PROTECTED]> writes:

> If you were thinking of a soltuion, where the filesystem itself
> supplies two different inode numbers based on the variant of the
> userspace interface, then I'm sorry to inform you, that this sort of
> interface is not likely to happen in the Linux kernel.

I don't see why not.  Why can't the file system tell what system call
was used (the legacy smallfile one or the largefile one)?  Surely
there is some way to find out, even if it's not as convenient as we'd
like.  And if there's no such way, then add one.

If it's too much trouble to implement this, then just issue 32-bit
hashes.  That's not perfect, but it'll be good enough for most casual
uses.  Or have it be a mount-time option whether to use 32-bit hashes
(to support smallfile legacy apps, albeit with some collisions
possible) or 64-bit hashes (to support modern apps, with little
possibility of collision).

There are lots of ways to go here, without having to port standard
utilities to substandard file systems (which has the side effect of
dumbing the utilities down and making them less secure).

> If you were thinking of automatically converting a 64bit inode number
> to a 32bit one (e.g. truncating it)

Yes, I was thinking of the _filesystem_ doing that.  Not glibc.  Not
the kernel.  The _filesystem_.  It's the filesystem's responsibility
to know that its inode numbers are somewhat bogus; the kernel and/or
glibc can't be expected to know that on their own.

> However, I realized that inode numbers generated by hashing the path
> have another fatal problem.  Namly that if you rename a file or
> directory, then the inode number will of course move with the name,

I was relying on your earlier statement that a file was uniquely
identified by its path, and that therefore a rename results in a
different file.  Those semantics are not great, but they're better
than nothing.  There must already be such problems with hard links
anyway, so this extra problem isn't that much worse.

>> Sure it is.  All it takes is sufficient resources.  You could maintain
>> an auxiliary inode table outside the file system, for example.
>
> And this could be an ever growing table, as the filesystem may not be
> aware of any removals on the server end.

That's not a problem.  If the table has garbage and gets too large
then the file system can garbage-collect it.

I don't seriously expect you to rush out and implement the inode table
right away.  All I'm trying to show is that it's indeed possible to
implement POSIX semantics, so that the claim that "it is
_NOT_POSSIBLE_ to make these filesystems conform to POSIX" is
incorrect.  With an auxiliary table, one can even implement
POSIX-style renames and hard-links.  The issue is not whether it is
_possible_ to conform to POSIX; it clearly is.  The issue is to find a
reasonable design point where the most commonly-run applications work
and where it doesn't cost too much to implement it.

Again, it is unreasonable to expect standard utilities to port around
file systems that don't have reliable inode numbers.  It breaks too
many other things: simplicity and security being the most important
two.  People will just have to learn to not trust running standard
utilities on file systems that lack reliable-enough inode numbers.


___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

Re: [bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Miklos Szeredi]

> > The legacy app will break regardless of how many files there are on
> > the filesystem, or even wheter it needs to use the inode number or
> > not.  It will break because the stat() family of syscalls will return
> > with an error.
> 
> I don't see why.  The kernel should be able to tell whether the app
> has called the old (smallfile) stat function, or the new stat
> function.  If the app calls the old stat function, the kernel can give
> it a 32-bit inode number.  If the app calls the new stat function, the
> kernel can give it a 64-bit inode number.  The two inode numbers do
> not have to agree.

There's also glibc in between the kernel and the app.  In fact the
EOVERFLOW error is generated by glibc, not the kernel.  The kernel
just quietly truncates the inode number AFAICS.

If you were thinking of a soltuion, where the filesystem itself
supplies two different inode numbers based on the variant of the
userspace interface, then I'm sorry to inform you, that this sort of
interface is not likely to happen in the Linux kernel.

If you were thinking of automatically converting a 64bit inode number
to a 32bit one (e.g. truncating it), that may be OK, I don't know why
glibc is more paranoid than the kernel in this respect.

However, I realized that inode numbers generated by hashing the path
have another fatal problem.  Namly that if you rename a file or
directory, then the inode number will of course move with the name,
but after that the stability of the number is not any more guaranteed.
It's back to where we started, only with a more complex scheme.

> This is not a perfect solution (for example, a smallfile legacy app
> can't print an inode number and give it to a modern app), but it's
> good enough.  Failure modes will be quite rare and easily explained,
> just as they're explained for the 2 GiB limitation on smallfile legacy
> aps (a problem that's considerably more important).
> 
> > it is _NOT_POSSIBLE_ to make these filesystems conform to POSIX.
> 
> Sure it is.  All it takes is sufficient resources.  You could maintain
> an auxiliary inode table outside the file system, for example.

And this could be an ever growing table, as the filesystem may not be
aware of any removals on the server end.

> But I'm trying to help generate a solution that's more efficient
> than that, one that is "good enough" to support the obsolescent
> software you're worried about.

Cool.  Still waiting for the breakthrough :)

Miklos


___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

Re: [bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Paul Eggert]

Miklos Szeredi <[EMAIL PROTECTED]> writes:

> The legacy app will break regardless of how many files there are on
> the filesystem, or even wheter it needs to use the inode number or
> not.  It will break because the stat() family of syscalls will return
> with an error.

I don't see why.  The kernel should be able to tell whether the app
has called the old (smallfile) stat function, or the new stat
function.  If the app calls the old stat function, the kernel can give
it a 32-bit inode number.  If the app calls the new stat function, the
kernel can give it a 64-bit inode number.  The two inode numbers do
not have to agree.

This is not a perfect solution (for example, a smallfile legacy app
can't print an inode number and give it to a modern app), but it's
good enough.  Failure modes will be quite rare and easily explained,
just as they're explained for the 2 GiB limitation on smallfile legacy
aps (a problem that's considerably more important).

> it is _NOT_POSSIBLE_ to make these filesystems conform to POSIX.

Sure it is.  All it takes is sufficient resources.  You could maintain
an auxiliary inode table outside the file system, for example.  But
I'm trying to help generate a solution that's more efficient than
that, one that is "good enough" to support the obsolescent software
you're worried about.


___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

Re: [bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Miklos Szeredi]

> >> If it's for legacy apps, tell people to recompile with largefile
> >> support.
> >
> > Paul, please!  It is ridiculous to require end users to recompile
> > their applications or kernels or anything.
> 
> A small-file legacy app, one that cannot handle files larger than 2
> GiB, is already quite a limited app these days.  Such a program will
> do just fine with 32-bit inode numbers in practice, as people won't
> expect much of it.  If someone complains that such an app doesn't work
> with a lot of files (e.g., the 470,000 files on your root file
> system), then you tell them "sorry, you've got a legacy app built for
> a small environment; please recompile it".  It's exactly what you'd
> tell them when their program doesn't work with a 2.5 GiB file.

You haven't been reading carefully enough.

The legacy app will break regardless of how many files there are on
the filesystem, or even wheter it needs to use the inode number or
not.  It will break because the stat() family of syscalls will return
with an error.

Believe me, you don't want this.

> >> There is a standard for this, a standard that reflects longstanding
> >> practice.
> >
> > Which is what I have been doing.
> 
> Then I'm afraid I'm totally lost.  If POSIX says that inodes must be
> unique and stable, and if the file system is conforming to POSIX, then
> what's the problem?

The problem (as I have stated _many_ times) is that it is
_NOT_POSSIBLE_ to make these filesystems conform to POSIX.  I'm
eagerly waiting for an idea that proves me wrong.

Your suggestion for using 64bit hashes came close, but it turns out to
be unworkable on the majority of systems.

Miklos


___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

Re: [bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Paul Eggert]

Miklos Szeredi <[EMAIL PROTECTED]> writes:

>> If it's for legacy apps, tell people to recompile with largefile
>> support.
>
> Paul, please!  It is ridiculous to require end users to recompile
> their applications or kernels or anything.

A small-file legacy app, one that cannot handle files larger than 2
GiB, is already quite a limited app these days.  Such a program will
do just fine with 32-bit inode numbers in practice, as people won't
expect much of it.  If someone complains that such an app doesn't work
with a lot of files (e.g., the 470,000 files on your root file
system), then you tell them "sorry, you've got a legacy app built for
a small environment; please recompile it".  It's exactly what you'd
tell them when their program doesn't work with a 2.5 GiB file.

But even if you disagree with me, and want to implement some other
solution for small-file apps that would cause small-file GNU 'find' to
break, then that's OK.  GNU 'find' is not compiled in small-file mode.
Just get the filesystem to work in largefile mode, and leave GNU
'find' alone.  (Similarly for GNU coreutils, etc.)

>> That's much better than asking people to rewrite thousands of apps.
>
> I think you are the only one asking for that.

The issue in question is much larger than GNU 'find'.  It applies to
many programs that use gnulib, and it applies to many other programs
that don't use gnulib.  There's no way we're going to rewrite them
all, particularly if the rewrites make the programs less secure on
file systems that conform to the standard.

> I'm not holding my breath for these bug reports, sorry.

It's better to take a proactive approach, and fix buggy areas when
warned about them, before users find out about them; particularly when
the failure mode is so obscure, as it will be here.

>> There is a standard for this, a standard that reflects longstanding
>> practice.
>
> Which is what I have been doing.

Then I'm afraid I'm totally lost.  If POSIX says that inodes must be
unique and stable, and if the file system is conforming to POSIX, then
what's the problem?


___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

Re: [bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Miklos Szeredi]

> >> If files are identified by the path, then you can hash the
> >> path. If you use a good 64-bit hash the chance of collision
> >> is practically zero. That's good enough.
> >
> > Yes.  And this solution is actually practical on pure 64bit
> > archs only.  On 32bit and dual archs it's not practial, because
> > legacy apps (those compiled without largefile support)
> 
> First, we're not talking about legacy apps here.  The applications
> we're talking about all have largefile support, as do the vast
> majority of other applications that I use daily.  So if you come up
> with a solution that works well with largefile apps, and does
> something sort-of-reasonable for legacy apps, that's fine.
> 
> Second, even 32-bit hashes are pretty good.  The chances of their
> screwing up in practice are quite small, if you have good hash
> functions.

For a perfect 32bit hash function the chance of a collision in a set
of 10,000 files is approx. 1%.  The chance of a collision in a set of
100,000 files is approx 70%.  I have 470,000 files on my root
partition, the chance of a collision within that is for all intents
and purposes 100%.

It may not matter in practice, but it would be hard to call that POSIX
conforming, which _very_ clearly states that inode numbers are
_unique_.

> > This is not a theoretical
> > possibility, I've had bug reports in this area.
> 
> If it's for legacy apps, tell people to recompile with largefile
> support.

Paul, please!  It is ridiculous to require end users to recompile
their applications or kernels or anything.  You are living in some
dream world I think, where the users serve the programmers, and not
the other way round.

> That's much better than asking people to rewrite thousands of apps.

I think you are the only one asking for that.

> (If you're getting lots of bug reports for legacy apps, then
> improve the quality of your 32-bit hash functions.  :-)
> 
> > Currently I have report of exactly 1 (one) application,
> > that breaks.
> 
> Sorry, that's not sufficient evidence.  I'm sure lots of other apps
> will break, but people won't necessarily know where to send the bug
> reports.

I'm not holding my breath for these bug reports, sorry.  Sshfs may be
a newcomer, but smbfs and fat has been around for quite some time.  If
they would break applications, those bugs would have been reported
already.

> There is a standard for this, a standard that reflects longstanding
> practice.

Which is what I have been doing.  Had I been using a hash function in
FUSE to get the inode number I would have been breaking long standing
practice.

> Just conform to the standard, and you'll be fine.

Neither uniqe-unstable or collinding-stable inode numbers conform to
POSIX.

Miklos


___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

Re: [bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Paul Eggert]

Miklos Szeredi <[EMAIL PROTECTED]> writes:

>> If files are identified by the path, then you can hash the
>> path. If you use a good 64-bit hash the chance of collision
>> is practically zero. That's good enough.
>
> Yes.  And this solution is actually practical on pure 64bit
> archs only.  On 32bit and dual archs it's not practial, because
> legacy apps (those compiled without largefile support)

First, we're not talking about legacy apps here.  The applications
we're talking about all have largefile support, as do the vast
majority of other applications that I use daily.  So if you come up
with a solution that works well with largefile apps, and does
something sort-of-reasonable for legacy apps, that's fine.

Second, even 32-bit hashes are pretty good.  The chances of their
screwing up in practice are quite small, if you have good hash
functions.

> This is not a theoretical
> possibility, I've had bug reports in this area.

If it's for legacy apps, tell people to recompile with largefile
support.  That's much better than asking people to rewrite thousands
of apps.  (If you're getting lots of bug reports for legacy apps, then
improve the quality of your 32-bit hash functions.  :-)

> Currently I have report of exactly 1 (one) application,
> that breaks.

Sorry, that's not sufficient evidence.  I'm sure lots of other apps
will break, but people won't necessarily know where to send the bug
reports.

There is a standard for this, a standard that reflects longstanding
practice.  Just conform to the standard, and you'll be fine.


___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

Re: [bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[James Youngman]

On 10/6/06, Miklos Szeredi <[EMAIL PROTECTED]> wrote:


Yes.  And this solution is actually practical on pure 64bit
archs only.  On 32bit and dual archs it's not practial, because
legacy apps (those compiled without largefile support) will
get an EOVERFLOW for stat if the inode number doesn't fit in
32 bits.  This is irrespective of whether the app actually
would use st_ino or not.


The problem already exists for NFS.  There are NFS implementations
with 64-bit inodes that either
(a) just don't return names of files whose inode number is >int32_max
fromreaddir
or
(b) Return the inode number modulo int32_max+1.

Some systems (for example IRIX I think) can be switched between the
two behaviours.  See also comments about CIFS at
http://www.ussg.iu.edu/hypermail/linux/kernel/0511.1/1045.html

James.


___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

[bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Miklos Szeredi]

Follow-up Comment #12, bug #17877 (project findutils):

> If files are identified by the path, then you can hash the
> path. If you use a good 64-bit hash the chance of collision
> is practically zero. That's good enough.

Yes.  And this solution is actually practical on pure 64bit
archs only.  On 32bit and dual archs it's not practial, because
legacy apps (those compiled without largefile support) will
get an EOVERFLOW for stat if the inode number doesn't fit in
32 bits.  This is irrespective of whether the app actually
would use st_ino or not.

So in fact 64 bit inode numbers would _break_ a lot of
currently installed systems.  This is not a theoretical
possibility, I've had bug reports in this area.

Any other solutions?

> I may sound a bit impatient here, but it's impatience for a
> reason. Knowledge of file serial numbers must be hardwired
> into thousands of programs. We can't change them all. If you
> have to support inodes by some inefficient means, then I'm
> afraid that's what you'll have to do. (Or, if you prefer,
> you can warn users of these file systems that they will
> break lots of programs. :-)

Currently I have report of exactly 1 (one) application,
that breaks.  That is an alpha version of find, for which
a solution is actually in the works (see gnulib-bugs ml).

So I think your impatience is rather unwarranted.


___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

[bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Paul Eggert]

Follow-up Comment #11, bug #17877 (project findutils):

>> if some Linux-based file systems can't provide stable
>> inode numbers, they should be fixed so that they do. It
>> shouldn't be that hard.
>
> It's not hard, it's impossible.  Take for example path
> based network filesystems (smbfs, sshfs).  For these the
> only identifier for files is the path.

If files are identified by the path, then you can hash the
path.  If you use a good 64-bit hash the chance of collision
is practically zero.  That's good enough.

I may sound a bit impatient here, but it's impatience for a
reason.  Knowledge of file serial numbers must be hardwired
into thousands of programs.  We can't change them all.  If you
have to support inodes by some inefficient means, then I'm
afraid that's what you'll have to do.  (Or, if you prefer,
you can warn users of these file systems that they will
break lots of programs.  :-)


___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

Re: Fwd: [bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Miklos Szeredi]

> > > But what about symlinks?
> > >
> > >   a   g
> > >b   h->a
> > > c
> > >  f->g
> > >
> > > The moment you traverse the f->g symlink above,
> > > the entire tree, a/b/c/f, is no longer referenced,
> > > so the h->a link may take you back to a new inode,
> > > and the cycle will not be detected.
> >
> > Right.  So find either holds a file descriptor for each symlink
> > traversed,
> 
> But at the time we traverse the symlink, we don't know it's a symlink.
>   Here's the order of events:
> 
> process 1 stats "." and remembers the resulting device and inode.
> process 1 stats c.   It is told that c is a directory, not a symlink.
> process 1 chdirs into c.  Now, CWD=/a/b/c, and that's where it thinks it is.
> process 1 stats ".." and compares the device and inode with the previous 
> value.
>   - these are the same so the security check succeeds.
> process 1 stats d.  It is told that d is a directory, not a symlink.
> process 2 renames d to d~, and creates a symlink d pointing at /g/h
> process 1 chdirs into d.  Now CWD=/g/h but it thinks this is /a/b/c/d
> process 1 stats ".." and compares the device and inode with the previous 
> value.
>   - these are DIFFERENT so the security check FAILS.  (as we would want)
> 
> So here, We stat ".." with CWD="/g/h" to determine that it is
> different to "/a/b/c".
> 
> (this logic is taken from oldfind, I'm much less sure how fts tackels
> the same situation)
> 
> > or just ignore this problems for inode-less filesystems
> 
> I'm very reluctant to do that.   It's the rare problems that the
> quality tools should worry about.  Common problem the user normally
> knows about and can cope with.
> 
> > (find -L is relatively rarely used).
> 
> It doesn't really matter that the feature is rarely used.  Security
> problems are bad, even if they only exist in rarely-used program
> features.

The worst that can happen is that the security check makes a false
alarm.  That in no way makes the security weaker.

> > I agree that compromises are unavoidable.
> 
> I guess the problem doesn't arise for SMBFS, since there are no
> symlinks available.Hence ofr the SMBFS case, one workaround could
> be to just skip the check, but there's no easy way to tel that it is
> safe to do that.
> 
> The problem is arguably worse for SSHFS, because SFTP doesn't support
> "ls -i" anyway, whereas the underlying filesystem could be susceptible
> to the kinds of attacks we're talking about.
> 
> > However the bug I originally reported was not a theoretical one, it
> > was reported for a _real_ FUSE filesystem used on gentoo (which
> > carries find-4.3) and very likely will be reported for
> > smbfs/cifs/fat/etc sooner or later.
> 
> Well the problem is more or less similar to the difficilties of NFSv3
> servers with NFSv2 clients, or 32-bit clients on systems where st_ino
> is 64 bits (for example HPUX 11i has this property, I think).

And no wonder they are similar, since in both cases the root of the
problem is how to map a large ID space into a small one.

Miklos


___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

Re: Fwd: [bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[James Youngman]

[+bug-findutils since we're discussing this example in that context]

On 10/5/06, Miklos Szeredi <[EMAIL PROTECTED]> wrote:

> But what about symlinks?
>
>   a   g
>b   h->a
> c
>  f->g
>
> The moment you traverse the f->g symlink above,
> the entire tree, a/b/c/f, is no longer referenced,
> so the h->a link may take you back to a new inode,
> and the cycle will not be detected.

Right.  So find either holds a file descriptor for each symlink
traversed,


But at the time we traverse the symlink, we don't know it's a symlink.
 Here's the order of events:

process 1 stats "." and remembers the resulting device and inode.
process 1 stats c.   It is told that c is a directory, not a symlink.
process 1 chdirs into c.  Now, CWD=/a/b/c, and that's where it thinks it is.
process 1 stats ".." and compares the device and inode with the previous value.
 - these are the same so the security check succeeds.
process 1 stats d.  It is told that d is a directory, not a symlink.
process 2 renames d to d~, and creates a symlink d pointing at /g/h
process 1 chdirs into d.  Now CWD=/g/h but it thinks this is /a/b/c/d
process 1 stats ".." and compares the device and inode with the previous value.
 - these are DIFFERENT so the security check FAILS.  (as we would want)

So here, We stat ".." with CWD="/g/h" to determine that it is
different to "/a/b/c".

(this logic is taken from oldfind, I'm much less sure how fts tackels
the same situation)


or just ignore this problems for inode-less filesystems


I'm very reluctant to do that.   It's the rare problems that the
quality tools should worry about.  Common problem the user normally
knows about and can cope with.


(find -L is relatively rarely used).


It doesn't really matter that the feature is rarely used.  Security
problems are bad, even if they only exist in rarely-used program
features.


I agree that compromises are unavoidable.


I guess the problem doesn't arise for SMBFS, since there are no
symlinks available.Hence ofr the SMBFS case, one workaround could
be to just skip the check, but there's no easy way to tel that it is
safe to do that.

The problem is arguably worse for SSHFS, because SFTP doesn't support
"ls -i" anyway, whereas the underlying filesystem could be susceptible
to the kinds of attacks we're talking about.


However the bug I originally reported was not a theoretical one, it
was reported for a _real_ FUSE filesystem used on gentoo (which
carries find-4.3) and very likely will be reported for
smbfs/cifs/fat/etc sooner or later.


Well the problem is more or less similar to the difficilties of NFSv3
servers with NFSv2 clients, or 32-bit clients on systems where st_ino
is 64 bits (for example HPUX 11i has this property, I think).

James.


___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

Re: [bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Miklos Szeredi]

> > Follow-up Comment #8, bug #17877 (project findutils):
> >
> > Thanks for the patch.  I've tested it and it doesn't seem to make any
> > difference.
> 
> Are you using the fts version of find 4.3.x?

Yes, find/find from 4.3.1+patch.

With find/oldfind the bug doesn't show up.

Miklos



___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

Re: [bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Miklos Szeredi]

> > Follow-up Comment #1, bug #17877 (project findutils):
> >
> > > There are several Linux filesystems (including smbfs, fat, and
> > > some FUSE based ones) which cannot provide stable inode numbers
> > > for unreferenced files or directories.
> 
> OP, can you provide a test case which demonstrates the problem?
> I will investigate when I have web and SSH access again next week.

The test case is to do "find . > /dev/null" on a large enough tree on
a machine with low enough memory, so that some of the inodes which fts
rememebers are evicted from the icache between the two stats.

Also attached a patch for fuse-2.5.3, which forces the kernel to drop
inodes from the cache immediately if there are no references to it.
This makes it easy to reliably reproduce the problem.

> > That's the problem.  If an OS can't provide stable inode numbers, it is not
> > POSIX compliant.  find shouldn't have to bend over backwards to work around
> > broken file systems.
> 
> I don't agree.   POSIX-compliant systems are allowed to support
> non-POSIX-compliant filesystems, and GNU findutils should be able to
> search them.
> 
> > You should be reporting this bug to the file system
> > people.
> 
> But you're right about this.

For some of the cases this is an intrinsic property of the filesystem,
not an implementation issue.  You cannot map a larger unique ID space
(e.g. path name) to a smaller unique ID space (st_ino) with bounded
memory requirements.

Miklos

Index: kernel/dir.c
===
RCS file: /cvsroot/fuse/fuse/kernel/dir.c,v
retrieving revision 1.101.2.1
diff -u -r1.101.2.1 dir.c
--- kernel/dir.c10 Apr 2006 08:56:41 -  1.101.2.1
+++ kernel/dir.c2 Oct 2006 10:12:29 -
@@ -185,12 +185,18 @@
 }
 #endif
 
+static int fuse_dentry_delete(struct dentry *entry)
+{
+   return 1;
+}
+
 static struct dentry_operations fuse_dentry_operations = {
 #ifdef KERNEL_2_6
.d_revalidate   = fuse_dentry_revalidate,
 #else
.d_revalidate   = fuse_dentry_revalidate_2_4,
 #endif
+   .d_delete = fuse_dentry_delete,
 };
 
 static inline int valid_mode(int m)
Index: kernel/inode.c
===
RCS file: /cvsroot/fuse/fuse/kernel/inode.c,v
retrieving revision 1.89
diff -u -r1.89 inode.c
--- kernel/inode.c  6 Jan 2006 18:29:39 -   1.89
+++ kernel/inode.c  2 Oct 2006 10:12:30 -
@@ -599,6 +599,7 @@
.destroy_inode  = fuse_destroy_inode,
.read_inode = fuse_read_inode,
.clear_inode= fuse_clear_inode,
+   .drop_inode = generic_delete_inode,
.put_super  = fuse_put_super,
.statfs = fuse_statfs,
.show_options   = fuse_show_options,


___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

[bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Miklos Szeredi]

Follow-up Comment #10, bug #17877 (project findutils):

> if some Linux-based file systems can't provide stable inode
> numbers, they should be fixed so that they do. It shouldn't
> be that hard.

It's not hard, it's impossible.  Take for example path based
network filesystems (smbfs, sshfs).   For these the only identifier for files
is the path.

How do you map the path to a unique stable integer identifier?


___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

[bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Paul Eggert]

Follow-up Comment #9, bug #17877 (project findutils):

Miklos Szeredi <[EMAIL PROTECTED]> writes:

> What the patch does is to release the inode as
> soon as there are no more references to it.

Sorry, but that's going to break lots of code.

It's not just gnulib; it's lots of other uses as well.  For
example, some shell scripts use "ls -i" to get a file's
inode number N, and "find -xdev ... -inum N" to find all
hard links to the same file.

Perhaps POSIX isn't clear enough on the subject, but the
intent was always that a file always retains the same inode
number, even if it's renamed or linked.

If some Linux-based file systems can't provide stable inode
numbers, they should be fixed so that they do.  It shouldn't
be that hard.


___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

Re: Fwd: [bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Jim Meyering]

"James Youngman" <[EMAIL PROTECTED]> wrote:
> On 10/4/06, Jim Meyering <[EMAIL PROTECTED]> wrote:
>
>> The resulting down side is that the old version of find is
>> susceptible to a nasty type of attack when traversing a directory
>> that is writable by another user.
>
> I think even oldfind (i.e. version 4.2.x and the alternate binary in
> 4.3.x) has sufficient defences against that.   The advantage that the
> fts-based find has over the older code is that it can traverse
> extremely deep directory hierarchies because it isn't recursive.   The
> fts-based find code is also simpler.

Oh, sorry to falsely impugn :-)
I forgot that the old one used full relative names and not chdir.


___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

Re: Fwd: [bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[James Youngman]

On 10/4/06, Jim Meyering <[EMAIL PROTECTED]> wrote:


The resulting down side is that the old version of find is
susceptible to a nasty type of attack when traversing a directory
that is writable by another user.


I think even oldfind (i.e. version 4.2.x and the alternate binary in
4.3.x) has sufficient defences against that.   The advantage that the
fts-based find has over the older code is that it can traverse
extremely deep directory hierarchies because it isn't recursive.   The
fts-based find code is also simpler.

James.


___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

Re: [bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[James Youngman]

On 10/4/06, Miklos Szeredi <[EMAIL PROTECTED]> wrote:


Follow-up Comment #8, bug #17877 (project findutils):

Thanks for the patch.  I've tested it and it doesn't seem to make any
difference.


Are you using the fts version of find 4.3.x?


___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

[bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Miklos Szeredi]

Follow-up Comment #8, bug #17877 (project findutils):

Thanks for the patch.  I've tested it and it doesn't seem to make any
difference.



___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

[bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[James Youngman]

Follow-up Comment #7, bug #17877 (project findutils):

findutils should work on systems that are themselves POSIX-compliant but
which support non-POSIX filesystems, at least where this is possible.

Please try the patch now attached to this bug on Savannah (from the
suggestion by Jim Meyering) and let me know if it fixes the problem.  
Thanks.
___

Additional Item Attachment:

File name: svannah-bug-17877-fix-attempt-001.patch Size:1 KB
First attempt at a fix


___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

Re: [bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[James Youngman]

On 9/28/06, Eric Blake <[EMAIL PROTECTED]> wrote:


Follow-up Comment #1, bug #17877 (project findutils):

> There are several Linux filesystems (including smbfs, fat, and
> some FUSE based ones) which cannot provide stable inode numbers
> for unreferenced files or directories.


OP, can you provide a test case which demonstrates the problem?
I will investigate when I have web and SSH access again next week.


That's the problem.  If an OS can't provide stable inode numbers, it is not
POSIX compliant.  find shouldn't have to bend over backwards to work around
broken file systems.


I don't agree.   POSIX-compliant systems are allowed to support
non-POSIX-compliant filesystems, and GNU findutils should be able to
search them.


You should be reporting this bug to the file system
people.


But you're right about this.


That said, it would be nice if fts could recognize such broken file systems
and work around their flaws.  But that is an issue for gnulib, not findutils,
since find 4.3.0 is the first version to use gnulib's fts instead of its own
homegrown traversal routines.


It's quite difficult to do this kind of recognition without stat()ing
each mount point in /etc/mtab and that can cause a hang on machines
which are clients of unresponsive NFS servers.

James.


___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

Re: [bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Jim Meyering]

Eric Blake <[EMAIL PROTECTED]> wrote:
...
>>   
>
> For the gnulib crowd, a summary of the above link is that find 4.3.0,
> which uses gnulib's fts, is sometimes reporting "No such file or
> directory" in the middle of traversal on non-POSIX file systems such as
> smbfs, cifs, sshfs, or fat, where the combination of the file system
> representation on disk and/or the remote server and caching issues makes
> it possible for a file's inode to change over time; or even for inodes to
> be reused in a short period of time on different files.
>
> Indeed, if the SAME file has two different inodes over time, that file
> system violates POSIX, but such systems do exist, and wreak havoc in
> fts()'s algorithm to traverse directories, because fts is not maintaining
> an open fd to each directory that it is in, but closes and then reopens
> the directories as it goes.  Yet, keeping fds open is expensive - on a
> pathologically deep hierarchy, there are systems that do not have enough
> fds available to traverse the entire hierachy with an open fd for every
> intermediate path, so you have to close and reopen directories as the
> algorithm progresses if you are to visit everything.
>
> On the opposite side of things, I don't know if fts() is or can be made
> robust to two distinct files being given the same inode due to an unlink
> and new file creation occuring during traversal, although I seem to recall
> an attempt in the past to make rm (which uses fts) more robust along these
> lines.

IMHO, it was more than an "attempt" to fix the bug in GNU rm :-)
Here's the thread:
  http://article.gmane.org/gmane.comp.gnu.core-utils.bugs/6614
Note that rm does not use fts.

But both rm and fts do use the same underlying cycle-detection module.
The sole place where dev/inode reuse matters is in cycle detection.
One difference is that rm uses the CYCLE_CHECK_REFLECT_CHDIR_UP
macro (from cycle-check.h) whenever it does a virtual chdir("..").
However, even that macro is necessary only when fts is using
its default (lazy, O(1)) cycle detection method.

I'll prepare a patch that makes fts use that macro in lazy
cycle-detection mode -- but not today.

For find, the easiest approach to avoiding failure due to a false-positive
cycle detection may be simply to add the FTS_TIGHT_CYCLE_CHECK option
to the flags used when calling fts_open.  That makes fts maintain a
small hash table of dev/inode pairs corresponding to the set of "active"
directories.  Since the directory dev/inode pairs in question cannot be
reused while a directory is active, in that mode, fts is not subject to
the problems I think you're referring to.


___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

Re: [bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Eric Blake]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

[Replying outside of savannah, and adding bug-gnulib, since ultimately,
the behavior in question is a result of gnulib's fts]

According to Geoff Clare on 9/30/2006 5:53 AM:
> Follow-up Comment #6, bug #17877 (project findutils):
> 
> Good point about inode numbers being reused.
> 
> This means that what you can tell from comparing inode numbers (obtained from
> pathnames), according to POSIX, is:
> 
> * If the inode numbers are different, the file is definitely a different
> file.
> 
> * If the inode numbers are the same, the file might be the same file or it
> might be a different file that has been assigned the same inode number.
> 
> The ambiguity in the second case can be avoided by obtaining the first inode
> number from an open fd (which is then kept open) instead of a pathname.  (Or
> an fd could be used to obtain the second inode number, as long as it was open
> at the time the first inode number was obtained.)
> 
> (All of the above assumes that the device numbers are the same.)
> 
> ___
> 
> Reply to this item at:
> 
>   

For the gnulib crowd, a summary of the above link is that find 4.3.0,
which uses gnulib's fts, is sometimes reporting "No such file or
directory" in the middle of traversal on non-POSIX file systems such as
smbfs, cifs, sshfs, or fat, where the combination of the file system
representation on disk and/or the remote server and caching issues makes
it possible for a file's inode to change over time; or even for inodes to
be reused in a short period of time on different files.

Indeed, if the SAME file has two different inodes over time, that file
system violates POSIX, but such systems do exist, and wreak havoc in
fts()'s algorithm to traverse directories, because fts is not maintaining
an open fd to each directory that it is in, but closes and then reopens
the directories as it goes.  Yet, keeping fds open is expensive - on a
pathologically deep hierarchy, there are systems that do not have enough
fds available to traverse the entire hierachy with an open fd for every
intermediate path, so you have to close and reopen directories as the
algorithm progresses if you are to visit everything.

On the opposite side of things, I don't know if fts() is or can be made
robust to two distinct files being given the same inode due to an unlink
and new file creation occuring during traversal, although I seem to recall
an attempt in the past to make rm (which uses fts) more robust along these
lines.

- --
Life is short - so eat dessert first!

Eric Blake [EMAIL PROTECTED]
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.2.1 (Cygwin)
Comment: Public key at home.comcast.net/~ericblake/eblake.gpg
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFFHnK684KuGfSFAYARAo0FAJ98yta6UE0vAgG9SnCeipt8kp2lSgCgv/iR
MNdPL+avqp9zVhW/7yJEua0=
=DnHu
-END PGP SIGNATURE-


___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

[bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Geoff Clare]

Follow-up Comment #6, bug #17877 (project findutils):

Good point about inode numbers being reused.

This means that what you can tell from comparing inode numbers (obtained from
pathnames), according to POSIX, is:

* If the inode numbers are different, the file is definitely a different
file.

* If the inode numbers are the same, the file might be the same file or it
might be a different file that has been assigned the same inode number.

The ambiguity in the second case can be avoided by obtaining the first inode
number from an open fd (which is then kept open) instead of a pathname.  (Or
an fd could be used to obtain the second inode number, as long as it was open
at the time the first inode number was obtained.)

(All of the above assumes that the device numbers are the same.)

___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

[bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Miklos Szeredi]

Follow-up Comment #5, bug #17877 (project findutils):

BTW, POSIX isn't all that clear about inode numbers.  Take this example:

touch x
rm x
mkfifo y

Is x and y the same file?  Obviously not.  Can they have the same inode
number?  Yes, filesystems do reuse inode numbers.

So is st_ino a unique identifier for a file within a filesystem?
Yes, but only at a given instant.  Of course you may know that a file has not
been deleted by holding a reference to it (file descriptor or cwd).  Otherwise
inode numbers are hints at best.


___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

[bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Miklos Szeredi]

Follow-up Comment #4, bug #17877 (project findutils):

Fair enough, I see your point.

However there are filesystems which simply don't have the notion of inode
numbers, or _anything_ that would indentify a file other than it's name.  And
it's not possible to map the name to a unique identifier with bounded memory
requirements.

So while these filesystems may not be POSIX compliant, find should continue
to support them.

Smbfs, cifs and sshfs are examples of this filesystem class.  I'm not sure
about FAT.  Cygwin may _appear_ to provide stable inode numbers for FAT with
some clever heuristics, but I don't think it can do that under all
conditions.

___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

[bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Geoff Clare]

Follow-up Comment #3, bug #17877 (project findutils):

POSIX does require a file's inode number to be stable.  To see this consider
what happens if you stat() a pathname and remember the st_dev and st_ino
values, then some time later you stat() the same pathname and observe that
the st_ino value is different.  POSIX says "The st_ino and st_dev fields
taken together uniquely identify the file within the system", therefore
st_ino being different means that the pathname now refers to a different
file.

___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

[bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Miklos Szeredi]

Follow-up Comment #2, bug #17877 (project findutils):

I've found the following about st_ino in POSIX:

In definitions:

  3.175 File Serial Number

  A per-file system unique identifier for a file.

In sys/stat.h:

  ino_t st_ino File serial number. 

  The st_ino and st_dev fields taken together uniquely identify the file
within the system.

I can't see any requirement about st_ino having to be stable, only that it
has to be unique.

Can you please forward this bug report to gnulib?

Thanks

___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

[bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[Eric Blake]

Follow-up Comment #1, bug #17877 (project findutils):

> There are several Linux filesystems (including smbfs, fat, and
> some FUSE based ones) which cannot provide stable inode numbers
> for unreferenced files or directories.

That's the problem.  If an OS can't provide stable inode numbers, it is not
POSIX compliant.  find shouldn't have to bend over backwards to work around
broken file systems.  You should be reporting this bug to the file system
people.  cygwin manages to provide stable inode numbers for FAT just fine.

That said, it would be nice if fts could recognize such broken file systems
and work around their flaws.  But that is an issue for gnulib, not findutils,
since find 4.3.0 is the first version to use gnulib's fts instead of its own
homegrown traversal routines.


___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils

[bug #17877] Invalid "No such file or directory" error on filesystem without stable inode numbers

[anonymous]

URL:
  

 Summary: Invalid "No such file or directory" error on
filesystem without stable inode numbers
 Project: findutils
Submitted by: None
Submitted on: Thursday 09/28/2006 at 11:24 UTC
Category: find
Severity: 3 - Normal
  Item Group: Wrong result
  Status: None
 Privacy: Public
 Assigned to: None
 Originator Name: Miklos Szeredi
Originator Email: [EMAIL PROTECTED]
 Open/Closed: Open
 Release: 4.3.0
   Fixed Release: None

___

Details:

fts_safe_changedir() in gnulib/lib/fts.c return ENOENT error if inode number
of a directory changed.

There are several Linux filesystems (including smbfs, fat, and some FUSE
based ones) which cannot provide stable inode numbers for unreferenced files
or directories.

In their cases find may in rare circumstances (if the cached in-core inode is
evicted from the cache) erronously fail to recurse into these directories.

This bug seems to have appeared in 4.3.0.




___

Carbon-Copy List:

CC Address  | Comment
+-
Available only the item webpage | Originator Email




___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/



___
Bug-findutils mailing list
Bug-findutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-findutils