Re: If not readdir() then what?

2007-04-16 Thread Neil Brown
On Monday April 16, [EMAIL PROTECTED] wrote: > > The challenge is making it be stable across inserts/deletes, never > mind reboots. And it's not a "little bit of cacheing"; in order to be > correct we would have to cache *forever*, since at least in theory an > NFS client could hold on to a

Re: If not readdir() then what?

2007-04-16 Thread Theodore Tso
On Mon, Apr 16, 2007 at 04:18:53PM +1000, Neil Brown wrote: > On Thursday April 12, [EMAIL PROTECTED] wrote: > > > > Unfortunately, in the NFS case if there are hash collisions, under the > > wrong circumstances the NFS client could loop forever trying to > > readdir() a directory stream. > > I

Re: If not readdir() then what?

2007-04-16 Thread Theodore Tso
On Mon, Apr 16, 2007 at 03:47:21PM +1000, Neil Brown wrote: > "my guess", "pretty much" really bother me. > > It sounds like "The largest anyone has asked for is 128bits, or let's > double it and hope that is enough until the next protocol revision". > Which was probably reasonable when NFSv2 was

Re: If not readdir() then what?

2007-04-16 Thread Neil Brown
On Thursday April 12, [EMAIL PROTECTED] wrote: > > Unfortunately, in the NFS case if there are hash collisions, under the > wrong circumstances the NFS client could loop forever trying to > readdir() a directory stream. I think that can be easily avoided by the filesystem by simply ensuring that

Re: If not readdir() then what?

2007-04-16 Thread Neil Brown
On Thursday April 12, [EMAIL PROTECTED] wrote: Unfortunately, in the NFS case if there are hash collisions, under the wrong circumstances the NFS client could loop forever trying to readdir() a directory stream. I think that can be easily avoided by the filesystem by simply ensuring that the

Re: If not readdir() then what?

2007-04-16 Thread Theodore Tso
On Mon, Apr 16, 2007 at 03:47:21PM +1000, Neil Brown wrote: my guess, pretty much really bother me. It sounds like The largest anyone has asked for is 128bits, or let's double it and hope that is enough until the next protocol revision. Which was probably reasonable when NFSv2 was being

Re: If not readdir() then what?

2007-04-16 Thread Theodore Tso
On Mon, Apr 16, 2007 at 04:18:53PM +1000, Neil Brown wrote: On Thursday April 12, [EMAIL PROTECTED] wrote: Unfortunately, in the NFS case if there are hash collisions, under the wrong circumstances the NFS client could loop forever trying to readdir() a directory stream. I think that

Re: If not readdir() then what?

2007-04-16 Thread Neil Brown
On Monday April 16, [EMAIL PROTECTED] wrote: The challenge is making it be stable across inserts/deletes, never mind reboots. And it's not a little bit of cacheing; in order to be correct we would have to cache *forever*, since at least in theory an NFS client could hold on to a cookie for

Re: If not readdir() then what?

2007-04-15 Thread Neil Brown
On Sunday April 15, [EMAIL PROTECTED] wrote: > On Thu, Apr 12, 2007 at 10:35:49AM -0700, H. Peter Anvin wrote: > > > > Any fixed size is too small. It should be a dynamic size. > > Idally it should be dynamic, but my guess is that if the cookie were a > fixed 256 bits, it would be sufficient

Re: If not readdir() then what?

2007-04-15 Thread Theodore Tso
On Thu, Apr 12, 2007 at 10:35:49AM -0700, H. Peter Anvin wrote: > J. Bruce Fields wrote: > >On Thu, Apr 12, 2007 at 08:21:16AM -0400, Theodore Tso wrote: > >>Again, compared to a directory fd cache, what you're proposing a huge > >>hit to the filesystem, and at the moment, given that

Re: If not readdir() then what?

2007-04-15 Thread Theodore Tso
On Thu, Apr 12, 2007 at 10:35:49AM -0700, H. Peter Anvin wrote: J. Bruce Fields wrote: On Thu, Apr 12, 2007 at 08:21:16AM -0400, Theodore Tso wrote: Again, compared to a directory fd cache, what you're proposing a huge hit to the filesystem, and at the moment, given that telldir/seekdir is

Re: If not readdir() then what?

2007-04-15 Thread Neil Brown
On Sunday April 15, [EMAIL PROTECTED] wrote: On Thu, Apr 12, 2007 at 10:35:49AM -0700, H. Peter Anvin wrote: Any fixed size is too small. It should be a dynamic size. Idally it should be dynamic, but my guess is that if the cookie were a fixed 256 bits, it would be sufficient for pretty

Re: If not readdir() then what?

2007-04-12 Thread H. Peter Anvin
J. Bruce Fields wrote: On Thu, Apr 12, 2007 at 08:21:16AM -0400, Theodore Tso wrote: Again, compared to a directory fd cache, what you're proposing a huge hit to the filesystem, and at the moment, given that telldir/seekdir is rarely used by everyone else, it's mainly NFS which is the main bad

Re: If not readdir() then what?

2007-04-12 Thread J. Bruce Fields
On Thu, Apr 12, 2007 at 08:21:16AM -0400, Theodore Tso wrote: > Again, compared to a directory fd cache, what you're proposing a huge > hit to the filesystem, and at the moment, given that telldir/seekdir > is rarely used by everyone else, it's mainly NFS which is the main bad > actor here by

Re: If not readdir() then what?

2007-04-12 Thread Theodore Tso
On Thu, Apr 12, 2007 at 03:57:41PM +1000, Neil Brown wrote: > But my perspective is that a solution in nfsd at-best a work-around. > Caching the whole 'struct file' when there is just a small bit that we > might want seems like a heavy hammer. The filesystem is in the best > place to know what

Re: If not readdir() then what?

2007-04-12 Thread Jörn Engel
On Thu, 12 April 2007 15:57:41 +1000, Neil Brown wrote: > > > However, the collision chain gives me quite a bit of headache. One > > would have to store each entry's position on the chain, deal with older > > entries getting deleted, newer entries getting removed, etc. All this > > requires a

Re: If not readdir() then what?

2007-04-12 Thread Neil Brown
On Thursday April 12, [EMAIL PROTECTED] wrote: > On Thu, 12 April 2007 11:46:41 +1000, Neil Brown wrote: > > > > I could argue that nfs came before ext3+dirindex, so ext3 should have > > been designed to work properly with NFS. You could argue that fixing > > it in nfsd fixes it for all

Re: If not readdir() then what?

2007-04-12 Thread Neil Brown
On Thursday April 12, [EMAIL PROTECTED] wrote: On Thu, 12 April 2007 11:46:41 +1000, Neil Brown wrote: I could argue that nfs came before ext3+dirindex, so ext3 should have been designed to work properly with NFS. You could argue that fixing it in nfsd fixes it for all filesystems. But

Re: If not readdir() then what?

2007-04-12 Thread Jörn Engel
On Thu, 12 April 2007 15:57:41 +1000, Neil Brown wrote: However, the collision chain gives me quite a bit of headache. One would have to store each entry's position on the chain, deal with older entries getting deleted, newer entries getting removed, etc. All this requires a lot of

Re: If not readdir() then what?

2007-04-12 Thread Theodore Tso
On Thu, Apr 12, 2007 at 03:57:41PM +1000, Neil Brown wrote: But my perspective is that a solution in nfsd at-best a work-around. Caching the whole 'struct file' when there is just a small bit that we might want seems like a heavy hammer. The filesystem is in the best place to know what needs

Re: If not readdir() then what?

2007-04-12 Thread J. Bruce Fields
On Thu, Apr 12, 2007 at 08:21:16AM -0400, Theodore Tso wrote: Again, compared to a directory fd cache, what you're proposing a huge hit to the filesystem, and at the moment, given that telldir/seekdir is rarely used by everyone else, it's mainly NFS which is the main bad actor here by

Re: If not readdir() then what?

2007-04-12 Thread H. Peter Anvin
J. Bruce Fields wrote: On Thu, Apr 12, 2007 at 08:21:16AM -0400, Theodore Tso wrote: Again, compared to a directory fd cache, what you're proposing a huge hit to the filesystem, and at the moment, given that telldir/seekdir is rarely used by everyone else, it's mainly NFS which is the main bad

Re: If not readdir() then what?

2007-04-11 Thread Jörn Engel
On Thu, 12 April 2007 11:46:41 +1000, Neil Brown wrote: > > I could argue that nfs came before ext3+dirindex, so ext3 should have > been designed to work properly with NFS. You could argue that fixing > it in nfsd fixes it for all filesystems. But I'm not sure either of > those arguments are

Re: If not readdir() then what?

2007-04-11 Thread Neil Brown
On Wednesday April 11, [EMAIL PROTECTED] wrote: > > Actually, no, we can't keep the collision chain count stable across a > create/delete even while the tree is cached. At least, not without > storing a huge amount of state associated with each page. (It would > be a lot more work than simply

Re: If not readdir() then what?

2007-04-11 Thread Neil Brown
On Wednesday April 11, [EMAIL PROTECTED] wrote: > On Thu, 12 Apr 2007, Neil Brown wrote: > > > For the second. > > You say that you " would need at least 96 bits in order to make that > > guarantee; 64 bits of hash, plus a 32-bit count value in the hash > > collision chain". I think 96 is a

Re: If not readdir() then what?

2007-04-11 Thread Jörn Engel
On Wed, 11 April 2007 16:23:21 -0700, H. Peter Anvin wrote: > David Lang wrote: > >On Thu, 12 Apr 2007, Neil Brown wrote: > > > >>For the second. > >> You say that you " would need at least 96 bits in order to make that > >> guarantee; 64 bits of hash, plus a 32-bit count value in the hash > >>

Re: If not readdir() then what?

2007-04-11 Thread H. Peter Anvin
David Lang wrote: On Thu, 12 Apr 2007, Neil Brown wrote: For the second. You say that you " would need at least 96 bits in order to make that guarantee; 64 bits of hash, plus a 32-bit count value in the hash collision chain". I think 96 is a bit greedy. Surely 48 bits of hash and 16 bits

Re: If not readdir() then what?

2007-04-11 Thread Theodore Tso
On Thu, Apr 12, 2007 at 08:32:05AM +1000, Neil Brown wrote: > For the first: > You are storing an internal tree representation of part of the > directory attached to the 'struct file'. > Would it be possible to store this attached to the page via the > ->private pointer? What would avoid

Re: If not readdir() then what?

2007-04-11 Thread David Lang
On Thu, 12 Apr 2007, Neil Brown wrote: For the second. You say that you " would need at least 96 bits in order to make that guarantee; 64 bits of hash, plus a 32-bit count value in the hash collision chain". I think 96 is a bit greedy. Surely 48 bits of hash and 16 bits of

Re: If not readdir() then what?

2007-04-11 Thread Neil Brown
On Wednesday April 11, [EMAIL PROTECTED] wrote: > On Wed, Apr 11, 2007 at 07:15:57AM +1000, Neil Brown wrote: > > > Neil, is this correct? > > > > It's just NFS. nfsd does open/getdents/close on every readdir > > request. > > That's... unfortunate. I like to think of it as "challenging" :-) >

Re: If not readdir() then what?

2007-04-11 Thread Theodore Tso
On Wed, Apr 11, 2007 at 07:15:57AM +1000, Neil Brown wrote: > > Neil, is this correct? > > It's just NFS. nfsd does open/getdents/close on every readdir > request. That's... unfortunate. Ext3 w/htree enable creates an in-memory red-black tree of a portion of the directory; this is typically

Re: If not readdir() then what?

2007-04-11 Thread Jan Engelhardt
On Apr 11 2007 07:15, Neil Brown wrote: >On Sunday April 8, [EMAIL PROTECTED] wrote: >> On Sun, 8 April 2007 11:11:20 -0700, H. Peter Anvin wrote: >> > >> > Well, the question is if you can keep the seekdir/telldir cookie around >> > as a pointer -- preferrably in userspace, of course. You

Re: If not readdir() then what?

2007-04-11 Thread Jan Engelhardt
On Apr 11 2007 07:15, Neil Brown wrote: On Sunday April 8, [EMAIL PROTECTED] wrote: On Sun, 8 April 2007 11:11:20 -0700, H. Peter Anvin wrote: Well, the question is if you can keep the seekdir/telldir cookie around as a pointer -- preferrably in userspace, of course. You would

Re: If not readdir() then what?

2007-04-11 Thread Theodore Tso
On Wed, Apr 11, 2007 at 07:15:57AM +1000, Neil Brown wrote: Neil, is this correct? It's just NFS. nfsd does open/getdents/close on every readdir request. That's... unfortunate. Ext3 w/htree enable creates an in-memory red-black tree of a portion of the directory; this is typically one

Re: If not readdir() then what?

2007-04-11 Thread Neil Brown
On Wednesday April 11, [EMAIL PROTECTED] wrote: On Wed, Apr 11, 2007 at 07:15:57AM +1000, Neil Brown wrote: Neil, is this correct? It's just NFS. nfsd does open/getdents/close on every readdir request. That's... unfortunate. I like to think of it as challenging :-) So the

Re: If not readdir() then what?

2007-04-11 Thread David Lang
On Thu, 12 Apr 2007, Neil Brown wrote: For the second. You say that you would need at least 96 bits in order to make that guarantee; 64 bits of hash, plus a 32-bit count value in the hash collision chain. I think 96 is a bit greedy. Surely 48 bits of hash and 16 bits of

Re: If not readdir() then what?

2007-04-11 Thread Theodore Tso
On Thu, Apr 12, 2007 at 08:32:05AM +1000, Neil Brown wrote: For the first: You are storing an internal tree representation of part of the directory attached to the 'struct file'. Would it be possible to store this attached to the page via the -private pointer? What would avoid the

Re: If not readdir() then what?

2007-04-11 Thread H. Peter Anvin
David Lang wrote: On Thu, 12 Apr 2007, Neil Brown wrote: For the second. You say that you would need at least 96 bits in order to make that guarantee; 64 bits of hash, plus a 32-bit count value in the hash collision chain. I think 96 is a bit greedy. Surely 48 bits of hash and 16 bits

Re: If not readdir() then what?

2007-04-11 Thread Jörn Engel
On Wed, 11 April 2007 16:23:21 -0700, H. Peter Anvin wrote: David Lang wrote: On Thu, 12 Apr 2007, Neil Brown wrote: For the second. You say that you would need at least 96 bits in order to make that guarantee; 64 bits of hash, plus a 32-bit count value in the hash collision chain. I

Re: If not readdir() then what?

2007-04-11 Thread Neil Brown
On Wednesday April 11, [EMAIL PROTECTED] wrote: On Thu, 12 Apr 2007, Neil Brown wrote: For the second. You say that you would need at least 96 bits in order to make that guarantee; 64 bits of hash, plus a 32-bit count value in the hash collision chain. I think 96 is a bit greedy.

Re: If not readdir() then what?

2007-04-11 Thread Neil Brown
On Wednesday April 11, [EMAIL PROTECTED] wrote: Actually, no, we can't keep the collision chain count stable across a create/delete even while the tree is cached. At least, not without storing a huge amount of state associated with each page. (It would be a lot more work than simply having

Re: If not readdir() then what?

2007-04-11 Thread Jörn Engel
On Thu, 12 April 2007 11:46:41 +1000, Neil Brown wrote: I could argue that nfs came before ext3+dirindex, so ext3 should have been designed to work properly with NFS. You could argue that fixing it in nfsd fixes it for all filesystems. But I'm not sure either of those arguments are likely

Re: If not readdir() then what?

2007-04-10 Thread Bernd Eckenfels
In article <[EMAIL PROTECTED]> you wrote: > Otherwise, the client would have to cache _all_ previous READDIR results > since the last opendir()/rewinddir() in order to be able to do its own > loop detection and that will obviously never scale for large directories > or for directories that change

Re: If not readdir() then what?

2007-04-10 Thread Trond Myklebust
On Wed, 2007-04-11 at 08:33 +1000, Neil Brown wrote: > A READDIR (aka getdents2) should take a directory handle, a cookie, > and a filename, and should return filenames and cookies. The > cookies may all be identical or may not. The filename might be used > by the filesystem, or it

Re: If not readdir() then what?

2007-04-10 Thread Neil Brown
On Tuesday April 10, [EMAIL PROTECTED] wrote: > On Wed, 2007-04-11 at 07:37 +1000, Neil Brown wrote: > > On Tuesday April 10, [EMAIL PROTECTED] wrote: > > > On Wed, 2007-04-11 at 07:12 +1000, Neil Brown wrote: > > > > > > > > Is there something that makes that interface problematic? > > > > > >

Re: If not readdir() then what?

2007-04-10 Thread Trond Myklebust
On Wed, 2007-04-11 at 07:37 +1000, Neil Brown wrote: > On Tuesday April 10, [EMAIL PROTECTED] wrote: > > On Wed, 2007-04-11 at 07:12 +1000, Neil Brown wrote: > > > > > > Is there something that makes that interface problematic? > > > > File deletions... > > How are they a problem ? > > There

Re: If not readdir() then what?

2007-04-10 Thread Bob Copeland
On 4/10/07, Neil Brown <[EMAIL PROTECTED]> wrote: 2/ Some data structure using an ordered search key that is based on the filename (e.g. a B-tree with a search key that is a hash of the filename). In the first case, you just use a fixed opaque cookie for location in a directory. In the

Re: If not readdir() then what?

2007-04-10 Thread Neil Brown
On Sunday April 8, [EMAIL PROTECTED] wrote: > On Sun, 8 April 2007 11:11:20 -0700, H. Peter Anvin wrote: > > > > Well, the question is if you can keep the seekdir/telldir cookie around > > as a pointer -- preferrably in userspace, of course. You would > > presumably garbage-collect them on

Re: If not readdir() then what?

2007-04-10 Thread Alan Cox
> A READDIR request contains either a cookie or a filename. Either mean > "This identifies the last name I got from you, give me the next one". > > Is there something that makes that interface problematic? Another client may have removed the name and re-added a new file with that name. At this

Re: If not readdir() then what?

2007-04-10 Thread Neil Brown
On Tuesday April 10, [EMAIL PROTECTED] wrote: > > Seems like it would simply make more sense for the server to be allowed > to determine what the size of the cookie should be. That is one possibility. but if the cookie gets too big, you substantially reduce the number of entries you can fit in

Re: If not readdir() then what?

2007-04-10 Thread Neil Brown
On Tuesday April 10, [EMAIL PROTECTED] wrote: > On Wed, 2007-04-11 at 07:12 +1000, Neil Brown wrote: > > > > Is there something that makes that interface problematic? > > File deletions... How are they a problem ? There are only two ways to organise a directory. 1/ Unsorted linear list of

Re: If not readdir() then what?

2007-04-10 Thread Neil Brown
On Monday April 9, [EMAIL PROTECTED] wrote: > On Mon, Apr 09, 2007 at 08:31:37AM -0400, Trond Myklebust wrote: > > That is a protocol limitation, not a client limitation. > > > > And after quickly checking RFC 3010, I see this limitation hasn't been > lifted in NFSv4. > > Speaking of which,

Re: If not readdir() then what?

2007-04-10 Thread Trond Myklebust
On Wed, 2007-04-11 at 07:12 +1000, Neil Brown wrote: > On Tuesday April 10, [EMAIL PROTECTED] wrote: > > The problem is that it is extremely hard to come up with an alternative > > that doesn't impose new conditions on what filesystems you can support. > > I seem to remember Hans Reiser making a

Re: If not readdir() then what?

2007-04-10 Thread H. Peter Anvin
Neil Brown wrote: On Tuesday April 10, [EMAIL PROTECTED] wrote: The problem is that it is extremely hard to come up with an alternative that doesn't impose new conditions on what filesystems you can support. I seem to remember Hans Reiser making a credible suggestion years ago when NFSv4 was

Re: If not readdir() then what?

2007-04-10 Thread Neil Brown
On Tuesday April 10, [EMAIL PROTECTED] wrote: > The problem is that it is extremely hard to come up with an alternative > that doesn't impose new conditions on what filesystems you can support. I seem to remember Hans Reiser making a credible suggestion years ago when NFSv4 was still in draft.

Re: If not readdir() then what?

2007-04-10 Thread Ulrich Drepper
On 4/10/07, H. Peter Anvin <[EMAIL PROTECTED]> wrote: It rather makes any user space accesses irrelevant. The main question seems to be if we can realistically increase the cookie size even to 64 bits. On 32-bit platforms, *not* using _FILE_OFFSET_BITS=64 is already today a stupid thing to

Re: If not readdir() then what?

2007-04-10 Thread Valdis . Kletnieks
ld also give a 'fixed view' when adding/deleting files > during readdir. What should happen if the directory contents go stale? If *two* systems cache the directory and then proceed to add/delete files, what happens? pgpKUxNQKTp2A.pgp Description: PGP signature

Re: If not readdir() then what?

2007-04-10 Thread H. Peter Anvin
Jan Engelhardt wrote: I had a thought, but I think it's not quite ripe.. NFS server sends the whole directory contents on NFS client opendir, so that the whole readdir/telldir/seekdir magic can happen on the client only... which would perhaps also enable a cheap telldir/seekdir, and would also

Re: If not readdir() then what?

2007-04-10 Thread Jan Engelhardt
On Apr 10 2007 10:37, Trond Myklebust wrote: > >NFS, OTOH, simply could not work without that requirement, since there >exists no file pointer to tell you where you are in a stream beyond >whatever the server manages to encode inside the opaque cookie+verifier. > >> But the fact of the matter is

Re: If not readdir() then what?

2007-04-10 Thread H. Peter Anvin
Ulrich Drepper wrote: On 4/10/07, Theodore Tso <[EMAIL PROTECTED]> wrote: That might work. But if in the long term we want to separate out what we can send back via telldir/seekdir, and some future new Posix interface, [...] With all these discussions about fixes for telldir, do we want to

Re: If not readdir() then what?

2007-04-10 Thread Trond Myklebust
On Tue, 2007-04-10 at 09:56 -0400, Theodore Tso wrote: > That might work. But if in the long term we want to separate out what > we can send back via telldir/seekdir, and some future new Posix > interface, I wonder if we might be better off defining a formal > interface which can be used by

Re: If not readdir() then what?

2007-04-10 Thread Ulrich Drepper
On 4/10/07, Theodore Tso <[EMAIL PROTECTED]> wrote: That might work. But if in the long term we want to separate out what we can send back via telldir/seekdir, and some future new Posix interface, [...] With all these discussions about fixes for telldir, do we want to persue an alternative

Re: If not readdir() then what?

2007-04-10 Thread Theodore Tso
On Mon, Apr 09, 2007 at 10:03:15AM -0400, Trond Myklebust wrote: > We could perhaps teach nfsd to open the file without the O_LARGEFILE > attribute in the case of NFSv2? That might work. But if in the long term we want to separate out what we can send back via telldir/seekdir, and some future

Re: If not readdir() then what?

2007-04-10 Thread Theodore Tso
On Mon, Apr 09, 2007 at 10:03:15AM -0400, Trond Myklebust wrote: We could perhaps teach nfsd to open the file without the O_LARGEFILE attribute in the case of NFSv2? That might work. But if in the long term we want to separate out what we can send back via telldir/seekdir, and some future new

Re: If not readdir() then what?

2007-04-10 Thread Ulrich Drepper
On 4/10/07, Theodore Tso [EMAIL PROTECTED] wrote: That might work. But if in the long term we want to separate out what we can send back via telldir/seekdir, and some future new Posix interface, [...] With all these discussions about fixes for telldir, do we want to persue an alternative

Re: If not readdir() then what?

2007-04-10 Thread Trond Myklebust
On Tue, 2007-04-10 at 09:56 -0400, Theodore Tso wrote: That might work. But if in the long term we want to separate out what we can send back via telldir/seekdir, and some future new Posix interface, I wonder if we might be better off defining a formal interface which can be used by NFSv2

Re: If not readdir() then what?

2007-04-10 Thread H. Peter Anvin
Ulrich Drepper wrote: On 4/10/07, Theodore Tso [EMAIL PROTECTED] wrote: That might work. But if in the long term we want to separate out what we can send back via telldir/seekdir, and some future new Posix interface, [...] With all these discussions about fixes for telldir, do we want to

Re: If not readdir() then what?

2007-04-10 Thread Jan Engelhardt
On Apr 10 2007 10:37, Trond Myklebust wrote: NFS, OTOH, simply could not work without that requirement, since there exists no file pointer to tell you where you are in a stream beyond whatever the server manages to encode inside the opaque cookie+verifier. But the fact of the matter is that if

Re: If not readdir() then what?

2007-04-10 Thread H. Peter Anvin
Jan Engelhardt wrote: I had a thought, but I think it's not quite ripe.. NFS server sends the whole directory contents on NFS client opendir, so that the whole readdir/telldir/seekdir magic can happen on the client only... which would perhaps also enable a cheap telldir/seekdir, and would also

Re: If not readdir() then what?

2007-04-10 Thread Valdis . Kletnieks
view' when adding/deleting files during readdir. What should happen if the directory contents go stale? If *two* systems cache the directory and then proceed to add/delete files, what happens? pgpKUxNQKTp2A.pgp Description: PGP signature

Re: If not readdir() then what?

2007-04-10 Thread Ulrich Drepper
On 4/10/07, H. Peter Anvin [EMAIL PROTECTED] wrote: It rather makes any user space accesses irrelevant. The main question seems to be if we can realistically increase the cookie size even to 64 bits. On 32-bit platforms, *not* using _FILE_OFFSET_BITS=64 is already today a stupid thing to do.

Re: If not readdir() then what?

2007-04-10 Thread Neil Brown
On Tuesday April 10, [EMAIL PROTECTED] wrote: The problem is that it is extremely hard to come up with an alternative that doesn't impose new conditions on what filesystems you can support. I seem to remember Hans Reiser making a credible suggestion years ago when NFSv4 was still in draft. It

Re: If not readdir() then what?

2007-04-10 Thread H. Peter Anvin
Neil Brown wrote: On Tuesday April 10, [EMAIL PROTECTED] wrote: The problem is that it is extremely hard to come up with an alternative that doesn't impose new conditions on what filesystems you can support. I seem to remember Hans Reiser making a credible suggestion years ago when NFSv4 was

Re: If not readdir() then what?

2007-04-10 Thread Trond Myklebust
On Wed, 2007-04-11 at 07:12 +1000, Neil Brown wrote: On Tuesday April 10, [EMAIL PROTECTED] wrote: The problem is that it is extremely hard to come up with an alternative that doesn't impose new conditions on what filesystems you can support. I seem to remember Hans Reiser making a

Re: If not readdir() then what?

2007-04-10 Thread Neil Brown
On Monday April 9, [EMAIL PROTECTED] wrote: On Mon, Apr 09, 2007 at 08:31:37AM -0400, Trond Myklebust wrote: That is a protocol limitation, not a client limitation. Groan And after quickly checking RFC 3010, I see this limitation hasn't been lifted in NFSv4. Speaking of which, right

Re: If not readdir() then what?

2007-04-10 Thread Neil Brown
On Tuesday April 10, [EMAIL PROTECTED] wrote: On Wed, 2007-04-11 at 07:12 +1000, Neil Brown wrote: Is there something that makes that interface problematic? File deletions... How are they a problem ? There are only two ways to organise a directory. 1/ Unsorted linear list of entries in

Re: If not readdir() then what?

2007-04-10 Thread Neil Brown
On Tuesday April 10, [EMAIL PROTECTED] wrote: Seems like it would simply make more sense for the server to be allowed to determine what the size of the cookie should be. That is one possibility. but if the cookie gets too big, you substantially reduce the number of entries you can fit in a

Re: If not readdir() then what?

2007-04-10 Thread Alan Cox
A READDIR request contains either a cookie or a filename. Either mean This identifies the last name I got from you, give me the next one. Is there something that makes that interface problematic? Another client may have removed the name and re-added a new file with that name. At this point

Re: If not readdir() then what?

2007-04-10 Thread Neil Brown
On Sunday April 8, [EMAIL PROTECTED] wrote: On Sun, 8 April 2007 11:11:20 -0700, H. Peter Anvin wrote: Well, the question is if you can keep the seekdir/telldir cookie around as a pointer -- preferrably in userspace, of course. You would presumably garbage-collect them on closedir()

Re: If not readdir() then what?

2007-04-10 Thread Bob Copeland
On 4/10/07, Neil Brown [EMAIL PROTECTED] wrote: 2/ Some data structure using an ordered search key that is based on the filename (e.g. a B-tree with a search key that is a hash of the filename). In the first case, you just use a fixed opaque cookie for location in a directory. In the second

Re: If not readdir() then what?

2007-04-10 Thread Trond Myklebust
On Wed, 2007-04-11 at 07:37 +1000, Neil Brown wrote: On Tuesday April 10, [EMAIL PROTECTED] wrote: On Wed, 2007-04-11 at 07:12 +1000, Neil Brown wrote: Is there something that makes that interface problematic? File deletions... How are they a problem ? There are only two ways

Re: If not readdir() then what?

2007-04-10 Thread Neil Brown
On Tuesday April 10, [EMAIL PROTECTED] wrote: On Wed, 2007-04-11 at 07:37 +1000, Neil Brown wrote: On Tuesday April 10, [EMAIL PROTECTED] wrote: On Wed, 2007-04-11 at 07:12 +1000, Neil Brown wrote: Is there something that makes that interface problematic? File deletions...

Re: If not readdir() then what?

2007-04-10 Thread Trond Myklebust
On Wed, 2007-04-11 at 08:33 +1000, Neil Brown wrote: A READDIR (aka getdents2) should take a directory handle, a cookie, and a filename, and should return filenames and cookies. The cookies may all be identical or may not. The filename might be used by the filesystem, or it might

Re: If not readdir() then what?

2007-04-10 Thread Bernd Eckenfels
In article [EMAIL PROTECTED] you wrote: Otherwise, the client would have to cache _all_ previous READDIR results since the last opendir()/rewinddir() in order to be able to do its own loop detection and that will obviously never scale for large directories or for directories that change

Re: If not readdir() then what?

2007-04-09 Thread Trond Myklebust
On Mon, 2007-04-09 at 18:34 +0200, Jan Engelhardt wrote: > On Apr 9 2007 10:03, Trond Myklebust wrote: > > >In practice, though, this sort of behaviour has to be managed carefully > >by the server. Forcing a client to re-read the entire contents of the > >directory doesn't really scale too

Re: If not readdir() then what?

2007-04-09 Thread Jan Engelhardt
On Apr 9 2007 10:03, Trond Myklebust wrote: >In practice, though, this sort of behaviour has to be managed carefully >by the server. Forcing a client to re-read the entire contents of the >directory doesn't really scale too well... What does the spec (readdir, and NFS READDIR) say about

Re: If not readdir() then what?

2007-04-09 Thread Trond Myklebust
On Mon, 2007-04-09 at 09:19 -0400, Theodore Tso wrote: > On Mon, Apr 09, 2007 at 08:31:37AM -0400, Trond Myklebust wrote: > > On Mon, 2007-04-09 at 13:09 +0200, Jörn Engel wrote: > > > That surely doesn't make life any easier for filesystem developers, I > > > agree. From that point of view, all

Re: If not readdir() then what?

2007-04-09 Thread Theodore Tso
On Mon, Apr 09, 2007 at 08:31:37AM -0400, Trond Myklebust wrote: > On Mon, 2007-04-09 at 13:09 +0200, Jörn Engel wrote: > > That surely doesn't make life any easier for filesystem developers, I > > agree. From that point of view, all telldir cookies should end their > > life at closedir time.

Re: If not readdir() then what?

2007-04-09 Thread Andreas Schwab
Jörn Engel <[EMAIL PROTECTED]> writes: > On Sun, 8 April 2007 21:44:26 -0400, Theodore Tso wrote: >> >> Well, Joern thought that rm -rf might relying on the telldir cookie >> being valid in precisely that circumstance. If that is true, I'd >> argue that this is a BUG in GNU coreutils that

Re: If not readdir() then what?

2007-04-09 Thread Trond Myklebust
On Mon, 2007-04-09 at 13:09 +0200, Jörn Engel wrote: > That surely doesn't make life any easier for filesystem developers, I > agree. From that point of view, all telldir cookies should end their > life at closedir time. For "rm -r" it would be sufficient if the nfs > client simply didn't

Re: If not readdir() then what?

2007-04-09 Thread Trond Myklebust
On Mon, 2007-04-09 at 13:09 +0200, Jörn Engel wrote: > That surely doesn't make life any easier for filesystem developers, I > agree. From that point of view, all telldir cookies should end their > life at closedir time. For "rm -r" it would be sufficient if the nfs > client simply didn't

Re: If not readdir() then what?

2007-04-09 Thread Jörn Engel
On Sun, 8 April 2007 21:44:26 -0400, Theodore Tso wrote: > > Well, Joern thought that rm -rf might relying on the telldir cookie > being valid in precisely that circumstance. If that is true, I'd > argue that this is a BUG in GNU coreutils that should be fixed... I heard it and accepted that

Re: If not readdir() then what?

2007-04-09 Thread Jörn Engel
On Sun, 8 April 2007 21:44:26 -0400, Theodore Tso wrote: Well, Joern thought that rm -rf might relying on the telldir cookie being valid in precisely that circumstance. If that is true, I'd argue that this is a BUG in GNU coreutils that should be fixed... I heard it and accepted that claim

Re: If not readdir() then what?

2007-04-09 Thread Trond Myklebust
On Mon, 2007-04-09 at 13:09 +0200, Jörn Engel wrote: That surely doesn't make life any easier for filesystem developers, I agree. From that point of view, all telldir cookies should end their life at closedir time. For rm -r it would be sufficient if the nfs client simply didn't seekdir at

Re: If not readdir() then what?

2007-04-09 Thread Trond Myklebust
On Mon, 2007-04-09 at 13:09 +0200, Jörn Engel wrote: That surely doesn't make life any easier for filesystem developers, I agree. From that point of view, all telldir cookies should end their life at closedir time. For rm -r it would be sufficient if the nfs client simply didn't seekdir at

Re: If not readdir() then what?

2007-04-09 Thread Andreas Schwab
Jörn Engel [EMAIL PROTECTED] writes: On Sun, 8 April 2007 21:44:26 -0400, Theodore Tso wrote: Well, Joern thought that rm -rf might relying on the telldir cookie being valid in precisely that circumstance. If that is true, I'd argue that this is a BUG in GNU coreutils that should be

Re: If not readdir() then what?

2007-04-09 Thread Theodore Tso
On Mon, Apr 09, 2007 at 08:31:37AM -0400, Trond Myklebust wrote: On Mon, 2007-04-09 at 13:09 +0200, Jörn Engel wrote: That surely doesn't make life any easier for filesystem developers, I agree. From that point of view, all telldir cookies should end their life at closedir time. For rm -r

Re: If not readdir() then what?

2007-04-09 Thread Trond Myklebust
On Mon, 2007-04-09 at 09:19 -0400, Theodore Tso wrote: On Mon, Apr 09, 2007 at 08:31:37AM -0400, Trond Myklebust wrote: On Mon, 2007-04-09 at 13:09 +0200, Jörn Engel wrote: That surely doesn't make life any easier for filesystem developers, I agree. From that point of view, all telldir

Re: If not readdir() then what?

2007-04-09 Thread Jan Engelhardt
On Apr 9 2007 10:03, Trond Myklebust wrote: In practice, though, this sort of behaviour has to be managed carefully by the server. Forcing a client to re-read the entire contents of the directory doesn't really scale too well... What does the spec (readdir, and NFS READDIR) say about duplicate

Re: If not readdir() then what?

2007-04-09 Thread Trond Myklebust
On Mon, 2007-04-09 at 18:34 +0200, Jan Engelhardt wrote: On Apr 9 2007 10:03, Trond Myklebust wrote: In practice, though, this sort of behaviour has to be managed carefully by the server. Forcing a client to re-read the entire contents of the directory doesn't really scale too well...

  1   2   >