Re: Bugs in knfsd -- Problem re-exporting an NFS share
> From [EMAIL PROTECTED] Sat Jan 6 16:36:57 2001 > Date: Sat, 6 Jan 2001 00:52:21 -0500 > From: Chris Siebenmann <[EMAIL PROTECTED]> > To: "Jorge L. deLyra" <[EMAIL PROTECTED]> > Subject: Re: Bugs in knfsd -- Problem re-exporting an NFS share > > You write: > [about re-exporting NFS mounts via NFS:] > | Well, I hope some solution is found, since this is an important feature. > | It would be nicer in knfsd, I think, but if that proves unpractical for > | some reason, your packet filter/forwarder might just be the answer. > > This is essentially impossible to do reliably; one way or another you > have to make tradeoffs. The basic problem is that of NFS filehandles. > NFS filehandles are fixed-size persisent opaque objects that uniquely > identify a given file to the server. Now, imagine that you are > attempting to re-export NFS partitions (from multiple systems) as well > as export some of your own local partitions. How do you come up with the > NFS filehandles that you're going to give clients, in general, and keep > the result persistent? > > You have to somehow add information (what server/filesystem the object > is on/for) to an existing, opaque block of information of the same size, > all of which may be significant, and that you can't make any assumptions > about the format of. There's no place for the information to go, unless > you create a potentially unboundedly large mapping table and store it > locally, or unless your code is only really going to work with known > sources of filehandles so you know how they're put together and thus > where you can stuff some extra information in. > > (unfsd's NFS filehandles are derived from file name and file location, > if I remember correctly, and can erroneously become invalid if the file > is moved around in the wrong way. That's the other way out, but it is > not 100% reliable, and if what someone wants to do with the server hits > this case hard you are truly out of luck.) OK, I think I got your point, we seem to be dealing with a limitation of the NFS protocol. However, Andi Kleen has this idea of a packet filtering and relay daemon. Since the packets of the private network cannot appear on the Internet, I guess there would have to be some kind of tunneling involved. Well, trying to elaborate on that, couldn't you solve the file-handling problem by building a local cache on disk in the front-end? That would reduce the re-exportation problem to the problem of exporting a local ext2 filesystem. I imagine something like this: let's say that latt.int is one home server on the Internet dfma.int is another home server on the Internet pmcs.int is the front-end, which is on both networks pmcs.prv is the name of the front end in the private network node.prv is any compute node in the private network latt.int:/home <---> mounted as /latt/home on pmcs.int dfma.int:/home <---> mounted as /dfma/home on pmcs.int ^ | re-export daemon: files requested | get cached into | V /rexp on pmcs (say, with paths /rexp/latt/home/..., etc) ^ | pmcs.prv:/rexp +---> mounted as /rexp on node.prv The re-export daemon on pmcs would have to keep in /rexp the files which are required via the mount of /rexp by the nodes, copying them from the appropriate Internet mount, and copying back into these mounts new files which show up in /rexp/. Would something like this be feasible? Cheers, Jorge L. deLyra, Associate Professor of Physics The University of Sao Paulo, IFUSP-DFMA For more information: finger [EMAIL PROTECTED] - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: Bugs in knfsd -- Problem re-exporting an NFS share
From [EMAIL PROTECTED] Sat Jan 6 16:36:57 2001 Date: Sat, 6 Jan 2001 00:52:21 -0500 From: Chris Siebenmann [EMAIL PROTECTED] To: "Jorge L. deLyra" [EMAIL PROTECTED] Subject: Re: Bugs in knfsd -- Problem re-exporting an NFS share You write: [about re-exporting NFS mounts via NFS:] | Well, I hope some solution is found, since this is an important feature. | It would be nicer in knfsd, I think, but if that proves unpractical for | some reason, your packet filter/forwarder might just be the answer. This is essentially impossible to do reliably; one way or another you have to make tradeoffs. The basic problem is that of NFS filehandles. NFS filehandles are fixed-size persisent opaque objects that uniquely identify a given file to the server. Now, imagine that you are attempting to re-export NFS partitions (from multiple systems) as well as export some of your own local partitions. How do you come up with the NFS filehandles that you're going to give clients, in general, and keep the result persistent? You have to somehow add information (what server/filesystem the object is on/for) to an existing, opaque block of information of the same size, all of which may be significant, and that you can't make any assumptions about the format of. There's no place for the information to go, unless you create a potentially unboundedly large mapping table and store it locally, or unless your code is only really going to work with known sources of filehandles so you know how they're put together and thus where you can stuff some extra information in. (unfsd's NFS filehandles are derived from file name and file location, if I remember correctly, and can erroneously become invalid if the file is moved around in the wrong way. That's the other way out, but it is not 100% reliable, and if what someone wants to do with the server hits this case hard you are truly out of luck.) OK, I think I got your point, we seem to be dealing with a limitation of the NFS protocol. However, Andi Kleen has this idea of a packet filtering and relay daemon. Since the packets of the private network cannot appear on the Internet, I guess there would have to be some kind of tunneling involved. Well, trying to elaborate on that, couldn't you solve the file-handling problem by building a local cache on disk in the front-end? That would reduce the re-exportation problem to the problem of exporting a local ext2 filesystem. I imagine something like this: let's say that latt.int is one home server on the Internet dfma.int is another home server on the Internet pmcs.int is the front-end, which is on both networks pmcs.prv is the name of the front end in the private network node.prv is any compute node in the private network latt.int:/home --- mounted as /latt/home on pmcs.int dfma.int:/home --- mounted as /dfma/home on pmcs.int ^ | re-export daemon: files requested | get cached into | V /rexp on pmcs (say, with paths /rexp/latt/home/..., etc) ^ | pmcs.prv:/rexp +--- mounted as /rexp on node.prv The re-export daemon on pmcs would have to keep in /rexp the files which are required via the mount of /rexp by the nodes, copying them from the appropriate Internet mount, and copying back into these mounts new files which show up in /rexp/. Would something like this be feasible? Cheers, Jorge L. deLyra, Associate Professor of Physics The University of Sao Paulo, IFUSP-DFMA For more information: finger [EMAIL PROTECTED] - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: Bugs in knfsd -- Problem re-exporting an NFS share
> You probably had a inode namespace collision. unfsd encodes the dev_t in > the upper in ext2 unused bits of the inode to be able to export multiple > file systems in a single export (knfsd dropped that broken feature) Well, not completely. There is now this option "nohide" on /etc/exports which does this. Another useful thing we use a lot here. With knfsd you still have to authorize (export) separately each local mount point (unfsd used to automatically authorize all the underlying tree), but with this option you can mount just once and go down into directories which are on different filesystems on the server. A very useful feature, specially while autofs is not able to do cascaded mounts. We use this for some sysadmin tasks on clients, which export their root to their server. I recognize the need for a robust NFS structure on Linux and understand that some restrictions on features might be needed for that. But all these features are quite nice and useful, so I hope ways are found to implement them in robust ways. Regards, Jorge L. deLyra, Associate Professor of Physics The University of Sao Paulo, IFUSP-DFMA For more information: finger [EMAIL PROTECTED] - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: Bugs in knfsd -- Problem re-exporting an NFS share
On Wed, Jan 03, 2001 at 10:49:54AM -0200, Jorge L. deLyra wrote: > > That's a bit surprising that you have so many problems with unfsd, I know > > lots of installations where it is (still) used successfully in its limits. > > I did recently some mainteance work on it to fix it for reiserfs. > > Well, let me qualify that better. The problems were not really bad before > 2.2.18. There where none in 2.0, by the way, as far as I remember. They > first showed up when we switched to 2.2, and were more frequent at the > beginning. When we got to 2.2.17 we had pretty much forgotten about them. > We have NFS exports on many machines around here and it only happened in > some, so it is difficult to pinpoint the problem. We were never quite sure > whether the problem was on the server or client, sometimes it looked one > way, sometimes another. Once there was a certain directory such that an ls > -lR on it was sure-fire to give an I/O error message and hang the mount. > Moving the contents to another directory and deleting the first one solved > it, don't ask me why. Very often a way to produce the problem was to sit > at the mount point and do a find for something. When a hang was produced, > you had to kill autofs, unmount by hand, maybe killing a few programs in > order not to get "busy" messages. Then just mount it again and it's OK. > Note that neither killing and restarting the server nor rebooting the > client seemed necessary. You just had to unmount and remount. You probably had a inode namespace collision. unfsd encodes the dev_t in the upper in ext2 unused bits of the inode to be able to export multiple file systems in a single export (knfsd dropped that broken feature) That sometimes fails, yielding duplicate inode numbers. When the collision happens for objects with different types (e.g. directory and file) then the client detects it immediately when the other object is still in its inode cache and marks the inode bad, giving these EIOs. Newer unfsd has various tuning options to avoid these collisions, but it does not always work. It depends on the actual device numbers and inode used so could suddenly appear on a system update. > > > (when you ignore locking and NFS over TCP ATM) All you need is to forward > > UDP packets. This can either be done by a normal user space daemon that > > uses portmap and just sends the packets out again, or alternatively by using > > UDP masquerading and appropiate routes on the internal boxes. > > Only forwarding packets is very different from full reexport support in knfsd > > and much simpler. > > Well, I hope some solution is found, since this is an important feature. > It would be nicer in knfsd, I think, but if that proves unpractical for > some reason, your packet filter/forwarder might just be the answer. It is impractical in knfsd for NFS for most other network file systems (especially those that have "artificial" inode numbers like smbfs). knfsd over FAT or amiga ffs is probably a problem too. Your problem case really sounds like you just need a proxy. -Andi - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: Bugs in knfsd -- Problem re-exporting an NFS share
> That's a bit surprising that you have so many problems with unfsd, I know > lots of installations where it is (still) used successfully in its limits. > I did recently some mainteance work on it to fix it for reiserfs. Well, let me qualify that better. The problems were not really bad before 2.2.18. There where none in 2.0, by the way, as far as I remember. They first showed up when we switched to 2.2, and were more frequent at the beginning. When we got to 2.2.17 we had pretty much forgotten about them. We have NFS exports on many machines around here and it only happened in some, so it is difficult to pinpoint the problem. We were never quite sure whether the problem was on the server or client, sometimes it looked one way, sometimes another. Once there was a certain directory such that an ls -lR on it was sure-fire to give an I/O error message and hang the mount. Moving the contents to another directory and deleting the first one solved it, don't ask me why. Very often a way to produce the problem was to sit at the mount point and do a find for something. When a hang was produced, you had to kill autofs, unmount by hand, maybe killing a few programs in order not to get "busy" messages. Then just mount it again and it's OK. Note that neither killing and restarting the server nor rebooting the client seemed necessary. You just had to unmount and remount. > (when you ignore locking and NFS over TCP ATM) All you need is to forward > UDP packets. This can either be done by a normal user space daemon that > uses portmap and just sends the packets out again, or alternatively by using > UDP masquerading and appropiate routes on the internal boxes. > Only forwarding packets is very different from full reexport support in knfsd > and much simpler. Well, I hope some solution is found, since this is an important feature. It would be nicer in knfsd, I think, but if that proves unpractical for some reason, your packet filter/forwarder might just be the answer. Best luck, Jorge L. deLyra, Associate Professor of Physics The University of Sao Paulo, IFUSP-DFMA For more information: finger [EMAIL PROTECTED] - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: Bugs in knfsd -- Problem re-exporting an NFS share
On Wed, Jan 03, 2001 at 09:14:21AM -0200, Jorge L. deLyra wrote: > > Date: Wed, 3 Jan 2001 11:52:36 +0100 > > From: Andi Kleen <[EMAIL PROTECTED]> > > To: "Jorge L. deLyra" <[EMAIL PROTECTED]> > > Cc: Neil Brown <[EMAIL PROTECTED]>, > > Andrzej Krzysztofowicz <[EMAIL PROTECTED]>, > > [EMAIL PROTECTED], kernel list <[EMAIL PROTECTED]> > > Subject: Re: Bugs in knfsd -- Problem re-exporting an NFS share > > > > On Wed, Jan 03, 2001 at 08:37:08AM -0200, Jorge L. deLyra wrote: > > > It would be nice if a way was found to implement this feature on knfsd. > > > > There is: just run unfsd too, bound to an own IP address to not conflict. > > Hum, interesting, I didn't know you could run both servers at the same > time. The problem is that the old server is not being developed any more, > as dully advised on the 2.2.18 kernel help messages. In fact, we have been > having some problems with it for some time, I/O errors and mounts hanging, > mostly. In fact, upgrading to 2.2.18 broke old NFS exports in some cases > around here, until we migrated to knfsd: gave us sure-fire I/O errors and > hangs. So this might be a stopgap measure for some time, but no permanent > solution. In any case, knfsd seems so much more solid, we want to use it! That's a bit surprising that you have so many problems with unfsd, I know lots of installations where it is (still) used successfully in its limits. I did recently some mainteance work on it to fix it for reiserfs. > > > More efficient would probably be a proxy though that just forwards packets. > > I see no reason in core NFS why multiple clients could not share a single > > multiplexed mount. It could need some lockd support though. > > I'm afraid you lost me here... If this would have to be some kind of new > daemon or kernel module that filters out NFS packets and forwards them, > wouldn't this be equivalent to making knfsd do the same thing? Humm, on > second thoughts, maybe not, because this beast would not have to interpret > the mount as a filesystem on the front end. Am I following you correctly? (when you ignore locking and NFS over TCP ATM) All you need is to forward UDP packets. This can either be done by a normal user space daemon that uses portmap and just sends the packets out again, or alternatively by using UDP masquerading and appropiate routes on the internal boxes. Only forwarding packets is very different from full reexport support in knfsd and much simpler. -Andi - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: Bugs in knfsd -- Problem re-exporting an NFS share
> Date: Wed, 3 Jan 2001 11:52:36 +0100 > From: Andi Kleen <[EMAIL PROTECTED]> > To: "Jorge L. deLyra" <[EMAIL PROTECTED]> > Cc: Neil Brown <[EMAIL PROTECTED]>, > Andrzej Krzysztofowicz <[EMAIL PROTECTED]>, > [EMAIL PROTECTED], kernel list <[EMAIL PROTECTED]> > Subject: Re: Bugs in knfsd -- Problem re-exporting an NFS share > > On Wed, Jan 03, 2001 at 08:37:08AM -0200, Jorge L. deLyra wrote: > > It would be nice if a way was found to implement this feature on knfsd. > > There is: just run unfsd too, bound to an own IP address to not conflict. Hum, interesting, I didn't know you could run both servers at the same time. The problem is that the old server is not being developed any more, as dully advised on the 2.2.18 kernel help messages. In fact, we have been having some problems with it for some time, I/O errors and mounts hanging, mostly. In fact, upgrading to 2.2.18 broke old NFS exports in some cases around here, until we migrated to knfsd: gave us sure-fire I/O errors and hangs. So this might be a stopgap measure for some time, but no permanent solution. In any case, knfsd seems so much more solid, we want to use it! > More efficient would probably be a proxy though that just forwards packets. > I see no reason in core NFS why multiple clients could not share a single > multiplexed mount. It could need some lockd support though. I'm afraid you lost me here... If this would have to be some kind of new daemon or kernel module that filters out NFS packets and forwards them, wouldn't this be equivalent to making knfsd do the same thing? Humm, on second thoughts, maybe not, because this beast would not have to interpret the mount as a filesystem on the front end. Am I following you correctly? Jorge L. deLyra, Associate Professor of Physics The University of Sao Paulo, IFUSP-DFMA For more information: finger [EMAIL PROTECTED] - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: Bugs in knfsd -- Problem re-exporting an NFS share
On Wed, Jan 03, 2001 at 08:37:08AM -0200, Jorge L. deLyra wrote: > It would be nice if a way was found to implement this feature on knfsd. There is: just run unfsd too, bound to an own IP address to not conflict. More efficient would probably be a proxy though that just forwards packets. I see no reason in core NFS why multiple clients could not share a single multiplexed mount. It could need some lockd support though. -Andi - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: Bugs in knfsd -- Problem re-exporting an NFS share
> This is not a supported configuration. You cannot export NFS mounted > filesystems with NFS. The protocol does not cope, and it > implementation doesn't even try. > NFS is for export local filesystems only. Just want to point out that this unsupported feature, which exists in the old user-space NFS server, is very useful under some circumstances. Case in point: you have a Beowulf-style parallel cluster on a private network, physically connected to the Internet only through the front-end, but with no direct routing to it, which saves addresses and security concerns (just imagine you have 1000 nodes); you want to import on the front-end various homes of users scattered around your lan, and then have the front-end re-export them to the cluster on the private network. Since the private network does not talk directly to the Internet, there is no danger of loopback mounts locking up. This scheme makes the use of the cluster, usually a resource shared among various groups and institutions, very convenient for all users: their own user configs and environment are available when they use the cluster, logging in through the front-end, and their data goes out transparently to their home systems. Very nice! It would be nice if a way was found to implement this feature on knfsd. Best regards, Jorge L. deLyra, Associate Professor of Physics The University of Sao Paulo, IFUSP-DFMA For more information: finger [EMAIL PROTECTED] - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: Bugs in knfsd -- Problem re-exporting an NFS share
This is not a supported configuration. You cannot export NFS mounted filesystems with NFS. The protocol does not cope, and it implementation doesn't even try. NFS is for export local filesystems only. Just want to point out that this unsupported feature, which exists in the old user-space NFS server, is very useful under some circumstances. Case in point: you have a Beowulf-style parallel cluster on a private network, physically connected to the Internet only through the front-end, but with no direct routing to it, which saves addresses and security concerns (just imagine you have 1000 nodes); you want to import on the front-end various homes of users scattered around your lan, and then have the front-end re-export them to the cluster on the private network. Since the private network does not talk directly to the Internet, there is no danger of loopback mounts locking up. This scheme makes the use of the cluster, usually a resource shared among various groups and institutions, very convenient for all users: their own user configs and environment are available when they use the cluster, logging in through the front-end, and their data goes out transparently to their home systems. Very nice! It would be nice if a way was found to implement this feature on knfsd. Best regards, Jorge L. deLyra, Associate Professor of Physics The University of Sao Paulo, IFUSP-DFMA For more information: finger [EMAIL PROTECTED] - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: Bugs in knfsd -- Problem re-exporting an NFS share
On Wed, Jan 03, 2001 at 08:37:08AM -0200, Jorge L. deLyra wrote: It would be nice if a way was found to implement this feature on knfsd. There is: just run unfsd too, bound to an own IP address to not conflict. More efficient would probably be a proxy though that just forwards packets. I see no reason in core NFS why multiple clients could not share a single multiplexed mount. It could need some lockd support though. -Andi - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: Bugs in knfsd -- Problem re-exporting an NFS share
Date: Wed, 3 Jan 2001 11:52:36 +0100 From: Andi Kleen [EMAIL PROTECTED] To: "Jorge L. deLyra" [EMAIL PROTECTED] Cc: Neil Brown [EMAIL PROTECTED], Andrzej Krzysztofowicz [EMAIL PROTECTED], [EMAIL PROTECTED], kernel list [EMAIL PROTECTED] Subject: Re: Bugs in knfsd -- Problem re-exporting an NFS share On Wed, Jan 03, 2001 at 08:37:08AM -0200, Jorge L. deLyra wrote: It would be nice if a way was found to implement this feature on knfsd. There is: just run unfsd too, bound to an own IP address to not conflict. Hum, interesting, I didn't know you could run both servers at the same time. The problem is that the old server is not being developed any more, as dully advised on the 2.2.18 kernel help messages. In fact, we have been having some problems with it for some time, I/O errors and mounts hanging, mostly. In fact, upgrading to 2.2.18 broke old NFS exports in some cases around here, until we migrated to knfsd: gave us sure-fire I/O errors and hangs. So this might be a stopgap measure for some time, but no permanent solution. In any case, knfsd seems so much more solid, we want to use it! More efficient would probably be a proxy though that just forwards packets. I see no reason in core NFS why multiple clients could not share a single multiplexed mount. It could need some lockd support though. I'm afraid you lost me here... If this would have to be some kind of new daemon or kernel module that filters out NFS packets and forwards them, wouldn't this be equivalent to making knfsd do the same thing? Humm, on second thoughts, maybe not, because this beast would not have to interpret the mount as a filesystem on the front end. Am I following you correctly? Jorge L. deLyra, Associate Professor of Physics The University of Sao Paulo, IFUSP-DFMA For more information: finger [EMAIL PROTECTED] - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: Bugs in knfsd -- Problem re-exporting an NFS share
On Wed, Jan 03, 2001 at 09:14:21AM -0200, Jorge L. deLyra wrote: Date: Wed, 3 Jan 2001 11:52:36 +0100 From: Andi Kleen [EMAIL PROTECTED] To: "Jorge L. deLyra" [EMAIL PROTECTED] Cc: Neil Brown [EMAIL PROTECTED], Andrzej Krzysztofowicz [EMAIL PROTECTED], [EMAIL PROTECTED], kernel list [EMAIL PROTECTED] Subject: Re: Bugs in knfsd -- Problem re-exporting an NFS share On Wed, Jan 03, 2001 at 08:37:08AM -0200, Jorge L. deLyra wrote: It would be nice if a way was found to implement this feature on knfsd. There is: just run unfsd too, bound to an own IP address to not conflict. Hum, interesting, I didn't know you could run both servers at the same time. The problem is that the old server is not being developed any more, as dully advised on the 2.2.18 kernel help messages. In fact, we have been having some problems with it for some time, I/O errors and mounts hanging, mostly. In fact, upgrading to 2.2.18 broke old NFS exports in some cases around here, until we migrated to knfsd: gave us sure-fire I/O errors and hangs. So this might be a stopgap measure for some time, but no permanent solution. In any case, knfsd seems so much more solid, we want to use it! That's a bit surprising that you have so many problems with unfsd, I know lots of installations where it is (still) used successfully in its limits. I did recently some mainteance work on it to fix it for reiserfs. More efficient would probably be a proxy though that just forwards packets. I see no reason in core NFS why multiple clients could not share a single multiplexed mount. It could need some lockd support though. I'm afraid you lost me here... If this would have to be some kind of new daemon or kernel module that filters out NFS packets and forwards them, wouldn't this be equivalent to making knfsd do the same thing? Humm, on second thoughts, maybe not, because this beast would not have to interpret the mount as a filesystem on the front end. Am I following you correctly? (when you ignore locking and NFS over TCP ATM) All you need is to forward UDP packets. This can either be done by a normal user space daemon that uses portmap and just sends the packets out again, or alternatively by using UDP masquerading and appropiate routes on the internal boxes. Only forwarding packets is very different from full reexport support in knfsd and much simpler. -Andi - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: Bugs in knfsd -- Problem re-exporting an NFS share
That's a bit surprising that you have so many problems with unfsd, I know lots of installations where it is (still) used successfully in its limits. I did recently some mainteance work on it to fix it for reiserfs. Well, let me qualify that better. The problems were not really bad before 2.2.18. There where none in 2.0, by the way, as far as I remember. They first showed up when we switched to 2.2, and were more frequent at the beginning. When we got to 2.2.17 we had pretty much forgotten about them. We have NFS exports on many machines around here and it only happened in some, so it is difficult to pinpoint the problem. We were never quite sure whether the problem was on the server or client, sometimes it looked one way, sometimes another. Once there was a certain directory such that an ls -lR on it was sure-fire to give an I/O error message and hang the mount. Moving the contents to another directory and deleting the first one solved it, don't ask me why. Very often a way to produce the problem was to sit at the mount point and do a find for something. When a hang was produced, you had to kill autofs, unmount by hand, maybe killing a few programs in order not to get "busy" messages. Then just mount it again and it's OK. Note that neither killing and restarting the server nor rebooting the client seemed necessary. You just had to unmount and remount. (when you ignore locking and NFS over TCP ATM) All you need is to forward UDP packets. This can either be done by a normal user space daemon that uses portmap and just sends the packets out again, or alternatively by using UDP masquerading and appropiate routes on the internal boxes. Only forwarding packets is very different from full reexport support in knfsd and much simpler. Well, I hope some solution is found, since this is an important feature. It would be nicer in knfsd, I think, but if that proves unpractical for some reason, your packet filter/forwarder might just be the answer. Best luck, Jorge L. deLyra, Associate Professor of Physics The University of Sao Paulo, IFUSP-DFMA For more information: finger [EMAIL PROTECTED] - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: Bugs in knfsd -- Problem re-exporting an NFS share
On Wed, Jan 03, 2001 at 10:49:54AM -0200, Jorge L. deLyra wrote: That's a bit surprising that you have so many problems with unfsd, I know lots of installations where it is (still) used successfully in its limits. I did recently some mainteance work on it to fix it for reiserfs. Well, let me qualify that better. The problems were not really bad before 2.2.18. There where none in 2.0, by the way, as far as I remember. They first showed up when we switched to 2.2, and were more frequent at the beginning. When we got to 2.2.17 we had pretty much forgotten about them. We have NFS exports on many machines around here and it only happened in some, so it is difficult to pinpoint the problem. We were never quite sure whether the problem was on the server or client, sometimes it looked one way, sometimes another. Once there was a certain directory such that an ls -lR on it was sure-fire to give an I/O error message and hang the mount. Moving the contents to another directory and deleting the first one solved it, don't ask me why. Very often a way to produce the problem was to sit at the mount point and do a find for something. When a hang was produced, you had to kill autofs, unmount by hand, maybe killing a few programs in order not to get "busy" messages. Then just mount it again and it's OK. Note that neither killing and restarting the server nor rebooting the client seemed necessary. You just had to unmount and remount. You probably had a inode namespace collision. unfsd encodes the dev_t in the upper in ext2 unused bits of the inode to be able to export multiple file systems in a single export (knfsd dropped that broken feature) That sometimes fails, yielding duplicate inode numbers. When the collision happens for objects with different types (e.g. directory and file) then the client detects it immediately when the other object is still in its inode cache and marks the inode bad, giving these EIOs. Newer unfsd has various tuning options to avoid these collisions, but it does not always work. It depends on the actual device numbers and inode used so could suddenly appear on a system update. (when you ignore locking and NFS over TCP ATM) All you need is to forward UDP packets. This can either be done by a normal user space daemon that uses portmap and just sends the packets out again, or alternatively by using UDP masquerading and appropiate routes on the internal boxes. Only forwarding packets is very different from full reexport support in knfsd and much simpler. Well, I hope some solution is found, since this is an important feature. It would be nicer in knfsd, I think, but if that proves unpractical for some reason, your packet filter/forwarder might just be the answer. It is impractical in knfsd for NFS for most other network file systems (especially those that have "artificial" inode numbers like smbfs). knfsd over FAT or amiga ffs is probably a problem too. Your problem case really sounds like you just need a proxy. -Andi - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: Bugs in knfsd -- Problem re-exporting an NFS share
You probably had a inode namespace collision. unfsd encodes the dev_t in the upper in ext2 unused bits of the inode to be able to export multiple file systems in a single export (knfsd dropped that broken feature) Well, not completely. There is now this option "nohide" on /etc/exports which does this. Another useful thing we use a lot here. With knfsd you still have to authorize (export) separately each local mount point (unfsd used to automatically authorize all the underlying tree), but with this option you can mount just once and go down into directories which are on different filesystems on the server. A very useful feature, specially while autofs is not able to do cascaded mounts. We use this for some sysadmin tasks on clients, which export their root to their server. I recognize the need for a robust NFS structure on Linux and understand that some restrictions on features might be needed for that. But all these features are quite nice and useful, so I hope ways are found to implement them in robust ways. Regards, Jorge L. deLyra, Associate Professor of Physics The University of Sao Paulo, IFUSP-DFMA For more information: finger [EMAIL PROTECTED] - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: Bugs in knfsd -- Problem re-exporting an NFS share
On Saturday December 30, [EMAIL PROTECTED] wrote: > > > On Friday December 29, [EMAIL PROTECTED] wrote: > > > Hi -- could you please CC me if you reply to this mail. > > > > > > A: /exports/A - Redhat 7.0 > > > B1/B2: mount /exports/A on /export/A from A - Redhat 6.2 > > > C: mount /exports/A on /mnt/A from B1 or B2 - Redhat 6.2 > > > > > > I use knfsd/nfs-utils on each machine. > > > > > > bash# ls /mnt/A > > > /mnt/A/A.txt: No such file or directory > > > > This is not a supported configuration. You cannot export NFS mounted > > filesystems with NFS. The protocol does not cope, and it > > implementation doesn't even try. > > NFS is for export local filesystems only. > > As I understand problem is somewhere else. > If this is intentionally unsupported configuration - OK. So why the error > appears ? The directory should be empty then. > > If the configuration is unsupported at the moment and the A.txt file is > located on A, some code that attempts to read re-exported files/directories > should be turned off (eg. #if 0). > > If the A.txt file is local for B1/B2 hosts, it is (IMHO) an obvious bug. > Sucgh a file should be hidden at the act of mounting. For both local and > remote access. > > Neil, could you tell us where the A.txt file is *really* located ? Well its a long story. The interface between knfsd and filesystems is not (currently) a clean one. The current interface to filesystems is dictated by the VFS layer, and it works primarily in terms of names. knfsd wants to work in terms of inode numbers (or something similar) and there is no such interface. knfsd does manage to do its job, but only by abusing part of the VFS interface - the "iget" routine which calls s_op->read_inode. iget is meant to be an private interface, only ever called by the filesystem itself (typically in the lookup operation) but, for ext2 at least, it works for knfsd, so knfsd uses it. Basically, knfsd assumes that is a filesystem has "read_inode" defined, then it can safely use iget. This works for ext2, but not for nfs. The nfs filesystem does define read_inode, but uses it quite differently to ext2. The net result is that when you try to map an inode number to an inode using iget with ext2 it always works, but if you do it with nfs it only works if the inode is already in the cache. So, when you: ls /mnt/A from C, knfsd on B1 (or B2) tries to find the directory 'A' and succeeds beause it is in cache. It does a readdir on this directory and gets a list of files including "A.txt". But when it tries to access A.txt, iget doesn't work for it, and it cannot find it. So A.txt is really on A, not on B1 or B2. But when you look at A.txt from C through B1 or B2 you only get a blurred image. You can see that it is there, but not what it is. Does that help? NeilBrown > > Regards >Andrzej > > BTW. AFAIR, I observed similar behaviour (files are visible but > inaccessible) while mounting a local filesystem at a busy directory > (eg.: mount /dev/fd0 .;ls -l) even in 2.2... > > -- > === > Andrzej M. Krzysztofowicz [EMAIL PROTECTED] > phone (48)(58) 347 14 61 > Faculty of Applied Phys. & Math., Technical University of Gdansk - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: Bugs in knfsd -- Problem re-exporting an NFS share
On Saturday December 30, [EMAIL PROTECTED] wrote: On Friday December 29, [EMAIL PROTECTED] wrote: Hi -- could you please CC me if you reply to this mail. A: /exports/A - Redhat 7.0 B1/B2: mount /exports/A on /export/A from A - Redhat 6.2 C: mount /exports/A on /mnt/A from B1 or B2 - Redhat 6.2 I use knfsd/nfs-utils on each machine. bash# ls /mnt/A /mnt/A/A.txt: No such file or directory This is not a supported configuration. You cannot export NFS mounted filesystems with NFS. The protocol does not cope, and it implementation doesn't even try. NFS is for export local filesystems only. As I understand problem is somewhere else. If this is intentionally unsupported configuration - OK. So why the error appears ? The directory should be empty then. If the configuration is unsupported at the moment and the A.txt file is located on A, some code that attempts to read re-exported files/directories should be turned off (eg. #if 0). If the A.txt file is local for B1/B2 hosts, it is (IMHO) an obvious bug. Sucgh a file should be hidden at the act of mounting. For both local and remote access. Neil, could you tell us where the A.txt file is *really* located ? Well its a long story. The interface between knfsd and filesystems is not (currently) a clean one. The current interface to filesystems is dictated by the VFS layer, and it works primarily in terms of names. knfsd wants to work in terms of inode numbers (or something similar) and there is no such interface. knfsd does manage to do its job, but only by abusing part of the VFS interface - the "iget" routine which calls s_op-read_inode. iget is meant to be an private interface, only ever called by the filesystem itself (typically in the lookup operation) but, for ext2 at least, it works for knfsd, so knfsd uses it. Basically, knfsd assumes that is a filesystem has "read_inode" defined, then it can safely use iget. This works for ext2, but not for nfs. The nfs filesystem does define read_inode, but uses it quite differently to ext2. The net result is that when you try to map an inode number to an inode using iget with ext2 it always works, but if you do it with nfs it only works if the inode is already in the cache. So, when you: ls /mnt/A from C, knfsd on B1 (or B2) tries to find the directory 'A' and succeeds beause it is in cache. It does a readdir on this directory and gets a list of files including "A.txt". But when it tries to access A.txt, iget doesn't work for it, and it cannot find it. So A.txt is really on A, not on B1 or B2. But when you look at A.txt from C through B1 or B2 you only get a blurred image. You can see that it is there, but not what it is. Does that help? NeilBrown Regards Andrzej BTW. AFAIR, I observed similar behaviour (files are visible but inaccessible) while mounting a local filesystem at a busy directory (eg.: mount /dev/fd0 .;ls -l) even in 2.2... -- === Andrzej M. Krzysztofowicz [EMAIL PROTECTED] phone (48)(58) 347 14 61 Faculty of Applied Phys. Math., Technical University of Gdansk - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: Bugs in knfsd -- Problem re-exporting an NFS share
> On Friday December 29, [EMAIL PROTECTED] wrote: > > Hi -- could you please CC me if you reply to this mail. > > > > A: /exports/A - Redhat 7.0 > > B1/B2: mount /exports/A on /export/A from A - Redhat 6.2 > > C: mount /exports/A on /mnt/A from B1 or B2 - Redhat 6.2 > > > > I use knfsd/nfs-utils on each machine. > > > > bash# ls /mnt/A > > /mnt/A/A.txt: No such file or directory > > This is not a supported configuration. You cannot export NFS mounted > filesystems with NFS. The protocol does not cope, and it > implementation doesn't even try. > NFS is for export local filesystems only. As I understand problem is somewhere else. If this is intentionally unsupported configuration - OK. So why the error appears ? The directory should be empty then. If the configuration is unsupported at the moment and the A.txt file is located on A, some code that attempts to read re-exported files/directories should be turned off (eg. #if 0). If the A.txt file is local for B1/B2 hosts, it is (IMHO) an obvious bug. Sucgh a file should be hidden at the act of mounting. For both local and remote access. Neil, could you tell us where the A.txt file is *really* located ? Regards Andrzej BTW. AFAIR, I observed similar behaviour (files are visible but inaccessible) while mounting a local filesystem at a busy directory (eg.: mount /dev/fd0 .;ls -l) even in 2.2... -- === Andrzej M. Krzysztofowicz [EMAIL PROTECTED] phone (48)(58) 347 14 61 Faculty of Applied Phys. & Math., Technical University of Gdansk - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: Bugs in knfsd -- Problem re-exporting an NFS share
On Friday December 29, [EMAIL PROTECTED] wrote: Hi -- could you please CC me if you reply to this mail. A: /exports/A - Redhat 7.0 B1/B2: mount /exports/A on /export/A from A - Redhat 6.2 C: mount /exports/A on /mnt/A from B1 or B2 - Redhat 6.2 I use knfsd/nfs-utils on each machine. bash# ls /mnt/A /mnt/A/A.txt: No such file or directory This is not a supported configuration. You cannot export NFS mounted filesystems with NFS. The protocol does not cope, and it implementation doesn't even try. NFS is for export local filesystems only. As I understand problem is somewhere else. If this is intentionally unsupported configuration - OK. So why the error appears ? The directory should be empty then. If the configuration is unsupported at the moment and the A.txt file is located on A, some code that attempts to read re-exported files/directories should be turned off (eg. #if 0). If the A.txt file is local for B1/B2 hosts, it is (IMHO) an obvious bug. Sucgh a file should be hidden at the act of mounting. For both local and remote access. Neil, could you tell us where the A.txt file is *really* located ? Regards Andrzej BTW. AFAIR, I observed similar behaviour (files are visible but inaccessible) while mounting a local filesystem at a busy directory (eg.: mount /dev/fd0 .;ls -l) even in 2.2... -- === Andrzej M. Krzysztofowicz [EMAIL PROTECTED] phone (48)(58) 347 14 61 Faculty of Applied Phys. Math., Technical University of Gdansk - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: Bugs in knfsd -- Problem re-exporting an NFS share
On Friday December 29, [EMAIL PROTECTED] wrote: > Hi -- could you please CC me if you reply to this mail. > > My problem is that I get an error when setting up the following > configuration: > > A: /exports/A - Redhat 7.0 > B1/B2: mount /exports/A on /export/A from A - Redhat 6.2 > C: mount /exports/A on /mnt/A from B1 or B2 - Redhat 6.2 > > I use knfsd/nfs-utils on each machine. > > bash# ls /mnt/A > /mnt/A/A.txt: No such file or directory > This is not a supported configuration. You cannot export NFS mounted filesystems with NFS. The protocol does not cope, and it implementation doesn't even try. NFS is for export local filesystems only. > > I searched for a while on deja.com, and there seemed to be some indications > that knfsd was bugged and that using the user-mode code would work. > However, no one replied specifically to my message, so I'm still not sure. > > BTW, what I tried to do was to set up a HA configuration of machines B1/B2 > using A as a "shared disk". > This is just to try out the HA software without buying more > hardware. Try "nbd" the network block device. That should be able to give a more realistic imitation of a share disk. NeilBrown > > Thanks in advance for any help! > > Best regards, > Frank Olsen > > PS Happy new year! > > > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to [EMAIL PROTECTED] > Please read the FAQ at http://www.tux.org/lkml/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
