Re: httpfs
:I don't really think that portalfs is the right thing to use to build :an httpfs with, but I would like to see how you managed to get your example :to work. Are you using stdout to create an anonymous file handle? What happens :if two processes concurrently read from /p/http/*? : :Duncan : :-- :_ :Duncan Barclay | God smiles upon the little children, You could certainly write a program to sit in the middle and cache the request to handle that case. The problem with portalfs is that you can't 'cd' into it or do directory operations on it, and filesystem operations such as lseek, fstat, and so forth cannot be intercepted. It would be the ultimate coolness if you could. We need a better solution then faking an NFS mount to be able to run *real* filesystems in user space. But, that aside, portalfs works just dandy for getting simple file handles from a path. -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: httpfs
On Sun, 18 Mar 2001, Matt Dillon wrote: You could certainly write a program to sit in the middle and cache the request to handle that case. The problem with portalfs is that you can't 'cd' into it or do directory operations on it, and filesystem operations such as lseek, fstat, and so forth cannot be intercepted. It would be the ultimate coolness if you could. We need a better solution then faking an NFS mount to be able to run *real* filesystems in user space. But, that aside, portalfs works just dandy for getting simple file handles from a path. Take a look at the XFS module included with Arla, and the Coda kernel module. They're both targetted at the idea that a userspace daemon will deal with open/close/directory requests, providing container vnodes for the actual files on demand, allowing the kernel to efficiently provide them to consumers. It's easy to imagine an HTTP backend daemon for them. The Arla kernel module is probably a bit more mature and better maintained; on the other hand, the Coda module is in our sys/ tree already. The OpenBSD folk have actually imported Arla into their distribution, which is actually not a bad idea now that OpenAFS is around... (Of course, we still need someone to port OpenAFS so that we have a free server -- with IFS on the server side, we should be able to exhibit a substantially simpler implementation with the same perform benefits as the AFS iopen() stuff :-) Robert N M Watson FreeBSD Core Team, TrustedBSD Project [EMAIL PROTECTED] NAI Labs, Safeport Network Services To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: httpfs
Peter Pentchev [EMAIL PROTECTED] writes: There was at the time - socketpair(2) had totally slipped my mind ;) Umm, you want pipe(2), not socketpair(2). DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: httpfs
On Sat, Mar 17, 2001 at 04:53:34PM +0100, Dag-Erling Smorgrav wrote: Peter Pentchev [EMAIL PROTECTED] writes: There was at the time - socketpair(2) had totally slipped my mind ;) Umm, you want pipe(2), not socketpair(2). Actually, I want socketpair(2). pipe(2) was what I used before, and that's the reason I had a read-only file descriptor - the portalfs architecture allows for only one fd to be returned, and pipe(2) provides a one-way pipe. I dup2'd stdout and stderr of the executed program to the child fd, and the parent could read its output, yet not write to its stdin. With socketpair(2), I can dup stdin, too, and have mount_portal return a two-way pipe/fd/socket to whoever requested it. At least, that's the common/standard/easiest way to create a two-way pipe on the same fd, described in APUE :) G'luck, Peter -- This would easier understand fewer had omitted. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: httpfs
Peter Pentchev [EMAIL PROTECTED] writes: On Sat, Mar 17, 2001 at 04:53:34PM +0100, Dag-Erling Smorgrav wrote: Peter Pentchev [EMAIL PROTECTED] writes: There was at the time - socketpair(2) had totally slipped my mind ;) Umm, you want pipe(2), not socketpair(2). Actually, I want socketpair(2). pipe(2) was what I used before, and that's the reason I had a read-only file descriptor - the portalfs architecture allows for only one fd to be returned, and pipe(2) provides a one-way pipe. Not in FreeBSD. DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: httpfs
On 17-Mar-01 Peter Pentchev wrote: Actually, I want socketpair(2). pipe(2) was what I used before, and that's the reason I had a read-only file descriptor - the portalfs architecture allows for only one fd to be returned, and pipe(2) provides a one-way pipe. I dup2'd stdout and stderr of the executed program to the child fd, and the parent could read its output, yet not write to its stdin. pipe's are bidirectional in FreeBSD.. --- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: httpfs
Peter Pentchev [EMAIL PROTECTED] wrote: What I did was implement an 'exec' portal method, which executes a program with given arguments, obtained from the path components and portal.conf rules, and returns a - basically read-only - descriptor connected to its stdout and stderr. Kind of simple, pipe(), fork(), dup2(), exec().. Nice. Is there any reason not to add some bidirectional support by connecting the descriptor to stdin as well? Tony. -- f.a.n.finch [EMAIL PROTECTED] [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: httpfs
On Fri, Mar 16, 2001 at 05:46:49AM +, Tony Finch wrote: Peter Pentchev [EMAIL PROTECTED] wrote: What I did was implement an 'exec' portal method, which executes a program with given arguments, obtained from the path components and portal.conf rules, and returns a - basically read-only - descriptor connected to its stdout and stderr. Kind of simple, pipe(), fork(), dup2(), exec().. Nice. Is there any reason not to add some bidirectional support by connecting the descriptor to stdin as well? There was at the time - socketpair(2) had totally slipped my mind ;) G'luck, Peter -- This sentence contradicts itself - or rather - well, no, actually it doesn't! To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: httpfs
I'd say it would be an excellent example of how to use portals if nothing else, given that almost nobody understands them. :) - Jordan To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: httpfs
On Sat, Mar 10, 2001 at 03:15:15AM -0800, Kris Kennaway wrote: A few of us were talking on IRC tonight about how cool it would be to have an httpfs filesystem -- then it occurred to me we almost have this already, in the form of the (under-utilised) portalfs. Portalfs works by handing off everything to a userland daemon which handles the actual transaction request, so you could easily imagine extending it to provide an http method similar to the tcp method it currently has for initiating tcp connections. One could probably make this more generic and finish implementing the undocumented 'exec' method (which currently exists as a stub): this would run an administrator-specified command (i.e. fixed in /etc/portal.conf) and pipe the output back to the user. A fully navigable httpfs (e.g. one you can ls and cd around in) is more work and probably requires a full stacking layer, but this would still be pretty cool. Is anyone feeling inspired to implement this? :-) OK, as I've mentioned in a private mail to Kris several hours after he sent out this message, I've done some initial hacking on mount_portal which lets me put: http/ exec http/ /usr/bin/fetch fetch -q -o - http://$1- into /etc/portal.conf, and then do cat /p/http/www.FreeBSD.org/handbook/ (the $1- part refers to path components below /p/http/; $1- means 'path components from 1 to last, separated by /') The code still needs a lot of cleanup before I would dare submit it for review and comments; my question is, should I bother^W^W^Wdoes this look like a reasonable extension to mount_portal, or are there other ways/places that such functionality should be implemented, if at all? G'luck, Peter -- If the meanings of 'true' and 'false' were switched, then this sentence wouldn't be false. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: httpfs
On 2001-03-10 13:36 -0500, Robert Watson [EMAIL PROTECTED] wrote: On Sat, 10 Mar 2001, Kris Kennaway wrote: A few of us were talking on IRC tonight about how cool it would be to have an httpfs filesystem -- then it occurred to me we almost have this already, in the form of the (under-utilised) portalfs. Portalfs works by handing off everything to a userland daemon which handles the actual transaction request, so you could easily imagine extending it to provide an http method similar to the tcp method it currently has for initiating tcp connections. I need not remind you that file systems front-ending onto random protocols are a bad idea for a huge number of reasons :-). Could you give me the three biggest reasons, IYO? I don't seem to know any of them. Thanks! Greg -- Gregory S. Sutter Five million battered women in mailto:[EMAIL PROTECTED] this country, and I've always http://www.zer0.org/~gsutter/ eaten mine plain... hkp://wwwkeys.pgp.net/0x845DFEDD To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
httpfs
A few of us were talking on IRC tonight about how cool it would be to have an httpfs filesystem -- then it occurred to me we almost have this already, in the form of the (under-utilised) portalfs. Portalfs works by handing off everything to a userland daemon which handles the actual transaction request, so you could easily imagine extending it to provide an http method similar to the tcp method it currently has for initiating tcp connections. One could probably make this more generic and finish implementing the undocumented 'exec' method (which currently exists as a stub): this would run an administrator-specified command (i.e. fixed in /etc/portal.conf) and pipe the output back to the user. A fully navigable httpfs (e.g. one you can ls and cd around in) is more work and probably requires a full stacking layer, but this would still be pretty cool. Is anyone feeling inspired to implement this? :-) Kris PGP signature
Re: httpfs
I don't have a link handy, but if you search for it, I'm sure it's out there. Linux had something like this called PerlFS which was much more generic. The general gist of it was you could use it to make fs's of things like http and ftp. It was extendable (I think) so you could make your own fs handlers out of perl scripts. I could be making alot of this up as this was a long time ago, and I'm too lazy/tired to pull up a web browser and look. In any event, it might be a good spot to look for ideas. If no one else has, I'll see what I can dig up in the morning. Night. -gordon On Sat, 10 Mar 2001, Kris Kennaway wrote: A few of us were talking on IRC tonight about how cool it would be to have an httpfs filesystem -- then it occurred to me we almost have this already, in the form of the (under-utilised) portalfs. Portalfs works by handing off everything to a userland daemon which handles the actual transaction request, so you could easily imagine extending it to provide an http method similar to the tcp method it currently has for initiating tcp connections. One could probably make this more generic and finish implementing the undocumented 'exec' method (which currently exists as a stub): this would run an administrator-specified command (i.e. fixed in /etc/portal.conf) and pipe the output back to the user. A fully navigable httpfs (e.g. one you can ls and cd around in) is more work and probably requires a full stacking layer, but this would still be pretty cool. Is anyone feeling inspired to implement this? :-) Kris To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: httpfs
On Mar 10, Kris Kennaway wrote: A few of us were talking on IRC tonight about how cool it would be to have an httpfs filesystem -- then it occurred to me we almost have this already, in the form of the (under-utilised) portalfs. Portalfs works by handing off everything to a userland daemon which handles the actual transaction request, so you could easily imagine extending it to provide an http method similar to the tcp method it currently has for initiating tcp connections. [...] Reminds me on something I read. http://zowie.metnet.navy.mil/~oleg/ftp/USENIX99/ --Mat -- Mathew Kanner [EMAIL PROTECTED] Sys Admin at large Obtuse quote: He [not me] understands: "This field of perception is void of perception of man." -- The Quintessence of Buddhism To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: httpfs
I need not remind you that file systems front-ending onto random protocols are a bad idea for a huge number of reasons :-). That said, you might take a look at Intermezzo, which someone has already refered to indirectly in response to your e-mail -- Intermezzo is a file system for Linux (based on Coda) that allows perl programs to provide the back-end for a file system. I assume this is done in the style of AFS/Arla/Coda by having a small kernel module that up-calls to userland using a /dev/whatever entry, and then a userland program that satisfies the requests in some or another way. If you're really interested in implementing yet-another-nasty-file-system-abomination, take a look at Arla and rip out the AFS bits and replace them with HTTP bits :-). Robert N M Watson FreeBSD Core Team, TrustedBSD Project [EMAIL PROTECTED] NAI Labs, Safeport Network Services On Sat, 10 Mar 2001, Kris Kennaway wrote: A few of us were talking on IRC tonight about how cool it would be to have an httpfs filesystem -- then it occurred to me we almost have this already, in the form of the (under-utilised) portalfs. Portalfs works by handing off everything to a userland daemon which handles the actual transaction request, so you could easily imagine extending it to provide an http method similar to the tcp method it currently has for initiating tcp connections. One could probably make this more generic and finish implementing the undocumented 'exec' method (which currently exists as a stub): this would run an administrator-specified command (i.e. fixed in /etc/portal.conf) and pipe the output back to the user. A fully navigable httpfs (e.g. one you can ls and cd around in) is more work and probably requires a full stacking layer, but this would still be pretty cool. Is anyone feeling inspired to implement this? :-) Kris To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
