On Wed, Jun 9, 2010 at 10:04 AM, Aaron Sherman <a...@ajs.com> wrote: > Has anyone begun to consider what kind of filesystem interface we want > for things like sftp, Amazon S3, Google Storage and other remote > storage possibilities? Is there any extant work out there, or should I > just start spit-balling?
In the absence of anything forthcoming and a totally arbitrary sense of urgency ;-) here's what I think I should do: IO::FileSystems (S32) gives us some basics and the Path role also provides some useful features. I will start there and build an IO::FileSystems::VFS roughly like: class IO::VFS is IO::FileSystems { ... # Session data if applicable has IO::VFS::Session $.session; # Many methods take a $context which, if supplied # will contain back-end specific data such as restart markers # or payment model information. I'll probably define # a role for the context parameter, but otherwise # leave it pretty loose as a back-end specific structure. # A simple operation that guarantees a round-trip to the filesystem method nop($context?) { ... } # list of sub-IO::VFS partitions/buckets/etc method targets($context?) { ... } method find_target($locator, $context?) { ... } # Means of acquiring file-level access through a VFS method find($locator, $enc = $.session.encoding, $context?) { ... } method glob(Str $matcher, $enc = $.session.encoding, $context?) { ... } # Like opening and writing to filehandle, but the operation is totally # opaque and might be a single call, senfile or anything else. # Note that this doesn't replace $obj.find($path).write(...) method put($locator, $data, $enc = $.session.encoding, $context?) { ... } # Atomic copy/rename, etc. are logically filesystem operations, even though # they might have counterparts at the file level. The distinction being that # at the filesystem level I never know nor care what the contents of the # file are, I just ask for an operation to be performed on a given path. method copy($from, $to, $enc = $.session.encoding, $context?) { ... } method rename($from, $to, $enc = $.session.encoding, $context?) { ... } method delete($locator, $enc = $.session.encoding, $context?) { ... } # service-level ACLs if any method acl($locator, $context?) { ... } } The general model I imagine would be something like: my IO::VFS::S3 $s3 .= new(); $s3.session.connect($amazonlogininfo); my $bucket = $s3.find_target($bucket_name); $bucket.put("quote.txt", "Now is the time for all good men...\n"); say "URI: ", $bucket.find("quote.txt").uri; or my IO::VFS::GoogleStorage $goog .= new(); $goog.session.connect($googlelogininfo); my $bucket = $goog.find_target($bucket_name); $bucket.put("quote.txt", "Now is the time for all good men...\n"); say "URI: ", $bucket.find("quote.txt").uri; or my IO::VFS::SFTP $sftp .= new(); $sftp.session.connect(:host<storage>, :user<ajs>, :password<iforgotit>); my $filesystem = $sftp.find_target("/tmp"); $filesystem.put("quote.txt", "Now is the time for all good men...\n"); say "URI: ", $filesystem.find("quote.txt").uri; # using sftp:... Notice that everything after $obj.session.connect is identical except for my choice of variable names. In fact, you shouldn't have to worry about what storage back-end you're using as long as you have a valid VFS handle. Really path names are the only thing that might trip you up. Thoughts? I think that in order to do this, I'll need the following support libraries which may or may not exist (I'll be looking into these): IO::FileSystems Path HTTP (requires socket IO, MIME, base64, etc.) Various crypto libs I don't intend to provide a finished implementation of any of these where they don't already exist (I may not even end up with a final implementation of the VFS layer), but at least I'll get far enough along that others who want to work on this will have a starting point, and I'll want to at least have a test that fakes its way all the way down to creating a remote file on all three services, even if most of the operations involve passing on blobs of data generated by equivalent calls in other languages. -- Aaron Sherman Email or GTalk: a...@ajs.com http://www.ajs.com/~ajs