On Sunday, 29 December 2013 at 15:05:31 UTC, Andrei Alexandrescu
wrote:
Well one question is what other successful designs could we use
as precedent? I don't know of any successful unified APIs for
regular/remote filesystems that also allow full local file
functionality. The closest abstraction I know of is the
FUSE-installed filesystems, but those adapt foreign file
systems to the Posix interface.
I guess file storage design is currently in flux and a bit
lacking in terms of APIs, partially because of cloud computing
and SSDs, so my suggestion would be to:
1. Provide os.linux, os.windows, os.osx, os.ios, os.posix for low
level local-storage access which closely resembles the
native/standard APIs.
2. Provide a novel hierarchy of abstraction for basic file I/O
that is highly portable and afford modular and extensible
implementation, and make those really easy to use in order to
encourage portable libraries for D. I don't think the overhead of
function calls through interfaces matters all that much because
the cost associated with context shifts for system calls will
dominate that by a solid margin (my assumption).
So basically on the most abstract and limited level you get a
key/value cache (memcache). The next level is key/value datastore
(couchdb/mongodb/amazon/google etc).
Then the ability to list all keys.
Then the ability list keys based on hiearchy (directories) etc.
If you can specify what kind of functionality your application
needs and set those constraints in a file-system context object
then you can set the level of portability for your application. I
think this would be a real advantage. Then let that file-system
object have a factory for "file-mode" and let that do the right
thing when you attempt to set the filetype as a MIME-constant
(e.g set the right extension or set the right attribute).
Another advantage is that you could create "safe virtual
filesystems" e.g. create a file-system context object that
resolve ".." and prefix all paths with "/tmp/" and set default
"file-mode" for created objects (like read-only and owner). Which
would make it easy to write more secure web-servers. You could
even specify a name-mangler and create a db-like filesystem for
arbitrary keys with automatic creation of directories (for
efficiency) if the abstraction level is right.
I don't think the overhead is all that much if it is done in a
modular fashion. You only import what you need. Finding the right
abstraction is not trivial, I agree. So you need to do some
analysis of what current storage-solutions provide to find the
common denominators.
(Having done some work with ndb on App Engine and the HttpRequest
stuff in Dart I see the advantage of wrapping up resource loading
as Futures. Mostly sugar, but uniform and easy to remember if
used on all resources. E.g. request all resources as futures,
then process them as they become available.)
