Extensions On Jan 23, 2012, at 4:33 AM, Paul Davis wrote:
> On Mon, Jan 23, 2012 at 3:03 AM, Randall Leeds <randall.le...@gmail.com> > wrote: >> On Sat, Dec 31, 2011 at 10:34, Jan Johannes <j...@ntr.io> wrote: >>> Dear favorite Document-database developer community :-) >>> This is my first message to the mailing list, so please forgive me and >>> gently correct me if i am breaking any communication standards whatsoeverŠ >> >> Nope! You're just bringing up a deep topic with a lot of text. >> I hope you didn't expect a speedy response! :) >> >>> >>> I am currently working on a WebDAV interface for CouchDB, that now gets >>> closer to its release date, somewhere in February or March and as i >>> approach that, I want to package everything in a proper CouchDB Module >>> format but the CouchDB Module system is, as i understood, in early >>> development or pre-development phase and needs a bit of work. >>> >>> So as i work on a CouchDB Module anyways i would like to contribute to the >>> Module system or start kicking the work on that of. >>> >>> Hence the following questions: >>> >>> Who started working on such a system or would be interested in talking >>> about the design decisions that need to be done before actual coding can >>> start? >>> (I know at least Jan Lehnardt expressed interest, but i can't quite >>> remember if he started actual work or just had plans for that.) >> >> Benoit is particularly interested and there is activity on the refuge >> project mailing list about it from Nicolas Dufour. >> I believe Jason Smith might be pretty interested since he's authored >> some "plugins" already. >> Others on the list are certainly interested. >> I'm keen to get a good module system and it's why I was in favor of >> the effort to get "couch-config" shipped as part of the install. >> >>> >>> If there is any documentation on planning such a system, could someone >>> please point me to the recourses? >> >> There's an umbrella ticket I made for modularizing the code base more: >> https://issues.apache.org/jira/browse/COUCHDB-1251 >> >> And these: >> >> Utility to help plugin developers manage paths >> https://issues.apache.org/jira/browse/COUCHDB-1012 >> >> support load of external erlang modules in couchdb. >> https://issues.apache.org/jira/browse/COUCHDB-1046 >> >>> >>> Lastly here is my two cents on a few topics that need to be addressed: >>> >>> - We should build the best module system there is for a database or >>> content management system to gain back a bit of the traction we lost >>> during the recent plateau phase of couchdb enthusiasm >> >> I find it hard to judge enthusiasm from the inside, but, yeah... a >> proper module system would rock. >> One of the things CouchDB struggles with in terms of image, but excels >> at in terms of function is scaling up and down. The only way to really >> make one project move from mobile phone to datacenter cluster is to >> modularize the build and deployment of key components. I think >> everyone agrees about this. CouchDB is the Gentoo of document >> databases in this way. >> >>> >>> - I think "module" is the preferable term, as extension, plugin or the >>> like sound like second class citizens, whereas module implies, that even a >>> core component of the system can be made into such an instance (Any other >>> suggestions?) >> >> Module is a little bit confusing because it conflicts with Erlang's >> definition. While the two would often be synonymous for many add-ons, >> it's not necessarily the case. >> I would go with calling them "applications" maybe. I've often thought >> that each such thing could correspond well to a single Erlang >> application and .app file, which Erlang has significant support and >> prior art for loading and unloading on the fly. But really, I don't >> care what color we paint this and long as it's neon awesome. >> >>> >>> - A CouchDB module should be packaged as a stand alone couchdb-document, >>> with the yet to be specified manifest beeing the json structure of the >>> document and the files beeing the attachment. >> >> I'm pretty sure I disagree here. I think this conflates two issues. >> The first, to me, is a good hooks system from which more interactions >> in the core could be scripted with sandboxed user code. Once again, I >> know I've talked to Benoit about this. The second, though, is the more >> first-class type of application that requires compiled Erlang >> bytecode. While these have some degree of portability[1] I >> instinctively cringe, from a security standpoint, at the idea of hot >> loading code from a document into the VM. While the document-module >> idea I find attractive from a share-ability and install-ability >> standpoint, I don't think it befits the general purpose module system. >> Although, a good hooks system may make certain kinds of modules work >> this way. >> >> [1] >> http://stackoverflow.com/questions/2255658/how-portable-are-erlang-beam-files >> >>> >>> - Discovery, Installation, Enabling, Disabling, Module-Specific >>> Configuration and Distribution should be part of Futon and introduce as >>> little new interface concepts as possible, also the futon interface >>> section "configuration" may be needed to be integrated and streamlined >>> quite nicely >> >> Again, this works the dynamic-language, hooks-based module, but I'm >> not sure it works for all modules. Kanso [2] is doing this to a large >> extent already, under the CouchApp frame, but I see no reason why a >> hooks system couldn't accommodate a new, more powerful kind of Kanso >> package. >> >> [2] http://kan.so/ >> >>> >>> - Even if a module is not possible to completely install via this method >>> (e.g. It depends on a os component, that is too low level etc.) such >>> modules should be as tightly integrated into the system as possible and >>> the manual admin installation steps should be documented in a way, so that >>> in a normal case no place but the module system needs to be accessed >> >> Agree. >> >>> >>> - on the top level the system should be the same for all OSes >>> >>> - The work packages i see at the moment for the structure i have in mind >>> would be: >>> >>> - Module repository: A CouchDB Instance, that hosts a collection of >>> CouchDB Modules >> >> See Kanso, above. >> >>> - Futon User Interface to browse available Modules in the configured >>> repositories, should be close to the Firefox plugin paradigms >> >> That would be neat. Or a separate 'shopping' utility for CouchApps. >> Sounds a bit like a combination between JChris' CouchApp Garden [3] >> and Max Ogden's Chrome Couch application [4]. Crossing these would >> Kanso would be really neat. >> >> [3] https://github.com/jchris/garden >> [4] https://github.com/maxogden/chromecouch >> >>> - Module Manifest: Descriptive CouchDoc, that references the attached >>> files or git repository, how files need to be fetched, packaged and what >>> goes where, also version dependencies, description, rating, >>> Module >>> Category, needed system privileges (basically if you need to be a couch >>> admin or if you need to be root on the os), needed >>> configuration, and >>> basic description of the configuration parameters >> >> I'd really prefer to punt on low-level privileges and just call this >> two separate things. The way to unify them, in my mind, is to create a >> hooks system and then to make a native module that listens for all >> hooks and delegates processing to design documents that indicate they >> handle certain hooks. In other words, build the un-privileged module >> system on top of the privileged one. Let administrators worry about >> file systems when installing the privileged ones. It's really not so >> bad. Sysadmins have been installing things for years, and actually >> probably prefer to install stuff on the command line than with a POST >> to /_replicate. The couch-config utility is there to facilitate this. >> What we probably need is an extra location for user-installed add-on >> applications that need to be added to the Erlang library path and used >> for loading configured applications. >> >>> - Futon User Interface to the installed modules, with options to >>> enable, >>> disable, uninstall, rate and configure >> >> Yes again. I like the gist, but I don't know if it should be bound up >> in Futon or not. >> >>> - The tools for installation/uninstallation and enabling/disabling as >>> well as packaging and dependency resolution >> >> That sounds like figuring out a good way to structure the >> configuration sections for modules. See the JIRA tickets I referenced >> for some discussion of that. >> >>> >>> Open questions: What existing projects could be made use of, maybe >>> there >>> is something great for the general erlang world, maybe there is a project >>> like zero install, that could be made use of, maybe that is not >>> necessary, what are your thoughts? >> >> I would also check out Agner: http://erlagner.org/ >> >> Basically, if we could get to the point where CouchDB administrators >> could use Agner to install new pieces I'd consider that a huge win. >> If, in the process, we get a modularized core and support for a >> design-document-level hooks system I'd consider that an epic win. I >> never say epic win. I would say it three times out loud. >> >>> >>> I wish you all a happy new year! >>> >>> >> >> Thanks :) >> -Randall > > +1 to pretty much everything Randall said. > > Minor caveat: application is about as overloaded in Erlang as module > (and process for that matter) when considering prior expectations of > non-Erlangians. There should be a word for these things. Plug-ins has > always been my word for them, but the second-class-ness that invokes > is understandable. Perhaps there's a perfect word, but perhaps > "component" is good enough? > > My other reaction is that installing "whatevers/components/widgets" > should *not* be a web thing. I understand the desire to make this user > friendly and the such, but as Randall points out, putting random > (fairly difficult to sandbox) code into a server/service environment > seems quite ungood. This basically hearkens back to the > disabled-by-default native view server. While its definitely > "possible" to provide some sort of sandboxed thing for Erlang > extensions, its a "plug all the leaks with fingers" type of possible. > If we're interested in allowing code execution on the server it should > be "here's you're iron box with carefully crafted ports of interaction > to the world" type of sandboxing. Ie, AppEngine vs erm, not-AppEngine. > > Other than that, yes. Erlang makes extensions super awesomely easy and > we should be embracing that more loudly. Both with community resources > for discovery as well as the technical adjustments to ease their > usage.
