Re: Roadmap 0.11?
On Tue, Feb 9, 2010 at 3:51 AM, Mark Hammond skippy.hamm...@gmail.com wrote: On 9/02/2010 7:05 AM, Per Ejeklint wrote: And will official windows builds be available with it? I've been putting together the official binaries for 0.10 and will be making some just as official for 0.11. I expect that for 0.11 and later though, the builds I put together will be listed on the main download page making them really official instead of just official :) http://people.apache.org/~mhammond/dist/ Mark. There Is an old threads about what will/could be in 0.11. I think it would be good to have a roadmap published. It's definitely needed for non developers people that want to have an idea of futures developments and orientation. - benoît
Roadmap 0.11?
Jan asked me to take this in dev@ so here I am. :-) The official roadmap page (http://couchdb.apache.org/roadmap.html) is a tiiiny little outdated. As I'm trying to convince my new customer to make an epic technology leap I need as much info as possible about the near future for CouchDB. 0.11 seems to feature rich enough for our production environment. So can you with insights comment on about when we will have a 0.11 release? And will official windows builds be available with it? Cheers, /Per
Re: roadmap 0.11/1.0
On Sun, Nov 1, 2009 at 8:15 PM, Chris Anderson jch...@apache.org wrote: On Sun, Nov 1, 2009 at 11:00 AM, Suhail Ahmed suhail...@gmail.com wrote: Hi, Any chance of seeing native erlang RPC protocol in 11 or soon there after? This may be part of the Cloudant clustering codebase. I can't speak for them, but from what I've heard it does inter-node communication in a native Erlang way. You could probably use this interface as a primary client interface as well, but what do I know? :) Chris cloudant is a clustered couchdb or a cluster system over couchdb ? - benoît
Re: roadmap 0.11/1.0
On Nov 2, 2009, at 4:19 AM, Benoit Chesneau wrote:
On Sun, Nov 1, 2009 at 8:15 PM, Chris Anderson jch...@apache.org
wrote:
On Sun, Nov 1, 2009 at 11:00 AM, Suhail Ahmed suhail...@gmail.com
wrote:
Hi,
Any chance of seeing native erlang RPC protocol in 11 or soon
there after?
This may be part of the Cloudant clustering codebase. I can't speak
for them, but from what I've heard it does inter-node communication
in
a native Erlang way. You could probably use this interface as a
primary client interface as well, but what do I know? :)
Chris
cloudant is a clustered couchdb or a cluster system over couchdb ?
- benoît
Hi Benoit, we (Cloudant) are developing a clustered CouchDB; that is,
the ability to shard a database across a variable number of CouchDB
instances and have any of those instances handle any HTTP API
request. The instances communicate with each other using distributed
Erlang. The distribution system is Dynamo-flavored consistent hashing
(actually borrowing significantly from dynomite), and view results are
merged and re-reduced at query time. We're still working hard on a
few technical issues (in particular, generalizing single-instance
update sequences to a distributed notion of this-happened-first for
a proper _changes feed), but I think most of the CouchDB 0.10.0 API is
in pretty good shape. We'll be releasing the source code soon; I'm
afraid I can't be any more specific at the moment.
As far as whether the code has the makings of an Erlang remote client
library well, yes and no. It turns out the CouchDB CRUD
operations mostly just work when you open the DB using an RPC call
to the remote node. Something like
gen_server:call({couch_server, Node}, {open, DbName, Options})
or even
rpc:call(Node, couch_db, open, [DbName, Options])
Once you get a #db{} record filled with remote Pids, you can use it
just like you would a local one. Pretty nice, that. It means that
hovercraft and CouchDB need relatively few adjustments in order to run
in different VMs.
Cheers, Adam
Re: roadmap 0.11/1.0
On Mon, Nov 2, 2009 at 6:59 PM, Adam Kocoloski kocol...@apache.org wrote:
On Nov 2, 2009, at 4:19 AM, Benoit Chesneau wrote:
On Sun, Nov 1, 2009 at 8:15 PM, Chris Anderson jch...@apache.org wrote:
On Sun, Nov 1, 2009 at 11:00 AM, Suhail Ahmed suhail...@gmail.com
wrote:
Hi,
Any chance of seeing native erlang RPC protocol in 11 or soon there
after?
This may be part of the Cloudant clustering codebase. I can't speak
for them, but from what I've heard it does inter-node communication in
a native Erlang way. You could probably use this interface as a
primary client interface as well, but what do I know? :)
Chris
cloudant is a clustered couchdb or a cluster system over couchdb ?
- benoît
Hi Benoit, we (Cloudant) are developing a clustered CouchDB; that is, the
ability to shard a database across a variable number of CouchDB instances
and have any of those instances handle any HTTP API request. The instances
communicate with each other using distributed Erlang. The distribution
system is Dynamo-flavored consistent hashing (actually borrowing
significantly from dynomite), and view results are merged and re-reduced at
query time. We're still working hard on a few technical issues (in
particular, generalizing single-instance update sequences to a distributed
notion of this-happened-first for a proper _changes feed), but I think
most of the CouchDB 0.10.0 API is in pretty good shape. We'll be releasing
the source code soon; I'm afraid I can't be any more specific at the
moment.
That's already a lot and really good to know :) I was thinking more
and more to it since i have special needs in term of storage for a
project. Thanks a lot to make it opensource.
As far as whether the code has the makings of an Erlang remote client
library well, yes and no. It turns out the CouchDB CRUD operations
mostly just work when you open the DB using an RPC call to the remote
node. Something like
gen_server:call({couch_server, Node}, {open, DbName, Options})
or even
rpc:call(Node, couch_db, open, [DbName, Options])
Once you get a #db{} record filled with remote Pids, you can use it just
like you would a local one. Pretty nice, that. It means that hovercraft and
CouchDB need relatively few adjustments in order to run in different VMs.
About that I had done some works on rpc calls based on hovercraft.
Code is here and wasn't finished :
http://github.com/benoitc/couchdb/blob/rpc/src/couchdb/couch_rpc.erl
- benoit
Re: roadmap 0.11/1.0
On Mon, Nov 2, 2009 at 7:10 PM, Benoit Chesneau bchesn...@gmail.com wrote: About that I had done some works on rpc calls based on hovercraft. Code is here and wasn't finished : http://github.com/benoitc/couchdb/blob/rpc/src/couchdb/couch_rpc.erl Reading this code I don't know why I used a gen_serv for that anyway maybe it could be useful. - benoit
Re: roadmap 0.11/1.0
On Nov 2, 2009, at 12:56 AM, Randall Leeds wrote: I'd like to add to see a patch to add filtering replication processes with an Erlang function. I think it would have lots of utility for any clustering solution that wants to get pushed into the trunk. I'm going to try to write the patch tomorrow. I think it seems fairly straightforward, but if anyone has some thoughts before I get started, let me know. Are you planning to filter on full documents, or just id/rev pairs?
Re: roadmap 0.11/1.0
On 1 Nov 2009, at 15:44, Benoit Chesneau wrote: Hi all, http://couchdb.apache.org/roadmap.html hasn't been updated. And in fact i'm really curious. What is the next things on the roadmap ? Also damien spoke in june to have a fixed release schedule (one every 6 months ?) is it still something in view ? This one definitely belongs on dev@ :) Cheers Jan --
Fwd: roadmap 0.11/1.0
was sent on user@ sorry for crossposting . -- Forwarded message -- From: Benoit Chesneau bchesn...@gmail.com Date: Sun, Nov 1, 2009 at 3:44 PM Subject: roadmap 0.11/1.0 To: u...@couchdb.apache.org Hi all, http://couchdb.apache.org/roadmap.html hasn't been updated. And in fact i'm really curious. What is the next things on the roadmap ? Also damien spoke in june to have a fixed release schedule (one every 6 months ?) is it still something in view ? - benoit
Re: roadmap 0.11/1.0
Hi, Any chance of seeing native erlang RPC protocol in 11 or soon there after? Cheers Suhail On Sun, Nov 1, 2009 at 6:48 PM, Chris Anderson jch...@apache.org wrote: Thanks Benoit for the topic, I've got a wishlist for 0.11 / 1.0 - some of it I'd be game to implement, some of it I think others would be better suited for: Accounts tab in Futon Eventually we'll need something like /_utils but not for developers. For now we just need a page in Futon where you can login and out, as well as manage user accounts if you are an admin. Listable _changes If you could format the changes feed with a _list style API, you could use it to drive XMPP or other protocols. I'm keen on building a version of Toast chat that works even in browsers that have JS disabled. I have a feeling the only person who's gonna write this patch is me. couchjs security The couchjs process is currently secure enough for the context where only admins can modify design documents. However, as CouchDB spreads, we're sure to see misconfigured instances out there. Also, making the JS capabilities more controlled will definitely protect against some attacks in shared hosting environments. (Eg those where many users have private dbs on the same node.) Currently JS functions can hack the sandbox to make http requests via curl. We need this functionality for the test suite, but not for the design document OS process. So we should use a command line flag to --enable-http that the test runner can use. We can also be more secure in our [reset] handling. Because there's no such thing as a real JS sandbox, if we move our [reset] handling to C, and have it swap out the JS context for a new one, we'll avoid the case where databases on the same node can spy on each other, by say, overwriting the emit() function to also store important values for later playback to the attacker db. There could be some performance impacts of a C-based reset (as it would involve compiling all of main.js after each reset). An alternate way to implement this is just to stop recycling processes in Erlang, and through them out after each use. I think that will get expensive (but maybe not much worse than C-based reset). This is trivial patch if anyone needs to run extra securely in a shared-host environment today. cron / event / changes handler Applications need to be able to trigger functionality in a periodic or event-based way. We could probably piggyback on _changes heartbeat to provide cron + event services. The idea is a design document function (event or cron or maybe trigger) that is called once for each item in the changes feed. This is the one I'm least sure about, but I've heard a lot of people request it. It's problematic because cron functionality isn't that useful unless it has side-effects, which brings the whole sandbox/http question up again. rewriter There's getting to be a pretty common pattern where people write CouchApps and then deploy them behind a rewrite proxy. We've already got rewrite patches floating around. It's just a matter of making the API decisions. clustering I've heard Cloudant has some clustering code, I'd definitely be willing to help with integration, and I'm sure there are other people who would as well. Cheers, Chris On Sun, Nov 1, 2009 at 7:02 AM, Benoit Chesneau bchesn...@gmail.com wrote: was sent on user@ sorry for crossposting . -- Forwarded message -- From: Benoit Chesneau bchesn...@gmail.com Date: Sun, Nov 1, 2009 at 3:44 PM Subject: roadmap 0.11/1.0 To: u...@couchdb.apache.org Hi all, http://couchdb.apache.org/roadmap.html hasn't been updated. And in fact i'm really curious. What is the next things on the roadmap ? Also damien spoke in june to have a fixed release schedule (one every 6 months ?) is it still something in view ? - benoit -- Chris Anderson http://jchrisa.net http://couch.io
Re: roadmap 0.11/1.0
On Sun, Nov 1, 2009 at 11:00 AM, Suhail Ahmed suhail...@gmail.com wrote: Hi, Any chance of seeing native erlang RPC protocol in 11 or soon there after? This may be part of the Cloudant clustering codebase. I can't speak for them, but from what I've heard it does inter-node communication in a native Erlang way. You could probably use this interface as a primary client interface as well, but what do I know? :) Chris Cheers Suhail On Sun, Nov 1, 2009 at 6:48 PM, Chris Anderson jch...@apache.org wrote: Thanks Benoit for the topic, I've got a wishlist for 0.11 / 1.0 - some of it I'd be game to implement, some of it I think others would be better suited for: Accounts tab in Futon Eventually we'll need something like /_utils but not for developers. For now we just need a page in Futon where you can login and out, as well as manage user accounts if you are an admin. Listable _changes If you could format the changes feed with a _list style API, you could use it to drive XMPP or other protocols. I'm keen on building a version of Toast chat that works even in browsers that have JS disabled. I have a feeling the only person who's gonna write this patch is me. couchjs security The couchjs process is currently secure enough for the context where only admins can modify design documents. However, as CouchDB spreads, we're sure to see misconfigured instances out there. Also, making the JS capabilities more controlled will definitely protect against some attacks in shared hosting environments. (Eg those where many users have private dbs on the same node.) Currently JS functions can hack the sandbox to make http requests via curl. We need this functionality for the test suite, but not for the design document OS process. So we should use a command line flag to --enable-http that the test runner can use. We can also be more secure in our [reset] handling. Because there's no such thing as a real JS sandbox, if we move our [reset] handling to C, and have it swap out the JS context for a new one, we'll avoid the case where databases on the same node can spy on each other, by say, overwriting the emit() function to also store important values for later playback to the attacker db. There could be some performance impacts of a C-based reset (as it would involve compiling all of main.js after each reset). An alternate way to implement this is just to stop recycling processes in Erlang, and through them out after each use. I think that will get expensive (but maybe not much worse than C-based reset). This is trivial patch if anyone needs to run extra securely in a shared-host environment today. cron / event / changes handler Applications need to be able to trigger functionality in a periodic or event-based way. We could probably piggyback on _changes heartbeat to provide cron + event services. The idea is a design document function (event or cron or maybe trigger) that is called once for each item in the changes feed. This is the one I'm least sure about, but I've heard a lot of people request it. It's problematic because cron functionality isn't that useful unless it has side-effects, which brings the whole sandbox/http question up again. rewriter There's getting to be a pretty common pattern where people write CouchApps and then deploy them behind a rewrite proxy. We've already got rewrite patches floating around. It's just a matter of making the API decisions. clustering I've heard Cloudant has some clustering code, I'd definitely be willing to help with integration, and I'm sure there are other people who would as well. Cheers, Chris On Sun, Nov 1, 2009 at 7:02 AM, Benoit Chesneau bchesn...@gmail.com wrote: was sent on user@ sorry for crossposting . -- Forwarded message -- From: Benoit Chesneau bchesn...@gmail.com Date: Sun, Nov 1, 2009 at 3:44 PM Subject: roadmap 0.11/1.0 To: u...@couchdb.apache.org Hi all, http://couchdb.apache.org/roadmap.html hasn't been updated. And in fact i'm really curious. What is the next things on the roadmap ? Also damien spoke in june to have a fixed release schedule (one every 6 months ?) is it still something in view ? - benoit -- Chris Anderson http://jchrisa.net http://couch.io -- Chris Anderson http://jchrisa.net http://couch.io
Re: roadmap 0.11/1.0
Hey Chris, I like hovercraft and it's simple and clean api to talk to couch. Are you talking about clustering over RPC? I know way too little about couch to mouth off like this, working my way to defining a structure for trees and graphs in couch is keeping me busy enough. Cheers Suhail On Sun, Nov 1, 2009 at 7:15 PM, Chris Anderson jch...@apache.org wrote: On Sun, Nov 1, 2009 at 11:00 AM, Suhail Ahmed suhail...@gmail.com wrote: Hi, Any chance of seeing native erlang RPC protocol in 11 or soon there after? This may be part of the Cloudant clustering codebase. I can't speak for them, but from what I've heard it does inter-node communication in a native Erlang way. You could probably use this interface as a primary client interface as well, but what do I know? :) Chris Cheers Suhail On Sun, Nov 1, 2009 at 6:48 PM, Chris Anderson jch...@apache.org wrote: Thanks Benoit for the topic, I've got a wishlist for 0.11 / 1.0 - some of it I'd be game to implement, some of it I think others would be better suited for: Accounts tab in Futon Eventually we'll need something like /_utils but not for developers. For now we just need a page in Futon where you can login and out, as well as manage user accounts if you are an admin. Listable _changes If you could format the changes feed with a _list style API, you could use it to drive XMPP or other protocols. I'm keen on building a version of Toast chat that works even in browsers that have JS disabled. I have a feeling the only person who's gonna write this patch is me. couchjs security The couchjs process is currently secure enough for the context where only admins can modify design documents. However, as CouchDB spreads, we're sure to see misconfigured instances out there. Also, making the JS capabilities more controlled will definitely protect against some attacks in shared hosting environments. (Eg those where many users have private dbs on the same node.) Currently JS functions can hack the sandbox to make http requests via curl. We need this functionality for the test suite, but not for the design document OS process. So we should use a command line flag to --enable-http that the test runner can use. We can also be more secure in our [reset] handling. Because there's no such thing as a real JS sandbox, if we move our [reset] handling to C, and have it swap out the JS context for a new one, we'll avoid the case where databases on the same node can spy on each other, by say, overwriting the emit() function to also store important values for later playback to the attacker db. There could be some performance impacts of a C-based reset (as it would involve compiling all of main.js after each reset). An alternate way to implement this is just to stop recycling processes in Erlang, and through them out after each use. I think that will get expensive (but maybe not much worse than C-based reset). This is trivial patch if anyone needs to run extra securely in a shared-host environment today. cron / event / changes handler Applications need to be able to trigger functionality in a periodic or event-based way. We could probably piggyback on _changes heartbeat to provide cron + event services. The idea is a design document function (event or cron or maybe trigger) that is called once for each item in the changes feed. This is the one I'm least sure about, but I've heard a lot of people request it. It's problematic because cron functionality isn't that useful unless it has side-effects, which brings the whole sandbox/http question up again. rewriter There's getting to be a pretty common pattern where people write CouchApps and then deploy them behind a rewrite proxy. We've already got rewrite patches floating around. It's just a matter of making the API decisions. clustering I've heard Cloudant has some clustering code, I'd definitely be willing to help with integration, and I'm sure there are other people who would as well. Cheers, Chris On Sun, Nov 1, 2009 at 7:02 AM, Benoit Chesneau bchesn...@gmail.com wrote: was sent on user@ sorry for crossposting . -- Forwarded message -- From: Benoit Chesneau bchesn...@gmail.com Date: Sun, Nov 1, 2009 at 3:44 PM Subject: roadmap 0.11/1.0 To: u...@couchdb.apache.org Hi all, http://couchdb.apache.org/roadmap.html hasn't been updated. And in fact i'm really curious. What is the next things on the roadmap ? Also damien spoke in june to have a fixed release schedule (one every 6 months ?) is it still something in view ? - benoit -- Chris Anderson http://jchrisa.net http://couch.io -- Chris Anderson http://jchrisa.net http://couch.io
Re: roadmap 0.11/1.0
On Nov 1, 2009, at 4:59 PM, Jan Lehnardt wrote: On 1 Nov 2009, at 19:48, Chris Anderson wrote: cron / event / changes handler Applications need to be able to trigger functionality in a periodic or event-based way. We could probably piggyback on _changes heartbeat to provide cron + event services. The idea is a design document function (event or cron or maybe trigger) that is called once for each item in the changes feed. This is the one I'm least sure about, but I've heard a lot of people request it. It's problematic because cron functionality isn't that useful unless it has side-effects, which brings the whole sandbox/ http question up again. We need this for 0.11 to allow continuous replication to survive server crashes. +1 This is pretty much the only feature I want before 1.0.0. Everything else in this thread so far isn't important enough to hold up 1.0.0. In my opinion, we should focus on fixing bugs, filling in small gaps in the APIs, better testing and optimizations. I think we should set a hard date for Feb 1 for feature freeze for a 0.11.0 branch. The next version, 0.11.0, once stabilized, should be considered a release candidate, followed by 1.0.0 once proven stable in production. Once we branch for 0.11.0, All new features will be in trunk for 2.0.0, and might be included in a later 1.1.0 release if needed. -Damien While we are at it, we should add means to trigger compaction periodically or when certain limits are reached (IIRC Adam mentioned a waste-factor for databases to see when compaction is worth triggering). This would be the first thing I'd start working for new features. We also have a few open ends in the API that need consolidation. e.g. making the show/list/update handler more RFC2616 aware (sending Location e.g.). There's a few more of these. We should probably work on them before working on new half-baked features :) -- Furthermore, more etap tests and a benchmarking suite would be great. Cheers Jan -- rewriter There's getting to be a pretty common pattern where people write CouchApps and then deploy them behind a rewrite proxy. We've already got rewrite patches floating around. It's just a matter of making the API decisions. clustering I've heard Cloudant has some clustering code, I'd definitely be willing to help with integration, and I'm sure there are other people who would as well. Cheers, Chris On Sun, Nov 1, 2009 at 7:02 AM, Benoit Chesneau bchesn...@gmail.com wrote: was sent on user@ sorry for crossposting . -- Forwarded message -- From: Benoit Chesneau bchesn...@gmail.com Date: Sun, Nov 1, 2009 at 3:44 PM Subject: roadmap 0.11/1.0 To: u...@couchdb.apache.org Hi all, http://couchdb.apache.org/roadmap.html hasn't been updated. And in fact i'm really curious. What is the next things on the roadmap ? Also damien spoke in june to have a fixed release schedule (one every 6 months ?) is it still something in view ? - benoit -- Chris Anderson http://jchrisa.net http://couch.io
Re: roadmap 0.11/1.0
On Sun, Nov 1, 2009 at 4:55 PM, Mark Hammond skippy.hamm...@gmail.com wrote: On 2/11/2009 9:13 AM, Paul Davis wrote: couchjs security ... The first step to securing cURL handlers I'm going to tackle is the basic all-or-nothing HTTP support. With something like --with-http as Chris suggests. Mark also made a suggestion on including a --with-http-host=127.0.0.1 or similar to support HTTP requests to a specific host only. This is probably doable but I'll have to think a bit harder on how to force that in the cURL handlers. My use-case is to avoid allowing full-blown cURL access to javascript while still allowing a couchdb 'external' to have access to its database. Specifically, I want to use the 'externals' mechanism to build an application specific API - an API which hides/abstracts multiple couchdb calls and subsequent data munging into a single call which is more relevant to the application's direct requirements. For example, if I have an external which is referenced via http://host:port/dbname/_external, the implementation of that external needs access to the referenced 'dbname', but no others. On IRC, there were 2 thoughts: * Paul mentioned the cmd-line switch to disable cURL - if we provided a new option that only allowed connections back to the same DB (probably by exposing an API where the host and DB name can't even be specified by the js code), it might still meet the general security requirements. * Benoit mentioned that if we changed the externals 'protocol' slightly, we could probably offer a javascript API back to the database which uses stdin/stdout to exchange requests and responses, allowing us to live without cURL support at all. Such an API may also be able to help allow cron jobs etc to have safe side-effects - the only side-effects they can perform are ones exposed by couchdb itself, not via generic http requests. The second option sounds better, but much more work. I'm willing to roll my sleeves up on this, so please share your thoughts (or ask me to expand on my use-case if necessary) Eventually I'd like to see an API to allow users to grant applications the right to use HTTP in an unrestricted way. Think of the HTTP requirement for an XML feed-reader CouchApp. It seems odd to me that CouchDB has so much application horsepower, but has to rely on browser + JSONP to be the API client of 3rd party services. Bridging this gap safely is important. I like your proposal of safe DB-only access via restricted curl. The original action.js I committed long ago [1] had an insecure implementation of this. Once we have something here, extending it for more nuanced models will be worth a lot. Chris [1] http://svn.apache.org/viewvc?view=revisionrevision=727141 Thanks, Mark -- Chris Anderson http://jchrisa.net http://couch.io
