Re: Roadmap process and merge procedure
Following up on this. We are two weeks away from the next release. Bob, you mentioned on IRC that you could help me with this. What time is convenient for you? On 17 April 2013 16:01, Noah Slater nsla...@apache.org wrote: I'll let someone more familiar with Git, and the conversations we had around this, answer. To be honest, I am less interested in debating the specifics of the proposal than I am about actually getting a proposal agreed upon, and putting it into practice. We are a little over a week away from the next bugfix release window, and a little over two months away from the next feature release window. I need someone to help me put this stuff into motion. We keep talking about the new release process / merge procedure, but it's been almost a year now, and nothing has happened. (And I am as much if not more so to blame for that as anyone else.) What would be great is if someone who was involved in fleshing out the original proposal (Bob, Paul, Jan, etc) could lend a hand and pair up with me for an evening to get all this put into motion. On 17 April 2013 15:40, Dirkjan Ochtman dirk...@ochtman.nl wrote: How does that impact master vs. 1.4.x? On Wed, Apr 17, 2013 at 4:36 PM, Noah Slater nsla...@apache.org wrote: The goal was that you only merge in features when they are ready, and come with tests, and docs, and what have you. And that you actually call a lazy consensus merge request on dev@ before you can merge in. On 17 April 2013 15:18, Dirkjan Ochtman dirk...@ochtman.nl wrote: On Wed, Apr 17, 2013 at 4:02 PM, Noah Slater nsla...@apache.org wrote: So that features can be merged into it as they become ready. Check out: http://wiki.apache.org/couchdb/Merge_Procedure Thoughts? Seems like you could call the next-feature-release branch master, and not have to start a new branch every time. Cheers, Dirkjan -- NS -- NS -- NS
Re: Roadmap process and merge procedure
Hey guys, Just following up on this, as it's been five days. And I can't move ahead without help. At a minimum, we need to create the 1.4.x branch. But I'd really like for us to properly document our merge procedure (which has languished because I don't know Git well enough to fill out the details) and then for us to call a vote on it, so that we can move forward as we originally planned to, with features only landing on a release branch with tests, docs, etc. If someone wants to volunteer to pair with me, perhaps we can knock this out? Thanks, On 12 April 2013 13:31, Noah Slater nsla...@apache.org wrote: Devs, With 1.3.0 out, it is time we revisit these docs: http://wiki.apache.org/couchdb/Roadmap_Process http://wiki.apache.org/couchdb/Merge_Procedure Our next bugfix release target is 1st May. Our next feature release target is 1st July. What do we need to do *now* to prepare for these? Do we need to create release branches? If yes, can we do it, and can we document it, so I can bake it into the release procedure. Do we need to communicate how to merge in work for the releases? Remember we discussed (and sort of half documented in that second wiki page) that to merge in code for a release, you have to do a VOTE on a merge request to dev, and you have to include tests, and docs, and what have you, and we do lazy consensus on it. If we're serious about this release cadence (and I think we all are) then we need to sort this stuff out ASAP, and communicate it, and actually put it into practice. I need your help though, because I am still learning Git, etc. So I am hoping that a few of you can weigh in on the current state of the merge procedure proposal, and maybe fix it up. Then, once we're happy with it, I can do the DISCUSS/VOTE dance on dev@ and make this official. I am super excited about this. Thanks, -- NS -- NS
Re: Roadmap process and merge procedure
On Wed, Apr 17, 2013 at 3:45 PM, Noah Slater nsla...@apache.org wrote: At a minimum, we need to create the 1.4.x branch. But I'd really like for us to properly document our merge procedure (which has languished because I don't know Git well enough to fill out the details) and then for us to call a vote on it, so that we can move forward as we originally planned to, with features only landing on a release branch with tests, docs, etc. Sorry, this might just be me, but why would you want to branch for 1.4.x now instead of, like, 2 (or 4) weeks before the intended release date? Cheers, Dirkjan
Re: Roadmap process and merge procedure
So that features can be merged into it as they become ready. Check out: http://wiki.apache.org/couchdb/Merge_Procedure Thoughts? On 17 April 2013 14:47, Dirkjan Ochtman dirk...@ochtman.nl wrote: On Wed, Apr 17, 2013 at 3:45 PM, Noah Slater nsla...@apache.org wrote: At a minimum, we need to create the 1.4.x branch. But I'd really like for us to properly document our merge procedure (which has languished because I don't know Git well enough to fill out the details) and then for us to call a vote on it, so that we can move forward as we originally planned to, with features only landing on a release branch with tests, docs, etc. Sorry, this might just be me, but why would you want to branch for 1.4.x now instead of, like, 2 (or 4) weeks before the intended release date? Cheers, Dirkjan -- NS
Re: Roadmap process and merge procedure
On Wed, Apr 17, 2013 at 4:02 PM, Noah Slater nsla...@apache.org wrote: So that features can be merged into it as they become ready. Check out: http://wiki.apache.org/couchdb/Merge_Procedure Thoughts? Seems like you could call the next-feature-release branch master, and not have to start a new branch every time. Cheers, Dirkjan
Re: Roadmap process and merge procedure
The goal was that you only merge in features when they are ready, and come with tests, and docs, and what have you. And that you actually call a lazy consensus merge request on dev@ before you can merge in. On 17 April 2013 15:18, Dirkjan Ochtman dirk...@ochtman.nl wrote: On Wed, Apr 17, 2013 at 4:02 PM, Noah Slater nsla...@apache.org wrote: So that features can be merged into it as they become ready. Check out: http://wiki.apache.org/couchdb/Merge_Procedure Thoughts? Seems like you could call the next-feature-release branch master, and not have to start a new branch every time. Cheers, Dirkjan -- NS
Re: Roadmap process and merge procedure
I'll let someone more familiar with Git, and the conversations we had around this, answer. To be honest, I am less interested in debating the specifics of the proposal than I am about actually getting a proposal agreed upon, and putting it into practice. We are a little over a week away from the next bugfix release window, and a little over two months away from the next feature release window. I need someone to help me put this stuff into motion. We keep talking about the new release process / merge procedure, but it's been almost a year now, and nothing has happened. (And I am as much if not more so to blame for that as anyone else.) What would be great is if someone who was involved in fleshing out the original proposal (Bob, Paul, Jan, etc) could lend a hand and pair up with me for an evening to get all this put into motion. On 17 April 2013 15:40, Dirkjan Ochtman dirk...@ochtman.nl wrote: How does that impact master vs. 1.4.x? On Wed, Apr 17, 2013 at 4:36 PM, Noah Slater nsla...@apache.org wrote: The goal was that you only merge in features when they are ready, and come with tests, and docs, and what have you. And that you actually call a lazy consensus merge request on dev@ before you can merge in. On 17 April 2013 15:18, Dirkjan Ochtman dirk...@ochtman.nl wrote: On Wed, Apr 17, 2013 at 4:02 PM, Noah Slater nsla...@apache.org wrote: So that features can be merged into it as they become ready. Check out: http://wiki.apache.org/couchdb/Merge_Procedure Thoughts? Seems like you could call the next-feature-release branch master, and not have to start a new branch every time. Cheers, Dirkjan -- NS -- NS
Re: Roadmap for the 1.3.0 release
On Feb 15, 2012, at 03:22 , Brian Mitchell wrote: Has there been any discussion around BigCouch integration strategies? It seems like it would fit the bill for the next undertaking on the general couch side. Does anyone from Cloudant have a suggestion for the timeline here? Cloudant havent't yet approached Apache CouchDB with anything regarding their announcement. Any other work from mobile builds and the like might be interesting to support. Were there any interesting changes to pull in from the mobile and embedded device ports? Also, I'm not sure where the view engine refactoring work ended up… I'll look at JIRA but maybe there are specific follow-ups to be made here. The biggest one IIRC was the use of emonk instead of an external Spidermonkey for iOS. Maybe we can make that a compile option like ./configure --enable-emonk-view-server so people can choose which one to run. Finally, for our included javascript libraries and Futon runtime, are we going to replace or update anything here? I'd imagine a newer jQuery could be an advantage for those doing CouchApps. Good that you bring this up. There's also other upstream dependencies that we should look at updating and we don't have a process for that, it usually happens whenever a developer feels like it. I haven't thought about this much yet, but maybe update all dependencies could be a todo before a release branch is created? Cheers Jan --
Re: Roadmap for the 1.3.0 release
On Feb 14, 2012, at 9:22 PM, Brian Mitchell wrote: Has there been any discussion around BigCouch integration strategies? It seems like it would fit the bill for the next undertaking on the general couch side. Does anyone from Cloudant have a suggestion for the timeline here? There's been a lot of discussion in #couchdb and #couchdb-dev but little on the ml. I'm not sure about timeline. There seems to be a lot of issues, most of them minor technical ones (the type that readily bog down once more than 3 people get involved). BigCouch embeds couchdb and was architected to be the clustering layer that couchdb lacks, so in that sense I think we're in pretty good shape. Ideally we'd have one common code base but it may be that some configuration of components is done at build time, perhaps driven by 3 types, mobile, single instance, and clustered Does it make sense to anyone to think of this in the opposite direction, .i.e. upgrade/enhance BigCouch to the latest couchdb and then call that couchdb 2.0? Any other work from mobile builds and the like might be interesting to support. Were there any interesting changes to pull in from the mobile and embedded device ports? Also, I'm not sure where the view engine refactoring work ended up… I'll look at JIRA but maybe there are specific follow-ups to be made here. Finally, for our included javascript libraries and Futon runtime, are we going to replace or update anything here? I'd imagine a newer jQuery could be an advantage for those doing CouchApps. Brian. On Tuesday, February 14, 2012 at 7:28 , Noah Slater wrote: So far we have: https://issues.apache.org/jira/browse/COUCHDB-1410
Re: Roadmap for the 1.3.0 release
On Wed, Feb 15, 2012 at 2:13 PM, Bob Dionne dio...@dionne-associates.com wrote: On Feb 14, 2012, at 9:22 PM, Brian Mitchell wrote: Has there been any discussion around BigCouch integration strategies? It seems like it would fit the bill for the next undertaking on the general couch side. Does anyone from Cloudant have a suggestion for the timeline here? There's been a lot of discussion in #couchdb and #couchdb-dev but little on the ml. I'm not sure about timeline. There seems to be a lot of issues, most of them minor technical ones (the type that readily bog down once more than 3 people get involved). BigCouch embeds couchdb and was architected to be the clustering layer that couchdb lacks, so in that sense I think we're in pretty good shape. Ideally we'd have one common code base but it may be that some configuration of components is done at build time, perhaps driven by 3 types, mobile, single instance, and clustered Does it make sense to anyone to think of this in the opposite direction, .i.e. upgrade/enhance BigCouch to the latest couchdb and then call that couchdb 2.0? I was thinking that having a single instance that you could upgrade as a cluster member with just a configuration swicth would be a better plan. With smart rebalancing I could create cluster really dynamically. I understand that currently it will be hard technically to do that since couch embedded in bigcouch has been modified to get some infos from the cluster (like the design doc, validate func ...) but it's still possible. Not sure if it should happen first, but I really wish we follow this way rather than creating different instances types. - benoît
Re: Roadmap for the 1.3.0 release
On Wed, Feb 15, 2012 at 11:48 AM, Jan Lehnardt j...@apache.org wrote: On Feb 15, 2012, at 03:22 , Brian Mitchell wrote: Any other work from mobile builds and the like might be interesting to support. Were there any interesting changes to pull in from the mobile and embedded device ports? Also, I'm not sure where the view engine refactoring work ended up… I'll look at JIRA but maybe there are specific follow-ups to be made here. The biggest one IIRC was the use of emonk instead of an external Spidermonkey for iOS. Maybe we can make that a compile option like ./configure --enable-emonk-view-server so people can choose which one to run. There is some work started about that in the refuge project using old work done by @davisp used in couchbase and my emonk_helper. I will provide patches asap. Finally, for our included javascript libraries and Futon runtime, are we going to replace or update anything here? I'd imagine a newer jQuery could be an advantage for those doing CouchApps. Good that you bring this up. There's also other upstream dependencies that we should look at updating and we don't have a process for that, it usually happens whenever a developer feels like it. I haven't thought about this much yet, but maybe update all dependencies could be a todo before a release branch is created? +1 for a ticket :) Also include support for mobile support in futon would be fine. (but i think we should only depends on jquery rather than including another framework) . - benoit
Re: Roadmap for the 1.3.0 release
That sounds really neat, a number of folks have asked for such a thing. Right, the ddocs, validation funs, etc.. currently aren't stored globally, which requires clustered calls to retrieve them On Feb 15, 2012, at 8:21 AM, Benoit Chesneau wrote: On Wed, Feb 15, 2012 at 2:13 PM, Bob Dionne dio...@dionne-associates.com wrote: On Feb 14, 2012, at 9:22 PM, Brian Mitchell wrote: Has there been any discussion around BigCouch integration strategies? It seems like it would fit the bill for the next undertaking on the general couch side. Does anyone from Cloudant have a suggestion for the timeline here? There's been a lot of discussion in #couchdb and #couchdb-dev but little on the ml. I'm not sure about timeline. There seems to be a lot of issues, most of them minor technical ones (the type that readily bog down once more than 3 people get involved). BigCouch embeds couchdb and was architected to be the clustering layer that couchdb lacks, so in that sense I think we're in pretty good shape. Ideally we'd have one common code base but it may be that some configuration of components is done at build time, perhaps driven by 3 types, mobile, single instance, and clustered Does it make sense to anyone to think of this in the opposite direction, .i.e. upgrade/enhance BigCouch to the latest couchdb and then call that couchdb 2.0? I was thinking that having a single instance that you could upgrade as a cluster member with just a configuration swicth would be a better plan. With smart rebalancing I could create cluster really dynamically. I understand that currently it will be hard technically to do that since couch embedded in bigcouch has been modified to get some infos from the cluster (like the design doc, validate func ...) but it's still possible. Not sure if it should happen first, but I really wish we follow this way rather than creating different instances types. - benoît
Re: Roadmap for the 1.3.0 release
So far we have: https://issues.apache.org/jira/browse/COUCHDB-1410 On Tue, Feb 14, 2012 at 12:16 PM, Noah Slater nsla...@tumbolia.org wrote: Devs, Please use this thread to discuss roadmap items for the 1.3.0 release. The current hot topic seems to be number handling. We'd like to formalise, improve, and document how we handle numbers. What else? Thanks, N
Re: Roadmap for the 1.3.0 release
Has there been any discussion around BigCouch integration strategies? It seems like it would fit the bill for the next undertaking on the general couch side. Does anyone from Cloudant have a suggestion for the timeline here? Any other work from mobile builds and the like might be interesting to support. Were there any interesting changes to pull in from the mobile and embedded device ports? Also, I'm not sure where the view engine refactoring work ended up… I'll look at JIRA but maybe there are specific follow-ups to be made here. Finally, for our included javascript libraries and Futon runtime, are we going to replace or update anything here? I'd imagine a newer jQuery could be an advantage for those doing CouchApps. Brian. On Tuesday, February 14, 2012 at 7:28 , Noah Slater wrote: So far we have: https://issues.apache.org/jira/browse/COUCHDB-1410
Re: roadmap
On Thu, Feb 10, 2011 at 11:44 AM, Jan Lehnardt j...@apache.org wrote: On 10 Feb 2011, at 17:29, Gabriel Farrell wrote: On Tue, Feb 8, 2011 at 11:37 AM, Robert Newson robert.new...@gmail.com wrote: You're absolutely right. 1.0.2 was ready to go quite some time ago but several bugs were found as we were releasing. We decided, as a team, that we couldn't ship with the bugs that were found, so we elected to fix them and delay the release. I think that was the right decision. We should only release when we feel the software is ready, not when some ultimately arbitrary day looms. I completely agree here: there should be no arbitrary release dates. However, I'm also in favor of increasing the frequency of minor point releases. Node.js is really inspiring in this area, with new minor point releases coming out every week or so (ooh, and 0.4.0 just got released 6 hours ago!). I know the process for an Apache project has more overhead, we don't have a BDFL, and the older community may not appreciate a release cycle that's quite so frantic (interpret that as you like), but there's still something to be learned from the rapid development and enthusiastic community around Node. Yup, I totally agree with node showing amazing momentum. But they do have the luxury of being able to break backwards compatibility with any release, really, and we don't have that :) — I think the 1.0.2 time frame was an outlier and that we are in pretty good shape to push maintenance releases quickly, if needed. — Now we only need to demonstrate that :) Ryan's been a bit more careful about backwards compatibility lately with the move to stable even and unstable odd branch releases. Backwards compatibility is a concern for any project as it matures. So, yeah, more agreement -- let's keep that concern in mind as we quickly push releases. :) Cheers Jan -- B. On Tue, Feb 8, 2011 at 4:32 PM, Noah Slater nsla...@apache.org wrote: On 8 Feb 2011, at 16:14, Dirkjan Ochtman wrote: Still, the problem I have is that it seems like there is a tendency to make releases large; it seems like there's little control against devs wanting to add their one more thing. Particularly for bugfix releases; from 1.0.1 it took almost 6 months for 1.0.2 to get released. In between, there were a little under 100 revisions on the 1.0.x branch, presumably most of those fixing bugs users could actually run into. It seems valuable to me if the community could have gotten some of these fixes sooner. I need someone else to weigh in on this, but I believe the reason was because of a few critical bugs that were being worked on. And not, as you suggest, because we were suffering from a Just One More Thing problem. I'd really need Jan or Chris to comment though as I use them as a conduit for judging this stuff.
Re: roadmap
On Tue, Feb 8, 2011 at 11:37 AM, Robert Newson robert.new...@gmail.com wrote: You're absolutely right. 1.0.2 was ready to go quite some time ago but several bugs were found as we were releasing. We decided, as a team, that we couldn't ship with the bugs that were found, so we elected to fix them and delay the release. I think that was the right decision. We should only release when we feel the software is ready, not when some ultimately arbitrary day looms. I completely agree here: there should be no arbitrary release dates. However, I'm also in favor of increasing the frequency of minor point releases. Node.js is really inspiring in this area, with new minor point releases coming out every week or so (ooh, and 0.4.0 just got released 6 hours ago!). I know the process for an Apache project has more overhead, we don't have a BDFL, and the older community may not appreciate a release cycle that's quite so frantic (interpret that as you like), but there's still something to be learned from the rapid development and enthusiastic community around Node. B. On Tue, Feb 8, 2011 at 4:32 PM, Noah Slater nsla...@apache.org wrote: On 8 Feb 2011, at 16:14, Dirkjan Ochtman wrote: Still, the problem I have is that it seems like there is a tendency to make releases large; it seems like there's little control against devs wanting to add their one more thing. Particularly for bugfix releases; from 1.0.1 it took almost 6 months for 1.0.2 to get released. In between, there were a little under 100 revisions on the 1.0.x branch, presumably most of those fixing bugs users could actually run into. It seems valuable to me if the community could have gotten some of these fixes sooner. I need someone else to weigh in on this, but I believe the reason was because of a few critical bugs that were being worked on. And not, as you suggest, because we were suffering from a Just One More Thing problem. I'd really need Jan or Chris to comment though as I use them as a conduit for judging this stuff.
Re: roadmap
On 10 Feb 2011, at 17:29, Gabriel Farrell wrote: On Tue, Feb 8, 2011 at 11:37 AM, Robert Newson robert.new...@gmail.com wrote: You're absolutely right. 1.0.2 was ready to go quite some time ago but several bugs were found as we were releasing. We decided, as a team, that we couldn't ship with the bugs that were found, so we elected to fix them and delay the release. I think that was the right decision. We should only release when we feel the software is ready, not when some ultimately arbitrary day looms. I completely agree here: there should be no arbitrary release dates. However, I'm also in favor of increasing the frequency of minor point releases. Node.js is really inspiring in this area, with new minor point releases coming out every week or so (ooh, and 0.4.0 just got released 6 hours ago!). I know the process for an Apache project has more overhead, we don't have a BDFL, and the older community may not appreciate a release cycle that's quite so frantic (interpret that as you like), but there's still something to be learned from the rapid development and enthusiastic community around Node. Yup, I totally agree with node showing amazing momentum. But they do have the luxury of being able to break backwards compatibility with any release, really, and we don't have that :) — I think the 1.0.2 time frame was an outlier and that we are in pretty good shape to push maintenance releases quickly, if needed. — Now we only need to demonstrate that :) Cheers Jan -- B. On Tue, Feb 8, 2011 at 4:32 PM, Noah Slater nsla...@apache.org wrote: On 8 Feb 2011, at 16:14, Dirkjan Ochtman wrote: Still, the problem I have is that it seems like there is a tendency to make releases large; it seems like there's little control against devs wanting to add their one more thing. Particularly for bugfix releases; from 1.0.1 it took almost 6 months for 1.0.2 to get released. In between, there were a little under 100 revisions on the 1.0.x branch, presumably most of those fixing bugs users could actually run into. It seems valuable to me if the community could have gotten some of these fixes sooner. I need someone else to weigh in on this, but I believe the reason was because of a few critical bugs that were being worked on. And not, as you suggest, because we were suffering from a Just One More Thing problem. I'd really need Jan or Chris to comment though as I use them as a conduit for judging this stuff.
Re: roadmap
Hi, On 9 Feb 2011, at 08:29, Benoit Chesneau wrote: It's not clear to me that couchdb would benefit by bundling clustering and search. Lucene has an approach that might work for us, namely where there's an explicit core project, with a number of supplementary parts. Releases are aligned for all of them, but the separation is pretty clear. I agree, wasn't speaking to make a bundle, but we could have official plugins supported by the apache project like hadoop or solr have. Something like Apache Extras might be of interest (though afaict it's really just a branded google code). https://blogs.apache.org/foundation/entry/the_apache_software_foundation_launches At the very least I think specific couchapps (implementations of wikis, blogs etc) should be encouraged to go there for hosting. I'm not sure couchdb-lucene, couchapp itself or geocouch (and other future plugins) are entirely suitable, but I guess it depends on how apache extras evolve over time. Cheers Simon
Re: roadmap
Hi, - a really good plugin story (geocouch, lucene search, easy to compile against couchdb sources) I don't know how possible this is (I can think of a number of issues without trying) but having these plugins uploadable into a database in a similar manner to views would be super sweet. It would mean you could find a vanilla couch hosting company and tailor it to your databases needs directly over HTTP. Cheers Simon
Re: roadmap
Randall Leeds wrote: My priorities are: - a really good embedding story (android, desktop apps, couchbase, etc ...) ... - a really good build story (particularly android, windows) Having recently worked with the Android port (see build instructions on wiki -- soon to be updated improved) we might be able to help with these things. We're releasing a collaborative mobile data collection product in the next few weeks that uses Open Data Kit technology and CouchDB for persistence and synchronization. More to come on this, obviously. Cheers, Matt -- Matt Adams Radical Dynamic www.radicaldynamic.com
Re: roadmap
On Tue, Feb 8, 2011 at 10:02 PM, Noah Slater nsla...@apache.org wrote: On 8 Feb 2011, at 20:53, Benoit Chesneau wrote: What is it supposed to mean ? A roadmap is a a detailed plan to guide progress toward a goal . Why couldn't we define goals ? I think Jan's point is that we use the JIRA roadmap as an advisory only, and never state that we are committing to it. That means that if I create a ticket for CouchDB to be able to read and send email, it doesn't hold-up the project. Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can. — Jamie Zawinski hum, ok yes a ticket is only a ticket. Although I was under the impression that we can give some priority to the tickets or close them if they doesn't enter in project goals.
Re: roadmap
On Tue, Feb 8, 2011 at 5:19 PM, Robert Newson robert.new...@gmail.com wrote: We do need to improve at releases, though I think we can all agree that 1.0.2 was just a particularly difficult one, pretty atypical. As for roadmap, I can see a need to revisit it. I'm not sure what should be on it, I suspect everyone has their pet feature or ten to add. So, I'm wary about fragmenting/diluting what we have by trying to put too much inside couchdb itself. Instead, I think Benoit is on to something by talking about pluggability. At the very least, couchdb can make it easy to embed and extend it in ways that are minimally likely to be broken by us in future releases. It's not clear to me that couchdb would benefit by bundling clustering and search. Lucene has an approach that might work for us, namely where there's an explicit core project, with a number of supplementary parts. Releases are aligned for all of them, but the separation is pretty clear. I agree, wasn't speaking to make a bundle, but we could have official plugins supported by the apache project like hadoop or solr have.
Re: roadmap
On Tue, Feb 8, 2011 at 6:42 PM, Peter Nolan peterwno...@gmail.com wrote: I would like a more formal direction on the future of couchapps. I am unsure how couchapps will proceed, but one thing i would like to see is the ability of your basic internet users to be able to form their own couchapps by easily integrating other couchapps into their databases, modifying to their content (most 'simple' users are content with just basic color and image changes). For example, a user may 'see' something cool on someones elses site/couch and would like to copy it to theirs. This 'copying' should be as easy as humanly possible. Any complications will only hinder the adoption of couchapps to the public in my opinion. Ideally this 'copying' should be done with a simple push of a button. This leads me to another need couchapps (and they're couchAPERS) must discuss - Standardization. I imagine it will be 'hard' if we don't begin to standardize some of the aspects of couchapps. For example - blog posts. There should be a standard way of 'inputting' blog posts into ones of couch, such that others may easily 'replicate' or 'pull from' peoples blogs and have them appear on their couches. Of course this 'standardization' should also not limit ones ability to change the code. There is a discussion started on the CouchApp mailing-list about that. I also think a couchapp spec is welcome, so we can develop compatible engine and clients. At the end we should only have to choose the hosting or the tool we like, period.
Re: roadmap
On 9 Feb 2011, at 09:26, Benoit Chesneau wrote: On Tue, Feb 8, 2011 at 10:02 PM, Noah Slater nsla...@apache.org wrote: On 8 Feb 2011, at 20:53, Benoit Chesneau wrote: What is it supposed to mean ? A roadmap is a a detailed plan to guide progress toward a goal . Why couldn't we define goals ? I think Jan's point is that we use the JIRA roadmap as an advisory only, and never state that we are committing to it. That means that if I create a ticket for CouchDB to be able to read and send email, it doesn't hold-up the project. Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can. — Jamie Zawinski hum, ok yes a ticket is only a ticket. Although I was under the impression that we can give some priority to the tickets or close them if they doesn't enter in project goals. All I'm saying is that just creating a ticket and assigning it a version number won't make us commit to delivering that. Of course we should organise all tickets into versions and come up with a sensible batch of stuff to work towards for every release. Cheers Jan --
Re: roadmap
We should be clear that just because Jira has that helpful 'Roadmap' panel, doesn't mean it's our official roadmap. It really isn't, though that is how Jira would like us to do things. I can't speak for everyone, but Jira, to me, is just a tool, it's not the boss of me. B. On 9 February 2011 12:30, Jan Lehnardt j...@apache.org wrote: On 9 Feb 2011, at 09:26, Benoit Chesneau wrote: On Tue, Feb 8, 2011 at 10:02 PM, Noah Slater nsla...@apache.org wrote: On 8 Feb 2011, at 20:53, Benoit Chesneau wrote: What is it supposed to mean ? A roadmap is a a detailed plan to guide progress toward a goal . Why couldn't we define goals ? I think Jan's point is that we use the JIRA roadmap as an advisory only, and never state that we are committing to it. That means that if I create a ticket for CouchDB to be able to read and send email, it doesn't hold-up the project. Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can. — Jamie Zawinski hum, ok yes a ticket is only a ticket. Although I was under the impression that we can give some priority to the tickets or close them if they doesn't enter in project goals. All I'm saying is that just creating a ticket and assigning it a version number won't make us commit to delivering that. Of course we should organise all tickets into versions and come up with a sensible batch of stuff to work towards for every release. Cheers Jan --
Re: roadmap
What do you mean by official here? On 9 Feb 2011, at 12:39, Robert Newson wrote: We should be clear that just because Jira has that helpful 'Roadmap' panel, doesn't mean it's our official roadmap. It really isn't, though that is how Jira would like us to do things. I can't speak for everyone, but Jira, to me, is just a tool, it's not the boss of me. B. On 9 February 2011 12:30, Jan Lehnardt j...@apache.org wrote: On 9 Feb 2011, at 09:26, Benoit Chesneau wrote: On Tue, Feb 8, 2011 at 10:02 PM, Noah Slater nsla...@apache.org wrote: On 8 Feb 2011, at 20:53, Benoit Chesneau wrote: What is it supposed to mean ? A roadmap is a a detailed plan to guide progress toward a goal . Why couldn't we define goals ? I think Jan's point is that we use the JIRA roadmap as an advisory only, and never state that we are committing to it. That means that if I create a ticket for CouchDB to be able to read and send email, it doesn't hold-up the project. Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can. — Jamie Zawinski hum, ok yes a ticket is only a ticket. Although I was under the impression that we can give some priority to the tickets or close them if they doesn't enter in project goals. All I'm saying is that just creating a ticket and assigning it a version number won't make us commit to delivering that. Of course we should organise all tickets into versions and come up with a sensible batch of stuff to work towards for every release. Cheers Jan --
Re: roadmap
On Tue, Feb 8, 2011 at 10:24, Benoit Chesneau bchesn...@gmail.com wrote: ... and you ? Most of all, I want a better schedule/insight into the release process. Even when reading the dev list, it's completely unclear when I might expect the next release or what the blockers are. Releases seem to just keep slipping, or maybe releasing isn't a very big priority, at least that's what it looks like from the outside. For the rest, I would like more documentation. The CouchOne API documentation is a step in the right direction, having good docs about the query server protocol would also be very helpful. Yes, I realize I can read the couchjs JS code, but that's not a substitute for good docs. The new Futon that Mikeal was working on would be nice to have. As for more pie-in-the-sky things, it would be nice to have more efficient view indexing (using multiple processes, for example -- map/reduce should enable that, right?) and an official way to do full-text search. Cheers, Dirkjan
Re: roadmap
On Tue, Feb 8, 2011 at 11:24 AM, Benoit Chesneau bchesn...@gmail.comwrote: ... and you ? - erlang api / plugin support - easier build process on Windows -juhani
Re: roadmap
+1 full-text search +1 documentation On Tue, Feb 8, 2011 at 4:07 AM, Juhani Ränkimies juh...@juranki.com wrote: On Tue, Feb 8, 2011 at 11:24 AM, Benoit Chesneau bchesn...@gmail.comwrote: ... and you ? - erlang api / plugin support - easier build process on Windows -juhani
Re: roadmap
On Tue, Feb 8, 2011 at 10:33 AM, Dirkjan Ochtman dirk...@ochtman.nl wrote: On Tue, Feb 8, 2011 at 10:24, Benoit Chesneau bchesn...@gmail.com wrote: ... and you ? Most of all, I want a better schedule/insight into the release process. Even when reading the dev list, it's completely unclear when I might expect the next release or what the blockers are. Releases seem to just keep slipping, or maybe releasing isn't a very big priority, at least that's what it looks like from the outside. For the rest, I would like more documentation. The CouchOne API documentation is a step in the right direction, having good docs about the query server protocol would also be very helpful. Yes, I realize I can read the couchjs JS code, but that's not a substitute for good docs. The new Futon that Mikeal was working on would be nice to have. As for more pie-in-the-sky things, it would be nice to have more efficient view indexing (using multiple processes, for example -- map/reduce should enable that, right?) and an official way to do full-text search. Cheers, Dirkjan +1 to the above
Re: roadmap
On Tue, Feb 8, 2011 at 16:57, Noah Slater nsla...@apache.org wrote: For step 2, we follow the roadmap that is generated in JIRA. That is, the roadmap is constantly maintained through ticket work and ticket maintenance. A link to this is even included prominently on the project homepage. We have found that maintaining a HTML roadmap separately to JIRA is too much trouble. Or, at least, we could infer that from the fact it was never updated by anyone. The roadmap (as a JIRA view) is effectively discussed on this list, and then codified through the tickets. I think that's Good Enough. And if it's not, then maybe we need to be smarter about how we use JIRA. Okay, the roadmap is somewhat useful, and I wasn't aware of it. Still, the problem I have is that it seems like there is a tendency to make releases large; it seems like there's little control against devs wanting to add their one more thing. Particularly for bugfix releases; from 1.0.1 it took almost 6 months for 1.0.2 to get released. In between, there were a little under 100 revisions on the 1.0.x branch, presumably most of those fixing bugs users could actually run into. It seems valuable to me if the community could have gotten some of these fixes sooner. I do realize the Apache release process is rather heavyweight, and that CouchDB only recently got a second RM, but it would still be nice if it was possible to have more timely releases. (And I also agree with till's sentiment about roadmaps.) Cheers, Dirkjan
Re: roadmap
We do need to improve at releases, though I think we can all agree that 1.0.2 was just a particularly difficult one, pretty atypical. As for roadmap, I can see a need to revisit it. I'm not sure what should be on it, I suspect everyone has their pet feature or ten to add. So, I'm wary about fragmenting/diluting what we have by trying to put too much inside couchdb itself. Instead, I think Benoit is on to something by talking about pluggability. At the very least, couchdb can make it easy to embed and extend it in ways that are minimally likely to be broken by us in future releases. It's not clear to me that couchdb would benefit by bundling clustering and search. Lucene has an approach that might work for us, namely where there's an explicit core project, with a number of supplementary parts. Releases are aligned for all of them, but the separation is pretty clear. B. On Tue, Feb 8, 2011 at 4:14 PM, Dirkjan Ochtman dirk...@ochtman.nl wrote: On Tue, Feb 8, 2011 at 16:57, Noah Slater nsla...@apache.org wrote: For step 2, we follow the roadmap that is generated in JIRA. That is, the roadmap is constantly maintained through ticket work and ticket maintenance. A link to this is even included prominently on the project homepage. We have found that maintaining a HTML roadmap separately to JIRA is too much trouble. Or, at least, we could infer that from the fact it was never updated by anyone. The roadmap (as a JIRA view) is effectively discussed on this list, and then codified through the tickets. I think that's Good Enough. And if it's not, then maybe we need to be smarter about how we use JIRA. Okay, the roadmap is somewhat useful, and I wasn't aware of it. Still, the problem I have is that it seems like there is a tendency to make releases large; it seems like there's little control against devs wanting to add their one more thing. Particularly for bugfix releases; from 1.0.1 it took almost 6 months for 1.0.2 to get released. In between, there were a little under 100 revisions on the 1.0.x branch, presumably most of those fixing bugs users could actually run into. It seems valuable to me if the community could have gotten some of these fixes sooner. I do realize the Apache release process is rather heavyweight, and that CouchDB only recently got a second RM, but it would still be nice if it was possible to have more timely releases. (And I also agree with till's sentiment about roadmaps.) Cheers, Dirkjan
Re: roadmap
On 8 Feb 2011, at 16:08, till wrote: IMHO a roadmap is defined by more than there's a new jira issue, we need to fix it with the next release. I think you're misunderstanding me. In JIRA, you can pin tickets to release versions. This is a perfectly good way of constructing a roadmap. And if the thing you want isn't in JIRA as a ticket to pin to a release version, that's a good opportunity for you to create one.
Re: roadmap
On 8 Feb 2011, at 16:14, Dirkjan Ochtman wrote: Still, the problem I have is that it seems like there is a tendency to make releases large; it seems like there's little control against devs wanting to add their one more thing. Particularly for bugfix releases; from 1.0.1 it took almost 6 months for 1.0.2 to get released. In between, there were a little under 100 revisions on the 1.0.x branch, presumably most of those fixing bugs users could actually run into. It seems valuable to me if the community could have gotten some of these fixes sooner. I need someone else to weigh in on this, but I believe the reason was because of a few critical bugs that were being worked on. And not, as you suggest, because we were suffering from a Just One More Thing problem. I'd really need Jan or Chris to comment though as I use them as a conduit for judging this stuff.
Re: roadmap
You're absolutely right. 1.0.2 was ready to go quite some time ago but several bugs were found as we were releasing. We decided, as a team, that we couldn't ship with the bugs that were found, so we elected to fix them and delay the release. I think that was the right decision. We should only release when we feel the software is ready, not when some ultimately arbitrary day looms. B. On Tue, Feb 8, 2011 at 4:32 PM, Noah Slater nsla...@apache.org wrote: On 8 Feb 2011, at 16:14, Dirkjan Ochtman wrote: Still, the problem I have is that it seems like there is a tendency to make releases large; it seems like there's little control against devs wanting to add their one more thing. Particularly for bugfix releases; from 1.0.1 it took almost 6 months for 1.0.2 to get released. In between, there were a little under 100 revisions on the 1.0.x branch, presumably most of those fixing bugs users could actually run into. It seems valuable to me if the community could have gotten some of these fixes sooner. I need someone else to weigh in on this, but I believe the reason was because of a few critical bugs that were being worked on. And not, as you suggest, because we were suffering from a Just One More Thing problem. I'd really need Jan or Chris to comment though as I use them as a conduit for judging this stuff.
Re: roadmap
I would like a more formal direction on the future of couchapps. I am unsure how couchapps will proceed, but one thing i would like to see is the ability of your basic internet users to be able to form their own couchapps by easily integrating other couchapps into their databases, modifying to their content (most 'simple' users are content with just basic color and image changes). For example, a user may 'see' something cool on someones elses site/couch and would like to copy it to theirs. This 'copying' should be as easy as humanly possible. Any complications will only hinder the adoption of couchapps to the public in my opinion. Ideally this 'copying' should be done with a simple push of a button. This leads me to another need couchapps (and they're couchAPERS) must discuss - Standardization. I imagine it will be 'hard' if we don't begin to standardize some of the aspects of couchapps. For example - blog posts. There should be a standard way of 'inputting' blog posts into ones of couch, such that others may easily 'replicate' or 'pull from' peoples blogs and have them appear on their couches. Of course this 'standardization' should also not limit ones ability to change the code. Same thing goes for messaging. It would be nice if we would start forming a standardized 'messaging' system such that messages posted on users A blog will (to steal from Jan) automagically appear on users B blog. (think I.R.C. amongst couches). In short, i believe the future of couchapps depends upon getting your mother to be able to make her own (and feel like she made her own). -Pete
Re: roadmap
I'm going to chime in here and say that improved build support for Android would be terrific. There are already some patches available for this and it would be nice to see them included in the official releases. Hopefully this doesn't equate with asking for ponies. I'd be happy to assist with testing or anything else that would help in this regard. Cheers, Matt -- Matt Adams Radical Dynamic www.radicaldynamic.com
Re: roadmap
On Tue, Feb 8, 2011 at 01:24, Benoit Chesneau bchesn...@gmail.com wrote: Hi all, With the announce of another /couchdb fork/project embedding couchdb/ I think it's the perfect time to define ourself for next releases. What will be CouchDB 1.2 or 2.0, what do we target, what is our timeline. What is the CouchDB core that will be used by other projects basically. Having a defined timeline, ie defined deadlines for freeze, release, ... is important in this context. It means that releases don't depend on external factors: we, the Apache CouchDB project follows its own agenda. There are other reasons for that of course. So can we try to define this time a good old roadmap? What will be the next couchdb? For me: - Plugin support - improved CouchApp engine - Improve possibilities to integrate CouchDB in other projects - Clean API, so couchdb features can be called more easily in other erlang programs or plugins Other feature I wish: - Official Apache CouchDB clustering layer plug - Official Search Plugin based on Apache Lucene that can be used like riak search or cloudant search ... and you ? - benoît I agree mostly with Benoit. My priorities are: - a really good embedding story (android, desktop apps, couchbase, etc ...) - a really good plugin story (geocouch, lucene search, easy to compile against couchdb sources) - a really good build story (particularly android, windows) - clean internal APIs -Randall
Re: roadmap
On 8 Feb 2011, at 17:32, Noah Slater wrote: On 8 Feb 2011, at 16:14, Dirkjan Ochtman wrote: Still, the problem I have is that it seems like there is a tendency to make releases large; it seems like there's little control against devs wanting to add their one more thing. Particularly for bugfix releases; from 1.0.1 it took almost 6 months for 1.0.2 to get released. In between, there were a little under 100 revisions on the 1.0.x branch, presumably most of those fixing bugs users could actually run into. It seems valuable to me if the community could have gotten some of these fixes sooner. I need someone else to weigh in on this, but I believe the reason was because of a few critical bugs that were being worked on. And not, as you suggest, because we were suffering from a Just One More Thing problem. I'd really need Jan or Chris to comment though as I use them as a conduit for judging this stuff. Robert already confirmed this, but I'd like to point out that Noah's analysis is apt. -- As for the suggestions for more transparency regarding what new features are being worked on and when do they land in which version I agree that we can do better and I'll take on doing some of the legwork here. I also like the proposed features, but I don't think committing to ship pony-features without seeing any code is a good idea — just to paint an extreme, so far nobody suggested that. Cheers Jan --
Re: roadmap
On Tue, Feb 8, 2011 at 9:40 PM, Jan Lehnardt j...@apache.org wrote: On 8 Feb 2011, at 17:32, Noah Slater wrote: On 8 Feb 2011, at 16:14, Dirkjan Ochtman wrote: Still, the problem I have is that it seems like there is a tendency to make releases large; it seems like there's little control against devs wanting to add their one more thing. Particularly for bugfix releases; from 1.0.1 it took almost 6 months for 1.0.2 to get released. In between, there were a little under 100 revisions on the 1.0.x branch, presumably most of those fixing bugs users could actually run into. It seems valuable to me if the community could have gotten some of these fixes sooner. I need someone else to weigh in on this, but I believe the reason was because of a few critical bugs that were being worked on. And not, as you suggest, because we were suffering from a Just One More Thing problem. I'd really need Jan or Chris to comment though as I use them as a conduit for judging this stuff. Robert already confirmed this, but I'd like to point out that Noah's analysis is apt. -- As for the suggestions for more transparency regarding what new features are being worked on and when do they land in which version I agree that we can do better and I'll take on doing some of the legwork here. I also like the proposed features, but I don't think committing to ship pony-features without seeing any code is a good idea — just to paint an extreme, so far nobody suggested that. Cheers Jan -- What is it supposed to mean ? A roadmap is a a detailed plan to guide progress toward a goal . Why couldn't we define goals ? - benoît
Re: roadmap
On 8 Feb 2011, at 20:53, Benoit Chesneau wrote: What is it supposed to mean ? A roadmap is a a detailed plan to guide progress toward a goal . Why couldn't we define goals ? I think Jan's point is that we use the JIRA roadmap as an advisory only, and never state that we are committing to it. That means that if I create a ticket for CouchDB to be able to read and send email, it doesn't hold-up the project. Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can. — Jamie Zawinski
Re: Roadmap 0.11?
On Tue, Feb 9, 2010 at 3:51 AM, Mark Hammond skippy.hamm...@gmail.com wrote: On 9/02/2010 7:05 AM, Per Ejeklint wrote: And will official windows builds be available with it? I've been putting together the official binaries for 0.10 and will be making some just as official for 0.11. I expect that for 0.11 and later though, the builds I put together will be listed on the main download page making them really official instead of just official :) http://people.apache.org/~mhammond/dist/ Mark. There Is an old threads about what will/could be in 0.11. I think it would be good to have a roadmap published. It's definitely needed for non developers people that want to have an idea of futures developments and orientation. - benoît
Re: roadmap 0.11/1.0
On Sun, Nov 1, 2009 at 8:15 PM, Chris Anderson jch...@apache.org wrote: On Sun, Nov 1, 2009 at 11:00 AM, Suhail Ahmed suhail...@gmail.com wrote: Hi, Any chance of seeing native erlang RPC protocol in 11 or soon there after? This may be part of the Cloudant clustering codebase. I can't speak for them, but from what I've heard it does inter-node communication in a native Erlang way. You could probably use this interface as a primary client interface as well, but what do I know? :) Chris cloudant is a clustered couchdb or a cluster system over couchdb ? - benoît
Re: roadmap 0.11/1.0
On Nov 2, 2009, at 4:19 AM, Benoit Chesneau wrote:
On Sun, Nov 1, 2009 at 8:15 PM, Chris Anderson jch...@apache.org
wrote:
On Sun, Nov 1, 2009 at 11:00 AM, Suhail Ahmed suhail...@gmail.com
wrote:
Hi,
Any chance of seeing native erlang RPC protocol in 11 or soon
there after?
This may be part of the Cloudant clustering codebase. I can't speak
for them, but from what I've heard it does inter-node communication
in
a native Erlang way. You could probably use this interface as a
primary client interface as well, but what do I know? :)
Chris
cloudant is a clustered couchdb or a cluster system over couchdb ?
- benoît
Hi Benoit, we (Cloudant) are developing a clustered CouchDB; that is,
the ability to shard a database across a variable number of CouchDB
instances and have any of those instances handle any HTTP API
request. The instances communicate with each other using distributed
Erlang. The distribution system is Dynamo-flavored consistent hashing
(actually borrowing significantly from dynomite), and view results are
merged and re-reduced at query time. We're still working hard on a
few technical issues (in particular, generalizing single-instance
update sequences to a distributed notion of this-happened-first for
a proper _changes feed), but I think most of the CouchDB 0.10.0 API is
in pretty good shape. We'll be releasing the source code soon; I'm
afraid I can't be any more specific at the moment.
As far as whether the code has the makings of an Erlang remote client
library well, yes and no. It turns out the CouchDB CRUD
operations mostly just work when you open the DB using an RPC call
to the remote node. Something like
gen_server:call({couch_server, Node}, {open, DbName, Options})
or even
rpc:call(Node, couch_db, open, [DbName, Options])
Once you get a #db{} record filled with remote Pids, you can use it
just like you would a local one. Pretty nice, that. It means that
hovercraft and CouchDB need relatively few adjustments in order to run
in different VMs.
Cheers, Adam
Re: roadmap 0.11/1.0
On Mon, Nov 2, 2009 at 6:59 PM, Adam Kocoloski kocol...@apache.org wrote:
On Nov 2, 2009, at 4:19 AM, Benoit Chesneau wrote:
On Sun, Nov 1, 2009 at 8:15 PM, Chris Anderson jch...@apache.org wrote:
On Sun, Nov 1, 2009 at 11:00 AM, Suhail Ahmed suhail...@gmail.com
wrote:
Hi,
Any chance of seeing native erlang RPC protocol in 11 or soon there
after?
This may be part of the Cloudant clustering codebase. I can't speak
for them, but from what I've heard it does inter-node communication in
a native Erlang way. You could probably use this interface as a
primary client interface as well, but what do I know? :)
Chris
cloudant is a clustered couchdb or a cluster system over couchdb ?
- benoît
Hi Benoit, we (Cloudant) are developing a clustered CouchDB; that is, the
ability to shard a database across a variable number of CouchDB instances
and have any of those instances handle any HTTP API request. The instances
communicate with each other using distributed Erlang. The distribution
system is Dynamo-flavored consistent hashing (actually borrowing
significantly from dynomite), and view results are merged and re-reduced at
query time. We're still working hard on a few technical issues (in
particular, generalizing single-instance update sequences to a distributed
notion of this-happened-first for a proper _changes feed), but I think
most of the CouchDB 0.10.0 API is in pretty good shape. We'll be releasing
the source code soon; I'm afraid I can't be any more specific at the
moment.
That's already a lot and really good to know :) I was thinking more
and more to it since i have special needs in term of storage for a
project. Thanks a lot to make it opensource.
As far as whether the code has the makings of an Erlang remote client
library well, yes and no. It turns out the CouchDB CRUD operations
mostly just work when you open the DB using an RPC call to the remote
node. Something like
gen_server:call({couch_server, Node}, {open, DbName, Options})
or even
rpc:call(Node, couch_db, open, [DbName, Options])
Once you get a #db{} record filled with remote Pids, you can use it just
like you would a local one. Pretty nice, that. It means that hovercraft and
CouchDB need relatively few adjustments in order to run in different VMs.
About that I had done some works on rpc calls based on hovercraft.
Code is here and wasn't finished :
http://github.com/benoitc/couchdb/blob/rpc/src/couchdb/couch_rpc.erl
- benoit
Re: roadmap 0.11/1.0
On Mon, Nov 2, 2009 at 7:10 PM, Benoit Chesneau bchesn...@gmail.com wrote: About that I had done some works on rpc calls based on hovercraft. Code is here and wasn't finished : http://github.com/benoitc/couchdb/blob/rpc/src/couchdb/couch_rpc.erl Reading this code I don't know why I used a gen_serv for that anyway maybe it could be useful. - benoit
Re: roadmap 0.11/1.0
On Nov 2, 2009, at 12:56 AM, Randall Leeds wrote: I'd like to add to see a patch to add filtering replication processes with an Erlang function. I think it would have lots of utility for any clustering solution that wants to get pushed into the trunk. I'm going to try to write the patch tomorrow. I think it seems fairly straightforward, but if anyone has some thoughts before I get started, let me know. Are you planning to filter on full documents, or just id/rev pairs?
Re: roadmap 0.11/1.0
On 1 Nov 2009, at 15:44, Benoit Chesneau wrote: Hi all, http://couchdb.apache.org/roadmap.html hasn't been updated. And in fact i'm really curious. What is the next things on the roadmap ? Also damien spoke in june to have a fixed release schedule (one every 6 months ?) is it still something in view ? This one definitely belongs on dev@ :) Cheers Jan --
Re: roadmap 0.11/1.0
Hi, Any chance of seeing native erlang RPC protocol in 11 or soon there after? Cheers Suhail On Sun, Nov 1, 2009 at 6:48 PM, Chris Anderson jch...@apache.org wrote: Thanks Benoit for the topic, I've got a wishlist for 0.11 / 1.0 - some of it I'd be game to implement, some of it I think others would be better suited for: Accounts tab in Futon Eventually we'll need something like /_utils but not for developers. For now we just need a page in Futon where you can login and out, as well as manage user accounts if you are an admin. Listable _changes If you could format the changes feed with a _list style API, you could use it to drive XMPP or other protocols. I'm keen on building a version of Toast chat that works even in browsers that have JS disabled. I have a feeling the only person who's gonna write this patch is me. couchjs security The couchjs process is currently secure enough for the context where only admins can modify design documents. However, as CouchDB spreads, we're sure to see misconfigured instances out there. Also, making the JS capabilities more controlled will definitely protect against some attacks in shared hosting environments. (Eg those where many users have private dbs on the same node.) Currently JS functions can hack the sandbox to make http requests via curl. We need this functionality for the test suite, but not for the design document OS process. So we should use a command line flag to --enable-http that the test runner can use. We can also be more secure in our [reset] handling. Because there's no such thing as a real JS sandbox, if we move our [reset] handling to C, and have it swap out the JS context for a new one, we'll avoid the case where databases on the same node can spy on each other, by say, overwriting the emit() function to also store important values for later playback to the attacker db. There could be some performance impacts of a C-based reset (as it would involve compiling all of main.js after each reset). An alternate way to implement this is just to stop recycling processes in Erlang, and through them out after each use. I think that will get expensive (but maybe not much worse than C-based reset). This is trivial patch if anyone needs to run extra securely in a shared-host environment today. cron / event / changes handler Applications need to be able to trigger functionality in a periodic or event-based way. We could probably piggyback on _changes heartbeat to provide cron + event services. The idea is a design document function (event or cron or maybe trigger) that is called once for each item in the changes feed. This is the one I'm least sure about, but I've heard a lot of people request it. It's problematic because cron functionality isn't that useful unless it has side-effects, which brings the whole sandbox/http question up again. rewriter There's getting to be a pretty common pattern where people write CouchApps and then deploy them behind a rewrite proxy. We've already got rewrite patches floating around. It's just a matter of making the API decisions. clustering I've heard Cloudant has some clustering code, I'd definitely be willing to help with integration, and I'm sure there are other people who would as well. Cheers, Chris On Sun, Nov 1, 2009 at 7:02 AM, Benoit Chesneau bchesn...@gmail.com wrote: was sent on user@ sorry for crossposting . -- Forwarded message -- From: Benoit Chesneau bchesn...@gmail.com Date: Sun, Nov 1, 2009 at 3:44 PM Subject: roadmap 0.11/1.0 To: u...@couchdb.apache.org Hi all, http://couchdb.apache.org/roadmap.html hasn't been updated. And in fact i'm really curious. What is the next things on the roadmap ? Also damien spoke in june to have a fixed release schedule (one every 6 months ?) is it still something in view ? - benoit -- Chris Anderson http://jchrisa.net http://couch.io
Re: roadmap 0.11/1.0
On Sun, Nov 1, 2009 at 11:00 AM, Suhail Ahmed suhail...@gmail.com wrote: Hi, Any chance of seeing native erlang RPC protocol in 11 or soon there after? This may be part of the Cloudant clustering codebase. I can't speak for them, but from what I've heard it does inter-node communication in a native Erlang way. You could probably use this interface as a primary client interface as well, but what do I know? :) Chris Cheers Suhail On Sun, Nov 1, 2009 at 6:48 PM, Chris Anderson jch...@apache.org wrote: Thanks Benoit for the topic, I've got a wishlist for 0.11 / 1.0 - some of it I'd be game to implement, some of it I think others would be better suited for: Accounts tab in Futon Eventually we'll need something like /_utils but not for developers. For now we just need a page in Futon where you can login and out, as well as manage user accounts if you are an admin. Listable _changes If you could format the changes feed with a _list style API, you could use it to drive XMPP or other protocols. I'm keen on building a version of Toast chat that works even in browsers that have JS disabled. I have a feeling the only person who's gonna write this patch is me. couchjs security The couchjs process is currently secure enough for the context where only admins can modify design documents. However, as CouchDB spreads, we're sure to see misconfigured instances out there. Also, making the JS capabilities more controlled will definitely protect against some attacks in shared hosting environments. (Eg those where many users have private dbs on the same node.) Currently JS functions can hack the sandbox to make http requests via curl. We need this functionality for the test suite, but not for the design document OS process. So we should use a command line flag to --enable-http that the test runner can use. We can also be more secure in our [reset] handling. Because there's no such thing as a real JS sandbox, if we move our [reset] handling to C, and have it swap out the JS context for a new one, we'll avoid the case where databases on the same node can spy on each other, by say, overwriting the emit() function to also store important values for later playback to the attacker db. There could be some performance impacts of a C-based reset (as it would involve compiling all of main.js after each reset). An alternate way to implement this is just to stop recycling processes in Erlang, and through them out after each use. I think that will get expensive (but maybe not much worse than C-based reset). This is trivial patch if anyone needs to run extra securely in a shared-host environment today. cron / event / changes handler Applications need to be able to trigger functionality in a periodic or event-based way. We could probably piggyback on _changes heartbeat to provide cron + event services. The idea is a design document function (event or cron or maybe trigger) that is called once for each item in the changes feed. This is the one I'm least sure about, but I've heard a lot of people request it. It's problematic because cron functionality isn't that useful unless it has side-effects, which brings the whole sandbox/http question up again. rewriter There's getting to be a pretty common pattern where people write CouchApps and then deploy them behind a rewrite proxy. We've already got rewrite patches floating around. It's just a matter of making the API decisions. clustering I've heard Cloudant has some clustering code, I'd definitely be willing to help with integration, and I'm sure there are other people who would as well. Cheers, Chris On Sun, Nov 1, 2009 at 7:02 AM, Benoit Chesneau bchesn...@gmail.com wrote: was sent on user@ sorry for crossposting . -- Forwarded message -- From: Benoit Chesneau bchesn...@gmail.com Date: Sun, Nov 1, 2009 at 3:44 PM Subject: roadmap 0.11/1.0 To: u...@couchdb.apache.org Hi all, http://couchdb.apache.org/roadmap.html hasn't been updated. And in fact i'm really curious. What is the next things on the roadmap ? Also damien spoke in june to have a fixed release schedule (one every 6 months ?) is it still something in view ? - benoit -- Chris Anderson http://jchrisa.net http://couch.io -- Chris Anderson http://jchrisa.net http://couch.io
Re: roadmap 0.11/1.0
Hey Chris, I like hovercraft and it's simple and clean api to talk to couch. Are you talking about clustering over RPC? I know way too little about couch to mouth off like this, working my way to defining a structure for trees and graphs in couch is keeping me busy enough. Cheers Suhail On Sun, Nov 1, 2009 at 7:15 PM, Chris Anderson jch...@apache.org wrote: On Sun, Nov 1, 2009 at 11:00 AM, Suhail Ahmed suhail...@gmail.com wrote: Hi, Any chance of seeing native erlang RPC protocol in 11 or soon there after? This may be part of the Cloudant clustering codebase. I can't speak for them, but from what I've heard it does inter-node communication in a native Erlang way. You could probably use this interface as a primary client interface as well, but what do I know? :) Chris Cheers Suhail On Sun, Nov 1, 2009 at 6:48 PM, Chris Anderson jch...@apache.org wrote: Thanks Benoit for the topic, I've got a wishlist for 0.11 / 1.0 - some of it I'd be game to implement, some of it I think others would be better suited for: Accounts tab in Futon Eventually we'll need something like /_utils but not for developers. For now we just need a page in Futon where you can login and out, as well as manage user accounts if you are an admin. Listable _changes If you could format the changes feed with a _list style API, you could use it to drive XMPP or other protocols. I'm keen on building a version of Toast chat that works even in browsers that have JS disabled. I have a feeling the only person who's gonna write this patch is me. couchjs security The couchjs process is currently secure enough for the context where only admins can modify design documents. However, as CouchDB spreads, we're sure to see misconfigured instances out there. Also, making the JS capabilities more controlled will definitely protect against some attacks in shared hosting environments. (Eg those where many users have private dbs on the same node.) Currently JS functions can hack the sandbox to make http requests via curl. We need this functionality for the test suite, but not for the design document OS process. So we should use a command line flag to --enable-http that the test runner can use. We can also be more secure in our [reset] handling. Because there's no such thing as a real JS sandbox, if we move our [reset] handling to C, and have it swap out the JS context for a new one, we'll avoid the case where databases on the same node can spy on each other, by say, overwriting the emit() function to also store important values for later playback to the attacker db. There could be some performance impacts of a C-based reset (as it would involve compiling all of main.js after each reset). An alternate way to implement this is just to stop recycling processes in Erlang, and through them out after each use. I think that will get expensive (but maybe not much worse than C-based reset). This is trivial patch if anyone needs to run extra securely in a shared-host environment today. cron / event / changes handler Applications need to be able to trigger functionality in a periodic or event-based way. We could probably piggyback on _changes heartbeat to provide cron + event services. The idea is a design document function (event or cron or maybe trigger) that is called once for each item in the changes feed. This is the one I'm least sure about, but I've heard a lot of people request it. It's problematic because cron functionality isn't that useful unless it has side-effects, which brings the whole sandbox/http question up again. rewriter There's getting to be a pretty common pattern where people write CouchApps and then deploy them behind a rewrite proxy. We've already got rewrite patches floating around. It's just a matter of making the API decisions. clustering I've heard Cloudant has some clustering code, I'd definitely be willing to help with integration, and I'm sure there are other people who would as well. Cheers, Chris On Sun, Nov 1, 2009 at 7:02 AM, Benoit Chesneau bchesn...@gmail.com wrote: was sent on user@ sorry for crossposting . -- Forwarded message -- From: Benoit Chesneau bchesn...@gmail.com Date: Sun, Nov 1, 2009 at 3:44 PM Subject: roadmap 0.11/1.0 To: u...@couchdb.apache.org Hi all, http://couchdb.apache.org/roadmap.html hasn't been updated. And in fact i'm really curious. What is the next things on the roadmap ? Also damien spoke in june to have a fixed release schedule (one every 6 months ?) is it still something in view ? - benoit -- Chris Anderson http://jchrisa.net http://couch.io -- Chris Anderson http://jchrisa.net http://couch.io
Re: roadmap 0.11/1.0
On Nov 1, 2009, at 4:59 PM, Jan Lehnardt wrote: On 1 Nov 2009, at 19:48, Chris Anderson wrote: cron / event / changes handler Applications need to be able to trigger functionality in a periodic or event-based way. We could probably piggyback on _changes heartbeat to provide cron + event services. The idea is a design document function (event or cron or maybe trigger) that is called once for each item in the changes feed. This is the one I'm least sure about, but I've heard a lot of people request it. It's problematic because cron functionality isn't that useful unless it has side-effects, which brings the whole sandbox/ http question up again. We need this for 0.11 to allow continuous replication to survive server crashes. +1 This is pretty much the only feature I want before 1.0.0. Everything else in this thread so far isn't important enough to hold up 1.0.0. In my opinion, we should focus on fixing bugs, filling in small gaps in the APIs, better testing and optimizations. I think we should set a hard date for Feb 1 for feature freeze for a 0.11.0 branch. The next version, 0.11.0, once stabilized, should be considered a release candidate, followed by 1.0.0 once proven stable in production. Once we branch for 0.11.0, All new features will be in trunk for 2.0.0, and might be included in a later 1.1.0 release if needed. -Damien While we are at it, we should add means to trigger compaction periodically or when certain limits are reached (IIRC Adam mentioned a waste-factor for databases to see when compaction is worth triggering). This would be the first thing I'd start working for new features. We also have a few open ends in the API that need consolidation. e.g. making the show/list/update handler more RFC2616 aware (sending Location e.g.). There's a few more of these. We should probably work on them before working on new half-baked features :) -- Furthermore, more etap tests and a benchmarking suite would be great. Cheers Jan -- rewriter There's getting to be a pretty common pattern where people write CouchApps and then deploy them behind a rewrite proxy. We've already got rewrite patches floating around. It's just a matter of making the API decisions. clustering I've heard Cloudant has some clustering code, I'd definitely be willing to help with integration, and I'm sure there are other people who would as well. Cheers, Chris On Sun, Nov 1, 2009 at 7:02 AM, Benoit Chesneau bchesn...@gmail.com wrote: was sent on user@ sorry for crossposting . -- Forwarded message -- From: Benoit Chesneau bchesn...@gmail.com Date: Sun, Nov 1, 2009 at 3:44 PM Subject: roadmap 0.11/1.0 To: u...@couchdb.apache.org Hi all, http://couchdb.apache.org/roadmap.html hasn't been updated. And in fact i'm really curious. What is the next things on the roadmap ? Also damien spoke in june to have a fixed release schedule (one every 6 months ?) is it still something in view ? - benoit -- Chris Anderson http://jchrisa.net http://couch.io
Re: roadmap 0.11/1.0
On Sun, Nov 1, 2009 at 4:55 PM, Mark Hammond skippy.hamm...@gmail.com wrote: On 2/11/2009 9:13 AM, Paul Davis wrote: couchjs security ... The first step to securing cURL handlers I'm going to tackle is the basic all-or-nothing HTTP support. With something like --with-http as Chris suggests. Mark also made a suggestion on including a --with-http-host=127.0.0.1 or similar to support HTTP requests to a specific host only. This is probably doable but I'll have to think a bit harder on how to force that in the cURL handlers. My use-case is to avoid allowing full-blown cURL access to javascript while still allowing a couchdb 'external' to have access to its database. Specifically, I want to use the 'externals' mechanism to build an application specific API - an API which hides/abstracts multiple couchdb calls and subsequent data munging into a single call which is more relevant to the application's direct requirements. For example, if I have an external which is referenced via http://host:port/dbname/_external, the implementation of that external needs access to the referenced 'dbname', but no others. On IRC, there were 2 thoughts: * Paul mentioned the cmd-line switch to disable cURL - if we provided a new option that only allowed connections back to the same DB (probably by exposing an API where the host and DB name can't even be specified by the js code), it might still meet the general security requirements. * Benoit mentioned that if we changed the externals 'protocol' slightly, we could probably offer a javascript API back to the database which uses stdin/stdout to exchange requests and responses, allowing us to live without cURL support at all. Such an API may also be able to help allow cron jobs etc to have safe side-effects - the only side-effects they can perform are ones exposed by couchdb itself, not via generic http requests. The second option sounds better, but much more work. I'm willing to roll my sleeves up on this, so please share your thoughts (or ask me to expand on my use-case if necessary) Eventually I'd like to see an API to allow users to grant applications the right to use HTTP in an unrestricted way. Think of the HTTP requirement for an XML feed-reader CouchApp. It seems odd to me that CouchDB has so much application horsepower, but has to rely on browser + JSONP to be the API client of 3rd party services. Bridging this gap safely is important. I like your proposal of safe DB-only access via restricted curl. The original action.js I committed long ago [1] had an insecure implementation of this. Once we have something here, extending it for more nuanced models will be worth a lot. Chris [1] http://svn.apache.org/viewvc?view=revisionrevision=727141 Thanks, Mark -- Chris Anderson http://jchrisa.net http://couch.io
Re: Roadmap discussion
On Tue, Feb 10, 2009 at 11:46 AM, Kerr Rainey kerr.rai...@gmail.com wrote: Is there still interest in stabilising a native erlang interface? -- Kerr Definitely. I was contemplating this a bit the other day. I wonder if it wouldn't be beneficial to create a couch_api.erl and just define an erlang api that maps to what other client libraries look like. Then if someone wants to peek into the internals they're free and we can maintain that we only support compatibility on that one file. Any way, just an idle thought. HTH, Paul Davis
Re: Roadmap discussion
* Full Text Search interface - We've had basically working patches for this floating around for a while. - It seems simple enough, we just need someone who comfortable in Java to step up to the plate and write a Lucene adapter. (Thanks!) I'm more than happy to look at this when I get time, I've been wondering where to start hacking on couch and we use solr at work (currently), so I would be able to justify some work time on it too Kev
Re: Roadmap discussion
I've made some progress on this, fwiw; http://github.com/rnewson/couchdb-lucene B. On Tue, Feb 10, 2009 at 12:27 PM, Kevin Jackson foamd...@gmail.com wrote: * Full Text Search interface - We've had basically working patches for this floating around for a while. - It seems simple enough, we just need someone who comfortable in Java to step up to the plate and write a Lucene adapter. (Thanks!) I'm more than happy to look at this when I get time, I've been wondering where to start hacking on couch and we use solr at work (currently), so I would be able to justify some work time on it too Kev
Re: Roadmap discussion
On Tue, Feb 10, 2009 at 8:53 AM, Paul Davis paul.joseph.da...@gmail.com wrote: On Tue, Feb 10, 2009 at 11:46 AM, Kerr Rainey kerr.rai...@gmail.com wrote: Is there still interest in stabilising a native erlang interface? -- Kerr Definitely. I was contemplating this a bit the other day. I wonder if it wouldn't be beneficial to create a couch_api.erl and just define an erlang api that maps to what other client libraries look like. Then if someone wants to peek into the internals they're free and we can maintain that we only support compatibility on that one file. I've been interfacing with the raw Erlang API for a commercial project. It works like a charm, the only trouble being that it isn't documented, and that it could change out from under me with no warning. (Although the second caveat isn't as bad as it sounds, because it probably won't change much.) From my experience, I'm having a hard time seeing how any additional code could help make the Erlang API official. The project I'm working on has a very specific data model (no updates, lots of parallel attachment writing, using the HTTP API for everything but the critical path...) and using the Erlang API has allowed me to cut out a lot of code paths (eg rev checking etc). Doing this wouldn't be safe for a general purpose API, but when you are interfacing in Erlang, you're not using a general purpose API anyway. I'm happy to have an Erlang API, but maybe it should wait til sometime after 0.9. I think the best way to ensure that it's maintained as stable would be to have an Erlang integration suite, which could double as documentation. It certainly wouldn't hurt to have more Erlang tests, so maybe we can file this feature under testing for now, and hope we get an Erlang test suite created by interested parties. Once we have the test suite we'll know what the Erlang API is. -- Chris Anderson http://jchris.mfdz.com
Re: Roadmap discussion
2009/2/10 Michael McDaniel couc...@autosys.us: ... also, an Erlang API that skips the JSON -convert- native Erlang terms translation overhead. Being as term translation is not necessary when talking 'directly' with the CDB engine (e.g. couch_query_servers:map_docs/2 could skip the JSON - term() translation if the view engine reads/writes native Erlang terms) Interesting. I'd certainly consider this another level further than what I was thinking of, or indeed would be thinking of using. There is probably a few levels where couch functionality could be exposed natively. I wonder how much doing this kind of bypassing for a native erlang view engine would complicate the code? Or would it give another clean layer? -- Kerr
Re: Roadmap discussion
On Mon, Feb 9, 2009 at 4:06 PM, Chris Anderson jch...@apache.org wrote: Couch Devs, From the beginning of this project, we've had a shared vision for CouchDB. The best documentation of that vision (in my mind) has been the text available at http://couchdb.apache.org/docs/overview.html However, the project has been evolving, and we are within sight of reaching many of the goals outlined in the Technical Overview. I think it would be helpful to outline our progress so far, and see how the project can grow. So far: * REST / MVCC / ACID Document Storage - This is solid (modulo _bulk_docs) - Append-only file structure has turned out to be a big win. * Incremental Map Reduce - This is also strong right now. It could be faster, but so far it's been fast enough. - Ready to implement view Etags. * Live database compaction - basically done. - View compaction would be a nice bonus. * Incremental Peer Based Replication - We're still working on non-buffering attachment replication. - Multi-node security model work is in progress. * Partitioning - I'm not sure how far we are along on this. Afaik, the plan is to split nodes into subnodes depending on load, essentially making a binary tree of Couches. My guess is that inner-nodes would be proxies and leaf nodes would handle disk. I think there's a dependency on getting the security model right. - View queries are just a big merge-sort across all the nodes. * Full Text Search interface - We've had basically working patches for this floating around for a while. - It seems simple enough, we just need someone who comfortable in Java to step up to the plate and write a Lucene adapter. (Thanks!) Not it! New parts: * External - This was originally implemented to support Full Text indexing. - People are using it as an interface to a whole range of app-servers and alternate indexers. * Show / List - Giving CouchDB the ability to serve directly non-JSON dynamic content types improves it's utility for offline applications. * Notification API - When Damien first mentioned the idea of holding open Comet-style connections for selective replication, it fit right into the bigger picture of Couch. I'm very interested in developments here. * Config - This is an outgrowth of supporting such a big system. Luckily now it's powerful enough to provide an easy way to hook custom Erlang modules into CouchDB's HTTP stack. * Stats - In-progress monitoring interface. * Alternate Storage and View Engines - There are interested parties here, but I think we're hoping to save this one until all the deeply subtle stuff Damien's working on comes together. I know there is a lot of stuff I've put on here, and there's a lot that I've missed. Please feel free to expand the outline. I'm hoping to discuss the best way for us to focus our energies. But first we should know where we stand and where we want to be. Chris -- Chris Anderson http://jchris.mfdz.com * Miscellaneous API extensions, nothing big on their own, but taken as whole make CouchDB more awesome. (multi-get, include_docs, stale=ok, etc) * The Status module/Futon page for tracking what background magic couchdb is doing.
Re: Roadmap discussion
On Feb 9, 2009, at 4:06 PM, Chris Anderson wrote: * Incremental Map Reduce - This is also strong right now. It could be faster, but so far it's been fast enough. I'd like to see built-in reductions for simple and commonly used things like count, sum, average, etc. When used it would avoid 90% of the current reduction costs. * Alternate Storage and View Engines - There are interested parties here, but I think we're hoping to save this one until all the deeply subtle stuff Damien's working on comes together. I think alternate view and reporting engines are a great idea, and I'd definitely like plug-ins for file attachment storage. Alternate storage engines aren't something that need to wait on me. -Damien
Re: Roadmap discussion
On 10/02/2009, at 9:24 AM, Damien Katz wrote: On Feb 9, 2009, at 4:06 PM, Chris Anderson wrote: * Incremental Map Reduce - This is also strong right now. It could be faster, but so far it's been fast enough. I'd like to see built-in reductions for simple and commonly used things like count, sum, average, etc. When used it would avoid 90% of the current reduction costs. * Alternate Storage and View Engines - There are interested parties here, but I think we're hoping to save this one until all the deeply subtle stuff Damien's working on comes together. I think alternate view and reporting engines are a great idea, and I'd definitely like plug-ins for file attachment storage. Alternate storage engines aren't something that need to wait on me. Any idea how _external is going to work in a partitioned database? For example, if _externals don't see a global update_seq, then mechanisms that currently work for e.g. geo-indexing or FTS won't be cluster-wide. Antony Blakey - CTO, Linkuistics Pty Ltd Ph: 0438 840 787 It is no measure of health to be well adjusted to a profoundly sick society. -- Jiddu Krishnamurti
