Chewbranca's Opinionated Vision for Apache CouchDB and Cloudant Dbcore
Hi all, Chewbranca here. I've been fairly quiet for the last few years for a number of reasons, but this year I've been becoming more involved again in the CouchDB community and various Cloudant communities. I put together this writeup a while ago to detail out what my personal views are on what I believe to be some of the highest priority focus points for CouchDB/Cloudant and targeted approaches for tackling these issues. Full disclosure, these are my own opinions discussing my views on direction for Cloudant and CouchDB; most of the views here are shared by a wider audience, but I want to be clear this is not a formal proposal from Cloudant but rather the musings of a long standing Cloudant and CouchDB member with plenty of strong opinions. The first draft of this document was done as an internal writeup with the intention of then creating a modified version that defines a set of core "stakeholders" such that the various tasks defined demonstrate value to the core stakeholders, but in particular, that all of this functionality is defined and executed in a way to bring clear value to the Apache CouchDB project and Apache CouchDB community. I strongly believe the majority of this functionality should be in the core CouchDB database itself and I want to ensure that it is done in a way to be clearly beneficial to _all_ stakeholders involved. I started adding in the discussion of target audience and stakeholders, but the breakout details for each of the discussion points below is incomplete. Forgive the lack of polish in this writeup, it's still a draft, but I've already sat on this longer than I wanted, and with the upcoming CouchDB developer meeting tomorrow, I felt it was better to get it out now in a rough form than worry about a completed writeup. We can always rip various pieces out and create more specific and complete writeups. I've got some additional writeups and ideas to get out to the mailing list as well, but this is a good starting point. A quick note on naming, "Dbcore" is the name of the version of CouchDB that Cloudant runs in production. Many years ago this was an actual fork, but since the 2.x merge we're essentially running CouchDB upstream with additional libraries added on for Cloudant specific functionality, like IAM auth and what not. The majority of the fun stuff was open sourced and incorporated into the 3.x codebase. This document switches between Dbcore and CouchDB in various areas, but again, this is more of a draft than a finished document. Thanks for your time and happy hacking! -Russell "Chewbranca" Branca == # Chewbranca's Immediate Vision for Cloudant Dbcore and Apache CouchDB Welcome to my vision of where to go with development and functionality of Cloudant's Dbcore and Apache CouchDB, for the immediate future in 2022/2023. This is an opinionated list of the things that I think are critically important and also readily achievable. We may or may not get to the things in this list, but here it is. There's much to do, much that could be done, and endless possibilities. This vision is targeted at the here and now, focused on immediately prevalent issues, constraints, and challenges, skewed towards heavy hitters in terms of time/resource investment to dividends paid. Not everything here is easy and simple, but everything here is cherry picked because of its significant impact, and also selected because it *needs* to be here. This vision is also not exhaustive, to neither Cloudant nor CouchDB. I make no claims that there isn't other important things to do, rather that the following list of things *is* of substantial importance and provides a path to paradigm shift the current state of Cloudant and CouchDB. ## Target Audience and Stakeholders I want this writeup to target a few different core groups of people and stakeholders to help understand motivations and help define scope. In particular, I'm targeting three core groups of people: * CouchDB users/operators * Cloudant customers * Cloudant operators As it stands today, these groups have very distinct interactions with the database, and while there will also be distinctions between users of the system and operators of the system, I think there's far more overlap between the three groups than is currently represented. In particular, all three of these groups need visibility into what the system is doing, what are the most active databases and documents, and what impact requests have on the system. Both users and operators need to have this visibility, and the more we can do to make it easier for everyone to see into the activity of the system, the better. The distinciton between Cloudant operators and CouchDB operators is considerably different as the tooling is fundamentally different, or rather heavily lacking in CouchDB and attached at the periphery in Cloudant; which is not ideal nor is sustainable in my opinion. I think we
Re: [VOTE] Set a finite default for max_attachment_size
A belated +1. Infinite attachment size is obviously an oversight, and practically speaking, I think gigabyte sized attachments are still way too big. I would love to see a solid binary attachment system with CouchDB, but neither couch_file nor FDB are likely to be it. -Russell On Mon, Feb 1, 2021 at 11:30 AM Joan Touzet wrote: > HI Donat, > > Point of order - when we do 72-hour votes, it's best to not count > weekends in that 72-hours. So, since you started on the 28th at 05:00 > UTC, I would have continued the vote until Feb 2 at 05:00 UTC. > > That said I am +1 on this too, long overdue. > > As to Eric's point, all we need to do is document that the setting can > be lifted and that we've tested "up to XGB" attachments as working in 3.x. > > 4.x change notes will show that the limit is reduced, and my opinion is > that concomitant with a 4.0 release there will be a new 3.x release that > lowers the defaults to match 4.x. (This may also need the changes > mentioned in Robert's other thread that are necessary to allow 3.x <-> > 4.x replication, that's still unclear.) For now, we don't need to reduce > down to 10MB or 5MB or whatever it is. > > -Joan > > On 01/02/2021 14:06, Bessenyei Balázs Donát wrote: > > This vote is now closed as there were three +1s, one +0 and no -1s and > > the 72 hours is up. I'll merge the PR. > > > > Thanks to all who voted! > > > > > > Donat > > > > > > On Mon, Feb 1, 2021 at 7:52 PM Eric Avdey wrote: > >> > >> Ok, fair enough, +0 from me with a note that I'd still prefer to see > this limit aligned with 4.x limits, so users wouldn't have to adjust to > this change twice. > >> > >> > >> Eric > >> > >>> On Feb 1, 2021, at 14:47, Nick Vatamaniuc wrote: > >>> > >>> I am +1 to lowering as it's better than infinity. > >>> > >>> But I also see Eric's point. I was surprised a while back just like > >>> Eric that I could successfully upload >1GB-sized files. So why not > >>> 0.5GB or 2GB? I am thinking 2GB was (is?) a common limit on some OSes > >>> and file systems (FAT32) since they use ints for file size and > >>> offsets. Since our attachment won't be saved as is in the file systems > >>> inside a .couch file 2GB may be too high, so 1GB as a limit makes > >>> sense to me. > >>> > >>> -Nick > >>> > >>> On Mon, Feb 1, 2021 at 1:25 PM Paul Davis > wrote: > > +1 > > Default unlimited seems like an oversight regardless of what we > change it to. > > On Mon, Feb 1, 2021 at 11:59 AM Eric Avdey wrote: > > > > Maybe I didn't express myself clear enough. Setting some finit > default is not a purpose, it's what you are doing and I'm asking what the > reason for this change. In other words I'm not asking what are you doing, > I'm asking why are you doing this. > > > > Introducing a new limit will be a breaking change to anoyone who > uploads attachments larger than that limit, obviously, so "assumed 1G is > large enough" sounds really arbitrary to me without any factual support for > that assumption. > > > > > > Eric > > > > > >> On Feb 1, 2021, at 13:15, Bessenyei Balázs Donát > wrote: > >> > >> The purpose of this vote / PR is to set _some_ finite default. I > went > >> with 1G as I assumed that would not break anyone's production > system. > >> I'd support decreasing that limit over time. > >> > >> The vote has been open for 72 hours now, but I believe it still > needs > >> two more +1s to pass. > >> > >> > >> Donat > >> > >> On Thu, Jan 28, 2021 at 10:44 PM Eric Avdey wrote: > >>> > >>> This got me curious and I tried to upload Ubuntu image as an > attachment. Interestingly CouchDB 3.x accepted first 1.4G of 2.8G file and > then returned proper 201 response with a new doc revision, which I certanly > didn't expect. Should say, that 1.4G seems suspiciously similar to a normal > memory limit for a 32 bit process. > >>> > >>> Putting this aside, I agree that uploading large attachments is an > anti-pattern and 1G seems excessive, hence my question. I'd expect this > number to be based on something and correlating it with a technical limit > in 4.x makes a lot of sense to me. > >>> > >>> > >>> Eric > >>> > >>> > On Jan 28, 2021, at 16:02, Robert Newson > wrote: > > Hi, > > I think a gigabyte is _very_ generous given our experience of > this feature in practice. > > In 4.x attachment size will necessarily be much more restrictive, > so it seems prudent to move toward that limit. > > I don’t think many folks (hopefully no one!) is routinely > inserting attachments over 1 gib today, I’d be fairly surprised if it even > works. > > B. > > > On 28 Jan 2021, at 19:42, Eric Avdey wrote: > > > > There is no justification neither here or on the PR for this > change, i.e. why this is done. Original infinity d
Re: [DISCUSS] Removing erlang 19 support
Oh that's fantastic, Joan! Glad to hear it. I'm excited to clear out some of the older Erlang versions, and I would love to see CouchDB and IBM/Cloudant stay more uptodate on Erlang releases. Thanks for all the work on the packaging front, that's awesome to see! -Russell On Mon, Jan 25, 2021 at 5:12 PM Joan Touzet wrote: > Hey Russell, have no fear. I'm happy to say your 2014 essay is no longer > an issue :) > > We've been shipping our couchdb binaries with Erlang Solutions' > pre-built Erlang for a very long time now, at least since 2.1.0 and > possibly since 2.0.0 released. When they don't provide it, we build > using kerl. Whatever version of Erlang comes with the OS doesn't matter > at all. In fact, we ship the exact same version of Erlang across every > OS/arch we support today. > > The only limitation is whether or not the specific version of Erlang > will *compile* on that specific OS/arch combo with the features we need > - this usually comes down to the version of openssl shipped by the > distro being compatible with Erlang's ssl module. There was a hiccup in > the transition fom 0.9.8 to 1.0.x, and again from 1.0.x to 1.1.x, but > that's all well in the past now. > > Cheers, > Joan "DRY" Touzet > > On 2021-01-25 3:40 p.m., Russell Branca wrote: > > I'm also +1 to removing Erlang 19. > > > > I wanted to reiterate what Newson said about Erlang Solutions providing > > Erlang packaging, and I think we should more strongly lean on options > like > > this rather than being dependent on the OS distros Erlang versions. Many > > years ago I wrote about the nuances with Erlang versions and CouchDB on > the > > R14/R15/R16 lines [1], and it's worth noting that multiple Debian > versions > > shipped versions of Erlang that were fundamentally broken for use with > > CouchDB. I think we even blocked a number of those versions in the > > rebar.config allowed versions. > > > > I don't know how much of an issue that is today, but there's certainly > > precedent for distro shipped Erlang versions not being an option. > > > > > > -Russell > > > > > > > > [1] > > > https://chewbranca.com/doc/trunk/archive/github/2014-05-07-on-the-viability-of-erlang-releases-and-couchdb.md > > > > > > > > On Mon, Jan 25, 2021 at 5:07 AM Bessenyei Balázs Donát < > bes...@apache.org> > > wrote: > > > >> Thank you all for the input! > >> > >> I'll remove erlang 19 for `couchdb-config` and we'll be able to refer > >> to this thread when we have to remove it anywhere else. > >> > >> > >> Donat > >> > >> On Sat, Jan 23, 2021 at 10:09 PM Adam Kocoloski > >> wrote: > >>> > >>> Ah, good research there Joan. +1 to Donat’s suggestion to drop support > >> for 19 from me. > >>> > >>> Adam > >>> > >>>> On Jan 22, 2021, at 4:49 PM, Joan Touzet wrote: > >>>> > >>>> On 2021-01-22 4:37 p.m., Robert Newson wrote: > >>>>> Iteresting. I’m actually surprised at the inversion here (that > >> CouchDB is dependent on IBM to confirm CouchDB’s stability). I’ve > always > >> agonised over even the perception that IBM/Cloudant is calling the > shots. I > >> appreciate the reassurance that running at scale provides, of course, I > >> just don’t think it can be our official position. > >>>> > >>>> It's a tough one. I was pretty aggressive on CouchDB running the very > >> latest until the scheduler collapse problem surfaced. After that, there > was > >> another problem (I can't recall) that was pretty serious too. I took a > >> wait-and-see attitude at that point, and after I didn't see IBM move > >> forward to a newer release, didn't move forward ourselves. Looks like we > >> ended up in deadlock! > >>>> > >>>> However! See your own comments on this: > >>>> > >>>> https://github.com/apache/couchdb/issues/3115#issuecomment-729031967 > >>>> > >>>> I knew there was something at the back of my head on this. Guess we're > >> both getting old ;) > >>>> > >>>>> On the core point of the thread, it seems there’s no barrier to > >> dropping Erlang 19 support, so I think we can go to a VOTE thread, > perhaps > >> best to wait till Monday for others to chime in on this discussion > though. > >>>> >
Re: [PROPOSAL] Archiving git branches
A belated +1 to archiving. Thanks Joan, and also thanks for keeping the remotes/origin/ioq-per-shard-or-user branch. -Russell On Wed, Oct 21, 2020 at 12:27 PM Joan Touzet wrote: > Hah! > > OK, it's all done. At present I cannot remove any of the #.#.# branches > as Infra has protected them. (I may or may not bother to open a ticket > on this.) > > Here's the remaining branch list: > > remotes/origin/1.3.x > remotes/origin/1.4.x > remotes/origin/1.5.x > remotes/origin/1.6.x > remotes/origin/1.x.x > remotes/origin/1278-add-clustered-db-info > remotes/origin/2.0.x > remotes/origin/2.1.x > remotes/origin/2.3.x > remotes/origin/2493-remove-auth-cache > remotes/origin/3.0.x > remotes/origin/3.x > remotes/origin/HEAD -> origin/main > remotes/origin/access > remotes/origin/bump-ibrowse > remotes/origin/feat/access-3.x > remotes/origin/feat/access-master-clean > remotes/origin/feat/add-same-site-secure/master > remotes/origin/ioq-per-shard-or-user > remotes/origin/main > remotes/origin/master > remotes/origin/prototype/fdb-layer-db-version-as-vstamps > remotes/origin/re-enable-most-elixir-tests > remotes/origin/record_last_compaction > remotes/origin/smoosh-update-operator-guide > remotes/origin/update-fauxton-1.2.6 > > That's about 6x shorter. Phew! > > Remember, if you are trying to recover a branch that is now gone, just: > > git checkout -b archive/ > > -Joan > > On 21/10/2020 14:59, Jan Lehnardt wrote: > > *makes chain saw noises* > > > > (trimming branches, get it?) > > > > Thanks Joan! > > > > Best > > Jan > > — > > > >> On 21. Oct 2020, at 20:23, Joan Touzet wrote: > >> > >> I am starting the work now. As there was no response, I'm going to only > >> keep these branches: > >> > >> 2.3.x > >> 3.x > >> main > >> master (with a README saying you're in the wrong place) > >> > >> plus these PR branches: > >> > >> bump-ibrowse (#3208) > >> smoosh-update-operator-guide (#3184) > >> re-enable-most-elixir-tests (#3175) > >> feat/add-same-site-secure/master (#3131) > >> feat/access-master-clean (#3038) > >> prototype/fdb-layer-db-version-as-vstamps (#2952) > >> feat/access-3.x (#2943) > >> ioq-per-shard-or-user (#1998) > >> 1278-add-clustered-db-info (#1443) > >> record_last_compaction (#1272) > >> > >> Any of the above that were targeted to prototype/fdb-layer or master > >> have been re-targeted to main (except a couple clustering-specific ones > >> which were set to 3.x). Please check if this is correct for your PRs. > >> > >> -Joan "clean all the things?" Touzet > >> > >> On 14/10/2020 13:19, Joan Touzet wrote: > >>> A reminder about this: I intend to start this work tomorrow. If you > have > >>> any PRs or branches you want left alone, speak now. > >>> > >>> Based on feedback I've received, it sounds like the prototype/fdb-* > >>> branches are now done? If this is **NOT** the case, speak up. > >>> > >>> -Joan > >>> > >>> On 07/10/2020 18:10, Joan Touzet wrote: > Hi there, > > I'd like to clean up our branches in git on the main couchdb repo. > This > would involve deleting some of our obsolete branches, after tagging > the > final revision on each branch. This way, we retain the history but the > branch no longer appears in the dropdown on GitHub, or in git branch > listings at the cli. > > Example process: > > git tag archive/1.3.x 1.3.x > git branch -d 1.3.x > git push origin :1.3.x > git push --tags > > If we ever needed the branch back, we just: > > git checkout -b 1.3.x archive/1.3.x > > I would propose to do this for all branches except: > > main > master (for now) > 2.3.x > 3.x > prototype/fdb-layer > > ...plus any branches that have been touched in the past 90 days, that > still have open PRs, or that someone specifically asks me to retain in > this thread. > > I'd also like to do this on couchdb-documentation and couchdb-fauxton. > > I would propose to do this about 1 week from now, let's say on October > 15th. > > Thoughts? > > -Joan "fall cleaning" Touzet > > >
Re: [ANNOUNCE] Bessenyei Balázs Donát elected as CouchDB committer
Congrats Donat! Welcome aboard! -Russell On Fri, Jan 15, 2021 at 10:17 AM Alessio 'Blaster' Biancalana < dottorblas...@apache.org> wrote: > Congratulations and welcome! > > Alessio > > On Fri, Jan 15, 2021 at 4:40 PM Dave Cottlehuber > wrote: > > > > > On 14. Jan 2021, at 21:48, Robert Newson wrote: > > > > > > > > Dear community, > > > > > > > > I am pleased to announce that the CouchDB Project Management > Committee > > has elected Bessenyei Balázs Donát as a CouchDB committer. > > > > > > > > Apache ID: bessbd > > > > > > > > Committers are given a binding vote in certain project decisions, as > > well as write access to public project infrastructure. > > > > > > > > This election was made in recognition of Donát's commitment to the > > project. We mean this in the sense of being loyal to the project and its > > interests. > > > > > > > > Please join me in extending a warm welcome to Donát! > > > > > > > > On behalf of the CouchDB PMC, > > > > > > > > Robert Newson > > > > Awesome! welcome Donát! > > > > A+ > > Dave > > >
Re: [DISCUSS] Removing erlang 19 support
I'm also +1 to removing Erlang 19. I wanted to reiterate what Newson said about Erlang Solutions providing Erlang packaging, and I think we should more strongly lean on options like this rather than being dependent on the OS distros Erlang versions. Many years ago I wrote about the nuances with Erlang versions and CouchDB on the R14/R15/R16 lines [1], and it's worth noting that multiple Debian versions shipped versions of Erlang that were fundamentally broken for use with CouchDB. I think we even blocked a number of those versions in the rebar.config allowed versions. I don't know how much of an issue that is today, but there's certainly precedent for distro shipped Erlang versions not being an option. -Russell [1] https://chewbranca.com/doc/trunk/archive/github/2014-05-07-on-the-viability-of-erlang-releases-and-couchdb.md On Mon, Jan 25, 2021 at 5:07 AM Bessenyei Balázs Donát wrote: > Thank you all for the input! > > I'll remove erlang 19 for `couchdb-config` and we'll be able to refer > to this thread when we have to remove it anywhere else. > > > Donat > > On Sat, Jan 23, 2021 at 10:09 PM Adam Kocoloski > wrote: > > > > Ah, good research there Joan. +1 to Donat’s suggestion to drop support > for 19 from me. > > > > Adam > > > > > On Jan 22, 2021, at 4:49 PM, Joan Touzet wrote: > > > > > > On 2021-01-22 4:37 p.m., Robert Newson wrote: > > >> Iteresting. I’m actually surprised at the inversion here (that > CouchDB is dependent on IBM to confirm CouchDB’s stability). I’ve always > agonised over even the perception that IBM/Cloudant is calling the shots. I > appreciate the reassurance that running at scale provides, of course, I > just don’t think it can be our official position. > > > > > > It's a tough one. I was pretty aggressive on CouchDB running the very > latest until the scheduler collapse problem surfaced. After that, there was > another problem (I can't recall) that was pretty serious too. I took a > wait-and-see attitude at that point, and after I didn't see IBM move > forward to a newer release, didn't move forward ourselves. Looks like we > ended up in deadlock! > > > > > > However! See your own comments on this: > > > > > > https://github.com/apache/couchdb/issues/3115#issuecomment-729031967 > > > > > > I knew there was something at the back of my head on this. Guess we're > both getting old ;) > > > > > >> On the core point of the thread, it seems there’s no barrier to > dropping Erlang 19 support, so I think we can go to a VOTE thread, perhaps > best to wait till Monday for others to chime in on this discussion though. > > > > > > More important is that we already committed changes on the main repo > re: Erlang 19 about 14 months ago by Paul: > > > > > > > https://github.com/apache/couchdb/commit/3594f2f1fc16903c1c383ebaf205d31c9c17fb3a > > > > > > I think that makes Donat's request pretty straightforward. > > > > > >> I also think that IBM Cloudant’s chosen Erlang release is in part > influenced by CouchDB’s lack of support for later versions and requirement > of compatible with older releases, which now appears illusory. > > > > > > If we're ready to move to 21 or 22 as a default, we're ready. Let's > hope the serious issues in 21/22 are at least mitigated. I'm happy to make > the 3.3 release (or whatever is next) use the very latest version of 21 or > 22 from GitHub, subject to community recommendations and encouragement. 23 > is still a WIP: https://github.com/apache/couchdb/issues/3115 > > > > > > -Jon > > > > > >> B. > > >>> On 22 Jan 2021, at 21:19, Joan Touzet wrote: > > >>> > > >>> On 22/01/2021 15:48, Robert Newson wrote: > > I’m +1 on dropping Erlang 19 support. Erlang is now on major > release 23. > > >>> > > >>> No problem here. > > >>> > > I’d further advocate a general policy of supporting only the most > recent 2 or 3 major releases of Erlang/OTP. > > > > The main (I think only?) reason to keep compatibility so far back > is because of the versions supported by some OS’es. I don’t feel that is a > strong reason given the couchdb install, in common with Erlang-based > projects, is self-contained. The existence of Erlang Solutions packages for > all common platforms is also a factor. > > >>> > > >>> That hasn't been the case for at least 2 years, if not longer. > > >>> > > >>> As the release engineer, I've been focused on stability for everyone. > > >>> This is largely driven by knowing that IBM/Cloudant largely run > CouchDB > > >>> on 20.x at scale. Standing on the shoulders of giants, our releases > run > > >>> the latest 20.x release at the time of binary generation. > > >>> > > >>> A few times recently issues cropped up in 21 and 22 that we didn't > > >>> encounter in our user base because, at scale, we are deployed on > > >>> 20.3.8.something. Some of these issues were non-trivial. (I'm off > today, > > >>> so I don't have the time to dig into the specifics until Monday.) > > >>> > > >>> So my $0.02 is that: if IBM/Cloudant is ready
Re: [VOTE] couchdb 4.0 transaction semantics
+1 -Russell On Fri, Jan 8, 2021 at 11:54 AM Joan Touzet wrote: > +1. > > This is for now I presume, as I thought that there was feeling about > relaxing this restriction somewhat for the 5.0 timeframe? Memory's dim. > > -Joan > > On 07/01/2021 06:00, Robert Newson wrote: > > Hi, > > > > Following on from the discussion at > https://lists.apache.org/thread.html/rac6c90c4ae03dc055c7e8be6eca1c1e173cf2f98d2afe6d018e62d29%40%3Cdev.couchdb.apache.org%3E > < > https://lists.apache.org/thread.html/rac6c90c4ae03dc055c7e8be6eca1c1e173cf2f98d2afe6d018e62d29@%3Cdev.couchdb.apache.org%3E > > > > > > The proposal is; > > > > "With the exception of the changes endpoint when in feed=continuous > mode, that all data-bearing responses from CouchDB are constructed from a > single, immutable snapshot of the database at the time of the request.” > > > > Paul Davis summarised the discussion in four bullet points, reiterated > here for context; > > > > 1. A single CouchDB API call should map to a single FDB transaction > > 2. We absolutely do not want to return a valid JSON response to any > > streaming API that hit a transaction boundary (because data > > loss/corruption) > > 3. We're willing to change the API requirements so that 2 is not an > issue. > > 4. None of this applies to continuous changes since that API call was > > never a single snapshot. > > > > > > Please vote accordingly, we’ll run this as lazy consensus per the bylaws > (https://couchdb.apache.org/bylaws.html#lazy < > https://couchdb.apache.org/bylaws.html#lazy>) > > > > B. > > > > >
Re: CouchDB running on the Raspberry Pi?
Fwiw I got CouchDB 2.x running on a Raspian Raspberry Pi 3.b recently without much fuss. Pretty sure I compiled Erlang from source. -Russell On Fri, Jan 19, 2018 at 4:12 AM Eric Clack wrote: > Hi Michael, Ben and Johs, > Thanks for your responses. > > I've found for my (admittedly small) projects Couch works pretty well on > the Pi (version 2). It's a great solution to the problem of where to store > metadata, and how to query it easily. The only performance problem I saw > was when reducing and grouping queries, e.g. here I used a memo to cache > results: > > https://github.com/ericclack/clojure-photo-bank/blob/master/src/clj/clojure_photo_bank/models/db.clj#L45 > ...running the query took about 5 seconds for around 4000 records. That's > not great, but for everything else Couch was easily fast enough. > > I'm going to explore Erlang packages for Raspbian and then look at > compiling a few different versions of CouchDB to see if I can at least get > some good bug reports to the right team. > > Many thanks, > -Eric. > > > > On 19 January 2018 at 08:11, Johs Ensby wrote: > > > Hi Eric, > > > > I think it would be great if you found a way to run couch on Raspberry > Pi, > > preferably Ubuntu Mate. > > Performance will surely not be great, but more important I would think it > > could be a great demonstration of how CouchDB could be a common platform > > from big clusters in the cloud to your very private system. > > > > johs:) > > > > > > > On 16 Jan 2018, at 23:28, benjamin.bast...@gmail.com wrote: > > > > > > I did this back quite a few years ago, and from what I remember IO > > > performance was pretty dreadful on an SD card. I'm not sure what the > > status > > > of COUCHDB-3287 is, but maybe a different storage engine would offer > > better > > > performance? > > > > > > Ben > > > > > > On Tue, Jan 16, 2018 at 2:08 PM, Michael Fair > > > > wrote: > > > > > >> This is similar to running Couch on mobile phones. > > >> Perhaps an alternative is starting with another "Couch Compatible" > > database > > >> that's lighter weight? > > >> > > >> Perhaps PouchDB running on Node.js comes to mind. > > >> This implements the replication/sync protocol between Couch compatible > > >> databases, without being an erlang based, sharded, distributed > backend. > > >> > > >> > > >> > > >> On Tue, Jan 16, 2018 at 2:44 AM, Eric Clack > > > >> wrote: > > >> > > >>> Hello CouchDB devs, > > >>> > > >>> Is there any interest in getting CouchDB running on the Raspberry Pi > > (an > > >>> AMD platform), running Raspbian Stretch? > > >>> > > >>> See my post here: > > >>> https://github.com/apache/couchdb/issues/1103 > > >>> > > >>> As I said in the post, I have time to contribute to the work and am > > >>> interested in learning more about CouchDB, Erlang, etc. > > >>> > > >>> Right now it would be useful for me to gauge interest as I need to > know > > >>> whether I should investigate alternative databases for my Pi > projects. > > >>> > > >>> Many thanks, > > >>> -Eric. > > >>> > > >>> -- > > >>> Eric Clack > > >>> e...@bn7.net > > >>> East Sussex, England. > > >>> > > >> > > > > > > > -- > Eric Clack > e...@bn7.net > East Sussex, England. >
Re: [RFC] On the Testing of CouchDB
ed. q=4 takes ~20s, q=2 ~10s and q=1 ~5s. I’m not > suggesting we set q=1 for all tests since q>1 is a behaviour we would > want to test as well, but maybe we can set q=2 when running the test > suite(s) for the time being. Shaving 25s off of a single test will get > us a long way with all tests ported. What do others think? > I'm not sure what's going on here, although Paul ran into similar situations where tests would run on my machine in under a second but take significantly longer on his machine. I think we should investigate this a bit more before taking any major action, but for a lot of tests running with a lower Q value would probably be fine. I think it's important we run with Q >= 2 so we at least run through the standard code paths of having to merge over multiple shard ranges for aggregate operations. Although that said, it could also be worthwhile to run tests with Q=1 to exercise the non merge code paths as well. > > @Nick: you introduced the -C / --config-overrides option to dev/run, > but I could never figure out how to apply it. That would be the easiest > place to make the cluster config change for the Elixir tests. > > `-c q=2` makes the nodes fail to start and I’m not sure how in this case, > the `[cluster]` section is meant. > > * * * > > Russell, Paul: do you think it is worth reaching out to the Elixir > community and ask if they are interested in helping out a little? If > you think its too early, we can wait with this. > Yeah I think that's a great idea and could be an excellent opportunity to get more folks involved in things. I do think we should probably figure out our branch/dev strategy prior to doing that, as me cherry-picking commits will eventually become onerous given sufficient branches. > * * * > > Thanks again! > > Best > Jan > -- > > No problem, really happy to see folks excited about this new engine! -Russell > > > > On 15. Dec 2017, at 02:03, Russell Branca wrote: > > > > Howdy folks! > > > > The testing of CouchDB is something that has seen focus and improvements > > for the last several years, for instance migrating the etap suite to > eunit, > > and updating the JS suite to run against clusters in 2.x. There's still > > improvements to be made, and that was one of the topics of the CouchDB > dev > > summit early in the year [1]. > > > > Before we go further, I want to clarify some nomenclature. I'm by no > means > > going to try and define unit testing vs integration testing vs quantum > > phase shift testing, but instead I want to focus on the distinction of > > where the testing takes place. Fundamentally, we have two places we test > > CouchDB: 1) at the Erlang VM level where we conduct assertions against > > module functions or process states; 2) at the HTTP level where we test > the > > behavior of CouchDB at the user level API. This post focuses entirely on > > the latter; that's not to say the former doesn't also merit attention, > just > > that the two are different enough that we can focus on them in isolation. > > > > So with that, let's chat about the current HTTP test suite in CouchDB. > This > > is the "JS suite" I referred to above, which is a custom built test suite > > written in Javascript and executed in the aging SpiderMonkey. The JS > suite > > has put in work for years, but it's showing it's age, and is a bit > awkward > > to work with and improve. However, I think the biggest issue with the JS > > suite is that it's utilized far less than it should be, and folks seem to > > avoid extending it or adding additional tests to it. There's been > > discussion for years about replacing said suite, but the discussions > > invariably got blocked on the bike shed of whether to rewrite the suite > in > > Javascript or Python. This thread provides a third option, with code! > > > > I started hacking on a replacement for the JS suite, this time written in > > Elixir. Overall I'm quite impressed with how it's come along, and have > some > > good examples to show. This is basically an Elixir app that has an HTTP > > client and then runs a series of tests that conduct tests against the > > CouchDB HTTP API and make assertions therein. > > > > You can find the current code in [2], and a comparison of the changes in > > [3]. The core HTTP client is only a handful of lines of codes and works > > quite well [4]. The utility functions used across all tests are located > in > > [5], and the tests themselves are in [6]. The existing test modules have > a > > 1:1 correspondence with the associated J
Re: [RFC] On the Testing of CouchDB
I just updated the README in the Elixir suite to have a todo list of tests to port, in case folks are looking for tests: https://github.com/apache/couchdb/blob/elixir-suite/test/elixir/README.md -Russell On Fri, Dec 15, 2017 at 9:46 AM Paul Davis wrote: > For `make check` it should be fairly straightforward to map the > current approach to it. I could probably knock that out fairly quickly > if you want me to give it a whirl. > > On Fri, Dec 15, 2017 at 11:42 AM, Russell Branca > wrote: > > Yeah just to reiterate what Paul said, the Elixir dev experience is > really > > nice and easy to get rolling with. I had no prior actual experience with > > Elixir and I was able to get things rolling in a few hours. > > > > RE Ben's question about diving in: please do! Just grab one of the > unported > > js suites and goto town. I've just been cherry-pick'ing things out of > > Paul's branch and we can continue to do the same until we get this more > > locked down. My goal with the porting is to keep chugging along and just > > get it knocked out, as I really don't think it will be overly onerous to > do > > so. And if anyone else wants to jump in, there's still a fair number of > > tests to port, just take your pick. > > > > One other thing that needs work is figuring out how to hook all this into > > "make check" and what not. I've mostly ignored that as this just points > at > > a CouchDB instance and can be run directly, but we'll need to sort that > out > > at some point. > > > > > > -Russell > > > > On Fri, Dec 15, 2017 at 9:03 AM Paul Davis > > wrote: > > > >> Hello everybody! > >> > >> I figured I should probably go ahead and chime in seeing as I've also > >> been playing around porting some of the tests in my free time between > >> ops shifts the last couple weeks. > >> > >> My first impression was that it was ridiculously easy to get involved. > >> On OS X at least, `brew install elixir` was enough to get a working > >> elixir installed (however, if you use kerl or erln8 you'll want have > >> to build an Erlang 20.x VM to use the brew package). I went from not > >> having Elixir installed to a full port of uuids.js with the config tag > >> logic written in about two hours one night. So far the Elixir docs and > >> seem very well written and put together. I'd say the worst part of > >> Elixir so far is that knowing Erlang I find myself searching for "How > >> do I do this Erlang thing in Elixir?" Which isn't as bad as it sounds. > >> The Elixir libraries have certainly had a considerable amount of > >> thought put into them to make them easy to use and remember. I find it > >> to be a lot like my experience when learning Python in that I may have > >> to Google once and then its muscle memory. As opposed to Erlang's > >> library where I'm constantly reading the lists manpage to remember > >> argument orderings and whether I want search or find versions etc. > >> > >> Which I guess is a long way of saying I'm rather liking the Elixir > >> development experience so far. > >> > >> That said, I'm currently about half way through porting replication.js > >> tests to Elixir. For the most part its fairly straightforward. My > >> current approach as we've done for the other modules is to do a direct > >> port. Once that's finished we'll want to break up that huge module > >> into a series of modules that share a lot of the utility functions. > >> One of the nice things about moving to Elixir is that its got a full > >> on development story rather than our current couchjs approach that > >> prevents sharing code easily between subsets of tests. > >> > >> For Ben's question on diving in, I'd do just that. I'd say leave a > >> note here about which module(s)? you're going to port so that we're > >> not duplicating efforts and then its basically just a matter of > >> getting Elixir installed. For that, here's a quick rundown on how I > >> got that working: > >> > >> $ brew update > >> $ brew install elixir > >> $ # wait for all the things... > >> $ iex # which fails cause I have an Erlang VM older than 20.0 as a > default > >> $ erln8 --fetch > >> $ erln8 --build --tag=OTP-20.1.6 --id=20.1.6 > >> $ # wait while erln8 does its thing > >> $ git clone https://github.com/apache/cou
Re: [RFC] On the Testing of CouchDB
s. And even just > initial impressions on toying around with Elixir. My only experience > with Elixir prior to this was reading through their quick > start/tutorial pages a couple of times to get a feeling for the syntax > but hadn't actually even typed it into an editor till last week. > > And that's all I've got for now. > > On Thu, Dec 14, 2017 at 11:57 PM, Benjamin Anderson > wrote: > > Slick! This seems like it's coming together really nicely. Can't argue > > with commits like "Prefer ?w=3 over hacky sleeps"[1] in any case. > > > >> I hope others have similar opinions after diving in! > > > > How should one dive in? Are you looking for others to help out with > > the ports, or just thinking aspirationally about future regular > > contributions to the test suite? > > > > -- > > b > > > > [1]: > https://github.com/apache/couchdb/commit/5bce2d98a298c25b77d8dcda19deeedb494cc289 > > > > On Thu, Dec 14, 2017 at 5:03 PM, Russell Branca > wrote: > >> Howdy folks! > >> > >> The testing of CouchDB is something that has seen focus and improvements > >> for the last several years, for instance migrating the etap suite to > eunit, > >> and updating the JS suite to run against clusters in 2.x. There's still > >> improvements to be made, and that was one of the topics of the CouchDB > dev > >> summit early in the year [1]. > >> > >> Before we go further, I want to clarify some nomenclature. I'm by no > means > >> going to try and define unit testing vs integration testing vs quantum > >> phase shift testing, but instead I want to focus on the distinction of > >> where the testing takes place. Fundamentally, we have two places we test > >> CouchDB: 1) at the Erlang VM level where we conduct assertions against > >> module functions or process states; 2) at the HTTP level where we test > the > >> behavior of CouchDB at the user level API. This post focuses entirely on > >> the latter; that's not to say the former doesn't also merit attention, > just > >> that the two are different enough that we can focus on them in > isolation. > >> > >> So with that, let's chat about the current HTTP test suite in CouchDB. > This > >> is the "JS suite" I referred to above, which is a custom built test > suite > >> written in Javascript and executed in the aging SpiderMonkey. The JS > suite > >> has put in work for years, but it's showing it's age, and is a bit > awkward > >> to work with and improve. However, I think the biggest issue with the JS > >> suite is that it's utilized far less than it should be, and folks seem > to > >> avoid extending it or adding additional tests to it. There's been > >> discussion for years about replacing said suite, but the discussions > >> invariably got blocked on the bike shed of whether to rewrite the suite > in > >> Javascript or Python. This thread provides a third option, with code! > >> > >> I started hacking on a replacement for the JS suite, this time written > in > >> Elixir. Overall I'm quite impressed with how it's come along, and have > some > >> good examples to show. This is basically an Elixir app that has an HTTP > >> client and then runs a series of tests that conduct tests against the > >> CouchDB HTTP API and make assertions therein. > >> > >> You can find the current code in [2], and a comparison of the changes in > >> [3]. The core HTTP client is only a handful of lines of codes and works > >> quite well [4]. The utility functions used across all tests are located > in > >> [5], and the tests themselves are in [6]. The existing test modules > have a > >> 1:1 correspondence with the associated JS suite test modules, and in > >> general are as direct of a port as possible. > >> > >> The test modules ported in their entirety or most of the way are: > >> > >> * all_docs.js > >> * basics.js > >> * config.js > >> * reduce.js > >> * rewrite.js > >> * uuids.js > >> * view_collation.js > >> > >> Paul has dove in and is responsible for a few of those test modules and > >> he's almost completed porting the replication.js suite as well. We > started > >> with the hard ones first, so for the most part the rest of the ports > should > >> be fairly smooth sailing. > >> > >> Here's a
[RFC] On the Testing of CouchDB
Howdy folks!
The testing of CouchDB is something that has seen focus and improvements
for the last several years, for instance migrating the etap suite to eunit,
and updating the JS suite to run against clusters in 2.x. There's still
improvements to be made, and that was one of the topics of the CouchDB dev
summit early in the year [1].
Before we go further, I want to clarify some nomenclature. I'm by no means
going to try and define unit testing vs integration testing vs quantum
phase shift testing, but instead I want to focus on the distinction of
where the testing takes place. Fundamentally, we have two places we test
CouchDB: 1) at the Erlang VM level where we conduct assertions against
module functions or process states; 2) at the HTTP level where we test the
behavior of CouchDB at the user level API. This post focuses entirely on
the latter; that's not to say the former doesn't also merit attention, just
that the two are different enough that we can focus on them in isolation.
So with that, let's chat about the current HTTP test suite in CouchDB. This
is the "JS suite" I referred to above, which is a custom built test suite
written in Javascript and executed in the aging SpiderMonkey. The JS suite
has put in work for years, but it's showing it's age, and is a bit awkward
to work with and improve. However, I think the biggest issue with the JS
suite is that it's utilized far less than it should be, and folks seem to
avoid extending it or adding additional tests to it. There's been
discussion for years about replacing said suite, but the discussions
invariably got blocked on the bike shed of whether to rewrite the suite in
Javascript or Python. This thread provides a third option, with code!
I started hacking on a replacement for the JS suite, this time written in
Elixir. Overall I'm quite impressed with how it's come along, and have some
good examples to show. This is basically an Elixir app that has an HTTP
client and then runs a series of tests that conduct tests against the
CouchDB HTTP API and make assertions therein.
You can find the current code in [2], and a comparison of the changes in
[3]. The core HTTP client is only a handful of lines of codes and works
quite well [4]. The utility functions used across all tests are located in
[5], and the tests themselves are in [6]. The existing test modules have a
1:1 correspondence with the associated JS suite test modules, and in
general are as direct of a port as possible.
The test modules ported in their entirety or most of the way are:
* all_docs.js
* basics.js
* config.js
* reduce.js
* rewrite.js
* uuids.js
* view_collation.js
Paul has dove in and is responsible for a few of those test modules and
he's almost completed porting the replication.js suite as well. We started
with the hard ones first, so for the most part the rest of the ports should
be fairly smooth sailing.
Here's an example of a very basic test:
```erlang
defmodule WelcomeTest do
use CouchTestCase
test "Welcome endpoint" do
assert Couch.get("/").body["couchdb"] == "Welcome", "Should say welcome"
end
end
```
As you can see, the `Couch` client is very simple HTTP client with
easy HTTP verb based methods. Let's look at a more complicated test
for asserting we can create documents in a database:
```erlang
@tag :with_db
test "Create a document and save it to the database", context do
resp = Couch.post("/#{context[:db_name]}", [body: %{:_id => "0",
:a => 1, :b => 1}])
assert resp.status_code == 201, "Should be 201 created"
assert resp.body["id"], "Id should be present"
assert resp.body["rev"], "Rev should be present"
resp2 = Couch.get("/#{context[:db_name]}/#{resp.body["id"]}")
assert resp2.body["_id"] == resp.body["id"], "Ids should match"
assert resp2.body["_rev"] == resp.body["rev"], "Revs should match"
end
```
This is fairly straightforward code to POST a new doc, make assertions
on the response, and then fetch the doc to make sure everything
matches up. What I really wanted to highlight here is the `@tag
:with_db` decorator. We can easily add custom "tags" to the tests to
simplify setup and teardown. That `:with_db` tag does two things, it
dynamically generates a random database name, and then takes care of
setup/teardown for creating and deleting said database for that
particular test. This is really useful and has been very nice to work
with so far. We also have tag functionality in place for executing a
test with a particular set of config options:
```erlang
@tag config: [
{"uuids", "algorithm", "utc_random"}
]
test "utc_random uuids are roughly random" do
resp = Couch.get("/_uuids", query: %{:count => 1000})
assert resp.status_code == 200
uuids = resp.body["uuids"]
assert String.length(Enum.at(uuids, 1)) == 32
# Assert no collisions
assert length(Enum.uniq(uuids)) == length(uuids)
# Assert rough ordering of UUIDs
u1 = String.slice(Enum.at(uuids, 1), 0..13
Re: [ANNOUNCE] Nick Vatamaniuc joins the PMC
Huzzah, congrats Nick! -Russell On Mon, Nov 13, 2017 at 5:27 AM Robert Kowalski wrote: > Welcome Nick, congrats! > > On Mon, Nov 13, 2017 at 1:24 PM, Andy Wenk wrote: > > Hey Nick - welcome on bord ;-) > > > > All the best > > > > Andy > > > > -- > > Andy Wenk > > Hamburg - Germany > > RockIt! > > > > GPG public key: > > http://pgp.mit.edu/pks/lookup?op=get&search=0x45D3565377F93D29 > > > > > > > >> On 13. Nov 2017, at 08:22, Garren Smith wrote: > >> > >> Welcome Nick. This is awesome. > >> > >> On Mon, Nov 13, 2017 at 7:30 AM, Nick Vatamaniuc > wrote: > >> > >>> Thanks everyone for the warm welcome. It's an honor to be a member of > the > >>> Apache CouchDB PMC. > >>> > >>> On Sun, Nov 12, 2017 at 8:29 AM, Robert Samuel Newson < > rnew...@apache.org> > >>> wrote: > >>> > Congratulations Nick, well deserved, and welcome aboard! > > B. > > > On 12 Nov 2017, at 03:39, Paul Davis > wrote: > > > > Hooray and welcome, Nick! > > > > > > On Sat, Nov 11, 2017 at 2:45 PM Joan Touzet > wrote: > > > >> Congratulations! Welcome, Nick! > >> > >> - Original Message - > >> From: "Alexander Shorin" > >> To: priv...@couchdb.apache.org > >> Sent: Saturday, 11 November, 2017 1:31:36 PM > >> Subject: Re: [ANNOUNCE] Nick Vatamaniuc joins the PMC > >> > >> Hooray to Nick! And welcome! > >> -- > >> ,,,^..^,,, > >> > >> > >> On Sat, Nov 11, 2017 at 4:40 PM, Jan Lehnardt > wrote: > >>> Forgot to CC private@ > >>> > Begin forwarded message: > > From: Jan Lehnardt > Subject: [ANNOUNCE] Nick Vatamaniuc joins the PMC > Date: 11. November 2017 at 17:38:39 GMT+1 > To: dev > Reply-To: dev@couchdb.apache.org > > Dear community, > > I am delighted to announce that Nick Vatamaniuc joins the Apache > >> CouchDB Project Management Committee today. > > Nick has made outstanding, sustained contributions to the project. > This > >> appointment is an official acknowledgement of their position within > >>> the > >> community, and our trust in their ability to provide oversight for > the > >> project. > > Everybody, please join me in congratulating Nick! > > On behalf of the CouchDB PMC, > Jan > — > > >>> > >> > > > >>> > > >
Re: [ANNOUNCE] Benjamin Anderson elected as CouchDB committer
Welcome aboard Ben! -Russell On Fri, Nov 4, 2016 at 6:10 PM, Benjamin Anderson wrote: > Thanks, everyone. Glad to be on board. :) > > -- > b > > On Wed, Nov 2, 2016 at 9:41 AM, Noah Slater wrote: > > Grats! Welcome! > > > > On Wed, 2 Nov 2016 at 05:23 Garren Smith wrote: > > > >> Awesome. Welcome Ben > >> > >> On Tuesday, 01 November 2016, Johannes Jörg Schmidt < > schm...@netzmerk.com> > >> wrote: > >> > >> > Welcome Benjamin! > >> > > >> > Am 01.11.2016 9:54 nachm. schrieb "Michelle Phung" < > >> > michelleph...@gmail.com > >> > >: > >> > > >> > > yay finally! :) & hi ben o/ > >> > > > >> > > - michelle > >> > > > >> > > > On Nov 1, 2016, at 4:45 PM, Jan Lehnardt >> > > wrote: > >> > > > > >> > > > Welcome Ben! :) > >> > > > > >> > > > > >> > > >> On 1 Nov 2016, at 20:46, Robert Samuel Newson < > rnew...@apache.org > >> > > > >> > > wrote: > >> > > >> > >> > > >> Dear community, > >> > > >> > >> > > >> I am pleased to announce that the CouchDB Project Management > >> Committee > >> > > has elected Benjamin Anderson as a CouchDB committer. > >> > > >> > >> > > >> Apache ID: banjiewen > >> > > >> > >> > > >> IRC nick: banjiewen > >> > > >> > >> > > >> Twitter: banjiewen > >> > > >> > >> > > >> Committers are given a binding vote in certain project > decisions, as > >> > > well as write access to public project infrastructure. > >> > > >> > >> > > >> This election was made in recognition of Ben's commitment to the > >> > > project. We mean this in the sense of being loyal to the project and > >> its > >> > > interests. > >> > > >> > >> > > >> Please join me in extending a warm welcome to Ben! > >> > > >> > >> > > >> On behalf of the CouchDB PMC, > >> > > >> > >> > > >> Robert Samuel Newson > >> > > >> > >> > > > > >> > > > -- > >> > > > Professional Support for Apache CouchDB: > >> > > > https://neighbourhood.ie/couchdb-support/ > >> > > > > >> > > > >> > > >> >
Re: Adding a node to cluster
Good catch, it should have included the db ;-)
-Russell
On Tuesday, September 6, 2016, Joey Samonte
wrote:
> In the example you gave for the replication, why is the target "
> test2.bar.com", and not "test2.bar.com/my_db"?
>
> > From: chewbra...@apache.org
> > Date: Tue, 6 Sep 2016 19:34:43 -0700
> > Subject: Re: Adding a node to cluster
> > To: dev@couchdb.apache.org
> >
> > CouchDB "replication" uses the HTTP API to replicate a database, whereas
> > the somewhat confusingly named "internal replication" is used within the
> > cluster itself as a way to synchronize state between the shard replicas.
> > The protocol for internal replication uses the Erlang distribution
> protocol
> > to communicate between nodes. So if your cluster is a connected set of
> > Erlang nodes then they will be able to synchronize database/shard state
> > amongst themselves.
> >
> >
> > As an example, consider a single node 1.x CouchDB server at
> > test1.foo.com:5984 where you want to replicate to a CouchDB 2.0 cluster
> at
> > test2.bar.com that has three db nodes and a load balancer, with the load
> > balancer actually having the test2.bar.com address. You communicate with
> > the cluster by way of the load balancer which will distribute requests to
> > all the nodes. The details of the cluster should be mostly transparent at
> > the HTTP layer, and so you can just do something like:
> >
> > curl -i -X POST test1.foo.com/_replicator -d '{"source":"
> test1.foo.com/my_db",
> > "target":"test2.bar.com"}'
> >
> > This will replicate "my_db" from test1 to test2 using the HTTP protocol
> > like you would expect from CouchDB. Within the test2 cluster the nodes
> will
> > communicate amongst themselves using internal replication (non HTTP) to
> > ensure the data is fully synchronized between the shard replicas.
> >
> > Hopefully that clarifies things a bit.
> >
> >
> > -Russell
> >
> >
> >
> > On Tuesday, September 6, 2016, Joey Samonte >
> > wrote:
> >
> > > I think I understand it. Hoping to find more documentation on it.
> > >
> > > > From: wickedg...@gmail.com
> > > > Date: Tue, 6 Sep 2016 15:55:00 -0700
> > > > Subject: Re: Adding a node to cluster
> > > > To: dev@couchdb.apache.org
> > > >
> > > > Nodes in the cluster do not replicate to one another.
> > > >
> > > > Replication takes place between databases. A single node isn't a
> > > > database; a clustered database spans multiple nodes.
> > > >
> > > > Each node has a black-box lump of data that happens to have a
> fraction
> > > > of a database inside, but that's an implementation detail. It's the
> > > > wrong level of abstraction to work with.
> > > >
> > > > On Tue, Sep 6, 2016 at 1:52 PM, Joey Samonte
> > > > > wrote:
> > > > > If you have two nodes in the cluster, the replication should be
> > > two-way between the nodes?
> > > > > What if there are three nodes? How should the replication be setup
> > > between them?
> > > > >
> > > > >> Subject: Re: Adding a node to cluster
> > > > >> From: sebastianrothbuc...@googlemail.com
>
> > > > >> Date: Mon, 5 Sep 2016 23:08:04 +0200
> > > > >> To: dev@couchdb.apache.org
> > > > >>
> > > > >> Hi,
> > > > >>
> > > > >> clustering and replication are indeed two (very) separate things -
> > > and you won't get a Cluster by setting up replication. Again: treat
> the two
> > > as separate. Clustering turns several shards (on several nodes) into
> one
> > > database (from an user/caller perspective) while replication happens
> > > _between_ databases.
> > > > >> Consequently, technical underpinnings differ as well, as Bob
> > > explained below.
> > > > >>
> > > > >> Hope that gets things in perspective a little...
> > > > >>
> > > > >> Best
> > > > >> Sebastian
> > > > >>
> > > > >> Von meinem iPhone gesendet
> > > > >>
> > > > >> > Am 05.09.2016 um 22:47 schrieb Joey Samonte <
> > > csharpdevelo...@hotmail.com >:
> > > > >> >
> > > > >> > Does this mean that setting up replication is separate from
> setting
> > > up clustering?
> > > > >> >
> > > > >> > Does replication needs to be bi-directional between nodes?
> > > > >> >
> > > > >> >> From: rnew...@apache.org
> > > > >> >> Subject: Re: Adding a node to cluster
> > > > >> >> Date: Thu, 25 Aug 2016 11:10:45 +0100
> > > > >> >> To: dev@couchdb.apache.org
> > > > >> >>
> > > > >> >> Ok, seems I've confused you.
> > > > >> >>
> > > > >> >> Couchdb replication occurs over http or https, as you know. The
> > > nodes in a couchdb 2.0 cluster do not communicate with each other over
> > > http. They use Erlang rpc. Erlang rpc can be configured for TLS
> > > encryption. It's in the Erlang faq and is fairly simple to set up in
> newer
> > > Erlang releases.
> > > > >> >>
> > > > >> >> I feel I owe an example of 2.0 cluster that exclusively uses
> TLS
> > > for all communications.
> > > > >> >>
> > > > >> >> Sent from my iPhone
> > > > >> >>
> > > > >> >>> On 24 Aug 2016, at 20:47, Joey Samonte <
> > > csharpdevelo...@hotmail.com > wrote:
> > > > >> >>>
> > > > >> >>>
Re: Adding a node to cluster
CouchDB "replication" uses the HTTP API to replicate a database, whereas
the somewhat confusingly named "internal replication" is used within the
cluster itself as a way to synchronize state between the shard replicas.
The protocol for internal replication uses the Erlang distribution protocol
to communicate between nodes. So if your cluster is a connected set of
Erlang nodes then they will be able to synchronize database/shard state
amongst themselves.
As an example, consider a single node 1.x CouchDB server at
test1.foo.com:5984 where you want to replicate to a CouchDB 2.0 cluster at
test2.bar.com that has three db nodes and a load balancer, with the load
balancer actually having the test2.bar.com address. You communicate with
the cluster by way of the load balancer which will distribute requests to
all the nodes. The details of the cluster should be mostly transparent at
the HTTP layer, and so you can just do something like:
curl -i -X POST test1.foo.com/_replicator -d '{"source":"test1.foo.com/my_db",
"target":"test2.bar.com"}'
This will replicate "my_db" from test1 to test2 using the HTTP protocol
like you would expect from CouchDB. Within the test2 cluster the nodes will
communicate amongst themselves using internal replication (non HTTP) to
ensure the data is fully synchronized between the shard replicas.
Hopefully that clarifies things a bit.
-Russell
On Tuesday, September 6, 2016, Joey Samonte
wrote:
> I think I understand it. Hoping to find more documentation on it.
>
> > From: wickedg...@gmail.com
> > Date: Tue, 6 Sep 2016 15:55:00 -0700
> > Subject: Re: Adding a node to cluster
> > To: dev@couchdb.apache.org
> >
> > Nodes in the cluster do not replicate to one another.
> >
> > Replication takes place between databases. A single node isn't a
> > database; a clustered database spans multiple nodes.
> >
> > Each node has a black-box lump of data that happens to have a fraction
> > of a database inside, but that's an implementation detail. It's the
> > wrong level of abstraction to work with.
> >
> > On Tue, Sep 6, 2016 at 1:52 PM, Joey Samonte
> > > wrote:
> > > If you have two nodes in the cluster, the replication should be
> two-way between the nodes?
> > > What if there are three nodes? How should the replication be setup
> between them?
> > >
> > >> Subject: Re: Adding a node to cluster
> > >> From: sebastianrothbuc...@googlemail.com
> > >> Date: Mon, 5 Sep 2016 23:08:04 +0200
> > >> To: dev@couchdb.apache.org
> > >>
> > >> Hi,
> > >>
> > >> clustering and replication are indeed two (very) separate things -
> and you won't get a Cluster by setting up replication. Again: treat the two
> as separate. Clustering turns several shards (on several nodes) into one
> database (from an user/caller perspective) while replication happens
> _between_ databases.
> > >> Consequently, technical underpinnings differ as well, as Bob
> explained below.
> > >>
> > >> Hope that gets things in perspective a little...
> > >>
> > >> Best
> > >> Sebastian
> > >>
> > >> Von meinem iPhone gesendet
> > >>
> > >> > Am 05.09.2016 um 22:47 schrieb Joey Samonte <
> csharpdevelo...@hotmail.com >:
> > >> >
> > >> > Does this mean that setting up replication is separate from setting
> up clustering?
> > >> >
> > >> > Does replication needs to be bi-directional between nodes?
> > >> >
> > >> >> From: rnew...@apache.org
> > >> >> Subject: Re: Adding a node to cluster
> > >> >> Date: Thu, 25 Aug 2016 11:10:45 +0100
> > >> >> To: dev@couchdb.apache.org
> > >> >>
> > >> >> Ok, seems I've confused you.
> > >> >>
> > >> >> Couchdb replication occurs over http or https, as you know. The
> nodes in a couchdb 2.0 cluster do not communicate with each other over
> http. They use Erlang rpc. Erlang rpc can be configured for TLS
> encryption. It's in the Erlang faq and is fairly simple to set up in newer
> Erlang releases.
> > >> >>
> > >> >> I feel I owe an example of 2.0 cluster that exclusively uses TLS
> for all communications.
> > >> >>
> > >> >> Sent from my iPhone
> > >> >>
> > >> >>> On 24 Aug 2016, at 20:47, Joey Samonte <
> csharpdevelo...@hotmail.com > wrote:
> > >> >>>
> > >> >>> What if we remove the reverse proxy and just set up the CouchDB
> nodes to allow only SSL connections, port 6984? https://wiki.apache.org/
> couchdb/How_to_enable_SSL
> > >> >>>
> > >> Subject: Re: Adding a node to cluster
> > >> From: rnew...@apache.org
> > >> Date: Wed, 24 Aug 2016 19:43:51 +0100
> > >> To: dev@couchdb.apache.org
> > >>
> > >> Assuming you mean a 2.0 cluster, no, all those nodes need to be
> able to communicate with erlang rpc (service discovery over port 4369 and
> then whatever port the node is running ong).
> > >>
> > >> > On 24 Aug 2016, at 12:36, Joey Samonte <
> csharpdevelo...@hotmail.com > wrote:
> > >> >
> > >> > Good day,
> > >> >
> > >> > Is it possible to add a node to a cluster from Fauxton if the
> remote host is behind a reverse proxy (nginx)
Re: Getting libraries to test RCs
N=1 might reduce some of the issues, but it won't eliminate the problem entirely. The fundamental issue is that the "_dbs" db, which contains a document corresponding to every clustered database in the system, does not provide immediate consistency guarantees, and cycling databases can result in conflicts arising in these docs. The docs contain the shard/node mappings and conflicts can cause different nodes to have different views of the world. It's important to remember that the "_dbs" db powers the db -> shards mapping and is a fundamental component of the quorum system, so unfortunately the standard clustered quorum semantics are not available in the "_dbs" db as it operates at a lower level. You can see the initial synchronization during bootup in [1] which circles its way back to [2] by way of mem3_sync_nodes.erl. You can further see where the "_dbs" db is a local db in the way in which shards are loaded in [3] and the fallback for creating the "_dbs" db in [4]. So in summary, the "_dbs" db operates at a lower level than the quorum system as the db is a core component that powers the shard mappings, and therefore uses a different approach for synchronization where each node has a full copy of the "_dbs" db and syncs directly with the other nodes. This is a known weak point as can be seen by the impact of cycling databases too quickly, and so recommended best practice is to not cycle databases quickly. Obviously this is not ideal, and this is one of the areas where a CP config store of some sort would be a significant boon, but bolting on a CP system to an AP system is fraught with a new set of complexities. (A clarification on N=1: with N=1 you only have one replica of the database, and the database exists on only one node. The rest of the nodes still need to get the updated "_dbs" db doc so they know where the database exists, because any node in the cluster can handle any request and it will need to know where the database exists. In general, you have one coordinating node and N replica nodes containing the N replicas (of each shard) for the given database. In a three node cluster with N=3, whatever coordinating node the request is handled by will also have a local shard replica, but this is a special case. In a cluster with more than 3 nodes, say 15 nodes, the coordinating node will only have a 3/15 chance to contain a local shard (assuming round robin load balancing across nodes). So basically every node must know where every database exists because every node can coordinate every request.) -Russell [1] https://github.com/apache/couchdb-mem3/blob/15615b295ec970ca9b12b7b54107a80b95149511/src/mem3_sync.erl#L234-L236 [2] https://github.com/apache/couchdb-mem3/blob/15615b295ec970ca9b12b7b54107a80b95149511/src/mem3_sync.erl#L230-L232 [3] https://github.com/apache/couchdb-mem3/blob/699308f510d335d05bfd0416ad5e893b68a7ec1d/src/mem3_shards.erl#L266-L283 [4] https://github.com/apache/couchdb-mem3/blob/699308f510d335d05bfd0416ad5e893b68a7ec1d/src/mem3_util.erl#L214-L222 On Fri, Sep 2, 2016 at 10:43 AM, Nolan Lawson wrote: > Thanks, Dale. That was my recollection as well. > > Basically PouchDB does PUT -> DELETE -> PUT between every test, so since > there are 1000s of tests, this race condition comes up pretty easily. We > can add a timeout or do a random DB name, but without doing that we don't > know if Couch 2.x is truly "passing" the test suite or not. > > I have some time this weekend, so I'll look into adding a patch to do the > workaround for Couch 2. I tend to side with Jan that in a clustered system > it can't reliably tell us when a database was truly deleted without > sacrificing the A in CAP. PouchDB users are already familiar with the weird > ways that databases start to behave when you actually DELETE them (e.g. > replication gets unreliable), hence workarounds like > https://www.npmjs.com/package/pouchdb-erase . In practice I expect PouchDB > users to never delete databases, so this is just an artifact of our test > suite IMO. > > –Nolan > > > On Fri, Sep 2, 2016 at 3:14 AM, Dale Harvey wrote: > > > In PouchDB we can look into a workaround that uses random names only when > > the tests are run against Couch 2.0, however I would really like to make > > sure that a database not being fully deleted when we get a successful > > confirmation of deletion is considered a bug, it has impacts beyond the > > test suite, its really hard to create a reliable system when there is no > > way for you to be certain when a database is deleted. > > > > Will found it easiest to reproduce this using concurrent scripts but > would > > like to clarify that Pouch doesnt run the test suite in parallel, this > bug > > can be hit by doing CREATE -> DELETE -> CREATE, its extremely hard to > nail > > down and reproduce (the similiar bug in PouchDB took many attempts + > > months). I will take a look at seeing if I can make an easier and clearer > > steps to reproduce. > > > > On 2 September 2016 at 11:01, Jan
Re: [ANNOUNCE] Garren Smith joins the PMC
Congrats Garren!! Glad to have you aboard. -Russell On Monday, October 19, 2015, Klaus Trainer wrote: > Congrats, Garren :) > > > On 10/19/2015 12:31 PM, Jan Lehnardt wrote: > > Dear community, > > > > I am delighted to announce that Garren Smith joins the Apache CouchDB > Project Management Committee today. > > > > Garren has made outstanding, sustained contributions to the project. > This appointment is an official acknowledgement of their position within > the community, and our trust in their ability to provide oversight for the > project. > > > > Everybody, please join me in congratulating Garren! > > > > On behalf of the CouchDB PMC, > > Jan > > >
Re: [ANNOUNCE] Michelle Phung joins the PMC
Woohoo, congrats Michelle!! -Russell On Monday, October 19, 2015, Klaus Trainer wrote: > Congrats, Michelle :) > > > On 10/19/2015 11:52 AM, Jan Lehnardt wrote: > > Dear community, > > > > I am delighted to announce that Michelle Phung joins the Apache CouchDB > Project Management Committee today. > > > > Michelle has made outstanding, sustained contributions to the project. > This appointment is an official acknowledgement of their position within > the community, and our trust in their ability to provide oversight for the > project. > > > > Everybody, please join me in congratulating Michelle! > > > > On behalf of the CouchDB PMC, > > Jan > > >
Re: 2.0 Progress
On Thursday, June 18, 2015, Jan Lehnardt wrote: > > > On 17 Jun 2015, at 17:50, Jan Lehnardt > > wrote: > > > > Hey all, > > > > Alexander, Bob and I had a bit of a brainstorming session today on what > is missing to get 2.0 out the door. We talked about missing features and > what to do about them. The following notes is what we think is best, but of > course, these are just suggestions and we’d love your feedback! Especially > on the TBD items. > > > > The notes are dense, let us know if you have any questions or need any > clarification :) > > > > * * * > > > > # Missing Features > > > > ## Availble in 1.x but missing in 2.x > > > > - [X] vhosts: Done. > > > > - [X] /_config on :5984 > > - no cluster wide config > > - per-node config can be set from any node: > > - only expose /_node_config/// > > - first stab of this already in master > > - Fauxton to offer “tabbed” interface for per-cluster config view > > - post 2.0, if we find a backing store that would give us a > cluster-consistent :5984/_config/section/key, we can enable that. > > > > > > - [X]: /_stats on :5984 > > - same as with /_config: > > - /_node_stats// > > > Updates on _node_config and _node_stats > > We will now propose /_node//_config and /_node//_stats with > the option of adding more endpoints there, later. > > The rest stays as is. +1 -Russell > > Best > Jan > -- > > > > > > > - [X] dynamic http endpoint configuration for :5984 > > - integrate Cloudant’s work that allows for erlang applications to > register their own routes in a priv/ file. Adding a route is done by adding > an app, changing a route is by updating an app and its priv/ file > > - /_config can be used to disable endpoints > > - TBD: what if two applications define the same endpoint? Options: > > - undefined > > - offending app doesn’t load > > - server doesn’t start / stops > > - define that endpoints can’t be overwritten, define strict sort > order for loading, load core apps preferred to ensure working node + > *handwaving* > > > > > > - [X] externals & proxy > > - keep both for 2.0 > > - add to chttpd routes loading: > > - 1. load all application routes > > - 2. load [external] and [proxy] routes > > - 3. listen to [external] and [proxy] config changes and adjust route > map at runtime > > > > > > - [X] logging configuration of level & file & apps ( > http://docs.couchdb.org/en/latest/config/logging.html) > > - dynamic configuration of the log file won’t be possible anymore, > because lager works differently > > - since the configuration for lager is more comprehensive, 1.x-style > configuration is no longer possible > > - TBD: however, it is possible to set the log level at runtime. Two > options: > > - setting /_config/log:backend/level to / couch_log > listens to config updates and calls > https://github.com/basho/lager#runtime-loglevel-changes > > - new endpoint, e.g. PUT /_log/backend/ > > - log atoms change, mark as BC break > > > > > > - [X] rewrites (?) > > - keep, should just work > > > > > > - [X] documentation > > - we have (or will have soon): API docs, upgrade/migration guides / new > features > > - we *don’t* have: how the cluster works (dynamo + couch specific > details) > >- needs help! > > > > - [X ] build script > > - finish unix version (Jan) > > - integrate with Mac app (Jan) > > - TBD: Windows > > - Can Nick North help? > > - maybe ship 2.0 with experimental Windows, since the code has > never run on Windows(?) > > > > > > ## Missing in 2.x > > > > - [X] single node mode > > - “single node mode” just means “cluster of one” > > - the chttp-layer might be slower than couch_httpd, if so, we have good > incentives to make this fast. > > - going from 1.x to 2.x-single-node will requires changes in how an app > uses the CouchDB API (as per our Breaking Changes documentation). > > - :5986 is 127.0.0.1-only and strongly discouraged for general use, ops > folks might want to use it. > > > > > > - [X] add/remove nodes, cluster rebalance > > - add/remove node needs documentation > > - rebalancing: > > - tool to help figure out what needs doing > > - might be ready by 2.0, post 2.0 otherwise > > > > * * * > > > > Best > > Jan > > -- > > > > -- > Professional Support for Apache CouchDB: > http://www.neighbourhood.ie/couchdb-support/ > >
Re: On Plugins and Extensibility
Hey Paul, Thanks for the great writeup! Couple of questions: How do priv/*.cfg files work with dynamic config updates? Seems like we'll lose the ability to write changes back to the config file, like we have with default.ini. Although I've been wondering for a while if allowing config:set/* to persist data is an anti-pattern compared to forcing persistent updates to be made directly in config files with the hopes they are done so in a VCS. Also, if we switch to priv/*.cfg how will users update those files? seems like we'll have to poll the file system for every one of those config files, or are you thinking that those files would only be updated as part of releases? Have you thought at all about using the mochiglobal.erl approach to regenerate a module with all the plugin dispatch compiled as functions? Seems like using shared ETS tables could become a source of contention for lots of concurrent requests. Proper benchmarks would provide better understanding as to whether this is actually an issue. -Russell On Thu, May 21, 2015 at 3:29 PM, Paul Davis wrote: > Hey everyone, > > So I've been meaning to write this email for sometime but have been > kept busy with lots of super fun things that are super fun. Anyway, I > just wanted to get this out there to start getting feed back from > everyone involved. > > Also, while this is called "Plugin Proposal" it shouldn't be confused > with the original couch_plugins use case. This is a lot lower level > (and may be used by something like couch_plugins if we go there again > in the future). Generally speaking this is just for cleaning up > internals and for people that want to run either minimal installs of > CouchDB or include the CouchDB in a larger Erlang application. > > Plugin Proposal > === > > Background > -- > > As we've grown the code base to include more and more applications > we're getting to the point where we've started adding various points > of extension in various ways. The best existing example is the > couch_stats application which loads stat/metric definitions from > applications. Henning Diedrich has some unmerged worked which looks to > follow a similar path for HTTP URL handlers. And Ilya Khlopotov has > some work for providing vendor specific hooks. > > While each of these have some overlaps in their intended use case, > they also share the fact that they've all implemented their own idea > of extensibility in slightly different ways. That's not necessarily > bad, but I think that we could reduce a lot of complexity if we take a > step back and write a utility application that could then be used to > support each of these features so that we can have both the > extensibility as well as simplify the implementation of each > individual feature. > > I'll start with a bit of background and then describe a general > approach as well as show some hopefully explicit example snippets of > how such a system might be used. Granted I haven't written out an > entire implementation of this so I may be off the mark in some places. > > Bikeshed First > -- > > I have no idea what we'd call this. We could repurpose the > couch_plugins app conceivably or make something new. For the the > purposes of this document I'll call it couch_epi (for extensible > plugin interface) and hopefully that's terrible enough someone will > think of a better name for the actual application. > > Requirements > > > The three major requirements I've thought of are: > > # Automatically discoverable > # Minimize apps that need to be started for tests > # Support release upgrades > > === Automatically Discoverable === > > The biggest thing here is that I don't want to require a change to a > default.ini or similar to enable or disable specific functionality > when we can already signify that by having the application present or > not. This is both for groups that may want to add new Erlang > applications to a release as well as anyone that wants to run a > minimal/embedded Couch. These are both obviously advanced uses but I > think are important given the number of ways that CouchDB is being > used. > > === Minimize the apps that need to be started for tests === > > This one I think should be obvious to anyone that's been writing unit > tests lately. There are some often silly places where we require > applications be started just to run some tests. For example, places > where we may want to call a function that's been instrumented and > requires couch_stats to have knowledge about the stat. > > === Support release upgrades === > > This one is obviously fairly advanced and limited in its audience but > its something I'd like to at least consider in the design. This comes > into effect for things like couch_stats that use a text file for its > extension method. The issue is that the release upgrade mechanics > don't provide any sort of signal that is easily usable to indicate > when this file has changed during an upgrade so w
[jira] [Commented] (COUCHDB-2657) if _metadata is there, creating a db does not work
[ https://issues.apache.org/jira/browse/COUCHDB-2657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500885#comment-14500885 ] Russell Branca commented on COUCHDB-2657: - I tracked this down to making a `fabric:open_doc` call from within the `cassim_metadata_cache` gen_server. The problem is that `fabric:open_doc` does a `receive` call while waiting for a worker response, and this intercepts any other messages sent in. I've got a patch out [1] that shows the rough solution. I did notice however that in the test case we're still getting a 409 error on the second request, so we'll want to make sure we handle that properly. Should we check and see if the values are the same? and if not reload? That might be worthwhile but potentially expensive. Alternatively we could automatically create the security doc on db creation. Also, I think COUCHDB-2659 might be a duplicate of this issue. [1] https://github.com/apache/couchdb-cassim/tree/2657-fix-cassim-fabric-calls > if _metadata is there, creating a db does not work > -- > > Key: COUCHDB-2657 > URL: https://issues.apache.org/jira/browse/COUCHDB-2657 > Project: CouchDB > Issue Type: Bug > Security Level: public(Regular issues) > Components: BigCouch, Database Core >Reporter: Robert Kowalski >Priority: Blocker > > with the new _setup_cluster endpoint a _metadata db is created. > when it exists, and we create a db, read it _all_doc immediatly [1] (Fauxton > does that) we get: > `document update conflict` > [1] > http://localhost:8000/sdfsdtest/_all_docs?startkey=%22_design%2F%22&endkey=%22_design0%22&include_docs=true&limit=501 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (COUCHDB-2659) CouchDB master failing the PouchDB test suite - live changes
[
https://issues.apache.org/jira/browse/COUCHDB-2659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500886#comment-14500886
]
Russell Branca commented on COUCHDB-2659:
-
Is this a duplicate of COUCHDB-2657?
> CouchDB master failing the PouchDB test suite - live changes
>
>
> Key: COUCHDB-2659
> URL: https://issues.apache.org/jira/browse/COUCHDB-2659
> Project: CouchDB
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: Database Core
>Reporter: Nolan Lawson
>
> See [this comment |
> https://github.com/pouchdb/pouchdb/pull/3722#issuecomment-91571425] for
> details. We have automated tests that pull in the latest CouchDB master
> branch and run the PouchDB test suite against it, and recently our tests
> started failing due to [apache/couchdb#313 |
> https://github.com/apache/couchdb/pull/313]. So I switched back to admin
> party mode, but now it's failing for another reason:
> {quote}
> 1) test.attachments.js-http #3074 live changes():
> Uncaught Database encountered an unknown error
> {quote}
> Instructions for running the PouchDB test suite are in our TESTING.md, it's
> pretty straightforward but basically you just want to point it to any CouchDB
> server like so and run the tests:
> {code}
> BAIL=0 COUCH_HOST=http://localhost:15984 npm test
> {code}
> {{BAIL=0}} will prevent bailing upon the first error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (COUCHDB-2657) if _metadata is there, creating a db does not work
[ https://issues.apache.org/jira/browse/COUCHDB-2657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500805#comment-14500805 ] Russell Branca commented on COUCHDB-2657: - Also the error I'm seeing with this is: [<<"gen_server:call/2 L182">>,<<"cassim_security:get_security_doc/1 L58">>,<<"cassim_security:get_security/2 L46">>,<<"chttpd_db:do_db_req/2 L263">>,<<"chttpd:handle_request/1 L210">>,<<"mochiweb_http:headers/5 L93">>,<<"proc_lib:init_p_do_apply/3 L237">>] > if _metadata is there, creating a db does not work > -- > > Key: COUCHDB-2657 > URL: https://issues.apache.org/jira/browse/COUCHDB-2657 > Project: CouchDB > Issue Type: Bug > Security Level: public(Regular issues) > Components: BigCouch, Database Core >Reporter: Robert Kowalski >Priority: Blocker > > with the new _setup_cluster endpoint a _metadata db is created. > when it exists, and we create a db, read it _all_doc immediatly [1] (Fauxton > does that) we get: > `document update conflict` > [1] > http://localhost:8000/sdfsdtest/_all_docs?startkey=%22_design%2F%22&endkey=%22_design0%22&include_docs=true&limit=501 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (COUCHDB-2657) if _metadata is there, creating a db does not work
[ https://issues.apache.org/jira/browse/COUCHDB-2657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500801#comment-14500801 ] Russell Branca commented on COUCHDB-2657: - Thanks for the report [~robertkowalski]. I ported the test script to Python [1] and I'm able to reproduce these issues. Deleting the _metadata db and rerunning the test works, but I haven't been able to test a fresh start of the dev cluster without the _metadata db as it keeps getting recreated. [~ilyak] where are you observing those failures? Is that during the first run of setup? I'm not seeing those. [1] https://gist.github.com/chewbranca/44e266140cbb74db9c47 > if _metadata is there, creating a db does not work > -- > > Key: COUCHDB-2657 > URL: https://issues.apache.org/jira/browse/COUCHDB-2657 > Project: CouchDB > Issue Type: Bug > Security Level: public(Regular issues) > Components: BigCouch, Database Core >Reporter: Robert Kowalski >Priority: Blocker > > with the new _setup_cluster endpoint a _metadata db is created. > when it exists, and we create a db, read it _all_doc immediatly [1] (Fauxton > does that) we get: > `document update conflict` > [1] > http://localhost:8000/sdfsdtest/_all_docs?startkey=%22_design%2F%22&endkey=%22_design0%22&include_docs=true&limit=501 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSSION] Move Fauxton to its own mailing list?
+1 On Tue, Apr 14, 2015 at 2:24 PM, Paul J Davis wrote: > +1 > > > > > On Apr 14, 2015, at 3:50 PM, Andy Wenk wrote: > > > > +1 > > > >> On 14 April 2015 at 21:02, Alexander Shorin wrote: > >> > >>> On Tue, Apr 14, 2015 at 5:09 PM, Jan Lehnardt wrote: > >>> We create new mailing list c...@couchdb.apache.org (or your favourite > >> bike shed) that gets all JIRA and GitHub traffic. > >>> > >>> dev@ then is for discussion and decisions for all code projects > >> (including Fauxton). > >>> > >>> If you want to keep track of tickets and pull requests, sign up for > code@ > >> . > >>> > >>> There is a bit of a discrepancy between discussions in JIRA and dev@, > >> but I’m willing to take that risk, as most people probably bulk-delete > all > >> JIRA mails anyway :) > >> > >> +1 > >> > >> > >> -- > >> ,,,^..^,,, > > > > > > > > -- > > Andy Wenk > > Hamburg - Germany > > RockIt! > > > > GPG fingerprint: C044 8322 9E12 1483 4FEC 9452 B65D 6BE3 9ED3 9588 > > > > https://people.apache.org/keys/committer/andywenk.asc >
Re: [PROPOSAL] Move transactional email out of dev@
Seems a bit odd to separate github PR comments from JIRA comments as the overlap on development related discussion can be large. As for actually separating out the messages, I'm -0. Email filters work really well for this sort of thing. -Russell On Tuesday, March 17, 2015, till wrote: > We use this in-house: https://github.com/heroku/devdigest/ > > Gives you the daily run-down on what happened on a set of given repos or in > an org. It's tweak-able (= open source, Ruby code). > > Till > > On Tue, Mar 17, 2015 at 4:40 PM, Noah Slater > wrote: > > > I am +1 on a monthly reminder. Do we have a volunteer to code that up? > > Could even be a weekly reminder. I'm imagining it would contain a > > summary of all open/active PRs, so people might be like "ooh, I am > > interested in that one" while the discussion is still happening! > > > > On 16 March 2015 at 23:04, Sebastian Rothbucher > > > wrote: > > > +1 with Joan and Alexander also, i.e. do a monthly thing instead of > > > blasting out dozens a day to dev@ => this is great, thanks for > bringing > > it > > > up! Sebastian > > > > > > On Mon, Mar 16, 2015 at 6:42 PM, Alexander Shorin > > > wrote: > > > > > >> On Mon, Mar 16, 2015 at 7:46 PM, Joan Touzet > wrote: > > >> > I'd be in support of an automated email monthly to dev@ that > reminds > > >> > people where to go to look for GH PRs, JIRA ticket updates, etc. > > >> > > >> Good idea btw. +1 > > >> > > >> -- > > >> ,,,^..^,,, > > >> > > > > > > > > -- > > Noah Slater > > https://twitter.com/nslater > > >
[jira] [Created] (COUCHDB-2629) Hide internal config details from other applications
Russell Branca created COUCHDB-2629:
---
Summary: Hide internal config details from other applications
Key: COUCHDB-2629
URL: https://issues.apache.org/jira/browse/COUCHDB-2629
Project: CouchDB
Issue Type: Bug
Security Level: public (Regular issues)
Components: Database Core
Reporter: Russell Branca
Throughout the code base we do things like `config:get("couch_httpd_auth",
"authentication_db", "_users")` which exposes internal details to other
applications that don't need to be aware of the particulars. This duplicates
code unnecessarily and causes problematic typos to go unnoticed like in
COUCHDB-2628.
One possible approach that I'm a fan of is to go into the application
responsible for relevant config value, and add a function to hide the details.
For instance, this is done in `cassim_metadata_cache:metadata_db`:
https://github.com/apache/couchdb-cassim/blob/windsor-merge/src/cassim_metadata_cache.erl#L58-L59
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (COUCHDB-2628) Resolve shard(s)_db naming discrepancies
[ https://issues.apache.org/jira/browse/COUCHDB-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14338879#comment-14338879 ] Russell Branca commented on COUCHDB-2628: - +1, thanks [~kxepal]! > Resolve shard(s)_db naming discrepancies > > > Key: COUCHDB-2628 > URL: https://issues.apache.org/jira/browse/COUCHDB-2628 > Project: CouchDB > Issue Type: Bug > Security Level: public(Regular issues) > Components: BigCouch > Reporter: Russell Branca > > Looks like there's a handful of discrepancies where `shard_db` is referenced > rather than `shards_db` which is the version actually used when creating the > relevant database [1]. A quick grep around shows about 8 uses of `shard_db`. > [1] > https://github.com/apache/couchdb-mem3/blob/master/src/mem3_shards.erl#L228-L229 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (COUCHDB-2628) Resolve shard(s)_db naming discrepancies
Russell Branca created COUCHDB-2628: --- Summary: Resolve shard(s)_db naming discrepancies Key: COUCHDB-2628 URL: https://issues.apache.org/jira/browse/COUCHDB-2628 Project: CouchDB Issue Type: Bug Security Level: public (Regular issues) Components: BigCouch Reporter: Russell Branca Looks like there's a handful of discrepancies where `shard_db` is referenced rather than `shards_db` which is the version actually used when creating the relevant database [1]. A quick grep around shows about 8 uses of `shard_db`. [1] https://github.com/apache/couchdb-mem3/blob/master/src/mem3_shards.erl#L228-L229 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [VOTE] accept Mango contribution from Cloudant IBM
+1 :D On Friday, December 19, 2014, Jan Lehnardt wrote: > +1, what a great holiday present :) > > Best > Jan > -- > > > On 18 Dec 2014, at 23:36 , Robert Kowalski > wrote: > > > > Hi list, > > > > Cloudant IBM wants to contribute the Mango code, a MongoDB API layer > > for CouchDB to the ASF. This needs a vote of the CouchDB project. The > > tarball is at > https://people.apache.org/~robertkowalski/dist/ip-clearance/mango-asf-ip-clearance-2015-12-18.tar.gz > > > > The SHA is: > > > > SHA512(mango-asf-ip-clearance-2015-12-18.tar.gz)= > > > 809b28fb0ccdab19401fb15a4d0f663ee7a51d2c6cbea4cd5e20621dfac16862f186c0375a3ef485775f78bbc29d88e42f7a6d8abe2d1ef20e96e226298f37ee > > > > > > The main repository is located at https://github.com/cloudant/mango - > > I also created the tarball from that source. > > > > We don't have a match for votings on code contributions in our bylaws, > > so I am chosing Majority Approval by the PMC (that is, 3 +1’s and no > > -1’s) and a 72 hour voting period beginning now (like the BigCouch > > vote in August) > > > > Best, > > Robert > >
[jira] [Commented] (COUCHDB-2422) Cassim and underscored databases
[
https://issues.apache.org/jira/browse/COUCHDB-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14216700#comment-14216700
]
Russell Branca commented on COUCHDB-2422:
-
A simple fix here is to just prepend a string like "db/" to the front of the id
[1]. This would solve the problem but I think we should also do this to better
represent that the document corresponds to settings for a database. This would
make it easier to distinguish between database specific docs and global docs.
[1]
https://github.com/apache/couchdb-cassim/blob/master/src/cassim_metadata_cache.erl#L73
> Cassim and underscored databases
>
>
> Key: COUCHDB-2422
> URL: https://issues.apache.org/jira/browse/COUCHDB-2422
> Project: CouchDB
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: Database Core
>Reporter: Alexander Shorin
>
> 1. Run dev cluster
> 2. Create {{_users}} database
> {code}
> http PUT http://localhost:15984/_users
> HTTP/1.1 201 Created
> Cache-Control: must-revalidate
> Content-Length: 12
> Content-Type: text/plain; charset=utf-8
> Date: Fri, 31 Oct 2014 10:36:19 GMT
> Location: http://localhost:15984/_users
> Server: CouchDB/9938fac (Erlang OTP/17)
> X-Couch-Request-ID: f273be89
> X-CouchDB-Body-Time: 0
> {"ok":true}
> {code}
> 3. Create cassim database
> {code}
> http PUT http://localhost:15984/cassim
> HTTP/1.1 201 Created
> Cache-Control: must-revalidate
> Content-Length: 12
> Content-Type: text/plain; charset=utf-8
> Date: Fri, 31 Oct 2014 10:37:06 GMT
> Location: http://localhost:15984/cassim
> Server: CouchDB/9938fac (Erlang OTP/17)
> X-Couch-Request-ID: 8bdcc974
> X-CouchDB-Body-Time: 0
> {"ok":true}
> {code}
> 4. Try to access {{_users}} database:
> {code}
> http http://localhost:15984/_users
> HTTP/1.1 400 Bad Request
> Cache-Control: must-revalidate
> Content-Length: 89
> Content-Type: text/plain; charset=utf-8
> Date: Fri, 31 Oct 2014 10:34:22 GMT
> Server: CouchDB/9938fac (Erlang OTP/17)
> X-Couch-Request-ID: e9f4a5b3
> X-CouchDB-Body-Time: 0
> {"error":"bad_request","reason":"Only reserved document ids may start with
> underscore."}
> {code}
> Oh wait...? This is because cassim creates documents with database name as
> the prefix and for {{_users}} and {{_replicator}} it clashes with
> restrictions about document names.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [DISCUSS] Removing "delayed_commits"
Jan Lehnardt writes: >> On 10 Nov 2014, at 16:11 , Alexander Shorin wrote: >> >> On Mon, Nov 10, 2014 at 6:06 PM, Jan Lehnardt wrote: On 10 Nov 2014, at 15:17 , Klaus Trainer wrote: Hey Jan, you didn't provide an argument, so I can only guess: Do you think that we shouldn't tackle that right now, as it would potentially delay the 2.0 release? >>> >>> One of my goals for CouchDB 2.0 is that people upgrading form 1.0 and >>> especially people who continue use CouchDB in a single-node scenario >>> have an easy time. Just breaking more things because we happen to be >>> bumping a version number is not a good motivation. Especially in our >>> new world of avoiding attaching marketing connotation to major release >>> versions (except 2.0 :), we can release CouchDB 3.0 and 4.0 shortly >>> after 2.0 if it means we break BC in a single way. If we keep BC breaks >>> to a minimum and make every transition up a major version as easy as >>> possible, we don’t run into a Python 3 situation that creates a major >>> schism in the user community that takes a decade to heal. >>> >>> Let’s not break things because we update the major version number, >>> instead, let’s update the major version number whenever we break >>> back backward compatibility. >> >> Suddenly, active delayed_commits is what is breaking CouchDB 2.0 - ask >> Russell how long he fight with database migrations from old 1.x to 2.0 >> until he found delayed_commits turned on. Also, I don't think that >> this change is breaking something expect benchmarks speed. > > I was referring to the original email that said: > >> I'd like to propose that we now (as we're already breaking API >> compatibility with the new major release) go the full way and >> remove that feature entirely. > > > I generally disagree with that reasoning as outlined above. > > I’d be happy to hear more about what problems Russell had, but it’s > the first time that info hits dev@ and it wasn’t brought up in this > thread so far. > > Best > Jan In general I think delayed_commits is a misfeature, but I think the big win is disabling it by default. In general I'm +1 to removing it. I've been bitten several times by delayed_commits being enabled without me realizing it causing significant departures from expected behavior, both during the database migrations work and also with eunit test port. Although admittedly both of those issues went away when delayed_commits was disabled. I think we should deprecate delayed commits in the 2.0 release. Switching it to disabled by default should be a good indication if people are effected by it as I imagine we'll get some feedback if that causes problems for anyone. After that I think we'll have a better idea if it's safe for removal. -- Russell
[jira] [Commented] (COUCHDB-2334) Metadata db "cassim" does not exist
[
https://issues.apache.org/jira/browse/COUCHDB-2334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14144038#comment-14144038
]
Russell Branca commented on COUCHDB-2334:
-
I'm not attached to having the db named cassim, "_meta" sounds fine to me.
Should we add an additional setting to use cassim for security?
> Metadata db "cassim" does not exist
> ---
>
> Key: COUCHDB-2334
> URL: https://issues.apache.org/jira/browse/COUCHDB-2334
> Project: CouchDB
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: BigCouch
>Reporter: Alexander Shorin
>
> And so happens every 5 minutes:
> {code}
> 2014-09-20 04:00:06.786 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:05:06.788 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:10:06.790 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:15:06.792 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:20:06.794 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:25:06.796 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:30:06.798 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:35:06.800 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:40:06.802 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:45:06.804 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:50:06.806 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:55:06.808 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 05:00:06.810 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 05:05:06.812 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 05:10:06.814 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (COUCHDB-2334) Metadata db "cassim" does not exist
[
https://issues.apache.org/jira/browse/COUCHDB-2334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14143725#comment-14143725
]
Russell Branca commented on COUCHDB-2334:
-
Logging at error level is probably overkill, we could switch that down to info
or notice. Newson, it's worth noting that using cassim for security properties
is enabled by the act of creating the cassim database, not by setting a
configuration value. The metadata cache process keeps checking for the cassim
database to be created and then will switch over to using it once it sees it
exists.
I'm not a huge fan of this approach, and I've been thinking we should add a
configuration setting for enabling cassim for security properties, and possibly
also a general flag for turning on cassim, although the creation of the cassim
database may be sufficient for that. If we add a config setting to enable
storing security properties in cassim, we can use cassim for other metadata
without having to force users to store security properties there as well.
> Metadata db "cassim" does not exist
> ---
>
> Key: COUCHDB-2334
> URL: https://issues.apache.org/jira/browse/COUCHDB-2334
> Project: CouchDB
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: BigCouch
>Reporter: Alexander Shorin
>
> And so happens every 5 minutes:
> {code}
> 2014-09-20 04:00:06.786 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:05:06.788 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:10:06.790 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:15:06.792 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:20:06.794 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:25:06.796 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:30:06.798 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:35:06.800 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:40:06.802 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:45:06.804 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:50:06.806 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:55:06.808 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 05:00:06.810 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 05:05:06.812 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 05:10:06.814 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (COUCHDB-2334) Metadata db "cassim" does not exist
[
https://issues.apache.org/jira/browse/COUCHDB-2334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russell Branca closed COUCHDB-2334.
---
Resolution: Won't Fix
This is expected behavior. Every five minutes we check to see if the database
exists, so that cassim can be enabled:
https://github.com/apache/couchdb-cassim/blob/windsor-merge/src/cassim_metadata_cache.erl#L113
> Metadata db "cassim" does not exist
> ---
>
> Key: COUCHDB-2334
> URL: https://issues.apache.org/jira/browse/COUCHDB-2334
> Project: CouchDB
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: BigCouch
>Reporter: Alexander Shorin
>
> And so happens every 5 minutes:
> {code}
> 2014-09-20 04:00:06.786 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:05:06.788 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:10:06.790 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:15:06.792 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:20:06.794 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:25:06.796 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:30:06.798 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:35:06.800 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:40:06.802 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:45:06.804 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:50:06.806 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 04:55:06.808 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 05:00:06.810 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 05:05:06.812 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> 2014-09-20 05:10:06.814 [error] node1@127.0.0.1 <0.341.0> Metadata db
> "cassim" does not exist
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (COUCHDB-2328) Chttpd _show/_update functions not working
[ https://issues.apache.org/jira/browse/COUCHDB-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Russell Branca closed COUCHDB-2328. --- > Chttpd _show/_update functions not working > -- > > Key: COUCHDB-2328 > URL: https://issues.apache.org/jira/browse/COUCHDB-2328 > Project: CouchDB > Issue Type: Bug > Security Level: public(Regular issues) > Components: HTTP Interface > Reporter: Russell Branca > Fix For: 2.0.0 > > > The _show and _update endpoints use `chttpd_external:send_external_response` > which does not add CORS headers. As a result OPTIONS requests to these > endpoints work, but the actual requests do not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (COUCHDB-2328) Chttpd _show/_update functions not working
[ https://issues.apache.org/jira/browse/COUCHDB-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Russell Branca resolved COUCHDB-2328. - Resolution: Fixed Fix Version/s: 2.0.0 > Chttpd _show/_update functions not working > -- > > Key: COUCHDB-2328 > URL: https://issues.apache.org/jira/browse/COUCHDB-2328 > Project: CouchDB > Issue Type: Bug > Security Level: public(Regular issues) > Components: HTTP Interface > Reporter: Russell Branca > Fix For: 2.0.0 > > > The _show and _update endpoints use `chttpd_external:send_external_response` > which does not add CORS headers. As a result OPTIONS requests to these > endpoints work, but the actual requests do not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (COUCHDB-2328) Chttpd _show/_update functions not working
Russell Branca created COUCHDB-2328: --- Summary: Chttpd _show/_update functions not working Key: COUCHDB-2328 URL: https://issues.apache.org/jira/browse/COUCHDB-2328 Project: CouchDB Issue Type: Bug Security Level: public (Regular issues) Components: HTTP Interface Reporter: Russell Branca The _show and _update endpoints use `chttpd_external:send_external_response` which does not add CORS headers. As a result OPTIONS requests to these endpoints work, but the actual requests do not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] Rewriting the CouchDB HTTP Layer
How long do we want to keep supporting R14B01? Hopefully we can jump ship over to 17 at some point and stay more current with Erlang releases. Both WebMachine and Cowboy supoort the REST-ful resource declarations which I'm a fan of, but using those would introduce breaking changes to the CouchDB API. Perhaps we make a 3.0 release with the HTTP changes and drop support for R14*? -Russell Paul Davis writes: > One ding against Cowboy is that Loïc isn't very interested in > supporting old releases so we'd have to make minor updates if we wan't > to keep supporting the R14B01 releases. Last time I did this it was > trivial but a bit of an annoyance that such a trivial change was > something I'd have to maintain downstream indefinitely whilst I needed > to support R14B01. It also means that future upgrades may be a bit > difficult if Cowboy ever starts embracing new language features like > maps that are just impossible to support on older VMs. > > On Wed, Aug 20, 2014 at 7:45 PM, Russell Branca > wrote: >> Thanks Andy! The next step I want to take is build another prototype in >> Cowboy and compare it with the web machine implementation. Hopefully will >> have some time for that over the weekend. >> >> -Russell >> On Aug 20, 2014 5:01 AM, "Andy Wenk" wrote: >> >>> Hey Russel, >>> >>> I have read your blog post about "Rewriting the CouchDB HTTP Layer". Thanks >>> a lot for that! >>> >>> As a note from a non-core-CouchDB-dev - this sounds great and very >>> reasonable. Making the code easier to test, removing unnecessary code >>> duplication, organising the code even better and making it easier to write >>> plugins are things, that will lead to better code and will make it easier >>> for devs to contribute. So all thumbs up! Great work! I hope the discussion >>> will lead to a good decision :) >>> >>> Cheers >>> >>> Andy >>> >>> >>> On 19 August 2014 02:27, Russell Branca wrote: >>> >>> > On Aug 17, 2014 8:15 PM, "Jason Smith" wrote: >>> > > >>> > > Hi, Russell. This is okay for a starting point but it is a bit vague. >>> > Could >>> > > you perhaps flesh out the plan and make it more comprehensive? >>> > > >>> > > ^^ That is a joke! >>> > > >>> > > Seriously, thank you very much for this analysis and plan. This is very >>> > > exciting! (Not least because the http codebase is the part I know best >>> > and >>> > > I can get excited about.) >>> > > >>> > >>> > Thanks! >>> > >>> > > One quick question that I don't see from your writeup: What version of >>> > > CouchDB are you thinking of targeting? 2.0? 2.1? 3.0? Is this >>> completely >>> > an >>> > > internal change, or does it affect users? >>> > > >>> > >>> > I think this is outside the scope of 2.0 given how close we are on that. >>> > There's a fair bit of legwork involved in doing the rewrite, so I >>> wouldn't >>> > want to block 2.0. So I think the question is 2.x or 3.0. If we go the >>> web >>> > machine route we should rework the api and definitely introduce backwards >>> > incompatible changes, so we would want to do a 3.0 release there. If we >>> > used cowboy we could mimic the current api and release 2.x. >>> > >>> > > For me, I am not so interested in an internal rewrite with zero >>> advantage >>> > > (besides "it's cleaner"), however I am am very interested to use the >>> > > rewrite for a better opportunity to explore plugin opportunities or >>> other >>> > > extensibility features. >>> > > >>> > > >>> > >>> > This rewrite needs to happen. The internals of the http layer need some >>> > serious love and there's a lot of duplication to remove. I think the big >>> > win here is that this would get the ball rolling on taking a closer look >>> at >>> > the various internal applications and figuring out what needs to be >>> > restructured. For instance, the next logical step after reworking the >>> http >>> > later is to standardize the clustered and local api modules, and there >>> was >>> > some great discussion in the Dev channel toda
Re: [DISCUSS] Rewriting the CouchDB HTTP Layer
Thanks Andy! The next step I want to take is build another prototype in Cowboy and compare it with the web machine implementation. Hopefully will have some time for that over the weekend. -Russell On Aug 20, 2014 5:01 AM, "Andy Wenk" wrote: > Hey Russel, > > I have read your blog post about "Rewriting the CouchDB HTTP Layer". Thanks > a lot for that! > > As a note from a non-core-CouchDB-dev - this sounds great and very > reasonable. Making the code easier to test, removing unnecessary code > duplication, organising the code even better and making it easier to write > plugins are things, that will lead to better code and will make it easier > for devs to contribute. So all thumbs up! Great work! I hope the discussion > will lead to a good decision :) > > Cheers > > Andy > > > On 19 August 2014 02:27, Russell Branca wrote: > > > On Aug 17, 2014 8:15 PM, "Jason Smith" wrote: > > > > > > Hi, Russell. This is okay for a starting point but it is a bit vague. > > Could > > > you perhaps flesh out the plan and make it more comprehensive? > > > > > > ^^ That is a joke! > > > > > > Seriously, thank you very much for this analysis and plan. This is very > > > exciting! (Not least because the http codebase is the part I know best > > and > > > I can get excited about.) > > > > > > > Thanks! > > > > > One quick question that I don't see from your writeup: What version of > > > CouchDB are you thinking of targeting? 2.0? 2.1? 3.0? Is this > completely > > an > > > internal change, or does it affect users? > > > > > > > I think this is outside the scope of 2.0 given how close we are on that. > > There's a fair bit of legwork involved in doing the rewrite, so I > wouldn't > > want to block 2.0. So I think the question is 2.x or 3.0. If we go the > web > > machine route we should rework the api and definitely introduce backwards > > incompatible changes, so we would want to do a 3.0 release there. If we > > used cowboy we could mimic the current api and release 2.x. > > > > > For me, I am not so interested in an internal rewrite with zero > advantage > > > (besides "it's cleaner"), however I am am very interested to use the > > > rewrite for a better opportunity to explore plugin opportunities or > other > > > extensibility features. > > > > > > > > > > This rewrite needs to happen. The internals of the http layer need some > > serious love and there's a lot of duplication to remove. I think the big > > win here is that this would get the ball rolling on taking a closer look > at > > the various internal applications and figuring out what needs to be > > restructured. For instance, the next logical step after reworking the > http > > later is to standardize the clustered and local api modules, and there > was > > some great discussion in the Dev channel today about that. > > > > The more we can decouple the various apps, the more easily we can extend > > CouchDB with plugins and new functionality. > > > > -Russell > > > > > > > > > > > On Mon, Aug 18, 2014 at 1:41 AM, Russell Branca > > > > wrote: > > > > > > > # Rewriting the CouchDB HTTP Layer > > > > > > > > With the light at the end of tunnel on the BigCouch merge, I thought > > > > it was time to get the conversation going on cleaning up the current > > > > HTTP stack duality. We've got a good opportunity to do some major > > > > cleanup, remove duplication, and really start more clearly separating > > > > the various components of CouchDB. > > > > > > > > > > > > ## Primary objectives > > > > > > > > * Consolidate down to one HTTP layer > > > > * Isolate HTTP functionality > > > > * Separate HTTP server from HTTP resources > > > > * Easy plugin integration > > > > * Build clustered/local API > > > > > > > > > > > > ### Consolidate down to one HTTP layer > > > > > > > > We currently have two HTTP layers, `couch_httpd` and `chttpd`. This > > > > was a useful construct when BigCouch was a separate application where > > > > isolating the clustered layer from the local layer was necessary, and > > > > quite useful. > > > > > > > > This is no longer the case, and we can significantly reduce code
Re: [DISCUSS] Rewriting the CouchDB HTTP Layer
On Aug 17, 2014 8:15 PM, "Jason Smith" wrote: > > Hi, Russell. This is okay for a starting point but it is a bit vague. Could > you perhaps flesh out the plan and make it more comprehensive? > > ^^ That is a joke! > > Seriously, thank you very much for this analysis and plan. This is very > exciting! (Not least because the http codebase is the part I know best and > I can get excited about.) > Thanks! > One quick question that I don't see from your writeup: What version of > CouchDB are you thinking of targeting? 2.0? 2.1? 3.0? Is this completely an > internal change, or does it affect users? > I think this is outside the scope of 2.0 given how close we are on that. There's a fair bit of legwork involved in doing the rewrite, so I wouldn't want to block 2.0. So I think the question is 2.x or 3.0. If we go the web machine route we should rework the api and definitely introduce backwards incompatible changes, so we would want to do a 3.0 release there. If we used cowboy we could mimic the current api and release 2.x. > For me, I am not so interested in an internal rewrite with zero advantage > (besides "it's cleaner"), however I am am very interested to use the > rewrite for a better opportunity to explore plugin opportunities or other > extensibility features. > > This rewrite needs to happen. The internals of the http layer need some serious love and there's a lot of duplication to remove. I think the big win here is that this would get the ball rolling on taking a closer look at the various internal applications and figuring out what needs to be restructured. For instance, the next logical step after reworking the http later is to standardize the clustered and local api modules, and there was some great discussion in the Dev channel today about that. The more we can decouple the various apps, the more easily we can extend CouchDB with plugins and new functionality. -Russell > > > On Mon, Aug 18, 2014 at 1:41 AM, Russell Branca > wrote: > > > # Rewriting the CouchDB HTTP Layer > > > > With the light at the end of tunnel on the BigCouch merge, I thought > > it was time to get the conversation going on cleaning up the current > > HTTP stack duality. We've got a good opportunity to do some major > > cleanup, remove duplication, and really start more clearly separating > > the various components of CouchDB. > > > > > > ## Primary objectives > > > > * Consolidate down to one HTTP layer > > * Isolate HTTP functionality > > * Separate HTTP server from HTTP resources > > * Easy plugin integration > > * Build clustered/local API > > > > > > ### Consolidate down to one HTTP layer > > > > We currently have two HTTP layers, `couch_httpd` and `chttpd`. This > > was a useful construct when BigCouch was a separate application where > > isolating the clustered layer from the local layer was necessary, and > > quite useful. > > > > This is no longer the case, and we can significantly reduce code > > duplication by consolidating down to one http layer. There are a > > number of places in the two apps where the code is nearly identical, > > except one calls out to `fabric` and the other calls out for > > `couch_*`. For instance, compare `couch_httpd_db:couch_doc_open/4` [1] > > with `chttpd_db:couch_doc_open/4` [2]. These are completely identical > > aside from whether it goes through the clustered layer, `fabric`, or > > through the local layer `couch_db`. > > > > There are plenty of other places with similar duplication. This is > > obviously ripe with opportunity to refactor and introduce some higher > > level abstractions to make the HTTP layer function independently of the > > document/database level APIs. > > > > > > ### Isolate HTTP functionality > > > > I don't think `couch_doc_open/4` has any business existing in > > the HTTP layer, we should move all non HTTP logic out. IMO the HTTP > > layer should only concern itself with: > > > > 1. Receiving the HTTP requests > > 2. Extracting out the request data into a standard data structure > > 3. Dispatch requests to the appropriate internal APIs > > 4. Forward the response > > > > Anything that doesn't fit into those four steps should be ripped out > > and moved elsewhere. For instance, the primary logic for determining the > > database redundancy and shard values is done in `chttpd_db` [3]. I > > would greatly prefer to see this logic in a database API. > > > > The more we can isolate HTTP logic from database logic the > > better. Once they are fu
[DISCUSS] Rewriting the CouchDB HTTP Layer
# Rewriting the CouchDB HTTP Layer
With the light at the end of tunnel on the BigCouch merge, I thought
it was time to get the conversation going on cleaning up the current
HTTP stack duality. We've got a good opportunity to do some major
cleanup, remove duplication, and really start more clearly separating
the various components of CouchDB.
## Primary objectives
* Consolidate down to one HTTP layer
* Isolate HTTP functionality
* Separate HTTP server from HTTP resources
* Easy plugin integration
* Build clustered/local API
### Consolidate down to one HTTP layer
We currently have two HTTP layers, `couch_httpd` and `chttpd`. This
was a useful construct when BigCouch was a separate application where
isolating the clustered layer from the local layer was necessary, and
quite useful.
This is no longer the case, and we can significantly reduce code
duplication by consolidating down to one http layer. There are a
number of places in the two apps where the code is nearly identical,
except one calls out to `fabric` and the other calls out for
`couch_*`. For instance, compare `couch_httpd_db:couch_doc_open/4` [1]
with `chttpd_db:couch_doc_open/4` [2]. These are completely identical
aside from whether it goes through the clustered layer, `fabric`, or
through the local layer `couch_db`.
There are plenty of other places with similar duplication. This is
obviously ripe with opportunity to refactor and introduce some higher
level abstractions to make the HTTP layer function independently of the
document/database level APIs.
### Isolate HTTP functionality
I don't think `couch_doc_open/4` has any business existing in
the HTTP layer, we should move all non HTTP logic out. IMO the HTTP
layer should only concern itself with:
1. Receiving the HTTP requests
2. Extracting out the request data into a standard data structure
3. Dispatch requests to the appropriate internal APIs
4. Forward the response
Anything that doesn't fit into those four steps should be ripped out
and moved elsewhere. For instance, the primary logic for determining the
database redundancy and shard values is done in `chttpd_db` [3]. I
would greatly prefer to see this logic in a database API.
The more we can isolate HTTP logic from database logic the
better. Once they are fully decoupled, then the HTTP layer is merely
one particular client interface on top of the core database. We also
get all the benefits of isolation for testing and what not.
Along these lines, I think we greatly overuse the #http{} record for
passing around request data, and instead you extract the body, and
then combine all of the user supplied headers and query string params
into a standard options list. This we can we completely separate
making database requests from the representation of the client
request.
### Separate HTTP server from HTTP resources.
I think everything I've said so far is pretty clear cut in terms of
it's _the_ logical thing to do, but separating the HTTP server from
the HTTP endpoints is less clearly defined. However, we do have
precedence for this and there are a number of solid benefits.
First, let me explain what I mean here. There are two pieces to an
HTTP stack, first there's the core HTTP engine that handles receiving
and responding to requests and other things along those lines, and
second there's the places where you supply your business logic and
figure what content to send to the user.
CouchDB has a handful of places using this aproach, where instead of
defining all the logic in the HTTP stack directly, we have auxilary
modules defined within the appropriate applications that specify how
any HTTP requests for that application are handled. A good clean
example of this approach is `couch_mrview_http` [4].
### Easy plugin integration
One big advantage of the above separation of HTTP resources is that it
provides a standard way of plugins hooking in new HTTP endpoints. The
more we can treat the "core" CouchDB applications as plugins, the more
easily it is to isolate and replace various parts of the stack.
### Build clustered/local API
The above example of `couch_doc_open/4` is a clear cut case where
we want to abstract the process of loading a document. Not all places
are as easily abstractable, but this is a great example of why I think
we should have a standard API on top of clustered and local layers,
where deciding which to use is based on a local/clustered flag, or
some other heuristic.
I've been toying around with the idea of making a request object of
some sort, is something like `couch_req:make(ReqBody, ReqOptions)`
that you can then pass to `couch_doc_api` or some such, but I don't
have any strong opinions on this.
## Where I've gotten so far: chttpd2, a proof of concept
I've hacked out an experimental WebMachine [5] based rewrite of the
HTTP stack called `chttpd2` [6]. This PoC follows the same ideas I've
outlined above, so I'll run back through the previous outlined items
and explain
Re: [DISCUSS] Chair rotation?
Late to the party but I'm +1 to Jan's one year term just starting. -Russell On Mon, Aug 11, 2014 at 4:22 AM, Andy Wenk wrote: > cool - thanks Noah! > On 11 August 2014 13:14, Noah Slater wrote: >> It looks like we reached consensus here. Jan's term started the day we >> voted the bylaws in. I will set a reminder for 12 months. :) >> >> On 8 August 2014 19:39, Noah Slater wrote: >> > Bob, individuals are Officers of the Foundation for the duration they >> > hold the position of Chair. You need to be an Officer (in American >> > corporation law) to carry out your duties as Chair. Nothing any more >> > complex than that. >> > >> > VP CouchDB is Jan's title as an Officer. Whoever is elected next will >> > also get the title VP CouchDB, and will also become an Officer. >> > Meanwhile, Jan will no longer be the Chair, the VP CouchDB, or an >> > Officer. >> > >> > Do you think this needs clarification in the bylaws? >> > >> > On 7 August 2014 00:51, Robert Samuel Newson wrote: >> >> >> >> The Chair is also and Officer of the ASF ( >> https://www.apache.org/foundation/faq.html#why-are-PMC-chairs-officers), >> so are we only able to select an existing Officer or does being elected as >> Chair cause one to become an Officer? If so, do they remain an Officer if >> subsequently replaced by another? >> >> >> >> B. >> >> >> >> On 6 Aug 2014, at 23:29, Andy Wenk wrote: >> >> >> >>> seconding Paul. Was my first thought also. Starting the term now, so >> that >> >>> we hold an election in 12 months. >> >>> >> >>> Cheers >> >>> >> >>> Andy >> >>> >> >>> >> >>> On 7 August 2014 00:08, Paul Davis >> wrote: >> >>> >> I'd lean towards having the term start now instead of end now. I can't >> come up with a super convincing argument either direction though. >> >> On Wed, Aug 6, 2014 at 4:58 PM, Noah Slater >> wrote: >> > Hello folks, >> > >> > It's time we have a discussion about the position of Chair. Jan has >> > held office for a while now, and has done a great job of it. But our >> > new bylaws specify that we will hold yearly reelections. >> > >> > We have a number of options here: >> > >> > 1. We say that Jan's position started on the day the bylaws were >> > ratified, and in 12 months we hold an election. >> > >> > 2. We say that Jan's position is up (having already held it for more >> > than 12 months) and we either nominated a new Chair or we hold a >> vote. >> > >> > Let's try to get general consensus about which option we want to >> take. >> > If there's clear agreement, we'll move forward with that option. >> > >> > Remember: our rationale for re-electing the Chair is that we think >> > that doing so is beneficial for the project. Giving new people a >> > chance to serve a term may stimulate activity and change, and so on. >> > It also helps familiarise as many people as possible with all parts >> of >> > the project. >> > >> > Jan has done a sterling job, and nobody can deny it. This discussion >> > isn't about whether we need to replace him. It's only about when we >> > set the end date of his current term. Immediately or 12 months in the >> > future. :) >> > >> > Also, please, whatever you do, please do not reply to this email with >> > a nomination. Such as: "I think Monty would make a good Chair." Chair >> > nomination itself will happen in private on the PMC list. >> > >> > Thanks, >> > >> > -- >> > Noah Slater >> > https://twitter.com/nslater >> >> >>> >> >>> >> >>> >> >>> -- >> >>> Andy Wenk >> >>> Hamburg - Germany >> >>> RockIt! >> >>> >> >>> GPG fingerprint: C044 8322 9E12 1483 4FEC 9452 B65D 6BE3 9ED3 9588 >> >>> >> >>> https://people.apache.org/keys/committer/andywenk.asc >> >> >> > >> > >> > >> > -- >> > Noah Slater >> > https://twitter.com/nslater >> >> >> >> -- >> Noah Slater >> https://twitter.com/nslater >> > -- > Andy Wenk > Hamburg - Germany > RockIt! > GPG fingerprint: C044 8322 9E12 1483 4FEC 9452 B65D 6BE3 9ED3 9588 > https://people.apache.org/keys/committer/andywenk.asc
Re: CouchDB 2.0: breaking the backward compatibility
I would also love to see _rev renamed, and I think it's a good opportunity to flip around all the meta info as well. I'm still partial to moving the relevant metadata into the headers, and no longer including any _* fields in the doc, but I know there are proponents on both sides of the coin there. The most recent proposal I could find is to move all the metadata into a '_' field [1]. In 2.0 I would like to see us move all metadata into headers or into the '_' field, and rename 'rev'. There's a lot of code overlap for the two so it seems like an opportune time to do it. I wonder if it's reasonable to make the use of a '_' field or exposed through headers configurable. I'm not sure it's a great idea to do so, but it's at least worth thinking about. Exposing conflicts by default is another thing I'm keen on. The question is how to make it "fail" loudly so that client libraries don't just think it's the document body. An aggressive approach send a list of conflict revs rather than a doc object which will break all existing parsers and require users to deal with. Then if you want a particular rev, you'll need to specify it in the request. We could also cleanup the API endpoints to make them more RESTful. IMO things like _all_dbs and _all_docs should be the top level endpoints and the current info endpoints moved to _info or some such. Along the lines of API cleanup is the capabilities engine. I think this would be a great thing to land, and if done properly could be a self defining REST endpoint showing all the things the server is capable of and how to reach them. -Russell [1] http://mail-archives.apache.org/mod_mbox/couchdb-dev/201312.mbox/%3c529de44c.4090...@bigbluehat.com%3E On Thu, Jul 17, 2014 at 2:14 AM, Robert Samuel Newson wrote: > Great point, +1 to just making that change on master right now. > > B. > > On 16 Jul 2014, at 22:35, Robert Kowalski wrote: > >> I would like to see 'JSONP responses should be sent with a >> "application/javascript"' (https://github.com/apache/couchdb/pull/236) >> beside the two merges in the 2.0 release - it is a small, but breaking >> change and the original issue is flying around in Jira for years. >> >> Best, >> Robert >> >> >> 2014-07-13 22:17 GMT+02:00 Robert Samuel Newson : >> >>> >>> Since we follow semantic versioning, the only meaning behind naming our >>> next release 2.0 and not 1.7 is that it contains backwards incompatible >>> changes. >>> >>> It’s for the CouchDB community as a whole to determine what is and isn’t >>> in a release. Certainly merging in bigcouch and rcouch are a huge part of >>> the 2.0 release, but they aren’t necessarily the only things. If they >>> hadn’t changed the API in incompatible ways, they wouldn’t cause a major >>> version bump. >>> >>> With that said then, I’m interested in hearing what else, besides the two >>> merges, we feel we want to take on in our first major revision bump in >>> approximately forever? At minimum, I would like to see a change that allows >>> us to use versions of spidermonkey released after 1.8.5, whatever that >>> change might be. >>> >>> B. >>> >>> On 13 Jul 2014, at 20:31, Joan Touzet wrote: >>> Improving the view server protocol is a great idea, but it is appropriate for a 2.0 timeframe? I would think it would make more sense in a 3.0 timeframe, given 2.0 is all about merging forks, not writing new features entirely from scratch. -Joan - Original Message - From: "Robert Samuel Newson" To: dev@couchdb.apache.org Sent: Sunday, July 13, 2014 8:52:40 AM Subject: Re: CouchDB 2.0: breaking the backward compatibility Adding mvcc for _security is a great idea (happily, Cloudant have done >>> so very recently, so I will be pulling that work over soon). A better view server protocol is also a great idea. On 13 Jul 2014, at 13:13, Samuel Williams < >>> space.ship.travel...@gmail.com> wrote: > > On 13/07/14 23:47, Alexander Shorin wrote: >> Our view server is compiles functions on each view index update >> instead of reusing inner cache. This is because of out-dated protocol: >> others design function are works differently from views. While it's >> good to change and improve query server protocol completely, this task >> requires more time to be done. We should have a least plan B to do >> small steps in good direction. > As already suggested, here is my proposal for 2.0 view/query server: > > >>> https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit > > I welcome people to suggest improvements/changes/ideas. > > Kind regards, > Samuel >>> >>> >
[jira] [Reopened] (COUCHDB-1998) Scripts and docs to boot dev cluster
[ https://issues.apache.org/jira/browse/COUCHDB-1998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Russell Branca reopened COUCHDB-1998: - Needs docs as described in the title. > Scripts and docs to boot dev cluster > > > Key: COUCHDB-1998 > URL: https://issues.apache.org/jira/browse/COUCHDB-1998 > Project: CouchDB > Issue Type: Improvement > Components: BigCouch >Reporter: Volker Mische >Assignee: Robert Newson > Fix For: 2.0.0 > > Time Spent: 96h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (COUCHDB-2095) Ignore epilogues in multipart/related MIME attachments
[ https://issues.apache.org/jira/browse/COUCHDB-2095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Russell Branca resolved COUCHDB-2095. - Resolution: Fixed Fix Version/s: 2.0.0 > Ignore epilogues in multipart/related MIME attachments > -- > > Key: COUCHDB-2095 > URL: https://issues.apache.org/jira/browse/COUCHDB-2095 > Project: CouchDB > Issue Type: Improvement > Security Level: public(Regular issues) > Components: BigCouch >Reporter: Paul Joseph Davis > Labels: merge > Fix For: 2.0.0 > > > Avoid regressions -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (COUCHDB-2098) Support efficient pagination of _all_dbs
[ https://issues.apache.org/jira/browse/COUCHDB-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Russell Branca resolved COUCHDB-2098. - Resolution: Fixed Fix Version/s: 2.0.0 Fixed with the move to couch_mrview. > Support efficient pagination of _all_dbs > > > Key: COUCHDB-2098 > URL: https://issues.apache.org/jira/browse/COUCHDB-2098 > Project: CouchDB > Issue Type: Improvement > Security Level: public(Regular issues) > Components: BigCouch >Reporter: Paul Joseph Davis > Labels: release > Fix For: 2.0.0 > > > Pull in some optimizations for paging _all_dbs in a cluster. Depends on > COUCHDB-1993 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (COUCHDB-2094) Support Last-Event-ID header in EventSource _changes feeds
[ https://issues.apache.org/jira/browse/COUCHDB-2094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Russell Branca resolved COUCHDB-2094. - Resolution: Fixed Fix Version/s: 2.0.0 > Support Last-Event-ID header in EventSource _changes feeds > -- > > Key: COUCHDB-2094 > URL: https://issues.apache.org/jira/browse/COUCHDB-2094 > Project: CouchDB > Issue Type: Improvement > Security Level: public(Regular issues) > Components: BigCouch >Reporter: Paul Joseph Davis > Labels: merge > Fix For: 2.0.0 > > > Avoid regressions -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (COUCHDB-2093) Provide _changes feeds using EventSource
[ https://issues.apache.org/jira/browse/COUCHDB-2093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Russell Branca resolved COUCHDB-2093. - Resolution: Fixed Fix Version/s: 2.0.0 > Provide _changes feeds using EventSource > > > Key: COUCHDB-2093 > URL: https://issues.apache.org/jira/browse/COUCHDB-2093 > Project: CouchDB > Issue Type: Improvement > Security Level: public(Regular issues) > Components: BigCouch >Reporter: Paul Joseph Davis > Labels: merge > Fix For: 2.0.0 > > > Avoid regressions -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (COUCHDB-2092) Allow _list functions to consume _all_docs
[ https://issues.apache.org/jira/browse/COUCHDB-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Russell Branca resolved COUCHDB-2092. - Resolution: Fixed Fix Version/s: 2.0.0 > Allow _list functions to consume _all_docs > -- > > Key: COUCHDB-2092 > URL: https://issues.apache.org/jira/browse/COUCHDB-2092 > Project: CouchDB > Issue Type: Improvement > Security Level: public(Regular issues) > Components: BigCouch >Reporter: Paul Joseph Davis > Labels: merge > Fix For: 2.0.0 > > > Avoid regressions -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (COUCHDB-2090) Test .couch file compatibility
[ https://issues.apache.org/jira/browse/COUCHDB-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14065262#comment-14065262 ] Russell Branca commented on COUCHDB-2090: - I've written things up in http://mail-archives.apache.org/mod_mbox/couchdb-dev/201406.mbox/browser with some sample .couch files and a script to test the things. > Test .couch file compatibility > -- > > Key: COUCHDB-2090 > URL: https://issues.apache.org/jira/browse/COUCHDB-2090 > Project: CouchDB > Issue Type: Improvement > Security Level: public(Regular issues) > Components: BigCouch >Reporter: Paul Joseph Davis > Labels: release > > There are a number of file format differences we need to test. Most of these > should be backwards compatible but its always possible an upgrade covered > issues historically. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (COUCHDB-2006) Add https support to BigCouch
[ https://issues.apache.org/jira/browse/COUCHDB-2006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14065254#comment-14065254 ] Russell Branca commented on COUCHDB-2006: - Duplicate of COUCHDB-2081. > Add https support to BigCouch > - > > Key: COUCHDB-2006 > URL: https://issues.apache.org/jira/browse/COUCHDB-2006 > Project: CouchDB > Issue Type: Improvement > Components: BigCouch >Reporter: Volker Mische > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (COUCHDB-1998) Scripts and docs to boot dev cluster
[ https://issues.apache.org/jira/browse/COUCHDB-1998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Russell Branca resolved COUCHDB-1998. - Resolution: Fixed Fix Version/s: 2.0.0 The dev/run script is functional, and also there is a dev/remsh script now. > Scripts and docs to boot dev cluster > > > Key: COUCHDB-1998 > URL: https://issues.apache.org/jira/browse/COUCHDB-1998 > Project: CouchDB > Issue Type: Improvement > Components: BigCouch >Reporter: Volker Mische >Assignee: Robert Newson > Fix For: 2.0.0 > > Time Spent: 96h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (COUCHDB-2006) Add https support to BigCouch
[ https://issues.apache.org/jira/browse/COUCHDB-2006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Russell Branca resolved COUCHDB-2006. - Resolution: Duplicate > Add https support to BigCouch > - > > Key: COUCHDB-2006 > URL: https://issues.apache.org/jira/browse/COUCHDB-2006 > Project: CouchDB > Issue Type: Improvement > Components: BigCouch >Reporter: Volker Mische > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (COUCHDB-2081) Support HTTPS in chttpd
[ https://issues.apache.org/jira/browse/COUCHDB-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Russell Branca resolved COUCHDB-2081. - Resolution: Fixed Fix Version/s: 2.0.0 This landed in chttpd. > Support HTTPS in chttpd > --- > > Key: COUCHDB-2081 > URL: https://issues.apache.org/jira/browse/COUCHDB-2081 > Project: CouchDB > Issue Type: Improvement > Security Level: public(Regular issues) > Components: BigCouch >Reporter: Paul Joseph Davis > Labels: release > Fix For: 2.0.0 > > > BigCouch relies on load balancers for SSL termination. We'll need to add the > options to chttpd directly. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (COUCHDB-1996) Switch logging for BigCouch
[ https://issues.apache.org/jira/browse/COUCHDB-1996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Russell Branca resolved COUCHDB-1996. - Resolution: Fixed Fix Version/s: 2.0.0 There is now a lager backed couch_log application. > Switch logging for BigCouch > --- > > Key: COUCHDB-1996 > URL: https://issues.apache.org/jira/browse/COUCHDB-1996 > Project: CouchDB > Issue Type: Improvement > Components: BigCouch >Reporter: Volker Mische > Fix For: 2.0.0 > > > The logging should be changed to either twig or lager -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (COUCHDB-1997) Make fabric endpoints work
[ https://issues.apache.org/jira/browse/COUCHDB-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Russell Branca resolved COUCHDB-1997. - Resolution: Won't Fix Not sure what this ticket is for. As far as I know fabric works with all the things. Feel free to reopen with more details if there's something else that needs to be addressed here. > Make fabric endpoints work > -- > > Key: COUCHDB-1997 > URL: https://issues.apache.org/jira/browse/COUCHDB-1997 > Project: CouchDB > Issue Type: Improvement > Components: BigCouch >Reporter: Volker Mische > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (COUCHDB-1993) Make fabric work with the refactored view engine
[ https://issues.apache.org/jira/browse/COUCHDB-1993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Russell Branca resolved COUCHDB-1993. - Resolution: Fixed Fix Version/s: 2.0.0 This landed a while ago. > Make fabric work with the refactored view engine > > > Key: COUCHDB-1993 > URL: https://issues.apache.org/jira/browse/COUCHDB-1993 > Project: CouchDB > Issue Type: Improvement > Components: BigCouch >Reporter: Volker Mische > Assignee: Russell Branca > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (COUCHDB-1960) Provide pagination for /_all_dbs
[
https://issues.apache.org/jira/browse/COUCHDB-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russell Branca resolved COUCHDB-1960.
-
Resolution: Fixed
Fix Version/s: 2.0.0
This is fixed for clustered interfaces. IMO the correct fix for the local
interfaces is to switch to using the dbs db for all interfaces, then we can
port this logic for the local interface.
> Provide pagination for /_all_dbs
>
>
> Key: COUCHDB-1960
> URL: https://issues.apache.org/jira/browse/COUCHDB-1960
> Project: CouchDB
> Issue Type: Improvement
> Components: Fauxton, HTTP Interface
>Reporter: Benjamin Young
> Fix For: 2.0.0
>
>
> The database-per-user design pattern has increased in popularity and
> consequently causes long lists of databases when one hits {{/_all_dbs}}
> To keep this sane, we need some pagination.
> {{?skip=10&limit=10}} style seems to be the simplest to implement (and there
> should be existing code from Cloudant for this one).
> Thanks!
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: rcouch merge: integration with the bigcouch branch questions
On Mon, Jul 7, 2014 at 8:50 PM, Benoit Chesneau wrote: > On Mon, Jul 7, 2014 at 11:44 PM, Russell Branca wrote: >> I switched to using chttpd in couch_mrview for delayed responses as it >> has more robust error handling logic (in particular it fixes a nasty >> bug where error messages were sent as full responses with headers in >> the stream) . >> >> Benoit, I don't understand what you mean by "Which makes this >> functions unsable with the standalone HTTP api." The change in >> couch_mrview to use chttpd should _not_ prevent the single node >> interface from operating, if you're encountering that then it's a bug. > > > It was badly phrased. I meant that apparently this change implies now > that I need to use chttpd for the view queries. I just had a quick > glance in the delayed response and not tested it, but doesn't it imply > to handle the request via a chttpd controller? > > https://github.com/apache/couchdb-chttpd/blob/1843-feature-bigcouch/src/chttpd.erl#L601-L607 > > Looks like it is a special response record. How the view queries are > now working on the couch_httpd api? > The record is opaque to the view stream callbacks. This just uses chttpd for sending delayed chunks, because it has the better implementation. >> >> IMO the logical next step for all this is to refactor the separate >> httpd layers into a single app and address the issues there, but I >> think that can wait for 2.x. >> >> Also, for "port the couch_mrview http modules to chttpd for the layer >> part *and* couch_httpd" I don't think this is the correct approach. I >> think we should define the core http layer in c(ouch_)httpd and then >> continue to use the pattern of the separate applications defining the >> relevant http bits. >> >> I've already done a lot of legwork to make the http view layer usable >> in both couch_httpd and chttpd with the long term game plan of >> consolidating down to only one http layer. For instance, the clustered >> view logic uses the existing couch_mrview_http code for actually >> sending the responses: >> https://github.com/apache/couchdb-chttpd/blob/master/src/chttpd_view.erl#L57. >> >> Compare these two functions: >> >> https://github.com/apache/couchdb-chttpd/blob/master/src/chttpd_view.erl#L49-L62 >> https://github.com/apache/couchdb-couch-mrview/blob/master/src/couch_mrview_http.erl#L191-L204 >> >> You can see they're almost identical, aside from >> `couch_mrview:query_view` vs `fabric:query_view`. I wanted to use the >> view layer as an example of removing unnecessary duplication in the >> two http layers, and to jump start the process of consolidation. >> Hopefully that gives a good idea on approach and the parts that we >> actually need to abstract. > > Maybe. I would have done the changes only in chttpd imo since they are > about a new features that won't be ship in 1.7. Especially since I > understand it some wants chttpd and couch_httpd may be merged in a way > or another. Keeping them separated until then would not impact any > future direction about it. > If we cut 1.7 IMO it should be off couchdb master with none of the merge. As far as the merge, it's including chttpd either way. > I implemented the feature using couch_httpd this week-end to support > the feature on the current version: > > https://github.com/rcouch/couchdb-couch-mrview/commit/4e0f29577c21fb6249fecbfe9f20a46b90f6ae3b > > The relevant part of it, s is that I didn't have to change the > couch_httpd application to support multi queries in view changes for > the _changes handler since this handler was cleanly separated from the > rest. > > For the 2.0 I think we should consider to completely make the http api > available as a *possible* transport and indeed only have one > application. but until then it worth to keep changes on it separated. > It would ease any work on that part later. > I'm a strong proponent for combining the http layers and then drastically decreasing the functionality to only handle the http side of things, and call out to the appropriate application for the database level logic. Believe me, I'm well aware of the overly coupled semantics of the current http stack, both in chttpd and couch_httpd. For instance the default clustering values for db creation are set in chttpd: https://github.com/apache/couchdb-chttpd/blob/master/src/chttpd_db.erl#L201-L216. This obviously is not ideal and most definitely needs to change, but it gets the job done for now. I personally would not block a 2.0 release to wait on a month or two of work to refactor the http layer. It's an optimization for code complexity (albeit a good one), not a feature relevant to people wanting to use BigCouch. > - benoit -Russell
Re: rcouch merge: integration with the bigcouch branch questions
I switched to using chttpd in couch_mrview for delayed responses as it has more robust error handling logic (in particular it fixes a nasty bug where error messages were sent as full responses with headers in the stream) . Benoit, I don't understand what you mean by "Which makes this functions unsable with the standalone HTTP api." The change in couch_mrview to use chttpd should _not_ prevent the single node interface from operating, if you're encountering that then it's a bug. IMO the logical next step for all this is to refactor the separate httpd layers into a single app and address the issues there, but I think that can wait for 2.x. Also, for "port the couch_mrview http modules to chttpd for the layer part *and* couch_httpd" I don't think this is the correct approach. I think we should define the core http layer in c(ouch_)httpd and then continue to use the pattern of the separate applications defining the relevant http bits. I've already done a lot of legwork to make the http view layer usable in both couch_httpd and chttpd with the long term game plan of consolidating down to only one http layer. For instance, the clustered view logic uses the existing couch_mrview_http code for actually sending the responses: https://github.com/apache/couchdb-chttpd/blob/master/src/chttpd_view.erl#L57. Compare these two functions: https://github.com/apache/couchdb-chttpd/blob/master/src/chttpd_view.erl#L49-L62 https://github.com/apache/couchdb-couch-mrview/blob/master/src/couch_mrview_http.erl#L191-L204 You can see they're almost identical, aside from `couch_mrview:query_view` vs `fabric:query_view`. I wanted to use the view layer as an example of removing unnecessary duplication in the two http layers, and to jump start the process of consolidation. Hopefully that gives a good idea on approach and the parts that we actually need to abstract. -Russell On Mon, Jul 7, 2014 at 2:04 PM, Benoit Chesneau wrote: > On Fri, Jul 4, 2014 at 6:38 PM, Paul Davis > wrote: >> On Fri, Jul 4, 2014 at 5:01 AM, Benoit Chesneau wrote: >>> >>> Some blocking points/questions I have about the integration of rcouch >>> and bigcouch. >>> >>> >>> - OTP release: the bigcouch and the rcouch branches are quite similar >>> but I still wonder about the usage of this configure script. Is this >>> really nedded? Can't we just rely on a makefile? >> >> >> I'd think it'd be possible to figure out something for this. I really >> don't consider it a "configure" script so much as a "bootstrap" >> script. Could very well be possible to put this into a Makefile. I >> don't think this is a merge blocker though as its a relatively minor >> change regardless of what we decide. >> >>> - HTTP api. some recent additions have been made in couch_mrview that >>> added chttpd usage. Which makes this functions unsable with the >>> standalone HTTP api. Also the couch application in the bigcouch >>> branch still contains couch_httpd* modules. Rcouch extracted them in a >>> full erlang app: couch_httpd. My question is do we still need the >>> legacy api (my understanding is that it is still used as standalone >>> node frontend in bigcouch) ? If yes I propose to have a better >>> separation in the api. There maybe 2 ways: >>> - port the couch_mrview http modules to chttpd for the layer part >>> *and* couch_httpd >>> - have different functions or better modules for chttpd and >>> couch_httpd in couch_mrview (couch_mrview_httpd_*.erl and >>> (couch_mrview_chttpd_*.erl)) >> >> >> The chttpd/couch_httpd_* split is a bit historical. The merge plan has >> been to maintain the separation through the merge and then have >> post-merge work that will parameterize them at runtime to work with >> either single-node or clustered APIs underneath. As to extracting them >> into a separate app that's not impossible but not something we've done >> in the BigCouch branch because it hasn't happened on master. I think >> it'd be a good idea to do post bigcouch merge when we're bringing in >> rcouch. > > The couch_httpd extraction is already done in rcouch. The question is > more having a mix of chttpd and couch_httpd in the couch_mrview app. >> >>> - couch_collate nif to replace couch_drv + ejson_compare nif . Is this >>> OK for you? It's extensively tested there. >> >> >> We don't use ejson_compare at all so from the BigCouch point of view >> it's just a matter of replacing couch_drv which would be fine by me. >> The only thing I remember about couch_collate was wanting to >> investigate the way that the collation contexts are moved around as it >> looked fairly complicated. But mostly I think that comes down to >> measuring how heavy weight of an operation it is to create and destroy >> those contexts. >> >>> - how the fauxton build is handled right now in the bigcouch branch. >>> What are requirements to make it automatic? (no manual installation). >>> Is there a way to disable it >> >> >> We haven't spent too much time on Fauxton yet so I don't have much to >> a
Re: info: quickcheck free for opensource projects
Agreed, we should get this going as soon as we can. How does the quickcheck CI server work with standard eunit tests? Do we need to do something to make sure only quickcheck tests work? Or do we just run all the tests? -Russell On Mon, Jun 30, 2014 at 1:22 AM, Benoit Chesneau wrote: > cool. Not sure what could be the roadmap on this. Like I see it maybe > the first thing to do is to integrate the changes on the test suite > from Alexander. Then we can probably add quickcheck test on each each > module using the macros to define if they are executed or not? > > > - benoit > > On Wed, Jun 11, 2014 at 7:41 PM, Jan Lehnardt wrote: >> >> On 11 Jun 2014, at 19:39 , Russell Branca wrote: >> >>> I'm a huge +1 to this. >>> >>> I've been trying to figure out a way to get us able to use the full version >>> of QuickCheck for a while now. John Hughes has been hinting that they found >>> a way to make the licensing work for open source, and it seems like this is >>> it. >>> >>> The full version of QuickCheck has some sweet features for testing out >>> state machines and also the PULSE scheduler which randomizes the execution >>> of processes to help discover race conditions: >>> http://www.quviq.com/features.html >>> >>> To clarify the questions about another "CI" server, I believe the reason >>> for this being released as a CI server is as a way to use the full version >>> of QuickCheck without them having to distribute it. >> >> Ah, apologies for missing that particular context. >> >> I’m still +100 on this :) >> >> Best >> Jan >> -- >> >>> >>> >>> -Russell >>> >>> >>> On Wed, Jun 11, 2014 at 4:13 AM, Jan Lehnardt wrote: >>> >>>> QC is not a CI tool. It’s more like an additional layer of more thorough >>>> unit testing that could (depending on their terms) run by our existing CI >>>> solutions. >>>> >>>> I’d be in favour of looking at how we can make it work! >>>> >>>> Best >>>> Jan >>>> -- >>>> >>>> On 11 Jun 2014, at 13:06 , Dirkjan Ochtman wrote: >>>> >>>>> n Wed, Jun 11, 2014 at 1:00 PM, Benoit Chesneau >>>> wrote: >>>>>> quickcheck made quickcheck-ci available for free for open-sources >>>> projects: >>>>>> >>>>>> http://quickcheck-ci.com/ >>>>>> >>>>>> It would be interresting to use it for couchdb imo. Thoughts? >>>>> >>>>> If we still use Travis, we already have 2 CI instances, and they have >>>>> not been able to prevent drawn out release processes like the one for >>>>> 1.6.0. Unless we somehow think this will magically solve all our CI >>>>> needs, I'd prefer to instead spend time on improving other parts of >>>>> the CI we already have. >>>>> >>>>> Cheers, >>>>> >>>>> Dirkjan >>>> >>>> >>
[jira] [Resolved] (COUCHDB-2097) Avoid performance regression with a single fd
[ https://issues.apache.org/jira/browse/COUCHDB-2097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Russell Branca resolved COUCHDB-2097. - Resolution: Fixed Resolving this as the performance changes are within tolerable limits. We can address specific performance issues that come up individually, but in general this looks acceptable. > Avoid performance regression with a single fd > - > > Key: COUCHDB-2097 > URL: https://issues.apache.org/jira/browse/COUCHDB-2097 > Project: CouchDB > Issue Type: Improvement > Security Level: public(Regular issues) > Components: BigCouch >Reporter: Paul Joseph Davis > Labels: release > Attachments: mike-wallace-fd-benchmarks.txt > > > Part of one of our large enhancements required that we remove a CouchDB > performance optimization on having two file handles to each .couch file. We > need to make sure that this doesn't negatively impact performance. -- This message was sent by Atlassian JIRA (v6.2#6252)
[REQUEST] Help test BigCouch file migration for .couch files
Hi all, I've put together a repo with sample .couch files from all point releases since CouchDB 1.1.1. Please follow the testing instructions here and let me know if the tests pass or fail: https://github.com/chewbranca/test_couch_file_migrations#running-the-tests Also it would be great to get people testing their own personal databases. I've tried to make a comprehensive list of doc types in the generated .couch files, but ideally we should test on as wide a variety of .couch files as possible. The setup instructions detail how to get a local BigCouch dev setup running, and show you how to copy a .couch file into BigCouch and then query it. Please bring in some of your .couch files and test them to see any unexpected behavior. Thanks! -Russell
Re: [ANNOUNCE] Lena Reinhard elected as CouchDB committer
Congrats Lena! -Russell On Sat, Jun 21, 2014 at 3:00 PM, Lena Reinhard wrote: > Thanks, everyone, for the warm welcome! I'm happy about the election and > very much looking forward to continuing working on CouchDB together with > everyone of you! > > All the best > > Lena > > > On 21.06.2014, at 21:27, Jan Lehnardt wrote: > > > > Welcome Lena, glad to have you on board! :) > > > > Best > > Jan > > -- > > > >> On 21 Jun 2014, at 16:59 , Noah Slater wrote: > >> > >> Dear community, > >> > >> I am pleased to announce that the CouchDB Project Management Committee > >> has elected Lena Reinhard as a CouchDB committer. > >> > >> Apache ID: lena > >> > >> IRC nick: ux > >> > >> Twitter: https://twitter.com/ux > >> > >> By default, external contributions to the project follow the > >> Review-Then-Commit model. Being a committer means you can follow the > >> Commit-Then-Review model. In other words, Lena can now make changes at > >> will. > >> > >> This election was made in recognition of Lena's existing contributions > >> and commitment to the project. > >> > >> Please join me in extending a warm welcome to Lena! > >> > >> On behalf of the CouchDB PMC, > >> > >> -- > >> Noah Slater > >> https://twitter.com/nslater > > >
Re: mutli view query
Hi Jean,
The top level field for multi view queries is "queries", try that instead
of "keys". Let me know if you're still having issues.
-Russell
On Jun 12, 2014 6:21 AM, "Jean-Felix Girard" wrote:
> Hi,
>
> One feature I look forward and want to test is the possibility to query a
> view with multiple "ranges" (startkey + endkey) -
> https://issues.apache.org/jira/browse/COUCHDB-523. I saw a commit to
> address it :
> https://github.com/apache/couchdb-couch-mrview/commit/3688736b81c8b8b6485cb136eea836bd729d152f.
> From my understanding, the bigcouch merge branch uses
> "couchdb-couch-mrview", master which include the commit. Is it really the
> case ?
>
> I tried but could not make it work.
>
> For instance:
>
> curl 'localhost:15984/test/_all_docs' -X POST -d
> '{"keys":[{"startkey":"a","endkey":"z"}]}' -H
> 'content-type:application/json' -0
>
>
> {"total_rows":5,"rows":[
> HTTP/1.0 500 Internal Server Error
> X-CouchDB-Body-Time: 0
> X-Couch-Request-ID: b1ad2ba2
> Server: CouchDB/c9a3fc1 (Erlang OTP/R16B03-1)
> Date: Thu, 12 Jun 2014 13:15:09 GMT
> Content-Type: text/plain; charset=utf-8
> Content-Length: 242
> Cache-Control: must-revalidate
>
> {"error":"badmatch","reason":"timeout","stack":["fabric_view_all_docs:go/5
> L76","couch_httpd:etag_maybe/2 L592","chttpd_db:all_docs_view/3
> L512","chttpd:handle_request/1 L206","mochiweb_http:headers/5
> L93","proc_lib:init_p_do_apply/3 L239"]}
>
>
> Any pointer would help!
>
> Thanks.
> --
> J
>
>
>
>
Re: info: quickcheck free for opensource projects
I'm a huge +1 to this. I've been trying to figure out a way to get us able to use the full version of QuickCheck for a while now. John Hughes has been hinting that they found a way to make the licensing work for open source, and it seems like this is it. The full version of QuickCheck has some sweet features for testing out state machines and also the PULSE scheduler which randomizes the execution of processes to help discover race conditions: http://www.quviq.com/features.html To clarify the questions about another "CI" server, I believe the reason for this being released as a CI server is as a way to use the full version of QuickCheck without them having to distribute it. -Russell On Wed, Jun 11, 2014 at 4:13 AM, Jan Lehnardt wrote: > QC is not a CI tool. It’s more like an additional layer of more thorough > unit testing that could (depending on their terms) run by our existing CI > solutions. > > I’d be in favour of looking at how we can make it work! > > Best > Jan > -- > > On 11 Jun 2014, at 13:06 , Dirkjan Ochtman wrote: > > > n Wed, Jun 11, 2014 at 1:00 PM, Benoit Chesneau > wrote: > >> quickcheck made quickcheck-ci available for free for open-sources > projects: > >> > >> http://quickcheck-ci.com/ > >> > >> It would be interresting to use it for couchdb imo. Thoughts? > > > > If we still use Travis, we already have 2 CI instances, and they have > > not been able to prevent drawn out release processes like the one for > > 1.6.0. Unless we somehow think this will magically solve all our CI > > needs, I'd prefer to instead spend time on improving other parts of > > the CI we already have. > > > > Cheers, > > > > Dirkjan > >
Re: Installing 1843-feature-bigcouch
Thanks for the report Andy. I'm not sure how far Newson has tested doing a
`make install`, I know personally I've only been running with the local dev
server for now.
The crash error in your log with mem3_shards:fold failing on a bad match is
an interesting one, and looks like an actual bug.
As for Futon, I think the general consensus is to drop it entirely for 2.0
and switch over to Fauxton.
Andy, thanks for starting to test single node CouchDB! It's definitely
important to make sure there aren't any regressions there and that things
still work as expected.
-Russell
On Sun, May 25, 2014 at 9:26 AM, Andy Wenk wrote:
> Hi Bob,
>
> I cloned a fresh couch repo and ran
>
> $ ./configure
> $ make
>
> Here are some warnings:
>
> Compiling priv/icu_driver/couch_icu_driver.c
> priv/icu_driver/couch_icu_driver.c:171:9: warning: incompatible pointer
> types initializing 'ErlDrvSSizeT (*)(ErlDrvData, unsigned int, char *,
> ErlDrvSizeT, char **, ErlDrvSizeT)' with an expression of type
> 'COUCH_SSIZET (ErlDrvData, unsigned int, char *, COUCH_SSIZET, char **,
> COUCH_SSIZET)' [-Wincompatible-pointer-types]
> couch_drv_control, /* F_PTR control, port_command callback */
> ^
> 1 warning generated.
>
> Compiling c_src/snappy/snappy.cc
> c_src/snappy/snappy.cc:516:13: warning: unused function 'ComputeTable'
> [-Wunused-function]
> static void ComputeTable() {
> ^
> 1 warning generated.
>
> When running
>
> $ make test
>
> I just received:
>
> ==> couchdb-1843-feature-bigcouch-3 (eunit)
>
> When running
>
> $ make install
>
> I received
>
> WARN: 'generate' command does not apply to directory
> /Users/andwen/project/couchdb/couchdb-1843-feature-bigcouch-3
> mkdir: /opt/couchdb: Permission denied
> make: *** [install] Error 1
>
> So the target is /opt/couchdb ... ? That's new ...
>
> With sudo it works for sure. I received this warning:
>
> WARN: 'generate' command does not apply to directory
> /Users/andwen/project/couchdb/couchdb-1843-feature-bigcouch-3
>
> When starting couchdb with
>
> $ sudo /opt/couchdb/bin/couchdb
>
> I received
>
> sudo /opt/couchdb/bin/couchdb⏎ ◼
> Apache CouchDB 6dcd33b is starting.
> 18:22:00.869 [info] Application lager started on node
> 'couchdb@andwen-2.local'
> 18:22:00.869 [info] Application couch_log started on node
> 'couchdb@andwen-2.local'
> 18:22:00.875 [info] Starting couch_sup
> 18:22:00.927 [notice] config: [couchdb] uuid set to
> b8bc739cea165a5795e4adca8be5f894 for reason nil
> 18:22:00.963 [info] open_result error {not_found,no_db_file} for _users
> 18:22:00.982 [info] needed 18.134 ms to open new _users
> 18:22:01.030 [info] open_result error {not_found,no_db_file} for
> _replicator
> 18:22:01.037 [info] needed 6.67 ms to open new _replicator
> Apache CouchDB has started. Time to relax.
> 18:22:01.050 [info] Apache CouchDB has started on http://127.0.0.1:5986/
> 18:22:01.050 [info] Application couch started on node
> 'couchdb@andwen-2.local'
> 18:22:01.064 [info] Application rexi started on node
> 'couchdb@andwen-2.local
> '
> 18:22:01.075 [error] Could not open file /opt/couchdb/var/lib/nodes.couch:
> no such file or directory
> 18:22:01.075 [info] open_result error {not_found,no_db_file} for nodes
> 18:22:01.081 [info] needed 5.419 ms to open new nodes
> 18:22:01.100 [error] Could not open file /opt/couchdb/var/lib/dbs.couch: no
> such file or directory
> 18:22:01.100 [info] Application mem3 started on node
> 'couchdb@andwen-2.local
> '
> 18:22:01.100 [info] open_result error {not_found,no_db_file} for dbs
> 18:22:01.100 [info] Application fabric started on node
> 'couchdb@andwen-2.local'
> 18:22:01.101 [error] Error in process <0.199.0> on node
> 'couchdb@andwen-2.local' with exit value:
>
> {{badmatch,file_exists},[{mem3_shards,fold,2,[{file,"src/mem3_shards.erl"},{line,91}]},{mem3_sync,initial_sync,1,[{file,"src/mem3_sync.erl"},{line,242}]}]}
>
>
> 18:22:01.106 [info] needed 5.706 ms to open new dbs
> 18:22:01.135 [info] Application chttpd started on node
> 'couchdb@andwen-2.local'
> 18:22:01.135 [info] Application couch_replicator started on node
> 'couchdb@andwen-2.local'
> 18:22:01.135 [info] Application ets_lru started on node
> 'couchdb@andwen-2.local'
> 18:22:01.153 [info] Application ddoc_cache started on node
> 'couchdb@andwen-2.local'
> 18:22:01.177 [info] Application runtime_tools started on node
> 'couchdb@andwen-2.local'
> 18:22:01.177 [info] Application snappy started on node
> 'couchdb@andwen-2.local'
>
> So looking at localhost:5986 I received
>
> {
>
>- couchdb: "Welcome",
>- uuid: "b8bc739cea165a5795e4adca8be5f894",
>- version: "6dcd33b"
>
> }
>
> And localhost:5986/_all_dbs/
>
> [
>
>- "_replicator",
>- "_users",
>- "dbs",
>- "nodes"
>
> ]
>
> Futon is not working at localhost:5986/_utils/
>
> So far ...
>
> Cheers
>
> Andy
>
> --
> Andy Wenk
> Hamburg - Germany
> RockIt!
>
> http://www.couchdb-buch.de
> http://www.pg-praxisbuch.de
>
> GPG fingerprin
On the Viability of Erlang Releases and CouchDB
# On the Viability of Erlang Releases and CouchDB There has been some discussion on what versions of Erlang CouchDB should support, and what versions of Erlang are detrimental to use. Sadly there were some pretty substantial problems in the R15 line and even parts of R16 that are landmines for CouchDB. This post will describe the current state of things and make some potential recommendations on approach. ## Scheduler Collapse It was discovered by Basho that R15* and R16B are susceptible to scheduler collapse. There's quite a bit of discussion and information in several threads [1] [2] [3] [4] [5]. So what is scheduler collapse? Erlang schedulers can be put to sleep when there is not sufficient work to occupy all schedulers, which saves on CPU and power consumption. When the schedulers that are still running go through enough reductions to pass the work balancing threshold, they can trigger a rebalance of work that will wake up sleeping schedulers. The other mechanism for sharing scheduler load is work stealing. A scheduler that does not have any work to do can steal work from other schedulers. However a scheduler that has gone to sleep cannot steal work, it has to be woken up separately. Now the real problem of scheduler collapse occurs when you take sleeping schedulers and long running NIFs and BIFs that do not report an appropriate amount of reductions. When you have NIFs and BIFs that don't report an appropriate amount of reductions, you can get into a situation where a long running function call will only show up as taking one reduction, and never hit the work balance threshold, causing that scheduler to be blocked during the operation and no additional schedulers getting woken up. I keep mentioning "NIFs and BIFs" because it's important to note that it is _not_ just user defined NIFs that are problematic, but also a number of Erlang BIFs that don't properly report reductions. Particularly relevant to CouchDB are the BIFs `term_to_binary` and `binary_to_term` which do _not_ behave properly, and each report a single reduction count, regardless of the size of the value passed to them. Given that every write CouchDB makes goes through `term_to_binary`, this is definitely not good. This problem is systemic to all versions of R15 and R16B. In R16B01, two changes were made to alleviate the problem. First, in OTP-11163 `term_to_binary` now uses an appropriate amount of reductions and will yield back to the scheduler. The second important change was the introduction of the `+sfwi` (Scheduler Forced Wakeup Interval) flag [6] which allows you to specify a time interval for a new watchdog process to check scheduler run queues and wake up sleeping schedulers if need be. These two changes help significantly, although from what I understand, they do not fully eliminate scheduler collapse. *NOTE*: the `+sfwi` is _not_ enabled by default, you must specify a greater than zero time interval to enable this. *WE NEED TO ENABLE THIS SETTING.* We should figure out a way to conditionally add this to vm.args or some such. On a side note, Basho runs R15B01 because they backported the `+sfwi` feature to R15B01 [7] [8]. They recommend running with `+sfwi 500` for a 500ms interval. It might be worth testing out different values, but 500 seems like a good starting point. For Riak 2.0, they will be building against R16B03-1 and 17.0 as their set of patches to R16B02 landed in R16B03-1 [9] [10]. ## R16B01 and the breaking of monitors So R16B01 sorted out the scheduler collapse issues, but unfortunately it also broke monitors, which immediately disqualifies this release as something we should recommend to users. The issues was fixed in OTP-11225 in R16B02. ## R16B02 and R16B03* I don't know of any catastrophic problems on the order of those described above in either of these releases. Basho fixed a number of unrelated bugs in R16B02 [9] [10] that have since landed in R16B03-1, which indicates we should probably prefer R16B03-1 over R16B02. R16B03 is also disqualified because it broke SSL and `erl_syntax`, resulting in the patched R16B03-1. ## R14 R14B01, R14B03, and R14B04 are known good stable releases of Erlang, and in my opinion the only known stable releases > R13 that don't present issues for CouchDB (I think R16B02/R16B03-1 are too new to declare stable yet). As for R14B02, there are some bad `ets` issues with that release. It's worth pointing out that there are two known bugs in R14B01, as Robert Newson explains: ``` There are two bugs in R14B01 that we do encounter, however. 1) Another 32/64 bit oops causes the vm to attempt to allocate huge amounts of ram (terabytes, or more) if it ever tries to allocate more than 2gib of ram at once. When this happens, the vm dies and is restarted. It’s annoying, but infrequent. 2) Sometimes when closing a file, the underlying file descriptor is *not* closed, though the erlang process exits. This is rare but still quite annoying. ``` # Recommendations for CouchDB In my
Re: [DISCUSS] Apache CouchDB Diversity Statement.
+1 On Thu, May 1, 2014 at 10:42 AM, Jan Lehnardt wrote: > +1 > > On 29 Apr 2014, at 11:23 , Robert Kowalski wrote: > > > +1 > > > > > > 2014-04-29 10:07 GMT+02:00 Garren Smith : > > > >> +1 This is a great idea. > >> > >> On 28 Apr 2014, at 11:34 PM, Paul Davis > >> wrote: > >> > >>> +1, especially to broadening the definition of contributions that we > >> recognize. > >>> > >>> On Mon, Apr 28, 2014 at 3:59 PM, Andy Wenk wrote: > two docs make sense imho. +1 > > Cheers > > Andy > > > On 28 April 2014 22:44, Robert Samuel Newson > >> wrote: > > > Count me in. > > > > B. > > > > On 28 Apr 2014, at 21:09, Noah Slater wrote: > > > >> Yep. Happy to see what the drafts look like. If two docs make sense, > >> two docs make sense. > >> > >> On 28 April 2014 22:04, Joan Touzet wrote: > >>> Nice. I personally would still like to see Diversity broken out in > a > > separate document, but the idea of a charter is very good and this > >> covers a > > lot of ground. > >>> > >>> This also dovetails very nicely with the mentorship ideas I have > > floated here and on IRC in the past. > >>> > >>> +1, let's do this! > >>> > >>> - Original Message - > >>> From: "Noah Slater" > >>> To: dev@couchdb.apache.org > >>> Sent: Monday, April 28, 2014 3:50:15 PM > >>> Subject: Re: [DISCUSS] Apache CouchDB Diversity Statement. > >>> > >>> I have two things I'd like to add. > >>> > >>> 1. I'd like to widen this bit and turn it into a PMC charter. > >>> Basically a document that sets out what the PMC values, our > >> commitment > >>> to diversity, and our pledge to the project. It's basically the > same > >>> idea, but with a flourishes. > >>> > >>> The closest we have to this at the moment is here: > >>> > >>> > > > >> > https://git-wip-us.apache.org/repos/asf?p=couchdb-admin.git;a=blob_plain;f=email/invite_pmc.txt;hb=HEAD > >>> > >>> I'd like to improve on this. > >>> > >>> 2. Technical ability is something I feel very strongly about. We've > >>> made a huge blunder IMO in requiring past contributors to be highly > >>> technical. I would like to introduce and officialise the same > policy > >>> as Apache Forrest. > >>> > >>> cf. https://forrest.apache.org/committed.html > >>> > >>> Summary: we recognise merit in four areas: > >>> > >>> (Co)mmunity - one must interact with others, and share vision and > > knowledge > >>> (P)roject - a clear vision and consensus are needed > >>> (Do)cumentation - without it, the stuff remains only in the minds > of > > the authors > >>> (C)ode - discussion goes nowhere without code > >>> > >>> I would like to think that we would elect someone who is blogging > for > >>> us, or doing marketing, or speaking a lot. These are all important > >>> people, and their importance to the project should be recognised. > >>> > >>> It's been about two years since I started pushing on this topic at > >> the > >>> PMC level and there have been big cultural changes. I am confident > >>> that I could elect people with contributions in any of these areas. > >>> But I want to make it official. > >>> > >>> This is what we value. This is what we will recognise. Our promise > to > >>> the community. > >>> > >>> > >>> > >>> -- > >>> Noah Slater > >>> https://twitter.com/nslater > >> > >> > >> > >> -- > >> Noah Slater > >> https://twitter.com/nslater > > > > > > > -- > Andy Wenk > Hamburg - Germany > RockIt! > > http://www.couchdb-buch.de > http://www.pg-praxisbuch.de > > GPG fingerprint: C044 8322 9E12 1483 4FEC 9452 B65D 6BE3 9ED3 9588 > > https://people.apache.org/keys/committer/andywenk.asc > >> > >> > >
Re: [ANNOUNCE] Joan Touzet joins the PMC
Congrats Joan! -Russell On Thu, Apr 10, 2014 at 10:55 AM, Noah Slater wrote: > Oh man, that made me nostalgic. I miss Monty. And I'm SO happy you're > joining the PMC. Welcome aboard Joan. :) > > On 10 April 2014 18:30, Joan Touzet wrote: > > Noah and the Apache CouchDB PMC, thank you for nominating me to join > > your number. > > > > Boy, how times have changed! In May 2009, I was a month or so into PhD > > research. I popped on IRC to ask my first CouchDB question - why a view > > was never returning - and stumbled upon a way to get invalid UTF-8 data > > into Couch, but not back out. Jan, Damien, Monty and Paul helped me out, > > though Paul was occasionally AFK looking at new apartment listings, and > > of course Monty was barking mad. > > > > People were friendly, engaged, and genuinely interested in the bug that > > became COUCHDB-345. It took 4 months to fix, and ended up with a patch > > set upstream to mochiweb as well. > > > > Since then I've been a non-stop user of CouchDB, including incorporating > > it into a p2p secure asset sharing system, an artefact repository for > > software release & deployment, as a backend for some top 10 mobile phone > > games and of course work on CouchDB, bigcouch and Cloudant themselves. > > > > I remain convinced that CouchDB is the sleeper NoSQL tech that's winning > > in the long run. > > > > We've got a lot of bugs to squash still, 2 big merges to get done and a > > whole lot of great requests and suggestions from the community at large. > > I look forward to doing everything I can to facilitate that work, and a > > few of my own patches as well. > > > > Rock on, > > Joan > > > > > > - Original Message - > > From: "Noah Slater" > > To: dev@couchdb.apache.org > > Sent: Thursday, April 10, 2014 4:57:24 AM > > Subject: [ANNOUNCE] Joan Touzet joins the PMC > > > > Dear community, > > > > I am delighted to announce that Joan Touzet joins the Apache CouchDB > > Project Management Committee today. > > > > Joan has made outstanding, sustained contributions to the project. > > This appointment is an official acknowledgement of their position > > within the community, and our trust in their ability to provide > > oversight for the project. > > > > Everybody, please join me in congratulating Joan! > > > > On behalf of the CouchDB PMC, > > > > -- > > Noah Slater > > https://twitter.com/nslater > > > > -- > Noah Slater > https://twitter.com/nslater >
[jira] [Commented] (COUCHDB-1993) Make fabric work with the refactored view engine
[ https://issues.apache.org/jira/browse/COUCHDB-1993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13963191#comment-13963191 ] Russell Branca commented on COUCHDB-1993: - Hey thanks [~benoitc]! I missed the notifications on your comments, will respond shortly. > Make fabric work with the refactored view engine > > > Key: COUCHDB-1993 > URL: https://issues.apache.org/jira/browse/COUCHDB-1993 > Project: CouchDB > Issue Type: Improvement > Components: BigCouch >Reporter: Volker Mische > Assignee: Russell Branca > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (COUCHDB-1993) Make fabric work with the refactored view engine
[ https://issues.apache.org/jira/browse/COUCHDB-1993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962301#comment-13962301 ] Russell Branca commented on COUCHDB-1993: - For posterity, I'm pasting the copy of my review request email here: This thread is a request for review on the integration of fabric and couch_mrview, although there are changes in chttpd as well. There are three pull requests, one for each repo: https://github.com/apache/couchdb-fabric/pull/1 https://github.com/apache/couchdb-couch-mrview/pull/1 https://github.com/apache/couchdb-chttpd/pull/1 Overall this is actually a sizable reduction in code. While doing the integration, I started down the path of deduplicating code the single node vs clustered http layers, and as a result these updates are a net negative in lines of code and I think does a good job of separating out the view logic from fabric and chttpd. There are a few items that still need to be addressed, but I think they are outside of the scope of this initial integration and/or could use wider discussion on the proper way to address them * Properly send metadata through fabric, more details in the commit message of: https://github.com/apache/couchdb-fabric/commit/aeac2dfe5014dc9fe1f10300669ea9244cae3a67 * Handle security databases, ie _users/_replication doc security * Figure out what approach to take for etags and standardize etag functions * Allow for sending update_seq in the view results Let's use this thread for wider discussion and use the PRs for comments on code, unless there's a better alternative. I'll be around today and tomorrow, but then I'll be away from my computer the following week on vacation, so it might be a delayed response before I can address any comments. Thanks for the review! > Make fabric work with the refactored view engine > > > Key: COUCHDB-1993 > URL: https://issues.apache.org/jira/browse/COUCHDB-1993 > Project: CouchDB > Issue Type: Improvement > Components: BigCouch > Reporter: Volker Mische >Assignee: Russell Branca > -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: [POLL] Erlang whitespace standards
Thanks for putting this together Joan. I left a few "other" comments in places where you did not have indent by four spaces. I personally think we should either follow the four space indent everywhere (python style), or just use whatever emacs erlang mode does, as that's the erlang community standard. -Russell On Mar 29, 2014 1:22 PM, "Jan Lehnardt" wrote: > Heya Joan, > > I love everything about this effort, great idea & execution, thanks! > > I cast my votes and hte results are fascinating to watch :) > > Best > Jan > -- > > On 28 Mar 2014, at 22:11 , Joan Touzet wrote: > > > Time for everyone's favourite topic: whitespace standards. > > > > I know many of you are fed up with not being able to auto format in > > your favourite editor and match the CouchDB Erlang coding standards, or > > receiving pull requests that are formatted poorly. > > > > I'd like to fix that with an appropriate whitespace standard, and > > supplementary plugins for vi and emacs that will just Do The Right Thing > > and let us all stop worrying about whitespace corrections in pull > > requests. > > > > The basic rules seem to be: > > > >* Indent everything 4 spaces (not 2 or 8, and no tabs) > >* Single blank lines between functions > >* No blank lines between guarded versions of the > > same function (e.g. couch_btree.erl#L36-L47) > > > > but beyond that there is some inconsistency in both the code and in > > discussion with other devs. > > > > Here's a short poll I threw together in Google Docs. I'd love it if you > > could take 5 minutes to reply to it. Early next week I'll summarize and > > post the results. Once we have agreement we can toss up a super small > > Markdown guide / wiki page / whatever and get started on the emacs/vi > > modifications to support it. > > > > > https://docs.google.com/forms/d/1b7KcQGgNbSCZVRwLjrUl5Z6C2TBx8X1btlU5fwrNHpg/edit# > > > > Thanks, > > Joan > >
Re: [ANNOUNCE] Robert Kowalski elected as CouchDB committer
Congrats and welcome board Robert! -Russell On Fri, Mar 28, 2014 at 8:50 AM, Dave Cottlehuber wrote: > Great to have you join us Robert :-)) > > On 28 March 2014 15:19, Nick North wrote: > > Congratulations, and welcome! > > > > Nick > > > > > > On 28 March 2014 13:26, Klaus Trainer wrote: > > > >> Great! Welcome Robert :) > >> > >> > >> On 03/28/2014 01:12 PM, Andy Wenk wrote: > >> > Dear community, > >> > > >> > I am pleased to announce that the CouchDB Project Management Committee > >> has > >> > elected Robert Kowalski as a CouchDB committer. > >> > > >> > Apache ID: robertkowalski > >> > > >> > IRC nick: robertkowalski > >> > > >> > Twitter: robinson_k > >> > > >> > By default, external contributions to the project follow the > >> > Review-Then-Commit model. Being a committer means you can follow the > >> > Commit-Then-Review model. In other words, Robert can now make changes > at > >> > will. > >> > > >> > This election was made in recognition of Robert's existing > contributions > >> > and commitment to the project. > >> > > >> > Please join me in extending a warm welcome to Robert! > >> > > >> > On behalf of the CouchDB PMC, > >> > > >> > Andy > >> > > >> >
[REVIEW] COUCHDB-1993 fabric + couch_mrview integration
Hi all, This thread is a request for review on the integration of fabric and couch_mrview, although there are changes in chttpd as well. There are three pull requests, one for each repo: https://github.com/apache/couchdb-fabric/pull/1 https://github.com/apache/couchdb-couch-mrview/pull/1 https://github.com/apache/couchdb-chttpd/pull/1 Overall this is actually a sizable reduction in code. While doing the integration, I started down the path of deduplicating code the single node vs clustered http layers, and as a result these updates are a net negative in lines of code and I think does a good job of separating out the view logic from fabric and chttpd. There are a few items that still need to be addressed, but I think they are outside of the scope of this initial integration and/or could use wider discussion on the proper way to address them * Properly send metadata through fabric, more details in the commit message of: https://github.com/apache/couchdb-fabric/commit/aeac2dfe5014dc9fe1f10300669ea9244cae3a67 * Handle security databases, ie _users/_replication doc security * Figure out what approach to take for etags and standardize etag functions * Allow for sending update_seq in the view results Let's use this thread for wider discussion and use the PRs for comments on code, unless there's a better alternative. I'll be around today and tomorrow, but then I'll be away from my computer the following week on vacation, so it might be a delayed response before I can address any comments. Thanks for the review! -Russell
Re: [ANNOUNCE] Max Thayer elected as CouchDB committer
Welcome Max! -Russell On Tue, Feb 25, 2014 at 3:33 AM, Andy Wenk wrote: > Hi Max - welcome to the club :) > > Cheers > > Andy > > > On 25 February 2014 12:20, Noah Slater wrote: > > > Dear community, > > > > I am pleased to announce that the CouchDB Project Management Committee > > has elected Max Thayer as a CouchDB committer. > > > > Apache ID: garbados > > > > IRC nick: garbados > > > > Twitter: garbados > > > > By default, external contributions to the project follow the > > Review-Then-Commit model. Being a committer means you can follow the > > Commit-Then-Review model. In other words, Max can now make changes at > > will. > > > > This election was made in recognition of Max's existing contributions > > and commitment to the project. > > > > Please join me in extending a warm welcome to Max! > > > > On behalf of the CouchDB PMC, > > > > -- > > Noah Slater > > https://twitter.com/nslater > > > > > > -- > Andy Wenk > Hamburg - Germany > RockIt! > > http://www.couchdb-buch.de > http://www.pg-praxisbuch.de > > GPG fingerprint: C044 8322 9E12 1483 4FEC 9452 B65D 6BE3 9ED3 9588 > > https://people.apache.org/keys/committer/andywenk.asc >
Re: [ANNOUNCE] Benjamin Young elected as CouchDB committer
Congrats Ben! -Russell On Tue, Feb 25, 2014 at 7:19 AM, Benoit Chesneau wrote: > On Tue, Feb 25, 2014 at 3:12 PM, Benjamin Young > wrote: > > > > On 2/25/14, 7:37 AM, Dave Cottlehuber wrote: > > On 25 February 2014 09:53, Robert Samuel Newson wrote: > > > Dear community, > > > > I am pleased to announce that the CouchDB Project Management > Committee > > has > > elected Benjamin Young as a CouchDB committer. > > > > Apache ID: bigbluehat > > > > IRC nick: bigbluehat > > > > Twitter: bigbluehat > >> > >> congratulations Ben! > >> > >> IIRC you were one of the first people I met wrt to CouchDB back at Couch > >> Camp 2010! > >> > > > > Yeah. We were roommates. :) > > > > Good memories. > > > > +1 to another one of those "wifi-from-the-trees" events. :D > > welcome on board :) - benoit >
Re: [ANNOUNCE] Andy Wenk joins the PMC
Welcome aboard Andy! -Russell On Sat, Feb 22, 2014 at 11:14 AM, Klaus Trainer wrote: > Hey, congrats Andy! > > > On 02/21/2014 09:09 PM, Noah Slater wrote: > > Dear community, > > > > I am delighted to announce that Andy Wenk joins the Apache CouchDB > > Project Management Committee today. > > > > Andy has made outstanding, sustained contributions to the project. > > This appointment is an official acknowledgement of his position within > > the community, and our trust in his ability to provide oversight for > > the project. > > > > Everybody, please join me in congratulating Andy! > > > > On behalf of the CouchDB PMC, > > > >
Re: [ANNOUNCE] Alexander Shorin joins the PMC
Congrats Alexander! (: -Russell On Sat, Feb 22, 2014 at 2:51 PM, Dave Cottlehuber wrote: > On 22. Februar 2014 at 18:59:19, Alexander Shorin (kxe...@gmail.com) > wrote: > > Thank you all for kind words and nice welcome! It's really a honor for > > me to join PMC. I'm happy to be there with you. Thanks! > > > > P.S. nice joke, Jan (: > > > > -- > > ,,,^..^,,, > > welcome on board Alex!! > > A+ > Dave > > >
[jira] [Commented] (COUCHDB-2079) Re-enable configurable HTTP handlers
[ https://issues.apache.org/jira/browse/COUCHDB-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13907678#comment-13907678 ] Russell Branca commented on COUCHDB-2079: - +1 to having an empty http layer that allows app to plug into it. I think it would also be interesting to update the handler functions to be less dependent on http, basically keep http isolated to chttpd, and have all the other things return erlang data structures. Basically make http a particular interface type, allowing for alternative interfaces or direct access. > Re-enable configurable HTTP handlers > > > Key: COUCHDB-2079 > URL: https://issues.apache.org/jira/browse/COUCHDB-2079 > Project: CouchDB > Issue Type: Improvement > Security Level: public(Regular issues) > Components: BigCouch >Reporter: Paul Joseph Davis > Labels: release > > chttpd hard codes HTTP handlers rather than lists them in the config files. > We'll need to revert this change. Though given how some apps (couch_mrview, > etc) have their own handlers we should consider making this configurable at > the app-level instead of the ini level in CouchDB. > I have a few ideas on this, mostly by removing all handler code from chttpd > and friends. The HTTPd app would then just start empty servers with no > handlers and each app would have a supervision tree process that would be > responsible for registering handlers. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
Re: [ANNOUNCE] Klaus Trainer elected as CouchDB committer
Welcome Klaus! -Russell On Wed, Feb 19, 2014 at 6:08 AM, Dave Cottlehuber wrote: > > > > > > On Tue, Feb 18, 2014 at 8:23 PM, Noah Slater wrote: > > > > > > > Dear community, > > > > > > > > > > > > > > I am pleased to announce that the CouchDB Project Management > Committee > > > > > > > has elected Klaus Trainer as a CouchDB committer. > > > > > > > > > > > > > > Apache ID: klaus_trainer > > > > > > > > > > > > > > IRC nick: klaus_trainer > > > > > > > > > > > > > > Twitter: KlausTrainer > > > > > > > > > > > > > > By default, external contributions to the project follow the > > > > > > > Review-Then-Commit model. Being a committer means you can > follow the > > > > > > > Commit-Then-Review model. In other words, Klaus can now make > changes > > > > > > > at will. > > > > > > > > > > > > > > This election was made in recognition of Klaus's existing > > > > > > > contributions and commitment to the project. > > > > > > > > > > > > > > Please join me in extending a warm welcome to Klaus! > > > > > > > > > > > > > > On behalf of the CouchDB PMC, > > > > > > > > > > > > > > -- > > > > > > > Noah Slater > > > > > > > https://twitter.com/nslater > > \o/ long awaited! > > A+ > Dave > > >
[jira] [Assigned] (COUCHDB-523) View API POST keys to retrieve multiple docs by key could also allow for multiple 'range' queries, i.e. an array of { startkey: .., endkey: ... } params in the POST
[
https://issues.apache.org/jira/browse/COUCHDB-523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russell Branca reassigned COUCHDB-523:
--
Assignee: Russell Branca (was: Joan Touzet)
> View API POST keys to retrieve multiple docs by key could also allow for
> multiple 'range' queries, i.e. an array of { startkey: .., endkey: ... }
> params in the POST
>
>
> Key: COUCHDB-523
> URL: https://issues.apache.org/jira/browse/COUCHDB-523
> Project: CouchDB
> Issue Type: Improvement
> Components: HTTP Interface
>Reporter: Nathan Stott
>Assignee: Russell Branca
>Priority: Minor
> Attachments: couch_httpd_view.erl, multi_start_end_key.diff,
> ranged_key_post.diff
>
>
> It would be useful if I could do a single POST to a view to retrieve multiple
> ranges specified by startkey, endkey.
> The format could be as follows:
> { "ranges": [ { "startkey": "a", "endkey": "c" }, { "startkey":"g",
> "endkey":"z" } ] }
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Assigned] (COUCHDB-1993) Make fabric work with the refactored view engine
[ https://issues.apache.org/jira/browse/COUCHDB-1993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Russell Branca reassigned COUCHDB-1993: --- Assignee: Russell Branca > Make fabric work with the refactored view engine > > > Key: COUCHDB-1993 > URL: https://issues.apache.org/jira/browse/COUCHDB-1993 > Project: CouchDB > Issue Type: Improvement > Components: BigCouch >Reporter: Volker Mische > Assignee: Russell Branca > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Closed] (COUCHDB-2045) Trace trap error while starting CouchDB
[
https://issues.apache.org/jira/browse/COUCHDB-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russell Branca closed COUCHDB-2045.
---
> Trace trap error while starting CouchDB
>
>
> Key: COUCHDB-2045
> URL: https://issues.apache.org/jira/browse/COUCHDB-2045
> Project: CouchDB
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: BigCouch
> Reporter: Russell Branca
>
> I ran into an error today in the 1843-feature-bigcouch branch, where running
> `./rel/dev1/bin/couchdb` would result in the VM crashing with a trace trap
> error. I tracked this down to the couch_primary_sup child spec for the
> `collation_driver` when calling `couch_drv:start_link()`, and more
> particularly, I was able to reproduce this error in a standard Erlang shell
> by running the command:
> `erl_ddll:load("/tmp/bigcouch/rel/dev1/lib/couch-2f254d9/priv",
> "couch_icu_driver").`
> After poking around with things, I ended up stumbling upon:
> http://openradar.appspot.com/7209349 (linked from
> http://erlang.org/pipermail/erlang-questions/2010-April/050563.html). I was
> able to resolve the error by doing `export
> DYLD_INSERT_LIBRARIES=/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation`
> in the shell prior to executing `./rel/dev1/bin/couchdb`.
> This seems like a logical addition to the flags set in
> `src/couch/rebar.config.script`, but I'm not familiar with CoreFoundation, so
> I don't have a recommendation on the proper way to do this across platforms.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (COUCHDB-2045) Trace trap error while starting CouchDB
[
https://issues.apache.org/jira/browse/COUCHDB-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13891033#comment-13891033
]
Russell Branca commented on COUCHDB-2045:
-
Thanks Newson, I can confirm the patch fixed the issue for me and I no longer
need to set the `DYLD_INSERT_LIBRARIES` environment variable.
> Trace trap error while starting CouchDB
>
>
> Key: COUCHDB-2045
> URL: https://issues.apache.org/jira/browse/COUCHDB-2045
> Project: CouchDB
> Issue Type: Bug
> Security Level: public(Regular issues)
> Components: BigCouch
> Reporter: Russell Branca
>
> I ran into an error today in the 1843-feature-bigcouch branch, where running
> `./rel/dev1/bin/couchdb` would result in the VM crashing with a trace trap
> error. I tracked this down to the couch_primary_sup child spec for the
> `collation_driver` when calling `couch_drv:start_link()`, and more
> particularly, I was able to reproduce this error in a standard Erlang shell
> by running the command:
> `erl_ddll:load("/tmp/bigcouch/rel/dev1/lib/couch-2f254d9/priv",
> "couch_icu_driver").`
> After poking around with things, I ended up stumbling upon:
> http://openradar.appspot.com/7209349 (linked from
> http://erlang.org/pipermail/erlang-questions/2010-April/050563.html). I was
> able to resolve the error by doing `export
> DYLD_INSERT_LIBRARIES=/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation`
> in the shell prior to executing `./rel/dev1/bin/couchdb`.
> This seems like a logical addition to the flags set in
> `src/couch/rebar.config.script`, but I'm not familiar with CoreFoundation, so
> I don't have a recommendation on the proper way to do this across platforms.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (COUCHDB-2045) Trace trap error while starting CouchDB
Russell Branca created COUCHDB-2045:
---
Summary: Trace trap error while starting CouchDB
Key: COUCHDB-2045
URL: https://issues.apache.org/jira/browse/COUCHDB-2045
Project: CouchDB
Issue Type: Bug
Security Level: public (Regular issues)
Components: BigCouch
Reporter: Russell Branca
I ran into an error today in the 1843-feature-bigcouch branch, where running
`./rel/dev1/bin/couchdb` would result in the VM crashing with a trace trap
error. I tracked this down to the couch_primary_sup child spec for the
`collation_driver` when calling `couch_drv:start_link()`, and more
particularly, I was able to reproduce this error in a standard Erlang shell by
running the command:
`erl_ddll:load("/tmp/bigcouch/rel/dev1/lib/couch-2f254d9/priv",
"couch_icu_driver").`
After poking around with things, I ended up stumbling upon:
http://openradar.appspot.com/7209349 (linked from
http://erlang.org/pipermail/erlang-questions/2010-April/050563.html). I was
able to resolve the error by doing `export
DYLD_INSERT_LIBRARIES=/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation`
in the shell prior to executing `./rel/dev1/bin/couchdb`.
This seems like a logical addition to the flags set in
`src/couch/rebar.config.script`, but I'm not familiar with CoreFoundation, so I
don't have a recommendation on the proper way to do this across platforms.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
Re: Pull Request comments
Wooo! Thanks Daniel! -Russell On Fri, Jan 24, 2014 at 7:32 AM, Benjamin Young wrote: > On 1/24/14, 8:49 AM, Garren Smith wrote: > >> Hi All, >> >> Thanks to the great work from Daniel aka Humbedooh, we now will receive >> pull request comments on the mailing list. We can also reply to the >> comments and they will be logged on the pull request. >> >> Cheers >> Garren >> > > Thanks, Daniel! >
Re: Migration Strategy [was Re: Git Repository Creation]
> 3. R15* & at least R16B01 and under are still prone to scheduler collapse. > > I am not sure if the > > “+st " hack in R16B02+ eases the pain or not, nor how badly this > > impacts heavy > > CouchDB users. This is the “wake up all schedulers every > to > > prevent collapse" > > issue. Read Scott Fritchie’s threads in Erlang-Questions [1] for details > & > > erl man[2]. > > > > > Mmm the link you refer say it is still better than in R14. Anyway this is > the right path to follow, we should collect any issues we have in mind in > using latest supported versions of Erlang and see how it impact us. > > I'm not sure where you see anything in DCH's link [1] that says R14 is worse. The relevant bits from that thread are "R15B0x's schedulers are broken", and "R16B's schedulers appear to be even more broken". The only mention of R14 I see on that thread is: "The CPU usage is noticeably lower than R14, but worryingly so.". I think we should support R14 until their is a viable stable alternative, or there is a sufficiently interesting upgrades to justify the switch. If we need to run forked versions of 3rd party dependencies until such time that we can upgrade in full, that seems like the logical path. Ideally we can figure out a way to use upstream versions directly without needing to fork. -Russell
Re: Git Repository Creation
The scheduler collapse problems in R15 and R16 are widely known and not resolved. Frankly, as developers of a database, we should strive to provide end users with the most reliable and best experience, which in my opinion means we should recommend R14B01. There is not a battle tested, reliable version of Erlang that has proven to solve the scheduler collapse problems, and until that time, I think it's unwise to remove support for R14. -Russell On Wed, Jan 22, 2014 at 6:20 AM, Benoit Chesneau wrote: > Robert, > > I understood what you meant. > > Imo the best thing would be creating a check list of the things that > prevent to go to a version greater than R14. Can you share the one you have > inside cloudant ? It will help us to reach a consensus also later to make > sure we can fix them in next Erlang releases. > > This is not that I want absolutely use the latest. If we stand on an old > and unmaintained release then we should know exactly why and check from > time to time if we still need to stay on this version. > > Thanks! > > - benoit > > > > > > On Wed, Jan 22, 2014 at 2:06 PM, Robert Samuel Newson >wrote: > > > > > I could have phrased it better, so I’ll do so now; > > > > R14 is still widely used in production and is very stable. R15 and R16 > > have known stability problems that affect deployments using NIF’s that > can > > potentially run for longer than a millisecond before returning control to > > the scheduler. > > > > I am not blackmailing the project but I hope you can understand how I > feel > > about your suggestion to remove the ability for Cloudant to continue > > working after we are making such a large contribution and, further, > seeking > > to move our active development to couchdb itself. > > > > B. > > > > On 22 Jan 2014, at 13:01, Benoit Chesneau wrote: > > > > > On Wed, Jan 22, 2014 at 1:58 PM, Dave Cottlehuber > > wrote: > > > > > >> On 22 January 2014 13:23, Benoit Chesneau > wrote: > > >>> On Wed, Jan 22, 2014 at 12:41 PM, Robert Samuel Newson > > >>> wrote: > > >>> > > Benoit, > > > > Cloudant requires R14 support, it would mean our contribution to > > couchdb > > becomes useless to us and we could not contribute further. > > > > >>> > > >>> Are you using blackmail? Is this the position of the Cloudant > company? > > >> > > >> Hi Benoit, > > >> > > >> Your comment reads like an ad hominem attack, and I don't think Bob's > > >> point, nor Bob, nor Cloudant, deserved that. > > >> > > > > > > My questions stand. The way it is formulated, and that's not the first > > > time, is not that clear at all. > > > > >
Re: we need more reviews
Huge +1 to more review. Let's also setup some code guidelines for Erlang/Javascript/C/HTML so we have an authoritative list of rules to follow to ensure code consistency, and similarly, let's get some guidelines for git commits messages as well, Tim Pope's article comes to mind: http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html. Whichever review tool we decide to use, we should use it universally. We should figure out a way to do github integration so that PR's show up in the tool, and then we update the PR with a link to the relevant review page. We also need a good way to initiate the analogue of a PR in whatever tool we use. One of the complications with the review process today, is that we're all accustomed to doing review through github PR's, but we can use github PR's for branches who exist solely on the ASF origin. I imagine this won't be an issue with Hive or Gerrit, but we should verify that we're not required to initiate code review from third party origins. On a side note, one of my personal nits with github PR's, that I hope might be resolved with one of these tools, is losing comments when the code changes. For instance, if I do a review and make 10 comments that each represent a line item that needs to be addressed, I would like to see a task list on the review indicating there are unfinished items to be addressed, basically a way to make a set of checkboxes necessary to complete the review. I find it odd that github doesn't do this, even more so by the fact that changing code to address comments will often mess with the commenting and hide the messages you posted, making it challenging to decipher whether the review items have been addressed. This is by no means a requirement to using any approach, just hoping that someone knows of a good way to approach this problem in whatever tool we use. -Russell On Wed, Jan 22, 2014 at 6:06 AM, Benoit Chesneau wrote: > On Wed, Jan 22, 2014 at 12:47 PM, Noah Slater wrote: > > > My first comment is: if we want more reviews, let's have more committers. > > > > We double our committer base in 2013, and the results look like this: > > > > https://www.ohloh.net/p/couchdb/analyses/latest/languages_summary > > > > And I see comments like this on StackOverflow: > > > > "I've recently noticed that Couch DB is back in heavy development." > > > > So I think we should continue to aggressively recruit more committers > > to the project in 2014. Excelsior! > > > > Welcoming new committers in the project is certainly a good thing but I > think it's orthogonal to the need of more review and team work. We should > find a good workflow now while we are still a limited number of committers > because It will be harder to enforce anything on a larger team. We should > settle on some guidelines now. It will also help us to attract more people > IMO. > > > > > > However, as for the way we do reviews... > > > > Infra ticket about Gerrit > > https://issues.apache.org/jira/browse/INFRA-2205 > > > > Effort seems to be mothballed. But I'm sure we could restart it if we > > were serious. > > > > However, we could use this: > > > > https://reviews.apache.org/groups/hive/ > > > > It is well integrated with Apache infrastructure already, sends mails > > to the mailing list, and so on. > > > > Happy to request an instance and we can experiment with it if we like. > > If it doesn't work out, we stop using it. No harm. Could make it > > entirely voluntary until we figure out a workflow that we all like. > > > > Should I do that? > > > > I thought gerrit was already integrated but any tool that could help us to > make the review is OK for me. How hive compares to gerrit? Could we also > plug it with github like gerrit [1] ? > > - benoît > > [1] https://gerrit.googlesource.com/plugins/github/ (and some others) > > > > > On 20 January 2014 15:30, Nick North wrote: > > > On 20 January 2014 12:26, Dale Harvey wrote: > > > > > >> I fully agree, its something I mentioned at the couchdb conf in > > vancouver, > > >> a review first system encourages contributions and has multiple > benefits > > >> > > >> * At least 2 people look at the code, less likely to push silly > > mistakes > > >> * Can codify and practice review rules > > >> * Its much easier to view the current activity in the project > > >> * Can bring up points of discussion before its too late > > >> > > >> But I think the most important thing is that it removes the burden of > > >> responsibility from the committer to the project as a whole, also > hugely > > >> beneficial is that it makes it particularly obvious when a patch has > > reach > > >> a stalemate and forces someone to make the call. > > >> > > >> For reference on PouchDB every committer is trusted to push code, > nobody > > >> (including myself) pushes their own code to master, it goes in the PR > > queue > > >> and gets a +1 from any other committer (who will usually push it), > thats > > >> essentially the same process we use at m
Re: Git Repository Creation
On Tue, Jan 21, 2014 at 10:51 AM, Alexander Shorin wrote: > On Tue, Jan 21, 2014 at 1:44 PM, Paul Davis > wrote: > > The repos for futon and jquery.couch.js are less > > clear. I'd prefer to avoid having infra create repos we don't intend > > to use beyond fancy archives. > > While I couldn't say for sure for Futon, jquery.couch.js is pretty > popular client which people uses, wanted to contribute to and I think > it shouldn't become abandoned deep within some archive repo. At some > point we're all agreed on the fact that's better to move it to > separate repo, no matter will this repo be under CouchDB flag or not - > but I think first one is better. > > -- > ,,,^..^,,, > Is anyone maintaining jquery.couch.js? There were some decent patches that slipped by due to lack of a maintainer (for instance the updates to use jquery promises). I don't think we should create a repo for jquery.couch.js unless we have someone who is planning on maintaining it long term. As for Futon, I think it's less relevant to move it to a separate repo because a) it's being replaced and b) there is minimal development happening there. -Russell
Re: improve the bigcouch and rcouch merges
Lot of great ideas in this thread. A number of the suggested items are feature and refactoring work, so I agree with Paul that we should separate those from the merges themselves, and concentrate on them after we've got the base pieces merged. Fabric works well for an internal API on the clustered bits. It could use a little work to make it more isolated from chttpd so that it's more directly usable. I think it makes sense to discuss making a more uniform API that provides similar functionality for clustered vs non clustered operations, but this is outside the scope of the merge. As Paul mentioned, I've got a number of ideas for ripping out auth into an isolated application that handles #user_ctx, roles, and user CRUD, which I want to dive into more after the merges. -Russell On Tue, Jan 21, 2014 at 5:41 AM, Robert Samuel Newson wrote: > > To follow on from Paul’s responses, we will be deleting fabric_rpc2 before > the merge. That exists only temporarily at Cloudant to ease upgrades, as > Paul notes, it has no place in the final source tree. > > B. > > On 21 Jan 2014, at 11:39, Paul Davis wrote: > > > Good points all around. Replying in line for context. > > > > On Tue, Jan 21, 2014 at 2:33 AM, Benoit Chesneau > wrote: > >> Hi all, > >> > >> I was reviewing the bigcouch and the rcouch branch this morning and I > >> think it is about time to start a real merge. Waiting that each merges > >> reach a working version before starting any work on the final product > >> is quite unproductive to say less. > >> > >> On the contrary what I see in the current branches let me think, it is a > >> good time to start some work to improve our code base in view of > >> achieving the 3 main goals we fixed a long time ago during the summit > >> and that were confirmed in December last year, ie.: > >> > >> - add a cluster facility to Apache CouchdDB > >> - allows Apache CouchDB to be embedded in other Erlang applications > >> (just like mnesia the standard database library in Erlang) > >> - make Apache CouchDB more *OTP*ish > >> > >> The changes in bigcouch are indeed quite more monolithic than the rcouch > >> changes by adapting Apache Couchdb to be able to run in a cluster > >> environment. While the cluster management part is quite isolated (mem3, > >> rexi, chttpd), others parts of bigcouch are wrapping or modifying the > >> apache couchdb internals: > >> > >> - fabric is adding fabric_rpc and fabric_rpc2 wrapping some modules and > >> functions in the couchdb core to be able to call them on the cluster. > >> It should be noted that the cluster part is using fabric to do all calls > >> to each couchdb nodes on the cluster and return/merge the results. > > > > For clarification the duplication of these modules is a side effect of > > running hot code upgrades. Conceptually fabric_rpc2 is the same as > > fabric_rpc, we just use the second module to do RPC API upgrades on > > live clusters. Not something to worry about extensively. > > > >> - some changes has been done to optimise the core: removing ref > >> counting, adding ets_lru to handle caching, db initialization has been > >> changed to look at the cluster to fetch ddocs to set validate funcs. > > > > The caching doesn't actually affect core. Its only used for clustered > > calls at the moment. The big thing here is definitely > > validate_doc_update functions being loaded lazily and the fact that > > single node code has to run clustered calls. I definitely agree that > > better internal APIs could probably clean this up. > > > >> - couch_replicator has been modified to work in a cluster > >> - bigcouch also changed the way the configuration is handled in couchdb, > >> making it more evented. Which is better than the current version but > >> still rely on an INI file. > > > > We did change the API to enforce some basic patterns to avoid common > > pitfalls in hot code upgrades but the fundamental operation is still > > the same. Unfortunately, as you point out, this doesn't change our > > reliance on INI files. > > > >> - logs are now handled by twig a specific application created by > >> Cloudant for that purpose. > >> > > > > We're looking to switch to lager as well. Twig was written pre-Lager > > and we haven't made the switch because it hasn't been a priority yet. > > We've got Lager queued up as a task to complete before the final > > merge. > > > >> > >> rcouch changes consist mostly in the following: > >> > >> - making Apache CouchDB an OTP release. It also extracts some code and > >> revisits the supervision like couch_httpd to transform it as a > >> standalone application, and make couch_replicator a full app > > > > For the record I removed all of the BigCouch stuff related to Erlang > > release handling because we started the merge when autotools was still > > a requirement. We do Erlang releases exclusively at Cloudant and > > switching to this is now going to be part of the merge. We do have > > some differences in some of the build tool
[jira] [Commented] (COUCHDB-1960) Provide pagination for /_all_dbs
[
https://issues.apache.org/jira/browse/COUCHDB-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13863177#comment-13863177
]
Russell Branca commented on COUCHDB-1960:
-
The BigCouch merge will change the list of databases to come from a database
all docs view, rather than a file system listing. As such, we'll want to
recommend standard view query params over using skip once that is brought in.
> Provide pagination for /_all_dbs
>
>
> Key: COUCHDB-1960
> URL: https://issues.apache.org/jira/browse/COUCHDB-1960
> Project: CouchDB
> Issue Type: Improvement
> Components: Fauxton, HTTP Interface
>Reporter: Benjamin Young
>
> The database-per-user design pattern has increased in popularity and
> consequently causes long lists of databases when one hits {{/_all_dbs}}
> To keep this sane, we need some pagination.
> {{?skip=10&limit=10}} style seems to be the simplest to implement (and there
> should be existing code from Cloudant for this one).
> Thanks!
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
Re: couchdb pull request: SOCKS 5 support for replication
This is very cool Newson! Big +1 for this landing. -Russell On Sat, Jan 4, 2014 at 10:35 AM, rnewson wrote: > GitHub user rnewson opened a pull request: > > https://github.com/apache/couchdb/pull/125 > > SOCKS 5 support for replication > > Notably, this allows replication over the Tor network (yes, DNS > resolution occurs over Tor also). > > > You can merge this pull request into a Git repository by running: > > $ git pull https://github.com/rnewson/couchdb socks5 > > Alternatively you can review and apply these changes as the patch at: > > https://github.com/apache/couchdb/pull/125.patch > > > commit 059fe032d5afe8f668effd2c3f81fa73ebe4031f > Author: Robert Newson > Date: 2014-01-04T17:32:00Z > > Add SOCKS5 module to ibrowse > > commit 3fbbc234a86600881d6a75d9cece4d2af8c168a3 > Author: Robert Newson > Date: 2014-01-04T17:31:40Z > > Pass protocol to ibrowse > > commit e529636f0a07c898ba9b1e788deaebffb7f871de > Author: Robert Newson > Date: 2014-01-04T17:57:22Z > > Clarify HTTP proxy fields and variables > > commit 65ba845f5705885b46842df93f8017af58841177 > Author: Robert Newson > Date: 2014-01-04T18:02:59Z > > Use SOCKS5 proxy if specified > > > >
Re: [ANNOUNCE] Nick North elected as CouchDB committer
Welcome Nick! -Russell On Thu, Jan 2, 2014 at 11:57 AM, Jan Lehnardt wrote: > Welcome aboard, Nick! :) > > Best > Jan > -- > > On 01 Jan 2014, at 20:24 , Dave Cottlehuber wrote: > > > Dear community, > > > > There's nothing like starting off the New Year with a New Committer!! > > > > I am pleased to announce that the CouchDB Project Management Committee > > has elected Nick North as a CouchDB committer. > > > >Apache ID: nicknorth > > > >IRC nick: NickN > > > >Twitter: @_shastra > > > > By default, external contributions to the project follow the > > Review-Then-Commit model. Being a committer means you can follow the > > Commit-Then-Review model. In other words, Nick can now make changes at > > will. > > > > This election was made in recognition of Nick's existing contributions > > and commitment to the project. > > > > Please join me in extending a warm welcome to Nick! > > > > On behalf of the CouchDB PMC, > > > > Dave Cottlehuber > >
[jira] [Commented] (COUCHDB-1963) Move from etap to eunit
[ https://issues.apache.org/jira/browse/COUCHDB-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13852259#comment-13852259 ] Russell Branca commented on COUCHDB-1963: - Sounds good [~kxepal]. I'm going to work on porting couch_index and couch_mrview. I'll push up a branch when I get somewhere with those. > Move from etap to eunit > --- > > Key: COUCHDB-1963 > URL: https://issues.apache.org/jira/browse/COUCHDB-1963 > Project: CouchDB > Issue Type: Improvement > Components: Test Suite >Reporter: Joan Touzet >Assignee: Alexander Shorin > Labels: etap, eunit, test > > Pursuant to an IRC/Twitter discussion, it seems that there is some > support ([~pjdavis1970], [~janl]) to move off of etap. Motivations include > etap being unmaintained for 2+ years, eunit shipping with Erlang, and least > importantly etap tests being completely broken on the Windows build. > There are only 63 .t files in the entire source tree; hopefully this is work > that can be weekend-tackled by one person, or could be tag-teamed with little > friction. > There was a subsequent discussion about the current, existing JS tests; while > there are issues with these and room for improvement, this bug is *strictly* > about our white-box etap unit tests and larger multi-instance tests that may > not best be accomplished by a JS framework. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (COUCHDB-1963) Move from etap to eunit
[ https://issues.apache.org/jira/browse/COUCHDB-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13852236#comment-13852236 ] Russell Branca commented on COUCHDB-1963: - +1 as well. I'm happy to help with this. Is there a branch where these changes are happening? Or should we just split it up per module or some such? > Move from etap to eunit > --- > > Key: COUCHDB-1963 > URL: https://issues.apache.org/jira/browse/COUCHDB-1963 > Project: CouchDB > Issue Type: Improvement > Components: Test Suite >Reporter: Joan Touzet >Assignee: Alexander Shorin > Labels: etap, eunit, test > > Pursuant to an IRC/Twitter discussion, it seems that there is some > support ([~pjdavis1970], [~janl]) to move off of etap. Motivations include > etap being unmaintained for 2+ years, eunit shipping with Erlang, and least > importantly etap tests being completely broken on the Windows build. > There are only 63 .t files in the entire source tree; hopefully this is work > that can be weekend-tackled by one person, or could be tag-teamed with little > friction. > There was a subsequent discussion about the current, existing JS tests; while > there are issues with these and room for improvement, this bug is *strictly* > about our white-box etap unit tests and larger multi-instance tests that may > not best be accomplished by a JS framework. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (COUCHDB-1960) Provide pagination for /_all_dbs
[
https://issues.apache.org/jira/browse/COUCHDB-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13846850#comment-13846850
]
Russell Branca commented on COUCHDB-1960:
-
I didn't spend much time optimizing the _all_dbs pagination because as Newson
mentioned in the other thread, it's doing a file system scan and returning all
the items. So if you have tens of thousands of databases, let alone millions,
you're going to be hosed either way. Although admittedly, making an additional
request per database doesn't help matters. When BigCouch is merged, the list of
databases will be represented in a database, and we will be able to do the
standard pagination queries on _all_dbs.
> Provide pagination for /_all_dbs
>
>
> Key: COUCHDB-1960
> URL: https://issues.apache.org/jira/browse/COUCHDB-1960
> Project: CouchDB
> Issue Type: Improvement
> Components: Fauxton, HTTP Interface
>Reporter: Benjamin Young
>
> The database-per-user design pattern has increased in popularity and
> consequently causes long lists of databases when one hits {{/_all_dbs}}
> To keep this sane, we need some pagination.
> {{?skip=10&limit=10}} style seems to be the simplest to implement (and there
> should be existing code from Cloudant for this one).
> Thanks!
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
