Chewbranca's Opinionated Vision for Apache CouchDB and Cloudant Dbcore

[Russell Branca]

Hi all,

Chewbranca here. I've been fairly quiet for the last few years for a number of
reasons, but this year I've been becoming more involved again in the CouchDB
community and various Cloudant communities. I put together this writeup a while
ago to detail out what my personal views are on what I believe to be some of the
highest priority focus points for CouchDB/Cloudant and targeted approaches
for tackling these issues. Full disclosure, these are my own opinions discussing
my views on direction for Cloudant and CouchDB; most of the views here are
shared by a wider audience, but I want to be clear this is not a formal proposal
from Cloudant but rather the musings of a long standing Cloudant and CouchDB
member with plenty of strong opinions.

The first draft of this document was done as an internal writeup with the
intention of then creating a modified version that defines a set of core
"stakeholders" such that the various tasks defined demonstrate value to the core
stakeholders, but in particular, that all of this functionality is defined and
executed in a way to bring clear value to the Apache CouchDB project and Apache
CouchDB community. I strongly believe the majority of this functionality should
be in the core CouchDB database itself and I want to ensure that it is done in a
way to be clearly beneficial to _all_ stakeholders involved. I started adding in
the discussion of target audience and stakeholders, but the breakout details for
each of the discussion points below is incomplete.

Forgive the lack of polish in this writeup, it's still a draft, but I've already
sat on this longer than I wanted, and with the upcoming CouchDB developer
meeting tomorrow, I felt it was better to get it out now in a rough form than
worry about a completed writeup. We can always rip various pieces out and create
more specific and complete writeups. I've got some additional writeups and ideas
to get out to the mailing list as well, but this is a good starting point.

A quick note on naming, "Dbcore" is the name of the version of CouchDB that
Cloudant runs in production. Many years ago this was an actual fork, but since
the 2.x merge we're essentially running CouchDB upstream with additional
libraries added on for Cloudant specific functionality, like IAM auth and what
not. The majority of the fun stuff was open sourced and incorporated into the
3.x codebase. This document switches between Dbcore and CouchDB in various
areas, but again, this is more of a draft than a finished document.

Thanks for your time and happy hacking!


-Russell "Chewbranca" Branca


==


# Chewbranca's Immediate Vision for Cloudant Dbcore and Apache CouchDB


Welcome to my vision of where to go with development and functionality of
Cloudant's Dbcore and Apache CouchDB, for the immediate future in 2022/2023.
This is an opinionated list of the things that I think are critically important
and also readily achievable. We may or may not get to the things in this list,
but here it is.

There's much to do, much that could be done, and endless possibilities. This
vision is targeted at the here and now, focused on immediately prevalent issues,
constraints, and challenges, skewed towards heavy hitters in terms of
time/resource investment to dividends paid. Not everything here is easy and
simple, but everything here is cherry picked because of its significant impact,
and also selected because it *needs* to be here.

This vision is also not exhaustive, to neither Cloudant nor CouchDB. I make no
claims that there isn't other important things to do, rather that the following
list of things *is* of substantial importance and provides a path to paradigm
shift the current state of Cloudant and CouchDB.


## Target Audience and Stakeholders

I want this writeup to target a few different core groups of people and
stakeholders to help understand motivations and help define scope. In
particular, I'm targeting three core groups of people:

  * CouchDB users/operators
  * Cloudant customers
  * Cloudant operators

As it stands today, these groups have very distinct interactions with the
database, and while there will also be distinctions between users of the system
and operators of the system, I think there's far more overlap between the three
groups than is currently represented. In particular, all three of these groups
need visibility into what the system is doing, what are the most active
databases and documents, and what impact requests have on the system. Both users
and operators need to have this visibility, and the more we can do to make it
easier for everyone to see into the activity of the system, the better.

The distinciton between Cloudant operators and CouchDB operators is considerably
different as the tooling is fundamentally different, or rather heavily lacking
in CouchDB and attached at the periphery in Cloudant; which is not ideal nor is
sustainable in my opinion.

I think we 

Re: [VOTE] Set a finite default for max_attachment_size

[Russell Branca]

A belated +1.

Infinite attachment size is obviously an oversight, and practically
speaking, I think gigabyte sized attachments are still way too big.

I would love to see a solid binary attachment system with CouchDB, but
neither couch_file nor FDB are likely to be it.


-Russell

On Mon, Feb 1, 2021 at 11:30 AM Joan Touzet  wrote:

> HI Donat,
>
> Point of order - when we do 72-hour votes, it's best to not count
> weekends in that 72-hours. So, since you started on the 28th at 05:00
> UTC, I would have continued the vote until Feb 2 at 05:00 UTC.
>
> That said I am +1 on this too, long overdue.
>
> As to Eric's point, all we need to do is document that the setting can
> be lifted and that we've tested "up to XGB" attachments as working in 3.x.
>
> 4.x change notes will show that the limit is reduced, and my opinion is
> that concomitant with a 4.0 release there will be a new 3.x release that
> lowers the defaults to match 4.x. (This may also need the changes
> mentioned in Robert's other thread that are necessary to allow 3.x <->
> 4.x replication, that's still unclear.) For now, we don't need to reduce
> down to 10MB or 5MB or whatever it is.
>
> -Joan
>
> On 01/02/2021 14:06, Bessenyei Balázs Donát wrote:
> > This vote is now closed as there were three +1s, one +0 and no -1s and
> > the 72 hours is up. I'll merge the PR.
> >
> > Thanks to all who voted!
> >
> >
> > Donat
> >
> >
> > On Mon, Feb 1, 2021 at 7:52 PM Eric Avdey  wrote:
> >>
> >> Ok, fair enough, +0 from me with a note that I'd still prefer to see
> this limit aligned with 4.x limits, so users wouldn't have to adjust to
> this change twice.
> >>
> >>
> >> Eric
> >>
> >>> On Feb 1, 2021, at 14:47, Nick Vatamaniuc  wrote:
> >>>
> >>> I am +1 to lowering as it's better than infinity.
> >>>
> >>> But I also see Eric's point. I was surprised a while back just like
> >>> Eric that I could successfully upload >1GB-sized files.  So why not
> >>> 0.5GB or 2GB? I am thinking 2GB was (is?) a common limit on some OSes
> >>> and file systems (FAT32) since they use ints for file size and
> >>> offsets. Since our attachment won't be saved as is in the file systems
> >>> inside a .couch file 2GB may be too high, so 1GB as a limit makes
> >>> sense to me.
> >>>
> >>> -Nick
> >>>
> >>> On Mon, Feb 1, 2021 at 1:25 PM Paul Davis 
> wrote:
> 
>  +1
> 
>  Default unlimited seems like an oversight regardless of what we
> change it to.
> 
>  On Mon, Feb 1, 2021 at 11:59 AM Eric Avdey  wrote:
> >
> > Maybe I didn't express myself clear enough. Setting some finit
> default is not a purpose, it's what you are doing and I'm asking what the
> reason for this change. In other words I'm not asking what are you doing,
> I'm asking why are you doing this.
> >
> > Introducing a new limit will be a breaking change to anoyone who
> uploads attachments larger than that limit, obviously, so "assumed 1G is
> large enough" sounds really arbitrary to me without any factual support for
> that assumption.
> >
> >
> > Eric
> >
> >
> >> On Feb 1, 2021, at 13:15, Bessenyei Balázs Donát 
> wrote:
> >>
> >> The purpose of this vote / PR is to set _some_ finite default. I
> went
> >> with 1G as I assumed that would not break anyone's production
> system.
> >> I'd support decreasing that limit over time.
> >>
> >> The vote has been open for 72 hours now, but I believe it still
> needs
> >> two more +1s to pass.
> >>
> >>
> >> Donat
> >>
> >> On Thu, Jan 28, 2021 at 10:44 PM Eric Avdey  wrote:
> >>>
> >>> This got me curious and I tried to upload Ubuntu image as an
> attachment. Interestingly CouchDB 3.x accepted first 1.4G of 2.8G file and
> then returned proper 201 response with a new doc revision, which I certanly
> didn't expect. Should say, that 1.4G seems suspiciously similar to a normal
> memory limit for a 32 bit process.
> >>>
> >>> Putting this aside, I agree that uploading large attachments is an
> anti-pattern and 1G seems excessive, hence my question. I'd expect this
> number to be based on something and correlating it with a  technical limit
> in 4.x makes a lot of sense to me.
> >>>
> >>>
> >>> Eric
> >>>
> >>>
>  On Jan 28, 2021, at 16:02, Robert Newson 
> wrote:
> 
>  Hi,
> 
>  I think a gigabyte is _very_ generous given our experience of
> this feature in practice.
> 
>  In 4.x attachment size will necessarily be much more restrictive,
> so it seems prudent to move toward that limit.
> 
>  I don’t think many folks (hopefully no one!) is routinely
> inserting attachments over 1 gib today, I’d be fairly surprised if it even
> works.
> 
>  B.
> 
> > On 28 Jan 2021, at 19:42, Eric Avdey  wrote:
> >
> > There is no justification neither here or on the PR for this
> change, i.e. why this is done. Original infinity 

Re: [DISCUSS] Removing erlang 19 support

[Russell Branca]

Oh that's fantastic, Joan! Glad to hear it. I'm excited to clear out some
of the older Erlang versions, and I would love to see CouchDB and
IBM/Cloudant stay more uptodate on Erlang releases. Thanks for all
the work on the packaging front, that's awesome to see!


-Russell

On Mon, Jan 25, 2021 at 5:12 PM Joan Touzet  wrote:

> Hey Russell, have no fear. I'm happy to say your 2014 essay is no longer
> an issue :)
>
> We've been shipping our couchdb binaries with Erlang Solutions'
> pre-built Erlang for a very long time now, at least since 2.1.0 and
> possibly since 2.0.0 released. When they don't provide it, we build
> using kerl. Whatever version of Erlang comes with the OS doesn't matter
> at all. In fact, we ship the exact same version of Erlang across every
> OS/arch we support today.
>
> The only limitation is whether or not the specific version of Erlang
> will *compile* on that specific OS/arch combo with the features we need
> - this usually comes down to the version of openssl shipped by the
> distro being compatible with Erlang's ssl module. There was a hiccup in
> the transition fom 0.9.8 to 1.0.x, and again from 1.0.x to 1.1.x, but
> that's all well in the past now.
>
> Cheers,
> Joan "DRY" Touzet
>
> On 2021-01-25 3:40 p.m., Russell Branca wrote:
> > I'm also +1 to removing Erlang 19.
> >
> > I wanted to reiterate what Newson said about Erlang Solutions providing
> > Erlang packaging, and I think we should more strongly lean on options
> like
> > this rather than being dependent on the OS distros Erlang versions. Many
> > years ago I wrote about the nuances with Erlang versions and CouchDB on
> the
> > R14/R15/R16 lines [1], and it's worth noting that multiple Debian
> versions
> > shipped versions of Erlang that were fundamentally broken for use with
> > CouchDB. I think we even blocked a number of those versions in the
> > rebar.config allowed versions.
> >
> > I don't know how much of an issue that is today, but there's certainly
> > precedent for distro shipped Erlang versions not being an option.
> >
> >
> > -Russell
> >
> >
> >
> > [1]
> >
> https://chewbranca.com/doc/trunk/archive/github/2014-05-07-on-the-viability-of-erlang-releases-and-couchdb.md
> >
> >
> >
> > On Mon, Jan 25, 2021 at 5:07 AM Bessenyei Balázs Donát <
> bes...@apache.org>
> > wrote:
> >
> >> Thank you all for the input!
> >>
> >> I'll remove erlang 19 for `couchdb-config` and we'll be able to refer
> >> to this thread when we have to remove it anywhere else.
> >>
> >>
> >> Donat
> >>
> >> On Sat, Jan 23, 2021 at 10:09 PM Adam Kocoloski 
> >> wrote:
> >>>
> >>> Ah, good research there Joan. +1 to Donat’s suggestion to drop support
> >> for 19 from me.
> >>>
> >>> Adam
> >>>
> >>>> On Jan 22, 2021, at 4:49 PM, Joan Touzet  wrote:
> >>>>
> >>>> On 2021-01-22 4:37 p.m., Robert Newson wrote:
> >>>>> Iteresting. I’m actually surprised at the inversion here (that
> >> CouchDB is dependent on  IBM to confirm CouchDB’s stability). I’ve
> always
> >> agonised over even the perception that IBM/Cloudant is calling the
> shots. I
> >> appreciate the reassurance that running at scale provides, of course, I
> >> just don’t think it can be our official position.
> >>>>
> >>>> It's a tough one. I was pretty aggressive on CouchDB running the very
> >> latest until the scheduler collapse problem surfaced. After that, there
> was
> >> another problem (I can't recall) that was pretty serious too. I took a
> >> wait-and-see attitude at that point, and after I didn't see IBM move
> >> forward to a newer release, didn't move forward ourselves. Looks like we
> >> ended up in deadlock!
> >>>>
> >>>> However! See your own comments on this:
> >>>>
> >>>> https://github.com/apache/couchdb/issues/3115#issuecomment-729031967
> >>>>
> >>>> I knew there was something at the back of my head on this. Guess we're
> >> both getting old ;)
> >>>>
> >>>>> On the core point of the thread, it seems there’s no barrier to
> >> dropping Erlang 19 support, so I think we can go to a VOTE thread,
> perhaps
> >> best to wait till Monday for others to chime in on this discussion
> though.
> >>>>
> >>>> More important is that we already committed changes on the main repo
> >> re: 

Re: [PROPOSAL] Archiving git branches

[Russell Branca]

A belated +1 to archiving. Thanks Joan, and also thanks for keeping the
remotes/origin/ioq-per-shard-or-user branch.


-Russell

On Wed, Oct 21, 2020 at 12:27 PM Joan Touzet  wrote:

> Hah!
>
> OK, it's all done. At present I cannot remove any of the #.#.# branches
> as Infra has protected them. (I may or may not bother to open a ticket
> on this.)
>
> Here's the remaining branch list:
>
>   remotes/origin/1.3.x
>   remotes/origin/1.4.x
>   remotes/origin/1.5.x
>   remotes/origin/1.6.x
>   remotes/origin/1.x.x
>   remotes/origin/1278-add-clustered-db-info
>   remotes/origin/2.0.x
>   remotes/origin/2.1.x
>   remotes/origin/2.3.x
>   remotes/origin/2493-remove-auth-cache
>   remotes/origin/3.0.x
>   remotes/origin/3.x
>   remotes/origin/HEAD -> origin/main
>   remotes/origin/access
>   remotes/origin/bump-ibrowse
>   remotes/origin/feat/access-3.x
>   remotes/origin/feat/access-master-clean
>   remotes/origin/feat/add-same-site-secure/master
>   remotes/origin/ioq-per-shard-or-user
>   remotes/origin/main
>   remotes/origin/master
>   remotes/origin/prototype/fdb-layer-db-version-as-vstamps
>   remotes/origin/re-enable-most-elixir-tests
>   remotes/origin/record_last_compaction
>   remotes/origin/smoosh-update-operator-guide
>   remotes/origin/update-fauxton-1.2.6
>
> That's about 6x shorter. Phew!
>
> Remember, if you are trying to recover a branch that is now gone, just:
>
>   git checkout -b  archive/
>
> -Joan
>
> On 21/10/2020 14:59, Jan Lehnardt wrote:
> > *makes chain saw noises*
> >
> > (trimming branches, get it?)
> >
> > Thanks Joan!
> >
> > Best
> > Jan
> > —
> >
> >> On 21. Oct 2020, at 20:23, Joan Touzet  wrote:
> >>
> >> I am starting the work now. As there was no response, I'm going to only
> >> keep these branches:
> >>
> >> 2.3.x
> >> 3.x
> >> main
> >> master (with a README saying you're in the wrong place)
> >>
> >> plus these PR branches:
> >>
> >> bump-ibrowse (#3208)
> >> smoosh-update-operator-guide (#3184)
> >> re-enable-most-elixir-tests (#3175)
> >> feat/add-same-site-secure/master (#3131)
> >> feat/access-master-clean (#3038)
> >> prototype/fdb-layer-db-version-as-vstamps (#2952)
> >> feat/access-3.x (#2943)
> >> ioq-per-shard-or-user (#1998)
> >> 1278-add-clustered-db-info (#1443)
> >> record_last_compaction (#1272)
> >>
> >> Any of the above that were targeted to prototype/fdb-layer or master
> >> have been re-targeted to main (except a couple clustering-specific ones
> >> which were set to 3.x). Please check if this is correct for your PRs.
> >>
> >> -Joan "clean all the things?" Touzet
> >>
> >> On 14/10/2020 13:19, Joan Touzet wrote:
> >>> A reminder about this: I intend to start this work tomorrow. If you
> have
> >>> any PRs or branches you want left alone, speak now.
> >>>
> >>> Based on feedback I've received, it sounds like the prototype/fdb-*
> >>> branches are now done? If this is **NOT** the case, speak up.
> >>>
> >>> -Joan
> >>>
> >>> On 07/10/2020 18:10, Joan Touzet wrote:
>  Hi there,
> 
>  I'd like to clean up our branches in git on the main couchdb repo.
> This
>  would involve deleting some of our obsolete branches, after tagging
> the
>  final revision on each branch. This way, we retain the history but the
>  branch no longer appears in the dropdown on GitHub, or in git branch
>  listings at the cli.
> 
>  Example process:
> 
>  git tag archive/1.3.x 1.3.x
>  git branch -d 1.3.x
>  git push origin :1.3.x
>  git push --tags
> 
>  If we ever needed the branch back, we just:
> 
>  git checkout -b 1.3.x archive/1.3.x
> 
>  I would propose to do this for all branches except:
> 
>  main
>  master (for now)
>  2.3.x
>  3.x
>  prototype/fdb-layer
> 
>  ...plus any branches that have been touched in the past 90 days, that
>  still have open PRs, or that someone specifically asks me to retain in
>  this thread.
> 
>  I'd also like to do this on couchdb-documentation and couchdb-fauxton.
> 
>  I would propose to do this about 1 week from now, let's say on October
>  15th.
> 
>  Thoughts?
> 
>  -Joan "fall cleaning" Touzet
> >
>

Re: [ANNOUNCE] Bessenyei Balázs Donát elected as CouchDB committer

[Russell Branca]

Congrats Donat! Welcome aboard!


-Russell

On Fri, Jan 15, 2021 at 10:17 AM Alessio 'Blaster' Biancalana <
dottorblas...@apache.org> wrote:

> Congratulations and welcome!
>
> Alessio
>
> On Fri, Jan 15, 2021 at 4:40 PM Dave Cottlehuber 
> wrote:
>
> > > > On 14. Jan 2021, at 21:48, Robert Newson  wrote:
> > > >
> > > > Dear community,
> > > >
> > > > I am pleased to announce that the CouchDB Project Management
> Committee
> > has elected Bessenyei Balázs Donát as a CouchDB committer.
> > > >
> > > > Apache ID: bessbd
> > > >
> > > > Committers are given a binding vote in certain project decisions, as
> > well as write access to public project infrastructure.
> > > >
> > > > This election was made in recognition of Donát's commitment to the
> > project. We mean this in the sense of being loyal to the project and its
> > interests.
> > > >
> > > > Please join me in extending a warm welcome to Donát!
> > > >
> > > > On behalf of the CouchDB PMC,
> > > >
> > > > Robert Newson
> >
> > Awesome! welcome Donát!
> >
> > A+
> > Dave
> >
>

Re: [DISCUSS] Removing erlang 19 support

[Russell Branca]

I'm also +1 to removing Erlang 19.

I wanted to reiterate what Newson said about Erlang Solutions providing
Erlang packaging, and I think we should more strongly lean on options like
this rather than being dependent on the OS distros Erlang versions. Many
years ago I wrote about the nuances with Erlang versions and CouchDB on the
R14/R15/R16 lines [1], and it's worth noting that multiple Debian versions
shipped versions of Erlang that were fundamentally broken for use with
CouchDB. I think we even blocked a number of those versions in the
rebar.config allowed versions.

I don't know how much of an issue that is today, but there's certainly
precedent for distro shipped Erlang versions not being an option.


-Russell



[1]
https://chewbranca.com/doc/trunk/archive/github/2014-05-07-on-the-viability-of-erlang-releases-and-couchdb.md



On Mon, Jan 25, 2021 at 5:07 AM Bessenyei Balázs Donát 
wrote:

> Thank you all for the input!
>
> I'll remove erlang 19 for `couchdb-config` and we'll be able to refer
> to this thread when we have to remove it anywhere else.
>
>
> Donat
>
> On Sat, Jan 23, 2021 at 10:09 PM Adam Kocoloski 
> wrote:
> >
> > Ah, good research there Joan. +1 to Donat’s suggestion to drop support
> for 19 from me.
> >
> > Adam
> >
> > > On Jan 22, 2021, at 4:49 PM, Joan Touzet  wrote:
> > >
> > > On 2021-01-22 4:37 p.m., Robert Newson wrote:
> > >> Iteresting. I’m actually surprised at the inversion here (that
> CouchDB is dependent on  IBM to confirm CouchDB’s stability). I’ve always
> agonised over even the perception that IBM/Cloudant is calling the shots. I
> appreciate the reassurance that running at scale provides, of course, I
> just don’t think it can be our official position.
> > >
> > > It's a tough one. I was pretty aggressive on CouchDB running the very
> latest until the scheduler collapse problem surfaced. After that, there was
> another problem (I can't recall) that was pretty serious too. I took a
> wait-and-see attitude at that point, and after I didn't see IBM move
> forward to a newer release, didn't move forward ourselves. Looks like we
> ended up in deadlock!
> > >
> > > However! See your own comments on this:
> > >
> > > https://github.com/apache/couchdb/issues/3115#issuecomment-729031967
> > >
> > > I knew there was something at the back of my head on this. Guess we're
> both getting old ;)
> > >
> > >> On the core point of the thread, it seems there’s no barrier to
> dropping Erlang 19 support, so I think we can go to a VOTE thread, perhaps
> best to wait till Monday for others to chime in on this discussion though.
> > >
> > > More important is that we already committed changes on the main repo
> re: Erlang 19 about 14 months ago by Paul:
> > >
> > >
> https://github.com/apache/couchdb/commit/3594f2f1fc16903c1c383ebaf205d31c9c17fb3a
> > >
> > > I think that makes Donat's request pretty straightforward.
> > >
> > >> I also think that IBM Cloudant’s chosen Erlang release is in part
> influenced by CouchDB’s lack of support for later versions and requirement
> of compatible with older releases, which now appears illusory.
> > >
> > > If we're ready to move to 21 or 22 as a default, we're ready. Let's
> hope the serious issues in 21/22 are at least mitigated. I'm happy to make
> the 3.3 release (or whatever is next) use the very latest version of 21 or
> 22 from GitHub, subject to community recommendations and encouragement. 23
> is still a WIP: https://github.com/apache/couchdb/issues/3115
> > >
> > > -Jon
> > >
> > >> B.
> > >>> On 22 Jan 2021, at 21:19, Joan Touzet  wrote:
> > >>>
> > >>> On 22/01/2021 15:48, Robert Newson wrote:
> >  I’m +1 on dropping Erlang 19 support. Erlang is now on major
> release 23.
> > >>>
> > >>> No problem here.
> > >>>
> >  I’d further advocate a general policy of supporting only the most
> recent 2 or 3 major releases of Erlang/OTP.
> > 
> >  The main (I think only?) reason to keep compatibility so far back
> is because of the versions supported by some OS’es. I don’t feel that is a
> strong reason given the couchdb install, in common with Erlang-based
> projects, is self-contained. The existence of Erlang Solutions packages for
> all common platforms is also a factor.
> > >>>
> > >>> That hasn't been the case for at least 2 years, if not longer.
> > >>>
> > >>> As the release engineer, I've been focused on stability for everyone.
> > >>> This is largely driven by knowing that IBM/Cloudant largely run
> CouchDB
> > >>> on 20.x at scale. Standing on the shoulders of giants, our releases
> run
> > >>> the latest 20.x release at the time of binary generation.
> > >>>
> > >>> A few times recently issues cropped up in 21 and 22 that we didn't
> > >>> encounter in our user base because, at scale, we are deployed on
> > >>> 20.3.8.something. Some of these issues were non-trivial. (I'm off
> today,
> > >>> so I don't have the time to dig into the specifics until Monday.)
> > >>>
> > >>> So my $0.02 is that: if IBM/Cloudant is 

Re: [VOTE] couchdb 4.0 transaction semantics

[Russell Branca]

+1


-Russell


On Fri, Jan 8, 2021 at 11:54 AM Joan Touzet  wrote:

> +1.
>
> This is for now I presume, as I thought that there was feeling about
> relaxing this restriction somewhat for the 5.0 timeframe? Memory's dim.
>
> -Joan
>
> On 07/01/2021 06:00, Robert Newson wrote:
> > Hi,
> >
> > Following on from the discussion at
> https://lists.apache.org/thread.html/rac6c90c4ae03dc055c7e8be6eca1c1e173cf2f98d2afe6d018e62d29%40%3Cdev.couchdb.apache.org%3E
> <
> https://lists.apache.org/thread.html/rac6c90c4ae03dc055c7e8be6eca1c1e173cf2f98d2afe6d018e62d29@%3Cdev.couchdb.apache.org%3E
> >
> >
> > The proposal is;
> >
> > "With the exception of the changes endpoint when in feed=continuous
> mode, that all data-bearing responses from CouchDB are constructed from a
> single, immutable snapshot of the database at the time of the request.”
> >
> > Paul Davis summarised the discussion in four bullet points, reiterated
> here for context;
> >
> > 1. A single CouchDB API call should map to a single FDB transaction
> > 2. We absolutely do not want to return a valid JSON response to any
> > streaming API that hit a transaction boundary (because data
> > loss/corruption)
> > 3. We're willing to change the API requirements so that 2 is not an
> issue.
> > 4. None of this applies to continuous changes since that API call was
> > never a single snapshot.
> >
> >
> > Please vote accordingly, we’ll run this as lazy consensus per the bylaws
> (https://couchdb.apache.org/bylaws.html#lazy <
> https://couchdb.apache.org/bylaws.html#lazy>)
> >
> > B.
> >
> >
>

Re: CouchDB running on the Raspberry Pi?

[Russell Branca]

Fwiw I got CouchDB 2.x running on a Raspian Raspberry Pi 3.b recently
without much fuss. Pretty sure I compiled Erlang from source.


-Russell

On Fri, Jan 19, 2018 at 4:12 AM Eric Clack  wrote:

> Hi Michael, Ben and Johs,
> Thanks for your responses.
>
> I've found for my (admittedly small) projects Couch works pretty well on
> the Pi (version 2). It's a great solution to the problem of where to store
> metadata, and how to query it easily. The only performance problem I saw
> was when reducing and grouping queries, e.g. here I used a memo to cache
> results:
>
> https://github.com/ericclack/clojure-photo-bank/blob/master/src/clj/clojure_photo_bank/models/db.clj#L45
> ...running the query took about 5 seconds for around 4000 records. That's
> not great, but for everything else Couch was easily fast enough.
>
> I'm going to explore Erlang packages for Raspbian and then look at
> compiling a few different versions of CouchDB to see if I can at least get
> some good bug reports to the right team.
>
> Many thanks,
> -Eric.
>
>
>
> On 19 January 2018 at 08:11, Johs Ensby  wrote:
>
> > Hi Eric,
> >
> > I think it would be great if you found a way to run couch on Raspberry
> Pi,
> > preferably Ubuntu Mate.
> > Performance will surely not be great, but more important I would think it
> > could be a great demonstration of how CouchDB could be a common platform
> > from big clusters in the cloud to your very private system.
> >
> > johs:)
> >
> >
> > > On 16 Jan 2018, at 23:28, benjamin.bast...@gmail.com wrote:
> > >
> > > I did this back quite a few years ago, and from what I remember IO
> > > performance was pretty dreadful on an SD card. I'm not sure what the
> > status
> > > of COUCHDB-3287 is, but maybe a different storage engine would offer
> > better
> > > performance?
> > >
> > > Ben
> > >
> > > On Tue, Jan 16, 2018 at 2:08 PM, Michael Fair  >
> > > wrote:
> > >
> > >> This is similar to running Couch on mobile phones.
> > >> Perhaps an alternative is starting with another "Couch Compatible"
> > database
> > >> that's lighter weight?
> > >>
> > >> Perhaps PouchDB running on Node.js comes to mind.
> > >> This implements the replication/sync protocol between Couch compatible
> > >> databases, without being an erlang based, sharded, distributed
> backend.
> > >>
> > >>
> > >>
> > >> On Tue, Jan 16, 2018 at 2:44 AM, Eric Clack  >
> > >> wrote:
> > >>
> > >>> Hello CouchDB devs,
> > >>>
> > >>> Is there any interest in getting CouchDB running on the Raspberry Pi
> > (an
> > >>> AMD platform), running Raspbian Stretch?
> > >>>
> > >>> See my post here:
> > >>> https://github.com/apache/couchdb/issues/1103
> > >>>
> > >>> As I said in the post, I have time to contribute to the work and am
> > >>> interested in learning more about CouchDB, Erlang, etc.
> > >>>
> > >>> Right now it would be useful for me to gauge interest as I need to
> know
> > >>> whether I should investigate alternative databases for my Pi
> projects.
> > >>>
> > >>> Many thanks,
> > >>> -Eric.
> > >>>
> > >>> --
> > >>> Eric Clack
> > >>> e...@bn7.net
> > >>> East Sussex, England.
> > >>>
> > >>
> >
> >
>
>
> --
> Eric Clack
> e...@bn7.net
> East Sussex, England.
>

Re: [RFC] On the Testing of CouchDB

[Russell Branca]

. q=4 takes ~20s, q=2 ~10s and q=1 ~5s. I’m not
> suggesting we set q=1 for all tests since q>1 is a behaviour we would
> want to test as well, but maybe we can set q=2 when running the test
> suite(s) for the time being. Shaving 25s off of a single test will get
> us a long way with all tests ported. What do others think?
>

I'm not sure what's going on here, although Paul ran into similar
situations where tests would run on my machine in under a second but take
significantly longer on his machine. I think we should investigate this a
bit more before taking any major action, but for a lot of tests running
with a lower Q value would probably be fine. I think it's important we run
with Q >= 2 so we at least run through the standard code paths of having to
merge over multiple shard ranges for aggregate operations. Although that
said, it could also be worthwhile to run tests with Q=1 to exercise the non
merge code paths as well.


>
> @Nick: you introduced the -C / --config-overrides option to dev/run,
> but I could never figure out how to apply it. That would be the easiest
> place to make the cluster config change for the Elixir tests.
>
> `-c q=2` makes the nodes fail to start and I’m not sure how in this case,
> the `[cluster]` section is meant.
>
> * * *
>
> Russell, Paul: do you think it is worth reaching out to the Elixir
> community and ask if they are interested in helping out a little? If
> you think its too early, we can wait with this.
>

Yeah I think that's a great idea and could be an excellent opportunity to
get more folks involved in things. I do think we should probably figure out
our branch/dev strategy prior to doing that, as me cherry-picking commits
will eventually become onerous given sufficient branches.


> * * *
>
> Thanks again!
>
> Best
> Jan
> --
>
>
No problem, really happy to see folks excited about this new engine!


-Russell


>
>
> > On 15. Dec 2017, at 02:03, Russell Branca <chewbra...@apache.org> wrote:
> >
> > Howdy folks!
> >
> > The testing of CouchDB is something that has seen focus and improvements
> > for the last several years, for instance migrating the etap suite to
> eunit,
> > and updating the JS suite to run against clusters in 2.x. There's still
> > improvements to be made, and that was one of the topics of the CouchDB
> dev
> > summit early in the year [1].
> >
> > Before we go further, I want to clarify some nomenclature. I'm by no
> means
> > going to try and define unit testing vs integration testing vs quantum
> > phase shift testing, but instead I want to focus on the distinction of
> > where the testing takes place. Fundamentally, we have two places we test
> > CouchDB: 1) at the Erlang VM level where we conduct assertions against
> > module functions or process states; 2) at the HTTP level where we test
> the
> > behavior of CouchDB at the user level API. This post focuses entirely on
> > the latter; that's not to say the former doesn't also merit attention,
> just
> > that the two are different enough that we can focus on them in isolation.
> >
> > So with that, let's chat about the current HTTP test suite in CouchDB.
> This
> > is the "JS suite" I referred to above, which is a custom built test suite
> > written in Javascript and executed in the aging SpiderMonkey. The JS
> suite
> > has put in work for years, but it's showing it's age, and is a bit
> awkward
> > to work with and improve. However, I think the biggest issue with the JS
> > suite is that it's utilized far less than it should be, and folks seem to
> > avoid extending it or adding additional tests to it. There's been
> > discussion for years about replacing said suite, but the discussions
> > invariably got blocked on the bike shed of whether to rewrite the suite
> in
> > Javascript or Python. This thread provides a third option, with code!
> >
> > I started hacking on a replacement for the JS suite, this time written in
> > Elixir. Overall I'm quite impressed with how it's come along, and have
> some
> > good examples to show. This is basically an Elixir app that has an HTTP
> > client and then runs a series of tests that conduct tests against the
> > CouchDB HTTP API and make assertions therein.
> >
> > You can find the current code in [2], and a comparison of the changes in
> > [3]. The core HTTP client is only a handful of lines of codes and works
> > quite well [4]. The utility functions used across all tests are located
> in
> > [5], and the tests themselves are in [6]. The existing test modules have
> a
> > 1:1 correspondence with the associated JS suite test modules, and in
> >

Re: [RFC] On the Testing of CouchDB

[Russell Branca]

I just updated the README in the Elixir suite to have a todo list of tests
to port, in case folks are looking for tests:
https://github.com/apache/couchdb/blob/elixir-suite/test/elixir/README.md


-Russell
On Fri, Dec 15, 2017 at 9:46 AM Paul Davis <paul.joseph.da...@gmail.com>
wrote:

> For `make check` it should be fairly straightforward to map the
> current approach to it. I could probably knock that out fairly quickly
> if you want me to give it a whirl.
>
> On Fri, Dec 15, 2017 at 11:42 AM, Russell Branca <chewbra...@apache.org>
> wrote:
> > Yeah just to reiterate what Paul said, the Elixir dev experience is
> really
> > nice and easy to get rolling with. I had no prior actual experience with
> > Elixir and I was able to get things rolling in a few hours.
> >
> > RE Ben's question about diving in: please do! Just grab one of the
> unported
> > js suites and goto town. I've just been cherry-pick'ing things out of
> > Paul's branch and we can continue to do the same until we get this more
> > locked down. My goal with the porting is to keep chugging along and just
> > get it knocked out, as I really don't think it will be overly onerous to
> do
> > so. And if anyone else wants to jump in, there's still a fair number of
> > tests to port, just take your pick.
> >
> > One other thing that needs work is figuring out how to hook all this into
> > "make check" and what not. I've mostly ignored that as this just points
> at
> > a CouchDB instance and can be run directly, but we'll need to sort that
> out
> > at some point.
> >
> >
> > -Russell
> >
> > On Fri, Dec 15, 2017 at 9:03 AM Paul Davis <paul.joseph.da...@gmail.com>
> > wrote:
> >
> >> Hello everybody!
> >>
> >> I figured I should probably go ahead and chime in seeing as I've also
> >> been playing around porting some of the tests in my free time between
> >> ops shifts the last couple weeks.
> >>
> >> My first impression was that it was ridiculously easy to get involved.
> >> On OS X at least, `brew install elixir` was enough to get a working
> >> elixir installed (however, if you use kerl or erln8 you'll want have
> >> to build an Erlang 20.x VM to use the brew package). I went from not
> >> having Elixir installed to a full port of uuids.js with the config tag
> >> logic written in about two hours one night. So far the Elixir docs and
> >> seem very well written and put together. I'd say the worst part of
> >> Elixir so far is that knowing Erlang I find myself searching for "How
> >> do I do this Erlang thing in Elixir?" Which isn't as bad as it sounds.
> >> The Elixir libraries have certainly had a considerable amount of
> >> thought put into them to make them easy to use and remember. I find it
> >> to be a lot like my experience when learning Python in that I may have
> >> to Google once and then its muscle memory. As opposed to Erlang's
> >> library where I'm constantly reading the lists manpage to remember
> >> argument orderings and whether I want search or find versions etc.
> >>
> >> Which I guess is a long way of saying I'm rather liking the Elixir
> >> development experience so far.
> >>
> >> That said, I'm currently about half way through porting replication.js
> >> tests to Elixir. For the most part its fairly straightforward. My
> >> current approach as we've done for the other modules is to do a direct
> >> port. Once that's finished we'll want to break up that huge module
> >> into a series of modules that share a lot of the utility functions.
> >> One of the nice things about moving to Elixir is that its got a full
> >> on development story rather than our current couchjs approach that
> >> prevents sharing code easily between subsets of tests.
> >>
> >> For Ben's question on diving in, I'd do just that. I'd say leave a
> >> note here about which module(s)? you're going to port so that we're
> >> not duplicating efforts and then its basically just a matter of
> >> getting Elixir installed. For that, here's a quick rundown on how I
> >> got that working:
> >>
> >> $ brew update
> >> $ brew install elixir
> >> $ # wait for all the things...
> >> $ iex # which fails cause I have an Erlang VM older than 20.0 as a
> default
> >> $ erln8 --fetch
> >> $ erln8 --build --tag=OTP-20.1.6 --id=20.1.6
> >> $ # wait while erln8 does its thing
> >> $ git clone https://github.com/apache/couchdb
> >>

Re: [RFC] On the Testing of CouchDB

[Russell Branca]

 this was reading through their quick
> start/tutorial pages a couple of times to get a feeling for the syntax
> but hadn't actually even typed it into an editor till last week.
>
> And that's all I've got for now.
>
> On Thu, Dec 14, 2017 at 11:57 PM, Benjamin Anderson
> <banjie...@apache.org> wrote:
> > Slick! This seems like it's coming together really nicely. Can't argue
> > with commits like "Prefer ?w=3 over hacky sleeps"[1] in any case.
> >
> >> I hope others have similar opinions after diving in!
> >
> > How should one dive in? Are you looking for others to help out with
> > the ports, or just thinking aspirationally about future regular
> > contributions to the test suite?
> >
> > --
> > b
> >
> > [1]:
> https://github.com/apache/couchdb/commit/5bce2d98a298c25b77d8dcda19deeedb494cc289
> >
> > On Thu, Dec 14, 2017 at 5:03 PM, Russell Branca <chewbra...@apache.org>
> wrote:
> >> Howdy folks!
> >>
> >> The testing of CouchDB is something that has seen focus and improvements
> >> for the last several years, for instance migrating the etap suite to
> eunit,
> >> and updating the JS suite to run against clusters in 2.x. There's still
> >> improvements to be made, and that was one of the topics of the CouchDB
> dev
> >> summit early in the year [1].
> >>
> >> Before we go further, I want to clarify some nomenclature. I'm by no
> means
> >> going to try and define unit testing vs integration testing vs quantum
> >> phase shift testing, but instead I want to focus on the distinction of
> >> where the testing takes place. Fundamentally, we have two places we test
> >> CouchDB: 1) at the Erlang VM level where we conduct assertions against
> >> module functions or process states; 2) at the HTTP level where we test
> the
> >> behavior of CouchDB at the user level API. This post focuses entirely on
> >> the latter; that's not to say the former doesn't also merit attention,
> just
> >> that the two are different enough that we can focus on them in
> isolation.
> >>
> >> So with that, let's chat about the current HTTP test suite in CouchDB.
> This
> >> is the "JS suite" I referred to above, which is a custom built test
> suite
> >> written in Javascript and executed in the aging SpiderMonkey. The JS
> suite
> >> has put in work for years, but it's showing it's age, and is a bit
> awkward
> >> to work with and improve. However, I think the biggest issue with the JS
> >> suite is that it's utilized far less than it should be, and folks seem
> to
> >> avoid extending it or adding additional tests to it. There's been
> >> discussion for years about replacing said suite, but the discussions
> >> invariably got blocked on the bike shed of whether to rewrite the suite
> in
> >> Javascript or Python. This thread provides a third option, with code!
> >>
> >> I started hacking on a replacement for the JS suite, this time written
> in
> >> Elixir. Overall I'm quite impressed with how it's come along, and have
> some
> >> good examples to show. This is basically an Elixir app that has an HTTP
> >> client and then runs a series of tests that conduct tests against the
> >> CouchDB HTTP API and make assertions therein.
> >>
> >> You can find the current code in [2], and a comparison of the changes in
> >> [3]. The core HTTP client is only a handful of lines of codes and works
> >> quite well [4]. The utility functions used across all tests are located
> in
> >> [5], and the tests themselves are in [6]. The existing test modules
> have a
> >> 1:1 correspondence with the associated JS suite test modules, and in
> >> general are as direct of a port as possible.
> >>
> >> The test modules ported in their entirety or most of the way are:
> >>
> >>   * all_docs.js
> >>   * basics.js
> >>   * config.js
> >>   * reduce.js
> >>   * rewrite.js
> >>   * uuids.js
> >>   * view_collation.js
> >>
> >> Paul has dove in and is responsible for a few of those test modules and
> >> he's almost completed porting the replication.js suite as well. We
> started
> >> with the hard ones first, so for the most part the rest of the ports
> should
> >> be fairly smooth sailing.
> >>
> >> Here's an example of a very basic test:
> >>
> >> ```erlang
> >> defmodule WelcomeTest do
> >>   use CouchTestC

[RFC] On the Testing of CouchDB

[Russell Branca]

Howdy folks!

The testing of CouchDB is something that has seen focus and improvements
for the last several years, for instance migrating the etap suite to eunit,
and updating the JS suite to run against clusters in 2.x. There's still
improvements to be made, and that was one of the topics of the CouchDB dev
summit early in the year [1].

Before we go further, I want to clarify some nomenclature. I'm by no means
going to try and define unit testing vs integration testing vs quantum
phase shift testing, but instead I want to focus on the distinction of
where the testing takes place. Fundamentally, we have two places we test
CouchDB: 1) at the Erlang VM level where we conduct assertions against
module functions or process states; 2) at the HTTP level where we test the
behavior of CouchDB at the user level API. This post focuses entirely on
the latter; that's not to say the former doesn't also merit attention, just
that the two are different enough that we can focus on them in isolation.

So with that, let's chat about the current HTTP test suite in CouchDB. This
is the "JS suite" I referred to above, which is a custom built test suite
written in Javascript and executed in the aging SpiderMonkey. The JS suite
has put in work for years, but it's showing it's age, and is a bit awkward
to work with and improve. However, I think the biggest issue with the JS
suite is that it's utilized far less than it should be, and folks seem to
avoid extending it or adding additional tests to it. There's been
discussion for years about replacing said suite, but the discussions
invariably got blocked on the bike shed of whether to rewrite the suite in
Javascript or Python. This thread provides a third option, with code!

I started hacking on a replacement for the JS suite, this time written in
Elixir. Overall I'm quite impressed with how it's come along, and have some
good examples to show. This is basically an Elixir app that has an HTTP
client and then runs a series of tests that conduct tests against the
CouchDB HTTP API and make assertions therein.

You can find the current code in [2], and a comparison of the changes in
[3]. The core HTTP client is only a handful of lines of codes and works
quite well [4]. The utility functions used across all tests are located in
[5], and the tests themselves are in [6]. The existing test modules have a
1:1 correspondence with the associated JS suite test modules, and in
general are as direct of a port as possible.

The test modules ported in their entirety or most of the way are:

  * all_docs.js
  * basics.js
  * config.js
  * reduce.js
  * rewrite.js
  * uuids.js
  * view_collation.js

Paul has dove in and is responsible for a few of those test modules and
he's almost completed porting the replication.js suite as well. We started
with the hard ones first, so for the most part the rest of the ports should
be fairly smooth sailing.

Here's an example of a very basic test:

```erlang
defmodule WelcomeTest do
  use CouchTestCase

  test "Welcome endpoint" do
assert Couch.get("/").body["couchdb"] == "Welcome", "Should say welcome"
  end

end

```


As you can see, the `Couch` client is very simple HTTP client with
easy HTTP verb based methods. Let's look at a more complicated test
for asserting we can create documents in a database:


```erlang

  @tag :with_db
  test "Create a document and save it to the database", context do
resp = Couch.post("/#{context[:db_name]}", [body: %{:_id => "0",
:a => 1, :b => 1}])
assert resp.status_code == 201, "Should be 201 created"
assert resp.body["id"], "Id should be present"
assert resp.body["rev"], "Rev should be present"

resp2 = Couch.get("/#{context[:db_name]}/#{resp.body["id"]}")
assert resp2.body["_id"] == resp.body["id"], "Ids should match"
assert resp2.body["_rev"] == resp.body["rev"], "Revs should match"
  end

```


This is fairly straightforward code to POST a new doc, make assertions
on the response, and then fetch the doc to make sure everything
matches up. What I really wanted to highlight here is the `@tag
:with_db` decorator. We can easily add custom "tags" to the tests to
simplify setup and teardown. That `:with_db` tag does two things, it
dynamically generates a random database name, and then takes care of
setup/teardown for creating and deleting said database for that
particular test. This is really useful and has been very nice to work
with so far. We also have tag functionality in place for executing a
test with a particular set of config options:


```erlang

  @tag config: [
{"uuids", "algorithm", "utc_random"}
  ]
  test "utc_random uuids are roughly random" do
resp = Couch.get("/_uuids", query: %{:count => 1000})
assert resp.status_code == 200
uuids = resp.body["uuids"]

assert String.length(Enum.at(uuids, 1)) == 32

# Assert no collisions
assert length(Enum.uniq(uuids)) == length(uuids)

# Assert rough ordering of UUIDs
u1 = String.slice(Enum.at(uuids, 1), 

Re: [ANNOUNCE] Nick Vatamaniuc joins the PMC

[Russell Branca]

Huzzah, congrats Nick!


-Russell

On Mon, Nov 13, 2017 at 5:27 AM Robert Kowalski  wrote:

> Welcome Nick, congrats!
>
> On Mon, Nov 13, 2017 at 1:24 PM, Andy Wenk  wrote:
> > Hey Nick - welcome on bord ;-)
> >
> > All the best
> >
> > Andy
> >
> > --
> > Andy Wenk
> > Hamburg - Germany
> > RockIt!
> >
> > GPG public key:
> > http://pgp.mit.edu/pks/lookup?op=get=0x45D3565377F93D29
> >
> >
> >
> >> On 13. Nov 2017, at 08:22, Garren Smith  wrote:
> >>
> >> Welcome Nick. This is awesome.
> >>
> >> On Mon, Nov 13, 2017 at 7:30 AM, Nick Vatamaniuc 
> wrote:
> >>
> >>> Thanks everyone for the warm welcome. It's an honor to be a member of
> the
> >>> Apache CouchDB PMC.
> >>>
> >>> On Sun, Nov 12, 2017 at 8:29 AM, Robert Samuel Newson <
> rnew...@apache.org>
> >>> wrote:
> >>>
>  Congratulations Nick, well deserved, and welcome aboard!
> 
>  B.
> 
> > On 12 Nov 2017, at 03:39, Paul Davis 
>  wrote:
> >
> > Hooray and welcome, Nick!
> >
> >
> > On Sat, Nov 11, 2017 at 2:45 PM Joan Touzet 
> wrote:
> >
> >> Congratulations! Welcome, Nick!
> >>
> >> - Original Message -
> >> From: "Alexander Shorin" 
> >> To: priv...@couchdb.apache.org
> >> Sent: Saturday, 11 November, 2017 1:31:36 PM
> >> Subject: Re: [ANNOUNCE] Nick Vatamaniuc joins the PMC
> >>
> >> Hooray to Nick! And welcome!
> >> --
> >> ,,,^..^,,,
> >>
> >>
> >> On Sat, Nov 11, 2017 at 4:40 PM, Jan Lehnardt 
> wrote:
> >>> Forgot to CC private@
> >>>
>  Begin forwarded message:
> 
>  From: Jan Lehnardt 
>  Subject: [ANNOUNCE] Nick Vatamaniuc joins the PMC
>  Date: 11. November 2017 at 17:38:39 GMT+1
>  To: dev 
>  Reply-To: dev@couchdb.apache.org
> 
>  Dear community,
> 
>  I am delighted to announce that Nick Vatamaniuc joins the Apache
> >> CouchDB Project Management Committee today.
> 
>  Nick has made outstanding, sustained contributions to the project.
>  This
> >> appointment is an official acknowledgement of their position within
> >>> the
> >> community, and our trust in their ability to provide oversight for
> the
> >> project.
> 
>  Everybody, please join me in congratulating Nick!
> 
>  On behalf of the CouchDB PMC,
>  Jan
>  —
> 
> >>>
> >>
> 
> 
> >>>
> >
>

Re: [ANNOUNCE] Benjamin Anderson elected as CouchDB committer

[Russell Branca]

Welcome aboard Ben!


-Russell

On Fri, Nov 4, 2016 at 6:10 PM, Benjamin Anderson 
wrote:

> Thanks, everyone. Glad to be on board. :)
>
> --
> b
>
> On Wed, Nov 2, 2016 at 9:41 AM, Noah Slater  wrote:
> > Grats! Welcome!
> >
> > On Wed, 2 Nov 2016 at 05:23 Garren Smith  wrote:
> >
> >> Awesome. Welcome Ben
> >>
> >> On Tuesday, 01 November 2016, Johannes Jörg Schmidt <
> schm...@netzmerk.com>
> >> wrote:
> >>
> >> > Welcome Benjamin!
> >> >
> >> > Am 01.11.2016 9:54 nachm. schrieb "Michelle Phung" <
> >> > michelleph...@gmail.com 
> >> > >:
> >> >
> >> > > yay finally! :) & hi ben o/
> >> > >
> >> > > - michelle
> >> > >
> >> > > > On Nov 1, 2016, at 4:45 PM, Jan Lehnardt  >> > > wrote:
> >> > > >
> >> > > > Welcome Ben! :)
> >> > > >
> >> > > >
> >> > > >> On 1 Nov 2016, at 20:46, Robert Samuel Newson <
> rnew...@apache.org
> >> > >
> >> > > wrote:
> >> > > >>
> >> > > >> Dear community,
> >> > > >>
> >> > > >> I am pleased to announce that the CouchDB Project Management
> >> Committee
> >> > > has elected Benjamin Anderson as a CouchDB committer.
> >> > > >>
> >> > > >>   Apache ID: banjiewen
> >> > > >>
> >> > > >>   IRC nick: banjiewen
> >> > > >>
> >> > > >>   Twitter: banjiewen
> >> > > >>
> >> > > >> Committers are given a binding vote in certain project
> decisions, as
> >> > > well as write access to public project infrastructure.
> >> > > >>
> >> > > >> This election was made in recognition of Ben's commitment to the
> >> > > project. We mean this in the sense of being loyal to the project and
> >> its
> >> > > interests.
> >> > > >>
> >> > > >> Please join me in extending a warm welcome to Ben!
> >> > > >>
> >> > > >> On behalf of the CouchDB PMC,
> >> > > >>
> >> > > >> Robert Samuel Newson
> >> > > >>
> >> > > >
> >> > > > --
> >> > > > Professional Support for Apache CouchDB:
> >> > > > https://neighbourhood.ie/couchdb-support/
> >> > > >
> >> > >
> >> >
> >>
>

Re: Adding a node to cluster

[Russell Branca]

Good catch, it should have included the db ;-)


-Russell

On Tuesday, September 6, 2016, Joey Samonte 
wrote:

> In the example you gave for the replication, why is the target "
> test2.bar.com", and not "test2.bar.com/my_db"?
>
> > From: chewbra...@apache.org 
> > Date: Tue, 6 Sep 2016 19:34:43 -0700
> > Subject: Re: Adding a node to cluster
> > To: dev@couchdb.apache.org 
> >
> > CouchDB "replication" uses the HTTP API to replicate a database, whereas
> > the somewhat confusingly named "internal replication" is used within the
> > cluster itself as a way to synchronize state between the shard replicas.
> > The protocol for internal replication uses the Erlang distribution
> protocol
> > to communicate between nodes. So if your cluster is a connected set of
> > Erlang nodes then they will be able to synchronize database/shard state
> > amongst themselves.
> >
> >
> > As an example, consider a single node 1.x CouchDB server at
> > test1.foo.com:5984 where you want to replicate to a CouchDB 2.0 cluster
> at
> > test2.bar.com that has three db nodes and a load balancer, with the load
> > balancer actually having the test2.bar.com address. You communicate with
> > the cluster by way of the load balancer which will distribute requests to
> > all the nodes. The details of the cluster should be mostly transparent at
> > the HTTP layer, and so you can just do something like:
> >
> > curl -i -X POST test1.foo.com/_replicator -d '{"source":"
> test1.foo.com/my_db",
> > "target":"test2.bar.com"}'
> >
> > This will replicate "my_db" from test1 to test2 using the HTTP protocol
> > like you would expect from CouchDB. Within the test2 cluster the nodes
> will
> > communicate amongst themselves using internal replication (non HTTP) to
> > ensure the data is fully synchronized between the shard replicas.
> >
> > Hopefully that clarifies things a bit.
> >
> >
> > -Russell
> >
> >
> >
> > On Tuesday, September 6, 2016, Joey Samonte  >
> > wrote:
> >
> > > I think I understand it. Hoping to find more documentation on it.
> > >
> > > > From: wickedg...@gmail.com  
> > > > Date: Tue, 6 Sep 2016 15:55:00 -0700
> > > > Subject: Re: Adding a node to cluster
> > > > To: dev@couchdb.apache.org  
> > > >
> > > > Nodes in the cluster do not replicate to one another.
> > > >
> > > > Replication takes place between databases. A single node isn't a
> > > > database; a clustered database spans multiple nodes.
> > > >
> > > > Each node has a black-box lump of data that happens to have a
> fraction
> > > > of a database inside, but that's an implementation detail. It's the
> > > > wrong level of abstraction to work with.
> > > >
> > > > On Tue, Sep 6, 2016 at 1:52 PM, Joey Samonte
> > > >  > wrote:
> > > > > If you have two nodes in the cluster, the replication should be
> > > two-way between the nodes?
> > > > > What if there are three nodes? How should the replication be setup
> > > between them?
> > > > >
> > > > >> Subject: Re: Adding a node to cluster
> > > > >> From: sebastianrothbuc...@googlemail.com 
> 
> > > > >> Date: Mon, 5 Sep 2016 23:08:04 +0200
> > > > >> To: dev@couchdb.apache.org  
> > > > >>
> > > > >> Hi,
> > > > >>
> > > > >> clustering and replication are indeed two (very) separate things -
> > > and you won't get a Cluster by setting up replication. Again: treat
> the two
> > > as separate. Clustering turns several shards (on several nodes) into
> one
> > > database (from an user/caller perspective) while replication happens
> > > _between_ databases.
> > > > >> Consequently, technical underpinnings differ as well, as Bob
> > > explained below.
> > > > >>
> > > > >> Hope that gets things in perspective a little...
> > > > >>
> > > > >> Best
> > > > >>   Sebastian
> > > > >>
> > > > >> Von meinem iPhone gesendet
> > > > >>
> > > > >> > Am 05.09.2016 um 22:47 schrieb Joey Samonte <
> > > csharpdevelo...@hotmail.com  >:
> > > > >> >
> > > > >> > Does this mean that setting up replication is separate from
> setting
> > > up clustering?
> > > > >> >
> > > > >> > Does replication needs to be bi-directional between nodes?
> > > > >> >
> > > > >> >> From: rnew...@apache.org  
> > > > >> >> Subject: Re: Adding a node to cluster
> > > > >> >> Date: Thu, 25 Aug 2016 11:10:45 +0100
> > > > >> >> To: dev@couchdb.apache.org  
> > > > >> >>
> > > > >> >> Ok, seems I've confused you.
> > > > >> >>
> > > > >> >> Couchdb replication occurs over http or https, as you know. The
> > > nodes in a couchdb 2.0 cluster do not communicate with each other over
> > > http. They use Erlang rpc. Erlang rpc can be configured for TLS
> > > encryption.  It's in the Erlang faq and is fairly simple to 

Re: Adding a node to cluster

[Russell Branca]

CouchDB "replication" uses the HTTP API to replicate a database, whereas
the somewhat confusingly named "internal replication" is used within the
cluster itself as a way to synchronize state between the shard replicas.
The protocol for internal replication uses the Erlang distribution protocol
to communicate between nodes. So if your cluster is a connected set of
Erlang nodes then they will be able to synchronize database/shard state
amongst themselves.


As an example, consider a single node 1.x CouchDB server at
test1.foo.com:5984 where you want to replicate to a CouchDB 2.0 cluster at
test2.bar.com that has three db nodes and a load balancer, with the load
balancer actually having the test2.bar.com address. You communicate with
the cluster by way of the load balancer which will distribute requests to
all the nodes. The details of the cluster should be mostly transparent at
the HTTP layer, and so you can just do something like:

curl -i -X POST test1.foo.com/_replicator -d '{"source":"test1.foo.com/my_db",
"target":"test2.bar.com"}'

This will replicate "my_db" from test1 to test2 using the HTTP protocol
like you would expect from CouchDB. Within the test2 cluster the nodes will
communicate amongst themselves using internal replication (non HTTP) to
ensure the data is fully synchronized between the shard replicas.

Hopefully that clarifies things a bit.


-Russell



On Tuesday, September 6, 2016, Joey Samonte 
wrote:

> I think I understand it. Hoping to find more documentation on it.
>
> > From: wickedg...@gmail.com 
> > Date: Tue, 6 Sep 2016 15:55:00 -0700
> > Subject: Re: Adding a node to cluster
> > To: dev@couchdb.apache.org 
> >
> > Nodes in the cluster do not replicate to one another.
> >
> > Replication takes place between databases. A single node isn't a
> > database; a clustered database spans multiple nodes.
> >
> > Each node has a black-box lump of data that happens to have a fraction
> > of a database inside, but that's an implementation detail. It's the
> > wrong level of abstraction to work with.
> >
> > On Tue, Sep 6, 2016 at 1:52 PM, Joey Samonte
> > > wrote:
> > > If you have two nodes in the cluster, the replication should be
> two-way between the nodes?
> > > What if there are three nodes? How should the replication be setup
> between them?
> > >
> > >> Subject: Re: Adding a node to cluster
> > >> From: sebastianrothbuc...@googlemail.com 
> > >> Date: Mon, 5 Sep 2016 23:08:04 +0200
> > >> To: dev@couchdb.apache.org 
> > >>
> > >> Hi,
> > >>
> > >> clustering and replication are indeed two (very) separate things -
> and you won't get a Cluster by setting up replication. Again: treat the two
> as separate. Clustering turns several shards (on several nodes) into one
> database (from an user/caller perspective) while replication happens
> _between_ databases.
> > >> Consequently, technical underpinnings differ as well, as Bob
> explained below.
> > >>
> > >> Hope that gets things in perspective a little...
> > >>
> > >> Best
> > >>   Sebastian
> > >>
> > >> Von meinem iPhone gesendet
> > >>
> > >> > Am 05.09.2016 um 22:47 schrieb Joey Samonte <
> csharpdevelo...@hotmail.com >:
> > >> >
> > >> > Does this mean that setting up replication is separate from setting
> up clustering?
> > >> >
> > >> > Does replication needs to be bi-directional between nodes?
> > >> >
> > >> >> From: rnew...@apache.org 
> > >> >> Subject: Re: Adding a node to cluster
> > >> >> Date: Thu, 25 Aug 2016 11:10:45 +0100
> > >> >> To: dev@couchdb.apache.org 
> > >> >>
> > >> >> Ok, seems I've confused you.
> > >> >>
> > >> >> Couchdb replication occurs over http or https, as you know. The
> nodes in a couchdb 2.0 cluster do not communicate with each other over
> http. They use Erlang rpc. Erlang rpc can be configured for TLS
> encryption.  It's in the Erlang faq and is fairly simple to set up in newer
> Erlang releases.
> > >> >>
> > >> >> I feel I owe an example of 2.0 cluster that exclusively uses TLS
> for all communications.
> > >> >>
> > >> >> Sent from my iPhone
> > >> >>
> > >> >>> On 24 Aug 2016, at 20:47, Joey Samonte <
> csharpdevelo...@hotmail.com > wrote:
> > >> >>>
> > >> >>> What if we remove the reverse proxy and just set up the CouchDB
> nodes to allow only SSL connections, port 6984? https://wiki.apache.org/
> couchdb/How_to_enable_SSL
> > >> >>>
> > >>  Subject: Re: Adding a node to cluster
> > >>  From: rnew...@apache.org 
> > >>  Date: Wed, 24 Aug 2016 19:43:51 +0100
> > >>  To: dev@couchdb.apache.org 
> > >> 
> > >>  Assuming you mean a 2.0 cluster, no, all those nodes need to be
> able to communicate with erlang rpc (service discovery over port 4369 and
> then whatever port the node is running ong).
> > >> 
> > >> > On 24 Aug 2016, at 12:36, Joey Samonte 

Re: Getting libraries to test RCs

[Russell Branca]

N=1 might reduce some of the issues, but it won't eliminate the problem
entirely. The fundamental issue is that the "_dbs" db, which contains a
document corresponding to every clustered database in the system, does not
provide immediate consistency guarantees, and cycling databases can result
in conflicts arising in these docs. The docs contain the shard/node
mappings and conflicts can cause different nodes to have different views of
the world.

It's important to remember that the "_dbs" db powers the db -> shards
mapping and is a fundamental component of the quorum system, so
unfortunately the standard clustered quorum semantics are not available in
the "_dbs" db as it operates at a lower level. You can see the initial
synchronization during bootup in [1] which circles its way back to [2] by
way of mem3_sync_nodes.erl. You can further see where the "_dbs" db is a
local db in the way in which shards are loaded in [3] and the fallback for
creating the "_dbs" db in [4].

So in summary, the "_dbs" db operates at a lower level than the quorum
system as the db is a core component that powers the shard mappings, and
therefore uses a different approach for synchronization where each node has
a full copy of the "_dbs" db and syncs directly with the other nodes. This
is a known weak point as can be seen by the impact of cycling databases too
quickly, and so recommended best practice is to not cycle databases
quickly. Obviously this is not ideal, and this is one of the areas where a
CP config store of some sort would be a significant boon, but bolting on a
CP system to an AP system is fraught with a new set of complexities.

(A clarification on N=1: with N=1 you only have one replica of the
database, and the database exists on only one node. The rest of the nodes
still need to get the updated "_dbs" db doc so they know where the database
exists, because any node in the cluster can handle any request and it will
need to know where the database exists. In general, you have one
coordinating node and N replica nodes containing the N replicas (of each
shard) for the given database. In a three node cluster with N=3, whatever
coordinating node the request is handled by will also have a local shard
replica, but this is a special case. In a cluster with more than 3 nodes,
say 15 nodes, the coordinating node will only have a 3/15 chance to contain
a local shard (assuming round robin load balancing across nodes). So
basically every node must know where every database exists because every
node can coordinate every request.)


-Russell


[1]
https://github.com/apache/couchdb-mem3/blob/15615b295ec970ca9b12b7b54107a80b95149511/src/mem3_sync.erl#L234-L236
[2]
https://github.com/apache/couchdb-mem3/blob/15615b295ec970ca9b12b7b54107a80b95149511/src/mem3_sync.erl#L230-L232
[3]
https://github.com/apache/couchdb-mem3/blob/699308f510d335d05bfd0416ad5e893b68a7ec1d/src/mem3_shards.erl#L266-L283
[4]
https://github.com/apache/couchdb-mem3/blob/699308f510d335d05bfd0416ad5e893b68a7ec1d/src/mem3_util.erl#L214-L222

On Fri, Sep 2, 2016 at 10:43 AM, Nolan Lawson  wrote:

> Thanks, Dale. That was my recollection as well.
>
> Basically PouchDB does PUT -> DELETE -> PUT between every test, so since
> there are 1000s of tests, this race condition comes up pretty easily. We
> can add a timeout or do a random DB name, but without doing that we don't
> know if Couch 2.x is truly "passing" the test suite or not.
>
> I have some time this weekend, so I'll look into adding a patch to do the
> workaround for Couch 2. I tend to side with Jan that in a clustered system
> it can't reliably tell us when a database was truly deleted without
> sacrificing the A in CAP. PouchDB users are already familiar with the weird
> ways that databases start to behave when you actually DELETE them (e.g.
> replication gets unreliable), hence workarounds like
> https://www.npmjs.com/package/pouchdb-erase . In practice I expect PouchDB
> users to never delete databases, so this is just an artifact of our test
> suite IMO.
>
> –Nolan
>
>
> On Fri, Sep 2, 2016 at 3:14 AM, Dale Harvey  wrote:
>
> > In PouchDB we can look into a workaround that uses random names only when
> > the tests are run against Couch 2.0, however I would really like to make
> > sure that a database not being fully deleted when we get a successful
> > confirmation of deletion is considered a bug, it has impacts beyond the
> > test suite, its really hard to create a reliable system when there is no
> > way for you to be certain when a database is deleted.
> >
> > Will found it easiest to reproduce this using concurrent scripts but
> would
> > like to clarify that Pouch doesnt run the test suite in parallel, this
> bug
> > can be hit by doing CREATE -> DELETE -> CREATE, its extremely hard to
> nail
> > down and reproduce (the similiar bug in PouchDB took many attempts +
> > months). I will take a look at seeing if I can make an easier and clearer
> > steps to 

Re: [ANNOUNCE] Michelle Phung joins the PMC

[Russell Branca]

Woohoo, congrats Michelle!!


-Russell

On Monday, October 19, 2015, Klaus Trainer  wrote:

> Congrats, Michelle :)
>
>
> On 10/19/2015 11:52 AM, Jan Lehnardt wrote:
> > Dear community,
> >
> > I am delighted to announce that Michelle Phung joins the Apache CouchDB
> Project Management Committee today.
> >
> > Michelle has made outstanding, sustained contributions to the project.
> This appointment is an official acknowledgement of their position within
> the community, and our trust in their ability to provide oversight for the
> project.
> >
> > Everybody, please join me in congratulating Michelle!
> >
> > On behalf of the CouchDB PMC,
> > Jan
> >
>

Re: [ANNOUNCE] Garren Smith joins the PMC

[Russell Branca]

Congrats Garren!! Glad to have you aboard.


-Russell

On Monday, October 19, 2015, Klaus Trainer  wrote:

> Congrats, Garren :)
>
>
> On 10/19/2015 12:31 PM, Jan Lehnardt wrote:
> > Dear community,
> >
> > I am delighted to announce that Garren Smith joins the Apache CouchDB
> Project Management Committee today.
> >
> > Garren has made outstanding, sustained contributions to the project.
> This appointment is an official acknowledgement of their position within
> the community, and our trust in their ability to provide oversight for the
> project.
> >
> > Everybody, please join me in congratulating Garren!
> >
> > On behalf of the CouchDB PMC,
> > Jan
> >
>

Re: 2.0 Progress

[Russell Branca]

On Thursday, June 18, 2015, Jan Lehnardt j...@apache.org wrote:


  On 17 Jun 2015, at 17:50, Jan Lehnardt j...@apache.org javascript:;
 wrote:
 
  Hey all,
 
  Alexander, Bob and I had a bit of a brainstorming session today on what
 is missing to get 2.0 out the door. We talked about missing features and
 what to do about them. The following notes is what we think is best, but of
 course, these are just suggestions and we’d love your feedback! Especially
 on the TBD items.
 
  The notes are dense, let us know if you have any questions or need any
 clarification :)
 
  * * *
 
  # Missing Features
 
  ## Availble in 1.x but missing in 2.x
 
  - [X] vhosts: Done.
 
  - [X] /_config on :5984
   - no cluster wide config
   - per-node config can be set from any node:
   - only expose /_node_config/node-fqdn/section/key
   - first stab of this already in master
   - Fauxton to offer “tabbed” interface for per-cluster config view
   - post 2.0, if we find a backing store that would give us a
 cluster-consistent :5984/_config/section/key, we can enable that.
 
 
  - [X]: /_stats on :5984
   - same as with /_config:
   - /_node_stats/fqdn/etc


 Updates on _node_config and _node_stats

 We will now propose /_node/fqdn/_config and /_node/fqdn/_stats with
 the option of adding more endpoints there, later.

 The rest stays as is.



+1


-Russell




 Best
 Jan
 --

 
 
  - [X] dynamic http endpoint configuration for :5984
   - integrate Cloudant’s work that allows for erlang applications to
 register their own routes in a priv/ file. Adding a route is done by adding
 an app, changing a route is by updating an app and its priv/ file
   - /_config can be used to disable endpoints
   - TBD: what if two applications define the same endpoint? Options:
   - undefined
   - offending app doesn’t load
   - server doesn’t start / stops
   - define that endpoints can’t be overwritten, define strict sort
 order for loading, load core apps preferred to ensure working node +
 *handwaving*
 
 
  - [X] externals  proxy
  - keep both for 2.0
  - add to chttpd routes loading:
- 1. load all application routes
- 2. load [external] and [proxy] routes
- 3. listen to [external] and [proxy] config changes and adjust route
 map at runtime
 
 
  - [X] logging configuration of level  file  apps (
 http://docs.couchdb.org/en/latest/config/logging.html)
   - dynamic configuration of the log file won’t be possible anymore,
 because lager works differently
   - since the configuration for lager is more comprehensive, 1.x-style
 configuration is no longer possible
   - TBD: however, it is possible to set the log level at runtime. Two
 options:
-  setting /_config/log:backend/level to logatom / couch_log
 listens to config updates and calls
 https://github.com/basho/lager#runtime-loglevel-changes
- new endpoint, e.g. PUT /_log/backend/logatom
   - log atoms change, mark as BC break
 
 
  - [X] rewrites (?)
   - keep, should just work
 
 
  - [X] documentation
   - we have (or will have soon): API docs, upgrade/migration guides / new
 features
   - we *don’t* have: how the cluster works (dynamo + couch specific
 details)
 - needs help!
 
  - [X ] build script
   - finish unix version (Jan)
   - integrate with Mac app (Jan)
   - TBD: Windows
- Can Nick North help?
- maybe ship 2.0 with experimental Windows, since the code has
 never run on Windows(?)
 
 
  ## Missing in 2.x
 
  - [X] single node mode
   - “single node mode” just means “cluster of one”
   - the chttp-layer might be slower than couch_httpd, if so, we have good
 incentives to make this fast.
   - going from 1.x to 2.x-single-node will requires changes in how an app
 uses the CouchDB API (as per our Breaking Changes documentation).
   - :5986 is 127.0.0.1-only and strongly discouraged for general use, ops
 folks might want to use it.
 
 
  - [X] add/remove nodes, cluster rebalance
   - add/remove node needs documentation
   - rebalancing:
- tool to help figure out what needs doing
- might be ready by 2.0, post 2.0 otherwise
 
  * * *
 
  Best
  Jan
  --
 

 --
 Professional Support for Apache CouchDB:
 http://www.neighbourhood.ie/couchdb-support/

Re: On Plugins and Extensibility

[Russell Branca]

Hey Paul,


Thanks for the great writeup!

Couple of questions:

How do priv/*.cfg files work with dynamic config updates? Seems like we'll
lose the ability to write changes back to the config file, like we have
with default.ini. Although I've been wondering for a while if allowing
config:set/* to persist data is an anti-pattern compared to forcing
persistent updates to be made directly in config files with the hopes they
are done so in a VCS. Also, if we switch to priv/*.cfg how will users
update those files? seems like we'll have to poll the file system for every
one of those config files, or are you thinking that those files would only
be updated as part of releases?

Have you thought at all about using the mochiglobal.erl approach to
regenerate a module with all the plugin dispatch compiled as functions?
Seems like using shared ETS tables could become a source of contention for
lots of concurrent requests. Proper benchmarks would provide better
understanding as to whether this is actually an issue.


-Russell

On Thu, May 21, 2015 at 3:29 PM, Paul Davis paul.joseph.da...@gmail.com
wrote:

 Hey everyone,

 So I've been meaning to write this email for sometime but have been
 kept busy with lots of super fun things that are super fun. Anyway, I
 just wanted to get this out there to start getting feed back from
 everyone involved.

 Also, while this is called Plugin Proposal it shouldn't be confused
 with the original couch_plugins use case. This is a lot lower level
 (and may be used by something like couch_plugins if we go there again
 in the future). Generally speaking this is just for cleaning up
 internals and for people that want to run either minimal installs of
 CouchDB or include the CouchDB in a larger Erlang application.

 Plugin Proposal
 ===

 Background
 --

 As we've grown the code base to include more and more applications
 we're getting to the point where we've started adding various points
 of extension in various ways. The best existing example is the
 couch_stats application which loads stat/metric definitions from
 applications. Henning Diedrich has some unmerged worked which looks to
 follow a similar path for HTTP URL handlers. And Ilya Khlopotov has
 some work for providing vendor specific hooks.

 While each of these have some overlaps in their intended use case,
 they also share the fact that they've all implemented their own idea
 of extensibility in slightly different ways. That's not necessarily
 bad, but I think that we could reduce a lot of complexity if we take a
 step back and write a utility application that could then be used to
 support each of these features so that we can have both the
 extensibility as well as simplify the implementation of each
 individual feature.

 I'll start with a bit of background and then describe a general
 approach as well as show some hopefully explicit example snippets of
 how such a system might be used. Granted I haven't written out an
 entire implementation of this so I may be off the mark in some places.

 Bikeshed First
 --

 I have no idea what we'd call this. We could repurpose the
 couch_plugins app conceivably or make something new. For the the
 purposes of this document I'll call it couch_epi (for extensible
 plugin interface) and hopefully that's terrible enough someone will
 think of a better name for the actual application.

 Requirements
 

 The three major requirements I've thought of are:

   # Automatically discoverable
   # Minimize apps that need to be started for tests
   # Support release upgrades

 === Automatically Discoverable ===

 The biggest thing here is that I don't want to require a change to a
 default.ini or similar to enable or disable specific functionality
 when we can already signify that by having the application present or
 not. This is both for groups that may want to add new Erlang
 applications to a release as well as anyone that wants to run a
 minimal/embedded Couch. These are both obviously advanced uses but I
 think are important given the number of ways that CouchDB is being
 used.

 === Minimize the apps that need to be started for tests ===

 This one I think should be obvious to anyone that's been writing unit
 tests lately. There are some often silly places where we require
 applications be started just to run some tests. For example, places
 where we may want to call a function that's been instrumented and
 requires couch_stats to have knowledge about the stat.

 === Support release upgrades ===

 This one is obviously fairly advanced and limited in its audience but
 its something I'd like to at least consider in the design. This comes
 into effect for things like couch_stats that use a text file for its
 extension method. The issue is that the release upgrade mechanics
 don't provide any sort of signal that is easily usable to indicate
 when this file has changed during an upgrade so we're left polling the
 file system which is less than 

[jira] [Commented] (COUCHDB-2659) CouchDB master failing the PouchDB test suite - live changes

[Russell Branca (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500886#comment-14500886
 ] 

Russell Branca commented on COUCHDB-2659:
-

Is this a duplicate of COUCHDB-2657?

 CouchDB master failing the PouchDB test suite - live changes
 

 Key: COUCHDB-2659
 URL: https://issues.apache.org/jira/browse/COUCHDB-2659
 Project: CouchDB
  Issue Type: Bug
  Security Level: public(Regular issues) 
  Components: Database Core
Reporter: Nolan Lawson

 See [this comment | 
 https://github.com/pouchdb/pouchdb/pull/3722#issuecomment-91571425] for 
 details. We have automated tests that pull in the latest CouchDB master 
 branch and run the PouchDB test suite against it, and recently our tests 
 started failing due to [apache/couchdb#313 | 
 https://github.com/apache/couchdb/pull/313]. So I switched back to admin 
 party mode, but now it's failing for another reason:
 {quote}
 1) test.attachments.js-http #3074 live changes():
  Uncaught Database encountered an unknown error
 {quote}
 Instructions for running the PouchDB test suite are in our TESTING.md, it's 
 pretty straightforward but basically you just want to point it to any CouchDB 
 server like so and run the tests:
 {code}
 BAIL=0 COUCH_HOST=http://localhost:15984 npm test
 {code}
 {{BAIL=0}} will prevent bailing upon the first error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (COUCHDB-2657) if _metadata is there, creating a db does not work

[Russell Branca (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500885#comment-14500885
 ] 

Russell Branca commented on COUCHDB-2657:
-

I tracked this down to making a `fabric:open_doc` call from within the 
`cassim_metadata_cache` gen_server. The problem is that `fabric:open_doc` does 
a `receive` call while waiting for a worker response, and this intercepts any 
other messages sent in. I've got a patch out [1] that shows the rough solution. 
I did notice however that in the test case we're still getting a 409 error on 
the second request, so we'll want to make sure we handle that properly. Should 
we check and see if the values are the same? and if not reload? That might be 
worthwhile but potentially expensive. Alternatively we could automatically 
create the security doc on db creation.

Also, I think COUCHDB-2659 might be a duplicate of this issue.



[1] https://github.com/apache/couchdb-cassim/tree/2657-fix-cassim-fabric-calls

 if _metadata is there, creating a db does not work
 --

 Key: COUCHDB-2657
 URL: https://issues.apache.org/jira/browse/COUCHDB-2657
 Project: CouchDB
  Issue Type: Bug
  Security Level: public(Regular issues) 
  Components: BigCouch, Database Core
Reporter: Robert Kowalski
Priority: Blocker

 with the new _setup_cluster endpoint a _metadata db is created.
 when it exists, and we create a db, read it _all_doc immediatly [1] (Fauxton 
 does that) we get:
 `document update conflict`
 [1] 
 http://localhost:8000/sdfsdtest/_all_docs?startkey=%22_design%2F%22endkey=%22_design0%22include_docs=truelimit=501



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (COUCHDB-2657) if _metadata is there, creating a db does not work

[Russell Branca (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500801#comment-14500801
 ] 

Russell Branca commented on COUCHDB-2657:
-

Thanks for the report [~robertkowalski]. I ported the test script to Python [1] 
and I'm able to reproduce these issues. Deleting the _metadata db and rerunning 
the test works, but I haven't been able to test a fresh start of the dev 
cluster without the _metadata db as it keeps getting recreated.

[~ilyak] where are you observing those failures? Is that during the first run 
of setup? I'm not seeing those.


[1] https://gist.github.com/chewbranca/44e266140cbb74db9c47

 if _metadata is there, creating a db does not work
 --

 Key: COUCHDB-2657
 URL: https://issues.apache.org/jira/browse/COUCHDB-2657
 Project: CouchDB
  Issue Type: Bug
  Security Level: public(Regular issues) 
  Components: BigCouch, Database Core
Reporter: Robert Kowalski
Priority: Blocker

 with the new _setup_cluster endpoint a _metadata db is created.
 when it exists, and we create a db, read it _all_doc immediatly [1] (Fauxton 
 does that) we get:
 `document update conflict`
 [1] 
 http://localhost:8000/sdfsdtest/_all_docs?startkey=%22_design%2F%22endkey=%22_design0%22include_docs=truelimit=501



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (COUCHDB-2657) if _metadata is there, creating a db does not work

[Russell Branca (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500805#comment-14500805
 ] 

Russell Branca commented on COUCHDB-2657:
-

Also the error I'm seeing with this is:


[gen_server:call/2 L182,cassim_security:get_security_doc/1 
L58,cassim_security:get_security/2 L46,chttpd_db:do_db_req/2 
L263,chttpd:handle_request/1 L210,mochiweb_http:headers/5 
L93,proc_lib:init_p_do_apply/3 L237]

 if _metadata is there, creating a db does not work
 --

 Key: COUCHDB-2657
 URL: https://issues.apache.org/jira/browse/COUCHDB-2657
 Project: CouchDB
  Issue Type: Bug
  Security Level: public(Regular issues) 
  Components: BigCouch, Database Core
Reporter: Robert Kowalski
Priority: Blocker

 with the new _setup_cluster endpoint a _metadata db is created.
 when it exists, and we create a db, read it _all_doc immediatly [1] (Fauxton 
 does that) we get:
 `document update conflict`
 [1] 
 http://localhost:8000/sdfsdtest/_all_docs?startkey=%22_design%2F%22endkey=%22_design0%22include_docs=truelimit=501



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSSION] Move Fauxton to its own mailing list?

[Russell Branca]

+1

On Tue, Apr 14, 2015 at 2:24 PM, Paul J Davis paul.joseph.da...@gmail.com
wrote:

 +1



  On Apr 14, 2015, at 3:50 PM, Andy Wenk andyw...@apache.org wrote:
 
  +1
 
  On 14 April 2015 at 21:02, Alexander Shorin kxe...@gmail.com wrote:
 
  On Tue, Apr 14, 2015 at 5:09 PM, Jan Lehnardt j...@apache.org wrote:
  We create new mailing list c...@couchdb.apache.org (or your favourite
  bike shed) that gets all JIRA and GitHub traffic.
 
  dev@ then is for discussion and decisions for all code projects
  (including Fauxton).
 
  If you want to keep track of tickets and pull requests, sign up for
 code@
  .
 
  There is a bit of a discrepancy between discussions in JIRA and dev@,
  but I’m willing to take that risk, as most people probably bulk-delete
 all
  JIRA mails anyway :)
 
  +1
 
 
  --
  ,,,^..^,,,
 
 
 
  --
  Andy Wenk
  Hamburg - Germany
  RockIt!
 
  GPG fingerprint: C044 8322 9E12 1483 4FEC 9452 B65D 6BE3 9ED3 9588
 
  https://people.apache.org/keys/committer/andywenk.asc

Re: [PROPOSAL] Move transactional email out of dev@

[Russell Branca]

Seems a bit odd to separate github PR comments from JIRA comments as the
overlap on development related discussion can be large.

As for actually separating out the messages, I'm -0. Email filters work
really well for this sort of thing.


-Russell

On Tuesday, March 17, 2015, till t...@php.net wrote:

 We use this in-house: https://github.com/heroku/devdigest/

 Gives you the daily run-down on what happened on a set of given repos or in
 an org. It's tweak-able (= open source, Ruby code).

 Till

 On Tue, Mar 17, 2015 at 4:40 PM, Noah Slater nsla...@apache.org
 javascript:; wrote:

  I am +1 on a monthly reminder. Do we have a volunteer to code that up?
  Could even be a weekly reminder. I'm imagining it would contain a
  summary of all open/active PRs, so people might be like ooh, I am
  interested in that one while the discussion is still happening!
 
  On 16 March 2015 at 23:04, Sebastian Rothbucher
  sebastianrothbuc...@googlemail.com javascript:; wrote:
   +1 with Joan and Alexander also, i.e. do a monthly thing instead of
   blasting out dozens a day to dev@ = this is great, thanks for
 bringing
  it
   up!   Sebastian
  
   On Mon, Mar 16, 2015 at 6:42 PM, Alexander Shorin kxe...@gmail.com
 javascript:;
  wrote:
  
   On Mon, Mar 16, 2015 at 7:46 PM, Joan Touzet woh...@apache.org
 javascript:; wrote:
I'd be in support of an automated email monthly to dev@ that
 reminds
people where to go to look for GH PRs, JIRA ticket updates, etc.
  
   Good idea btw. +1
  
   --
   ,,,^..^,,,
  
 
 
 
  --
  Noah Slater
  https://twitter.com/nslater
 

[jira] [Created] (COUCHDB-2628) Resolve shard(s)_db naming discrepancies

[Russell Branca (JIRA)]

Russell Branca created COUCHDB-2628:
---

 Summary: Resolve shard(s)_db naming discrepancies
 Key: COUCHDB-2628
 URL: https://issues.apache.org/jira/browse/COUCHDB-2628
 Project: CouchDB
  Issue Type: Bug
  Security Level: public (Regular issues)
  Components: BigCouch
Reporter: Russell Branca


Looks like there's a handful of discrepancies where `shard_db` is referenced 
rather than `shards_db` which is the version actually used when creating the 
relevant database [1]. A quick grep around shows about 8 uses of `shard_db`.



[1] 
https://github.com/apache/couchdb-mem3/blob/master/src/mem3_shards.erl#L228-L229



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (COUCHDB-2628) Resolve shard(s)_db naming discrepancies

[Russell Branca (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338879#comment-14338879
 ] 

Russell Branca commented on COUCHDB-2628:
-

+1, thanks [~kxepal]!

 Resolve shard(s)_db naming discrepancies
 

 Key: COUCHDB-2628
 URL: https://issues.apache.org/jira/browse/COUCHDB-2628
 Project: CouchDB
  Issue Type: Bug
  Security Level: public(Regular issues) 
  Components: BigCouch
Reporter: Russell Branca

 Looks like there's a handful of discrepancies where `shard_db` is referenced 
 rather than `shards_db` which is the version actually used when creating the 
 relevant database [1]. A quick grep around shows about 8 uses of `shard_db`.
 [1] 
 https://github.com/apache/couchdb-mem3/blob/master/src/mem3_shards.erl#L228-L229



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (COUCHDB-2629) Hide internal config details from other applications

[Russell Branca (JIRA)]

Russell Branca created COUCHDB-2629:
---

 Summary: Hide internal config details from other applications
 Key: COUCHDB-2629
 URL: https://issues.apache.org/jira/browse/COUCHDB-2629
 Project: CouchDB
  Issue Type: Bug
  Security Level: public (Regular issues)
  Components: Database Core
Reporter: Russell Branca


Throughout the code base we do things like `config:get(couch_httpd_auth, 
authentication_db, _users)` which exposes internal details to other 
applications that don't need to be aware of the particulars. This duplicates 
code unnecessarily and causes problematic typos to go unnoticed like in 
COUCHDB-2628.

One possible approach that I'm a fan of is to go into the application 
responsible for relevant config value, and add a function to hide the details. 
For instance, this is done in `cassim_metadata_cache:metadata_db`: 
https://github.com/apache/couchdb-cassim/blob/windsor-merge/src/cassim_metadata_cache.erl#L58-L59



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] accept Mango contribution from Cloudant IBM

[Russell Branca]

+1 :D

On Friday, December 19, 2014, Jan Lehnardt j...@apache.org wrote:

 +1, what a great holiday present :)

 Best
 Jan
 --

  On 18 Dec 2014, at 23:36 , Robert Kowalski r...@kowalski.gd
 javascript:; wrote:
 
  Hi list,
 
  Cloudant IBM wants to contribute the Mango code, a MongoDB API layer
  for CouchDB to the ASF. This needs a vote of the CouchDB project. The
  tarball is at
 https://people.apache.org/~robertkowalski/dist/ip-clearance/mango-asf-ip-clearance-2015-12-18.tar.gz
 
  The SHA is:
 
  SHA512(mango-asf-ip-clearance-2015-12-18.tar.gz)=
 
 809b28fb0ccdab19401fb15a4d0f663ee7a51d2c6cbea4cd5e20621dfac16862f186c0375a3ef485775f78bbc29d88e42f7a6d8abe2d1ef20e96e226298f37ee
 
 
  The main repository is located at https://github.com/cloudant/mango -
  I also created the tarball from that source.
 
  We don't have a match for votings on code contributions in our bylaws,
  so I am chosing Majority Approval by the PMC (that is, 3 +1’s and no
  -1’s) and a 72 hour voting period beginning now (like the BigCouch
  vote in August)
 
  Best,
  Robert

[jira] [Commented] (COUCHDB-2422) Cassim and underscored databases

[Russell Branca (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14216700#comment-14216700
 ] 

Russell Branca commented on COUCHDB-2422:
-

A simple fix here is to just prepend a string like db/ to the front of the id 
[1]. This would solve the problem but I think we should also do this to better 
represent that the document corresponds to settings for a database. This would 
make it easier to distinguish between database specific docs and global docs.


[1] 
https://github.com/apache/couchdb-cassim/blob/master/src/cassim_metadata_cache.erl#L73

 Cassim and underscored databases
 

 Key: COUCHDB-2422
 URL: https://issues.apache.org/jira/browse/COUCHDB-2422
 Project: CouchDB
  Issue Type: Bug
  Security Level: public(Regular issues) 
  Components: Database Core
Reporter: Alexander Shorin

 1. Run dev cluster
 2. Create {{_users}} database
 {code}
 http PUT http://localhost:15984/_users
 HTTP/1.1 201 Created
 Cache-Control: must-revalidate
 Content-Length: 12
 Content-Type: text/plain; charset=utf-8
 Date: Fri, 31 Oct 2014 10:36:19 GMT
 Location: http://localhost:15984/_users
 Server: CouchDB/9938fac (Erlang OTP/17)
 X-Couch-Request-ID: f273be89
 X-CouchDB-Body-Time: 0
 {ok:true}
 {code}
 3. Create cassim database
 {code}
 http PUT http://localhost:15984/cassim
 HTTP/1.1 201 Created
 Cache-Control: must-revalidate
 Content-Length: 12
 Content-Type: text/plain; charset=utf-8
 Date: Fri, 31 Oct 2014 10:37:06 GMT
 Location: http://localhost:15984/cassim
 Server: CouchDB/9938fac (Erlang OTP/17)
 X-Couch-Request-ID: 8bdcc974
 X-CouchDB-Body-Time: 0
 {ok:true}
 {code}
 4. Try to access {{_users}} database:
 {code}
 http http://localhost:15984/_users
 HTTP/1.1 400 Bad Request
 Cache-Control: must-revalidate
 Content-Length: 89
 Content-Type: text/plain; charset=utf-8
 Date: Fri, 31 Oct 2014 10:34:22 GMT
 Server: CouchDB/9938fac (Erlang OTP/17)
 X-Couch-Request-ID: e9f4a5b3
 X-CouchDB-Body-Time: 0
 {error:bad_request,reason:Only reserved document ids may start with 
 underscore.}
 {code}
 Oh wait...? This is because cassim creates documents with database name as 
 the prefix and for {{_users}} and {{_replicator}} it clashes with 
 restrictions about document names.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Removing delayed_commits

[Russell Branca]

Jan Lehnardt writes:

 On 10 Nov 2014, at 16:11 , Alexander Shorin kxe...@gmail.com wrote:
 
 On Mon, Nov 10, 2014 at 6:06 PM, Jan Lehnardt j...@apache.org wrote:
 On 10 Nov 2014, at 15:17 , Klaus Trainer klaus_trai...@posteo.de wrote:
 
 Hey Jan,
 
 you didn't provide an argument, so I can only guess: Do you think that
 we shouldn't tackle that right now, as it would potentially delay the
 2.0 release?
 
 One of my goals for CouchDB 2.0 is that people upgrading form 1.0 and
 especially people who continue use CouchDB in a single-node scenario
 have an easy time. Just breaking more things because we happen to be
 bumping a version number is not a good motivation. Especially in our
 new world of avoiding attaching marketing connotation to major release
 versions (except 2.0 :), we can release CouchDB 3.0 and 4.0 shortly
 after 2.0 if it means we break BC in a single way. If we keep BC breaks
 to a minimum and make every transition up a major version as easy as
 possible, we don’t run into a Python 3 situation that creates a major
 schism in the user community that takes a decade to heal.
 
 Let’s not break things because we update the major version number,
 instead, let’s update the major version number whenever we break
 back backward compatibility.
 
 Suddenly, active delayed_commits is what is breaking CouchDB 2.0 - ask
 Russell how long he fight with database migrations from old 1.x to 2.0
 until he found delayed_commits turned on.  Also, I don't think that
 this change is breaking something expect benchmarks speed.

 I was referring to the original email that said:

 I'd like to propose that we now (as we're already breaking API
 compatibility with the new major release) go the full way and
 remove that feature entirely.


 I generally disagree with that reasoning as outlined above.

 I’d be happy to hear more about what problems Russell had, but it’s
 the first time that info hits dev@ and it wasn’t brought up in this
 thread so far.

 Best
 Jan

In general I think delayed_commits is a misfeature, but I think the big
win is disabling it by default. In general I'm +1 to removing it.

I've been bitten several times by delayed_commits being enabled without
me realizing it causing significant departures from expected behavior,
both during the database migrations work and also with eunit test
port. Although admittedly both of those issues went away when
delayed_commits was disabled.

I think we should deprecate delayed commits in the 2.0
release. Switching it to disabled by default should be a good indication
if people are effected by it as I imagine we'll get some feedback if
that causes problems for anyone. After that I think we'll have a better
idea if it's safe for removal.


--
Russell

[jira] [Commented] (COUCHDB-2334) Metadata db cassim does not exist

[Russell Branca (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14143725#comment-14143725
 ] 

Russell Branca commented on COUCHDB-2334:
-

Logging at error level is probably overkill, we could switch that down to info 
or notice. Newson, it's worth noting that using cassim for security properties 
is enabled by the act of creating the cassim database, not by setting a 
configuration value. The metadata cache process keeps checking for the cassim 
database to be created and then will switch over to using it once it sees it 
exists.

I'm not a huge fan of this approach, and I've been thinking we should add a 
configuration setting for enabling cassim for security properties, and possibly 
also a general flag for turning on cassim, although the creation of the cassim 
database may be sufficient for that. If we add a config setting to enable 
storing security properties in cassim, we can use cassim for other metadata 
without having to force users to store security properties there as well.

 Metadata db cassim does not exist
 ---

 Key: COUCHDB-2334
 URL: https://issues.apache.org/jira/browse/COUCHDB-2334
 Project: CouchDB
  Issue Type: Bug
  Security Level: public(Regular issues) 
  Components: BigCouch
Reporter: Alexander Shorin

 And so happens every 5 minutes:
 {code}
 2014-09-20 04:00:06.786 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:05:06.788 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:10:06.790 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:15:06.792 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:20:06.794 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:25:06.796 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:30:06.798 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:35:06.800 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:40:06.802 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:45:06.804 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:50:06.806 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:55:06.808 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 05:00:06.810 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 05:05:06.812 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 05:10:06.814 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (COUCHDB-2334) Metadata db cassim does not exist

[Russell Branca (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14144038#comment-14144038
 ] 

Russell Branca commented on COUCHDB-2334:
-

I'm not attached to having the db named cassim, _meta sounds fine to me. 
Should we add an additional setting to use cassim for security?

 Metadata db cassim does not exist
 ---

 Key: COUCHDB-2334
 URL: https://issues.apache.org/jira/browse/COUCHDB-2334
 Project: CouchDB
  Issue Type: Bug
  Security Level: public(Regular issues) 
  Components: BigCouch
Reporter: Alexander Shorin

 And so happens every 5 minutes:
 {code}
 2014-09-20 04:00:06.786 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:05:06.788 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:10:06.790 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:15:06.792 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:20:06.794 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:25:06.796 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:30:06.798 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:35:06.800 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:40:06.802 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:45:06.804 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:50:06.806 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:55:06.808 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 05:00:06.810 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 05:05:06.812 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 05:10:06.814 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (COUCHDB-2334) Metadata db cassim does not exist

[Russell Branca (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-2334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russell Branca closed COUCHDB-2334.
---
Resolution: Won't Fix

This is expected behavior. Every five minutes we check to see if the database 
exists, so that cassim can be enabled: 
https://github.com/apache/couchdb-cassim/blob/windsor-merge/src/cassim_metadata_cache.erl#L113

 Metadata db cassim does not exist
 ---

 Key: COUCHDB-2334
 URL: https://issues.apache.org/jira/browse/COUCHDB-2334
 Project: CouchDB
  Issue Type: Bug
  Security Level: public(Regular issues) 
  Components: BigCouch
Reporter: Alexander Shorin

 And so happens every 5 minutes:
 {code}
 2014-09-20 04:00:06.786 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:05:06.788 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:10:06.790 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:15:06.792 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:20:06.794 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:25:06.796 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:30:06.798 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:35:06.800 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:40:06.802 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:45:06.804 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:50:06.806 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 04:55:06.808 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 05:00:06.810 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 05:05:06.812 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 2014-09-20 05:10:06.814 [error] node1@127.0.0.1 0.341.0 Metadata db 
 cassim does not exist
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (COUCHDB-2328) Chttpd _show/_update functions not working

[Russell Branca (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russell Branca resolved COUCHDB-2328.
-
   Resolution: Fixed
Fix Version/s: 2.0.0

 Chttpd _show/_update functions not working
 --

 Key: COUCHDB-2328
 URL: https://issues.apache.org/jira/browse/COUCHDB-2328
 Project: CouchDB
  Issue Type: Bug
  Security Level: public(Regular issues) 
  Components: HTTP Interface
Reporter: Russell Branca
 Fix For: 2.0.0


 The _show and _update endpoints use `chttpd_external:send_external_response` 
 which does not add CORS headers. As a result OPTIONS requests to these 
 endpoints work, but the actual requests do not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (COUCHDB-2328) Chttpd _show/_update functions not working

[Russell Branca (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russell Branca closed COUCHDB-2328.
---

 Chttpd _show/_update functions not working
 --

 Key: COUCHDB-2328
 URL: https://issues.apache.org/jira/browse/COUCHDB-2328
 Project: CouchDB
  Issue Type: Bug
  Security Level: public(Regular issues) 
  Components: HTTP Interface
Reporter: Russell Branca
 Fix For: 2.0.0


 The _show and _update endpoints use `chttpd_external:send_external_response` 
 which does not add CORS headers. As a result OPTIONS requests to these 
 endpoints work, but the actual requests do not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (COUCHDB-2328) Chttpd _show/_update functions not working

[Russell Branca (JIRA)]

Russell Branca created COUCHDB-2328:
---

 Summary: Chttpd _show/_update functions not working
 Key: COUCHDB-2328
 URL: https://issues.apache.org/jira/browse/COUCHDB-2328
 Project: CouchDB
  Issue Type: Bug
  Security Level: public (Regular issues)
  Components: HTTP Interface
Reporter: Russell Branca


The _show and _update endpoints use `chttpd_external:send_external_response` 
which does not add CORS headers. As a result OPTIONS requests to these 
endpoints work, but the actual requests do not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Rewriting the CouchDB HTTP Layer

[Russell Branca]

How long do we want to keep supporting R14B01? Hopefully we can jump
ship over to 17 at some point and stay more current with Erlang
releases.

Both WebMachine and Cowboy supoort the REST-ful resource declarations
which I'm a fan of, but using those would introduce breaking changes to
the CouchDB API. Perhaps we make a 3.0 release with the HTTP changes and
drop support for R14*?


-Russell

Paul Davis writes:

 One ding against Cowboy is that Loïc isn't very interested in
 supporting old releases so we'd have to make minor updates if we wan't
 to keep supporting the R14B01 releases. Last time I did this it was
 trivial but a bit of an annoyance that such a trivial change was
 something I'd have to maintain downstream indefinitely whilst I needed
 to support R14B01. It also means that future upgrades may be a bit
 difficult if Cowboy ever starts embracing new language features like
 maps that are just impossible to support on older VMs.

 On Wed, Aug 20, 2014 at 7:45 PM, Russell Branca russ...@chewbranca.com 
 wrote:
 Thanks Andy! The next step I want to take is build another prototype in
 Cowboy and compare it with the web machine  implementation. Hopefully will
 have some time for that over the weekend.

 -Russell
 On Aug 20, 2014 5:01 AM, Andy Wenk andyw...@apache.org wrote:

 Hey Russel,

 I have read your blog post about Rewriting the CouchDB HTTP Layer. Thanks
 a lot for that!

 As a note from a non-core-CouchDB-dev - this sounds great and very
 reasonable. Making the code easier to test, removing unnecessary code
 duplication, organising the code even better and making it easier to write
 plugins are things, that will lead to better code and will make it easier
 for devs to contribute. So all thumbs up! Great work! I hope the discussion
 will lead to a good decision :)

 Cheers

 Andy


 On 19 August 2014 02:27, Russell Branca russ...@chewbranca.com wrote:

  On Aug 17, 2014 8:15 PM, Jason Smith jason.h.sm...@gmail.com wrote:
  
   Hi, Russell. This is okay for a starting point but it is a bit vague.
  Could
   you perhaps flesh out the plan and make it more comprehensive?
  
   ^^ That is a joke!
  
   Seriously, thank you very much for this analysis and plan. This is very
   exciting! (Not least because the http codebase is the part I know best
  and
   I can get excited about.)
  
 
  Thanks!
 
   One quick question that I don't see from your writeup: What version of
   CouchDB are you thinking of targeting? 2.0? 2.1? 3.0? Is this
 completely
  an
   internal change, or does it affect users?
  
 
  I think this is outside the scope of 2.0 given how close we are on that.
  There's a fair bit of legwork involved in doing the rewrite, so I
 wouldn't
  want to block 2.0. So I think the question is 2.x or 3.0. If we go the
 web
  machine route we should rework the api and definitely introduce backwards
  incompatible changes, so we would want to do a 3.0 release there. If we
  used cowboy we could mimic the current api and release 2.x.
 
   For me, I am not so interested in an internal rewrite with zero
 advantage
   (besides it's cleaner), however I am am very interested to use the
   rewrite for a better opportunity to explore plugin opportunities or
 other
   extensibility features.
  
  
 
  This rewrite needs to happen. The internals of the http layer need some
  serious love and there's a lot of duplication to remove. I think the big
  win here is that this would get the ball rolling on taking a closer look
 at
  the various internal applications and figuring out what needs to be
  restructured. For instance, the next logical step after reworking the
 http
  later is to standardize the clustered and local api modules, and there
 was
  some great discussion in the Dev channel today about that.
 
  The more we can decouple the various apps, the more easily we can extend
  CouchDB with plugins and new functionality.
 
  -Russell
 
  
  
   On Mon, Aug 18, 2014 at 1:41 AM, Russell Branca chewbra...@apache.org
 
   wrote:
  
# Rewriting the CouchDB HTTP Layer
   
With the light at the end of tunnel on the BigCouch merge, I thought
it was time to get the conversation going on cleaning up the current
HTTP stack duality. We've got a good opportunity to do some major
cleanup, remove duplication, and really start more clearly separating
the various components of CouchDB.
   
   
## Primary objectives
   
* Consolidate down to one HTTP layer
* Isolate HTTP functionality
* Separate HTTP server from HTTP resources
* Easy plugin integration
* Build clustered/local API
   
   
### Consolidate down to one HTTP layer
   
We currently have two HTTP layers, `couch_httpd` and `chttpd`. This
was a useful construct when BigCouch was a separate application where
isolating the clustered layer from the local layer was necessary, and
quite useful.
   
This is no longer the case, and we can significantly reduce code
duplication

Re: [DISCUSS] Rewriting the CouchDB HTTP Layer

[Russell Branca]

Thanks Andy! The next step I want to take is build another prototype in
Cowboy and compare it with the web machine  implementation. Hopefully will
have some time for that over the weekend.

-Russell
On Aug 20, 2014 5:01 AM, Andy Wenk andyw...@apache.org wrote:

 Hey Russel,

 I have read your blog post about Rewriting the CouchDB HTTP Layer. Thanks
 a lot for that!

 As a note from a non-core-CouchDB-dev - this sounds great and very
 reasonable. Making the code easier to test, removing unnecessary code
 duplication, organising the code even better and making it easier to write
 plugins are things, that will lead to better code and will make it easier
 for devs to contribute. So all thumbs up! Great work! I hope the discussion
 will lead to a good decision :)

 Cheers

 Andy


 On 19 August 2014 02:27, Russell Branca russ...@chewbranca.com wrote:

  On Aug 17, 2014 8:15 PM, Jason Smith jason.h.sm...@gmail.com wrote:
  
   Hi, Russell. This is okay for a starting point but it is a bit vague.
  Could
   you perhaps flesh out the plan and make it more comprehensive?
  
   ^^ That is a joke!
  
   Seriously, thank you very much for this analysis and plan. This is very
   exciting! (Not least because the http codebase is the part I know best
  and
   I can get excited about.)
  
 
  Thanks!
 
   One quick question that I don't see from your writeup: What version of
   CouchDB are you thinking of targeting? 2.0? 2.1? 3.0? Is this
 completely
  an
   internal change, or does it affect users?
  
 
  I think this is outside the scope of 2.0 given how close we are on that.
  There's a fair bit of legwork involved in doing the rewrite, so I
 wouldn't
  want to block 2.0. So I think the question is 2.x or 3.0. If we go the
 web
  machine route we should rework the api and definitely introduce backwards
  incompatible changes, so we would want to do a 3.0 release there. If we
  used cowboy we could mimic the current api and release 2.x.
 
   For me, I am not so interested in an internal rewrite with zero
 advantage
   (besides it's cleaner), however I am am very interested to use the
   rewrite for a better opportunity to explore plugin opportunities or
 other
   extensibility features.
  
  
 
  This rewrite needs to happen. The internals of the http layer need some
  serious love and there's a lot of duplication to remove. I think the big
  win here is that this would get the ball rolling on taking a closer look
 at
  the various internal applications and figuring out what needs to be
  restructured. For instance, the next logical step after reworking the
 http
  later is to standardize the clustered and local api modules, and there
 was
  some great discussion in the Dev channel today about that.
 
  The more we can decouple the various apps, the more easily we can extend
  CouchDB with plugins and new functionality.
 
  -Russell
 
  
  
   On Mon, Aug 18, 2014 at 1:41 AM, Russell Branca chewbra...@apache.org
 
   wrote:
  
# Rewriting the CouchDB HTTP Layer
   
With the light at the end of tunnel on the BigCouch merge, I thought
it was time to get the conversation going on cleaning up the current
HTTP stack duality. We've got a good opportunity to do some major
cleanup, remove duplication, and really start more clearly separating
the various components of CouchDB.
   
   
## Primary objectives
   
* Consolidate down to one HTTP layer
* Isolate HTTP functionality
* Separate HTTP server from HTTP resources
* Easy plugin integration
* Build clustered/local API
   
   
### Consolidate down to one HTTP layer
   
We currently have two HTTP layers, `couch_httpd` and `chttpd`. This
was a useful construct when BigCouch was a separate application where
isolating the clustered layer from the local layer was necessary, and
quite useful.
   
This is no longer the case, and we can significantly reduce code
duplication by consolidating down to one http layer. There are a
number of places in the two apps where the code is nearly identical,
except one calls out to `fabric` and the other calls out for
`couch_*`. For instance, compare `couch_httpd_db:couch_doc_open/4`
 [1]
with `chttpd_db:couch_doc_open/4` [2]. These are completely identical
aside from whether it goes through the clustered layer, `fabric`, or
through the local layer `couch_db`.
   
There are plenty of other places with similar duplication. This is
obviously ripe with opportunity to refactor and introduce some higher
level abstractions to make the HTTP layer function independently of
 the
document/database level APIs.
   
   
### Isolate HTTP functionality
   
I don't think `couch_doc_open/4` has any business existing in
the HTTP layer, we should move all non HTTP logic out. IMO the HTTP
layer should only concern itself with:
   
1. Receiving the HTTP requests
2. Extracting out the request data

Re: [DISCUSS] Rewriting the CouchDB HTTP Layer

[Russell Branca]

On Aug 17, 2014 8:15 PM, Jason Smith jason.h.sm...@gmail.com wrote:

 Hi, Russell. This is okay for a starting point but it is a bit vague.
Could
 you perhaps flesh out the plan and make it more comprehensive?

 ^^ That is a joke!

 Seriously, thank you very much for this analysis and plan. This is very
 exciting! (Not least because the http codebase is the part I know best and
 I can get excited about.)


Thanks!

 One quick question that I don't see from your writeup: What version of
 CouchDB are you thinking of targeting? 2.0? 2.1? 3.0? Is this completely
an
 internal change, or does it affect users?


I think this is outside the scope of 2.0 given how close we are on that.
There's a fair bit of legwork involved in doing the rewrite, so I wouldn't
want to block 2.0. So I think the question is 2.x or 3.0. If we go the web
machine route we should rework the api and definitely introduce backwards
incompatible changes, so we would want to do a 3.0 release there. If we
used cowboy we could mimic the current api and release 2.x.

 For me, I am not so interested in an internal rewrite with zero advantage
 (besides it's cleaner), however I am am very interested to use the
 rewrite for a better opportunity to explore plugin opportunities or other
 extensibility features.



This rewrite needs to happen. The internals of the http layer need some
serious love and there's a lot of duplication to remove. I think the big
win here is that this would get the ball rolling on taking a closer look at
the various internal applications and figuring out what needs to be
restructured. For instance, the next logical step after reworking the http
later is to standardize the clustered and local api modules, and there was
some great discussion in the Dev channel today about that.

The more we can decouple the various apps, the more easily we can extend
CouchDB with plugins and new functionality.

-Russell



 On Mon, Aug 18, 2014 at 1:41 AM, Russell Branca chewbra...@apache.org
 wrote:

  # Rewriting the CouchDB HTTP Layer
 
  With the light at the end of tunnel on the BigCouch merge, I thought
  it was time to get the conversation going on cleaning up the current
  HTTP stack duality. We've got a good opportunity to do some major
  cleanup, remove duplication, and really start more clearly separating
  the various components of CouchDB.
 
 
  ## Primary objectives
 
  * Consolidate down to one HTTP layer
  * Isolate HTTP functionality
  * Separate HTTP server from HTTP resources
  * Easy plugin integration
  * Build clustered/local API
 
 
  ### Consolidate down to one HTTP layer
 
  We currently have two HTTP layers, `couch_httpd` and `chttpd`. This
  was a useful construct when BigCouch was a separate application where
  isolating the clustered layer from the local layer was necessary, and
  quite useful.
 
  This is no longer the case, and we can significantly reduce code
  duplication by consolidating down to one http layer. There are a
  number of places in the two apps where the code is nearly identical,
  except one calls out to `fabric` and the other calls out for
  `couch_*`. For instance, compare `couch_httpd_db:couch_doc_open/4` [1]
  with `chttpd_db:couch_doc_open/4` [2]. These are completely identical
  aside from whether it goes through the clustered layer, `fabric`, or
  through the local layer `couch_db`.
 
  There are plenty of other places with similar duplication. This is
  obviously ripe with opportunity to refactor and introduce some higher
  level abstractions to make the HTTP layer function independently of the
  document/database level APIs.
 
 
  ### Isolate HTTP functionality
 
  I don't think `couch_doc_open/4` has any business existing in
  the HTTP layer, we should move all non HTTP logic out. IMO the HTTP
  layer should only concern itself with:
 
  1. Receiving the HTTP requests
  2. Extracting out the request data into a standard data structure
  3. Dispatch requests to the appropriate internal APIs
  4. Forward the response
 
  Anything that doesn't fit into those four steps should be ripped out
  and moved elsewhere. For instance, the primary logic for determining the
  database redundancy and shard values is done in `chttpd_db` [3]. I
  would greatly prefer to see this logic in a database API.
 
  The more we can isolate HTTP logic from database logic the
  better. Once they are fully decoupled, then the HTTP layer is merely
  one particular client interface on top of the core database. We also
  get all the benefits of isolation for testing and what not.
 
  Along these lines, I think we greatly overuse the #http{} record for
  passing around request data, and instead you extract the body, and
  then combine all of the user supplied headers and query string params
  into a standard options list. This we can we completely separate
  making database requests from the representation of the client
  request.
 
 
  ### Separate HTTP server from HTTP resources

[DISCUSS] Rewriting the CouchDB HTTP Layer

[Russell Branca]

# Rewriting the CouchDB HTTP Layer

With the light at the end of tunnel on the BigCouch merge, I thought
it was time to get the conversation going on cleaning up the current
HTTP stack duality. We've got a good opportunity to do some major
cleanup, remove duplication, and really start more clearly separating
the various components of CouchDB.


## Primary objectives

* Consolidate down to one HTTP layer
* Isolate HTTP functionality
* Separate HTTP server from HTTP resources
* Easy plugin integration
* Build clustered/local API


### Consolidate down to one HTTP layer

We currently have two HTTP layers, `couch_httpd` and `chttpd`. This
was a useful construct when BigCouch was a separate application where
isolating the clustered layer from the local layer was necessary, and
quite useful.

This is no longer the case, and we can significantly reduce code
duplication by consolidating down to one http layer. There are a
number of places in the two apps where the code is nearly identical,
except one calls out to `fabric` and the other calls out for
`couch_*`. For instance, compare `couch_httpd_db:couch_doc_open/4` [1]
with `chttpd_db:couch_doc_open/4` [2]. These are completely identical
aside from whether it goes through the clustered layer, `fabric`, or
through the local layer `couch_db`.

There are plenty of other places with similar duplication. This is
obviously ripe with opportunity to refactor and introduce some higher
level abstractions to make the HTTP layer function independently of the
document/database level APIs.


### Isolate HTTP functionality

I don't think `couch_doc_open/4` has any business existing in
the HTTP layer, we should move all non HTTP logic out. IMO the HTTP
layer should only concern itself with:

1. Receiving the HTTP requests
2. Extracting out the request data into a standard data structure
3. Dispatch requests to the appropriate internal APIs
4. Forward the response

Anything that doesn't fit into those four steps should be ripped out
and moved elsewhere. For instance, the primary logic for determining the
database redundancy and shard values is done in `chttpd_db` [3]. I
would greatly prefer to see this logic in a database API.

The more we can isolate HTTP logic from database logic the
better. Once they are fully decoupled, then the HTTP layer is merely
one particular client interface on top of the core database. We also
get all the benefits of isolation for testing and what not.

Along these lines, I think we greatly overuse the #http{} record for
passing around request data, and instead you extract the body, and
then combine all of the user supplied headers and query string params
into a standard options list. This we can we completely separate
making database requests from the representation of the client
request.


### Separate HTTP server from HTTP resources.

I think everything I've said so far is pretty clear cut in terms of
it's _the_ logical thing to do, but separating the HTTP server from
the HTTP endpoints is less clearly defined. However, we do have
precedence for this and there are a number of solid benefits.

First, let me explain what I mean here. There are two pieces to an
HTTP stack, first there's the core HTTP engine that handles receiving
and responding to requests and other things along those lines, and
second there's the places where you supply your business logic and
figure what content to send to the user.

CouchDB has a handful of places using this aproach, where instead of
defining all the logic in the HTTP stack directly, we have auxilary
modules defined within the appropriate applications that specify how
any HTTP requests for that application are handled. A good clean
example of this approach is `couch_mrview_http` [4].


### Easy plugin integration

One big advantage of the above separation of HTTP resources is that it
provides a standard way of plugins hooking in new HTTP endpoints. The
more we can treat the core CouchDB applications as plugins, the more
easily it is to isolate and replace various parts of the stack.


### Build clustered/local API

The above example of `couch_doc_open/4` is a clear cut case where
we want to abstract the process of loading a document. Not all places
are as easily abstractable, but this is a great example of why I think
we should have a standard API on top of clustered and local layers,
where deciding which to use is based on a local/clustered flag, or
some other heuristic.

I've been toying around with the idea of making a request object of
some sort, is something like `couch_req:make(ReqBody, ReqOptions)`
that you can then pass to `couch_doc_api` or some such, but I don't
have any strong opinions on this.


## Where I've gotten so far: chttpd2, a proof of concept

I've hacked out an experimental WebMachine [5] based rewrite of the
HTTP stack called `chttpd2` [6]. This PoC follows the same ideas I've
outlined above, so I'll run back through the previous outlined items
and explain 

Re: [DISCUSS] Chair rotation?

[Russell Branca]

Late to the party but I'm +1 to Jan's one year term just starting.


-Russell

On Mon, Aug 11, 2014 at 4:22 AM, Andy Wenk andyw...@apache.org wrote:

 cool - thanks Noah!
 On 11 August 2014 13:14, Noah Slater nsla...@apache.org wrote:
 It looks like we reached consensus here. Jan's term started the day we
 voted the bylaws in. I will set a reminder for 12 months. :)

 On 8 August 2014 19:39, Noah Slater nsla...@apache.org wrote:
  Bob, individuals are Officers of the Foundation for the duration they
  hold the position of Chair. You need to be an Officer (in American
  corporation law) to carry out your duties as Chair. Nothing any more
  complex than that.
 
  VP CouchDB is Jan's title as an Officer. Whoever is elected next will
  also get the title VP CouchDB, and will also become an Officer.
  Meanwhile, Jan will no longer be the Chair, the VP CouchDB, or an
  Officer.
 
  Do you think this needs clarification in the bylaws?
 
  On 7 August 2014 00:51, Robert Samuel Newson rnew...@apache.org wrote:
 
  The Chair is also and Officer of the ASF (
 https://www.apache.org/foundation/faq.html#why-are-PMC-chairs-officers),
 so are we only able to select an existing Officer or does being elected as
 Chair cause one to become an Officer? If so, do they remain an Officer if
 subsequently replaced by another?
 
  B.
 
  On 6 Aug 2014, at 23:29, Andy Wenk andyw...@apache.org wrote:
 
  seconding Paul. Was my first thought also. Starting the term now, so
 that
  we hold an election in 12 months.
 
  Cheers
 
  Andy
 
 
  On 7 August 2014 00:08, Paul Davis paul.joseph.da...@gmail.com
 wrote:
 
  I'd lean towards having the term start now instead of end now. I can't
  come up with a super convincing argument either direction though.
 
  On Wed, Aug 6, 2014 at 4:58 PM, Noah Slater nsla...@apache.org
 wrote:
  Hello folks,
 
  It's time we have a discussion about the position of Chair. Jan has
  held office for a while now, and has done a great job of it. But our
  new bylaws specify that we will hold yearly reelections.
 
  We have a number of options here:
 
  1. We say that Jan's position started on the day the bylaws were
  ratified, and in 12 months we hold an election.
 
  2. We say that Jan's position is up (having already held it for more
  than 12 months) and we either nominated a new Chair or we hold a
 vote.
 
  Let's try to get general consensus about which option we want to
 take.
  If there's clear agreement, we'll move forward with that option.
 
  Remember: our rationale for re-electing the Chair is that we think
  that doing so is beneficial for the project. Giving new people a
  chance to serve a term may stimulate activity and change, and so on.
  It also helps familiarise as many people as possible with all parts
 of
  the project.
 
  Jan has done a sterling job, and nobody can deny it. This discussion
  isn't about whether we need to replace him. It's only about when we
  set the end date of his current term. Immediately or 12 months in the
  future. :)
 
  Also, please, whatever you do, please do not reply to this email with
  a nomination. Such as: I think Monty would make a good Chair. Chair
  nomination itself will happen in private on the PMC list.
 
  Thanks,
 
  --
  Noah Slater
  https://twitter.com/nslater
 
 
 
 
  --
  Andy Wenk
  Hamburg - Germany
  RockIt!
 
  GPG fingerprint: C044 8322 9E12 1483 4FEC 9452 B65D 6BE3 9ED3 9588
 
  https://people.apache.org/keys/committer/andywenk.asc
 
 
 
 
  --
  Noah Slater
  https://twitter.com/nslater



 --
 Noah Slater
 https://twitter.com/nslater

 -- 
 Andy Wenk
 Hamburg - Germany
 RockIt!
 GPG fingerprint: C044 8322 9E12 1483 4FEC 9452 B65D 6BE3 9ED3 9588
  https://people.apache.org/keys/committer/andywenk.asc

[jira] [Resolved] (COUCHDB-1993) Make fabric work with the refactored view engine

[Russell Branca (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-1993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russell Branca resolved COUCHDB-1993.
-

   Resolution: Fixed
Fix Version/s: 2.0.0

This landed a while ago.

 Make fabric work with the refactored view engine
 

 Key: COUCHDB-1993
 URL: https://issues.apache.org/jira/browse/COUCHDB-1993
 Project: CouchDB
  Issue Type: Improvement
  Components: BigCouch
Reporter: Volker Mische
Assignee: Russell Branca
 Fix For: 2.0.0






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (COUCHDB-1997) Make fabric endpoints work

[Russell Branca (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russell Branca resolved COUCHDB-1997.
-

Resolution: Won't Fix

Not sure what this ticket is for. As far as I know fabric works with all the 
things. Feel free to reopen with more details if there's something else that 
needs to be addressed here.

 Make fabric endpoints work
 --

 Key: COUCHDB-1997
 URL: https://issues.apache.org/jira/browse/COUCHDB-1997
 Project: CouchDB
  Issue Type: Improvement
  Components: BigCouch
Reporter: Volker Mische





--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (COUCHDB-1996) Switch logging for BigCouch

[Russell Branca (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-1996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russell Branca resolved COUCHDB-1996.
-

   Resolution: Fixed
Fix Version/s: 2.0.0

There is now a lager backed couch_log application.

 Switch logging for BigCouch
 ---

 Key: COUCHDB-1996
 URL: https://issues.apache.org/jira/browse/COUCHDB-1996
 Project: CouchDB
  Issue Type: Improvement
  Components: BigCouch
Reporter: Volker Mische
 Fix For: 2.0.0


 The logging should be changed to either twig or lager



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (COUCHDB-2081) Support HTTPS in chttpd

[Russell Branca (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russell Branca resolved COUCHDB-2081.
-

   Resolution: Fixed
Fix Version/s: 2.0.0

This landed in chttpd.

 Support HTTPS in chttpd
 ---

 Key: COUCHDB-2081
 URL: https://issues.apache.org/jira/browse/COUCHDB-2081
 Project: CouchDB
  Issue Type: Improvement
  Security Level: public(Regular issues) 
  Components: BigCouch
Reporter: Paul Joseph Davis
  Labels: release
 Fix For: 2.0.0


 BigCouch relies on load balancers for SSL termination. We'll need to add the 
 options to chttpd directly.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (COUCHDB-1998) Scripts and docs to boot dev cluster

[Russell Branca (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-1998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russell Branca resolved COUCHDB-1998.
-

   Resolution: Fixed
Fix Version/s: 2.0.0

The dev/run script is functional, and also there is a dev/remsh script now.

 Scripts and docs to boot dev cluster
 

 Key: COUCHDB-1998
 URL: https://issues.apache.org/jira/browse/COUCHDB-1998
 Project: CouchDB
  Issue Type: Improvement
  Components: BigCouch
Reporter: Volker Mische
Assignee: Robert Newson
 Fix For: 2.0.0

  Time Spent: 96h
  Remaining Estimate: 0h





--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (COUCHDB-2006) Add https support to BigCouch

[Russell Branca (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-2006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russell Branca resolved COUCHDB-2006.
-

Resolution: Duplicate

 Add https support to BigCouch
 -

 Key: COUCHDB-2006
 URL: https://issues.apache.org/jira/browse/COUCHDB-2006
 Project: CouchDB
  Issue Type: Improvement
  Components: BigCouch
Reporter: Volker Mische





--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (COUCHDB-2090) Test .couch file compatibility

[Russell Branca (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065262#comment-14065262
 ] 

Russell Branca commented on COUCHDB-2090:
-

I've written things up in 
http://mail-archives.apache.org/mod_mbox/couchdb-dev/201406.mbox/browser with 
some sample .couch files and a script to test the things.

 Test .couch file compatibility
 --

 Key: COUCHDB-2090
 URL: https://issues.apache.org/jira/browse/COUCHDB-2090
 Project: CouchDB
  Issue Type: Improvement
  Security Level: public(Regular issues) 
  Components: BigCouch
Reporter: Paul Joseph Davis
  Labels: release

 There are a number of file format differences we need to test. Most of these 
 should be backwards compatible but its always possible an upgrade covered 
 issues historically.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (COUCHDB-2092) Allow _list functions to consume _all_docs

[Russell Branca (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russell Branca resolved COUCHDB-2092.
-

   Resolution: Fixed
Fix Version/s: 2.0.0

 Allow _list functions to consume _all_docs
 --

 Key: COUCHDB-2092
 URL: https://issues.apache.org/jira/browse/COUCHDB-2092
 Project: CouchDB
  Issue Type: Improvement
  Security Level: public(Regular issues) 
  Components: BigCouch
Reporter: Paul Joseph Davis
  Labels: merge
 Fix For: 2.0.0


 Avoid regressions



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (COUCHDB-2093) Provide _changes feeds using EventSource

[Russell Branca (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-2093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russell Branca resolved COUCHDB-2093.
-

   Resolution: Fixed
Fix Version/s: 2.0.0

 Provide _changes feeds using EventSource
 

 Key: COUCHDB-2093
 URL: https://issues.apache.org/jira/browse/COUCHDB-2093
 Project: CouchDB
  Issue Type: Improvement
  Security Level: public(Regular issues) 
  Components: BigCouch
Reporter: Paul Joseph Davis
  Labels: merge
 Fix For: 2.0.0


 Avoid regressions



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (COUCHDB-2094) Support Last-Event-ID header in EventSource _changes feeds

[Russell Branca (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-2094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russell Branca resolved COUCHDB-2094.
-

   Resolution: Fixed
Fix Version/s: 2.0.0

 Support Last-Event-ID header in EventSource _changes feeds
 --

 Key: COUCHDB-2094
 URL: https://issues.apache.org/jira/browse/COUCHDB-2094
 Project: CouchDB
  Issue Type: Improvement
  Security Level: public(Regular issues) 
  Components: BigCouch
Reporter: Paul Joseph Davis
  Labels: merge
 Fix For: 2.0.0


 Avoid regressions



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (COUCHDB-2098) Support efficient pagination of _all_dbs

[Russell Branca (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russell Branca resolved COUCHDB-2098.
-

   Resolution: Fixed
Fix Version/s: 2.0.0

Fixed with the move to couch_mrview.

 Support efficient pagination of _all_dbs
 

 Key: COUCHDB-2098
 URL: https://issues.apache.org/jira/browse/COUCHDB-2098
 Project: CouchDB
  Issue Type: Improvement
  Security Level: public(Regular issues) 
  Components: BigCouch
Reporter: Paul Joseph Davis
  Labels: release
 Fix For: 2.0.0


 Pull in some optimizations for paging _all_dbs in a cluster. Depends on 
 COUCHDB-1993



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (COUCHDB-2095) Ignore epilogues in multipart/related MIME attachments

[Russell Branca (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-2095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russell Branca resolved COUCHDB-2095.
-

   Resolution: Fixed
Fix Version/s: 2.0.0

 Ignore epilogues in multipart/related MIME attachments
 --

 Key: COUCHDB-2095
 URL: https://issues.apache.org/jira/browse/COUCHDB-2095
 Project: CouchDB
  Issue Type: Improvement
  Security Level: public(Regular issues) 
  Components: BigCouch
Reporter: Paul Joseph Davis
  Labels: merge
 Fix For: 2.0.0


 Avoid regressions



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Reopened] (COUCHDB-1998) Scripts and docs to boot dev cluster

[Russell Branca (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-1998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russell Branca reopened COUCHDB-1998:
-


Needs docs as described in the title.

 Scripts and docs to boot dev cluster
 

 Key: COUCHDB-1998
 URL: https://issues.apache.org/jira/browse/COUCHDB-1998
 Project: CouchDB
  Issue Type: Improvement
  Components: BigCouch
Reporter: Volker Mische
Assignee: Robert Newson
 Fix For: 2.0.0

  Time Spent: 96h
  Remaining Estimate: 0h





--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: CouchDB 2.0: breaking the backward compatibility

[Russell Branca]

I would also love to see _rev renamed, and I think it's a good
opportunity to flip around all the meta info as well. I'm still
partial to moving the relevant metadata into the headers, and no
longer including any _* fields in the doc, but I know there are
proponents on both sides of the coin there. The most recent proposal I
could find is to move all the metadata into a '_' field [1]. In 2.0 I
would like to see us move all metadata into headers or into the '_'
field, and rename 'rev'. There's a lot of code overlap for the two so
it seems like an opportune time to do it.

I wonder if it's reasonable to make the use of a '_' field or exposed
through headers configurable. I'm not sure it's a great idea to do so,
but it's at least worth thinking about.

Exposing conflicts by default is another thing I'm keen on. The
question is how to make it fail loudly so that client libraries
don't just think it's the document body. An aggressive approach send a
list of conflict revs rather than a doc object which will break all
existing parsers and require users to deal with. Then if you want a
particular rev, you'll need to specify it in the request.

We could also cleanup the API endpoints to make them more RESTful. IMO
things like _all_dbs and _all_docs should be the top level endpoints
and the current info endpoints moved to _info or some such.

Along the lines of API cleanup is the capabilities engine. I think
this would be a great thing to land, and if done properly could be a
self defining REST endpoint showing all the things the server is
capable of and how to reach them.



-Russell


[1] 
http://mail-archives.apache.org/mod_mbox/couchdb-dev/201312.mbox/%3c529de44c.4090...@bigbluehat.com%3E

On Thu, Jul 17, 2014 at 2:14 AM, Robert Samuel Newson
rnew...@apache.org wrote:
 Great point, +1 to just making that change on master right now.

 B.

 On 16 Jul 2014, at 22:35, Robert Kowalski r...@kowalski.gd wrote:

 I would like to see 'JSONP responses should be sent with a
 application/javascript' (https://github.com/apache/couchdb/pull/236)
 beside the two merges in the 2.0 release - it is a small, but breaking
 change and the original issue is flying around in Jira for years.

 Best,
 Robert


 2014-07-13 22:17 GMT+02:00 Robert Samuel Newson rnew...@apache.org:


 Since we follow semantic versioning, the only meaning behind naming our
 next release 2.0 and not 1.7 is that it contains backwards incompatible
 changes.

 It’s for the CouchDB community as a whole to determine what is and isn’t
 in a release. Certainly merging in bigcouch and rcouch are a huge part of
 the 2.0 release, but they aren’t necessarily the only things. If they
 hadn’t changed the API in incompatible ways, they wouldn’t cause a major
 version bump.

 With that said then, I’m interested in hearing what else, besides the two
 merges, we feel we want to take on in our first major revision bump in
 approximately forever? At minimum, I would like to see a change that allows
 us to use versions of spidermonkey released after 1.8.5, whatever that
 change might be.

 B.

 On 13 Jul 2014, at 20:31, Joan Touzet woh...@apache.org wrote:

 Improving the view server protocol is a great idea, but it is appropriate
 for a 2.0 timeframe? I would think it would make more sense in a 3.0
 timeframe, given 2.0 is all about merging forks, not writing new features
 entirely from scratch.

 -Joan

 - Original Message -
 From: Robert Samuel Newson rnew...@apache.org
 To: dev@couchdb.apache.org
 Sent: Sunday, July 13, 2014 8:52:40 AM
 Subject: Re: CouchDB 2.0: breaking the backward compatibility


 Adding mvcc for _security is a great idea (happily, Cloudant have done
 so very recently, so I will be pulling that work over soon).

 A better view server protocol is also a great idea.


 On 13 Jul 2014, at 13:13, Samuel Williams 
 space.ship.travel...@gmail.com wrote:


 On 13/07/14 23:47, Alexander Shorin wrote:
 Our view server is compiles functions on each view index update
 instead of reusing inner cache. This is because of out-dated protocol:
 others design function are works differently from views. While it's
 good to change and improve query server protocol completely, this task
 requires more time to be done. We should have a least plan B to do
 small steps in good direction.
 As already suggested, here is my proposal for 2.0 view/query server:


 https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit

 I welcome people to suggest improvements/changes/ideas.

 Kind regards,
 Samuel

[jira] [Resolved] (COUCHDB-1960) Provide pagination for /_all_dbs

[Russell Branca (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russell Branca resolved COUCHDB-1960.
-

   Resolution: Fixed
Fix Version/s: 2.0.0

This is fixed for clustered interfaces. IMO the correct fix for the local 
interfaces is to switch to using the dbs db for all interfaces, then we can 
port this logic for the local interface.

 Provide pagination for /_all_dbs
 

 Key: COUCHDB-1960
 URL: https://issues.apache.org/jira/browse/COUCHDB-1960
 Project: CouchDB
  Issue Type: Improvement
  Components: Fauxton, HTTP Interface
Reporter: Benjamin Young
 Fix For: 2.0.0


 The database-per-user design pattern has increased in popularity and 
 consequently causes long lists of databases when one hits {{/_all_dbs}}
 To keep this sane, we need some pagination.
 {{?skip=10limit=10}} style seems to be the simplest to implement (and there 
 should be existing code from Cloudant for this one).
 Thanks!



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: rcouch merge: integration with the bigcouch branch questions

[Russell Branca]

On Mon, Jul 7, 2014 at 8:50 PM, Benoit Chesneau bchesn...@gmail.com wrote:
 On Mon, Jul 7, 2014 at 11:44 PM, Russell Branca chewbra...@apache.org wrote:
 I switched to using chttpd in couch_mrview for delayed responses as it
 has more robust error handling logic (in particular it fixes a nasty
 bug where error messages were sent as full responses with headers in
 the stream) .

 Benoit, I don't understand what you mean by Which makes this
 functions unsable with the standalone HTTP api. The change in
 couch_mrview to use chttpd should _not_ prevent the single node
 interface from operating, if you're encountering that then it's a bug.


 It was badly phrased. I meant that apparently this change implies now
 that I need to use chttpd for the view queries. I just had a quick
 glance in the delayed response and not tested it, but doesn't it imply
 to handle the request via a chttpd controller?

 https://github.com/apache/couchdb-chttpd/blob/1843-feature-bigcouch/src/chttpd.erl#L601-L607

 Looks like it is a special response record. How the view queries are
 now working on the couch_httpd api?


The record is opaque to the view stream callbacks. This just uses
chttpd for sending delayed chunks, because it has the better
implementation.


 IMO the logical next step for all this is to refactor the separate
 httpd layers into a single app and address the issues there, but I
 think that can wait for 2.x.

 Also, for port the couch_mrview http modules to chttpd for the layer
 part *and* couch_httpd I don't think this is the correct approach. I
 think we should define the core http layer in c(ouch_)httpd and then
 continue to use the pattern of the separate applications defining the
 relevant http bits.

 I've already done a lot of legwork to make the http view layer usable
 in both couch_httpd and chttpd with the long term game plan of
 consolidating down to only one http layer. For instance, the clustered
 view logic uses the existing couch_mrview_http code for actually
 sending the responses:
 https://github.com/apache/couchdb-chttpd/blob/master/src/chttpd_view.erl#L57.

 Compare these two functions:

 https://github.com/apache/couchdb-chttpd/blob/master/src/chttpd_view.erl#L49-L62
 https://github.com/apache/couchdb-couch-mrview/blob/master/src/couch_mrview_http.erl#L191-L204

 You can see they're almost identical, aside from
 `couch_mrview:query_view` vs `fabric:query_view`. I wanted to use the
 view layer as an example of removing unnecessary duplication in the
 two http layers, and to jump start the process of consolidation.
 Hopefully that gives a good idea on approach and the parts that we
 actually need to abstract.

 Maybe. I would have done the changes only in chttpd imo since they are
 about a new features that won't be ship in 1.7. Especially since I
 understand it some wants chttpd and couch_httpd may be merged in a way
 or another. Keeping them separated until then would not impact any
 future direction about it.


If we cut 1.7 IMO it should be off couchdb master with none of the
merge. As far as the merge, it's including chttpd either way.

 I implemented  the feature using couch_httpd this week-end to support
 the feature on the current version:

 https://github.com/rcouch/couchdb-couch-mrview/commit/4e0f29577c21fb6249fecbfe9f20a46b90f6ae3b

 The relevant part of it, s is that I didn't have to change the
 couch_httpd application to support multi queries in view changes for
 the _changes handler since this handler was cleanly separated from the
 rest.

 For the 2.0 I think we should consider to completely make the http api
 available as a *possible* transport and indeed only have one
 application. but until then it worth to keep changes on it separated.
 It would ease any work on that part later.


I'm a strong proponent for combining the http layers and then
drastically decreasing the functionality to only handle the http side
of things, and call out to the appropriate application for the
database level logic. Believe me, I'm well aware of the overly coupled
semantics of the current http stack, both in chttpd and couch_httpd.
For instance the default clustering values for db creation are set in
chttpd: 
https://github.com/apache/couchdb-chttpd/blob/master/src/chttpd_db.erl#L201-L216.
This obviously is not ideal and most definitely needs to change, but
it gets the job done for now. I personally would not block a 2.0
release to wait on a month or two of work to refactor the http layer.
It's an optimization for code complexity (albeit a good one), not a
feature relevant to people wanting to use BigCouch.


 - benoit


-Russell

Re: info: quickcheck free for opensource projects

[Russell Branca]

Agreed, we should get this going as soon as we can. How does the
quickcheck CI server work with standard eunit tests? Do we need to do
something to make sure only quickcheck tests work? Or do we just run
all the tests?


-Russell

On Mon, Jun 30, 2014 at 1:22 AM, Benoit Chesneau bchesn...@gmail.com wrote:
 cool. Not sure what could be the roadmap on this. Like I see it maybe
 the first thing to do is to integrate  the changes on the test suite
 from Alexander. Then we can probably add quickcheck test on each each
 module using the macros to define if they are executed or not?


 - benoit

 On Wed, Jun 11, 2014 at 7:41 PM, Jan Lehnardt j...@apache.org wrote:

 On 11 Jun 2014, at 19:39 , Russell Branca chewbra...@apache.org wrote:

 I'm a huge +1 to this.

 I've been trying to figure out a way to get us able to use the full version
 of QuickCheck for a while now. John Hughes has been hinting that they found
 a way to make the licensing work for open source, and it seems like this is
 it.

 The full version of QuickCheck has some sweet features for testing out
 state machines and also the PULSE scheduler which randomizes the execution
 of processes to help discover race conditions:
 http://www.quviq.com/features.html

 To clarify the questions about another CI server, I believe the reason
 for this being released as a CI server is as a way to use the full version
 of QuickCheck without them having to distribute it.

 Ah, apologies for missing that particular context.

 I’m still +100 on this :)

 Best
 Jan
 --



 -Russell


 On Wed, Jun 11, 2014 at 4:13 AM, Jan Lehnardt j...@apache.org wrote:

 QC is not a CI tool. It’s more like an additional layer of more thorough
 unit testing that could (depending on their terms) run by our existing CI
 solutions.

 I’d be in favour of looking at how we can make it work!

 Best
 Jan
 --

 On 11 Jun 2014, at 13:06 , Dirkjan Ochtman dirk...@ochtman.nl wrote:

 n Wed, Jun 11, 2014 at 1:00 PM, Benoit Chesneau bchesn...@gmail.com
 wrote:
 quickcheck made quickcheck-ci available for free for open-sources
 projects:

 http://quickcheck-ci.com/

 It would be interresting to use it for couchdb imo. Thoughts?

 If we still use Travis, we already have 2 CI instances, and they have
 not been able to prevent drawn out release processes like the one for
 1.6.0. Unless we somehow think this will magically solve all our CI
 needs, I'd prefer to instead spend time on improving other parts of
 the CI we already have.

 Cheers,

 Dirkjan

[jira] [Resolved] (COUCHDB-2097) Avoid performance regression with a single fd

[Russell Branca (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-2097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russell Branca resolved COUCHDB-2097.
-

Resolution: Fixed

Resolving this as the performance changes are within tolerable limits. We can 
address specific performance issues that come up individually, but in general 
this looks acceptable.

 Avoid performance regression with a single fd
 -

 Key: COUCHDB-2097
 URL: https://issues.apache.org/jira/browse/COUCHDB-2097
 Project: CouchDB
  Issue Type: Improvement
  Security Level: public(Regular issues) 
  Components: BigCouch
Reporter: Paul Joseph Davis
  Labels: release
 Attachments: mike-wallace-fd-benchmarks.txt


 Part of one of our large enhancements required that we remove a CouchDB 
 performance optimization on having two file handles to each .couch file. We 
 need to make sure that this doesn't negatively impact performance.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: [ANNOUNCE] Lena Reinhard elected as CouchDB committer

[Russell Branca]

Congrats Lena!


-Russell


On Sat, Jun 21, 2014 at 3:00 PM, Lena Reinhard l...@thehoodiefirm.com
wrote:

 Thanks, everyone, for the warm welcome! I'm happy about the election and
 very much looking forward to continuing working on CouchDB together with
 everyone of you!

 All the best

 Lena

  On 21.06.2014, at 21:27, Jan Lehnardt j...@apache.org wrote:
 
  Welcome Lena, glad to have you on board! :)
 
  Best
  Jan
  --
 
  On 21 Jun 2014, at 16:59 , Noah Slater nsla...@apache.org wrote:
 
  Dear community,
 
  I am pleased to announce that the CouchDB Project Management Committee
  has elected Lena Reinhard as a CouchDB committer.
 
Apache ID: lena
 
IRC nick: ux
 
Twitter: https://twitter.com/ux
 
  By default, external contributions to the project follow the
  Review-Then-Commit model. Being a committer means you can follow the
  Commit-Then-Review model. In other words, Lena can now make changes at
  will.
 
  This election was made in recognition of Lena's existing contributions
  and commitment to the project.
 
  Please join me in extending a warm welcome to Lena!
 
  On behalf of the CouchDB PMC,
 
  --
  Noah Slater
  https://twitter.com/nslater
 

Re: mutli view query

[Russell Branca]

Hi Jean,

The top level field for multi view queries is queries, try that instead
of keys. Let me know if you're still having issues.

-Russell
On Jun 12, 2014 6:21 AM, Jean-Felix Girard jeanfel...@icloud.com wrote:

 Hi,

 One feature I look forward and want to test is the possibility to query a
 view with multiple ranges (startkey + endkey) -
 https://issues.apache.org/jira/browse/COUCHDB-523.  I saw a commit to
 address it :
 https://github.com/apache/couchdb-couch-mrview/commit/3688736b81c8b8b6485cb136eea836bd729d152f.
  From my understanding, the bigcouch merge branch uses
 couchdb-couch-mrview, master which include the commit.  Is it really the
 case ?

 I tried but could not make it work.

 For instance:

 curl 'localhost:15984/test/_all_docs' -X POST -d
 '{keys:[{startkey:a,endkey:z}]}' -H
 'content-type:application/json' -0


 {total_rows:5,rows:[
 HTTP/1.0 500 Internal Server Error
 X-CouchDB-Body-Time: 0
 X-Couch-Request-ID: b1ad2ba2
 Server: CouchDB/c9a3fc1 (Erlang OTP/R16B03-1)
 Date: Thu, 12 Jun 2014 13:15:09 GMT
 Content-Type: text/plain; charset=utf-8
 Content-Length: 242
 Cache-Control: must-revalidate

 {error:badmatch,reason:timeout,stack:[fabric_view_all_docs:go/5
 L76,couch_httpd:etag_maybe/2 L592,chttpd_db:all_docs_view/3
 L512,chttpd:handle_request/1 L206,mochiweb_http:headers/5
 L93,proc_lib:init_p_do_apply/3 L239]}


 Any pointer would help!

 Thanks.
 --
 J

Re: info: quickcheck free for opensource projects

[Russell Branca]

I'm a huge +1 to this.

I've been trying to figure out a way to get us able to use the full version
of QuickCheck for a while now. John Hughes has been hinting that they found
a way to make the licensing work for open source, and it seems like this is
it.

The full version of QuickCheck has some sweet features for testing out
state machines and also the PULSE scheduler which randomizes the execution
of processes to help discover race conditions:
http://www.quviq.com/features.html

To clarify the questions about another CI server, I believe the reason
for this being released as a CI server is as a way to use the full version
of QuickCheck without them having to distribute it.


-Russell


On Wed, Jun 11, 2014 at 4:13 AM, Jan Lehnardt j...@apache.org wrote:

 QC is not a CI tool. It’s more like an additional layer of more thorough
 unit testing that could (depending on their terms) run by our existing CI
 solutions.

 I’d be in favour of looking at how we can make it work!

 Best
 Jan
 --

 On 11 Jun 2014, at 13:06 , Dirkjan Ochtman dirk...@ochtman.nl wrote:

  n Wed, Jun 11, 2014 at 1:00 PM, Benoit Chesneau bchesn...@gmail.com
 wrote:
  quickcheck made quickcheck-ci available for free for open-sources
 projects:
 
  http://quickcheck-ci.com/
 
  It would be interresting to use it for couchdb imo. Thoughts?
 
  If we still use Travis, we already have 2 CI instances, and they have
  not been able to prevent drawn out release processes like the one for
  1.6.0. Unless we somehow think this will magically solve all our CI
  needs, I'd prefer to instead spend time on improving other parts of
  the CI we already have.
 
  Cheers,
 
  Dirkjan

Re: Installing 1843-feature-bigcouch

[Russell Branca]

Thanks for the report Andy. I'm not sure how far Newson has tested doing a
`make install`, I know personally I've only been running with the local dev
server for now.

The crash error in your log with mem3_shards:fold failing on a bad match is
an interesting one, and looks like an actual bug.

As for Futon, I think the general consensus is to drop it entirely for 2.0
and switch over to Fauxton.

Andy, thanks for starting to test single node CouchDB! It's definitely
important to make sure there aren't any regressions there and that things
still work as expected.


-Russell


On Sun, May 25, 2014 at 9:26 AM, Andy Wenk a...@nms.de wrote:

 Hi Bob,

 I cloned a fresh couch repo and ran

 $ ./configure
 $ make

 Here are some warnings:

 Compiling priv/icu_driver/couch_icu_driver.c
 priv/icu_driver/couch_icu_driver.c:171:9: warning: incompatible pointer
 types initializing 'ErlDrvSSizeT (*)(ErlDrvData, unsigned int, char *,
 ErlDrvSizeT, char **, ErlDrvSizeT)' with an expression of type
 'COUCH_SSIZET (ErlDrvData, unsigned int, char *, COUCH_SSIZET, char **,
 COUCH_SSIZET)' [-Wincompatible-pointer-types]
 couch_drv_control,  /* F_PTR control, port_command callback */
 ^
 1 warning generated.

 Compiling c_src/snappy/snappy.cc
 c_src/snappy/snappy.cc:516:13: warning: unused function 'ComputeTable'
 [-Wunused-function]
 static void ComputeTable() {
 ^
 1 warning generated.

 When running

 $ make test

 I just received:

 == couchdb-1843-feature-bigcouch-3 (eunit)

 When running

 $ make install

 I received

 WARN:  'generate' command does not apply to directory
 /Users/andwen/project/couchdb/couchdb-1843-feature-bigcouch-3
 mkdir: /opt/couchdb: Permission denied
 make: *** [install] Error 1

 So the target is /opt/couchdb ... ? That's new ...

 With sudo it works for sure. I received this warning:

 WARN:  'generate' command does not apply to directory
 /Users/andwen/project/couchdb/couchdb-1843-feature-bigcouch-3

 When starting couchdb with

 $ sudo /opt/couchdb/bin/couchdb

 I received

  sudo /opt/couchdb/bin/couchdb⏎ ◼
 Apache CouchDB 6dcd33b is starting.
 18:22:00.869 [info] Application lager started on node
 'couchdb@andwen-2.local'
 18:22:00.869 [info] Application couch_log started on node
 'couchdb@andwen-2.local'
 18:22:00.875 [info] Starting couch_sup
 18:22:00.927 [notice] config: [couchdb] uuid set to
 b8bc739cea165a5795e4adca8be5f894 for reason nil
 18:22:00.963 [info] open_result error {not_found,no_db_file} for _users
 18:22:00.982 [info] needed 18.134 ms to open new _users
 18:22:01.030 [info] open_result error {not_found,no_db_file} for
 _replicator
 18:22:01.037 [info] needed 6.67 ms to open new _replicator
 Apache CouchDB has started. Time to relax.
 18:22:01.050 [info] Apache CouchDB has started on http://127.0.0.1:5986/
 18:22:01.050 [info] Application couch started on node
 'couchdb@andwen-2.local'
 18:22:01.064 [info] Application rexi started on node
 'couchdb@andwen-2.local
 '
 18:22:01.075 [error] Could not open file /opt/couchdb/var/lib/nodes.couch:
 no such file or directory
 18:22:01.075 [info] open_result error {not_found,no_db_file} for nodes
 18:22:01.081 [info] needed 5.419 ms to open new nodes
 18:22:01.100 [error] Could not open file /opt/couchdb/var/lib/dbs.couch: no
 such file or directory
 18:22:01.100 [info] Application mem3 started on node
 'couchdb@andwen-2.local
 '
 18:22:01.100 [info] open_result error {not_found,no_db_file} for dbs
 18:22:01.100 [info] Application fabric started on node
 'couchdb@andwen-2.local'
 18:22:01.101 [error] Error in process 0.199.0 on node
 'couchdb@andwen-2.local' with exit value:

 {{badmatch,file_exists},[{mem3_shards,fold,2,[{file,src/mem3_shards.erl},{line,91}]},{mem3_sync,initial_sync,1,[{file,src/mem3_sync.erl},{line,242}]}]}


 18:22:01.106 [info] needed 5.706 ms to open new dbs
 18:22:01.135 [info] Application chttpd started on node
 'couchdb@andwen-2.local'
 18:22:01.135 [info] Application couch_replicator started on node
 'couchdb@andwen-2.local'
 18:22:01.135 [info] Application ets_lru started on node
 'couchdb@andwen-2.local'
 18:22:01.153 [info] Application ddoc_cache started on node
 'couchdb@andwen-2.local'
 18:22:01.177 [info] Application runtime_tools started on node
 'couchdb@andwen-2.local'
 18:22:01.177 [info] Application snappy started on node
 'couchdb@andwen-2.local'

 So looking at localhost:5986 I received

 {

- couchdb: Welcome,
- uuid: b8bc739cea165a5795e4adca8be5f894,
- version: 6dcd33b

 }

 And localhost:5986/_all_dbs/

 [

- _replicator,
- _users,
- dbs,
- nodes

 ]

 Futon is not working at localhost:5986/_utils/

 So far ...

 Cheers

 Andy

 --
 Andy Wenk
 Hamburg - Germany
 RockIt!

 http://www.couchdb-buch.de
 http://www.pg-praxisbuch.de

 GPG fingerprint: C044 8322 9E12 1483 4FEC 9452 B65D 6BE3 9ED3 9588

 https://people.apache.org/keys/committer/andywenk.asc

On the Viability of Erlang Releases and CouchDB

[Russell Branca]

# On the Viability of Erlang Releases and CouchDB

There has been some discussion on what versions of Erlang CouchDB
should support, and what versions of Erlang are detrimental to
use. Sadly there were some pretty substantial problems in the R15 line
and even parts of R16 that are landmines for CouchDB. This post will
describe the current state of things and make some potential
recommendations on approach.

## Scheduler Collapse

It was discovered by Basho that R15* and R16B are susceptible to
scheduler collapse. There's quite a bit of discussion and
information in several threads [1] [2] [3] [4] [5].

So what is scheduler collapse? Erlang schedulers can be put to sleep
when there is not sufficient work to occupy all schedulers, which
saves on CPU and power consumption. When the schedulers that are still
running go through enough reductions to pass the work balancing
threshold, they can trigger a rebalance of work that will wake up
sleeping schedulers. The other mechanism for sharing scheduler load is
work stealing. A scheduler that does not have any work to do can
steal work from other schedulers. However a scheduler that has gone
to sleep cannot steal work, it has to be woken up separately.

Now the real problem of scheduler collapse occurs when you take
sleeping schedulers and long running NIFs and BIFs that do not report
an appropriate amount of reductions. When you have NIFs and BIFs that
don't report an appropriate amount of reductions, you can get into a
situation where a long running function call will only show up as
taking one reduction, and never hit the work balance threshold,
causing that scheduler to be blocked during the operation and no
additional schedulers getting woken up.

I keep mentioning NIFs and BIFs because it's important to note that
it is _not_ just user defined NIFs that are problematic, but also a
number of Erlang BIFs that don't properly report reductions.
Particularly relevant to CouchDB are the BIFs `term_to_binary` and
`binary_to_term` which do _not_ behave properly, and each report a
single reduction count, regardless of the size of the value passed to
them. Given that every write CouchDB makes goes through
`term_to_binary`, this is definitely not good.

This problem is systemic to all versions of R15 and R16B. In R16B01,
two changes were made to alleviate the problem. First, in OTP-11163
`term_to_binary` now uses an appropriate amount of reductions and will
yield back to the scheduler. The second important change was the
introduction of the `+sfwi` (Scheduler Forced Wakeup Interval) flag
[6] which allows you to specify a time interval for a new watchdog
process to check scheduler run queues and wake up sleeping schedulers
if need be. These two changes help significantly, although from what I
understand, they do not fully eliminate scheduler collapse.

*NOTE*: the `+sfwi` is _not_ enabled by default, you must specify a
greater than zero time interval to enable this. *WE NEED TO ENABLE
THIS SETTING.* We should figure out a way to conditionally add this
to vm.args or some such.

On a side note, Basho runs R15B01 because they backported the `+sfwi`
feature to R15B01 [7] [8]. They recommend running with `+sfwi 500` for
a 500ms interval. It might be worth testing out different values, but
500 seems like a good starting point. For Riak 2.0, they will be
building against R16B03-1 and 17.0 as their set of patches to R16B02
landed in R16B03-1 [9] [10].

## R16B01 and the breaking of monitors

So R16B01 sorted out the scheduler collapse issues, but unfortunately
it also broke monitors, which immediately disqualifies this release as
something we should recommend to users. The issues was fixed in
OTP-11225 in R16B02.

## R16B02 and R16B03*

I don't know of any catastrophic problems on the order of those
described above in either of these releases. Basho fixed a number of
unrelated bugs in R16B02 [9] [10] that have since landed in R16B03-1,
which indicates we should probably prefer R16B03-1 over R16B02. R16B03
is also disqualified because it broke SSL and `erl_syntax`, resulting
in the patched R16B03-1.

## R14

R14B01, R14B03, and R14B04 are known good stable releases of Erlang,
and in my opinion the only known stable releases  R13 that don't
present issues for CouchDB (I think R16B02/R16B03-1 are too new to
declare stable yet). As for R14B02, there are some bad `ets` issues
with that release.

It's worth pointing out that there are two known bugs in R14B01, as
Robert Newson explains:

```
There are two bugs in R14B01 that we do encounter, however. 1) Another
32/64 bit oops causes the vm to attempt to allocate huge amounts of
ram (terabytes, or more) if it ever tries to allocate more than 2gib
of ram at once. When this happens, the vm dies and is restarted. It’s
annoying, but infrequent. 2) Sometimes when closing a file, the
underlying file descriptor is *not* closed, though the erlang process
exits. This is rare but still quite annoying.
```


# Recommendations for CouchDB

In my 

Re: [DISCUSS] Apache CouchDB Diversity Statement.

[Russell Branca]

+1


On Thu, May 1, 2014 at 10:42 AM, Jan Lehnardt j...@apache.org wrote:

 +1

 On 29 Apr 2014, at 11:23 , Robert Kowalski r...@kowalski.gd wrote:

  +1
 
 
  2014-04-29 10:07 GMT+02:00 Garren Smith gar...@apache.org:
 
  +1 This is a great idea.
 
  On 28 Apr 2014, at 11:34 PM, Paul Davis paul.joseph.da...@gmail.com
  wrote:
 
  +1, especially to broadening the definition of contributions that we
  recognize.
 
  On Mon, Apr 28, 2014 at 3:59 PM, Andy Wenk a...@nms.de wrote:
  two docs make sense imho. +1
 
  Cheers
 
  Andy
 
 
  On 28 April 2014 22:44, Robert Samuel Newson rnew...@apache.org
  wrote:
 
  Count me in.
 
  B.
 
  On 28 Apr 2014, at 21:09, Noah Slater nsla...@apache.org wrote:
 
  Yep. Happy to see what the drafts look like. If two docs make sense,
  two docs make sense.
 
  On 28 April 2014 22:04, Joan Touzet jo...@atypical.net wrote:
  Nice. I personally would still like to see Diversity broken out in
 a
  separate document, but the idea of a charter is very good and this
  covers a
  lot of ground.
 
  This also dovetails very nicely with the mentorship ideas I have
  floated here and on IRC in the past.
 
  +1, let's do this!
 
  - Original Message -
  From: Noah Slater nsla...@apache.org
  To: dev@couchdb.apache.org
  Sent: Monday, April 28, 2014 3:50:15 PM
  Subject: Re: [DISCUSS] Apache CouchDB Diversity Statement.
 
  I have two things I'd like to add.
 
  1. I'd like to widen this bit and turn it into a PMC charter.
  Basically a document that sets out what the PMC values, our
  commitment
  to diversity, and our pledge to the project. It's basically the
 same
  idea, but with a flourishes.
 
  The closest we have to this at the moment is here:
 
 
 
 
 https://git-wip-us.apache.org/repos/asf?p=couchdb-admin.git;a=blob_plain;f=email/invite_pmc.txt;hb=HEAD
 
  I'd like to improve on this.
 
  2. Technical ability is something I feel very strongly about. We've
  made a huge blunder IMO in requiring past contributors to be highly
  technical. I would like to introduce and officialise the same
 policy
  as Apache Forrest.
 
  cf. https://forrest.apache.org/committed.html
 
  Summary: we recognise merit in four areas:
 
  (Co)mmunity - one must interact with others, and share vision and
  knowledge
  (P)roject - a clear vision and consensus are needed
  (Do)cumentation - without it, the stuff remains only in the minds
 of
  the authors
  (C)ode - discussion goes nowhere without code
 
  I would like to think that we would elect someone who is blogging
 for
  us, or doing marketing, or speaking a lot. These are all important
  people, and their importance to the project should be recognised.
 
  It's been about two years since I started pushing on this topic at
  the
  PMC level and there have been big cultural changes. I am confident
  that I could elect people with contributions in any of these areas.
  But I want to make it official.
 
  This is what we value. This is what we will recognise. Our promise
 to
  the community.
 
 
 
  --
  Noah Slater
  https://twitter.com/nslater
 
 
 
  --
  Noah Slater
  https://twitter.com/nslater
 
 
 
 
  --
  Andy Wenk
  Hamburg - Germany
  RockIt!
 
  http://www.couchdb-buch.de
  http://www.pg-praxisbuch.de
 
  GPG fingerprint: C044 8322 9E12 1483 4FEC 9452 B65D 6BE3 9ED3 9588
 
  https://people.apache.org/keys/committer/andywenk.asc
 
 

Re: [ANNOUNCE] Joan Touzet joins the PMC

[Russell Branca]

Congrats Joan!


-Russell


On Thu, Apr 10, 2014 at 10:55 AM, Noah Slater nsla...@apache.org wrote:

 Oh man, that made me nostalgic. I miss Monty. And I'm SO happy you're
 joining the PMC. Welcome aboard Joan. :)

 On 10 April 2014 18:30, Joan Touzet jo...@atypical.net wrote:
  Noah and the Apache CouchDB PMC, thank you for nominating me to join
  your number.
 
  Boy, how times have changed! In May 2009, I was a month or so into PhD
  research. I popped on IRC to ask my first CouchDB question - why a view
  was never returning - and stumbled upon a way to get invalid UTF-8 data
  into Couch, but not back out. Jan, Damien, Monty and Paul helped me out,
  though Paul was occasionally AFK looking at new apartment listings, and
  of course Monty was barking mad.
 
  People were friendly, engaged, and genuinely interested in the bug that
  became COUCHDB-345. It took 4 months to fix, and ended up with a patch
  set upstream to mochiweb as well.
 
  Since then I've been a non-stop user of CouchDB, including incorporating
  it into a p2p secure asset sharing system, an artefact repository for
  software release  deployment, as a backend for some top 10 mobile phone
  games and of course work on CouchDB, bigcouch and Cloudant themselves.
 
  I remain convinced that CouchDB is the sleeper NoSQL tech that's winning
  in the long run.
 
  We've got a lot of bugs to squash still, 2 big merges to get done and a
  whole lot of great requests and suggestions from the community at large.
  I look forward to doing everything I can to facilitate that work, and a
  few of my own patches as well.
 
  Rock on,
  Joan
 
 
  - Original Message -
  From: Noah Slater nsla...@apache.org
  To: dev@couchdb.apache.org
  Sent: Thursday, April 10, 2014 4:57:24 AM
  Subject: [ANNOUNCE] Joan Touzet joins the PMC
 
  Dear community,
 
  I am delighted to announce that Joan Touzet joins the Apache CouchDB
  Project Management Committee today.
 
  Joan has made outstanding, sustained contributions to the project.
  This appointment is an official acknowledgement of their position
  within the community, and our trust in their ability to provide
  oversight for the project.
 
  Everybody, please join me in congratulating Joan!
 
  On behalf of the CouchDB PMC,
 
  --
  Noah Slater
  https://twitter.com/nslater



 --
 Noah Slater
 https://twitter.com/nslater

[jira] [Commented] (COUCHDB-1993) Make fabric work with the refactored view engine

[Russell Branca (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-1993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963191#comment-13963191
 ] 

Russell Branca commented on COUCHDB-1993:
-

Hey thanks [~benoitc]! I missed the notifications on your comments, will 
respond shortly.




 Make fabric work with the refactored view engine
 

 Key: COUCHDB-1993
 URL: https://issues.apache.org/jira/browse/COUCHDB-1993
 Project: CouchDB
  Issue Type: Improvement
  Components: BigCouch
Reporter: Volker Mische
Assignee: Russell Branca





--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (COUCHDB-1993) Make fabric work with the refactored view engine

[Russell Branca (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-1993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962301#comment-13962301
 ] 

Russell Branca commented on COUCHDB-1993:
-

For posterity, I'm pasting the copy of my review request email here:

This thread is a request for review on the integration of fabric and 
couch_mrview, although there are changes in chttpd as well. There are three 
pull requests, one for each repo:


https://github.com/apache/couchdb-fabric/pull/1
https://github.com/apache/couchdb-couch-mrview/pull/1
https://github.com/apache/couchdb-chttpd/pull/1


Overall this is actually a sizable reduction in code. While doing the 
integration, I started down the path of deduplicating code the single node vs 
clustered http layers, and as a result these updates are a net negative in 
lines of code and I think does a good job of separating out the view logic from 
fabric and chttpd.

There are a few items that still need to be addressed, but I think they are 
outside of the scope of this initial integration and/or could use wider 
discussion on the proper way to address them

  * Properly send metadata through fabric, more details in the commit message 
of: 
https://github.com/apache/couchdb-fabric/commit/aeac2dfe5014dc9fe1f10300669ea9244cae3a67

  * Handle security databases, ie _users/_replication doc security

  * Figure out what approach to take for etags and standardize etag functions

  * Allow for sending update_seq in the view results


Let's use this thread for wider discussion and use the PRs for comments on 
code, unless there's a better alternative. I'll be around today and tomorrow, 
but then I'll be away from my computer the following week on vacation, so it 
might be a delayed response before I can address any comments. Thanks for the 
review!


 Make fabric work with the refactored view engine
 

 Key: COUCHDB-1993
 URL: https://issues.apache.org/jira/browse/COUCHDB-1993
 Project: CouchDB
  Issue Type: Improvement
  Components: BigCouch
Reporter: Volker Mische
Assignee: Russell Branca





--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: [POLL] Erlang whitespace standards

[Russell Branca]

Thanks for putting this together Joan. I left a few other comments in
places where you did not have indent by four spaces. I personally think we
should either follow the four space indent everywhere (python style), or
just use whatever emacs erlang mode does, as that's the erlang community
standard.

-Russell
On Mar 29, 2014 1:22 PM, Jan Lehnardt j...@apache.org wrote:

 Heya Joan,

 I love everything about this effort, great idea  execution, thanks!

 I cast my votes and hte results are fascinating to watch :)

 Best
 Jan
 --

 On 28 Mar 2014, at 22:11 , Joan Touzet woh...@apache.org wrote:

  Time for everyone's favourite topic: whitespace standards.
 
  I know many of you are fed up with not being able to auto format in
  your favourite editor and match the CouchDB Erlang coding standards, or
  receiving pull requests that are formatted poorly.
 
  I'd like to fix that with an appropriate whitespace standard, and
  supplementary plugins for vi and emacs that will just Do The Right Thing
  and let us all stop worrying about whitespace corrections in pull
  requests.
 
  The basic rules seem to be:
 
 * Indent everything 4 spaces (not 2 or 8, and no tabs)
 * Single blank lines between functions
 * No blank lines between guarded versions of the
   same function (e.g. couch_btree.erl#L36-L47)
 
  but beyond that there is some inconsistency in both the code and in
  discussion with other devs.
 
  Here's a short poll I threw together in Google Docs. I'd love it if you
  could take 5 minutes to reply to it. Early next week I'll summarize and
  post the results. Once we have agreement we can toss up a super small
  Markdown guide / wiki page / whatever and get started on the emacs/vi
  modifications to support it.
 
 
 https://docs.google.com/forms/d/1b7KcQGgNbSCZVRwLjrUl5Z6C2TBx8X1btlU5fwrNHpg/edit#
 
  Thanks,
  Joan

[REVIEW] COUCHDB-1993 fabric + couch_mrview integration

[Russell Branca]

Hi all,


This thread is a request for review on the integration of fabric and
couch_mrview, although there are changes in chttpd as well. There are three
pull requests, one for each repo:


https://github.com/apache/couchdb-fabric/pull/1
https://github.com/apache/couchdb-couch-mrview/pull/1
https://github.com/apache/couchdb-chttpd/pull/1


Overall this is actually a sizable reduction in code. While doing the
integration, I started down the path of deduplicating code the single node
vs clustered http layers, and as a result these updates are a net negative
in lines of code and I think does a good job of separating out the view
logic from fabric and chttpd.

There are a few items that still need to be addressed, but I think they are
outside of the scope of this initial integration and/or could use wider
discussion on the proper way to address them

  * Properly send metadata through fabric, more details in the commit
message of:
https://github.com/apache/couchdb-fabric/commit/aeac2dfe5014dc9fe1f10300669ea9244cae3a67

  * Handle security databases, ie _users/_replication doc security

  * Figure out what approach to take for etags and standardize etag
functions

  * Allow for sending update_seq in the view results


Let's use this thread for wider discussion and use the PRs for comments on
code, unless there's a better alternative. I'll be around today and
tomorrow, but then I'll be away from my computer the following week on
vacation, so it might be a delayed response before I can address any
comments. Thanks for the review!


-Russell

Re: [ANNOUNCE] Robert Kowalski elected as CouchDB committer

[Russell Branca]

Congrats and welcome board Robert!


-Russell


On Fri, Mar 28, 2014 at 8:50 AM, Dave Cottlehuber d...@jsonified.com wrote:

 Great to have you join us Robert :-))

 On 28 March 2014 15:19, Nick North nort...@gmail.com wrote:
  Congratulations, and welcome!
 
  Nick
 
 
  On 28 March 2014 13:26, Klaus Trainer klaus_trai...@posteo.de wrote:
 
  Great!  Welcome Robert :)
 
 
  On 03/28/2014 01:12 PM, Andy Wenk wrote:
   Dear community,
  
   I am pleased to announce that the CouchDB Project Management Committee
  has
   elected Robert Kowalski as a CouchDB committer.
  
   Apache ID: robertkowalski
  
   IRC nick: robertkowalski
  
   Twitter: robinson_k
  
   By default, external contributions to the project follow the
   Review-Then-Commit model. Being a committer means you can follow the
   Commit-Then-Review model. In other words, Robert can now make changes
 at
   will.
  
   This election was made in recognition of Robert's existing
 contributions
   and commitment to the project.
  
   Please join me in extending a warm welcome to Robert!
  
   On behalf of the CouchDB PMC,
  
   Andy
  
 

Re: [ANNOUNCE] Benjamin Young elected as CouchDB committer

[Russell Branca]

Congrats Ben!


-Russell


On Tue, Feb 25, 2014 at 7:19 AM, Benoit Chesneau bchesn...@gmail.comwrote:

 On Tue, Feb 25, 2014 at 3:12 PM, Benjamin Young byo...@bigbluehat.com
 wrote:
 
  On 2/25/14, 7:37 AM, Dave Cottlehuber wrote:
 
  On 25 February 2014 09:53, Robert Samuel Newson wrote:
 
  Dear community,
 
  I am pleased to announce that the CouchDB Project Management
 Committee
  has
  elected Benjamin Young as a CouchDB committer.
 
  Apache ID: bigbluehat
 
  IRC nick: bigbluehat
 
  Twitter: bigbluehat
 
  congratulations Ben!
 
  IIRC you were one of the first people I met wrt to CouchDB back at Couch
  Camp 2010!
 
 
  Yeah. We were roommates. :)
 
  Good memories.
 
  +1 to another one of those wifi-from-the-trees events. :D

 welcome on board :) - benoit

Re: [ANNOUNCE] Max Thayer elected as CouchDB committer

[Russell Branca]

Welcome Max!


-Russell


On Tue, Feb 25, 2014 at 3:33 AM, Andy Wenk a...@nms.de wrote:

 Hi Max - welcome to the club :)

 Cheers

 Andy


 On 25 February 2014 12:20, Noah Slater nsla...@apache.org wrote:

  Dear community,
 
  I am pleased to announce that the CouchDB Project Management Committee
  has elected Max Thayer as a CouchDB committer.
 
  Apache ID: garbados
 
  IRC nick: garbados
 
  Twitter: garbados
 
  By default, external contributions to the project follow the
  Review-Then-Commit model. Being a committer means you can follow the
  Commit-Then-Review model. In other words, Max can now make changes at
  will.
 
  This election was made in recognition of Max's existing contributions
  and commitment to the project.
 
  Please join me in extending a warm welcome to Max!
 
  On behalf of the CouchDB PMC,
 
  --
  Noah Slater
  https://twitter.com/nslater
 



 --
 Andy Wenk
 Hamburg - Germany
 RockIt!

 http://www.couchdb-buch.de
 http://www.pg-praxisbuch.de

 GPG fingerprint: C044 8322 9E12 1483 4FEC 9452 B65D 6BE3 9ED3 9588

 https://people.apache.org/keys/committer/andywenk.asc

Re: [ANNOUNCE] Alexander Shorin joins the PMC

[Russell Branca]

Congrats Alexander! (:


-Russell


On Sat, Feb 22, 2014 at 2:51 PM, Dave Cottlehuber d...@jsonified.com wrote:

 On 22. Februar 2014 at 18:59:19, Alexander Shorin (kxe...@gmail.com)
 wrote:
  Thank you all for kind words and nice welcome! It's really a honor for
  me to join PMC. I'm happy to be there with you. Thanks!
 
  P.S. nice joke, Jan (:
 
  --
  ,,,^..^,,,

 welcome on board Alex!!

 A+
 Dave

Re: [ANNOUNCE] Andy Wenk joins the PMC

[Russell Branca]

Welcome aboard Andy!


-Russell


On Sat, Feb 22, 2014 at 11:14 AM, Klaus Trainer klaus_trai...@posteo.dewrote:

 Hey, congrats Andy!


 On 02/21/2014 09:09 PM, Noah Slater wrote:
  Dear community,
 
  I am delighted to announce that Andy Wenk joins the Apache CouchDB
  Project Management Committee today.
 
  Andy has made outstanding, sustained contributions to the project.
  This appointment is an official acknowledgement of his position within
  the community, and our trust in his ability to provide oversight for
  the project.
 
  Everybody, please join me in congratulating Andy!
 
  On behalf of the CouchDB PMC,
 

Re: [ANNOUNCE] Klaus Trainer elected as CouchDB committer

[Russell Branca]

Welcome Klaus!


-Russell


On Wed, Feb 19, 2014 at 6:08 AM, Dave Cottlehuber d...@jsonified.com wrote:

  On Tue, Feb 18, 2014 at 8:23 PM, Noah Slater wrote:
   Dear community,
  
   I am pleased to announce that the CouchDB Project Management
 Committee
   has elected Klaus Trainer as a CouchDB committer.
  
   Apache ID: klaus_trainer
  
   IRC nick: klaus_trainer
  
   Twitter: KlausTrainer
  
   By default, external contributions to the project follow the
   Review-Then-Commit model. Being a committer means you can
 follow the
   Commit-Then-Review model. In other words, Klaus can now make
 changes
   at will.
  
   This election was made in recognition of Klaus's existing
   contributions and commitment to the project.
  
   Please join me in extending a warm welcome to Klaus!
  
   On behalf of the CouchDB PMC,
  
   --
   Noah Slater
   https://twitter.com/nslater

 \o/ long awaited!

 A+
 Dave

[jira] [Commented] (COUCHDB-2079) Re-enable configurable HTTP handlers

[Russell Branca (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13907678#comment-13907678
 ] 

Russell Branca commented on COUCHDB-2079:
-

+1 to having an empty http layer that allows app to plug into it.

I think it would also be interesting to update the handler functions to be less 
dependent on http, basically keep http isolated to chttpd, and have all the 
other things return erlang data structures. Basically make http a particular 
interface type, allowing for alternative interfaces or direct access.

 Re-enable configurable HTTP handlers
 

 Key: COUCHDB-2079
 URL: https://issues.apache.org/jira/browse/COUCHDB-2079
 Project: CouchDB
  Issue Type: Improvement
  Security Level: public(Regular issues) 
  Components: BigCouch
Reporter: Paul Joseph Davis
  Labels: release

 chttpd hard codes HTTP handlers rather than lists them in the config files. 
 We'll need to revert this change. Though given how some apps (couch_mrview, 
 etc) have their own handlers we should consider making this configurable at 
 the app-level instead of the ini level in CouchDB.
 I have a few ideas on this, mostly by removing all handler code from chttpd 
 and friends. The HTTPd app would then just start empty servers with no 
 handlers and each app would have a supervision tree process that would be 
 responsible for registering handlers.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Assigned] (COUCHDB-523) View API POST keys to retrieve multiple docs by key could also allow for multiple 'range' queries, i.e. an array of { startkey: .., endkey: ... } params in the POST

[Russell Branca (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russell Branca reassigned COUCHDB-523:
--

Assignee: Russell Branca  (was: Joan Touzet)

 View API POST keys to retrieve multiple docs by key could also allow for 
 multiple 'range' queries, i.e. an array of { startkey: .., endkey: ... } 
 params in the POST
 

 Key: COUCHDB-523
 URL: https://issues.apache.org/jira/browse/COUCHDB-523
 Project: CouchDB
  Issue Type: Improvement
  Components: HTTP Interface
Reporter: Nathan Stott
Assignee: Russell Branca
Priority: Minor
 Attachments: couch_httpd_view.erl, multi_start_end_key.diff, 
 ranged_key_post.diff


 It would be useful if I could do a single POST to a view to retrieve multiple 
 ranges specified by startkey, endkey.
 The format could be as follows:
 { ranges: [ { startkey: a, endkey: c }, { startkey:g, 
 endkey:z } ] }



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Assigned] (COUCHDB-1993) Make fabric work with the refactored view engine

[Russell Branca (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-1993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russell Branca reassigned COUCHDB-1993:
---

Assignee: Russell Branca

 Make fabric work with the refactored view engine
 

 Key: COUCHDB-1993
 URL: https://issues.apache.org/jira/browse/COUCHDB-1993
 Project: CouchDB
  Issue Type: Improvement
  Components: BigCouch
Reporter: Volker Mische
Assignee: Russell Branca





--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (COUCHDB-2045) Trace trap error while starting CouchDB

[Russell Branca (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13891033#comment-13891033
 ] 

Russell Branca commented on COUCHDB-2045:
-

Thanks Newson, I can confirm the patch fixed the issue for me and I no longer 
need to set the `DYLD_INSERT_LIBRARIES` environment variable.

 Trace trap error while starting CouchDB 
 

 Key: COUCHDB-2045
 URL: https://issues.apache.org/jira/browse/COUCHDB-2045
 Project: CouchDB
  Issue Type: Bug
  Security Level: public(Regular issues) 
  Components: BigCouch
Reporter: Russell Branca

 I ran into an error today in the 1843-feature-bigcouch branch, where running 
 `./rel/dev1/bin/couchdb` would result in the VM crashing with a trace trap 
 error. I tracked this down to the couch_primary_sup child spec for the 
 `collation_driver` when calling `couch_drv:start_link()`, and more 
 particularly, I was able to reproduce this error in a standard Erlang shell 
 by running the command: 
 `erl_ddll:load(/tmp/bigcouch/rel/dev1/lib/couch-2f254d9/priv, 
 couch_icu_driver).`
 After poking around with things, I ended up stumbling upon: 
 http://openradar.appspot.com/7209349 (linked from 
 http://erlang.org/pipermail/erlang-questions/2010-April/050563.html). I was 
 able to resolve the error by doing `export 
 DYLD_INSERT_LIBRARIES=/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation`
  in the shell prior to executing `./rel/dev1/bin/couchdb`.
 This seems like a logical addition to the flags set in 
 `src/couch/rebar.config.script`, but I'm not familiar with CoreFoundation, so 
 I don't have a recommendation on the proper way to do this across platforms.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Closed] (COUCHDB-2045) Trace trap error while starting CouchDB

[Russell Branca (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russell Branca closed COUCHDB-2045.
---


 Trace trap error while starting CouchDB 
 

 Key: COUCHDB-2045
 URL: https://issues.apache.org/jira/browse/COUCHDB-2045
 Project: CouchDB
  Issue Type: Bug
  Security Level: public(Regular issues) 
  Components: BigCouch
Reporter: Russell Branca

 I ran into an error today in the 1843-feature-bigcouch branch, where running 
 `./rel/dev1/bin/couchdb` would result in the VM crashing with a trace trap 
 error. I tracked this down to the couch_primary_sup child spec for the 
 `collation_driver` when calling `couch_drv:start_link()`, and more 
 particularly, I was able to reproduce this error in a standard Erlang shell 
 by running the command: 
 `erl_ddll:load(/tmp/bigcouch/rel/dev1/lib/couch-2f254d9/priv, 
 couch_icu_driver).`
 After poking around with things, I ended up stumbling upon: 
 http://openradar.appspot.com/7209349 (linked from 
 http://erlang.org/pipermail/erlang-questions/2010-April/050563.html). I was 
 able to resolve the error by doing `export 
 DYLD_INSERT_LIBRARIES=/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation`
  in the shell prior to executing `./rel/dev1/bin/couchdb`.
 This seems like a logical addition to the flags set in 
 `src/couch/rebar.config.script`, but I'm not familiar with CoreFoundation, so 
 I don't have a recommendation on the proper way to do this across platforms.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (COUCHDB-2045) Trace trap error while starting CouchDB

[Russell Branca (JIRA)]

Russell Branca created COUCHDB-2045:
---

 Summary: Trace trap error while starting CouchDB 
 Key: COUCHDB-2045
 URL: https://issues.apache.org/jira/browse/COUCHDB-2045
 Project: CouchDB
  Issue Type: Bug
  Security Level: public (Regular issues)
  Components: BigCouch
Reporter: Russell Branca


I ran into an error today in the 1843-feature-bigcouch branch, where running 
`./rel/dev1/bin/couchdb` would result in the VM crashing with a trace trap 
error. I tracked this down to the couch_primary_sup child spec for the 
`collation_driver` when calling `couch_drv:start_link()`, and more 
particularly, I was able to reproduce this error in a standard Erlang shell by 
running the command: 
`erl_ddll:load(/tmp/bigcouch/rel/dev1/lib/couch-2f254d9/priv, 
couch_icu_driver).`

After poking around with things, I ended up stumbling upon: 
http://openradar.appspot.com/7209349 (linked from 
http://erlang.org/pipermail/erlang-questions/2010-April/050563.html). I was 
able to resolve the error by doing `export 
DYLD_INSERT_LIBRARIES=/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation`
 in the shell prior to executing `./rel/dev1/bin/couchdb`.

This seems like a logical addition to the flags set in 
`src/couch/rebar.config.script`, but I'm not familiar with CoreFoundation, so I 
don't have a recommendation on the proper way to do this across platforms.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: Pull Request comments

[Russell Branca]

Wooo! Thanks Daniel!


-Russell


On Fri, Jan 24, 2014 at 7:32 AM, Benjamin Young byo...@bigbluehat.comwrote:

 On 1/24/14, 8:49 AM, Garren Smith wrote:

 Hi All,

 Thanks to the great work from Daniel aka Humbedooh, we now will receive
 pull request comments on the mailing list. We can also reply to the
 comments and they will be logged on the pull request.

 Cheers
 Garren


 Thanks, Daniel!

Re: Migration Strategy [was Re: Git Repository Creation]

[Russell Branca]

 3. R15*  at least R16B01 and under are still prone to scheduler collapse.
  I am not sure if the
  “+st msecs  hack in R16B02+ eases the pain or not, nor how badly this
  impacts heavy
  CouchDB users. This is the “wake up all schedulers every milliseconds
 to
  prevent collapse
  issue. Read Scott Fritchie’s threads in Erlang-Questions [1] for details
 
  erl man[2].
 
 
 Mmm the link you refer say it is still better than in R14. Anyway this is
 the right path to follow, we should collect any issues we have in mind in
 using latest supported versions of Erlang and see how it impact us.


I'm not sure where you see anything in DCH's link [1] that says R14 is
worse. The relevant bits from that thread are R15B0x's schedulers are
broken, and R16B's schedulers appear to be even more broken. The only
mention of R14 I see on that thread is: The CPU usage is noticeably lower
than R14, but worryingly so..

I think we should support R14 until their is a viable stable alternative,
or there is a sufficiently interesting upgrades to justify the switch. If
we need to run forked versions of 3rd party dependencies until such time
that we can upgrade in full, that seems like the logical path. Ideally we
can figure out a way to use upstream versions directly without needing to
fork.


-Russell

Re: we need more reviews

[Russell Branca]

Huge +1 to more review.

Let's also setup some code guidelines for Erlang/Javascript/C/HTML so we
have an authoritative list of rules to follow to ensure code consistency,
and similarly, let's get some guidelines for git commits messages as well,
Tim Pope's article comes to mind:
http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html.

Whichever review tool we decide to use, we should use it universally. We
should figure out a way to do github integration so that PR's show up in
the tool, and then we update the PR with a link to the relevant review
page. We also need a good way to initiate the analogue of a PR in whatever
tool we use. One of the complications with the review process today, is
that we're all accustomed to doing review through github PR's, but we can
use github PR's for branches who exist solely on the ASF origin. I imagine
this won't be an issue with Hive or Gerrit, but we should verify that we're
not required to initiate code review from third party origins.

On a side note, one of my personal nits with github PR's, that I hope might
be resolved with one of these tools, is losing comments when the code
changes. For instance, if I do a review and make 10 comments that each
represent a line item that needs to be addressed, I would like to see a
task list on the review indicating there are unfinished items to be
addressed, basically a way to make a set of checkboxes necessary to
complete the review. I find it odd that github doesn't do this, even more
so by the fact that changing code to address comments will often mess with
the commenting and hide the messages you posted, making it challenging to
decipher whether the review items have been addressed. This is by no means
a requirement to using any approach, just hoping that someone knows of a
good way to approach this problem in whatever tool we use.


-Russell


On Wed, Jan 22, 2014 at 6:06 AM, Benoit Chesneau bchesn...@gmail.comwrote:

 On Wed, Jan 22, 2014 at 12:47 PM, Noah Slater nsla...@apache.org wrote:

  My first comment is: if we want more reviews, let's have more committers.
 
  We double our committer base in 2013, and the results look like this:
 
  https://www.ohloh.net/p/couchdb/analyses/latest/languages_summary
 
  And I see comments like this on StackOverflow:
 
  I've recently noticed that Couch DB is back in heavy development.


  So I think we should continue to aggressively recruit more committers
  to the project in 2014. Excelsior!
 

 Welcoming new committers in the project is certainly a good thing but I
 think it's orthogonal to the need of more review and team work. We should
 find a good workflow now while we are still a limited number of committers
 because It will be harder to enforce anything on a larger team. We should
 settle on some guidelines now. It will also help us to attract more people
 IMO.


 
  However, as for the way we do reviews...
 
  Infra ticket about Gerrit
  https://issues.apache.org/jira/browse/INFRA-2205
 
  Effort seems to be mothballed. But I'm sure we could restart it if we
  were serious.
 
  However, we could use this:
 
  https://reviews.apache.org/groups/hive/
 
  It is well integrated with Apache infrastructure already, sends mails
  to the mailing list, and so on.
 
  Happy to request an instance and we can experiment with it if we like.
  If it doesn't work out, we stop using it. No harm. Could make it
  entirely voluntary until we figure out a workflow that we all like.
 
  Should I do that?
 

 I thought gerrit was already integrated but any tool that could help us to
 make the review is OK for me. How hive compares to gerrit? Could we also
 plug it with github like gerrit [1] ?

 - benoît

 [1] https://gerrit.googlesource.com/plugins/github/ (and some others)

 
  On 20 January 2014 15:30, Nick North nort...@gmail.com wrote:
   On 20 January 2014 12:26, Dale Harvey d...@arandomurl.com wrote:
  
   I fully agree, its something I mentioned at the couchdb conf in
  vancouver,
   a review first system encourages contributions and has multiple
 benefits
  
* At least 2 people look at the code, less likely to push silly
  mistakes
* Can codify and practice review rules
* Its much easier to view the current activity in the project
* Can bring up points of discussion before its too late
  
   But I think the most important thing is that it removes the burden of
   responsibility from the committer to the project as a whole, also
 hugely
   beneficial is that it makes it particularly obvious when a patch has
  reach
   a stalemate and forces someone to make the call.
  
   For reference on PouchDB every committer is trusted to push code,
 nobody
   (including myself) pushes their own code to master, it goes in the PR
  queue
   and gets a +1 from any other committer (who will usually push it),
 thats
   essentially the same process we use at mozilla and coming to think of
 it
   any other project I have worked on, any commiter has the ability to
 -1 

Re: Git Repository Creation

[Russell Branca]

The scheduler collapse problems in R15 and R16 are widely known and not
resolved. Frankly, as developers of a database, we should strive to provide
end users with the most reliable and best experience, which in my opinion
means we should recommend R14B01. There is not a battle tested, reliable
version of Erlang that has proven to solve the scheduler collapse problems,
and until that time, I think it's unwise to remove support for R14.


-Russell


On Wed, Jan 22, 2014 at 6:20 AM, Benoit Chesneau bchesn...@gmail.comwrote:

 Robert,

 I understood what you meant.

 Imo the best thing would be creating a check list of the things that
 prevent to go to a version greater than R14. Can you share the one you have
 inside cloudant ? It will help us to reach a consensus also later to make
 sure we can fix them in next Erlang releases.

 This is not that I want absolutely use the latest. If we stand on an old
 and unmaintained release then we should know exactly why and check from
 time to time if we still need to stay on this version.

 Thanks!

 - benoit





 On Wed, Jan 22, 2014 at 2:06 PM, Robert Samuel Newson rnew...@apache.org
 wrote:

 
  I could have phrased it better, so I’ll do so now;
 
  R14 is still widely used in production and is very stable. R15 and R16
  have known stability problems that affect deployments using NIF’s that
 can
  potentially run for longer than a millisecond before returning control to
  the scheduler.
 
  I am not blackmailing the project but I hope you can understand how I
 feel
  about your suggestion to remove the ability for Cloudant to continue
  working after we are making such a large contribution and, further,
 seeking
  to move our active development to couchdb itself.
 
  B.
 
  On 22 Jan 2014, at 13:01, Benoit Chesneau bchesn...@gmail.com wrote:
 
   On Wed, Jan 22, 2014 at 1:58 PM, Dave Cottlehuber d...@jsonified.com
  wrote:
  
   On 22 January 2014 13:23, Benoit Chesneau bchesn...@gmail.com
 wrote:
   On Wed, Jan 22, 2014 at 12:41 PM, Robert Samuel Newson
   rnew...@apache.orgwrote:
  
   Benoit,
  
   Cloudant requires R14 support, it would mean our contribution to
  couchdb
   becomes useless to us and we could not contribute further.
  
  
   Are you using blackmail? Is this the position of the Cloudant
 company?
  
   Hi Benoit,
  
   Your comment reads like an ad hominem attack, and I don't think Bob's
   point, nor Bob, nor Cloudant, deserved that.
  
  
   My questions stand. The way it is formulated, and that's not the first
   time, is not that clear at all.
 
 

Re: Git Repository Creation

[Russell Branca]

On Tue, Jan 21, 2014 at 10:51 AM, Alexander Shorin kxe...@gmail.com wrote:

 On Tue, Jan 21, 2014 at 1:44 PM, Paul Davis paul.joseph.da...@gmail.com
 wrote:
  The repos for futon and jquery.couch.js are less
  clear. I'd prefer to avoid having infra create repos we don't intend
  to use beyond fancy archives.

 While I couldn't say for sure for Futon, jquery.couch.js is pretty
 popular client which people uses, wanted to contribute to and I think
 it shouldn't become abandoned deep within some archive repo. At some
 point we're all agreed on the fact that's better to move it to
 separate repo, no matter will this repo be under CouchDB flag or not -
 but I think first one is better.

 --
 ,,,^..^,,,



Is anyone maintaining jquery.couch.js? There were some decent patches that
slipped by due to lack of a maintainer (for instance the updates to use
jquery promises). I don't think we should create a repo for jquery.couch.js
unless we have someone who is planning on maintaining it long term.

As for Futon, I think it's less relevant to move it to a separate repo
because a) it's being replaced and b) there is minimal development
happening there.


-Russell

Re: couchdb pull request: SOCKS 5 support for replication

[Russell Branca]

This is very cool Newson! Big +1 for this landing.


-Russell


On Sat, Jan 4, 2014 at 10:35 AM, rnewson g...@git.apache.org wrote:

 GitHub user rnewson opened a pull request:

 https://github.com/apache/couchdb/pull/125

 SOCKS 5 support for replication

 Notably, this allows replication over the Tor network (yes, DNS
 resolution occurs over Tor also).


 You can merge this pull request into a Git repository by running:

 $ git pull https://github.com/rnewson/couchdb socks5

 Alternatively you can review and apply these changes as the patch at:

 https://github.com/apache/couchdb/pull/125.patch

 
 commit 059fe032d5afe8f668effd2c3f81fa73ebe4031f
 Author: Robert Newson rnew...@apache.org
 Date:   2014-01-04T17:32:00Z

 Add SOCKS5 module to ibrowse

 commit 3fbbc234a86600881d6a75d9cece4d2af8c168a3
 Author: Robert Newson rnew...@apache.org
 Date:   2014-01-04T17:31:40Z

 Pass protocol to ibrowse

 commit e529636f0a07c898ba9b1e788deaebffb7f871de
 Author: Robert Newson rnew...@apache.org
 Date:   2014-01-04T17:57:22Z

 Clarify HTTP proxy fields and variables

 commit 65ba845f5705885b46842df93f8017af58841177
 Author: Robert Newson rnew...@apache.org
 Date:   2014-01-04T18:02:59Z

 Use SOCKS5 proxy if specified

 

[jira] [Commented] (COUCHDB-1960) Provide pagination for /_all_dbs

[Russell Branca (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863177#comment-13863177
 ] 

Russell Branca commented on COUCHDB-1960:
-

The BigCouch merge will change the list of databases to come from a database 
all docs view, rather than a file system listing. As such, we'll want to 
recommend standard view query params over using skip once that is brought in.

 Provide pagination for /_all_dbs
 

 Key: COUCHDB-1960
 URL: https://issues.apache.org/jira/browse/COUCHDB-1960
 Project: CouchDB
  Issue Type: Improvement
  Components: Fauxton, HTTP Interface
Reporter: Benjamin Young

 The database-per-user design pattern has increased in popularity and 
 consequently causes long lists of databases when one hits {{/_all_dbs}}
 To keep this sane, we need some pagination.
 {{?skip=10limit=10}} style seems to be the simplest to implement (and there 
 should be existing code from Cloudant for this one).
 Thanks!



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: [ANNOUNCE] Nick North elected as CouchDB committer

[Russell Branca]

Welcome Nick!


-Russell


On Thu, Jan 2, 2014 at 11:57 AM, Jan Lehnardt j...@apache.org wrote:

 Welcome aboard, Nick! :)

 Best
 Jan
 --

 On 01 Jan 2014, at 20:24 , Dave Cottlehuber d...@jsonified.com wrote:

  Dear community,
 
  There's nothing like starting off the New Year with a New Committer!!
 
  I am pleased to announce that the CouchDB Project Management Committee
  has elected Nick North as a CouchDB committer.
 
 Apache ID: nicknorth
 
 IRC nick: NickN
 
 Twitter: @_shastra
 
  By default, external contributions to the project follow the
  Review-Then-Commit model. Being a committer means you can follow the
  Commit-Then-Review model. In other words, Nick can now make changes at
  will.
 
  This election was made in recognition of Nick's existing contributions
  and commitment to the project.
 
  Please join me in extending a warm welcome to Nick!
 
  On behalf of the CouchDB PMC,
 
  Dave Cottlehuber

[jira] [Commented] (COUCHDB-1963) Move from etap to eunit

[Russell Branca (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13852236#comment-13852236
 ] 

Russell Branca commented on COUCHDB-1963:
-

+1 as well. I'm happy to help with this. Is there a branch where these changes 
are happening? Or should we just split it up per module or some such?

 Move from etap to eunit
 ---

 Key: COUCHDB-1963
 URL: https://issues.apache.org/jira/browse/COUCHDB-1963
 Project: CouchDB
  Issue Type: Improvement
  Components: Test Suite
Reporter: Joan Touzet
Assignee: Alexander Shorin
  Labels: etap, eunit, test

 Pursuant to an IRC/Twitter discussion, it seems that there is some
 support ([~pjdavis1970], [~janl]) to move off of etap. Motivations include 
 etap being unmaintained for 2+ years, eunit shipping with Erlang, and least 
 importantly etap tests being completely broken on the Windows build.
 There are only 63 .t files in the entire source tree; hopefully this is work 
 that can be weekend-tackled by one person, or could be tag-teamed with little 
 friction.
 There was a subsequent discussion about the current, existing JS tests; while 
 there are issues with these and room for improvement, this bug is *strictly* 
 about our white-box etap unit tests and larger multi-instance tests that may 
 not best be accomplished by a JS framework.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Commented] (COUCHDB-1960) Provide pagination for /_all_dbs

[Russell Branca (JIRA)]

[ 
https://issues.apache.org/jira/browse/COUCHDB-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13846850#comment-13846850
 ] 

Russell Branca commented on COUCHDB-1960:
-

I didn't spend much time optimizing the _all_dbs pagination because as Newson 
mentioned in the other thread, it's doing a file system scan and returning all 
the items. So if you have tens of thousands of databases, let alone millions, 
you're going to be hosed either way. Although admittedly, making an additional 
request per database doesn't help matters. When BigCouch is merged, the list of 
databases will be represented in a database, and we will be able to do the 
standard pagination queries on _all_dbs.

 Provide pagination for /_all_dbs
 

 Key: COUCHDB-1960
 URL: https://issues.apache.org/jira/browse/COUCHDB-1960
 Project: CouchDB
  Issue Type: Improvement
  Components: Fauxton, HTTP Interface
Reporter: Benjamin Young

 The database-per-user design pattern has increased in popularity and 
 consequently causes long lists of databases when one hits {{/_all_dbs}}
 To keep this sane, we need some pagination.
 {{?skip=10limit=10}} style seems to be the simplest to implement (and there 
 should be existing code from Cloudant for this one).
 Thanks!



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Re: [PROPOSAL] new underscore namespacing

[Russell Branca]

I'm +1 to the idea of ripping out the metadata fields from the doc bodies
entirely and adding them to the headers. I think this makes a much cleaner
separation of concerns. However, one thing that jumps out at me as a
potential gotcha, is grabbing a large list of conflicts, or some other
piece of metadata that can be much larger than allowable lengths in
headers. We'll need to come up with an approach there, although perhaps
it's a good opportunity to rethink how we bubble up conflicts to users, and
find some way to make it more obvious when there is a conflict.


-Russell


On Tue, Dec 3, 2013 at 9:25 AM, Brian Mitchell br...@strmpnk.co wrote:

 On Tue, Dec 3, 2013 at 10:35 AM, Volker Mische volker.mis...@gmail.com
 wrote:
  I personally would prefer to have the meta information completely
  separate from the document. I know there have been discussion in the
  past to even have them separate in the backend (but that's not the point
  of this proposal).
 
  So the API for the view function could change to `function(doc, meta)`.
  This way you could store in your document whatever you like.

 This is convenient in some cases but I really do like how CouchDB
 documents are self-contained and self-representing. Having to rely on
 an outside context to know things like identity and revision seems
 unfortunate.

 Most cases I've interacted with CouchDB would require at least that
 minimal _id and _rev. Separating them might seem okay if you just look
 at a map function with two arguments but it gets messier if you look
 at client code in the applications that use CouchDB. Many do things
 like:

 my_doc = db.get('doc')
 ...
 db.put(my_doc)

 Juggling this context separately will require extra work or at least
 force CouchDB clients to reinvent some way to wrap up and store
 metadata. My experience writing client code against Riak, which
 externalizes this metadata, was not nearly as clean as working with
 CouchDB since everything needed some sort of wrapper where CouchDB
 could many times get away with just a simple value.

 On the side of internals, there is certainly some work to be done to
 lift things in and out of this special field, but it could be done w/o
 forcing clients to live with their world split in two.

 Brian.

Re: [discuss] couchapp.org down, couchapp.org spammed

[Russell Branca]

In its current form, couchapp.org is a relic that has been mostly abandoned
and is not maintainable by the community. I strongly feel we should move
the couchapp documentation into official documentation. I see two relevant
points of interest in this discussion. First is the notion of couchapps as
the self contained application platform that utilizes show/list functions,
and whether these should be included in CouchDB. I don't think this is the
important issue at hand.

The bigger issue is that the defacto method of defining design documents is
to use one of the many couchapp tools, and the only place this is
documented as a whole is couchapp.org. This is a disservice to the
community, and one that needs to be resolved. This is a constant source of
confusion to new users who quickly realize the futility of defining design
docs in the browser, and get lost when told just use a couchapp, and then
they inevitably end up clicking on the top google result for couchapp,
couchapp.org. We need to have this properly documented in the official
documentation so that the process is fully defined for new users.

A couple of options for approach would be to formalize the folder
definition of a couchapp and list tools known to be compatible, or to
officially bless a tool like Erica. While I do want to see Fauxton provide
powerful editors for all the different function types, I don't think this
is sufficient as people typically want to keep their design docs raw code
under SCM. Whatever approach is taken, I think the number one priority here
is ensuring proper documentation explaining to users best practices for
defining and maintaining design docs.


-Russell


On Sat, Nov 30, 2013 at 2:57 PM, Filippo Fadda 
filippo.fa...@programmazione.it wrote:

 List and show are not stored procedure. We already discussed that Benoit.

 http://wiki.apache.org/couchdb/Formatting_with_Show_and_List

 From the documentation: Show and list functions are side effect free and
 idempotent. They can not make additional HTTP requests against CouchDB.
 Their purpose is to render JSON documents in other formats.

 Apples are not pears. I'm not buying it, stop selling list and show like
 stored proc. A stored procedure, in general, is not side effect free or
 idempotent.

  If you just want a database then any KV is enough for your purpose. If
 now
  you want a DBMS which at least I want, then you probably want a way to
  store procedures and such to adapt the query language to your needs.
 Guess,
  what is the purpose of lists and shows?

 Not in the way they are. To me, actually list and show are useless. I'm
 sorry, this is what I think. Can we just move on?

 -Filippo

[jira] [Created] (COUCHDB-1922) CORS bug with reduce_headers and ?SIMPLE_HEADERS

[Russell Branca (JIRA)]

Russell Branca created COUCHDB-1922:
---

 Summary: CORS bug with reduce_headers and ?SIMPLE_HEADERS
 Key: COUCHDB-1922
 URL: https://issues.apache.org/jira/browse/COUCHDB-1922
 Project: CouchDB
  Issue Type: Bug
  Components: HTTP Interface
Reporter: Russell Branca


The current implementation of couch_httpd_cors:reduce_headers0/3 has a bug in 
matching against couch_httpd_cors:member_nocase/2, where the atom `true` should 
actually be the atom `false`: [1].

This currently has the effect of never removing the disallowed elements from 
the list, as desired. The immediate fix of `s/true/false/` on that line breaks 
two additional tests that expect the Server header to be passed through to 
the response, because Server is not in the list `?SIMPLE_HEADERS` [2], nor 
should it be as per the spec [3].

We'll want to construct a list of allowed headers that is the union of the 
simple headers and the allowed CouchDB headers, like Server.

[1] 
https://github.com/apache/couchdb/blob/master/src/couchdb/couch_httpd_cors.erl#L248
[2] 
https://github.com/apache/couchdb/blob/master/src/couchdb/couch_httpd_cors.erl#L35-L37
[3] http://www.w3.org/TR/cors/#simple-header



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Resolved] (COUCHDB-1922) CORS bug with reduce_headers and ?SIMPLE_HEADERS

[Russell Branca (JIRA)]

 [ 
https://issues.apache.org/jira/browse/COUCHDB-1922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Russell Branca resolved COUCHDB-1922.
-

Resolution: Fixed
  Assignee: Russell Branca

 CORS bug with reduce_headers and ?SIMPLE_HEADERS
 

 Key: COUCHDB-1922
 URL: https://issues.apache.org/jira/browse/COUCHDB-1922
 Project: CouchDB
  Issue Type: Bug
  Components: HTTP Interface
Reporter: Russell Branca
Assignee: Russell Branca

 The current implementation of couch_httpd_cors:reduce_headers0/3 has a bug in 
 matching against couch_httpd_cors:member_nocase/2, where the atom `true` 
 should actually be the atom `false`: [1].
 This currently has the effect of never removing the disallowed elements from 
 the list, as desired. The immediate fix of `s/true/false/` on that line 
 breaks two additional tests that expect the Server header to be passed 
 through to the response, because Server is not in the list 
 `?SIMPLE_HEADERS` [2], nor should it be as per the spec [3].
 We'll want to construct a list of allowed headers that is the union of the 
 simple headers and the allowed CouchDB headers, like Server.
 [1] 
 https://github.com/apache/couchdb/blob/master/src/couchdb/couch_httpd_cors.erl#L248
 [2] 
 https://github.com/apache/couchdb/blob/master/src/couchdb/couch_httpd_cors.erl#L35-L37
 [3] http://www.w3.org/TR/cors/#simple-header



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: [ANNOUNCE] Andy Wenk elected as CouchDB committer

[Russell Branca]

Welcome aboard Andy!


-Russell


On Wed, Nov 6, 2013 at 9:42 AM, Andy Wenk a...@nms.de wrote:

 Hey everyone,

 I am really honoured to have the opportunity to be a part of the Apache
 CouchDB project. I am a fan and user since 2008 an I love this project and
 the community - namely you!

 My first focus will be the integration of L10n into CouchDB. I hope I can
 motivate a lot of people to help. But I am also interested in all the other
 stuff in CouchDB - especially in the Erlang parts - but wait - I have to
 learn a lot ;.)

 So - well ... thanks to all of you ;-)

 Let's Rock!

 Andy


 On 6 November 2013 18:17, Simon Metson si...@cloudant.com wrote:

  W00t!
 
 
  On Wednesday, 6 November 2013 at 17:15, Dirkjan Ochtman wrote:
 
   On Wed, Nov 6, 2013 at 6:07 PM, Noah Slater nsla...@apache.org(mailto:
  nsla...@apache.org) wrote:
I am pleased to announce that the CouchDB Project Management
 Committee
has elected Andy Wenk as a CouchDB committer.
  
  
  
   \o/ Welcome!
  
   Cheers,
  
   Dirkjan
 
 
 


 --
 Andy Wenk
 Hamburg - Germany
 RockIt!

 http://www.couchdb-buch.de
 http://www.pg-praxisbuch.de

 GPG fingerprint: 3B6D 07F8 8732 CD61 D078  EF7A 63BA 495D 0F15 F21F