3.3.3 packages update

[Nick Vatamaniuc]

Hi everyone,

Debian and RPM 3.3.3 packages were rebuilt and updated with the Erlang
version 24.3.4.15. That version fixes a memory leak
https://github.com/erlang/otp/issues/7834

There were no other CouchDB source changes. The updated deb package
has version 3.3.3-1 and the rpm one has version 3.3.3.1-1

Cheers,
-Nick

[RESULT] (Was: [VOTE] Release Apache CouchDB 3.3.3-rc1)

[Nick Vatamaniuc]

Dear community,

The vote has now closed.

The results are:
+1 - 4 votes
+0 - votes
-0 -  votes
-1 - votes

The vote passes.

Thank you to everyone who participated!
-Nick

Re: [VOTE] Release Apache CouchDB 3.3.3

[Nick Vatamaniuc]

signature: ok
checksum: match
make check: pass
make release: ok
fauxton verify install: pass

env:
   erlang 24, ubuntu 22.04.03

+1

On Fri, Dec 1, 2023 at 1:09 PM Jiahui Li  wrote:
>
> Hi,
>
> +1
>
> Signature: ok
> Checksums: match
> make check: passed
> make release: ok
>
> env:
>   - macOS/x86_64 14.1.1, Erlang 24.3.4.14, Elixir 1.15.7, Python 3.11.6, 
> SpiderMonkey 91
>   - macOS/x86_64 14.1.1, Erlang 25.3.2.7, Elixir 1.15.7, Python 3.11.6, 
> SpiderMonkey 91
>
> Thanks for preparing the release!
>
>
> Jiahui Li (Jessica)
> 
> From: Nick Vatamaniuc 
> Sent: Wednesday, November 29, 2023 1:22 PM
> To: dev@couchdb.apache.org 
> Subject: [EXTERNAL] [VOTE] Release Apache CouchDB 3.3.3
>
> Dear community,
>
> I would like to propose that we release Apache CouchDB 3.3.3.
>
> Candidate release notes:
> https://github.com/apache/couchdb/blob/3.3.x/src/docs/src/whatsnew/3.3.rst
>
> We encourage the whole community to download and test these release
> artefacts so that any critical issues can be resolved before the
> release is made. Everyone is free to vote on this release, so dig
> right in! (Only PMC members have binding votes, but they depend on
> community feedback to gauge if an official release is ready to be
> made.)
>
> The release artefacts we are voting on are available here:
> https://dist.apache.org/repos/dist/dev/couchdb/source/3.3.3/rc.1/
>
> There, you will find a tarball, a GPG signature, and the SHA256 checksum.
>
> Please follow the test procedure here:
> https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release
>
> Please remember that "RC1" is an annotation. If the vote passes, these
> artefacts will be released as Apache CouchDB 3.3.3.
>
> Please cast your votes now.
>
> Thanks,
> -Nick

[VOTE] Release Apache CouchDB 3.3.3

[Nick Vatamaniuc]

Dear community,

I would like to propose that we release Apache CouchDB 3.3.3.

Candidate release notes:
https://github.com/apache/couchdb/blob/3.3.x/src/docs/src/whatsnew/3.3.rst

We encourage the whole community to download and test these release
artefacts so that any critical issues can be resolved before the
release is made. Everyone is free to vote on this release, so dig
right in! (Only PMC members have binding votes, but they depend on
community feedback to gauge if an official release is ready to be
made.)

The release artefacts we are voting on are available here:
https://dist.apache.org/repos/dist/dev/couchdb/source/3.3.3/rc.1/

There, you will find a tarball, a GPG signature, and the SHA256 checksum.

Please follow the test procedure here:
https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release

Please remember that "RC1" is an annotation. If the vote passes, these
artefacts will be released as Apache CouchDB 3.3.3.

Please cast your votes now.

Thanks,
-Nick

Re: [ANNOUNCE] Jiahui Li elected as CouchDB committer

[Nick Vatamaniuc]

Congratulations, Jessica!


On Thu, Aug 24, 2023 at 2:03 PM Ronny Berndt  wrote:
>
> Dear community,
>
> I am pleased to announce that the CouchDB Project Management Committee has 
> elected Jiahui Li as a CouchDB committer.
>
> Apache ID: jiahuili430
>
> Slack nick: Jessica Li
>
> Committers are given a binding vote in certain project decisions, as well as 
> write access to public project infrastructure.
>
> This election was made in recognition of Jiahui's commitment to the project. 
> We mean this in the sense of being loyal to the project and its interests.
>
> Please join me in extending a warm welcome to Jiahui!
>
> On behalf of the CouchDB PMC,
> Ronny

Re: [ANNOUNCE] Gábor Páli elected as CouchDB committer

[Nick Vatamaniuc]

Welcome and congratulations, Gábor!

On Thu, Aug 24, 2023 at 1:58 PM Ronny Berndt  wrote:
>
> Dear community,
>
> I am pleased to announce that the CouchDB Project Management Committee has 
> elected Gábor Páli as a CouchDB committer.
>
> Apache ID: pgj
>
> Slack nick: pgj
>
> Committers are given a binding vote in certain project decisions, as well as 
> write access to public project infrastructure.
>
> This election was made in recognition of Gábor's commitment to the project. 
> We mean this in the sense of being loyal to the project and its interests.
>
> Please join me in extending a warm welcome to Gábor!
>
> On behalf of the CouchDB PMC,
> Ronny

Re: Out of disk handler proposal

[Nick Vatamaniuc]

Looks great, and a nice way to use the Erlang diskmon functionality.

Would it make sense to allow writes which delete data to allow users
to recover some space. Technically it could be a "deleted" update with
a large body, but in most cases users would not do that and would
actually delete data.

Another interesting thing is what happens at the coordinator level.
Currently the patch behavior (I think?) is if at least one worker
(node) responds with "insufficient_storage" the whole request stops
and returns a 507. That's probably the simpler approach. Would there
be a case where the disk sizes on nodes are not equal, and some nodes
would become full while others have more space. In that case we could
consider "insufficient_storage" errors as maintenance mode or
"rexi_EXIT" errors. However, that might hide the problem at the API
level as we'd still return 202 for doc updates.  (But hopefully
diskmon would emit emergency or critical level logs at least).
Another, tangentially related error condition, is when file systems
are remounted read-only on error. In that case disk writes return
"erofs" errors. We had an issue to improve the behavior in that case,
treating them as a maintenance mode / rexi_EXIT error :
https://github.com/apache/couchdb/issues/4168

What if the disk does get full (the internal replicator catches up or
brings in data from other nodes), would it make sense to also handle
"enospc" errors similar to "insufficient_storage" errors?

In diskmon is there a way to raise a warning before we hit the
"insufficient_storage" and at least emit something the logs and/or
have a _node/_local/alarms API to let users check if there are any
warnings or alarms set? That's to give users a chance to manage their
disk capacity before API requests start failing with 507? I don't
think there is anything in the current API for them to get an
indication how close they are to filling the disk.

Cheers,
-Nick

On Thu, Jun 29, 2023 at 10:09 AM Robert Newson  wrote:
>
> Hi All,
>
> out of disk handler
>
> I propose to enhance CouchDB to monitor disk occupancy and react 
> automatically as free space becomes scarce. I've written a working prototype 
> at: https://github.com/apache/couchdb/compare/main...out-of-disk-handler
>
> The `diskmon` application is part of Erlang/OTP and I suggest we use that as 
> the base, since it supports all the platforms we support (and a few more).
>
> The patch reacts differently depending on whether it is database_dir or 
> view_index_dir that runs out of space (of course they might both run out of 
> space at the same time in the common case that the same device is used for 
> both), namely;
>
> 1) Clustered database updates are prohibited (a 507 Insufficient Storage 
> error is returned)
> 2) Background indexing is suspended (no new jobs will be started)
> 3) Querying a stale view is prohibited (a 507 Insufficient Storage error is 
> returned)
> 4) Querying an up-to-date view is permitted
>
> The goal being to leave internal replication running (to avoid data loss) and 
> compaction (as the only action that reduces disk occupancy). I can see adding 
> an option to suspend _all_ writing at, say, 99% full, in order to avoid 
> hitting actual end of disk, but have not coded this up in the branch so far.
>
> At the moment these all activate at once, which I think is not how we want to 
> do this.
>
> I suggest that we have configuration options for;
>
> 1) a global toggle to activate the out of disk handler
> 2) a parameter for the used disk percentage of view_index_dir at which we 
> suspend background indexing, defaulting to 80
> 3) a parameter for the used disk percentage of view_index_dir at which we 
> refuse to update stale indexes, defaulting to 90
> 4) a parameter for the used disk percentage of database_dir at which we 
> suspend writes, defaulting to 90.
>
> What do we all think?
>
>
> B.

Re: Upgrade and reboot of ci-couchdb.a.o

[Nick Vatamaniuc]

Sunday would work. Thanks, Gavin!

Cheers,
-Nick

On Thu, Jun 1, 2023 at 2:23 AM Gavin McDonald  wrote:
>
> Hi All,
>
> I would like to schedule downtime to upgrade Jenkins on ci-couchdb.a.o and
> to also update OS packages and do a reboot.
>
> Would Sunday at around 10am UTC be ok for that?
>
> TIA
>
> --
>
> *Gavin McDonald*
> Systems Administrator
> ASF Infrastructure Team

[DISCUSS] Make Erlang 24 our minimum supported version

[Nick Vatamaniuc]

What do we think about moving to Erlang 24 as our minimum supported version?

24 has been the base version of our package releases for a while, and
we've also been running it at Cloudant for more than 6 months now
without any issues.

Besides speedier JIT, there are some handy-dandy maps functions, and
raw file delete functionality which would be nice to use without the
extra macros around them.

What do we think?

-Nick

[RESULT] (Was [VOTE] Release Apache CouchDB 3.3.2)

[Nick Vatamaniuc]

Dear community,

The vote has now closed.

Thank you to everyone who participated!

The results are:
+1 - 3 votes
+0 - 0 votes
-0 - 0 votes
-1 - 0 votes

The vote for 3.3.2 release passed.

Thank you,
-Nick

[RESULT] (was [VOTE] Release Apache CouchDB 3.2.3)

[Nick Vatamaniuc]

Dear community,

The vote has now closed.

Thank you to everyone who participated!

The results are:
+1 - 3 votes
+0 - 0 votes
-0 - 0 votes
-1 - 0 votes

The vote for 3.2.3 release passed.

Thanks,
-Nick

Re: [VOTE] Release Apache CouchDB 3.2.3

[Nick Vatamaniuc]

+1

Signature: ok
Checksums: match.
make check: pass
make release, fauxton "verify installation": works

env: macos x86_64, python 3.8.16, erlang 23.3.4.18, elixir 1.12.3

Thanks,
-Nick

On Sun, Apr 16, 2023 at 4:14 AM Nick Vatamaniuc  wrote:
>
> Dear community,
>
> I would like to propose that we release Apache CouchDB 3.2.3.
>
> Candidate release notes:
> https://github.com/apache/couchdb/blob/3.2.3-docs/src/docs/src/whatsnew/3.2.rst
>
> We encourage the whole community to download and test these release
> artefacts so that any critical issues can be resolved before the
> release is made. Everyone is free to vote on this release, so dig
> right in! (Only PMC members have binding votes, but they depend on
> community feedback to gauge if an official release is ready to be
> made.)
>
> The release artefacts we are voting on are available here:
> https://dist.apache.org/repos/dist/dev/couchdb/source/3.2.3/rc.1/
>
> There, you will find a tarball, a GPG signature, and SHA256/SHA512 checksums.
>
> Please follow the test procedure here:
> https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release
>
> Since 3.2.3 is based on an older release, there is a Python
> compatibility issue where some tests may not work with the later
> version. I used Python 3.8, Erlang 23 and Elixir 1.12.3
>
> Please remember that "RC1" is an annotation. If the vote passes, these
> artefacts will be released as Apache CouchDB 3.2.3.
>
> Please cast your votes now.
>
> Thanks,
> -Nick

Re: [VOTE] Release Apache CouchDB 3.3.2

[Nick Vatamaniuc]

Minor type correction "I would like to propose that we release Apache
CouchDB 3.3.0." should be "I would like to propose that we release
Apache CouchDB 3.3.2".

My own +1:

Signature: ok.
Checksum: match.
make check: pass
make release, fauxton "verify installation": works

env: macos x86_64, python 3.10.7, erlang 24.3.4.9, elixir 1.13.4

Cheers,
-Nick

On Sun, Apr 16, 2023 at 4:06 AM Nick Vatamaniuc  wrote:
>
> Dear community,
>
> I would like to propose that we release Apache CouchDB 3.3.0.
>
> Candidate release notes:
> https://github.com/apache/couchdb/blob/3.3.x/src/docs/src/whatsnew/3.3.rst
>
> We encourage the whole community to download and test these release
> artefacts so that any critical issues can be resolved before the
> release is made. Everyone is free to vote on this release, so dig
> right in! (Only PMC members have binding votes, but they depend on
> community feedback to gauge if an official release is ready to be
> made.)
>
> The release artefacts we are voting on are available here:
> https://dist.apache.org/repos/dist/dev/couchdb/source/3.3.2/rc.1/
>
> There, you will find a tarball, a GPG signature, and SHA256/SHA512 checksums.
>
> Please follow the test procedure here:
> https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release
>
> Please remember that "RC1" is an annotation. If the vote passes, these
> artefacts will be released as Apache CouchDB 3.3.2.
>
> Please cast your votes now.
>
> Thanks,
> -Nick

[VOTE] Release Apache CouchDB 3.2.3

[Nick Vatamaniuc]

Dear community,

I would like to propose that we release Apache CouchDB 3.2.3.

Candidate release notes:
https://github.com/apache/couchdb/blob/3.2.3-docs/src/docs/src/whatsnew/3.2.rst

We encourage the whole community to download and test these release
artefacts so that any critical issues can be resolved before the
release is made. Everyone is free to vote on this release, so dig
right in! (Only PMC members have binding votes, but they depend on
community feedback to gauge if an official release is ready to be
made.)

The release artefacts we are voting on are available here:
https://dist.apache.org/repos/dist/dev/couchdb/source/3.2.3/rc.1/

There, you will find a tarball, a GPG signature, and SHA256/SHA512 checksums.

Please follow the test procedure here:
https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release

Since 3.2.3 is based on an older release, there is a Python
compatibility issue where some tests may not work with the later
version. I used Python 3.8, Erlang 23 and Elixir 1.12.3

Please remember that "RC1" is an annotation. If the vote passes, these
artefacts will be released as Apache CouchDB 3.2.3.

Please cast your votes now.

Thanks,
-Nick

[VOTE] Release Apache CouchDB 3.3.2

[Nick Vatamaniuc]

Dear community,

I would like to propose that we release Apache CouchDB 3.3.0.

Candidate release notes:
https://github.com/apache/couchdb/blob/3.3.x/src/docs/src/whatsnew/3.3.rst

We encourage the whole community to download and test these release
artefacts so that any critical issues can be resolved before the
release is made. Everyone is free to vote on this release, so dig
right in! (Only PMC members have binding votes, but they depend on
community feedback to gauge if an official release is ready to be
made.)

The release artefacts we are voting on are available here:
https://dist.apache.org/repos/dist/dev/couchdb/source/3.3.2/rc.1/

There, you will find a tarball, a GPG signature, and SHA256/SHA512 checksums.

Please follow the test procedure here:
https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release

Please remember that "RC1" is an annotation. If the vote passes, these
artefacts will be released as Apache CouchDB 3.3.2.

Please cast your votes now.

Thanks,
-Nick

Subject: [ANNOUNCE] Ronny Berndt joins the PMC

[Nick Vatamaniuc]

Dear community,

I am delighted to announce that Ronny Berndt joins the Apache CouchDB
Project Management Committee today.

Ronny has made outstanding, sustained contributions to the project.
This appointment is an official acknowledgement of their position
within the community, and our trust in their ability to provide
oversight for the project.

Everybody, please join me in congratulating Ronny!

On behalf of the CouchDB PMC,
-Nick Vatamaniuc

Re: [DISCUSS] Covering indexes for Mango RFC

[Nick Vatamaniuc]

That looks like a nice improvement, Mike. Thanks for making the RFC.

> One question I had on this is whether RFCs should be frozen once approved, or 
> whether I should look to keep it up to date with changes in the code? The 
> Bylaws don't really cover whether they are living documents or records of 
> what the community agreed on at a point in time.

It could make sense to keep them updated with major changes to the
design, along with the documentation. They are in the main code
repository so that should make it easier to keep them in sync. We
could add version updates / notes to the RFC sections to make the
design evolution more apparent. IETF uses "updates #ref" or "obsoletes
#ref" format https://www.rfc-editor.org/rfc/rfc7322.html#section-4.1.4
with newer ones replacing the older ones. But that might work better
for a complete redesign or larger changes.

Cheers,
-Nick

On Fri, Jan 27, 2023 at 8:49 AM Mike Rhodes  wrote:
>
> All,
>
> I've added an RFC to introduce covering index functionality to Mango JSON 
> indexes. It's in a pull request at:
>
> https://github.com/apache/couchdb/pull/4410
>
> The aim is to allow users to create index and query combinations that operate 
> as fast as views when searching for data (as opposed to aggregating it via 
> reduces). It's a logical extension to my recent fields pushdown PR 
> (https://github.com/apache/couchdb/pull/4394). Overall, I'm interested in 
> addressing some of the low- and mid-hanging query performance fruit within 
> Mango.
>
> This work touches more places than the previous work, so I figured that it 
> was RFC worthy. And it'd be nice not to create a PR with some fatal flaw to 
> it :D
>
> One question I had on this is whether RFCs should be frozen once approved, or 
> whether I should look to keep it up to date with changes in the code? The 
> Bylaws don't really cover whether they are living documents or records of 
> what the community agreed on at a point in time.
>
> Anyway, cheers for having a look,
> --
> Mike.

Re: [VOTE] Release Apache CouchDB 3.3.1-RC2

[Nick Vatamaniuc]

+1

MacOS, x86-64, Erlang 23

sig: ok
checksums: ok
make check: ok
make release: ok
fauxton verify: ok
multiple fauxton replication working: ok

Cheers,
-Nick

On Fri, Jan 6, 2023 at 10:34 AM Jan Lehnardt  wrote:
>
> Convenience macOS binaries are up for arm64 and x86_64:
>
>https://dist.apache.org/repos/dist/dev/couchdb/binary/mac/3.3.1/rc.2/
>
> Best
> Jan
> —
>
> > On 6. Jan 2023, at 12:11, Jan Lehnardt  wrote:
> >
> > Dear community,
> >
> > I would like to propose that we release Apache CouchDB 3.3.1.
> >
> > Changes since RC1:
> >
> >  - allow starting of more than one replication job (D’OH)
> >
> > Candidate release notes:
> >
> >   https://docs.couchdb.org/en/latest/whatsnew/3.3.html
> >
> > We encourage the whole community to download and test these release 
> > artefacts so that any critical issues can be resolved before the release is 
> > made. Everyone is free to vote on this release, so dig right in! (Only PMC 
> > members have binding votes, but they depend on community feedback to gauge 
> > if an official release is ready to be made.)
> >
> > The release artefacts we are voting on are available here:
> >
> >   https://dist.apache.org/repos/dist/dev/couchdb/source/3.3.1/rc.2/ 
> > 
> >
> > There, you will find a tarball, a GPG signature, and SHA256/SHA512 
> > checksums.
> >
> > Please follow the test procedure here:
> >
> >   
> > https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release
> >
> > Please remember that "RC2" is an annotation. If the vote passes, these 
> > artefacts will be released as Apache CouchDB 3.3.1.
> >
> > Please cast your votes now.
> >
> > Thanks,
> > Jan
> > —
>
>

Re: [VOTE] Release Apache CouchDB 3.3.1-RC1

[Nick Vatamaniuc]

Thanks for making the RC, Jan.

The signature, checksums and make check passed. However, I was playing
with Fauxton, creating multiple replications and noticed all except
one were starting but then not making progress. So I think we'll have
to -1 to include the https://github.com/apache/couchdb/pull/4353 fix
in RC2.


On Thu, Jan 5, 2023 at 11:04 AM Jan Lehnardt  wrote:
>
> Dear community,
>
> I would like to propose that we release Apache CouchDB 3.3.1.
>
> Candidate release notes:
>
> https://docs.couchdb.org/en/latest/whatsnew/3.3.html
>
> We encourage the whole community to download and test these release artefacts 
> so that any critical issues can be resolved before the release is made. 
> Everyone is free to vote on this release, so dig right in! (Only PMC members 
> have binding votes, but they depend on community feedback to gauge if an 
> official release is ready to be made.)
>
> The release artefacts we are voting on are available here:
>
> https://dist.apache.org/repos/dist/dev/couchdb/source/3.3.1/rc.1/
>
> There, you will find a tarball, a GPG signature, and SHA256/SHA512 checksums.
>
> Please follow the test procedure here:
>
> 
> https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release
>
> Please remember that "RC1" is an annotation. If the vote passes, these 
> artefacts will be released as Apache CouchDB 3.3.1.
>
> Please cast your votes now.
>
> Thanks,
> Jan
> —

[RESULT] (Was: [VOTE] Release Apache CouchDB 3.3.0-rc.2)

[Nick Vatamaniuc]

Dear community,

The vote has now closed with a passing result.

Thank you to everyone who participated. We got a total of 7 +1 votes,
including 3 +1 PMC votes, which officially makes this a passing vote.

Thank you,
-Nick

Re: [VOTE] Release Apache CouchDB 3.3.0 (RC2)

[Nick Vatamaniuc]

My own vote: +1

Ubuntu 22.04, x86-64, Erlang 23

Sig: ok
Checksums: ok
make check: pass
make release: works
Fauxton "Verify Installation": pass


I think we need one more PMC member +1 vote to make it official.

Thanks,
-Nick



On Fri, Dec 23, 2022 at 2:07 PM Nick Vatamaniuc  wrote:
>
> Thank you, all for testing. It's looking encouraging so far!
>
> We happen to have some deb and rpm packages available as well from the
> last CI on main
> https://ci-couchdb.apache.org/blue/organizations/jenkins/jenkins-cm1%2FFullPlatformMatrix/detail/main/438/artifacts/
> Those are plain rpm and deb packages not full yum or apt repos, but it
> may be possible to use them for tests when the build dependencies are
> not present or it's tricky to set things up.
>
> Cheers!
> -Nick
>
>
> On Fri, Dec 23, 2022 at 11:39 AM Ronny Berndt  wrote:
> >
> > Hi,
> >
> > here is the missing „see below“ section:
> >
> > First run with error:
> >
> > PartitionViewUpdateTest [test/elixir/test/partition_view_update_test.exs]
> >   * test purge removes view rows (212.0ms) [L#80]
> >   * test view updates properly remove old keys (211.8ms) [L#9]
> >   * test query with update=false works (3352.6ms) [L#34]
> >
> >   1) test query with update=false works (PartitionViewUpdateTest)
> >  test/elixir/test/partition_view_update_test.exs:34
> >  ** (RuntimeError) timed out after 30371 ms
> >  code: retry_until(fn ->
> >  stacktrace:
> >(couchdbtest 0.1.0) test/elixir/lib/couch/dbtest.ex:423: 
> > Couch.DBTest.retry_until/4
> >test/elixir/test/partition_view_update_test.exs:63: (test)
> >
> >   * test purged conflict changes view rows (264.5ms) [L#108]
> >
> >
> > Running test again:
> >
> > PartitionViewUpdateTest [test/elixir/test/partition_view_update_test.exs]
> >   * test view updates properly remove old keys (888.7ms) [L#9]
> >   * test purged conflict changes view rows (315.8ms) [L#108]
> >   * test purge removes view rows (264.1ms) [L#80]
> >
> > ReshardHelpers [test/elixir/test/reshard_helpers.exs]
> >
> > PartitionHelpers [test/elixir/test/partition_helpers.exs]
> >
> >
> > Finished in 2.1 seconds (0.00s async, 2.1s sync)
> > 4 tests, 0 failures
> >
> > Merry Christmas.
> >
> > /Ronny
> >
> >
> > > Am 23.12.2022 um 14:30 schrieb Ronny Berndt :
> > >
> > > Hi,
> > >
> > > +1
> > >
> > > Windows 10 Pro / Version 22H2 / OS build 19045.2364
> > > Erlang 24.3.4.6
> > > Elixir 1.13.4
> > > Spidermonkey 91
> > >
> > > sig: ok
> > > checksums: ok
> > > make check: ok (1 test fail at first run, see below)
> > > make release: ok
> > > build msi: ok
> > > fauxton verify: ok
> > > db & doc creation: ok
> > >
> > > Great work!
> > >
> > > /Ronny
> > >
> > >
> > >>
> > >> On Thu, Dec 22, 2022 at 12:11 AM Nick Vatamaniuc 
> > >> wrote:
> > >>
> > >>> Dear community,
> > >>>
> > >>> I would like to propose that we release Apache CouchDB 3.3.0.
> > >>>
> > >>> Candidate release notes:
> > >>> https://docs.couchdb.org/en/latest/whatsnew/3.3.html
> > >>>
> > >>> Changes since RC1:
> > >>> https://github.com/apache/couchdb/compare/3.3.0-RC1...3.3.0-RC2
> > >>>
> > >>> We encourage the whole community to download and test these release
> > >>> artefacts so that any critical issues can be resolved before the
> > >>> release is made. Everyone is free to vote on this release, so dig
> > >>> right in! (Only PMC members have binding votes, but they depend on
> > >>> community feedback to gauge if an official release is ready to be
> > >>> made.)
> > >>>
> > >>> The release artefacts we are voting on are available here:
> > >>> https://dist.apache.org/repos/dist/dev/couchdb/source/3.3.0/rc.2/
> > >>>
> > >>> There, you will find a tarball, a GPG signature, and SHA256/SHA512
> > >>> checksums.
> > >>>
> > >>> Please follow the test procedure here:
> > >>>
> > >>> https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release
> > >>>
> > >>> Please remember that "RC2" is an annotation. If the vote passes, these
> > >>> artefacts will be released as Apache CouchDB 3.3.0.
> > >>>
> > >>> Please cast your votes now.
> > >>>
> > >>> Thanks,
> > >>> -Nick
> > >>>
> > >
> >

Re: [VOTE] Release Apache CouchDB 3.3.0 (RC2)

[Nick Vatamaniuc]

Thank you, all for testing. It's looking encouraging so far!

We happen to have some deb and rpm packages available as well from the
last CI on main
https://ci-couchdb.apache.org/blue/organizations/jenkins/jenkins-cm1%2FFullPlatformMatrix/detail/main/438/artifacts/
Those are plain rpm and deb packages not full yum or apt repos, but it
may be possible to use them for tests when the build dependencies are
not present or it's tricky to set things up.

Cheers!
-Nick


On Fri, Dec 23, 2022 at 11:39 AM Ronny Berndt  wrote:
>
> Hi,
>
> here is the missing „see below“ section:
>
> First run with error:
>
> PartitionViewUpdateTest [test/elixir/test/partition_view_update_test.exs]
>   * test purge removes view rows (212.0ms) [L#80]
>   * test view updates properly remove old keys (211.8ms) [L#9]
>   * test query with update=false works (3352.6ms) [L#34]
>
>   1) test query with update=false works (PartitionViewUpdateTest)
>  test/elixir/test/partition_view_update_test.exs:34
>  ** (RuntimeError) timed out after 30371 ms
>  code: retry_until(fn ->
>  stacktrace:
>(couchdbtest 0.1.0) test/elixir/lib/couch/dbtest.ex:423: 
> Couch.DBTest.retry_until/4
>test/elixir/test/partition_view_update_test.exs:63: (test)
>
>   * test purged conflict changes view rows (264.5ms) [L#108]
>
>
> Running test again:
>
> PartitionViewUpdateTest [test/elixir/test/partition_view_update_test.exs]
>   * test view updates properly remove old keys (888.7ms) [L#9]
>   * test purged conflict changes view rows (315.8ms) [L#108]
>   * test purge removes view rows (264.1ms) [L#80]
>
> ReshardHelpers [test/elixir/test/reshard_helpers.exs]
>
> PartitionHelpers [test/elixir/test/partition_helpers.exs]
>
>
> Finished in 2.1 seconds (0.00s async, 2.1s sync)
> 4 tests, 0 failures
>
> Merry Christmas.
>
> /Ronny
>
>
> > Am 23.12.2022 um 14:30 schrieb Ronny Berndt :
> >
> > Hi,
> >
> > +1
> >
> > Windows 10 Pro / Version 22H2 / OS build 19045.2364
> > Erlang 24.3.4.6
> > Elixir 1.13.4
> > Spidermonkey 91
> >
> > sig: ok
> > checksums: ok
> > make check: ok (1 test fail at first run, see below)
> > make release: ok
> > build msi: ok
> > fauxton verify: ok
> > db & doc creation: ok
> >
> > Great work!
> >
> > /Ronny
> >
> >
> >>
> >> On Thu, Dec 22, 2022 at 12:11 AM Nick Vatamaniuc 
> >> wrote:
> >>
> >>> Dear community,
> >>>
> >>> I would like to propose that we release Apache CouchDB 3.3.0.
> >>>
> >>> Candidate release notes:
> >>> https://docs.couchdb.org/en/latest/whatsnew/3.3.html
> >>>
> >>> Changes since RC1:
> >>> https://github.com/apache/couchdb/compare/3.3.0-RC1...3.3.0-RC2
> >>>
> >>> We encourage the whole community to download and test these release
> >>> artefacts so that any critical issues can be resolved before the
> >>> release is made. Everyone is free to vote on this release, so dig
> >>> right in! (Only PMC members have binding votes, but they depend on
> >>> community feedback to gauge if an official release is ready to be
> >>> made.)
> >>>
> >>> The release artefacts we are voting on are available here:
> >>> https://dist.apache.org/repos/dist/dev/couchdb/source/3.3.0/rc.2/
> >>>
> >>> There, you will find a tarball, a GPG signature, and SHA256/SHA512
> >>> checksums.
> >>>
> >>> Please follow the test procedure here:
> >>>
> >>> https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release
> >>>
> >>> Please remember that "RC2" is an annotation. If the vote passes, these
> >>> artefacts will be released as Apache CouchDB 3.3.0.
> >>>
> >>> Please cast your votes now.
> >>>
> >>> Thanks,
> >>> -Nick
> >>>
> >
>

[VOTE] Release Apache CouchDB 3.3.0 (RC2)

[Nick Vatamaniuc]

Dear community,

I would like to propose that we release Apache CouchDB 3.3.0.

Candidate release notes:
https://docs.couchdb.org/en/latest/whatsnew/3.3.html

Changes since RC1:
https://github.com/apache/couchdb/compare/3.3.0-RC1...3.3.0-RC2

We encourage the whole community to download and test these release
artefacts so that any critical issues can be resolved before the
release is made. Everyone is free to vote on this release, so dig
right in! (Only PMC members have binding votes, but they depend on
community feedback to gauge if an official release is ready to be
made.)

The release artefacts we are voting on are available here:
https://dist.apache.org/repos/dist/dev/couchdb/source/3.3.0/rc.2/

There, you will find a tarball, a GPG signature, and SHA256/SHA512 checksums.

Please follow the test procedure here:
https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release

Please remember that "RC2" is an annotation. If the vote passes, these
artefacts will be released as Apache CouchDB 3.3.0.

Please cast your votes now.

Thanks,
-Nick

Re: Subject: [VOTE] Release Apache CouchDB 3.3.0

[Nick Vatamaniuc]

Hi everyone,

The vote failed. We will create an RC2 shortly.

Thanks Jan and Ronny for making great progress to make the Windows build work.

Thanks to Johannes for trying out the release and discovering network
isolation issues. We should be able to build the release again from
the dist tarball without needing any network access. Running the test
suite (make check) still requires a network for now.

Thanks to Juanjo for jumping and assisting us with the Elixir test suite.

Cheers,
-Nick

On Mon, Dec 19, 2022 at 1:08 PM Jan Lehnardt  wrote:
>
> Hey all,
>
> > On 17. Dec 2022, at 19:06, Johannes J. Schmidt 
> >  wrote:
> >
> > - Fauxton verify install: works mostly, except for the Replication test, 
> > see https://github.com/apache/couchdb/issues/4313
>
>
> This has been fixed: https://github.com/apache/couchdb/pull/4317
>
>
> * * *
>
> > On 19. Dec 2022, at 11:18, Ronny Berndt  wrote:
>
>
>
> -1 for the Windows build.
>
> Good find! — We tracked this down to jiffy getting rebar3 support in version 
> 1.1.0 and fixes to that support in 1.1.1. We shipped our last release with 
> jiffy 1.0.9 which directly preceded 1.1.0 (tho our tag is CouchDB-1.0.9). As 
> far as I can tell there are no substantial fixes* in 1.1.0+ so we might as 
> well unblock our release by reverting to jiffy 1.0.9.
>
> * https://github.com/davisp/jiffy/compare/1.0.9...1.1.1
>
> PR here: https://github.com/apache/couchdb/pull/4320
>
> * * *
>
> Best
> Jan
> —
>
>
> > Johannes
> >
> >
> > --- Original Message ---
> > On Saturday, December 17th, 2022 at 07:04, Nick Vatamaniuc 
> >  wrote:
> >
> >
> >>
> >>
> >> My own +1
> >>
> >> Sig: ok
> >> Checksums: ok
> >> make check: passes
> >> make release: works
> >> Fauxton verify install: works
> >> Creating database and docs in Fauxton: works
> >>
> >> Tested on macos 12, x86-64, erlang 23, spidermonkey 1.8.5
> >>
> >> Cheers,
> >> -Nick
> >>
> >> On Sat, Dec 17, 2022 at 12:16 AM Nick Vatamaniuc vatam...@apache.org wrote:
> >>
> >>> Dear community,
> >>>
> >>> I would like to propose that we release Apache CouchDB 3.3.0.
> >>>
> >>> Candidate release notes:
> >>> https://docs.couchdb.org/en/latest/whatsnew/3.3.html
> >>>
> >>> We encourage the whole community to download and test these release
> >>> artefacts so that any critical issues can be resolved before the
> >>> release is made. Everyone is free to vote on this release, so dig
> >>> right in! (Only PMC members have binding votes, but they depend on
> >>> community feedback to gauge if an official release is ready to be
> >>> made.)
> >>>
> >>> The release artefacts we are voting on are available here:
> >>> https://dist.apache.org/repos/dist/dev/couchdb/source/3.3.0/rc.1/
> >>>
> >>> There, you will find a tarball, a GPG signature, and SHA256/SHA512 
> >>> checksums.
> >>>
> >>> Please follow the test procedure here:
> >>> https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release
> >>>
> >>> Please remember that "RC1" is an annotation. If the vote passes, these
> >>> artefacts will be released as Apache CouchDB 3.3.0.
> >>>
> >>> Please cast your votes now.
> >>>
> >>> Thanks,
> >>> -Nick
>

Re: Subject: [VOTE] Release Apache CouchDB 3.3.0

[Nick Vatamaniuc]

My own +1

Sig: ok
Checksums: ok
make check: passes
make release: works
Fauxton verify install: works
Creating database and docs in Fauxton: works

Tested on macos 12, x86-64, erlang 23, spidermonkey 1.8.5

Cheers,
-Nick

On Sat, Dec 17, 2022 at 12:16 AM Nick Vatamaniuc  wrote:
>
> Dear community,
>
> I would like to propose that we release Apache CouchDB 3.3.0.
>
> Candidate release notes:
> https://docs.couchdb.org/en/latest/whatsnew/3.3.html
>
> We encourage the whole community to download and test these release
> artefacts so that any critical issues can be resolved before the
> release is made. Everyone is free to vote on this release, so dig
> right in! (Only PMC members have binding votes, but they depend on
> community feedback to gauge if an official release is ready to be
> made.)
>
> The release artefacts we are voting on are available here:
> https://dist.apache.org/repos/dist/dev/couchdb/source/3.3.0/rc.1/
>
> There, you will find a tarball, a GPG signature, and SHA256/SHA512 checksums.
>
> Please follow the test procedure here:
> https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release
>
> Please remember that "RC1" is an annotation. If the vote passes, these
> artefacts will be released as Apache CouchDB 3.3.0.
>
> Please cast your votes now.
>
> Thanks,
> -Nick

Subject: [VOTE] Release Apache CouchDB 3.3.0

[Nick Vatamaniuc]

Dear community,

I would like to propose that we release Apache CouchDB 3.3.0.

Candidate release notes:
https://docs.couchdb.org/en/latest/whatsnew/3.3.html

We encourage the whole community to download and test these release
artefacts so that any critical issues can be resolved before the
release is made. Everyone is free to vote on this release, so dig
right in! (Only PMC members have binding votes, but they depend on
community feedback to gauge if an official release is ready to be
made.)

The release artefacts we are voting on are available here:
https://dist.apache.org/repos/dist/dev/couchdb/source/3.3.0/rc.1/

There, you will find a tarball, a GPG signature, and SHA256/SHA512 checksums.

Please follow the test procedure here:
https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release

Please remember that "RC1" is an annotation. If the vote passes, these
artefacts will be released as Apache CouchDB 3.3.0.

Please cast your votes now.

Thanks,
-Nick

[NOTICE] CouchDB 3.3 release this month

[Nick Vatamaniuc]

Dear community,

This is advance notice that we plan to do a 3.3 release this month.

Please use this time to review any outstanding changes to be made,
merging them in as necessary. The current list of changes we expect to
land for this release is here:


https://github.com/apache/couchdb/issues?q=is%3Aopen+is%3Aissue+milestone%3A3.3.0

The current release notes draft is here:

https://github.com/apache/couchdb/pull/4257

There are few more pending changes to the whatsnew from recent PRs
which just merged, but the bulk of the release notes should be there
(thanks doing those Jan and Ronny!)

Please read this through and suggest any changes.

Thanks,
-Nick

Re: [DISCUSS] Integrate couchdb-config into the main repo

[Nick Vatamaniuc]

Good idea, Ronny. And thanks for providing the summary!

+1 to integrate b64url, khash, and ets_lru as well

Cheers,
-Nick

On Wed, Oct 26, 2022 at 12:12 PM Ronny Berndt  wrote:
>
> Great work, thank you for this Nick!
>
> What about hose candidates too?
>
> b64url,
> ets_lru
> khash
>
>
> Original discussion at Slack (because of the 90 days limit of slack,
> I will post the part of the discussion):
>
> vatamane
>   21:51 Uhr Noticed it was kind of a pain to manage the separate config 
> app and wondering it's it's worth keeping at a separate repo
>   21:51 Uhr Sent a question to the mailing list about it
>   21:52 Uhr Waiting on travis CI to pick up the job and start running it 
> kind of was the trigger for it
>   21:52 Uhr that and the extra tag + extra PR, all for a minor change in 
> the code...
>
>
> big-r
>   22:15 Uhr Can this discussion be extended to other "external" couchdb-* 
> repositories?
>   22:17 Uhr What was the main reason to separate them and are they used 
> by external apps?
>
>
> jaydoane
>   23:09 Uhr you mean these?
> DepDescs = [
> %% Independent Apps
> {config,   "config",   {tag, 
> "2.2.0"}},
> {b64url,   "b64url",   {tag, 
> "1.0.3"}},
> {ets_lru,  "ets-lru",  {tag, 
> "1.1.0"}},
> {khash,"khash",{tag, 
> "1.1.0"}},
> {snappy,   "snappy",   {tag, 
> "CouchDB-1.0.8"}},
>
> %% %% Non-Erlang deps
> {fauxton,  {url, 
> "https://github.com/apache/couchdb-fauxton"},
> {tag, "v1.2.8"}, [raw]},
>
> %% Third party deps
> {folsom,   "folsom",   {tag, 
> "CouchDB-0.8.4"}},
> {hyper,"hyper",{tag, 
> "CouchDB-2.2.0-7"}},
> {ibrowse,  "ibrowse",  {tag, 
> "CouchDB-4.4.2-5"}},
> {jiffy,"jiffy",{tag, 
> "1.1.1"}},
> {mochiweb, "mochiweb", {tag, 
> "v3.1.0"}},
> {meck, "meck", {tag, 
> "0.9.2"}},
> {recon,"recon",{tag, 
> "2.5.2"}}
> ].
>
> big-r
>   23:27 Uhr Yes, the first part…
>
> jaydoane
>   23:30 Uhr I suspect in general they were left separate after the Great 
> Unification because they could be used independently of CouchDB,
> but the headaches caused by keeping them separate may 
> not be worth it, as Nick pointed out
>
> vatamane
>   23:54 Uhr khash and b64url could potentially be used externally, they 
> are less tied to CouchDB
>   23:55 Uhr khash though could probably be replaced with maps these days
>   23:56 Uhr ets_lru is a single .erl file
>   23:57 Uhr that could be in couch_util or something
>   23:59 Uhr.@big-r yeah good idea, I can see ets_lru, khash and 
> b64url as good candidates
>
>
> vatamane
>   00:01 Uhr b64url is also being replaced by the otp base64 url + an 
> urlsafe alphabet too, there is a good chance users would just pick that
> 
> https://github.com/erlang/otp/commit/05e61dc7eb568cc5a5db965dcc3534fb6c9aa66d 
> eventually
> (not that I know if anyone our b64url anyway) 
> (bearbeitet)
>
> rnewson
>   10:13 Uhr it's partly that they can be used without couchdb but also an 
> attempt to preserve that decoupling. we've been back and forth on it.
> It seems pretty clear that no one is interested in 
> using those components separately so I'm onboard with folding them back into 
> the main repo
>
>
> /Ronny
>
> > Am 26.10.2022 um 17:39 schrieb Nick Vatamaniuc :
> >
> > Thanks for the feedback, everyone.
> >
> > config application has now been integrated into main in
> > https://github.com/apache/couchdb/pull/4242
> >
> >

Re: [DISCUSS] Integrate couchdb-config into the main repo

[Nick Vatamaniuc]

Thanks for the feedback, everyone.

config application has now been integrated into main in
https://github.com/apache/couchdb/pull/4242


On Tue, Oct 4, 2022 at 3:23 AM Jan Lehnardt  wrote:
>
> No objections.
>
> Best
> Jan
>
> —
> Professional Support for Apache CouchDB:
> https://neighbourhood.ie/couchdb-support/
>
> 24/7 Observation for your CouchDB Instances:
> https://opservatory.app
>
> > On 28. Sep 2022, at 21:32, Nick Vatamaniuc  wrote:
> >
> > Hi everyone,
> >
> > What do we think about integrating couchdb-config into the main couchdb 
> > repo?
> >
> > Currently it's a separate repo, with a different CI (Travis), which
> > takes quite a bit of time waiting to schedule runs, has a different
> > configuration file etc. Any updates to config needs a new tag and a
> > corresponding bump and a separate PR in the main repo.
> >
> > Over the years it has been intermingled with couch_log and other
> > CouchDB bits and that wouldn't make it as appealing to use as a
> > separate application, so let's at least make life easier for us? What
> > does everyone think?
> >
> > Thanks,
> > -Nick
>

[DISCUSS] Integrate couchdb-config into the main repo

[Nick Vatamaniuc]

Hi everyone,

What do we think about integrating couchdb-config into the main couchdb repo?

Currently it's a separate repo, with a different CI (Travis), which
takes quite a bit of time waiting to schedule runs, has a different
configuration file etc. Any updates to config needs a new tag and a
corresponding bump and a separate PR in the main repo.

Over the years it has been intermingled with couch_log and other
CouchDB bits and that wouldn't make it as appealing to use as a
separate application, so let's at least make life easier for us? What
does everyone think?

Thanks,
-Nick

[ANNOUNCE] CouchDB docs are now in the main repo

[Nick Vatamaniuc]

Hi everyone,

Just a quick announcement that CouchDB docs are now part of the main
CouchDB repo in src/docs as discussed in
https://lists.apache.org/thread/x4lc6vhthj1vkt2xpd0ox5osh959qsc4. The
other major change was upgrading to the latest Shinx 5.1.1 package.

The two relevant PRs are:
  * https://github.com/apache/couchdb/pull/4164
  * https://github.com/apache/couchdb/pull/4174

One of the objections to integrating the docs was the docs CI ran much
quicker, and we wouldn't want to lose that. It turns out we were able
to "fast-forward" the CI for docs only changes. How that happens can
be seen in https://github.com/apache/couchdb/pull/4164#issuecomment-1239442611

Some of the benefits of this change are:
   * Easier to create feature PRs: it can be a single commit, updating
both the code and the docs.
   * Easier to synchronize docs and code changes, especially during
the release process.
   * Faster ./configure runs, as it doesn't have to clone a separate docs repo.
   * Easier to set up and run doc builds. There is a `./setup.sh`
script which should create the python virtualenv and install all the
requirements.
   * Some of the sphinx extensions have been cleaned up: for instance,
httpdomain is a proper pip dependency now, as opposed being vendored
in.
   * "Edit on Github'' link on the documentation front page now works
and points to the src/docs readme file.

Thanks to Ronny, Jay, Jan, and everyone else who participated for
their help with ideas and PR reviews!

Cheers,
-Nick

Re: CouchDB Code Reload and Debugging with Erlang LS Extension

[Nick Vatamaniuc]

Thanks for sharing, Zach

I gave it a try and it worked as advertised.

Found that clicking on the "fringe" next to the code line
automatically sets a breakpoint so it doesn't need a manual M-x
dap-breakpoint-add!

Cheers,
-Nick

On Thu, Sep 1, 2022 at 9:36 PM Zach Lankton  wrote:
>
> Hello Team,
>
> I am relatively new to contributing to the CouchDB code base and have been
> working primarily on improving the VSCode Devcontainer and some beginner
> bugs.
>
> Recently, I've been working on getting code reloading and debugging to work
> with the CouchDB code base using the Erlang LS extension.  I started with
> the VSCode DevContainer but recently challenged myself to learn enough
> emacs to get this working there as well.
>
> My overall goal is to improve the onboarding of new developers to the code
> base.  I really like the (mostly) one-click launch of the dev container and
> how quickly you can be working with the code without all the hassle and
> heartache of the traditional method of setting up a dev environment and
> installing all the dependencies.  Additionally, this makes for excellent
> isolation from your main host machine and can be thrown away easily when
> you are finished.  I plan to continue making improvements in this regard
> while learning my way around the project and Erlang in general.
>
> In the meantime, for those interested in trying out code reloading and
> debugging with the erlang_ls extension, I have written instructions on
> getting started.  These instructions are for Ubuntu and emacs but should be
> similar, if not identical, for other OS's and IDE's.
>
> I would love to hear from anyone trying this out!
>
> https://gist.github.com/zachlankton/e3300672a75c8ed7fd5b77093c0e8738
>
> --
> Thanks!
> Zach Lankton

Re: [DISCUSS] Should roles_claim_name be marked as deprecated and replaced in favor of roles_claim_path

[Nick Vatamaniuc]

If RCPs have all the functionality of RCNs, I think it makes sense to
deprecate RCNs.

On Tue, Aug 30, 2022 at 9:02 AM Ronny Berndt  wrote:
>
> Overview
>
> In a JWT token it is possible to add an attribute for role claims.
> If the roles are presented as top-level attribute like
>
> {
>   "couchdb-roles": [
> "my_role_1",
> "my_role_2"
>   ]
> }
>
> and setting the parameter roles_claim_name in the config file to
>
> [jwt_auth]
> roles_claim_name = couchdb-roles
>
> CouchDB was able to read that attributed and take over that roles.
> This doesn't work, if the claim roles are nested, eg:
>
> {
>   "my" :{
> "nested": {
>"couchdb-roles": [
>  "my_role_1",
>  "my_role_2"
>]
> }
>   }
> }
>
> To allow this and for backwards compatibility, a new config parameter 
> `roles_claim_path`
> is introduced to allow nested role claims. To allow the example from above, 
> yo can use
> the following syntax:
>
> [jwt_auth]
> roles_claim_path = my.nested.couchdb-roles
>
> It is now possible to specify nested (& unnested "\." prevents interpreting 
> as nested)
> JSON paths in role_claim_path, like
> roles_claim_path = foo.bar\.zonk.baz\.buu.baa.baa\.bee.roles which is 
> equivalent to
>
> "foo": {
>   "bar.zonk": {
> "baz.buu": {
>   "baa": {
> "baa.bee": {
>   "roles": [
> "my_nested_role_1",
> "my_nested_role_2"
>   ]
> }
>   }
> }
>   }
> }
>
> After merging the functionality with PR#4041 [3], I wanted to know, if we 
> should
> deprecated the existing parameter `roles_claim_name` (RCN) in favor of 
> `roles_claim_path`
> (RCP). RCP has all the functionality of RCN plus it allows nested & unnested 
> JWT JSON
> role claims. If `roles_claim_path` is defined, then `roles_claim_name` is 
> ignored.
>
> In the docs PR#737, I already "marked" RCN as deprecated in the next version 
> (3.3),
> because RCP acts as successor of RCN. We could also remove the deprecation 
> note and
> have both options in CouchDB with overlapping functionality.
>
> What do you think?
>
> Related Issues or Pull Requests:
>
> #3758 [1]
> #3166 [2]
>
> [1] https://github.com/apache/couchdb/issues/3758
> [2] https://github.com/apache/couchdb/pull/3166
> [3] https://github.com/apache/couchdb/pull/4041
> [4] https://github.com/apache/couchdb-documentation/pull/737

[DISCUSSION] _bulk_get support for the replicator

[Nick Vatamaniuc]

Hi everyone,

I implemented _bulk_get support for the replicator in
https://github.com/apache/couchdb/pull/4144. I tried to make it as
backwards compatible as possible to replicate with older endpoints and
PouchDB. Some preliminary benchmarks showed a decent speedup of 30% or
so.

There is a silly chart in the PR comments showing the effect of the
exponential moving average to make it less mysterious where some of
those constants came from. I had opted to make some of those constants
hard coded, to avoid adding even more replication options, but can be
convinced otherwise. Feel free to suggest or comment in the PR.

Also, to add some visibility regarding _bulk_get effectiveness I added
two replication stats fields. Not sure I picked the best names, let me
know if you have any better suggestions or perhaps not have them at
all?

Thanks,
-Nick

[DISCUSSION] Make Erlang 23 the minimum supported distribution

[Nick Vatamaniuc]

Hello everyone,

I'd like to propose making Erlang 23 the minimum supported Erlang
distribution. We have accumulated a lot of ifdefs and other cruft
supporting various APIs and syntactic constructs since Erlang 20. With
Erlang 25 just released it could be a good time to do some cleanup.

We could remove ifdefs for crypto, time functions, the non-syntactic
get_stacktrace() macro, as well as have access to nifty new
functionality like persistent terms, atomic counters and a few other
things.

If we want to switch to rebar3, that would also be easier with Erlang
23 being the minimum.

What do we think?

Thanks,
-Nick

Re: [DISCUSSION] Rename 3.x branch to main and include docs in the main repo

[Nick Vatamaniuc]

It looks like we could do the branch switch fairly easily. Here is what I did:

1) Disabled main protection in .asf.yaml for main branch
2) Pushed fdbmain as branch off of main
3) Force pushed 3.x to replace main (since protection was removed)
4) Re-enabled protection for main and fdbmain
5) Replaced old 3.x branch with a single readme marker file

If you have any PRs against 3.x, rebase them on top of main
If you have any PRs against main, rebase them on top of fdbmain

Cheers,
-Nick


On Mon, Jun 6, 2022 at 12:07 PM Nick Vatamaniuc  wrote:
>
> > I would like to suggest that we explore options with automation to try
> to help remember all doc changes for a PR. I.e. the checkbox in the PR
> template can help, but maybe we could also auto-label PRs with
> "no-doc-changes" to remind reviewers, and/or have triggers for PRs
> that do and don't touch files under docs?
>
> That's an interesting idea, Will. Something like that could work. Add
> a PR or commit label in Github perhaps? Usually what happens is that
> the docs PR follows the main code PR. So first the main PR is reviewed
> / merged, then the contributor would remember to make a separate docs
> PR. But sometimes (including myself quite often) we forget to do that.
> That's why I thought keeping it in one pull request would make it a
> bit easier to not forget and keep both exactly in sync.
>
> > Is it possible to distinguish what has been changed in a repository?
>
> That could work, Ronny! In a pull request CI job we can check the diff
> between that pull request and the branch we're merging into (main,
> 3.x). Then, see if there are any file changes to src/docs/* and
> trigger or skip various jenkins stages based on that. So if changes
> are only to docs, just run the docs CI quickly and skip the whole code
> CI. This would be for pull requests CI only. The full CI, which runs
> after the merge to main, already runs docs + fauxton + CI builds
> anyway.
>
> Regards,
> -Nick
>
>
> On Sat, Jun 4, 2022 at 2:07 PM Nick Vatamaniuc  wrote:
> >
> > Looking at https://docs.readthedocs.io/en/stable/versions.html, they
> > recognize an explicit "stable" tag or just picks the latest semantic
> > version branch or tag. They use https://peps.python.org/pep-0440/
> > rules to parse and that allows for a ".postN" suffix for a tag/branch.
> > So we could have a 3.2.2.post1 branch if we don't want to maintain a
> > stable tag. We could even create that branch on demand from a 3.2.2
> > tag when we have docs updates after 3.2.2 is out.
> >
> > This would work with either scheme - docs in the main repo or separate
> > as they are now.  The idea of keeping them in the main couchdb repo
> > (provided we can sort out the CI so usability of docs-only vs
> > code-only PRs doesn't suffer), would just keep the extra branching and
> > cherry-picking in one repo only.
> >
> > Cheers,
> > -Nick
> >
> >
> >
> > On Sat, Jun 4, 2022 at 5:04 AM Jan Lehnardt  wrote:
> > >
> > > Heya Nick,
> > >
> > > thanks for taking my raw idea to the larger group here :)
> > >
> > > The one other point that started the whole thing is our publishing of 
> > > docs.
> > >
> > > We have a stable and a latest tag where latest just tracks the main branch
> > > on the docs repo and stable we move manually when a new release comes out.
> > >
> > > What does happen though is that after a release, say 3.2.2, we improve the
> > > documentation for something that is valid for 3.2.2. If there are no other
> > > changes, we can choose to move the stable tag to that commit and get
> > > improved documentation as stable for all users online, even though that
> > > strictly didn’t exist at the time of the 3.2.2 release.
> > >
> > > Now, if we first add a new feature in the 3.x line on the couchdb repo and
> > > document that in the main branch of the docs repo, and then the docs
> > > improvement from before, we can not move the stable tag on the docs repo
> > > as that would publish the docs for a not-yet-released feature and the
> > > improvement is not available  until we release whatever comes after 3.2.2,
> > > which might be a considerable time later.
> > >
> > > My thinking was that it might be neat to follow the branch structure of 
> > > the
> > > couchdb repo in the docs repo, so that we can make this more easily, and
> > > from there we got to moving docs into the main repo to avoid double
> > > bookkeeping, but that still doesn’t help us in the second scenario 
> > > outlined
> &

Re: [DISCUSSION] Rename 3.x branch to main and include docs in the main repo

[Nick Vatamaniuc]

> I would like to suggest that we explore options with automation to try
to help remember all doc changes for a PR. I.e. the checkbox in the PR
template can help, but maybe we could also auto-label PRs with
"no-doc-changes" to remind reviewers, and/or have triggers for PRs
that do and don't touch files under docs?

That's an interesting idea, Will. Something like that could work. Add
a PR or commit label in Github perhaps? Usually what happens is that
the docs PR follows the main code PR. So first the main PR is reviewed
/ merged, then the contributor would remember to make a separate docs
PR. But sometimes (including myself quite often) we forget to do that.
That's why I thought keeping it in one pull request would make it a
bit easier to not forget and keep both exactly in sync.

> Is it possible to distinguish what has been changed in a repository?

That could work, Ronny! In a pull request CI job we can check the diff
between that pull request and the branch we're merging into (main,
3.x). Then, see if there are any file changes to src/docs/* and
trigger or skip various jenkins stages based on that. So if changes
are only to docs, just run the docs CI quickly and skip the whole code
CI. This would be for pull requests CI only. The full CI, which runs
after the merge to main, already runs docs + fauxton + CI builds
anyway.

Regards,
-Nick


On Sat, Jun 4, 2022 at 2:07 PM Nick Vatamaniuc  wrote:
>
> Looking at https://docs.readthedocs.io/en/stable/versions.html, they
> recognize an explicit "stable" tag or just picks the latest semantic
> version branch or tag. They use https://peps.python.org/pep-0440/
> rules to parse and that allows for a ".postN" suffix for a tag/branch.
> So we could have a 3.2.2.post1 branch if we don't want to maintain a
> stable tag. We could even create that branch on demand from a 3.2.2
> tag when we have docs updates after 3.2.2 is out.
>
> This would work with either scheme - docs in the main repo or separate
> as they are now.  The idea of keeping them in the main couchdb repo
> (provided we can sort out the CI so usability of docs-only vs
> code-only PRs doesn't suffer), would just keep the extra branching and
> cherry-picking in one repo only.
>
> Cheers,
> -Nick
>
>
>
> On Sat, Jun 4, 2022 at 5:04 AM Jan Lehnardt  wrote:
> >
> > Heya Nick,
> >
> > thanks for taking my raw idea to the larger group here :)
> >
> > The one other point that started the whole thing is our publishing of docs.
> >
> > We have a stable and a latest tag where latest just tracks the main branch
> > on the docs repo and stable we move manually when a new release comes out.
> >
> > What does happen though is that after a release, say 3.2.2, we improve the
> > documentation for something that is valid for 3.2.2. If there are no other
> > changes, we can choose to move the stable tag to that commit and get
> > improved documentation as stable for all users online, even though that
> > strictly didn’t exist at the time of the 3.2.2 release.
> >
> > Now, if we first add a new feature in the 3.x line on the couchdb repo and
> > document that in the main branch of the docs repo, and then the docs
> > improvement from before, we can not move the stable tag on the docs repo
> > as that would publish the docs for a not-yet-released feature and the
> > improvement is not available  until we release whatever comes after 3.2.2,
> > which might be a considerable time later.
> >
> > My thinking was that it might be neat to follow the branch structure of the
> > couchdb repo in the docs repo, so that we can make this more easily, and
> > from there we got to moving docs into the main repo to avoid double
> > bookkeeping, but that still doesn’t help us in the second scenario outlined
> > above.
> >
> > To make that work we’d need to have a post-release branch for each release
> > that we can merge improvement-only doc commits to and that we can freely
> > move the stable docs tag forward with. It doesn’t matter if we do this on
> > the docs or the couchdb repo with included docs, but it means a little more
> > bookkeeping with branches:
> >
> > - all doc commits go into release branches (main/3.x etc)
> > - improvement-only commits are also merged into a hypothetical
> >   3.x-post-release-doc-commits[1] branch
> >
> > To me this feels like a little overhead for a nice gain, but I’d like to
> > hear from y’all if this is worth it.
> >
> > * * *
> >
> > I’m am +1 on moving 3.x to main and moving main to fdbmain in the couchdb 
> > repo.
> >
> > I have no strong opinion on moving docs into the main repo, as that doesn’t
> > solve 

Re: [DISCUSSION] Rename 3.x branch to main and include docs in the main repo

[Nick Vatamaniuc]

Looking at https://docs.readthedocs.io/en/stable/versions.html, they
recognize an explicit "stable" tag or just picks the latest semantic
version branch or tag. They use https://peps.python.org/pep-0440/
rules to parse and that allows for a ".postN" suffix for a tag/branch.
So we could have a 3.2.2.post1 branch if we don't want to maintain a
stable tag. We could even create that branch on demand from a 3.2.2
tag when we have docs updates after 3.2.2 is out.

This would work with either scheme - docs in the main repo or separate
as they are now.  The idea of keeping them in the main couchdb repo
(provided we can sort out the CI so usability of docs-only vs
code-only PRs doesn't suffer), would just keep the extra branching and
cherry-picking in one repo only.

Cheers,
-Nick



On Sat, Jun 4, 2022 at 5:04 AM Jan Lehnardt  wrote:
>
> Heya Nick,
>
> thanks for taking my raw idea to the larger group here :)
>
> The one other point that started the whole thing is our publishing of docs.
>
> We have a stable and a latest tag where latest just tracks the main branch
> on the docs repo and stable we move manually when a new release comes out.
>
> What does happen though is that after a release, say 3.2.2, we improve the
> documentation for something that is valid for 3.2.2. If there are no other
> changes, we can choose to move the stable tag to that commit and get
> improved documentation as stable for all users online, even though that
> strictly didn’t exist at the time of the 3.2.2 release.
>
> Now, if we first add a new feature in the 3.x line on the couchdb repo and
> document that in the main branch of the docs repo, and then the docs
> improvement from before, we can not move the stable tag on the docs repo
> as that would publish the docs for a not-yet-released feature and the
> improvement is not available  until we release whatever comes after 3.2.2,
> which might be a considerable time later.
>
> My thinking was that it might be neat to follow the branch structure of the
> couchdb repo in the docs repo, so that we can make this more easily, and
> from there we got to moving docs into the main repo to avoid double
> bookkeeping, but that still doesn’t help us in the second scenario outlined
> above.
>
> To make that work we’d need to have a post-release branch for each release
> that we can merge improvement-only doc commits to and that we can freely
> move the stable docs tag forward with. It doesn’t matter if we do this on
> the docs or the couchdb repo with included docs, but it means a little more
> bookkeeping with branches:
>
> - all doc commits go into release branches (main/3.x etc)
> - improvement-only commits are also merged into a hypothetical
>   3.x-post-release-doc-commits[1] branch
>
> To me this feels like a little overhead for a nice gain, but I’d like to
> hear from y’all if this is worth it.
>
> * * *
>
> I’m am +1 on moving 3.x to main and moving main to fdbmain in the couchdb 
> repo.
>
> I have no strong opinion on moving docs into the main repo, as that doesn’t
> solve the problem I’m most interested in, and with adding complexity about CI
> builds, that might not be worth the hassle, but I’m also not stopping anyone
> from putting in the work :)
>
> [1]: ridiculously long name chosen to avoid getting distracted with 
> bikeshedding.
>
> Best
> Jan
> —
> Professional Support for Apache CouchDB:
> https://neighbourhood.ie/couchdb-support/
>
> *24/7 Observation for your CouchDB Instances:
> https://opservatory.app
>
> > On 2. Jun 2022, at 21:40, Nick Vatamaniuc  wrote:
> >
> > Hi everyone,
> >
> > In a #dev slack thread we were discussing improvements to how we tag
> > our documentation repo. There were two proposals to simplify the
> > development and release process, and we wanted to see what the rest of
> > the community thought about it. Those two ideas are:
> >
> > 1. Rename couchdb repo's main branch to fdbmain and rename 3.x to main.
> >
> > From an ergonomic point of view, there is more development on the 3.x
> > branch so having it as main makes more sense. It can help with:
> >   * Issues auto-closing when PRs are merged
> >   * Github code search works better in the default branch
> >
> > 2. Move docs to the main repo.
> >
> > We noticed that the docs repo tags/branches can get out-of-sync with
> > the main couchdb repo. We have been merging features in main when they
> > apply only to 3.2.x and it requires care to keep track of what gets
> > merged and ported between branches. The idea is to simplify and make
> > it automatic so docs always follow the main repo so merging and
> > porting happens in one place only.
> >
> > What does everyone think?
> >
> > Thanks,
> > -Nick
>

Re: [DISCUSSION] Rename 3.x branch to main and include docs in the main repo

[Nick Vatamaniuc]

Good point about the CI, Jonathan. I think if we do this there might
be a way to skip the full CI if only docs changed.

As for historical reason, others would have to correct me here, but I
think there was a trend to split projects into smaller repositories,
one per Erlang application (roughly).  Then, eventually, the trend
went back to having a single repo.

Regards,
-Nick

On Fri, Jun 3, 2022 at 8:04 AM Jonathan Hall  wrote:
>
> On 6/2/22 21:40, Nick Vatamaniuc wrote:
>
> > 2. Move docs to the main repo.
> >
> > We noticed that the docs repo tags/branches can get out-of-sync with
> > the main couchdb repo. We have been merging features in main when they
> > apply only to 3.2.x and it requires care to keep track of what gets
> > merged and ported between branches. The idea is to simplify and make
> > it automatic so docs always follow the main repo so merging and
> > porting happens in one place only.
>
> I think I'm in favor of this. I can think of several times when it would
> have simplified things.  But I'm curious: Why hasn't this been done
> before? Or why did we start with separate repos? Are there any drawbacks
> we should be considering?
>
> For example, would CI or automated builds be made more difficult or slow?
>
> > What does everyone think?
> >
> > Thanks,
> > -Nick

[DISCUSSION] Rename 3.x branch to main and include docs in the main repo

[Nick Vatamaniuc]

Hi everyone,

In a #dev slack thread we were discussing improvements to how we tag
our documentation repo. There were two proposals to simplify the
development and release process, and we wanted to see what the rest of
the community thought about it. Those two ideas are:

1. Rename couchdb repo's main branch to fdbmain and rename 3.x to main.

>From an ergonomic point of view, there is more development on the 3.x
branch so having it as main makes more sense. It can help with:
   * Issues auto-closing when PRs are merged
   * Github code search works better in the default branch

2. Move docs to the main repo.

We noticed that the docs repo tags/branches can get out-of-sync with
the main couchdb repo. We have been merging features in main when they
apply only to 3.2.x and it requires care to keep track of what gets
merged and ported between branches. The idea is to simplify and make
it automatic so docs always follow the main repo so merging and
porting happens in one place only.

What does everyone think?

Thanks,
-Nick

Re: [ANNOUNCE] Ronny Berndt elected as CouchDB committer

[Nick Vatamaniuc]

Congratulations, Ronny.
Welcome aboard!

On Fri, Apr 8, 2022 at 3:37 AM Jan Lehnardt  wrote:
>
> Dear community,
>
> I am pleased to announce that the CouchDB Project Management Committee has 
> elected Ronny Berndt as a CouchDB committer.
>
> Apache ID: ronny
>
> Slack nick: big-r
>
> Committers are given a binding vote in certain project decisions, as well as 
> write access to public project infrastructure.
>
> This election was made in recognition of Ronny's commitment to the project. 
> We mean this in the sense of being loyal to the project and its interests.
>
> Please join me in extending a warm welcome to Ronny!
>
> On behalf of the CouchDB PMC,
> Jan
> —

Re: [DISCUSS] Handle libicu upgrades better

[Nick Vatamaniuc]

Hi everyone,

The pull request is open and ready for review:
https://github.com/apache/couchdb/pull/3889

In summary, I think we managed to track the collation versions in
views in a backwards compatible manner. There is a new view info map
in the header which can be used in the future to record additional
metadata about views (couchjs versions?, etc). With a slight bit of
trickery, the views can also be transparently downgraded to the 3.2.1
version without a signature change or a view reset.

Views which have more than one collator version will be submitted for
re-compaction to the already existing smoosh upgrade views channel.
The automatic trigger can be disabled as well with a new config
option.

As far as HTTP API differences these two endpoints now emit additional
collator version info:
  _design/*/_info "view_index" has a "collator_versions" list which
shows the list of versions for that particular view
  _node/*/_versions "collation_driver" object has a new
"collator_version" which shows the collator version
An example of this can be seen in
https://github.com/apache/couchdb/pull/3889#issuecomment-1022643789

Thanks to everyone who participated in the discussion! Let's continue
the review in the PR comments

Cheers,
-Nick

On Tue, Jan 25, 2022 at 10:15 AM Nick Vatamaniuc  wrote:
>
> Good idea, Will, to return the current collator version in the
> `/_node/_local/_versions` output. We return the collation algorithm
> and the library versions, however, since we switched to tracking the
> opaque collator version, it's good to also expose that too.
>
> On Tue, Jan 25, 2022 at 9:47 AM Will Young  wrote:
> >
> > Hi Nick,
> >
> >   The view _info setup looks good to me. Maybe it would be helpful to
> > print the current runtime's collator and icu versions somewhere like
> > the / meta or  /node/ _system endpoint? I think that would provide a
> > way to cross-reference to alleviate the drawback of the collator being
> > the least human readable version (though only to find a more readable
> > version for the views that are from the current runtime,) and maybe to
> > debug oddities like a cluster somehow having nodes that are out of
> > sync on libicu versions, or just to make it easier to check if dbs are
> > going to be rebuilt after an update. Of course there are also other
> > ways for an admin to examine the current runtime and workout versions
> > so it is probably a question of how frequently it will come up.
> >
> > Thanks,
> > Will
> >
> > Am Di., 25. Jan. 2022 um 07:45 Uhr schrieb Nick Vatamaniuc 
> > :
> > >
> > > Another update regarding the draft PR.
> > >
> > > There are now upgrade tests to see how we handle older 2.x, 3.2.1, and
> > > views with multiple collator versions in them.
> > >
> > > The last commit modifies the _design/*/_info API to return the list of
> > > collator versions to the user and wanted to see what everyone thought
> > > about it:
> > >
> > > https://github.com/apache/couchdb/pull/3889#issuecomment-1020861208
> > >
> > > Thanks,
> > > -Nick
> > >
> > > On Tue, Jan 11, 2022 at 1:06 PM Nick Vatamaniuc  
> > > wrote:
> > > >
> > > > I threw together a draft PR https://github.com/apache/couchdb/pull/3889
> > > >
> > > > Would that work? There are two tricks there - re-using a field
> > > > position from an older <2.3.1 format, this should allow transparently
> > > > downgrading back to 3.2.1 as we ignore that field there. Also, used a
> > > > map  so it should allow adding extra info to the views in the future
> > > > (custom collation tailorings?).
> > > >
> > > > Thanks,
> > > > -Nick
> > > >
> > > > On Sat, Nov 20, 2021 at 12:32 PM Nick Vatamaniuc  
> > > > wrote:
> > > > >
> > > > > Thanks, Adam. And thanks for the tip about the view header, Bob.
> > > > >
> > > > > Wonder if a disk version would make sense for views. Noticed Eric did
> > > > > a nice job transparently migrating 2.x -> 3.x view files when we
> > > > > removed key seq indices. Perhaps something like that would work for
> > > > > adding a collator version.
> > > > >
> > > > > Cheers,
> > > > > -Nick
> > > > >
> > > > > On Fri, Nov 19, 2021 at 9:09 AM Adam Kocoloski  
> > > > > wrote:
> > > > > >
> > > > > > That seems like a smart solut

Re: [DISCUSS] Handle libicu upgrades better

[Nick Vatamaniuc]

Good idea, Will, to return the current collator version in the
`/_node/_local/_versions` output. We return the collation algorithm
and the library versions, however, since we switched to tracking the
opaque collator version, it's good to also expose that too.

On Tue, Jan 25, 2022 at 9:47 AM Will Young  wrote:
>
> Hi Nick,
>
>   The view _info setup looks good to me. Maybe it would be helpful to
> print the current runtime's collator and icu versions somewhere like
> the / meta or  /node/ _system endpoint? I think that would provide a
> way to cross-reference to alleviate the drawback of the collator being
> the least human readable version (though only to find a more readable
> version for the views that are from the current runtime,) and maybe to
> debug oddities like a cluster somehow having nodes that are out of
> sync on libicu versions, or just to make it easier to check if dbs are
> going to be rebuilt after an update. Of course there are also other
> ways for an admin to examine the current runtime and workout versions
> so it is probably a question of how frequently it will come up.
>
> Thanks,
> Will
>
> Am Di., 25. Jan. 2022 um 07:45 Uhr schrieb Nick Vatamaniuc 
> :
> >
> > Another update regarding the draft PR.
> >
> > There are now upgrade tests to see how we handle older 2.x, 3.2.1, and
> > views with multiple collator versions in them.
> >
> > The last commit modifies the _design/*/_info API to return the list of
> > collator versions to the user and wanted to see what everyone thought
> > about it:
> >
> > https://github.com/apache/couchdb/pull/3889#issuecomment-1020861208
> >
> > Thanks,
> > -Nick
> >
> > On Tue, Jan 11, 2022 at 1:06 PM Nick Vatamaniuc  wrote:
> > >
> > > I threw together a draft PR https://github.com/apache/couchdb/pull/3889
> > >
> > > Would that work? There are two tricks there - re-using a field
> > > position from an older <2.3.1 format, this should allow transparently
> > > downgrading back to 3.2.1 as we ignore that field there. Also, used a
> > > map  so it should allow adding extra info to the views in the future
> > > (custom collation tailorings?).
> > >
> > > Thanks,
> > > -Nick
> > >
> > > On Sat, Nov 20, 2021 at 12:32 PM Nick Vatamaniuc  
> > > wrote:
> > > >
> > > > Thanks, Adam. And thanks for the tip about the view header, Bob.
> > > >
> > > > Wonder if a disk version would make sense for views. Noticed Eric did
> > > > a nice job transparently migrating 2.x -> 3.x view files when we
> > > > removed key seq indices. Perhaps something like that would work for
> > > > adding a collator version.
> > > >
> > > > Cheers,
> > > > -Nick
> > > >
> > > > On Fri, Nov 19, 2021 at 9:09 AM Adam Kocoloski  
> > > > wrote:
> > > > >
> > > > > That seems like a smart solution Nick.
> > > > >
> > > > > Adam
> > > > >
> > > > > > On Nov 19, 2021, at 7:28 AM, Robert Newson  wrote:
> > > > > >
> > > > > > Noting that the upgrade channel for views was misconceived (by me) 
> > > > > > as there is no version number in the header for them. You’d need to 
> > > > > > add it.
> > > > > >
> > > > > > B.
> > > > > >
> > > > > >> On 18 Nov 2021, at 07:12, Nick Vatamaniuc  
> > > > > >> wrote:
> > > > > >>
> > > > > >> Thinking more about this issue I wonder if we can avoid resetting 
> > > > > >> and
> > > > > >> rebuilding everything from scratch, and instead, let the upgrade
> > > > > >> happen in the background, while still serving the existing view 
> > > > > >> data.
> > > > > >>
> > > > > >> The realization was that collation doesn't affect the emitted keys 
> > > > > >> and
> > > > > >> values themselves, only their order in the view b-trees. That means
> > > > > >> we'd just have to rebuild b-trees, and that is exactly what our 
> > > > > >> view
> > > > > >> compactor already does.
> > > > > >>
> > > > > >> When we detect a libicu version discrepancy we'd submit the view 
> > > > > >> for
> > > > > >> compaction. We even

Re: [DISCUSS] Handle libicu upgrades better

[Nick Vatamaniuc]

Another update regarding the draft PR.

There are now upgrade tests to see how we handle older 2.x, 3.2.1, and
views with multiple collator versions in them.

The last commit modifies the _design/*/_info API to return the list of
collator versions to the user and wanted to see what everyone thought
about it:

https://github.com/apache/couchdb/pull/3889#issuecomment-1020861208

Thanks,
-Nick

On Tue, Jan 11, 2022 at 1:06 PM Nick Vatamaniuc  wrote:
>
> I threw together a draft PR https://github.com/apache/couchdb/pull/3889
>
> Would that work? There are two tricks there - re-using a field
> position from an older <2.3.1 format, this should allow transparently
> downgrading back to 3.2.1 as we ignore that field there. Also, used a
> map  so it should allow adding extra info to the views in the future
> (custom collation tailorings?).
>
> Thanks,
> -Nick
>
> On Sat, Nov 20, 2021 at 12:32 PM Nick Vatamaniuc  wrote:
> >
> > Thanks, Adam. And thanks for the tip about the view header, Bob.
> >
> > Wonder if a disk version would make sense for views. Noticed Eric did
> > a nice job transparently migrating 2.x -> 3.x view files when we
> > removed key seq indices. Perhaps something like that would work for
> > adding a collator version.
> >
> > Cheers,
> > -Nick
> >
> > On Fri, Nov 19, 2021 at 9:09 AM Adam Kocoloski  wrote:
> > >
> > > That seems like a smart solution Nick.
> > >
> > > Adam
> > >
> > > > On Nov 19, 2021, at 7:28 AM, Robert Newson  wrote:
> > > >
> > > > Noting that the upgrade channel for views was misconceived (by me) as 
> > > > there is no version number in the header for them. You’d need to add it.
> > > >
> > > > B.
> > > >
> > > >> On 18 Nov 2021, at 07:12, Nick Vatamaniuc  wrote:
> > > >>
> > > >> Thinking more about this issue I wonder if we can avoid resetting and
> > > >> rebuilding everything from scratch, and instead, let the upgrade
> > > >> happen in the background, while still serving the existing view data.
> > > >>
> > > >> The realization was that collation doesn't affect the emitted keys and
> > > >> values themselves, only their order in the view b-trees. That means
> > > >> we'd just have to rebuild b-trees, and that is exactly what our view
> > > >> compactor already does.
> > > >>
> > > >> When we detect a libicu version discrepancy we'd submit the view for
> > > >> compaction. We even have a dedicated "upgrade" [1] channel in smoosh
> > > >> which handles file version format upgrades, but we'll tweak that logic
> > > >> to trigger on libicu version mismatches as well.
> > > >>
> > > >> Would this work? Does anyone see any issue with that approach?
> > > >>
> > > >> [1] 
> > > >> https://github.com/apache/couchdb/blob/3.x/src/smoosh/src/smoosh_server.erl#L435-L442
> > > >>
> > > >> Cheers,
> > > >> -Nick
> > > >>
> > > >>
> > > >>
> > > >>> On Fri, Oct 29, 2021 at 7:01 PM Nick Vatamaniuc  
> > > >>> wrote:
> > > >>>
> > > >>> Hello everyone,
> > > >>>
> > > >>> CouchDB by default uses the libicu library to sort its view rows.
> > > >>> When views are built, we do not record or track the version of the
> > > >>> collation algorithm. The issue is that the ICU library may modify the
> > > >>> collation order between major libicu versions, and when that happens,
> > > >>> views built with the older versions may experience data loss. I wanted
> > > >>> to discuss the option to record the libicu collator version in each
> > > >>> view then warn the user when there is a mismatch. Also, optionally
> > > >>> ignore the mismatch, or automatically rebuild the views.
> > > >>>
> > > >>> Imagine, for example, searching patient records using start/end keys.
> > > >>> It could be possible that, say, the first letter of their name now
> > > >>> collates differently in a new libicu. That would prevent the patient
> > > >>> record from showing up in the view results for some important
> > > >>> procedure or medication. Users might not even be aware of this kind of
> > > >>> data loss occurring, there wo

Re: [DISCUSS] Handle libicu upgrades better

[Nick Vatamaniuc]

Interesting idea, Will, about possibly using the collation functions
from the query server side. There is no current mechanism to do it;
we'd have to invent it.

If we could reliably detect the couchjs libicu library version, we
could try to track it separately from the libicu used to sort view
keys. But don't think it's exposed as a JS library call (like we have
for get_libicu_version in the NIF module). But if we tracked it, and
there was a version mismatch, we wouldn't even be able to use the
trick from above to recompact the view, and we'd have to fully reset
it.

I noticed in your link how there is a mode to disable libicu linking
'--without-intl-api',  which turns off some APIs on the JS side. One
way to ensure we don't need to track libicu versions linked to the
collator is to disable its usage :-)  At first it seems rather
unusual, however it could provide some stability guarantee about the
views not becoming invalid after couchjs is upgraded. (There is of
course the chance that the other APIs users used in the new JS engine
somehow generate different data on the newer engine, which would also
invalidate the old views. It would have to be libicu, say some math
operations or one other string processing functions).

Perhaps we should just track couchjs versions and engine types in the
view file headers like we're starting to do with libicu versions? I
feel like we might need that at some point, but also it feels like a
future effort. Since we'd have to handle full view resets, warnings,
user assertions about their view / js engine being compatible etc.

Cheers,
-Nick

On Thu, Jan 13, 2022 at 12:14 PM Will Young  wrote:
>
> I would be a little hesitant to rely on the version mozilla wraps up
> reliably working since they modify it a little and don't actually use
> its normal build system:
> https://firefox-source-docs.mozilla.org/intl/icu.html#internationalization-in-spidermonkey-and-gecko
>
> That icu->spidermonkey->couch creates the longest and most fragile
> dependency chain when doing a full windows build is frustrating
> especially since its not really clear if anyone would be making use of
> the Intl, string.prototype.normalization() or similar functionality in
> spidermonkey. If C functions to access things from icu were being
> explicitly registered in the JS context from the query server code
> that could break this dependency chain and be able to disable intl and
> know what to do explicitly when icu really should be used. That would
> also make it easier to replace spidermonkey with a more minimalist JS
> interpreter.
>
> -Will
>
> Am Do., 13. Jan. 2022 um 17:38 Uhr schrieb Nick Vatamaniuc 
> :
> >
> > Hi Ronny,
> >
> > If it makes it easier to build on some platforms it could make sense.
> > Or find some way for both of them to point to a single libicu library.
> >
> > On some OSes (ex. Linux distros), dynamically linking to a system
> > libicu also makes sense because it's often the easiest way to get
> > security updates. libicu has had quite a number of high risk CVEs over
> > the years [1]
> >
> > [1] https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=icu
> >
> > -Nick
> >
> > On Wed, Jan 12, 2022 at 2:47 PM Ronny Berndt
> >  wrote:
> > >
> > > Hi,
> > >
> > > to prevent different versions of the ICU libs, why don’t use the shipped 
> > > version
> > > of the libs from the spidermonkey tree (use only esr versions) and link 
> > > against those in the build process
> > > and don’t rely on the system version?
> > >
> > > The windows version of CouchDB isn’t available for the actual version and 
> > > the build process for this
> > > os stucks at the moment. Maybe it is a broader discussion and maybe it is 
> > > a good idea to combine
> > > this with the erlang version update process ([DISCUSS] Erlang version 
> > > update process for convenience binaries).
> > >
> > > - Ronny
> > >
> > > > Am 12.01.2022 um 16:31 schrieb Will Young :
> > > >
> > > > Hi Nick,
> > > >
> > > >  I like the way this breaks down the problem into something that can
> > > > work with the existing maintenance mechanisms. On the UCA version it
> > > > looks to me like the major version tracks the last unicode version
> > > > that had a collation change (version 9.0?), while the ICU version is
> > > > changing with each release which would be more frequent than actual
> > > > collation changes. Looking at the ICU release notes I get the
> > > > impression that the frequency of change may inbetween because of bug
> > > > fixes or additions to u

Re: [DISCUSS] Handle libicu upgrades better

[Nick Vatamaniuc]

Hi Ronny,

If it makes it easier to build on some platforms it could make sense.
Or find some way for both of them to point to a single libicu library.

On some OSes (ex. Linux distros), dynamically linking to a system
libicu also makes sense because it's often the easiest way to get
security updates. libicu has had quite a number of high risk CVEs over
the years [1]

[1] https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=icu

-Nick

On Wed, Jan 12, 2022 at 2:47 PM Ronny Berndt
 wrote:
>
> Hi,
>
> to prevent different versions of the ICU libs, why don’t use the shipped 
> version
> of the libs from the spidermonkey tree (use only esr versions) and link 
> against those in the build process
> and don’t rely on the system version?
>
> The windows version of CouchDB isn’t available for the actual version and the 
> build process for this
> os stucks at the moment. Maybe it is a broader discussion and maybe it is a 
> good idea to combine
> this with the erlang version update process ([DISCUSS] Erlang version update 
> process for convenience binaries).
>
> - Ronny
>
> > Am 12.01.2022 um 16:31 schrieb Will Young :
> >
> > Hi Nick,
> >
> >  I like the way this breaks down the problem into something that can
> > work with the existing maintenance mechanisms. On the UCA version it
> > looks to me like the major version tracks the last unicode version
> > that had a collation change (version 9.0?), while the ICU version is
> > changing with each release which would be more frequent than actual
> > collation changes. Looking at the ICU release notes I get the
> > impression that the frequency of change may inbetween because of bug
> > fixes or additions to unicode that directly get a differing order in
> > the root collation. I.e. ICU 54 seems like a clean match of UCA
> > version and collation change while it seems like 59 could have changed
> > some emoji sort orders that may already have been reflected in 58's
> > UCA version?
> >
> > Another question I have about ICU synchronization is spidermonkey's
> > use of ICU. Since all build instructions keep erlang and mozjs'
> > linking to the same system ICU, I think there could never be a need to
> > record an ICU related version from the query server, but I've never
> > seen instructions to set locales in relation the query server or do
> > anything to ensure a function is using the root collator, so I don't
> > think the build setup reflects an actual need for spidermonkey to be
> > truly in sync on aspects of icu like collation setup and everything
> > important is happening in the erlang/nifs?
> > Thanks,
> > -Will
> >
> > Am Di., 11. Jan. 2022 um 19:06 Uhr schrieb Nick Vatamaniuc 
> > :
> >>
> >> I threw together a draft PR https://github.com/apache/couchdb/pull/3889
> >>
> >> Would that work? There are two tricks there - re-using a field
> >> position from an older <2.3.1 format, this should allow transparently
> >> downgrading back to 3.2.1 as we ignore that field there. Also, used a
> >> map  so it should allow adding extra info to the views in the future
> >> (custom collation tailorings?).
> >>
> >> Thanks,
> >> -Nick
> >>
> >> On Sat, Nov 20, 2021 at 12:32 PM Nick Vatamaniuc  
> >> wrote:
> >>>
> >>> Thanks, Adam. And thanks for the tip about the view header, Bob.
> >>>
> >>> Wonder if a disk version would make sense for views. Noticed Eric did
> >>> a nice job transparently migrating 2.x -> 3.x view files when we
> >>> removed key seq indices. Perhaps something like that would work for
> >>> adding a collator version.
> >>>
> >>> Cheers,
> >>> -Nick
> >>>
> >>> On Fri, Nov 19, 2021 at 9:09 AM Adam Kocoloski  
> >>> wrote:
> >>>>
> >>>> That seems like a smart solution Nick.
> >>>>
> >>>> Adam
> >>>>
> >>>>> On Nov 19, 2021, at 7:28 AM, Robert Newson  wrote:
> >>>>>
> >>>>> Noting that the upgrade channel for views was misconceived (by me) as 
> >>>>> there is no version number in the header for them. You’d need to add it.
> >>>>>
> >>>>> B.
> >>>>>
> >>>>>> On 18 Nov 2021, at 07:12, Nick Vatamaniuc  wrote:
> >>>>>>
> >>>>>> Thinking more about this issue I wonder if we can avoid resetting and
> >>>>>> rebuilding everything from scratch, and instead, let the 

Re: [DISCUSS] Handle libicu upgrades better

[Nick Vatamaniuc]

Hi Will,

Thanks for taking a look.

We currently do not support any custom tailorings, so I was hoping
that using the UCA version would be enough to indicate an
incompatibility. It has the advantage of being human readable and
matches the official UCA versions published so we can track changes
easier. But, perhaps, a better option to use is ucol_getVersion() [1].
This version would capture the UCA version, data tables and possible
tailorings if we add them in the future. The downside is it's opaque
so it would be hard to associate with a particular libicu version.
Postgres currently uses ucol_getVersion, so I think we'll switch to
that as well.

Good point about synchronizing the ICU with spidermonkey. If both are
linked to a shared libicu library, we could query the libicu version
at runtime for info/debug purposes (with GET /_node/_local/_versions).
That will tell us the version used. Otherwise, if spidermonkey is
built with a statically linked libicu it could have a completely
separate version.  And in principle, I could see how a different
version of JS engine and how it compares strings would emit different
keys and values. That was primarily why I thought of adding a metadata
map item to the view header so we could capture things like that in
the future. Presumably, then we'd have to completely reset and rebuild
the views from scratch.

[1] 
https://unicode-org.github.io/icu-docs/apidoc/dev/icu4c/ucol_8h.html#a0f98dd01ba7a64069ade6f0fda13528d

Cheers,
-Nick

On Wed, Jan 12, 2022 at 10:39 AM Will Young  wrote:
>
> Hi Nick,
>
>   I like the way this breaks down the problem into something that can
> work with the existing maintenance mechanisms. On the UCA version it
> looks to me like the major version tracks the last unicode version
> that had a collation change (version 9.0?), while the ICU version is
> changing with each release which would be more frequent than actual
> collation changes. Looking at the ICU release notes I get the
> impression that the frequency of change may inbetween because of bug
> fixes or additions to unicode that directly get a differing order in
> the root collation. I.e. ICU 54 seems like a clean match of UCA
> version and collation change while it seems like 59 could have changed
> some emoji sort orders that may already have been reflected in 58's
> UCA version?
>
> Another question I have about ICU synchronization is spidermonkey's
> use of ICU. Since all build instructions keep erlang and mozjs'
> linking to the same system ICU, I think there could never be a need to
> record an ICU related version from the query server, but I've never
> seen instructions to set locales in relation the query server or do
> anything to ensure a function is using the root collator, so I don't
> think the build setup reflects an actual need for spidermonkey to be
> truly in sync on aspects of icu like collation setup and everything
> important is happening in the erlang/nifs?
>  Thanks,
> -Will
>
> Am Di., 11. Jan. 2022 um 19:06 Uhr schrieb Nick Vatamaniuc 
> :
> >
> > I threw together a draft PR https://github.com/apache/couchdb/pull/3889
> >
> > Would that work? There are two tricks there - re-using a field
> > position from an older <2.3.1 format, this should allow transparently
> > downgrading back to 3.2.1 as we ignore that field there. Also, used a
> > map  so it should allow adding extra info to the views in the future
> > (custom collation tailorings?).
> >
> > Thanks,
> > -Nick
> >
> > On Sat, Nov 20, 2021 at 12:32 PM Nick Vatamaniuc  wrote:
> > >
> > > Thanks, Adam. And thanks for the tip about the view header, Bob.
> > >
> > > Wonder if a disk version would make sense for views. Noticed Eric did
> > > a nice job transparently migrating 2.x -> 3.x view files when we
> > > removed key seq indices. Perhaps something like that would work for
> > > adding a collator version.
> > >
> > > Cheers,
> > > -Nick
> > >
> > > On Fri, Nov 19, 2021 at 9:09 AM Adam Kocoloski  
> > > wrote:
> > > >
> > > > That seems like a smart solution Nick.
> > > >
> > > > Adam
> > > >
> > > > > On Nov 19, 2021, at 7:28 AM, Robert Newson  wrote:
> > > > >
> > > > > Noting that the upgrade channel for views was misconceived (by me) as 
> > > > > there is no version number in the header for them. You’d need to add 
> > > > > it.
> > > > >
> > > > > B.
> > > > >
> > > > >> On 18 Nov 2021, at 07:12, Nick Vatamaniuc  wrote:
> > > > >>
> > > > >> Thinking more about this issue I wonder if we can

Re: [DISCUSS] Handle libicu upgrades better

[Nick Vatamaniuc]

I threw together a draft PR https://github.com/apache/couchdb/pull/3889

Would that work? There are two tricks there - re-using a field
position from an older <2.3.1 format, this should allow transparently
downgrading back to 3.2.1 as we ignore that field there. Also, used a
map  so it should allow adding extra info to the views in the future
(custom collation tailorings?).

Thanks,
-Nick

On Sat, Nov 20, 2021 at 12:32 PM Nick Vatamaniuc  wrote:
>
> Thanks, Adam. And thanks for the tip about the view header, Bob.
>
> Wonder if a disk version would make sense for views. Noticed Eric did
> a nice job transparently migrating 2.x -> 3.x view files when we
> removed key seq indices. Perhaps something like that would work for
> adding a collator version.
>
> Cheers,
> -Nick
>
> On Fri, Nov 19, 2021 at 9:09 AM Adam Kocoloski  wrote:
> >
> > That seems like a smart solution Nick.
> >
> > Adam
> >
> > > On Nov 19, 2021, at 7:28 AM, Robert Newson  wrote:
> > >
> > > Noting that the upgrade channel for views was misconceived (by me) as 
> > > there is no version number in the header for them. You’d need to add it.
> > >
> > > B.
> > >
> > >> On 18 Nov 2021, at 07:12, Nick Vatamaniuc  wrote:
> > >>
> > >> Thinking more about this issue I wonder if we can avoid resetting and
> > >> rebuilding everything from scratch, and instead, let the upgrade
> > >> happen in the background, while still serving the existing view data.
> > >>
> > >> The realization was that collation doesn't affect the emitted keys and
> > >> values themselves, only their order in the view b-trees. That means
> > >> we'd just have to rebuild b-trees, and that is exactly what our view
> > >> compactor already does.
> > >>
> > >> When we detect a libicu version discrepancy we'd submit the view for
> > >> compaction. We even have a dedicated "upgrade" [1] channel in smoosh
> > >> which handles file version format upgrades, but we'll tweak that logic
> > >> to trigger on libicu version mismatches as well.
> > >>
> > >> Would this work? Does anyone see any issue with that approach?
> > >>
> > >> [1] 
> > >> https://github.com/apache/couchdb/blob/3.x/src/smoosh/src/smoosh_server.erl#L435-L442
> > >>
> > >> Cheers,
> > >> -Nick
> > >>
> > >>
> > >>
> > >>> On Fri, Oct 29, 2021 at 7:01 PM Nick Vatamaniuc  
> > >>> wrote:
> > >>>
> > >>> Hello everyone,
> > >>>
> > >>> CouchDB by default uses the libicu library to sort its view rows.
> > >>> When views are built, we do not record or track the version of the
> > >>> collation algorithm. The issue is that the ICU library may modify the
> > >>> collation order between major libicu versions, and when that happens,
> > >>> views built with the older versions may experience data loss. I wanted
> > >>> to discuss the option to record the libicu collator version in each
> > >>> view then warn the user when there is a mismatch. Also, optionally
> > >>> ignore the mismatch, or automatically rebuild the views.
> > >>>
> > >>> Imagine, for example, searching patient records using start/end keys.
> > >>> It could be possible that, say, the first letter of their name now
> > >>> collates differently in a new libicu. That would prevent the patient
> > >>> record from showing up in the view results for some important
> > >>> procedure or medication. Users might not even be aware of this kind of
> > >>> data loss occurring, there won't be any error in the API or warning in
> > >>> the logs.
> > >>>
> > >>> I was thinking how to solve this. There were a few commits already to
> > >>> cleanup our collation drivers [1], expose libicu and collation
> > >>> algorithm version in the new _versions endpoint [2], and some other
> > >>> minor fixes in that area. As the next steps we could:
> > >>>
> > >>> 1) Modify our views to keep track of the collation algorithm
> > >>> version. We could attempt to transparently upgrade the view header
> > >>> format -- read the old view file, update the header with an extra
> > >>> libicu collation version field, that updates the signature, and then,
> > >>> save the file wi

Re: GH Workflows

[Nick Vatamaniuc]

Option 1 sounds good - picking a variety of OS and packaged SM
versions. For 91 we may have to add a new OS into the mix - Fedora 35.
I saw it was the only distro packaging mozjs91
(https://pkgs.org/search/?q=mozjs91)

-Nick

On Fri, Nov 26, 2021 at 12:38 PM Adam Kocoloski  wrote:
>
> I see a couple of possible options:
>
> 1) Implicitly test against a matrix of SM versions because we use the 
> packages that are shipped with each supported OS in the CI matrix, e.g. 
> CentOS8 -> SM60, Ubuntu 20.04 -> SM68, Debian 11 -> SM78 (that last one’s not 
> in Jenkins CI yet I think). In this model we only add official support for 
> new SM versions when they show up in a supported distro.
>
> 2) Reinvigorate the couchdb-pkg machinery we use to generate SM 1.8.5 
> packages to publish .debs for all supported SM versions, then add a new 
> matrix build to verify support.
>
> I guess the key question is whether we expect users to continue to depend on 
> distro packages for their actual production installation. If so, then I’m not 
> sure a separate matrix offers a great return on investment. We could easily 
> end up in a situation where CouchDB works with the SM package that we built 
> for CI, but not the one the distro maintainers ended up publishing. That 
> said, if someone wants to do the work, great!
>
> An option to trigger a manual integration run against a specific version of 
> SM also seems like an OK compromise to me.
>
> Adam
>
> > On Nov 26, 2021, at 5:23 AM, Jan Lehnardt  wrote:
> >
> > Hey all,
> >
> > we recently received a nice contribution (SpiderMonkey 91 support) as a 
> > community contribution:
> >
> >https://github.com/apache/couchdb/pull/3842/files
> >
> > It includes a GH workflow that includes building said SM version to prove 
> > that it works, which is very nice.
> >
> > We currently don’t really test a matrix of SM versions, and we probably 
> > don’t want to build sm from source on each CI run.
> >
> > My question to you is how you think we should handle this?
> >
> > Best
> > Jan
> > —
> > Professional Support for Apache CouchDB:
> > https://neighbourhood.ie/couchdb-support/
> >
> > 24/7 Observation for your CouchDB Instances:
> > https://opservatory.app
> >
>

Re: [DISCUSS] EUnit test files location

[Nick Vatamaniuc]

Using rebar 3.15.2 would work for now. Good find.

For _build directory, a good number of dependencies already have their
own unit tests for PRs, so they should be ok. Some we just skip
altogether anyway
https://github.com/apache/couchdb/blob/main/Makefile#L76
(folsom,meck,mochiweb,triq,proper,snappy,bcrypt,hyper,ibrowse).

But the issue is that those dependencies just won't run their own
tests or that they won't even be available as dependencies for other
eunit tests. If it's later, we may have to do some additional magic,
like you suggested.

Thanks,
-Nick

On Fri, Nov 26, 2021 at 8:57 AM Adam Kocoloski  wrote:
>
> The pc plugin has handled everything I’ve thrown at it so far. I recall Paul 
> mentioning some issue with transitive dependencies; I’ll keep an eye out for 
> any trouble. I didn’t try anything with Fauxton and docs yet.
>
> Rebar3 supported Erlang 20 as recently as 3.15.2, so I can imagine an option 
> to just use that version for Erlang 20 / 21 builds and the latest version for 
> 22+. That’s what I ended up doing for the GH Actions CI for the b64url lib. 
> For what it’s worth, the rebar3 team only intends to support 3 major releases 
> at a time going forward, see https://github.com/erlang/rebar3/pull/2555 
> <https://github.com/erlang/rebar3/pull/2555>. They argue that this is 
> consistent with the support statement from Erlang/OTP itself.
>
> One notable difference I’ve found so far is that our current setup considers 
> dependencies to be peers to the internal apps in the umbrella couchdb repo: 
> we put all of them in src/, we run their tests as part of our test suite, 
> etc. Rebar3 installs dependencies to the _build directory and does not test 
> them. If it’s essential that we consider code in these other repositories to 
> be part of an Apache CouchDB source release we might need to get a little 
> creative.
>
> As an aside, I didn’t intend to weigh in on the rebar3 vs. mix debate here. I 
> can see good arguments for both. I just found myself gaining some familiarity 
> with rebar3 recently, and figured there would be a fair amount of common work 
> involved in aligning our build system to either modern option.
>
> Cheers, Adam
>
> > On Nov 24, 2021, at 2:07 PM, Nick Vatamaniuc  wrote:
> >
> > Good idea to try rebar3! I tried it a few years back and got stumped
> > at building some NIFs and a few other issues. Most of those can be
> > overcome with a bit of work.
> >
> > I saw you used the pc compiler for NIFs, and I think rebar3 can set
> > overrides now too
> > https://github.com/blt/port_compiler#use-with-existing-dependency for
> > individual dependencies. Perhaps the issue I encountered might be
> > solved that way.
> >
> > The other issue was our "raw" dependencies - fauxton and docs. The
> > approach I had tried was to make them look like Erlang apps. I don't
> > recall how successful that was. But I think there are some newer
> > things like https://rebar3.readme.io/docs/custom-dep-resources which
> > we could leverage there.
> >
> > One more issue I recently discovered was that rebar3 is not Erlang 20
> > compatible. We ship our releases with the Erlang 20 as the minimum
> > version and if we update to use rebar3 our release branch might be
> > left behind. If the issue is just using Tag:Error:Stack catch
> > patterns, we could potentially have our own fork of rebar3, as we
> > already do for rebar2.
> >
> > Cheers,
> > -Nick
> >
> >
> > On Tue, Nov 23, 2021 at 5:39 PM Adam Kocoloski  wrote:
> >>
> >> Bah, ignore the message below. The issue isn’t that EUnit has a problem 
> >> with nested directories, it’s that it expects each test module to have a 
> >> `_tests` suffix, not a `_test` one. We’re inconsistent about adhering to 
> >> that rule in our codebase I’ll submit a PR to fix the module names so no 
> >> one else bumps into that issue.
> >>
> >> Adam
> >>
> >>> On Nov 23, 2021, at 3:48 PM, Adam Kocoloski  wrote:
> >>>
> >>> Hi,
> >>>
> >>> TL;DR - I think it would be a good idea to move EUnit tests back into 
> >>> `test` instead of `test/eunit`.
> >>>
> >>> Recently I’ve been spending some time porting specific libraries to use 
> >>> rebar3 and I thought it might be a nice project to hack on supporting 
> >>> rebar3 in CouchDB proper during the holiday weekend. I noticed a few 
> >>> small issues, but one in particular is that it was difficult to get 
> >>> rebar3 to pick up unit tests that were in subdirectories of `test`.
> >>>
> >>> Eve

Re: [DISCUSS] EUnit test files location

[Nick Vatamaniuc]

Good idea to try rebar3! I tried it a few years back and got stumped
at building some NIFs and a few other issues. Most of those can be
overcome with a bit of work.

I saw you used the pc compiler for NIFs, and I think rebar3 can set
overrides now too
https://github.com/blt/port_compiler#use-with-existing-dependency for
individual dependencies. Perhaps the issue I encountered might be
solved that way.

The other issue was our "raw" dependencies - fauxton and docs. The
approach I had tried was to make them look like Erlang apps. I don't
recall how successful that was. But I think there are some newer
things like https://rebar3.readme.io/docs/custom-dep-resources which
we could leverage there.

One more issue I recently discovered was that rebar3 is not Erlang 20
compatible. We ship our releases with the Erlang 20 as the minimum
version and if we update to use rebar3 our release branch might be
left behind. If the issue is just using Tag:Error:Stack catch
patterns, we could potentially have our own fork of rebar3, as we
already do for rebar2.

Cheers,
-Nick


On Tue, Nov 23, 2021 at 5:39 PM Adam Kocoloski  wrote:
>
> Bah, ignore the message below. The issue isn’t that EUnit has a problem with 
> nested directories, it’s that it expects each test module to have a `_tests` 
> suffix, not a `_test` one. We’re inconsistent about adhering to that rule in 
> our codebase I’ll submit a PR to fix the module names so no one else bumps 
> into that issue.
>
> Adam
>
> > On Nov 23, 2021, at 3:48 PM, Adam Kocoloski  wrote:
> >
> > Hi,
> >
> > TL;DR - I think it would be a good idea to move EUnit tests back into 
> > `test` instead of `test/eunit`.
> >
> > Recently I’ve been spending some time porting specific libraries to use 
> > rebar3 and I thought it might be a nice project to hack on supporting 
> > rebar3 in CouchDB proper during the holiday weekend. I noticed a few small 
> > issues, but one in particular is that it was difficult to get rebar3 to 
> > pick up unit tests that were in subdirectories of `test`.
> >
> > Eventually I figured out that you could tell the compiler to do recursive 
> > compilation on these test directories and that worked for running the 
> > end-to-end test suite, but then I found that rebar3’s `—app` flag was still 
> > broken, e.g. if I ran `rebar3 eunit —app couch_log` it would only find 
> > tests in the actual source modules but not any of the modules in the 
> > test/eunit directory. I thought this might be an inconsistency bug in 
> > rebar3, but I realized that when you invoked rebar3 this way it was really 
> > just calling `eunit:test({application, couch_log})`, and EUnit doesn’t 
> > allow any configurability of its search path when invoked this way.
> >
> > Bottom line, it feels like we’re going against the grain of eunit itself 
> > when we try to move the eunit tests to anywhere else other than `test`. 
> > Currently we have a small handful of apps following this pattern:
> >
> > chttpd
> > couch
> > couch_epi
> > couch_log
> > couch_prometheus
> > couch_replicator
> >
> > Of those, only the first two have any ExUnit unit tests that (I think) 
> > motivated the creation of separate eunit and exunit tests in the first 
> > place. Would it be OK if we moved those test modules back into `test` so we 
> > can stay on the “paved path” for EUnit usage going forward?
> >
> > Cheers, Adam
>

Re: [DISCUSS] Handle libicu upgrades better

[Nick Vatamaniuc]

Thanks, Adam. And thanks for the tip about the view header, Bob.

Wonder if a disk version would make sense for views. Noticed Eric did
a nice job transparently migrating 2.x -> 3.x view files when we
removed key seq indices. Perhaps something like that would work for
adding a collator version.

Cheers,
-Nick

On Fri, Nov 19, 2021 at 9:09 AM Adam Kocoloski  wrote:
>
> That seems like a smart solution Nick.
>
> Adam
>
> > On Nov 19, 2021, at 7:28 AM, Robert Newson  wrote:
> >
> > Noting that the upgrade channel for views was misconceived (by me) as there 
> > is no version number in the header for them. You’d need to add it.
> >
> > B.
> >
> >> On 18 Nov 2021, at 07:12, Nick Vatamaniuc  wrote:
> >>
> >> Thinking more about this issue I wonder if we can avoid resetting and
> >> rebuilding everything from scratch, and instead, let the upgrade
> >> happen in the background, while still serving the existing view data.
> >>
> >> The realization was that collation doesn't affect the emitted keys and
> >> values themselves, only their order in the view b-trees. That means
> >> we'd just have to rebuild b-trees, and that is exactly what our view
> >> compactor already does.
> >>
> >> When we detect a libicu version discrepancy we'd submit the view for
> >> compaction. We even have a dedicated "upgrade" [1] channel in smoosh
> >> which handles file version format upgrades, but we'll tweak that logic
> >> to trigger on libicu version mismatches as well.
> >>
> >> Would this work? Does anyone see any issue with that approach?
> >>
> >> [1] 
> >> https://github.com/apache/couchdb/blob/3.x/src/smoosh/src/smoosh_server.erl#L435-L442
> >>
> >> Cheers,
> >> -Nick
> >>
> >>
> >>
> >>> On Fri, Oct 29, 2021 at 7:01 PM Nick Vatamaniuc  
> >>> wrote:
> >>>
> >>> Hello everyone,
> >>>
> >>> CouchDB by default uses the libicu library to sort its view rows.
> >>> When views are built, we do not record or track the version of the
> >>> collation algorithm. The issue is that the ICU library may modify the
> >>> collation order between major libicu versions, and when that happens,
> >>> views built with the older versions may experience data loss. I wanted
> >>> to discuss the option to record the libicu collator version in each
> >>> view then warn the user when there is a mismatch. Also, optionally
> >>> ignore the mismatch, or automatically rebuild the views.
> >>>
> >>> Imagine, for example, searching patient records using start/end keys.
> >>> It could be possible that, say, the first letter of their name now
> >>> collates differently in a new libicu. That would prevent the patient
> >>> record from showing up in the view results for some important
> >>> procedure or medication. Users might not even be aware of this kind of
> >>> data loss occurring, there won't be any error in the API or warning in
> >>> the logs.
> >>>
> >>> I was thinking how to solve this. There were a few commits already to
> >>> cleanup our collation drivers [1], expose libicu and collation
> >>> algorithm version in the new _versions endpoint [2], and some other
> >>> minor fixes in that area. As the next steps we could:
> >>>
> >>> 1) Modify our views to keep track of the collation algorithm
> >>> version. We could attempt to transparently upgrade the view header
> >>> format -- read the old view file, update the header with an extra
> >>> libicu collation version field, that updates the signature, and then,
> >>> save the file with the new header and new signature. This avoids view
> >>> rebuilds, just records the collator version in the view and moves the
> >>> files to a new name.
> >>>
> >>> 2) Do what PostgreSQL does, and 2a) emit a warning with the view
> >>> results when the current libicu version doesn't match the version in
> >>> the view [3]. That means altering the view results to add a "warning":
> >>> "..." field. Another alternative 2b) is emit a warning in the
> >>> _design/$ddoc/_info only. Users would have to know that after an OS
> >>> version upgrade, or restoring backups, to make sure to look at their
> >>> _design/$ddoc/_info for each db for each ddoc. Of course, there may be
> >>> users which used the 

Re: [DISCUSS] erlfmt the 3.x branch

[Nick Vatamaniuc]

I gave it a try in https://github.com/apache/couchdb/pull/3837

There are two separate commits - tooling and actual formatting. The
Erlang version check didn't seem too bad to implement. Hopefully soon
we'll jump to 21+ and can clean that code up.

>From the main branch I ported all the formatting exceptions and
applied them to 3.x, one was in couch_debug and one in chttpd module.

Thanks,
-Nick

On Thu, Nov 18, 2021 at 11:50 AM Nick Vatamaniuc  wrote:
>
> Hi Jay,
>
> I don't really agree with many of the choices erlfmt makes either. I
> personally like emilio's [1] syntax rules better. But, I think there
> is still a net benefit to having a standardized format that is
> automatically applied and enforced. The automated part helps when
> reviewing PRs / contributions and keeping it consistent in the future.
> And it's odd that main was reformatted but 3.x isn't currently.
>
> I think with piecemeal conversion we'd lose the automatic enforcement
> or we'd have to add exceptions allow/deny lists for files.
>
> For blame I think it might be helpful to just have one commit for the
> reformatting and one commit for make/dev* scripts so it's easy to jump
> before and after the one commit that only does reformatting. I'll try
> to group the PR that way (2 separate commits). This way current
> pending PR can also bring first rebase the on the tooling, so they can
> reformat their commits, then rebase again on the reformatted code base
> to hopefully get a lot less conflicts.
>
> [1] https://github.com/cloudant-labs/emilio
>
> Regards,
> -Nick
>
> On Thu, Nov 18, 2021 at 11:25 AM Jay Doane  wrote:
> >
> > Hi Nick,
> >
> > As with main, my objections to a mass reformatting of hundreds of files
> > boils down to:
> >
> > 1. Making it more difficult to understand the logical changes of the code
> > (via blame) after reformatting
> > 2. I don't agree with all the opinions of the formatter, nor does my
> > editor/mode
> >
> > Is this something that could be done piecemeal for the purpose of
> > backporting/diffing, or do you feel it's necessary to reformat everything
> > at once?
> >
> > Thanks,
> > Jay
> >
> > On Wed, Nov 17, 2021 at 11:24 PM Nick Vatamaniuc  wrote:
> >
> > > Hi all,
> > >
> > > I had noticed we had agreed to use erlfmt but only ran it on main and
> > > not 3.x. That makes it harder to port changes or diff-ing modules
> > > sometimes. Would there be any objections to a pull request
> > > reformatting 3.x?
> > >
> > > The only issue is that we may have to avoid running erlfmt if we
> > > detect Erlang 20, but that's just a few if / else checks. The CI
> > > checks on 21+ would still check and fail if erlfmt check fails.
> > >
> > > Thanks,
> > > -Nick
> > >

Re: [DISCUSS] erlfmt the 3.x branch

[Nick Vatamaniuc]

Hi Jay,

I don't really agree with many of the choices erlfmt makes either. I
personally like emilio's [1] syntax rules better. But, I think there
is still a net benefit to having a standardized format that is
automatically applied and enforced. The automated part helps when
reviewing PRs / contributions and keeping it consistent in the future.
And it's odd that main was reformatted but 3.x isn't currently.

I think with piecemeal conversion we'd lose the automatic enforcement
or we'd have to add exceptions allow/deny lists for files.

For blame I think it might be helpful to just have one commit for the
reformatting and one commit for make/dev* scripts so it's easy to jump
before and after the one commit that only does reformatting. I'll try
to group the PR that way (2 separate commits). This way current
pending PR can also bring first rebase the on the tooling, so they can
reformat their commits, then rebase again on the reformatted code base
to hopefully get a lot less conflicts.

[1] https://github.com/cloudant-labs/emilio

Regards,
-Nick

On Thu, Nov 18, 2021 at 11:25 AM Jay Doane  wrote:
>
> Hi Nick,
>
> As with main, my objections to a mass reformatting of hundreds of files
> boils down to:
>
> 1. Making it more difficult to understand the logical changes of the code
> (via blame) after reformatting
> 2. I don't agree with all the opinions of the formatter, nor does my
> editor/mode
>
> Is this something that could be done piecemeal for the purpose of
> backporting/diffing, or do you feel it's necessary to reformat everything
> at once?
>
> Thanks,
> Jay
>
> On Wed, Nov 17, 2021 at 11:24 PM Nick Vatamaniuc  wrote:
>
> > Hi all,
> >
> > I had noticed we had agreed to use erlfmt but only ran it on main and
> > not 3.x. That makes it harder to port changes or diff-ing modules
> > sometimes. Would there be any objections to a pull request
> > reformatting 3.x?
> >
> > The only issue is that we may have to avoid running erlfmt if we
> > detect Erlang 20, but that's just a few if / else checks. The CI
> > checks on 21+ would still check and fail if erlfmt check fails.
> >
> > Thanks,
> > -Nick
> >

[DISCUSS] erlfmt the 3.x branch

[Nick Vatamaniuc]

Hi all,

I had noticed we had agreed to use erlfmt but only ran it on main and
not 3.x. That makes it harder to port changes or diff-ing modules
sometimes. Would there be any objections to a pull request
reformatting 3.x?

The only issue is that we may have to avoid running erlfmt if we
detect Erlang 20, but that's just a few if / else checks. The CI
checks on 21+ would still check and fail if erlfmt check fails.

Thanks,
-Nick

Re: [DISCUSS] Handle libicu upgrades better

[Nick Vatamaniuc]

Thinking more about this issue I wonder if we can avoid resetting and
rebuilding everything from scratch, and instead, let the upgrade
happen in the background, while still serving the existing view data.

The realization was that collation doesn't affect the emitted keys and
values themselves, only their order in the view b-trees. That means
we'd just have to rebuild b-trees, and that is exactly what our view
compactor already does.

When we detect a libicu version discrepancy we'd submit the view for
compaction. We even have a dedicated "upgrade" [1] channel in smoosh
which handles file version format upgrades, but we'll tweak that logic
to trigger on libicu version mismatches as well.

Would this work? Does anyone see any issue with that approach?

[1] 
https://github.com/apache/couchdb/blob/3.x/src/smoosh/src/smoosh_server.erl#L435-L442

Cheers,
-Nick



On Fri, Oct 29, 2021 at 7:01 PM Nick Vatamaniuc  wrote:
>
> Hello everyone,
>
> CouchDB by default uses the libicu library to sort its view rows.
> When views are built, we do not record or track the version of the
> collation algorithm. The issue is that the ICU library may modify the
> collation order between major libicu versions, and when that happens,
> views built with the older versions may experience data loss. I wanted
> to discuss the option to record the libicu collator version in each
> view then warn the user when there is a mismatch. Also, optionally
> ignore the mismatch, or automatically rebuild the views.
>
> Imagine, for example, searching patient records using start/end keys.
> It could be possible that, say, the first letter of their name now
> collates differently in a new libicu. That would prevent the patient
> record from showing up in the view results for some important
> procedure or medication. Users might not even be aware of this kind of
> data loss occurring, there won't be any error in the API or warning in
> the logs.
>
> I was thinking how to solve this. There were a few commits already to
> cleanup our collation drivers [1], expose libicu and collation
> algorithm version in the new _versions endpoint [2], and some other
> minor fixes in that area. As the next steps we could:
>
>   1) Modify our views to keep track of the collation algorithm
> version. We could attempt to transparently upgrade the view header
> format -- read the old view file, update the header with an extra
> libicu collation version field, that updates the signature, and then,
> save the file with the new header and new signature. This avoids view
> rebuilds, just records the collator version in the view and moves the
> files to a new name.
>
>   2) Do what PostgreSQL does, and 2a) emit a warning with the view
> results when the current libicu version doesn't match the version in
> the view [3]. That means altering the view results to add a "warning":
> "..." field. Another alternative 2b) is emit a warning in the
> _design/$ddoc/_info only. Users would have to know that after an OS
> version upgrade, or restoring backups, to make sure to look at their
> _design/$ddoc/_info for each db for each ddoc. Of course, there may be
> users which used the "raw" collation option, or know they are using
> just the plain ASCII character sets in their views. So we'd have a
> configuration setting to ignore the warnings as well.
>
>   3) Users who see the warning, could then either rebuild the view
> with the new collator library manually, or it could happen
> automatically based on a configuration option, basically "when
> collator versions are miss-matched, invalidate and rebuild all the
> views".
>
>   4) We'd have a way for the users to assert (POST a ddoc update) that
> they double-checked the new ICU version and are convinced that a
> particular view would not experience data loss with the new collator.
> That should make the warning go away, and the view to not be rebuilt.
> This can't be just a naive "collator" option setting as both per-view
> and per-design options are used when computing the view signature, and
> any changes there would result in the view being rebuilt. Perhaps we
> can add it to the design docs as a separate option which is excluded
> from the signature hash, like the "autoupdate" setting for background
> index builder ("collation_version_accept"?). PostgreSQL also offers
> this option with the ALTER COLLATION ... REFRESH VERSION command [3]
>
> What do we think, is this a reasonable approach? Is there something
> easier / simpler we can do?
>
> Thanks!
> -Nick
>
> [1] 
> https://github.com/apache/couchdb/pull/3746/commits/28f26f52fe2e170d98658311dafa8198d96b8061
> [2] 
> https://github.com/apache/couchdb/commit/c1bb4e4856edd93255d75ebe158b4da38bbf
> [3] https://www.postgresql.org/docs/13/sql-altercollation.html

Re: [DISCUSS] improving visibility for CouchDB-maintained independent Erlang apps

[Nick Vatamaniuc]

Nicely done, Adam!

+1 for most of the PR except one. Perhaps there is a way to push the
CI image to the couchdbdev account instead of your personal one.

As for voting for making these proper releases, I am not really
looking forward to the SVN pushes, signing, checksumming, and
collecting votes for these deps. Especially for a NIF with just 2
functions in it.  Wonder what can be simplified there. Can voting be
done in a GH release draft comments section as opposed a dev@couchdb.
What about SVN pushes, can we automate pushing the GH release .tgz
artifact to SVN with a bot or something? Or does calling them "tags"
instead of releases make a difference?

Thanks,
-Nick

On Tue, Nov 16, 2021 at 3:02 PM Adam Kocoloski  wrote:
>
> Hi,
>
> I’ve spent some cycles on this work over the past few weeks and wanted to 
> summarize my progress and propose some next steps. I focused on the following 
> two apps:
>
> erlfdb (Erlang API for FoundationDB)
> b64url (Base64 encoder / decoder with URL-safe scheme)
>
> I’ve submitted PRs to implement the following functionality from my initial 
> mail:
>
> > - rebar3 compatibility
> > - project READMEs that clarify the app is independent of (but maintained 
> > by) CouchDB
> > - activating GH Issues
>
> I’ve also introduced some additional enhancements:
>
> - devcontainer configurations to simplify dev environment
> - CI using GitHub Actions to verify support for Linux + Windows and 
> Erlang/OTP 22-24
> - branch protection configurations
> - automated code coverage reports sent to coveralls.io
>
> In the few days since I added GH Issues to erlfdb we’ve had a request for 
> packages on hex.pm, which was the next item on the list :) As we discussed, 
> this community should produce official source releases upstream of any 
> packages that get published. I’m willing to take a shot at running a 
> simplified release process for these other repos, though I might need to 
> “phone a friend” the first time through as I’ve never run a CouchDB release. 
> Sound good?
>
> In the meantime, I’ve got a few PRs that could use a +1 if folks have the 
> time:
>
> https://github.com/apache/couchdb-erlfdb/pull/35
> https://github.com/apache/couchdb-erlfdb/pull/42
> https://github.com/apache/couchdb-erlfdb/pull/43
> https://github.com/apache/couchdb-b64url/pull/8
>
> Cheers, Adam
>

Re: [VOTE] Release Apache CouchDB 3.2.1

[Nick Vatamaniuc]

I created rpm/deb packages and pushed them to JFrog. Please test them
as per: 
https://docs.couchdb.org/en/stable/install/unix.html#installation-using-the-apache-couchdb-convenience-binary-packages.
Note that the RPM package signing key has changed. Docs updated
accordingly.

I had also built x86 and ARM docker images
https://hub.docker.com/r/apache/couchdb/tags. Please test those as
well. The 3.2.1 tag is aliased to 3.2, 3, and "latest".

Thanks for the build instructions and the nicely done couchdb-pkg and
couchdb-docker repos, Joan!

Cheers,
-Nick

On Sat, Nov 6, 2021 at 1:06 PM Jan Lehnardt  wrote:
>
> Heya Joan,
>
> thanks for the heads-up. Nick and I are tackling these this time around :)
>
> Best
> Jan
> —
>
> > On 3. Nov 2021, at 08:07, Joan Touzet  wrote:
> >
> > Hello there,
> >
> > Sorry, I'm unavailable to help test and vote on releases.
> >
> > I took notes during the 3.2.0 release as to what was needed to do the 
> > binary releases without me. Someone from the PMC, specifically, should pick 
> > this up for now.
> >
> > https://gist.github.com/wohali/9f159bbdb6ef2c51a50d1e2326fc1d5b
> >
> > Any questions, feel free to send me a note offlist.
> >
> > Cheers,
> > Joan "too busy for a pithy quote" Touzet
> >
> > On 02/11/2021 22:00, Nick Vatamaniuc wrote:
> >> Dear community,
> >> I would like to propose that we release Apache CouchDB 3.2.1
> >> Candidate release notes:
> >> https://docs.couchdb.org/en/latest/whatsnew/3.2.html#version-3-2-1
> >> We encourage the whole community to download and test these release
> >> artefacts so that any critical issues can be resolved before the
> >> release is made. Everyone is free to vote on this release, so dig
> >> right in! (Only PMC members have binding votes, but they depend on
> >> community feedback to gauge if an official release is ready to be
> >> made.)
> >> The release artefacts we are voting on are available here:
> >> https://dist.apache.org/repos/dist/dev/couchdb/source/3.2.1/rc.1/
> >> There, you will find a tarball, a GPG signature, and SHA256/SHA512 
> >> checksums.
> >> Please follow the test procedure here:
> >> 
> >> https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release
> >> Please remember that "RC1" is an annotation. If the vote passes, these
> >> artefacts will be released as Apache CouchDB 3.2.1
> >> Please cast your votes now.
> >> Thanks,
> >> -Nick
>

Re: [VOTE] Release Apache CouchDB 3.2.1

[Nick Vatamaniuc]

Dear community,

The vote has now closed.
Thank you to everyone who participated!

The results are:
+1 - 4 (3 binding) votes
+0 - 0 votes
-0 - 0 votes
-1 - 0 votes

The vote result is +1 (passed)

Thanks,
-Nick

On Wed, Nov 3, 2021 at 1:12 PM Jay Doane  wrote:
>
> +1
>
> GPG signature, checksums, `make check`, fauxton: all good
>
> macOS 11.6.1, OTP 23.3.4.8, SpiderMonkey 60
>
> Small nit on release notes:
> > there are less corner cases
> s/less/fewer/
>
> Very nice work, Nick!
>
>
> On Tue, Nov 2, 2021 at 7:00 PM Nick Vatamaniuc  wrote:
>
> > Dear community,
> >
> > I would like to propose that we release Apache CouchDB 3.2.1
> >
> > Candidate release notes:
> >
> > https://docs.couchdb.org/en/latest/whatsnew/3.2.html#version-3-2-1
> >
> > We encourage the whole community to download and test these release
> > artefacts so that any critical issues can be resolved before the
> > release is made. Everyone is free to vote on this release, so dig
> > right in! (Only PMC members have binding votes, but they depend on
> > community feedback to gauge if an official release is ready to be
> > made.)
> >
> > The release artefacts we are voting on are available here:
> >
> > https://dist.apache.org/repos/dist/dev/couchdb/source/3.2.1/rc.1/
> >
> > There, you will find a tarball, a GPG signature, and SHA256/SHA512
> > checksums.
> >
> > Please follow the test procedure here:
> >
> >
> > https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release
> >
> > Please remember that "RC1" is an annotation. If the vote passes, these
> > artefacts will be released as Apache CouchDB 3.2.1
> >
> > Please cast your votes now.
> >
> > Thanks,
> > -Nick
> >

Re: [VOTE] Release Apache CouchDB 3.2.1

[Nick Vatamaniuc]

My own vote: +1

sig: ok
checksums: ok
make check: ok
Fauxton self-check: ok

Ubuntu 20.04, Erlang 23, Spidermonkey 68

Cheers,
-Nick

On Tue, Nov 2, 2021 at 10:00 PM Nick Vatamaniuc  wrote:
>
> Dear community,
>
> I would like to propose that we release Apache CouchDB 3.2.1
>
> Candidate release notes:
>
> https://docs.couchdb.org/en/latest/whatsnew/3.2.html#version-3-2-1
>
> We encourage the whole community to download and test these release
> artefacts so that any critical issues can be resolved before the
> release is made. Everyone is free to vote on this release, so dig
> right in! (Only PMC members have binding votes, but they depend on
> community feedback to gauge if an official release is ready to be
> made.)
>
> The release artefacts we are voting on are available here:
>
> https://dist.apache.org/repos/dist/dev/couchdb/source/3.2.1/rc.1/
>
> There, you will find a tarball, a GPG signature, and SHA256/SHA512 checksums.
>
> Please follow the test procedure here:
>
> 
> https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release
>
> Please remember that "RC1" is an annotation. If the vote passes, these
> artefacts will be released as Apache CouchDB 3.2.1
>
> Please cast your votes now.
>
> Thanks,
> -Nick

[VOTE] Release Apache CouchDB 3.2.1

[Nick Vatamaniuc]

Dear community,

I would like to propose that we release Apache CouchDB 3.2.1

Candidate release notes:

https://docs.couchdb.org/en/latest/whatsnew/3.2.html#version-3-2-1

We encourage the whole community to download and test these release
artefacts so that any critical issues can be resolved before the
release is made. Everyone is free to vote on this release, so dig
right in! (Only PMC members have binding votes, but they depend on
community feedback to gauge if an official release is ready to be
made.)

The release artefacts we are voting on are available here:

https://dist.apache.org/repos/dist/dev/couchdb/source/3.2.1/rc.1/

There, you will find a tarball, a GPG signature, and SHA256/SHA512 checksums.

Please follow the test procedure here:

https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release

Please remember that "RC1" is an annotation. If the vote passes, these
artefacts will be released as Apache CouchDB 3.2.1

Please cast your votes now.

Thanks,
-Nick

[DISCUSS] Handle libicu upgrades better

[Nick Vatamaniuc]

Hello everyone,

CouchDB by default uses the libicu library to sort its view rows.
When views are built, we do not record or track the version of the
collation algorithm. The issue is that the ICU library may modify the
collation order between major libicu versions, and when that happens,
views built with the older versions may experience data loss. I wanted
to discuss the option to record the libicu collator version in each
view then warn the user when there is a mismatch. Also, optionally
ignore the mismatch, or automatically rebuild the views.

Imagine, for example, searching patient records using start/end keys.
It could be possible that, say, the first letter of their name now
collates differently in a new libicu. That would prevent the patient
record from showing up in the view results for some important
procedure or medication. Users might not even be aware of this kind of
data loss occurring, there won't be any error in the API or warning in
the logs.

I was thinking how to solve this. There were a few commits already to
cleanup our collation drivers [1], expose libicu and collation
algorithm version in the new _versions endpoint [2], and some other
minor fixes in that area. As the next steps we could:

  1) Modify our views to keep track of the collation algorithm
version. We could attempt to transparently upgrade the view header
format -- read the old view file, update the header with an extra
libicu collation version field, that updates the signature, and then,
save the file with the new header and new signature. This avoids view
rebuilds, just records the collator version in the view and moves the
files to a new name.

  2) Do what PostgreSQL does, and 2a) emit a warning with the view
results when the current libicu version doesn't match the version in
the view [3]. That means altering the view results to add a "warning":
"..." field. Another alternative 2b) is emit a warning in the
_design/$ddoc/_info only. Users would have to know that after an OS
version upgrade, or restoring backups, to make sure to look at their
_design/$ddoc/_info for each db for each ddoc. Of course, there may be
users which used the "raw" collation option, or know they are using
just the plain ASCII character sets in their views. So we'd have a
configuration setting to ignore the warnings as well.

  3) Users who see the warning, could then either rebuild the view
with the new collator library manually, or it could happen
automatically based on a configuration option, basically "when
collator versions are miss-matched, invalidate and rebuild all the
views".

  4) We'd have a way for the users to assert (POST a ddoc update) that
they double-checked the new ICU version and are convinced that a
particular view would not experience data loss with the new collator.
That should make the warning go away, and the view to not be rebuilt.
This can't be just a naive "collator" option setting as both per-view
and per-design options are used when computing the view signature, and
any changes there would result in the view being rebuilt. Perhaps we
can add it to the design docs as a separate option which is excluded
from the signature hash, like the "autoupdate" setting for background
index builder ("collation_version_accept"?). PostgreSQL also offers
this option with the ALTER COLLATION ... REFRESH VERSION command [3]

What do we think, is this a reasonable approach? Is there something
easier / simpler we can do?

Thanks!
-Nick

[1] 
https://github.com/apache/couchdb/pull/3746/commits/28f26f52fe2e170d98658311dafa8198d96b8061
[2] 
https://github.com/apache/couchdb/commit/c1bb4e4856edd93255d75ebe158b4da38bbf
[3] https://www.postgresql.org/docs/13/sql-altercollation.html

Re: [DISCUSS] improving visibility for CouchDB-maintained independent Erlang apps

[Nick Vatamaniuc]

Good idea, Adam. Some of those are pretty nice applications.

Technically there are difficulties around rebar3 compatibility for
NIFs. It might be easier to make them developer friendly after we
switched the whole CouchDB project to rebar3. The non-NIF apps can be
used already as source dependencies. For example: {ets_lru, {git,
"https://github.com/apache/couchdb-ets-lru.git;, {tag, "1.1.0"}}}. We
also create git tags when we update them so it's easy to refer to
those, instead of just plain commit ids. But documentation is still
missing and that would be nice to add.

Regarding releases, anyone can clone these repos and use them, but I
guess we cannot say they are ASF project "releases". Is that a correct
interpretation? Not sure if adding a disclaimer to the docs might
help, too.

There is also another direction we can take some applications like
config, for example. I think it might make sense to pull that one into
the main repo. Over the years it got tangled enough with CouchDB
specifics (couch_log, etc) that having it as separate is more of an
inconvenience than a benefit at this point.

Cheers,
-Nick

On Fri, Oct 29, 2021 at 9:53 AM Adam Kocoloski  wrote:
>
> Ah! That .asf.yaml configuration is neat, thanks for pointing it out.
>
> Good point on the source releases — do you have anything in particular in 
> mind when you talk about making it simpler for the community at large?
>
> Adam
>
> > On Oct 28, 2021, at 10:48 PM, Joan Touzet  wrote:
> >
> > On 28/10/2021 16:44, Adam Kocoloski wrote:
> >> I think we could benefit from making these projects more visible. Paul’s 
> >> jiffy library for JSON processing is a nice counterexample where he's 
> >> gotten non-trivial contributions from the broader Erlang community by 
> >> putting a little distance between it and CouchDB. Do others agree?
> >
> > No opinion here, but:
> >
> >> If so, I’ve been thinking about some next steps that would help on that 
> >> front:
> >> - activating GH Issues (and maybe even Discussions?)
> >
> > The only ASF thing is that we must ensure that all of the Issues traffic 
> > ends up on a mailing list for archival purposes. (I think Infra are still 
> > working on mirroring Discussions traffic.)
> >
> > Mirroring that traffic and enabling issues is in fact self-serve now:
> >
> > https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-Notificationsettingsforrepositories
> >
> > and
> >
> > https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-Repositoryfeatures
> >
> > If you want Discussions you have to open an Infra ticket.
> >
> >> - releases published in Hex? (bit worried about the interaction with ASF 
> >> release requirements here)
> >
> > "Real" source releases would still have to go through voting and 3 +1 PMC 
> > votes - this could be made simpler for the community at large. with 
> > documentation on how to test these applications independently of CouchDB.
> >
> > Binary releases aren't "real" releases, but they would need to be paired 
> > with actual ASF releases due to policy. No problem with them being on Hex 
> > just as there's no problem with us being on Docker or any other binary 
> > stream, just so long as we do the "official" dance first.
> >
> > -Joan
>

Re: [VOTE] Release CouchDB 3.2.0 (RC2 round)

[Nick Vatamaniuc]

It has been 72+ hours. Vote passes. This becomes the 3.2.0 CouchDB release

Thank you for testing and voting!

Cheers,
-Nick

On Wed, Oct 6, 2021 at 9:06 AM Jan Lehnardt  wrote:
>
> +1
>
> sigs check out, Fauxton looks good, validating the install works too.
>
> `make check` passes on Erlang 22, but `make elixir` tests fail[1] with
> Erlang 20 & 21 I didn’t have time to look into this too much, but it
> looks like there is something in the build that requires Erlang 22 for
> elixir tests.
>
> `make eunit` passes in 20 and 21 as well.
>
> Tested with Spidermonkey 60.
>
> Nicely done everyone!
>
> Best
> Jan
> —
> [1] with Erlang 20, many repeated lines of:
>
> (no error logger present) error: "beam/beam_load.c(1863): Error loading 
> module error_logger:\n  This BEAM file was compiled for a later version of 
> the run-time system than 20.\n  To fix this, please recompile this module 
> with an 20 compiler.\n  (Use of opcode 164; this emulator supports only up to 
> 159.)\n”
>
> With Erlang 21, an infinite loop of:
>
> (no logger present) unexpected logger message: 
> {log,error,"~s~n",["beam/beam_load.c(1878): Error loading module file:\n  
> This BEAM file was compiled for a later version of the run-time system than 
> 21.\n  To fix this, please recompile this module with an 21 compiler.\n  (Use 
> of opcode 168; this emulator supports only up to 
> 163.)\n"],#{error_logger=>#{emulator=>true,tag=>error},gl=><0.0.0>,pid=><0.22.0>,time=>1633523875486343}}
>
>
> that can only be stopped with `kill -9 `
>
>
>
> > On 5. Oct 2021, at 21:30, Nick Vatamaniuc  wrote:
> >
> > Dear community,
> >
> > I would like to propose that we release Apache CouchDB 3.2.0.
> >
> > Candidate release notes:
> >
> >https://docs.couchdb.org/en/latest/whatsnew/3.2.html
> >
> > This is the RC2 round. Changes since last round:
> >https://github.com/apache/couchdb/compare/3.2.0-RC1...3.2.0-RC2
> >
> > We encourage the whole community to download and test these release
> > artefacts so that any critical issues can be resolved before the
> > release is made. Everyone is free to vote on this release, so dig
> > right in! (Only PMC members have binding votes, but they depend on
> > community feedback to gauge if an official release is ready to be
> > made.)
> >
> > The release artefacts we are voting on are available here:
> >
> >https://dist.apache.org/repos/dist/dev/couchdb/source/3.2.0/rc.2/
> >
> > There, you will find a tarball, a GPG signature, and SHA256/SHA512 
> > checksums.
> >
> > Please follow the test procedure here:
> >
> >
> > https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release
> >
> > Please remember that "RC2" is an annotation. If the vote passes, these
> > artefacts will be released as Apache CouchDB 3.2.0.
> >
> > Please cast your votes now.
> >
> > Thanks,
> > -Nick
>

Re: [VOTE] Release CouchDB 3.2.0 (RC2 round)

[Nick Vatamaniuc]

+1

Tested on MacOS 11.6, Erlang 20

Signature: ok
Checksums: ok
make check: ok
make release and starting bin/couchdb: ok
http://localhost:5984/_utils (Fauxton): ok
 - Created 2 DBs
 - Setup a replication job between them
 - Docs from first db replicated to the second db

-Nick

On Tue, Oct 5, 2021 at 4:49 PM Joan Touzet  wrote:
>
> Tested on Windows 10. Checksums match. Signature OK but still needs
> web-of-trust:
> -
> gpg: assuming signed data in 'apache-couchdb-3.2.0-RC2.tar.gz'
> gpg: Signature made 05/10/2021 15:21:31 Eastern Summer Time
> gpg:using RSA key 0BD7A98499C4AB41C910EE65FC04DFBC9657A78E
> gpg: Good signature from "Nicolae Vatamaniuc "
> [unknown]
> gpg: aka "default " [unknown]
> gpg: WARNING: This key is not certified with a trusted signature!
> gpg:  There is no indication that the signature belongs to the
> owner.
> Primary key fingerprint: 0BD7 A984 99C4 AB41 C910  EE65 FC04 DFBC 9657 A78E
> -
>
> make -f Makefile.win check PASSes.
>
> Windows .msi package builds and installs on another machine:
>
> {"couchdb":"Welcome","version":"3.2.0","git_sha":"efb409bba","uuid":"c0a4fee320344dc32290b064608baceb","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The
> Apache Software Foundation"}}
>
> git_sha matches the last checkin by nickva to bump the docs for 3.2.0-RC2.
>
> Fauxton Verify Installation button comes back with all checkmarks.
>
> +1 - and good work everyone!
>
> -Joan "please no more 7AM woodchippers outside my front door" Touzet
>
> On 05/10/2021 15:30, Nick Vatamaniuc wrote:
> > Dear community,
> >
> > I would like to propose that we release Apache CouchDB 3.2.0.
> >
> > Candidate release notes:
> >
> > https://docs.couchdb.org/en/latest/whatsnew/3.2.html
> >
> > This is the RC2 round. Changes since last round:
> > https://github.com/apache/couchdb/compare/3.2.0-RC1...3.2.0-RC2
> >
> > We encourage the whole community to download and test these release
> > artefacts so that any critical issues can be resolved before the
> > release is made. Everyone is free to vote on this release, so dig
> > right in! (Only PMC members have binding votes, but they depend on
> > community feedback to gauge if an official release is ready to be
> > made.)
> >
> > The release artefacts we are voting on are available here:
> >
> > https://dist.apache.org/repos/dist/dev/couchdb/source/3.2.0/rc.2/
> >
> > There, you will find a tarball, a GPG signature, and SHA256/SHA512 
> > checksums.
> >
> > Please follow the test procedure here:
> >
> > 
> > https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release
> >
> > Please remember that "RC2" is an annotation. If the vote passes, these
> > artefacts will be released as Apache CouchDB 3.2.0.
> >
> > Please cast your votes now.
> >
> > Thanks,
> > -Nick
> >

[VOTE] Release CouchDB 3.2.0 (RC2 round)

[Nick Vatamaniuc]

Dear community,

I would like to propose that we release Apache CouchDB 3.2.0.

Candidate release notes:

https://docs.couchdb.org/en/latest/whatsnew/3.2.html

This is the RC2 round. Changes since last round:
https://github.com/apache/couchdb/compare/3.2.0-RC1...3.2.0-RC2

We encourage the whole community to download and test these release
artefacts so that any critical issues can be resolved before the
release is made. Everyone is free to vote on this release, so dig
right in! (Only PMC members have binding votes, but they depend on
community feedback to gauge if an official release is ready to be
made.)

The release artefacts we are voting on are available here:

https://dist.apache.org/repos/dist/dev/couchdb/source/3.2.0/rc.2/

There, you will find a tarball, a GPG signature, and SHA256/SHA512 checksums.

Please follow the test procedure here:

https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release

Please remember that "RC2" is an annotation. If the vote passes, these
artefacts will be released as Apache CouchDB 3.2.0.

Please cast your votes now.

Thanks,
-Nick

Re: [VOTE] Release Apache CouchDB 3.1.2

[Nick Vatamaniuc]

Thank you, all!

It looks like the vote passed and this will become the 3.1.2 release.

-Nick

On Tue, Sep 28, 2021 at 7:41 PM Joan Touzet  wrote:
>
> On 28/09/2021 10:53, Nick Vatamaniuc wrote:
> > Thanks for taking a look, Joan
> >
> > I just highlighted the docs in the email diff but the tarball should
> > contain the differences and they should match the differences between
> > the 3.1.1 and 3.1.2-RC1 tags in git.
>
> Got it. I missed the -documentation part of the URL.
>
> Tested on Windows. Checksums confirmed. Signature matches, but Nick you
> need to get yourself to a key signing party! :D
>
> > gpg: Good signature from "Nicolae Vatamaniuc " 
> > [unknown]
> > gpg: aka "default " [unknown]
> > gpg: WARNING: This key is not certified with a trusted signature!
> > gpg:  There is no indication that the signature belongs to the 
> > owner.
>
> I have a repeatable error on make check:
>
> test/javascript/tests\replicator_db_bad_rep_id.js
> Error: false
> Trace back (most recent call first):
>
>  52:47: test/javascript/test_setup.js
>   T
>  41:5: test/javascript/tests\replicator_db_bad_rep_id.js
>   rep_doc_with_bad_rep_id
>  88:5: test/javascript/tests\replicator_db_bad_rep_id.js
>   couchTests.replicator_db_bad_rep_id
>  50:5: test/javascript/cli_runner.js
>   runTest
>  63:1: test/javascript/cli_runner.js
>
> However, because the JS test suite is finally all gone in 3.2/3.x, and
> no one else is reporting this failure on Linux, and the new eunit tests
> pass, AND because this part of the code didn't change, I'm going to
> chalk it up to weirdness and ignore it.
>
> Windows package installed locally and verified functional thru the
> Fauxton wizard:
>
> {"couchdb":"Welcome","version":"3.1.2","git_sha":"572b68e72","uuid":"c0a4fee320344dc32290b064608baceb","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The
> Apache Software Foundation"}}
>
> +1
>
> >
> > Regards,
> > -Nick
> >
> > On Mon, Sep 27, 2021 at 8:36 PM Joan Touzet  wrote:
> >>
> >> Nick, that diff shows documentation only. Something seems wrong with
> >> this release.
> >>
> >> Recommend we hold the vote until this can be cleared up?
> >>
> >> -Joan
> >>
> >> On 27/09/2021 08:53, Nick Vatamaniuc wrote:
> >>> Dear community,
> >>>
> >>> I would like to propose that we release Apache CouchDB 3.1.2
> >>>
> >>> Changes since 3.1.1
> >>>
> >>> 
> >>> https://github.com/apache/couchdb-documentation/compare/3.1.1...3.1.x?expand=1
> >>>
> >>> This is a minor release where we backport a few features from the
> >>> pending 3.2 release to the 3.1.x branch.
> >>>
> >>> We encourage the whole community to download and test these release
> >>> artefacts so that any critical issues can be resolved before the
> >>> release is made. Everyone is free to vote on this release, so dig
> >>> right in! (Only PMC members have binding votes, but they depend on
> >>> community feedback to gauge if an official release is ready to be
> >>> made.)
> >>>
> >>> The release artefacts we are voting on are available here:
> >>>
> >>> https://dist.apache.org/repos/dist/dev/couchdb/source/3.1.2/rc.1/
> >>>
> >>> There, you will find a tarball, a GPG signature, and SHA256/SHA512 
> >>> checksums.
> >>>
> >>> Please follow the test procedure here:
> >>>
> >>> 
> >>> https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release
> >>>
> >>> Please remember that "RC1" is an annotation. If the vote passes, these
> >>> artefacts will be released as Apache CouchDB 3.1.2
> >>>
> >>> Please cast your votes.
> >>>
> >>> Thanks,
> >>> -Nick
> >>>

Re: [VOTE] Release Apache CouchDB 3.2.0

[Nick Vatamaniuc]

Thanks everyone for verifying

RC1 voting fails and we will issue an other release candidate

So far the issues are:

  1) Building fails on Windows : Thanks Joan, for verifying and debugging.

  2) For the FreeBSD build issue Dave found a fix
https://github.com/apache/couchdb/pull/3770 and it's already merge.
Thanks, Dave!

  3) couch_prometheus_e2e_tests:109: deny_prometheus_http...*failed* :
this looks like a network stack issue difference. We test that when we
disable the additional port we get a econnrefused error (passed
through from the kernel / libc) but on FreeBSD it might come in as a
timeout. Is that a configuration option that can be tweaked on FreeBSD
to return econnrefused, we could also update the test to accept
timeout as a valid response but it's a generic test run timeout so
that seems not as optimal.
https://github.com/apache/couchdb/blob/3.x/src/couch_prometheus/test/eunit/couch_prometheus_e2e_tests.erl#L109

-Nick

On Wed, Sep 29, 2021 at 11:45 AM Dave Cottlehuber  wrote:
>
> On Mon, 27 Sep 2021, at 22:59, Nick Vatamaniuc wrote:
> > Dear community,
> >
> > I would like to propose that we release Apache CouchDB 3.2.0.
> >
> > Candidate release notes:
> >
> > https://docs.couchdb.org/en/latest/whatsnew/3.2.html
> >
> > We encourage the whole community to download and test these release
> > artefacts so that any critical issues can be resolved before the
> > release is made. Everyone is free to vote on this release, so dig
> > right in! (Only PMC members have binding votes, but they depend on
> > community feedback to gauge if an official release is ready to be
> > made.)
> >
> > The release artefacts we are voting on are available here:
> >
> > https://dist.apache.org/repos/dist/dev/couchdb/source/3.2.0/rc.1/
> >
> > There, you will find a tarball, a GPG signature, and SHA256/SHA512 
> > checksums.
> >
> > Please follow the test procedure here:
> >
> > 
> > https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release
> >
> > Please remember that "RC1" is an annotation. If the vote passes, these
> > artefacts will be released as Apache CouchDB 3.2.0.
> >
> > Please cast your votes now.
> >
> > Thanks,
> > -Nick
>
> -1
>
> But with Jan and Nick's help we have a nice little patch for this:
>
> https://github.com/apache/couchdb/pull/3770
>
> - FreeBSD 13.0-RELEASE p4 amd64
> - elixir 1.12.3
> - erlang 24.0.6
> - spidermonkey 78
> - python3.8.12
> - default CC (clang 11.0.1)
>
> After applying the patch I get this error from prometheus tests:
>
> couch_prometheus_e2e_tests:109: deny_prometheus_http...*failed*
> in function couch_prometheus_e2e_tests:'-deny_prometheus_http/1-fun-1-'/1 
> (test/eunit/couch_prometheus_e2e_tests.erl, line 109)
> in call from eunit_test:run_testfun/1 (eunit_test.erl, line 71)
> in call from eunit_proc:run_test/1 (eunit_proc.erl, line 522)
> in call from eunit_proc:with_timeout/3 (eunit_proc.erl, line 347)
> in call from eunit_proc:handle_test/2 (eunit_proc.erl, line 505)
> in call from eunit_proc:tests_inorder/3 (eunit_proc.erl, line 447)
> in call from eunit_proc:with_timeout/3 (eunit_proc.erl, line 337)
> in call from eunit_proc:run_group/2 (eunit_proc.erl, line 561)
> **error:{assertEqual,[{module,couch_prometheus_e2e_tests},
>   {line,109},
>   {expression,"Response"},
>   {expected,{error,{conn_failed,{error,econnrefused,
>   {value,{error,req_timedout}}]}
>   output:<<"">>
>
> couch_prometheus_e2e_tests:80: node_see_updated_metrics...ok
> [done in 31.151 s]
>   [done in 32.485 s]
>
> A+
> Dave

Re: [VOTE] Release Apache CouchDB 3.2.0

[Nick Vatamaniuc]

Thanks for checking, Joan.

What version of Erlang was it?

I recall we excluded erl_interface from rebar and enc (port compiler)
so we could compile on Erlang 23+, but I don't think we ever updated
the windows flags:

A few instances:

 
https://github.com/apache/couchdb-rebar/blob/ee85b6920474e6e1b9c1802b092e022720b6b676/src/rebar_port_compiler.erl#L723-L724

 
https://github.com/davisp/erlang-native-compiler/commit/0302390774db4104bfb46383d2819ed06ce78d1f#diff-fb2f03770e7203b48a9f4581e5bf983c213191888b048438f33e52acfcc42232R745-R746

Would removing those lines help? Sadly I don't have a Windows dev
setup to give it a try.

-Nick


On Tue, Sep 28, 2021 at 8:02 PM Joan Touzet  wrote:
>
> Unfortunately, I must vote -1 for 3.2.0. On Windows, I cannot compile
> the Jiffy we are including in 3.2 anymore due to a linking error:
>
> > LINK : fatal error LNK1181: cannot open input file 'erl_interface.lib'
>
> This is the same system that just finished compiling and testing 3.1.2,
> no other changes.
>
> I don't have more time for this tonight, so I'll have to investigate
> tomorrow or later this week.
>
> -Joan
>
> On 28/09/2021 07:19, Juanjo Rodríguez wrote:
> > +1
> >
> > Tested on Ubuntu 20.04, Erlang 22 and Elixir 1.9.1
> > Signature: Ok
> > Checksums: Ok
> > Tests: (configure --spidermonkey-version 68 make check): Pass
> >
> > Additional info: Also pass LightCouch Java Client and Cloudant Sync for
> > Android test suites.
> >
> > Thanks!!
> >
> >
> > El mar, 28 sept 2021 a las 1:00, Nick Vatamaniuc ()
> > escribió:
> >
> >> Dear community,
> >>
> >> I would like to propose that we release Apache CouchDB 3.2.0.
> >>
> >> Candidate release notes:
> >>
> >> https://docs.couchdb.org/en/latest/whatsnew/3.2.html
> >>
> >> We encourage the whole community to download and test these release
> >> artefacts so that any critical issues can be resolved before the
> >> release is made. Everyone is free to vote on this release, so dig
> >> right in! (Only PMC members have binding votes, but they depend on
> >> community feedback to gauge if an official release is ready to be
> >> made.)
> >>
> >> The release artefacts we are voting on are available here:
> >>
> >> https://dist.apache.org/repos/dist/dev/couchdb/source/3.2.0/rc.1/
> >>
> >> There, you will find a tarball, a GPG signature, and SHA256/SHA512
> >> checksums.
> >>
> >> Please follow the test procedure here:
> >>
> >>
> >> https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release
> >>
> >> Please remember that "RC1" is an annotation. If the vote passes, these
> >> artefacts will be released as Apache CouchDB 3.2.0.
> >>
> >> Please cast your votes now.
> >>
> >> Thanks,
> >> -Nick
> >>
> >

Re: [VOTE] Release Apache CouchDB 3.1.2

[Nick Vatamaniuc]

Thanks for taking a look, Joan

I just highlighted the docs in the email diff but the tarball should
contain the differences and they should match the differences between
the 3.1.1 and 3.1.2-RC1 tags in git.

Regards,
-Nick

On Mon, Sep 27, 2021 at 8:36 PM Joan Touzet  wrote:
>
> Nick, that diff shows documentation only. Something seems wrong with
> this release.
>
> Recommend we hold the vote until this can be cleared up?
>
> -Joan
>
> On 27/09/2021 08:53, Nick Vatamaniuc wrote:
> > Dear community,
> >
> > I would like to propose that we release Apache CouchDB 3.1.2
> >
> > Changes since 3.1.1
> >
> > 
> > https://github.com/apache/couchdb-documentation/compare/3.1.1...3.1.x?expand=1
> >
> > This is a minor release where we backport a few features from the
> > pending 3.2 release to the 3.1.x branch.
> >
> > We encourage the whole community to download and test these release
> > artefacts so that any critical issues can be resolved before the
> > release is made. Everyone is free to vote on this release, so dig
> > right in! (Only PMC members have binding votes, but they depend on
> > community feedback to gauge if an official release is ready to be
> > made.)
> >
> > The release artefacts we are voting on are available here:
> >
> > https://dist.apache.org/repos/dist/dev/couchdb/source/3.1.2/rc.1/
> >
> > There, you will find a tarball, a GPG signature, and SHA256/SHA512 
> > checksums.
> >
> > Please follow the test procedure here:
> >
> > 
> > https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release
> >
> > Please remember that "RC1" is an annotation. If the vote passes, these
> > artefacts will be released as Apache CouchDB 3.1.2
> >
> > Please cast your votes.
> >
> > Thanks,
> > -Nick
> >

[VOTE] Release Apache CouchDB 3.2.0

[Nick Vatamaniuc]

Dear community,

I would like to propose that we release Apache CouchDB 3.2.0.

Candidate release notes:

https://docs.couchdb.org/en/latest/whatsnew/3.2.html

We encourage the whole community to download and test these release
artefacts so that any critical issues can be resolved before the
release is made. Everyone is free to vote on this release, so dig
right in! (Only PMC members have binding votes, but they depend on
community feedback to gauge if an official release is ready to be
made.)

The release artefacts we are voting on are available here:

https://dist.apache.org/repos/dist/dev/couchdb/source/3.2.0/rc.1/

There, you will find a tarball, a GPG signature, and SHA256/SHA512 checksums.

Please follow the test procedure here:

https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release

Please remember that "RC1" is an annotation. If the vote passes, these
artefacts will be released as Apache CouchDB 3.2.0.

Please cast your votes now.

Thanks,
-Nick

Re: [VOTE] Release Apache CouchDB 3.1.2

[Nick Vatamaniuc]

My own vote:

Signature: Ok
Checksums: Ok
Tests (make check): Pass

+1

On Mon, Sep 27, 2021 at 8:53 AM Nick Vatamaniuc  wrote:
>
> Dear community,
>
> I would like to propose that we release Apache CouchDB 3.1.2
>
> Changes since 3.1.1
>
> 
> https://github.com/apache/couchdb-documentation/compare/3.1.1...3.1.x?expand=1
>
> This is a minor release where we backport a few features from the
> pending 3.2 release to the 3.1.x branch.
>
> We encourage the whole community to download and test these release
> artefacts so that any critical issues can be resolved before the
> release is made. Everyone is free to vote on this release, so dig
> right in! (Only PMC members have binding votes, but they depend on
> community feedback to gauge if an official release is ready to be
> made.)
>
> The release artefacts we are voting on are available here:
>
> https://dist.apache.org/repos/dist/dev/couchdb/source/3.1.2/rc.1/
>
> There, you will find a tarball, a GPG signature, and SHA256/SHA512 checksums.
>
> Please follow the test procedure here:
>
> 
> https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release
>
> Please remember that "RC1" is an annotation. If the vote passes, these
> artefacts will be released as Apache CouchDB 3.1.2
>
> Please cast your votes.
>
> Thanks,
> -Nick

[VOTE] Release Apache CouchDB 3.1.2

[Nick Vatamaniuc]

Dear community,

I would like to propose that we release Apache CouchDB 3.1.2

Changes since 3.1.1


https://github.com/apache/couchdb-documentation/compare/3.1.1...3.1.x?expand=1

This is a minor release where we backport a few features from the
pending 3.2 release to the 3.1.x branch.

We encourage the whole community to download and test these release
artefacts so that any critical issues can be resolved before the
release is made. Everyone is free to vote on this release, so dig
right in! (Only PMC members have binding votes, but they depend on
community feedback to gauge if an official release is ready to be
made.)

The release artefacts we are voting on are available here:

https://dist.apache.org/repos/dist/dev/couchdb/source/3.1.2/rc.1/

There, you will find a tarball, a GPG signature, and SHA256/SHA512 checksums.

Please follow the test procedure here:

https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release

Please remember that "RC1" is an annotation. If the vote passes, these
artefacts will be released as Apache CouchDB 3.1.2

Please cast your votes.

Thanks,
-Nick

Re: CouchDB 3.1.1 Compaction

[Nick Vatamaniuc]

Hi Farhan,

You can configure time windows during which compaction would take
place. See 
https://docs.couchdb.org/en/stable/maintenance/compaction.html#scheduling-windows

Cheers,
-Nick


On Thu, Aug 12, 2021 at 5:05 AM Farhan Faiz  wrote:
>
> Hi,
>
> I want to know when does Compaction occurs by default in CouchDB 3.1.1.
> I have not set anything, everything is the default. And compaction occurs 
> every week on different days which is causing heavy load on the servers. 
> Please advise me on the best practice for reducing the load during compaction.

Re: Reformat src files with `erlfmt` on `main`

[Nick Vatamaniuc]

Good idea, Donát! Thanks for giving it a try.

Even though I personally think emilio has a better style, and I don't
agree with some of the choices erlfmt makes, I still think it would
still be a net benefit to having consistency if we get automated code
reformatting. So +1 overall.

If there are modules or areas we really don't want reformatted, erlfmt
allows comment pragmas (%% erlfmt-ignore [1]) to disable re-formatting
for some sections of code. There are also some suggestions to do minor
manual refactorings which result in a better formatted code [2] , so
we could apply a few of those in some places.

A few more ideas and comments I had were in the PR comment
https://github.com/apache/couchdb/pull/3568#issuecomment-842569320

[1] https://github.com/WhatsApp/erlfmt#ignoring-formatting
[2] https://github.com/WhatsApp/erlfmt#manual-interventions

Cheers,
-Nick

On Tue, May 18, 2021 at 2:13 PM Bessenyei Balázs Donát
 wrote:
>
> Hi dev@couchdb,
>
> To eliminate the need for formatting-related comments and thus
> unnecessary cycles in PRs, I've invested a little time to see if we
> could use a formatter on `main` [1].
> The PR reformats `.erl` files in `src` and the script [2] included
> shows that the compiled binaries match "before" and "after".
> The formatter used in the PR is `erlfmt` [3] which is an opinionated
> [4] tool so it's more of a "take it or leave it" as-is. (We could try
> using other formatters if we want in case people want formatting but
> think the choices `erlfmt` makes are unacceptable.)
> Some members of the CouchDB dev community already left some great
> comments on the PR and I haven't seen any strong opposition so far,
> but I wanted to make sure more people are aware of this.
> If you have any questions, comments or concerns (or objections),
> please let me know.
>
>
> Thank you,
>
> Donat
>
>
> [1]: https://github.com/apache/couchdb/pull/3568
> [2]: 
> https://github.com/apache/couchdb/pull/3568/files#diff-7adfbc2d8dba4d4ff49ff2b760b81c006097f20f412ea2007f899042fd0de98a
> [3]: https://github.com/WhatsApp/erlfmt
> [4]: 
> https://github.com/WhatsApp/erlfmt#comparison-with-other-erlang-formatters

Re: [DISCUSSION] Clean up non-functioning applications from main

[Nick Vatamaniuc]

I created the PR https://github.com/apache/couchdb/pull/3503

The first set of commits remove and cleanup applications, settings and
url handlers. Then, the rest of the commits focus on fixing the
functionality to adjust for the removal: updating include paths,
switching to use the newer couch_views utility modules, and making
sure couch unit tests run. For couch_server, I had started to replace
it but, since it's called in quite a few places and the PR was getting
quite large, I stopped with just pairing it down to the minimal
functionality. I tried to add descriptive commit messages for some
issues encountered. The tricker one is probably where couch_views is
updated with utility functions from the old couch_mrview and
couch_index applications.

Thanks,
-Nick

On Tue, Apr 13, 2021 at 11:03 AM Paul Davis  wrote:
>
> Another +1 to removing as much as possible and building back anything
> that we feel is appropriate for main.
>
> On Mon, Apr 12, 2021 at 1:31 PM Robert Newson  wrote:
> >
> > +1 to all the proposed cuts.
> >
> > I’m keen to see couch_server.erl itself go, so its remaining uses need new 
> > homes (couch_passwords an obvious choice for the hashing referred to, etc).
> >
> > I’m inferring that neither purge and global_changes work on main anyway, 
> > but they can still be called and will route to 3.x code. Agree that it’s 
> > better to stub those out (send a 503 I guess?) in the short term and either 
> > re-implement on FDB or (as Joan said) vote on their permanent removal. 
> > (Noting that a much better implementation of purge and global_changes seems 
> > possible with FDB though less clear if the effort is justified).
> >
> > So, in brief, remove absolutely all the obsoleted, unreachable code as soon 
> > as possible, then once the dust has settled we can see if there are obvious 
> > gaps we should fill in before the 4.0 release.
> >
> > B.
> >
> > > On 12 Apr 2021, at 18:51, Nick Vatamaniuc  wrote:
> > >
> > > The current versions of those apps rely on mem3, clustering, adding
> > > nodes, etc and they will trail behind the 3.x versions since
> > > developers wouldn't think to port those updates to main since they are
> > > simply non-functional there. Most of those apps have to be re-written
> > > from scratch and it would be better to start from the recent working
> > > versions on 3.x.  The tests for those apps don't really fail as we get
> > > green builds on PR branches to main. We simply don't run them at all
> > > and only run a subset of applications (fabric, couch_jobs, couch_views
> > > and a few others).
> > >
> > > Don't think this is about a 4.x release per-se. This is mainly about
> > > cleaning up, reducing the cognitive load of anyone jumping in trying
> > > to work on main and seeing applications and endpoints calling into
> > > non-existing applications.
> > >
> > > -Nick
> > >
> > >
> > > -Nick
> > >
> > > On Mon, Apr 12, 2021 at 1:13 PM Joan Touzet  wrote:
> > >>
> > >> Generally +1 with one major reservation:
> > >>
> > >> On 12/04/2021 12:25, Nick Vatamaniuc wrote:
> > >>> * Some applications we want to have in main, but the way they are
> > >>> implemented currently rely completely or mostly on 3.x code: purge
> > >>> logic, couch_peruser, global_changes, setup. I am thinking it may be
> > >>> better to remove them from main as we'll have them on the 3.x branch
> > >>> they'll be recent (working) there. When we're ready to fix them up, we
> > >>> can copy that code from there to the main branch.
> > >>
> > >> If the intent is to release 4.0 with them, then I would suggest keeping
> > >> them there and allowing their tests to fail so we know that a "failing
> > >> main" means that the product isn't ready to release yet.
> > >>
> > >> If we are pushing these out past a 4.0 release, then that decision needs
> > >> to be made formally.
> > >>
> > >> Parenthetically, we try to avoid "code owners" here, but usually fixes
> > >> to couch_peruser and setup fall to Jan, while purge and global_changes I
> > >> *believe* have generally been made by IBM/Cloudant.
> > >>
> > >> -Joan "not sure main is ready to be called 4.0 yet anyway" Touzet
> >

Re: [DISCUSSION] Clean up non-functioning applications from main

[Nick Vatamaniuc]

The current versions of those apps rely on mem3, clustering, adding
nodes, etc and they will trail behind the 3.x versions since
developers wouldn't think to port those updates to main since they are
simply non-functional there. Most of those apps have to be re-written
from scratch and it would be better to start from the recent working
versions on 3.x.  The tests for those apps don't really fail as we get
green builds on PR branches to main. We simply don't run them at all
and only run a subset of applications (fabric, couch_jobs, couch_views
and a few others).

Don't think this is about a 4.x release per-se. This is mainly about
cleaning up, reducing the cognitive load of anyone jumping in trying
to work on main and seeing applications and endpoints calling into
non-existing applications.

-Nick


-Nick

On Mon, Apr 12, 2021 at 1:13 PM Joan Touzet  wrote:
>
> Generally +1 with one major reservation:
>
> On 12/04/2021 12:25, Nick Vatamaniuc wrote:
> >  * Some applications we want to have in main, but the way they are
> > implemented currently rely completely or mostly on 3.x code: purge
> > logic, couch_peruser, global_changes, setup. I am thinking it may be
> > better to remove them from main as we'll have them on the 3.x branch
> > they'll be recent (working) there. When we're ready to fix them up, we
> > can copy that code from there to the main branch.
>
> If the intent is to release 4.0 with them, then I would suggest keeping
> them there and allowing their tests to fail so we know that a "failing
> main" means that the product isn't ready to release yet.
>
> If we are pushing these out past a 4.0 release, then that decision needs
> to be made formally.
>
> Parenthetically, we try to avoid "code owners" here, but usually fixes
> to couch_peruser and setup fall to Jan, while purge and global_changes I
> *believe* have generally been made by IBM/Cloudant.
>
> -Joan "not sure main is ready to be called 4.0 yet anyway" Touzet

[DISCUSSION] Clean up non-functioning applications from main

[Nick Vatamaniuc]

Hi all,

Currently we have a bunch of 3.x applications in the main branch which
are not functional. Some even start supervisors and expose HTTP
endpoints, like _reshard (mem3). Initially the idea was to keep those
applications there to make it easy to merge fixes back and forth, but
I wonder if it is time to clean them up.

I gave this a try locally and got it to where we pass `make check`.
Here is what it looks like:

 * mem3, rexi, ken, smoosh, ddoc_cache, dreyfus are pretty easy to remove.

 * couch_mrview and couch_index : a bit trickier as those are called
from couch_views for some utility functions. In some cases there is
some duplicated code, for example we have two ddoc_to_mrst copies with
one a one line difference:
https://github.com/apache/couchdb/blob/main/src/couch_mrview/src/couch_mrview_util.erl#L142-L194
vs 
https://github.com/apache/couchdb/blob/main/src/couch_views/src/couch_views_util.erl#L31-L84.
I managed to consolidate that into couch_views and remove couch_mrview
and couch_index

 * couch app : Half of the modules here can be removed, btree,
couch_file, most of the couch_server functionality, couch_compress
along with snappy. But couch_server is still used in many parts of the
code to get the couch version, as the backend of the _uuid endpoint
and also for hashing passwords. I have stripped it down to do just
those 3 things.

 * Some applications we want to have in main, but the way they are
implemented currently rely completely or mostly on 3.x code: purge
logic, couch_peruser, global_changes, setup. I am thinking it may be
better to remove them from main as we'll have them on the 3.x branch
they'll be recent (working) there. When we're ready to fix them up, we
can copy that code from there to the main branch.

What does everyone think?

Thanks,
-Nick

Re: Removing "node" field from replicator "/_scheduler/{jobs | docs}"

[Nick Vatamaniuc]

That would work. Good idea


On Mon, Apr 5, 2021 at 3:12 PM Robert Newson  wrote:
>
> Several good points in there, Nick.
>
> How about a config toggle? A single config setting that decides if any 
> endpoint exposes “erlangness” (so at least node names, pids or registered 
> names). To be applied to active_tasks output also. Defaulting to true for 
> compatibility.
>
> > On 5 Apr 2021, at 17:17, Nick Vatamaniuc  wrote:
> >
> > The "node" field can be helpful in determining where the background
> > task runs even if nodes are not connected in a mesh. Nodes could still
> > be named something like replication_node_1, replication_node_2, etc.
> > Even in 3.x, the replicator doesn't rely on the nodes being meshed all
> > that much, the jobs start independently on each node based on how
> > _replicator doc shards are distributed.
> >
> > A few more thoughts around it:
> >
> > * If we're going to remove "node", should we also remove the "pid"
> > field? It seems odd to have one but not the other...
> >
> > * One of the reasons node and pid were added to _scheduler/*
> > endpoints was to eliminate the need for the users to ever look in
> > _active_tasks to check the status of their replication jobs. In other
> > words, having to tell a user "if you want stats look in
> > _scheduler/jobs, but if you want to find the node look in
> > _active_tasks".
> >
> >  * What about _active_tasks and "node" and "pid" there? Both view
> > indexing and replication jobs have those fields, should we remove them
> > too?
> >
> > So I think overall I am neutral (-0) on the idea. I can see how it may
> > be odd to have those internal details in there to start with, but I am
> > not sure meshing alone is a good reason either way. And if we're going
> > to do it, it may be better to be consistent with "pid" and "node" and
> > in regards to _active_tasks as well.
> >
> > (As a fun aside, the node and pid was once helpful in our test
> > environment in discovering that a different kube cluster was picking
> > up and executing indexing jobs due to a misconfigured [fabric]
> > fdb_directory config. Both clusters were sharing the same FDB cluster
> > underneath).
> >
> > Cheers,
> > -Nick
> >
> > On Fri, Apr 2, 2021 at 6:00 PM Bessenyei Balázs Donát  
> > wrote:
> >>
> >> I support removing obsolete fields from responses.
> >> I also support tracking API changes.
> >>
> >>
> >> Donat
> >>
> >> On Fri, Apr 2, 2021 at 10:23 PM Robert Newson  wrote:
> >>>
> >>> +1 to removing “node” on main (but not 3.x branch).
> >>>
> >>> B.
> >>>
> >>>> On 2 Apr 2021, at 21:11, Ilya Khlopotov  wrote:
> >>>>
> >>>> Hi,
> >>>>
> >>>> In FDB world there wouldn't be erlang mesh as far as I can tell. In this 
> >>>> situation the `node` field in the response from `/_scheduler/jobs`  and 
> >>>> `/_scheduler/docs` doesn't make sense.
> >>>>
> >>>> We could either remove the field or set it to `None`. I propose complete 
> >>>> removal.
> >>>>
> >>>> I also propose to establish a formal process to track API changes 
> >>>> formally. Sooner or latter we would need to compile list of changes 
> >>>> between versions. In case of a rewrite on top of FDB I suspect the 
> >>>> archeology process wouldn't be easy. We could create a github issue 
> >>>> template which would set necessary labels for example.
> >>>>
> >>>> Best regards,
> >>>> iilyak
> >>>>
> >>>
>

Re: [DISCUSS] API versioning redux

[Nick Vatamaniuc]

I think that's a good compromise if we want to bring more users along
from 3.x to 4.x.

In respect to existing users I can see there being at least 4
different categories:

1. Users who know about the snapshotting behavior of 1.x-3.x, and
their applications rely on it for correctness.
2. Users who don't know about the snapshotting, and their applications
rely on it for correctness.
3. Users who know about the snapshotting behavior, but their
applications don't rely on it for correctness.
4. Users who don't know about the snapshotting behavior and their
applications don't rely on it for correctness

I worry about 1 and especially 2. Documenting the differences in
behavior there would be crucial. Hopefully after learning about the
feature a bit more, the majority of those users will be able to tell
if it affects their application or not. At the same time, the users in
category 3 and 4 won't feel like they need to do a lot of grunt work
for seemingly no reason, if they want to use 4.x for scalability or
other reasons like not having to deal with spurious conflicts.

Good point that being explicit about versions would allow us to not
worry as much about in-place backward compatibility. I like the idea
of the v4 API getting all kinds of new and interesting features:
transactional PATCH across multiple docs, bulk docs with transactional
guarantees and so on -- it would offer enough of a  "carrot" for users
to look forward to and want to use the new API, instead of being
forced by the "stick" of the old API being removed.

As for the warnings and global vs per endpoint, I think global and no
warnings might be a good start?  We can always add warnings and
per-endpoint schemas later, but it's harder to remove them once we've
already added them.


-Nick

On Mon, Feb 22, 2021 at 8:48 PM Adam Kocoloski  wrote:
>
> Hi all,
>
> Several times in recent months we’ve had discussions on this list about 
> behavior changes in CouchDB 4.0 that boil down to tradeoffs between
>
> a) maintaining compatibility with current CouchDB APIs, and
> b) capitalizing on the transactional semantics of FoundationDB
>
> An example would be the support for large result sets in a view response: do 
> we stitch together multiple transactions to support very large responses, or 
> provide new pagination APIs and guarantee that each response is generated 
> from a single isolated snapshot?
>
> I’d like to suggest that we “have our cake and eat it, too” by introducing 
> versioned APIs as previously suggested by Ilya[1]. We’d have a V3 API that 
> strives to maximize compatibility with CouchDB 3.x, and a V4 API that still 
> looks and feels like CouchDB, but introduces breaking changes where necessary 
> to provide well-defined transactional semantics. I think this explicit 
> approach to API evolution could make life easier for client library 
> maintainers, and also free us up to improve the API without needing to be 
> quite as careful about in-place backwards compatibility.
>
> [1]: 
> https://lists.apache.org/thread.html/rcc742c0fdca0363bb338b54526045720868597ea35ee6842aef174e0%40%3Cdev.couchdb.apache.org%3E
>
> What do you think? Cheers, Adam

Re: Removing "node" field from replicator "/_scheduler/{jobs | docs}"

[Nick Vatamaniuc]

The "node" field can be helpful in determining where the background
task runs even if nodes are not connected in a mesh. Nodes could still
be named something like replication_node_1, replication_node_2, etc.
Even in 3.x, the replicator doesn't rely on the nodes being meshed all
that much, the jobs start independently on each node based on how
_replicator doc shards are distributed.

A few more thoughts around it:

 * If we're going to remove "node", should we also remove the "pid"
field? It seems odd to have one but not the other...

 * One of the reasons node and pid were added to _scheduler/*
endpoints was to eliminate the need for the users to ever look in
_active_tasks to check the status of their replication jobs. In other
words, having to tell a user "if you want stats look in
_scheduler/jobs, but if you want to find the node look in
_active_tasks".

  * What about _active_tasks and "node" and "pid" there? Both view
indexing and replication jobs have those fields, should we remove them
too?

So I think overall I am neutral (-0) on the idea. I can see how it may
be odd to have those internal details in there to start with, but I am
not sure meshing alone is a good reason either way. And if we're going
to do it, it may be better to be consistent with "pid" and "node" and
in regards to _active_tasks as well.

(As a fun aside, the node and pid was once helpful in our test
environment in discovering that a different kube cluster was picking
up and executing indexing jobs due to a misconfigured [fabric]
fdb_directory config. Both clusters were sharing the same FDB cluster
underneath).

Cheers,
-Nick

On Fri, Apr 2, 2021 at 6:00 PM Bessenyei Balázs Donát  wrote:
>
> I support removing obsolete fields from responses.
> I also support tracking API changes.
>
>
> Donat
>
> On Fri, Apr 2, 2021 at 10:23 PM Robert Newson  wrote:
> >
> > +1 to removing “node” on main (but not 3.x branch).
> >
> > B.
> >
> > > On 2 Apr 2021, at 21:11, Ilya Khlopotov  wrote:
> > >
> > > Hi,
> > >
> > > In FDB world there wouldn't be erlang mesh as far as I can tell. In this 
> > > situation the `node` field in the response from `/_scheduler/jobs`  and 
> > > `/_scheduler/docs` doesn't make sense.
> > >
> > > We could either remove the field or set it to `None`. I propose complete 
> > > removal.
> > >
> > > I also propose to establish a formal process to track API changes 
> > > formally. Sooner or latter we would need to compile list of changes 
> > > between versions. In case of a rewrite on top of FDB I suspect the 
> > > archeology process wouldn't be easy. We could create a github issue 
> > > template which would set necessary labels for example.
> > >
> > > Best regards,
> > > iilyak
> > >
> >

Re: [VOTE] Set a finite default for max_attachment_size

[Nick Vatamaniuc]

I am +1 to lowering as it's better than infinity.

But I also see Eric's point. I was surprised a while back just like
Eric that I could successfully upload >1GB-sized files.  So why not
0.5GB or 2GB? I am thinking 2GB was (is?) a common limit on some OSes
and file systems (FAT32) since they use ints for file size and
offsets. Since our attachment won't be saved as is in the file systems
inside a .couch file 2GB may be too high, so 1GB as a limit makes
sense to me.

-Nick

On Mon, Feb 1, 2021 at 1:25 PM Paul Davis  wrote:
>
> +1
>
> Default unlimited seems like an oversight regardless of what we change it to.
>
> On Mon, Feb 1, 2021 at 11:59 AM Eric Avdey  wrote:
> >
> > Maybe I didn't express myself clear enough. Setting some finit default is 
> > not a purpose, it's what you are doing and I'm asking what the reason for 
> > this change. In other words I'm not asking what are you doing, I'm asking 
> > why are you doing this.
> >
> > Introducing a new limit will be a breaking change to anoyone who uploads 
> > attachments larger than that limit, obviously, so "assumed 1G is large 
> > enough" sounds really arbitrary to me without any factual support for that 
> > assumption.
> >
> >
> > Eric
> >
> >
> > > On Feb 1, 2021, at 13:15, Bessenyei Balázs Donát  
> > > wrote:
> > >
> > > The purpose of this vote / PR is to set _some_ finite default. I went
> > > with 1G as I assumed that would not break anyone's production system.
> > > I'd support decreasing that limit over time.
> > >
> > > The vote has been open for 72 hours now, but I believe it still needs
> > > two more +1s to pass.
> > >
> > >
> > > Donat
> > >
> > > On Thu, Jan 28, 2021 at 10:44 PM Eric Avdey  wrote:
> > >>
> > >> This got me curious and I tried to upload Ubuntu image as an attachment. 
> > >> Interestingly CouchDB 3.x accepted first 1.4G of 2.8G file and then 
> > >> returned proper 201 response with a new doc revision, which I certanly 
> > >> didn't expect. Should say, that 1.4G seems suspiciously similar to a 
> > >> normal memory limit for a 32 bit process.
> > >>
> > >> Putting this aside, I agree that uploading large attachments is an 
> > >> anti-pattern and 1G seems excessive, hence my question. I'd expect this 
> > >> number to be based on something and correlating it with a  technical 
> > >> limit in 4.x makes a lot of sense to me.
> > >>
> > >>
> > >> Eric
> > >>
> > >>
> > >>> On Jan 28, 2021, at 16:02, Robert Newson  wrote:
> > >>>
> > >>> Hi,
> > >>>
> > >>> I think a gigabyte is _very_ generous given our experience of this 
> > >>> feature in practice.
> > >>>
> > >>> In 4.x attachment size will necessarily be much more restrictive, so it 
> > >>> seems prudent to move toward that limit.
> > >>>
> > >>> I don’t think many folks (hopefully no one!) is routinely inserting 
> > >>> attachments over 1 gib today, I’d be fairly surprised if it even works.
> > >>>
> > >>> B.
> > >>>
> >  On 28 Jan 2021, at 19:42, Eric Avdey  wrote:
> > 
> >  There is no justification neither here or on the PR for this change, 
> >  i.e. why this is done. Original infinity default was set to preserve 
> >  previous behaviour, this change will inadvertently break workflow for 
> >  users who upload large attachment and haven't set explicit default, so 
> >  why is it fine to do now? There might be some discussion around this 
> >  somewhere, but it'd be nice to include it here for sake of people like 
> >  me who's out of the loop.
> > 
> >  Also 1G limit seems arbitrary - how was it choosen?
> > 
> > 
> >  Thanks,
> >  Eric
> > 
> > 
> > 
> > > On Jan 28, 2021, at 01:46, Bessenyei Balázs Donát  
> > > wrote:
> > >
> > > Hi All,
> > >
> > > In https://github.com/apache/couchdb/pull/3347 I'm proposing to set a
> > > finite default for max_attachment_size .
> > > The PR is approved, but as per Ilya's request, I'd like to call for a
> > > lazy majority vote here.
> > > The vote will remain open for at least 72 hours from now.
> > >
> > > Please let me know if you have any questions, comments or concerns.
> > >
> > >
> > > Donat
> > 
> > >>>
> > >>
> >

Re: [DISCUSS] Replicator scheduler improvement

[Nick Vatamaniuc]

I created an RFC based on the discussion above:
https://github.com/apache/couchdb-documentation/pull/617

-Nick

On Sun, Jan 17, 2021 at 12:22 AM Nick Vatamaniuc  wrote:
>
> On Fri, Jan 15, 2021 at 6:41 PM Joan Touzet  wrote:
> >
> > On 15/01/2021 17:51, Nick Vatamaniuc wrote:> You're absolutely right.
> > There are two places where job priority comes
> > > into play: one is when jobs are started, and then the other is when
> > > they are stopped. When starting jobs, both continuous and one-shot
> > > (normal) jobs are at the same priority. However when jobs are stopped,
> > > we try to stop only the continuous jobs. The only time we'd stop
> > > one-shot jobs is when the operator manually lowers the max_jobs config
> > > limit. This means there is an imbalance between the starting and
> > > stopping algorithms and eventually a large number of one-shot
> > > replications could completely crowd out and replace all the continuous
> > > ones. Continuous ones would periodically stop, and if the new ones
> > > started are one-shot, those would slowly accumulate over time.
> >
> > I do think this is a concern, but I also think it can be addressed
> > separately from this proposal. I don't have a good answer re: fixing it.
> >
> > > The idea is to still guarantee fairness with hopefully an easy to
> > > understand algorithm. For the 10 different replicator dbs we still
> > > round-robin among them, and pick the jobs with the lowest start time
> > > from each in turn, then, do the same thing next cycle, until we have
> > > enough jobs picked out. So for example if we had this setup:
> > >
> > > db1: j4
> > > db2: j3, j5
> > > db3: j1, j2
> > >
> > > Where the index for the jobs is the start time (j1 was last stated
> > > before j2), we'd pick them as j1, j3, j4 during the first round-robin
> > > selection loop, then we'd be left with
> > >
> > > db1 :
> > > db2 : j5
> > > db3 : j2
> > >
> > > Next we'd pick j2 and j5 and so on. So overall we'd get
> > > j1,j3,j4,j2,j5. If we look at the dbs they came from it would be:
> > > db3,db2,db1,db3,db2. In case db1 had a 98 jobs and db2 just 2:
> > >
> > > db1: j1, j2, j3 - j41, j43 - j99
> > > db2: j42, j100
> > >
> > > The algorithm would pick: j1, j42, j2, j100, j3, j4, j5, j6 - j99. The
> > > dbs they'd come from would be db1,db2,db1,db2,db1,db1,db1,... to make
> > > sure all the dbs get a fair chance to run their jobs. If there is one
> > > db only, the algorithm doesn't change from the current one. However,
> > > one thing worrying me here is this implies either a lot of sorting or
> > > keeping a scheduler queue per db, so things could end up being quite
> > > expensive performance-wise.
> >
> > So this doesn't really let us do any sort of priority queue, which is
> > unfortunate. You can take your higher priority tasks and move them
> > outside of the db that has all of your background tasks in it, but if
> > you have > max_jobs/num_replicator_dbs in that separate db, some will
> > still get ignored. Fair share would provide a way to ensure starvation
> > doesn't occur.
> >
>
> I think my idea would just be a simpler version of fair share where
> the shares are always equal amongst the dbs, and without the need to
> explicitly handle the concept of a "share". Jobs would just start and
> stop based on a db round-robin schedule, then based on start times. So
> the few jobs in a separate db would get a higher priority compared to
> the priority of a db with lots of jobs in it. But as you said, that
> would not be an explicit priority mechanism so might be awkward to use
> it to boost priorities.
>
> > (Also consider the situation where num_replicator_dbs > max_jobs...)
> >
>
> Was this in regards to taking the higher priority tasks and moving
> them outside of the db? In the original idea we should handle the case
> where num_replicator_dbs > max_jobs: during each scheduling cycle, the
> dbs with jobs with lower start times at the front would get picked
> first. But it would mean that some of the dbs don't get scheduled
> during a scheduling cycle, they might have to wait for the next one or
> the one after, etc. But that's unavoidable anyway. The key is that it
> would still be fair across multiple scheduling cycles.
>
> > >> Would a fair-share scheduler [1] make sense here? We can't borrow many
> > >> of the OS-level scheduling ideas because we don't have the same insigh

Re: [DISCUSS] Removing erlang 19 support

[Nick Vatamaniuc]

+1 on dropping Erlang 19

It's been a best-effort support for it anyway, if I remember the last
discussion about it correctly.

I looked at the failure in the CI and it looks like we're failing on
the ceil/1 function in couch_emsort.erl. We already shipped 3.1.1 with
that change and the commit is 10 months old.  We haven't seen much
feedback about users not being to run on 19 so far, so don't see a big
issue with dropping it officially from CI and rebar.config.script

Fair point on following Cloudant's base version, Joan. I was involved
with the last transition from 17 to 20 and we spent a significant
amount of engineering effort discovering various incompatibilities and
even bugs in Erlang 20.

Cheers,
-Nick

On Fri, Jan 22, 2021 at 4:19 PM Joan Touzet  wrote:
>
> On 22/01/2021 15:48, Robert Newson wrote:
> > I’m +1 on dropping Erlang 19 support. Erlang is now on major release 23.
>
> No problem here.
>
> > I’d further advocate a general policy of supporting only the most recent 2 
> > or 3 major releases of Erlang/OTP.
> >
> > The main (I think only?) reason to keep compatibility so far back is 
> > because of the versions supported by some OS’es. I don’t feel that is a 
> > strong reason given the couchdb install, in common with Erlang-based 
> > projects, is self-contained. The existence of Erlang Solutions packages for 
> > all common platforms is also a factor.
>
> That hasn't been the case for at least 2 years, if not longer.
>
> As the release engineer, I've been focused on stability for everyone.
> This is largely driven by knowing that IBM/Cloudant largely run CouchDB
> on 20.x at scale. Standing on the shoulders of giants, our releases run
> the latest 20.x release at the time of binary generation.
>
> A few times recently issues cropped up in 21 and 22 that we didn't
> encounter in our user base because, at scale, we are deployed on
> 20.3.8.something. Some of these issues were non-trivial. (I'm off today,
> so I don't have the time to dig into the specifics until Monday.)
>
> So my $0.02 is that: if IBM/Cloudant is ready to move to something newer
> at scale, I'm ready to release binaries on a newer Erlang by default.
>
> The alternative (running newer Erlangs in the binary distributions than
> IBM/Cloudant run in production) could possibly be perceived as treating
> our open source customers as guinea pigs. I'd rather not risk that
> perception, but am willing to be convinced otherwise.
>
> -Joan
>
> >
> > B.
> >
> >> On 22 Jan 2021, at 19:54, Bessenyei Balázs Donát  wrote:
> >>
> >> Hi All,
> >>
> >> CI for https://github.com/apache/couchdb-config appears to be broken.
> >> I wanted to fix it in
> >> https://github.com/apache/couchdb-config/pull/34/files , but I'm
> >> getting issues with erlang 19. Are we okay with dropping 19 support
> >> there?
> >>
> >> On a different note: are we okay with dropping erlang 19 support
> >> overall in couch project(s)?
> >>
> >>
> >> Thank you,
> >> Donat
> >

Re: [DISCUSS] Replicator scheduler improvement

[Nick Vatamaniuc]

On Fri, Jan 15, 2021 at 6:41 PM Joan Touzet  wrote:
>
> On 15/01/2021 17:51, Nick Vatamaniuc wrote:> You're absolutely right.
> There are two places where job priority comes
> > into play: one is when jobs are started, and then the other is when
> > they are stopped. When starting jobs, both continuous and one-shot
> > (normal) jobs are at the same priority. However when jobs are stopped,
> > we try to stop only the continuous jobs. The only time we'd stop
> > one-shot jobs is when the operator manually lowers the max_jobs config
> > limit. This means there is an imbalance between the starting and
> > stopping algorithms and eventually a large number of one-shot
> > replications could completely crowd out and replace all the continuous
> > ones. Continuous ones would periodically stop, and if the new ones
> > started are one-shot, those would slowly accumulate over time.
>
> I do think this is a concern, but I also think it can be addressed
> separately from this proposal. I don't have a good answer re: fixing it.
>
> > The idea is to still guarantee fairness with hopefully an easy to
> > understand algorithm. For the 10 different replicator dbs we still
> > round-robin among them, and pick the jobs with the lowest start time
> > from each in turn, then, do the same thing next cycle, until we have
> > enough jobs picked out. So for example if we had this setup:
> >
> > db1: j4
> > db2: j3, j5
> > db3: j1, j2
> >
> > Where the index for the jobs is the start time (j1 was last stated
> > before j2), we'd pick them as j1, j3, j4 during the first round-robin
> > selection loop, then we'd be left with
> >
> > db1 :
> > db2 : j5
> > db3 : j2
> >
> > Next we'd pick j2 and j5 and so on. So overall we'd get
> > j1,j3,j4,j2,j5. If we look at the dbs they came from it would be:
> > db3,db2,db1,db3,db2. In case db1 had a 98 jobs and db2 just 2:
> >
> > db1: j1, j2, j3 - j41, j43 - j99
> > db2: j42, j100
> >
> > The algorithm would pick: j1, j42, j2, j100, j3, j4, j5, j6 - j99. The
> > dbs they'd come from would be db1,db2,db1,db2,db1,db1,db1,... to make
> > sure all the dbs get a fair chance to run their jobs. If there is one
> > db only, the algorithm doesn't change from the current one. However,
> > one thing worrying me here is this implies either a lot of sorting or
> > keeping a scheduler queue per db, so things could end up being quite
> > expensive performance-wise.
>
> So this doesn't really let us do any sort of priority queue, which is
> unfortunate. You can take your higher priority tasks and move them
> outside of the db that has all of your background tasks in it, but if
> you have > max_jobs/num_replicator_dbs in that separate db, some will
> still get ignored. Fair share would provide a way to ensure starvation
> doesn't occur.
>

I think my idea would just be a simpler version of fair share where
the shares are always equal amongst the dbs, and without the need to
explicitly handle the concept of a "share". Jobs would just start and
stop based on a db round-robin schedule, then based on start times. So
the few jobs in a separate db would get a higher priority compared to
the priority of a db with lots of jobs in it. But as you said, that
would not be an explicit priority mechanism so might be awkward to use
it to boost priorities.

> (Also consider the situation where num_replicator_dbs > max_jobs...)
>

Was this in regards to taking the higher priority tasks and moving
them outside of the db? In the original idea we should handle the case
where num_replicator_dbs > max_jobs: during each scheduling cycle, the
dbs with jobs with lower start times at the front would get picked
first. But it would mean that some of the dbs don't get scheduled
during a scheduling cycle, they might have to wait for the next one or
the one after, etc. But that's unavoidable anyway. The key is that it
would still be fair across multiple scheduling cycles.

> >> Would a fair-share scheduler [1] make sense here? We can't borrow many
> >> of the OS-level scheduling ideas because we don't have the same insight
> >> into the runtime characteristics of our jobs, but fair share might be a
> >> good match.
> >>
> >> Here's one way it could be implemented:
> >>
> >> * Each replicator DB could represent a logical grouping of jobs, e.g.
> >>background continuous backup jobs, one-shot migration jobs,
> >>high-priority user requests, etc.
> >>
> >> * An INI-level config could determine how these groups are handled:
> >>
> >>* 100% fair (default) -- total available job slots a

Re: [DISCUSS] Replicator scheduler improvement

[Nick Vatamaniuc]

On Thu, Jan 14, 2021 at 4:51 PM Joan Touzet  wrote:
>
> On 2021-01-12 12:55 p.m., Nick Vatamaniuc wrote:
> > Hello everyone
>
> Hi (Dr.) Nick! ;)

"One quick application of Spiffy and we got ourselves a better scheduler!"

I had to look up the Simpson reference as I had completely forgotten
about him :-)

> > I wanted to see what we thought about adding a scheduling improvement
> > to the replicator. Specifically adding round-robin fairness amongst
> > different _replicator dbs
>
> We've needed some sort of priority system for a while -- critically, in
> some cases, our failure to offer something has forced my hand into
> recommending use of external replication scheduling techniques. I'd
> never thought of using multiple databases as a way of handling this
> problem, though.
>
> > Currently, the scheduler runs all the jobs in the system fairly. It
> > does it by using the jobs' "last started" timestamp to select which
> > jobs to stop and start next. So each scheduling cycle, the jobs which
> > ran the longest (have the lowest start time) get stopped, then, from
> > the list of pending jobs we also pick those with the lowest start
> > times (since they waited the longest) to run next.
>
> There is also some difference between continuous and one-shot
> replication jobs, isn't there? I seem to recall one-shot jobs get
> priority over continuous ones.

You're absolutely right. There are two places where job priority comes
into play: one is when jobs are started, and then the other is when
they are stopped. When starting jobs, both continuous and one-shot
(normal) jobs are at the same priority. However when jobs are stopped,
we try to stop only the continuous jobs. The only time we'd stop
one-shot jobs is when the operator manually lowers the max_jobs config
limit. This means there is an imbalance between the starting and
stopping algorithms and eventually a large number of one-shot
replications could completely crowd out and replace all the continuous
ones. Continuous ones would periodically stop, and if the new ones
started are one-shot, those would slowly accumulate over time.

> > However, this algorithm can be unfair among replication jobs from
> > different _replicator dbs. For example, if max_jobs=10 and one
> > _replicator db has a 1000 jobs and another has only one doc then,
> > that one job might have to wait for quite a while to get its turn to
> >  run. If we picked fairly amongst _replicator dbs, then for example,
> >  the one job would get at least one of the 10 max_jobs run "slots",
> > and the rest of the 9 slots would go to the replicator db with 1000
> > docs. If there would be 11 _replicator dbs, then a docs from the ones
> > with the lowest start times amongst them would be picked first.
>
> That does sound unfair. I wasn't aware of this "gotcha" with multiple
> replicator DBs (in the field I rarely see >1 replicator DB) but it
> intuitively makes sense - all replicator DBs are unioned in the
> scheduler's queue and treated like a single, big DB. (There is basically
> no advantage to multiple _replicator DBs today then, other than perhaps
> security / obscurity reasons, is there?)

Exactly, they are unioned without any regard to how they ended up in
the scheduler. Today we could use multiple db to group replication
jobs for security or for management, like you suggested. You can
delete a whole _replicator db and stop a whole group of jobs at once.
Or you could replicate the _replicator dbs themselves around even!

> > This feature would also allow running some quick replication jobs,
> > when there is already a full queue main _replicator db jobs by
> > creating a "quickjobs/_replicator" db, and insert these new jobs
> > there. With this new scheme they would start running right away even
> > if the queue is full with jobs from the main _replicator db. Another,
> > related idea I had, was to add per job user-configurable priorities:
> > high/default/low or numeric. However, that scheme is not as good as
> > it could lead to permanent starvation of jobs while the round-robin
> > db scheduling scheme still guarantees all jobs will eventually run.
>
> I think this proposal can be improved. I like the idea of leveraging
> multiple replicator DBs to provide some sort of priority tweak, but I am
> not sure I can predict what will happen in the case of e.g. 10 different
> replicator DBs fighting with each other in the scheduler as you've laid
> it out.

The idea is to still guarantee fairness with hopefully an easy to
understand algorithm. For the 10 different replicator dbs we still
round-robin among them, and pick the jobs with the lowest start time
from each in turn, then, do the same thing ne

Re: [ANNOUNCE] Bessenyei Balázs Donát elected as CouchDB committer

[Nick Vatamaniuc]

Congrats Donat and welcome!

-Nick


On Thu, Jan 14, 2021 at 3:55 PM Joan Touzet  wrote:
>
> Congratulations Bessenyei! Do you have a nickname (other than bessbd)?
>
> -Joan
>
> On 2021-01-14 3:48 p.m., Robert Newson wrote:
> > Dear community,
> >
> > I am pleased to announce that the CouchDB Project Management Committee has 
> > elected Bessenyei Balázs Donát as a CouchDB committer.
> >
> > Apache ID: bessbd
> >
> > Committers are given a binding vote in certain project decisions, as well 
> > as write access to public project infrastructure.
> >
> > This election was made in recognition of Donát's commitment to the project. 
> > We mean this in the sense of being loyal to the project and its interests.
> >
> > Please join me in extending a warm welcome to Donát!
> >
> > On behalf of the CouchDB PMC,
> >
> > Robert Newson
> >

[DISCUSS] Replicator scheduler improvement

[Nick Vatamaniuc]

Hello everyone

I wanted to see what we thought about adding a scheduling improvement
to the replicator. Specifically adding round-robin fairness amongst
different _replicator dbs.

Currently, the scheduler runs all the jobs in the system fairly. It
does it by using the jobs' "last started" timestamp to select which
jobs to stop and start next. So each scheduling cycle, the jobs which
ran the longest (have the lowest start time) get stopped, then, from
the list of pending jobs we also pick those with the lowest start
times (since they waited the longest) to run next. However, this
algorithm can be unfair among replication jobs from different
_replicator dbs. For example, if max_jobs=10 and one _replicator db
has a 1000 jobs and another has only one doc then, that one job might
have to wait for quite a while to get its turn to run. If we picked
fairly amongst _replicator dbs, then for example, the one job would
get at least one of the 10 max_jobs run "slots", and the rest of the 9
slots would go to the replicator db with 1000 docs. If there would be
11 _replicator dbs, then a docs from the ones with the lowest start
times amongst them would be picked first.

This feature would also allow running some quick replication jobs,
when there is already a full queue main _replicator db jobs by
creating a "quickjobs/_replicator" db, and insert these new jobs
there. With this new scheme they would start running right away even
if the queue is full with jobs from the main _replicator db. Another,
related idea I had, was to add per job user-configurable priorities:
high/default/low or numeric. However, that scheme is not as good as it
could lead to permanent starvation of jobs while the round-robin db
scheduling scheme still guarantees all jobs will eventually run.

Would this feature be useful to have? I was thinking of giving it a
try on a branch. I suspect implementing this for 3.x might be a tad
simpler since scheduling is done in memory independently on each
separate node so was thinking of starting there first. For main
(future 4.x) we might require some coordination state to live in FDB
and would have to possibly patch up couch_jobs to know about
priorities.

Cheers,
-Nick

Re: [VOTE] couchdb 4.0 transaction semantics

[Nick Vatamaniuc]

> I withdraw my vote until I can get a clearer view. Nick would you mind
re-stating?

Not at all! The longer version and other considerations was stated in
my last reply to the discussion thread so I assumed that was accepted
as a consensus since nobody replied arguing otherwise.

https://lists.apache.org/thread.html/r45bff6ca4339f775df631f47e77657afbca83ee0ef03c6aa1a1d45cb%40%3Cdev.couchdb.apache.org%3E

But the gist of it is that existing (< 3.x) replicators won't be able
to replicate non-continuous (normal) changes from >= 4.x endpoints.

Regards,
-Nick

On Sat, Jan 9, 2021 at 1:26 AM Joan Touzet  wrote:
>
> Wait, what? I thought you agreed with this approach in that thread.
>
> I withdraw my vote until I can get a clearer view. Nick would you mind
> re-stating?
>
> -Joan
>
> On 2021-01-08 11:37 p.m., Nick V wrote:
> > +1 for 1 through 3
> >
> > -1 for 4  as I think the exception should apply to normal change feeds as 
> > well, as described in the thread
> >
> > Cheers,
> > -Nick
> >
> >> On Jan 8, 2021, at 17:12, Joan Touzet  wrote:
> >>
> >> Thanks, then it's a solid +1 from me.
> >>
> >> -Joan
> >>
> >>> On 2021-01-08 4:13 p.m., Robert Newson wrote:
> >>> You are probably thinking of a possible “group commit”. That is 
> >>> anticipated and not contradicted by this proposal. This proposal is 
> >>> explicitly about not using multiple states of the database for a single 
> >>> doc lookup, view query, etc.
> > On 8 Jan 2021, at 19:53, Joan Touzet  wrote:
> 
>  +1.
> 
>  This is for now I presume, as I thought that there was feeling about
>  relaxing this restriction somewhat for the 5.0 timeframe? Memory's dim.
> 
>  -Joan
> 
>  On 07/01/2021 06:00, Robert Newson wrote:
> > Hi,
> >
> > Following on from the discussion at 
> > https://lists.apache.org/thread.html/rac6c90c4ae03dc055c7e8be6eca1c1e173cf2f98d2afe6d018e62d29%40%3Cdev.couchdb.apache.org%3E
> >  
> > 
> >
> > The proposal is;
> >
> > "With the exception of the changes endpoint when in feed=continuous 
> > mode, that all data-bearing responses from CouchDB are constructed from 
> > a single, immutable snapshot of the database at the time of the 
> > request.”
> >
> > Paul Davis summarised the discussion in four bullet points, reiterated 
> > here for context;
> >
> > 1. A single CouchDB API call should map to a single FDB transaction
> > 2. We absolutely do not want to return a valid JSON response to any
> > streaming API that hit a transaction boundary (because data
> > loss/corruption)
> > 3. We're willing to change the API requirements so that 2 is not an 
> > issue.
> > 4. None of this applies to continuous changes since that API call was
> > never a single snapshot.
> >
> >
> > Please vote accordingly, we’ll run this as lazy consensus per the 
> > bylaws (https://couchdb.apache.org/bylaws.html#lazy 
> > )
> >
> > B.
> >
> >

Re: [DISCUSS] Deprecate custom reduce functions

[Nick Vatamaniuc]

> I gather that we perceive (probably rightfully so) that most custom
reduce functions are poorly written, and hurt performance. Is this the
primary/only reason to disable it? Just to give users one more hoop to
jump through before shooting themselves in the foot?

One more hoop to jump through before shooting themselves in the foot
is a good way of putting it, and explains my reasoning. Sorry, I
should have expanded a bit why I voted the way I did.

Cheers,
-Nick


On Thu, Oct 15, 2020 at 11:14 AM Jonathan Hall  wrote:
>
> I'd like to read some clarifying comments about _why_ we want to disable
> it by default. Maybe they've already been made, and I missed them.
>
> I gather that we perceive (probably rightfully so) that most custom
> reduce functions are poorly written, and hurt performance. Is this the
> primary/only reason to disable it? Just to give users one more hoop to
> jump through before shooting themselves in the foot?
>
> The original message on 10/13 from Robert Newson began by explaining
> some technical difficulties surrounding custom reduce functions, but
> then explained that those have been resolved.
>
> Maybe this muddied the issue in my mind--are there still any _technical_
> reasons to disable custom reduce by default, or is it purely the
> (mis)usability issue?
>
> While it may change with more information, at the moment, my feelings on
> the two issues below are:
>
> -0.5 for deprecate custom reduce functions
> -0.0 for disable custom reduce functions by default
>
> Jonathan
>
>
> On 10/13/20 10:21 PM, Robert Samuel Newson wrote:
> > Nick, let's broaden the thread to two questions then;
> >
> > 1) Deprecate custom reduce functions
> > 2) Disable custom reduce functions by default, but don't deprecate them.
> >
> >
> >
> >> On 13 Oct 2020, at 21:16, Nick Vatamaniuc  wrote:
> >>
> >> In case of _sum, like Joan mentioned, we can emit objects or arrays
> >> and the built-in _sum will add the values of the fields together:
> >>
> >> So  {"map": 'function(d){ emit(d._id, {"bar":1, "foo":2, "baz":3});
> >> }',  "reduce" : '_sum' } for 10 docs would produce {"bar": 10, "baz":
> >> 30, "foo": 20}.
> >>
> >> As for the deprecation, I wouldn't necessarily call for deprecation
> >> but I can see leaving it disabled by default and let the users enable
> >> it if they want to. If we see that there is a good demand for custom
> >> functions, and it is annoying for users to have to enable it, we could
> >> revert it back to enabled by default or like it was discussed, or, try
> >> to add more built-in reducers.
> >>
> >> Cheers,
> >> -Nick
> >>
> >> On Tue, Oct 13, 2020 at 3:38 PM Jonathan Hall  wrote:
> >>> So looking through the code that uses this, it looks like the main use
> >>> I've had for custom reduce functions is summing multiple values at
> >>> once.  A rough equivalent of 'SELECT SUM(foo),SUM(bar),SUM(baz)'.
> >>>
> >>> The first thing that comes to mind to duplicate this functionality
> >>> without a custom reduce function would mean building one unique index
> >>> for each value that needs to be summed, which I expect would be a lot
> >>> less efficient.
> >>>
> >>> But maybe I'm overlooking a more clever and efficient alternative.
> >>>
> >>> Jonathan
> >>>
> >>>
> >>> On 10/13/20 6:31 PM, Robert Samuel Newson wrote:
> >>>> Hi,
> >>>>
> >>>> Yes, that's what I'm referring to, the javascript reduce function.
> >>>>
> >>>> I'm curious what you do with custom reduce that isn't covered by the 
> >>>> built-in reduces?
> >>>>
> >>>> I also think if custom reduce was disabled by default that we would be 
> >>>> motivated to expand this set of built-in reduce functions.
> >>>>
> >>>> B.
> >>>>
> >>>>> On 13 Oct 2020, at 17:06, Jonathan Hall  wrote:
> >>>>>
> >>>>> To be clear, by "custom reduce functions" you mean this 
> >>>>> (https://docs.couchdb.org/en/stable/ddocs/ddocs.html#reduce-and-rereduce-functions)?
> >>>>>
> >>>>> So by default, only built-in reduce functions could be used 
> >>>>> (https://docs.couchdb.org/en/stable/ddocs/ddocs.html#built-in-reduce-functions)?
> >>>>>
> &g

Re: [DISCUSS] Deprecate custom reduce functions

[Nick Vatamaniuc]

Makes sense. I am +1 to #2 (disable by default), and +0 neutral on #1
(deprecate)

On Tue, Oct 13, 2020 at 4:21 PM Robert Samuel Newson  wrote:
>
>
> Nick, let's broaden the thread to two questions then;
>
> 1) Deprecate custom reduce functions
> 2) Disable custom reduce functions by default, but don't deprecate them.
>
>
>
> > On 13 Oct 2020, at 21:16, Nick Vatamaniuc  wrote:
> >
> > In case of _sum, like Joan mentioned, we can emit objects or arrays
> > and the built-in _sum will add the values of the fields together:
> >
> > So  {"map": 'function(d){ emit(d._id, {"bar":1, "foo":2, "baz":3});
> > }',  "reduce" : '_sum' } for 10 docs would produce {"bar": 10, "baz":
> > 30, "foo": 20}.
> >
> > As for the deprecation, I wouldn't necessarily call for deprecation
> > but I can see leaving it disabled by default and let the users enable
> > it if they want to. If we see that there is a good demand for custom
> > functions, and it is annoying for users to have to enable it, we could
> > revert it back to enabled by default or like it was discussed, or, try
> > to add more built-in reducers.
> >
> > Cheers,
> > -Nick
> >
> > On Tue, Oct 13, 2020 at 3:38 PM Jonathan Hall  wrote:
> >>
> >> So looking through the code that uses this, it looks like the main use
> >> I've had for custom reduce functions is summing multiple values at
> >> once.  A rough equivalent of 'SELECT SUM(foo),SUM(bar),SUM(baz)'.
> >>
> >> The first thing that comes to mind to duplicate this functionality
> >> without a custom reduce function would mean building one unique index
> >> for each value that needs to be summed, which I expect would be a lot
> >> less efficient.
> >>
> >> But maybe I'm overlooking a more clever and efficient alternative.
> >>
> >> Jonathan
> >>
> >>
> >> On 10/13/20 6:31 PM, Robert Samuel Newson wrote:
> >>> Hi,
> >>>
> >>> Yes, that's what I'm referring to, the javascript reduce function.
> >>>
> >>> I'm curious what you do with custom reduce that isn't covered by the 
> >>> built-in reduces?
> >>>
> >>> I also think if custom reduce was disabled by default that we would be 
> >>> motivated to expand this set of built-in reduce functions.
> >>>
> >>> B.
> >>>
> >>>> On 13 Oct 2020, at 17:06, Jonathan Hall  wrote:
> >>>>
> >>>> To be clear, by "custom reduce functions" you mean this 
> >>>> (https://docs.couchdb.org/en/stable/ddocs/ddocs.html#reduce-and-rereduce-functions)?
> >>>>
> >>>> So by default, only built-in reduce functions could be used 
> >>>> (https://docs.couchdb.org/en/stable/ddocs/ddocs.html#built-in-reduce-functions)?
> >>>>
> >>>> If my understanding is correct, I guess I find it a but surprising. I've 
> >>>> always thought of map/reduce of one of the core features of CouchDB, so 
> >>>> to see half of that turned off (even if it can be re-enabled) makes me 
> >>>> squint a bit. And it is a feature I use, so I would not be in favor of 
> >>>> deprecating it entirely, without a clear proposal/documentation for an 
> >>>> alternative/work-around.
> >>>>
> >>>> Based on the explanation below, it doesn't sound like there's a 
> >>>> technical reason to deprecate it, but rather a user-experience reason. 
> >>>> Is this correct?
> >>>>
> >>>> If my understanding is correct, I'm not excited about the proposal, but 
> >>>> before I dive further into my thoughts, I'd like confirmation that I 
> >>>> actually understand the proposal, and am not worried about something 
> >>>> else ;)
> >>>>
> >>>> Jonathan
> >>>>
> >>>>
> >>>> On 10/13/20 5:48 PM, Robert Samuel Newson wrote:
> >>>>> Hi All,
> >>>>>
> >>>>> As part of CouchDB 4.0, which moves the storage tier of CouchDB into 
> >>>>> FoundationDB, we have struggled to reproduce the full map/reduce 
> >>>>> functionality. Happily this has now happened, and that work is now 
> >>>>> merged to the couchdb main branch.
> >>>>>
> >>>>> This functionality includes the use of custom (javascript) reduce 
> >>>>> functions. It is my experience that these are very often problematic, 
> >>>>> in that much more often than not the functions do not significantly 
> >>>>> reduce the input parameters into a smaller result (indeed, sometimes 
> >>>>> the output is the same or larger than the input).
> >>>>>
> >>>>> To that end, I'm asking if we should deprecate the feature entirely.
> >>>>>
> >>>>> In scope for this thread is the middle ground proposal that Paul Davis 
> >>>>> has written up here;
> >>>>>
> >>>>> https://github.com/apache/couchdb/pull/3214
> >>>>>
> >>>>> Where custom reduces are not allowed by default but can be enabled.
> >>>>>
> >>>>> The core _ability_ to do custom reduces will always been maintained, 
> >>>>> this is intrinsic to the design of ebtree, the structure we use on top 
> >>>>> of FoundationDB to hold and maintain intermediate reduce values.
> >>>>>
> >>>>> My view is that we should merge #3214 and disable custom reduces by 
> >>>>> default.
> >>>>>
> >>>>> B.
> >>>>>
>

Re: [DISCUSS] Deprecate custom reduce functions

[Nick Vatamaniuc]

In case of _sum, like Joan mentioned, we can emit objects or arrays
and the built-in _sum will add the values of the fields together:

So  {"map": 'function(d){ emit(d._id, {"bar":1, "foo":2, "baz":3});
}',  "reduce" : '_sum' } for 10 docs would produce {"bar": 10, "baz":
30, "foo": 20}.

As for the deprecation, I wouldn't necessarily call for deprecation
but I can see leaving it disabled by default and let the users enable
it if they want to. If we see that there is a good demand for custom
functions, and it is annoying for users to have to enable it, we could
revert it back to enabled by default or like it was discussed, or, try
to add more built-in reducers.

Cheers,
-Nick

On Tue, Oct 13, 2020 at 3:38 PM Jonathan Hall  wrote:
>
> So looking through the code that uses this, it looks like the main use
> I've had for custom reduce functions is summing multiple values at
> once.  A rough equivalent of 'SELECT SUM(foo),SUM(bar),SUM(baz)'.
>
> The first thing that comes to mind to duplicate this functionality
> without a custom reduce function would mean building one unique index
> for each value that needs to be summed, which I expect would be a lot
> less efficient.
>
> But maybe I'm overlooking a more clever and efficient alternative.
>
> Jonathan
>
>
> On 10/13/20 6:31 PM, Robert Samuel Newson wrote:
> > Hi,
> >
> > Yes, that's what I'm referring to, the javascript reduce function.
> >
> > I'm curious what you do with custom reduce that isn't covered by the 
> > built-in reduces?
> >
> > I also think if custom reduce was disabled by default that we would be 
> > motivated to expand this set of built-in reduce functions.
> >
> > B.
> >
> >> On 13 Oct 2020, at 17:06, Jonathan Hall  wrote:
> >>
> >> To be clear, by "custom reduce functions" you mean this 
> >> (https://docs.couchdb.org/en/stable/ddocs/ddocs.html#reduce-and-rereduce-functions)?
> >>
> >> So by default, only built-in reduce functions could be used 
> >> (https://docs.couchdb.org/en/stable/ddocs/ddocs.html#built-in-reduce-functions)?
> >>
> >> If my understanding is correct, I guess I find it a but surprising. I've 
> >> always thought of map/reduce of one of the core features of CouchDB, so to 
> >> see half of that turned off (even if it can be re-enabled) makes me squint 
> >> a bit. And it is a feature I use, so I would not be in favor of 
> >> deprecating it entirely, without a clear proposal/documentation for an 
> >> alternative/work-around.
> >>
> >> Based on the explanation below, it doesn't sound like there's a technical 
> >> reason to deprecate it, but rather a user-experience reason. Is this 
> >> correct?
> >>
> >> If my understanding is correct, I'm not excited about the proposal, but 
> >> before I dive further into my thoughts, I'd like confirmation that I 
> >> actually understand the proposal, and am not worried about something else 
> >> ;)
> >>
> >> Jonathan
> >>
> >>
> >> On 10/13/20 5:48 PM, Robert Samuel Newson wrote:
> >>> Hi All,
> >>>
> >>> As part of CouchDB 4.0, which moves the storage tier of CouchDB into 
> >>> FoundationDB, we have struggled to reproduce the full map/reduce 
> >>> functionality. Happily this has now happened, and that work is now merged 
> >>> to the couchdb main branch.
> >>>
> >>> This functionality includes the use of custom (javascript) reduce 
> >>> functions. It is my experience that these are very often problematic, in 
> >>> that much more often than not the functions do not significantly reduce 
> >>> the input parameters into a smaller result (indeed, sometimes the output 
> >>> is the same or larger than the input).
> >>>
> >>> To that end, I'm asking if we should deprecate the feature entirely.
> >>>
> >>> In scope for this thread is the middle ground proposal that Paul Davis 
> >>> has written up here;
> >>>
> >>> https://github.com/apache/couchdb/pull/3214
> >>>
> >>> Where custom reduces are not allowed by default but can be enabled.
> >>>
> >>> The core _ability_ to do custom reduces will always been maintained, this 
> >>> is intrinsic to the design of ebtree, the structure we use on top of 
> >>> FoundationDB to hold and maintain intermediate reduce values.
> >>>
> >>> My view is that we should merge #3214 and disable custom reduces by 
> >>> default.
> >>>
> >>> B.
> >>>

Re: [DISCUSS] Rename default branch to `main`

[Nick Vatamaniuc]

That looks great. Good call to archive and remove some of those repositories.

On Wed, Sep 16, 2020 at 11:52 AM Paul Davis  wrote:
>
> I'll create the branches on all of the appropriate repositories today
> and start looking at Jenkins requirements. I'll hold off on filing the
> infra ticket until tomorrow so that folks have time to double check my
> sanity.
>
> On Wed, Sep 16, 2020 at 10:50 AM Paul Davis  
> wrote:
> >
> > Also, I added an easier to digest list of repositories with their
> > planned actions to the gist [1]. I'm repasting here for historical
> > tracing.
> >
> > Create `main` branch, update default branch and branch protections
> > ===
> >
> > * couchdb
> > * couchdb-admin
> > * couchdb-b64url
> > * couchdb-bear
> > * couchdb-ci
> > * couchdb-config
> > * couchdb-docker
> > * couchdb-documentation
> > * couchdb-erlfdb
> > * couchdb-escodegen
> > * couchdb-esprima
> > * couchdb-ets-lru
> > * couchdb-fauxton
> > * couchdb-folsom
> > * couchdb-glazier
> > * couchdb-helm
> > * couchdb-hqueue
> > * couchdb-hyper
> > * couchdb-ibrowse
> > * couchdb-ioq
> > * couchdb-jaeger-passage
> > * couchdb-jiffy
> > * couchdb-khash
> > * couchdb-local
> > * couchdb-meck
> > * couchdb-mochiweb
> > * couchdb-nano
> > * couchdb-passage
> > * couchdb-pkg
> > * couchdb-rebar
> > * couchdb-recon
> > * couchdb-snappy
> > * couchdb-thrift-protocol
> >
> > Have infra set `asf-site` as default branch
> > ===
> >
> > * couchdb-www
> >
> > Empty repositories to delete
> > ===
> >
> > * couchdb-fauxton-server
> > * couchdb-javascript-tests
> > * couchdb-query-server-spidermonkey
> >
> > Repositories to archive
> > ===
> >
> > * couchdb-cassim
> > * couchdb-chttpd
> > * couchdb-couch
> > * couchdb-couch-collate
> > * couchdb-couch-dbupdates
> > * couchdb-couch-epi
> > * couchdb-couch-event
> > * couchdb-couch-httpd
> > * couchdb-couch-index
> > * couchdb-couch-log
> > * couchdb-couch-log-lager
> > * couchdb-couch-mrview
> > * couchdb-couch-plugins
> > * couchdb-couch-replicator
> > * couchdb-couch-stats
> > * couchdb-ddoc-cache
> > * couchdb-erlang-bcrypt
> > * couchdb-erlang-tests
> > * couchdb-examples
> > * couchdb-fabric
> > * couchdb-futon
> > * couchdb-global-changes
> > * couchdb-goldrush
> > * couchdb-jquery-couch
> > * couchdb-lager
> > * couchdb-mango
> > * couchdb-mem3
> > * couchdb-nmo
> > * couchdb-oauth
> > * couchdb-peruser
> > * couchdb-query-server-node
> > * couchdb-rexi
> > * couchdb-setup
> > * couchdb-smoosh
> > * couchdb-triq
> > * couchdb-twig
> >
> >
> > [1] https://gist.github.com/davisp/9de8fa167812f80356d4990e390c9351
> >
> > On Wed, Sep 16, 2020 at 10:49 AM Paul Davis  
> > wrote:
> > >
> > > Right. I figure that's basically an ASF version of the `gh-pages` branch?
> > >
> > > On Wed, Sep 16, 2020 at 10:39 AM Joan Touzet  wrote:
> > > >
> > > >
> > > > On 16/09/2020 11:39, Paul Davis wrote:
> > > > > On Wed, Sep 16, 2020 at 10:32 AM Joan Touzet  
> > > > > wrote:
> > > > >>
> > > > >>
> > > > >>
> > > > >> On 16/09/2020 10:57, Paul Davis wrote:
> > > > >>> Hey all,
> > > > >>>
> > > > >>> Here's a list of all CouchDB related repositories with a few quick
> > > > >>> stats and my read on their status and requirements. Can I get some
> > > > >>> eyeballs on this to double check before I submit a ticket to infra
> > > > >>> for doing our branch renaming updates?
> > > > >>>
> > > > >>> https://gist.github.com/davisp/9de8fa167812f80356d4990e390c9351
> > > > >>>
> > > > >>> There are a few repos with comments I had when I wasn't 100% sure on
> > > > >>> the status. For ease those are:
> > > > >>>
> > > > >>> couchdb-couch-collate - I couldn't easily tell if this was still
> > > > >>> used for Windows builds
> > > > >>
> > > > >> Nope
> > > > >>
> > > > >>> couchdb-fauxton-server - This is an empty repo, should we have it
> > > > >>> deleted?
> > > > >>
> > > > >> Sure
> > > > >>
> > > > >>> couchdb-jquery-couch - Should this be archived? Has PouchDB/nano
> > > > >>> replaced it?
> > > > >>
> > > > >> If I recall correctly this was part of Futon and 1.x releases?
> > > > >>
> > > > >>> couchdb-nmo - Should this be archived?
> > > > >>
> > > > >> Very old code from 2015 from Robert Kowalski to help set up
> > > > >> clusters/etc. I don't know anything about it, and it appears
> > > > >> unmaintained. +1 to archive
> > > > >>
> > > > >>> couchdb-oauth - I couldn't find this used anywhere, should we 
> > > > >>> archive
> > > > >>
> > > > >> I remember using this extensively! 1.x asset. As we no longer 
> > > > >> officially
> > > > >> support it (or CouchDB "plugins" in this form), +1 to archive
> > > > >>
> > > > >>> couchdb-www - Should this be archived or included in the rename?
> > > > >>
> > > > >> We already 

Re: [VOTE] Release Apache CouchDB 3.1.1 (RC2)

[Nick Vatamaniuc]

Environment:
  Ubuntu 18.04.5, x86_64
  $ asdf current
elixir 1.9.4-otp-22
erlang 22.2.3
 $ cat /etc/apt/sources.list.d/couchdb-bintray.list
deb https://apache.bintray.com/couchdb-deb bionic main

Sig: ok
Checksums: ok
Configure, make & make check: ok
Build release, add admin & start: ok
Used Fauxton to:
  - configure cluster: ok
  - verify install: ok
  - create dbs: ok
  - create docs: ok
  - create replications between dbs on same cluster: ok
  - links to docs: ok

+1

Thank you for creating the release, Joan!

-Nick

On Mon, Sep 14, 2020 at 10:15 PM Joan Touzet  wrote:
>
> Hi everyone,
>
> There have been no votes on this release. Are people available to try it
> out? Tomorrow, I will be able to complete the binary builds and submit
> my own vote.
>
> Remember, 3.1.1 will not be cut unless 3 +1 votes, minimum, arrive from
> committers, with no blockers by convention. The Mac situation needs to
> be investigated. It's not clear to me that there is consensus on
> blocking the release on the new request header toggle, but if there is,
> I could cut a 3.1.1-RC3 _without_ that change tomorrow if consensus
> materialises overnight.
>
> Note that I will be on holiday starting end of the week for 2.5 weeks.
> If 3.1.1 doesn't cut from this RC, for whatever reason, the next
> opportunity I will have to turn the crank will be 5 October 2020.
>
> -Joan
>
> On 2020-09-11 6:53 p.m., Joan Touzet wrote:
> > Dear community,
> >
> > I would like to propose that we release Apache CouchDB 3.1.1.
> >
> > Changes since the last round:
> >
> >  https://github.com/apache/couchdb/compare/3.1.1-RC1...3.1.1-RC2
> >
> > Candidate release notes:
> >
> >  https://docs.couchdb.org/en/latest/whatsnew/3.1.html
> >
> > We encourage the whole community to download and test these release
> > artefacts so that any critical issues can be resolved before the release
> > is made. Everyone is free to vote on this release, so dig right in!
> > (Only PMC members have binding votes, but they depend on community
> > feedback to gauge if an official release is ready to be made.)
> >
> > The release artefacts we are voting on are available here:
> >
> >  https://dist.apache.org/repos/dist/dev/couchdb/source/3.1.1/rc.2/
> >
> > There, you will find a tarball, a GPG signature, and SHA256/SHA512
> > checksums.
> >
> > Please follow the test procedure here:
> >
> >
> > https://cwiki.apache.org/confluence/display/COUCHDB/Testing+a+Source+Release
> >
> >
> > Please remember that "RC2" is an annotation. If the vote passes, these
> > artefacts will be released as Apache CouchDB 3.1.1.
> >
> > Because of the weekend, this vote will remain open until 5PM ET (UTC-4),
> > Tuesday, 15 September 2020.
> >
> > Please cast your votes now.
> >
> > Thanks,
> > Joan "once more unto the breech, dear friends" Touzet

Re: [ANNOUNCE] Glynn Bird joins the PMC

[Nick Vatamaniuc]

Congrats, Glynn!

-Nick

On 9/14/20 10:07 PM, Joan Touzet wrote:

Congratulations Glynn - and welcome!

-Joan

On 2020-09-14 12:22 p.m., Michelle P wrote:


Dear community,

I am delighted to announce that Glynn Bird joins the Apache CouchDB 
Project Management Committee today.


Glynn has made outstanding, sustained contributions to the project. 
This appointment is an official acknowledgement of their position 
within the community, and our trust in their ability to provide 
oversight for the project.


Everybody, please join me in congratulating Glynn!

On behalf of the CouchDB PMC,
Michelle

Re: [DISCUSS] Rename default branch to `main`

[Nick Vatamaniuc]

+1

This conversation kind of spilled into the other thread about merging,
so I replied there as well about main vs master and rebasing. Sorry
about the confusion.

On Wed, Sep 9, 2020 at 2:05 PM Robert Samuel Newson  wrote:
>
> I'm +1 in favour of renaming to 'main'.
>
>
>
> > On 9 Sep 2020, at 18:26, Alessio 'Blaster' Biancalana 
> >  wrote:
> >
> > I'm not against nor in favor :-D
> > Words matter but in my opinion git's master was never _that_ master.
> > Anyway, if it bothers someone... let's do this!
> >
> > Concerning open PRs, I don't know, I think original authors can easily
> > rebase. Also, the next release will cut some stuff open I think, so maybe
> > we'll find a feasible time slot to do so.
> >
> > Alessio
> >
> > On Wed, Sep 9, 2020 at 5:40 PM Paul Davis 
> > wrote:
> >
> >> Howdy Folks!
> >>
> >> Words matter. I've just started a thread on merging all of the
> >> FoundationDB work into mainline development and thought this would be
> >> a good time to bring up a separate discussion on renaming our default
> >> branch.
> >>
> >> Personally, I've got a few projects where I used `main` for the
> >> mainline development branch. I find it to be a fairly natural shift
> >> because I tab-complete everything on the command line. I'd be open to
> >> other suggestions but I'm also hoping this doesn't devolve into a
> >> bikeshed on what we end up picking.
> >>
> >> For mechanics, what I'm thinking is that when we finish up the last
> >> rebase of the FoundationDB work that instead of actually pushing the
> >> merge/rebase button we just rename the branch and then change the
> >> default branch on GitHub and close the PR.
> >>
> >> Thoughts?
> >>
> >> Paul
> >>
>

Re: Announcing Opservatory: 24/7 CouchDB Observation and Analysis

[Nick Vatamaniuc]

That looks really nice, Jan. Thanks for sharing!

On Wed, Sep 9, 2020 at 1:29 PM Alessio 'Blaster' Biancalana
 wrote:
>
> Wow, that's _amazing_. I'm glad seeing this kind of effort in an ecosystem
> like the one of CouchDB, cool stuff!
>
> Alessio
>
> On Wed, Sep 9, 2020 at 6:03 PM Jan Lehnardt  wrote:
>
> > Dear CouchDB Community,
> >
> > I’m happy to announce Opservatory https://opservatory.app, my company
> > Neighbourhoodie’s latest product for monitoring your CouchDB instances.
> >
> > Opservatory knows CouchDB better than any single human could. We’ve put
> > our combined multi-decade experience with supporting CouchDB
> > installations of all shapes into one tool that makes sure you are not
> > running into any trouble, ever.
> >
> > It continuously monitors your CouchDB installations and runs a number
> > of checks to ensure your CouchDBs are running correctly, fast and
> > securely. It also checks for conditions and configurations that have
> > the potential to cause production issues in the future.
> >
> > It compares metrics against settings to make real-time recommendations
> > for improving your CouchDB instance or cluster. It evaluates
> > configuration, metrics, and database information to ensure smooth
> > operation. It even evaluates your JavaScript code in design documents
> > for common mistakes and performance pitfalls.
> >
> > Here are just a few examples of the checks that Opservatory runs
> > periodically:
> >
> >   - If you are just starting out with CouchDB, Opservatory checks if
> > you’ve got any quirks in your setup and configuration and gives you
> > recommendations for well-established best practices for running CouchDB
> > correctly, fast and securely.
> >
> >   - If your production cluster ran into trouble and you don’t know
> > where to start looking for things to fix, Opservatory will give you the
> > top list of issues that you can resolve to make your problem(s) go away.
> >
> >   - If your intern uploaded a design doc with an inefficient map
> > function (again), Opservatory will let you know, but it also catches
> > more common patterns used by seasoned CouchDB developers that can be
> > improved.
> >
> >   - If you create a new database and forget to set the correct
> > _security object, Opservatory will warn you about publicly accessible
> > data that you might want to make private as soon as you get the
> > notification.
> >
> > Opservatory has over 80 checks that cover the five main areas of every
> > CouchDB installation: setup, configuration, metrics, databases and
> > queries/design docs. We have at least as many more in the backlog, and
> > we are constantly adding more as we find more issues during our
> > professional support service work.
> >
> > You can configure Opservatory to receive alerts via email or Slack, and
> > it covers all CouchDB versions: 1.x, 2.x and 3.x. It even works with
> > Cloudant!
> >
> > And while we are very happy with where we are already, we have big
> > plans to make maintaining CouchDB in production even easier with
> > features round capacity planning and seamless growth as well as helping
> > to automate common maintenance tasks safely.
> >
> > If you have any questions, I’m happy to answer off-list.
> >
> > If you think this is interesting, we welcome you to sign up for a free
> > trial at https://opservatory.app.
> >
> > Best
> > Jan
> > —
> > Professional Support for Apache CouchDB:
> > https://neighbourhood.ie/couchdb-support/
> >
> > 24/7 Observation for your CouchDB Instances:
> > https://opservatory.app
> >
> >
> >

Re: Is it time to merge prototype/fdb-layer to master?

[Nick Vatamaniuc]

Could we rename prototype/fdb-layer to main and it will be the base of
4.x? There might be a few things we could bring in from master into
main like elixir test improvements, tooling, but it would be a smaller
diff than bringing everything from prototype/fdb-layer into master.

Then master can be renamed to old-master or something like that and
we'd see if there are any commits that were intended to go into 3.x
but never made it. Once those are moved over, we can delete it too or
leave it around for a few years, with a note saying "use main or 3.x"
branches.

Nick

On Wed, Sep 9, 2020 at 12:28 PM Paul Davis  wrote:
>
> The merge on this turned out to be a lot more straightforward so I
> think its probably the way to go. I've got a failing test in
> couch_views_active_tasks_test but it appears to be flaky rather than a
> merge error. I'll work though getting `make check` to complete and
> then send another update.
>
> https://github.com/apache/couchdb/tree/prototype/fdb-layer-final-merge
> https://github.com/apache/couchdb/commit/873ccb4882f2e984c25f59ad0fd0a0677b9d4477
>
> On Wed, Sep 9, 2020 at 10:29 AM Paul Davis  
> wrote:
> >
> > Howdy folks!
> >
> > I've just gone through a rebase of `prototype/fdb-layer` against
> > master. Its not quite finished because the ebtree import went wrong
> > during rebase due to a weirdness of the history.
> >
> > I have a PR up for the rebase into master for people to look at [1].
> > Although the more important comparison is likely with the current
> > `prototype/fdb-layer` that can be found at [2].
> >
> > Given the ebtree aspect, as well as the fact that I get labeled as the
> > committer for all commits when doing a rebase I'm also wondering if we
> > shouldn't turn this into a merge in this instance. I'll work up a
> > second branch that shows that diff as well that we could then rebase
> > onto master.
> >
> > Regardless, I'd appreciate if we could get some eyeballs on the diff
> > and then finally merge this work to the default branch so its the main
> > line development going forward.
> >
> > Paul
> >
> > [1] https://github.com/apache/couchdb/pull/3137
> > [2] 
> > https://github.com/apache/couchdb/compare/prototype/fdb-layer...prototype/fdb-layer-final-rebase

Re: [DISCUSS] Creating new deleted documents in CouchDB 4

[Nick Vatamaniuc]

I think we came to a consensus to maintain compatibility. Thank you
all who participated in the discussion

Cheers,
-Nick

On Tue, Sep 1, 2020 at 4:40 PM Joan Touzet  wrote:
>
> Same - keep for now, choose to deprecate later.
>
> Remember by our semver policy this would mean the earliest this endpoint
> could be removed would be CouchDB 5.0 (!)
>
> -Joan "time keeps on slippin', slippin'..." Touzet
>
> On 2020-09-01 4:35 p.m., Jonathan Hall wrote:
> > Thanks for the explanation.
> >
> > I concur, I prefer compatibility, but as I'm not coding it, I'll defer the 
> > decision to others.
> >
> > Jonathan
> >
> >
> > On Sep 1, 2020, 10:30 PM, at 10:30 PM, Paul Davis 
> >  wrote:
> >> Replication of deletions isn't affected due to the new_edits=false
> >> flag like you guessed. This is purely "interactively creating a new
> >> document that is deleted". Its a fairly minor edge case in that the
> >> document must not exist. Any other attempt to "revive" a deleted doc
> >> into a deleted state will fail with a conflict on 3.x.
> >>
> >> I'm +0 for compatibility. Its not a significant amount of work to
> >> implement the behavior and we can always deprecate it in the future to
> >> remove the weird edge case of "document that has never existed can be
> >> created in a deleted state".
> >>
> >> On Tue, Sep 1, 2020 at 3:26 PM Jonathan Hall  wrote:
> >>>
> >>> Isn't compatibility required to support replication of deleted
> >> documents? Or does creation of a deleted document work with
> >> new_edits=false?
> >>>
> >>>
> >>>
> >>> On Sep 1, 2020, 10:16 PM, at 10:16 PM, Nick Vatamaniuc
> >>  wrote:
> >>>> Hi everyone,
> >>>>
> >>>> While running PouchDB replication unit tests against the CouchDB 4
> >>>> replicator PR branch (thanks to Garren Smith, who helped set up the
> >>>> tests), we had noticed a doc update API incompatibility between
> >>>> CouchDB 3.x/PouchDB and the prototype/fdb-layer branch: CouchDB
> >>>> 3.x/PouchDB allow creating new deleted documents and
> >>>> prototype/fdb-layer branch doesn't.
> >>>>
> >>>> For example:
> >>>>
> >>>> $ http put $DB1/mydb/doc1 _deleted:='true' a=b
> >>>> HTTP/1.1 200 OK
> >>>>
> >>>> {
> >>>> "id": "doc1",
> >>>> "ok": true,
> >>>> "rev": "1-ad7eb689fcae75e7a7edb57dc1f30939"
> >>>> }
> >>>>
> >>>> $ http $DB1/mydb/doc1?deleted=true
> >>>> HTTP/1.1 200 OK
> >>>>
> >>>> {
> >>>> "_deleted": true,
> >>>> "_id": "doc1",
> >>>> "_rev": "1-ad7eb689fcae75e7a7edb57dc1f30939",
> >>>> "a": "b"
> >>>> }
> >>>>
> >>>> On prototype/fdb-layer it returns a 409 conflict error
> >>>>
> >>>> I opened a PR to make the prototype/fdb-layer branch behave the same
> >>>> and keep the API compatibility, but also wanted to see what the
> >>>> community thinks.
> >>>>
> >>>> https://github.com/apache/couchdb/pull/3123
> >>>>
> >>>> Would we want to keep compatibility with CouchDB 3.x/PouchDB or,
> >>>> return a conflict (409), like the prototype/fdb-layer branch does?
> >>>>
> >>>> My vote is for compatibility.
> >>>>
> >>>> Thanks,
> >>>> -Nick
> >

[DISCUSS] Creating new deleted documents in CouchDB 4

[Nick Vatamaniuc]

Hi everyone,

While running PouchDB replication unit tests against the CouchDB 4
replicator PR branch (thanks to Garren Smith, who helped set up the
tests), we had noticed a doc update API incompatibility between
CouchDB 3.x/PouchDB and the prototype/fdb-layer branch: CouchDB
3.x/PouchDB allow creating new deleted documents and
prototype/fdb-layer branch doesn't.

For example:

$ http put $DB1/mydb/doc1 _deleted:='true' a=b
HTTP/1.1 200 OK

{
"id": "doc1",
"ok": true,
"rev": "1-ad7eb689fcae75e7a7edb57dc1f30939"
}

$ http $DB1/mydb/doc1?deleted=true
HTTP/1.1 200 OK

{
"_deleted": true,
"_id": "doc1",
"_rev": "1-ad7eb689fcae75e7a7edb57dc1f30939",
"a": "b"
}

On prototype/fdb-layer it returns a 409 conflict error

I opened a PR to make the prototype/fdb-layer branch behave the same
and keep the API compatibility, but also wanted to see what the
community thinks.

https://github.com/apache/couchdb/pull/3123

Would we want to keep compatibility with CouchDB 3.x/PouchDB or,
return a conflict (409), like the prototype/fdb-layer branch does?

My vote is for compatibility.

Thanks,
-Nick