[VOTE] Apache CouchDB 1.0.3 Release
This is the release vote for Apache CouchDB 1.0.3 Changes in this release: * Fixed compatibility issues with Erlang R14B02. * Fix bug that allows invalid UTF-8 after valid escapes. * The query parameter `include_docs` now honors the parameter `conflicts`. This applies to queries against map views, _all_docs and _changes. * Added support for inclusive_end with reduce views. * More performant queries against _changes and _all_docs when using the `include_docs` parameter. * Enabled replication over IPv6. * Fixed for crashes in continuous and filtered changes feeds. * Fixed error when restarting replications in OTP R14B02. * Upgrade ibrowse to version 2.2.0. * Fixed bug when using a filter and a limit of 1. * Fixed OAuth signature computation in OTP R14B02. * Handle passwords with : in them. * Made compatible with jQuery 1.5.x. * Etap tests no longer require use of port 5984. They now use a randomly selected port so they won't clash with a running CouchDB. * Windows builds now require ICU >= 4.4.0 and Erlang >= R14B03. See COUCHDB-1152, and COUCHDB-963 + OTP-9139 for more information. We encourage the whole community to download and test these release artifacts so that any critical issues can be resolved before the release is made. Everyone is free to vote on this release. Please report your results and vote to this thread. We are voting on the following release artifacts: http://people.apache.org/~davisp/dist/1.0.3-rc1/ Instructions for validating the release tarball can be found here: http://people.apache.org/~davisp/dist/ Instructions for testing the build artefacts can be found here: http://wiki.apache.org/couchdb/Test_procedure These artifacts have been built from the 1.0.3 tag in Subversion: http://svn.apache.org/repos/asf/couchdb/tags/1.0.3/ At some point this weekend you will be bored with nothing to do for ten to fifteen minutes, this is when you should vote.
Re: New write performance optimizations coming
On Fri, Jun 24, 2011 at 12:59 PM, Randall Leeds wrote: > On Fri, Jun 24, 2011 at 03:36, Robert Dionne > wrote: >> This is interesting work, I notice some substantial changes to couch_btree, >> a new query_modify_raw, etc.. >> >> I'm wondering though if we'd be better off to base these changes on the re >> factored version of couch_btree that davisp has[1]. I haven't looked at it >> too closely or tested with it but if I recall the goal was first to achieve >> a more readable version with identical semantics so that we could then move >> forward with improvements. >> >> >> [1] >> https://github.com/davisp/couchdb/commit/37c1c9b4b90f6c0f3c22b75dfb2ae55c8b708ab1 >> >> > > I think the only thing holding that back was a good benchmarking. > Can we throw these new benchmarks at that branch? > The concerns expressed last time were due to the switch to using the cmp_keys function. I think it was Randall that had the idea to rewrite the get_node write_node functions to do something in ets to try and focus in on testing just the btree changes. I'd also note that I haven't pulled in the recent changes from trunk that include data sizes and compression into the new branch, so it'll need a bit of work there as well.
Re: New write performance optimizations coming
On Fri, Jun 24, 2011 at 03:36, Robert Dionne
wrote:
> This is interesting work, I notice some substantial changes to couch_btree, a
> new query_modify_raw, etc..
>
> I'm wondering though if we'd be better off to base these changes on the re
> factored version of couch_btree that davisp has[1]. I haven't looked at it
> too closely or tested with it but if I recall the goal was first to achieve
> a more readable version with identical semantics so that we could then move
> forward with improvements.
>
>
> [1]
> https://github.com/davisp/couchdb/commit/37c1c9b4b90f6c0f3c22b75dfb2ae55c8b708ab1
>
>
I think the only thing holding that back was a good benchmarking.
Can we throw these new benchmarks at that branch?
>
>
> On Jun 24, 2011, at 6:06 AM, Filipe David Manana wrote:
>
>> Thanks Adam.
>>
>> Don't get too scared :) Ignore the commit history and just look at
>> github's "Files changed" tab, the modification summary is:
>>
>> "Showing 19 changed files with 730 additions and 402 deletions."
>>
>> More than half of those commits were merges with trunk, many snappy
>> refactorings (before it was added to trunk) and other experiments that
>> were reverted after.
>> We'll try to break this into 2 or 3 patches.
>>
>> So the single patch is something relatively small:
>> https://github.com/fdmanana/couchdb/compare/async_file_writes_no_test.diff
>>
>> On Fri, Jun 24, 2011 at 4:05 AM, Adam Kocoloski wrote:
>>> Hi Damien, I'd like to see these 220 commits rebased into a set of logical
>>> patches against trunk. It'll make the review easier and will help future
>>> devs track down any bugs that are introduced. Best,
>>>
>>> Adam
>>>
>>> On Jun 23, 2011, at 6:49 PM, Damien Katz wrote:
>>>
Hi everyone,
As it’s known by many of you, Filipe and I have been working on improving
performance, specially write performance [1]. This work has been public in
the Couchbase github account since the beginning, and the non Couchbase
specific changes are now isolated in [2] and [3].
In [3] there’s an Erlang module that is used to test the performance when
writing and updating batches of documents with concurrency, which was
used, amongst other tools, to measure the performance gains. This module
bypasses the network stack and the JSON parsing, so that basically it
allows us to see more easily how significant the changes in couch_file,
couch_db and couch_db_updater are.
The main and most important change is asynchronous writes. The file module
no longer blocks callers until the write calls complete. Instead they
immediately reply to the caller with the position in the file where the
data is going to be written to. The data is then sent to a dedicated loop
process that is continuously writing the data it receives, from the
couch_file gen_server, to disk (and batching when possible). This allows
callers (such as the db updater for.e.g.) to issue write calls and keep
doing other work (preparing documents, etc) while the writes are being
done in parallel. After issuing all the writes, callers simply call the
new ‘flush’ function in the couch_file gen_server, which will block the
caller until everything was effectively written to disk - normally this
flush call ends up not blocking the caller or it blocks it for a very
small period.
There are other changes such as avoiding 2 btree lookups per document ID
(COUCHDB-1084 [4]), faster sorting in the updater (O(n log n) vs O(n^2))
and avoid sorting already sorted lists in the updater.
Checking if attachments are compressible was also moved into a new
module/process. We verified this took much CPU time when all or most of
the documents to write/update have attachments - building the regexps and
matching against them for every single attachment is surprisingly
expensive.
There’s also a new couch_db:update_doc/s flag named ‘optimistic’ which
basically changes the behaviour to write the document bodies before
entering the updater and skip some attachment related checks (duplicated
names for e.g.). This flag is not yet exposed to the HTTP api, but it
could be via an X-Optimistic-Write header in the doc PUT/POST requests and
_bulk_docs for e.g. We’ve seen this as good when the client knows that the
documents to write don’t exist yet in the database and we aren’t already
IO bound, such as when SSDs are used.
We used relaximation, Filipe’s basho bench based tests [5] and the Erlang
test module mentioned before [6, 7], exposed via the HTTP . Here follow
some benchmark results.
# Using the Erlang test module (test output)
## 1Kb documents, 10 concurrent writers, batches of 500 docs
trunk before snappy was added:
{"db":"load_test","total":10,"batch":500,"concurrency":10,"
[jira] [Commented] (COUCHDB-1197) trunk no longer builds on Windows
[ https://issues.apache.org/jira/browse/COUCHDB-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054410#comment-13054410 ] Filipe Manana commented on COUCHDB-1197: Dave, no idea at what version we are, but definitely we're behind version 2.0.0, which according to the release notes brings significant performance gains. I'm trying it out and see if no issues, maybe update trunk with v2.0.3 > trunk no longer builds on Windows > - > > Key: COUCHDB-1197 > URL: https://issues.apache.org/jira/browse/COUCHDB-1197 > Project: CouchDB > Issue Type: Bug > Components: Build System, JavaScript View Server > Environment: Windows 7 Enterprise x64 > Cygwin > MS Visual Studio 2008 Express >Reporter: Dave Cottlehuber > Labels: cygwin, windows > Fix For: 1.2 > > Attachments: COUCHDB-1197_fix_ejson.patch, > COUCHDB-1197_fix_ejson_v2.patch, COUCHDB-1197_fix_libcurl.patch > > > ./configure fails > - can no longer correctly find libcurl (after COUCHDB-1042) and instead > compiles against cygwin's curl which is *bad*. Patch attached to resolve this. > - finds jsapi.h correctly but can no longer use it. Work by dch to identify > when it broke and how to fix this underway. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1197) trunk no longer builds on Windows
[ https://issues.apache.org/jira/browse/COUCHDB-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054382#comment-13054382 ] Dave Cottlehuber commented on COUCHDB-1197: --- Already done, just checked & its now committed but slightly modified form: https://github.com/lloyd/yajl/commit/e2d080612f566205d16a65fdad05618f92774d4c I can't see from src/ejson/yajl what version we are at; what makes sense to do next? > trunk no longer builds on Windows > - > > Key: COUCHDB-1197 > URL: https://issues.apache.org/jira/browse/COUCHDB-1197 > Project: CouchDB > Issue Type: Bug > Components: Build System, JavaScript View Server > Environment: Windows 7 Enterprise x64 > Cygwin > MS Visual Studio 2008 Express >Reporter: Dave Cottlehuber > Labels: cygwin, windows > Fix For: 1.2 > > Attachments: COUCHDB-1197_fix_ejson.patch, > COUCHDB-1197_fix_ejson_v2.patch, COUCHDB-1197_fix_libcurl.patch > > > ./configure fails > - can no longer correctly find libcurl (after COUCHDB-1042) and instead > compiles against cygwin's curl which is *bad*. Patch attached to resolve this. > - finds jsapi.h correctly but can no longer use it. Work by dch to identify > when it broke and how to fix this underway. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: New write performance optimizations coming
This is interesting work, I notice some substantial changes to couch_btree, a
new query_modify_raw, etc..
I'm wondering though if we'd be better off to base these changes on the re
factored version of couch_btree that davisp has[1]. I haven't looked at it too
closely or tested with it but if I recall the goal was first to achieve
a more readable version with identical semantics so that we could then move
forward with improvements.
[1]
https://github.com/davisp/couchdb/commit/37c1c9b4b90f6c0f3c22b75dfb2ae55c8b708ab1
On Jun 24, 2011, at 6:06 AM, Filipe David Manana wrote:
> Thanks Adam.
>
> Don't get too scared :) Ignore the commit history and just look at
> github's "Files changed" tab, the modification summary is:
>
> "Showing 19 changed files with 730 additions and 402 deletions."
>
> More than half of those commits were merges with trunk, many snappy
> refactorings (before it was added to trunk) and other experiments that
> were reverted after.
> We'll try to break this into 2 or 3 patches.
>
> So the single patch is something relatively small:
> https://github.com/fdmanana/couchdb/compare/async_file_writes_no_test.diff
>
> On Fri, Jun 24, 2011 at 4:05 AM, Adam Kocoloski wrote:
>> Hi Damien, I'd like to see these 220 commits rebased into a set of logical
>> patches against trunk. It'll make the review easier and will help future
>> devs track down any bugs that are introduced. Best,
>>
>> Adam
>>
>> On Jun 23, 2011, at 6:49 PM, Damien Katz wrote:
>>
>>> Hi everyone,
>>>
>>> As it’s known by many of you, Filipe and I have been working on improving
>>> performance, specially write performance [1]. This work has been public in
>>> the Couchbase github account since the beginning, and the non Couchbase
>>> specific changes are now isolated in [2] and [3].
>>> In [3] there’s an Erlang module that is used to test the performance when
>>> writing and updating batches of documents with concurrency, which was used,
>>> amongst other tools, to measure the performance gains. This module bypasses
>>> the network stack and the JSON parsing, so that basically it allows us to
>>> see more easily how significant the changes in couch_file, couch_db and
>>> couch_db_updater are.
>>>
>>> The main and most important change is asynchronous writes. The file module
>>> no longer blocks callers until the write calls complete. Instead they
>>> immediately reply to the caller with the position in the file where the
>>> data is going to be written to. The data is then sent to a dedicated loop
>>> process that is continuously writing the data it receives, from the
>>> couch_file gen_server, to disk (and batching when possible). This allows
>>> callers (such as the db updater for.e.g.) to issue write calls and keep
>>> doing other work (preparing documents, etc) while the writes are being done
>>> in parallel. After issuing all the writes, callers simply call the new
>>> ‘flush’ function in the couch_file gen_server, which will block the caller
>>> until everything was effectively written to disk - normally this flush call
>>> ends up not blocking the caller or it blocks it for a very small period.
>>>
>>> There are other changes such as avoiding 2 btree lookups per document ID
>>> (COUCHDB-1084 [4]), faster sorting in the updater (O(n log n) vs O(n^2))
>>> and avoid sorting already sorted lists in the updater.
>>>
>>> Checking if attachments are compressible was also moved into a new
>>> module/process. We verified this took much CPU time when all or most of the
>>> documents to write/update have attachments - building the regexps and
>>> matching against them for every single attachment is surprisingly expensive.
>>>
>>> There’s also a new couch_db:update_doc/s flag named ‘optimistic’ which
>>> basically changes the behaviour to write the document bodies before
>>> entering the updater and skip some attachment related checks (duplicated
>>> names for e.g.). This flag is not yet exposed to the HTTP api, but it could
>>> be via an X-Optimistic-Write header in the doc PUT/POST requests and
>>> _bulk_docs for e.g. We’ve seen this as good when the client knows that the
>>> documents to write don’t exist yet in the database and we aren’t already IO
>>> bound, such as when SSDs are used.
>>>
>>> We used relaximation, Filipe’s basho bench based tests [5] and the Erlang
>>> test module mentioned before [6, 7], exposed via the HTTP . Here follow
>>> some benchmark results.
>>>
>>>
>>> # Using the Erlang test module (test output)
>>>
>>> ## 1Kb documents, 10 concurrent writers, batches of 500 docs
>>>
>>> trunk before snappy was added:
>>>
>>> {"db":"load_test","total":10,"batch":500,"concurrency":10,"rounds":10,"delayed_commits":false,"optimistic":false,"total_time_ms":270071}
>>>
>>> trunk:
>>>
>>> {"db":"load_test","total":10,"batch":500,"concurrency":10,"rounds":10,"delayed_commits":false,"optimistic":false,"total_time_ms":157328}
>>>
>>> trunk + async writ
[jira] [Commented] (COUCHDB-1197) trunk no longer builds on Windows
[ https://issues.apache.org/jira/browse/COUCHDB-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054344#comment-13054344 ] Filipe Manana commented on COUCHDB-1197: Dave, the yajl change, do you think you can contribute it upstream? > trunk no longer builds on Windows > - > > Key: COUCHDB-1197 > URL: https://issues.apache.org/jira/browse/COUCHDB-1197 > Project: CouchDB > Issue Type: Bug > Components: Build System, JavaScript View Server > Environment: Windows 7 Enterprise x64 > Cygwin > MS Visual Studio 2008 Express >Reporter: Dave Cottlehuber > Labels: cygwin, windows > Fix For: 1.2 > > Attachments: COUCHDB-1197_fix_ejson.patch, > COUCHDB-1197_fix_ejson_v2.patch, COUCHDB-1197_fix_libcurl.patch > > > ./configure fails > - can no longer correctly find libcurl (after COUCHDB-1042) and instead > compiles against cygwin's curl which is *bad*. Patch attached to resolve this. > - finds jsapi.h correctly but can no longer use it. Work by dch to identify > when it broke and how to fix this underway. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-1197) trunk no longer builds on Windows
[ https://issues.apache.org/jira/browse/COUCHDB-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Cottlehuber updated COUCHDB-1197: -- Attachment: COUCHDB-1197_fix_ejson_v2.patch Thanks benoitc for advice on how to handle the Makefile. > trunk no longer builds on Windows > - > > Key: COUCHDB-1197 > URL: https://issues.apache.org/jira/browse/COUCHDB-1197 > Project: CouchDB > Issue Type: Bug > Components: Build System, JavaScript View Server > Environment: Windows 7 Enterprise x64 > Cygwin > MS Visual Studio 2008 Express >Reporter: Dave Cottlehuber > Labels: cygwin, windows > Fix For: 1.2 > > Attachments: COUCHDB-1197_fix_ejson.patch, > COUCHDB-1197_fix_ejson_v2.patch, COUCHDB-1197_fix_libcurl.patch > > > ./configure fails > - can no longer correctly find libcurl (after COUCHDB-1042) and instead > compiles against cygwin's curl which is *bad*. Patch attached to resolve this. > - finds jsapi.h correctly but can no longer use it. Work by dch to identify > when it broke and how to fix this underway. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1197) trunk no longer builds on Windows
[ https://issues.apache.org/jira/browse/COUCHDB-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054337#comment-13054337 ] Dave Cottlehuber commented on COUCHDB-1197: --- new attachment COUCHDB-1197_fix_ejson_v2.patch completes ejson fix (Makefile.am and yajl_gen.c) > trunk no longer builds on Windows > - > > Key: COUCHDB-1197 > URL: https://issues.apache.org/jira/browse/COUCHDB-1197 > Project: CouchDB > Issue Type: Bug > Components: Build System, JavaScript View Server > Environment: Windows 7 Enterprise x64 > Cygwin > MS Visual Studio 2008 Express >Reporter: Dave Cottlehuber > Labels: cygwin, windows > Fix For: 1.2 > > Attachments: COUCHDB-1197_fix_ejson.patch, > COUCHDB-1197_fix_libcurl.patch > > > ./configure fails > - can no longer correctly find libcurl (after COUCHDB-1042) and instead > compiles against cygwin's curl which is *bad*. Patch attached to resolve this. > - finds jsapi.h correctly but can no longer use it. Work by dch to identify > when it broke and how to fix this underway. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: New write performance optimizations coming
Thanks Adam.
Don't get too scared :) Ignore the commit history and just look at
github's "Files changed" tab, the modification summary is:
"Showing 19 changed files with 730 additions and 402 deletions."
More than half of those commits were merges with trunk, many snappy
refactorings (before it was added to trunk) and other experiments that
were reverted after.
We'll try to break this into 2 or 3 patches.
So the single patch is something relatively small:
https://github.com/fdmanana/couchdb/compare/async_file_writes_no_test.diff
On Fri, Jun 24, 2011 at 4:05 AM, Adam Kocoloski wrote:
> Hi Damien, I'd like to see these 220 commits rebased into a set of logical
> patches against trunk. It'll make the review easier and will help future
> devs track down any bugs that are introduced. Best,
>
> Adam
>
> On Jun 23, 2011, at 6:49 PM, Damien Katz wrote:
>
>> Hi everyone,
>>
>> As it’s known by many of you, Filipe and I have been working on improving
>> performance, specially write performance [1]. This work has been public in
>> the Couchbase github account since the beginning, and the non Couchbase
>> specific changes are now isolated in [2] and [3].
>> In [3] there’s an Erlang module that is used to test the performance when
>> writing and updating batches of documents with concurrency, which was used,
>> amongst other tools, to measure the performance gains. This module bypasses
>> the network stack and the JSON parsing, so that basically it allows us to
>> see more easily how significant the changes in couch_file, couch_db and
>> couch_db_updater are.
>>
>> The main and most important change is asynchronous writes. The file module
>> no longer blocks callers until the write calls complete. Instead they
>> immediately reply to the caller with the position in the file where the data
>> is going to be written to. The data is then sent to a dedicated loop process
>> that is continuously writing the data it receives, from the couch_file
>> gen_server, to disk (and batching when possible). This allows callers (such
>> as the db updater for.e.g.) to issue write calls and keep doing other work
>> (preparing documents, etc) while the writes are being done in parallel.
>> After issuing all the writes, callers simply call the new ‘flush’ function
>> in the couch_file gen_server, which will block the caller until everything
>> was effectively written to disk - normally this flush call ends up not
>> blocking the caller or it blocks it for a very small period.
>>
>> There are other changes such as avoiding 2 btree lookups per document ID
>> (COUCHDB-1084 [4]), faster sorting in the updater (O(n log n) vs O(n^2)) and
>> avoid sorting already sorted lists in the updater.
>>
>> Checking if attachments are compressible was also moved into a new
>> module/process. We verified this took much CPU time when all or most of the
>> documents to write/update have attachments - building the regexps and
>> matching against them for every single attachment is surprisingly expensive.
>>
>> There’s also a new couch_db:update_doc/s flag named ‘optimistic’ which
>> basically changes the behaviour to write the document bodies before entering
>> the updater and skip some attachment related checks (duplicated names for
>> e.g.). This flag is not yet exposed to the HTTP api, but it could be via an
>> X-Optimistic-Write header in the doc PUT/POST requests and _bulk_docs for
>> e.g. We’ve seen this as good when the client knows that the documents to
>> write don’t exist yet in the database and we aren’t already IO bound, such
>> as when SSDs are used.
>>
>> We used relaximation, Filipe’s basho bench based tests [5] and the Erlang
>> test module mentioned before [6, 7], exposed via the HTTP . Here follow some
>> benchmark results.
>>
>>
>> # Using the Erlang test module (test output)
>>
>> ## 1Kb documents, 10 concurrent writers, batches of 500 docs
>>
>> trunk before snappy was added:
>>
>> {"db":"load_test","total":10,"batch":500,"concurrency":10,"rounds":10,"delayed_commits":false,"optimistic":false,"total_time_ms":270071}
>>
>> trunk:
>>
>> {"db":"load_test","total":10,"batch":500,"concurrency":10,"rounds":10,"delayed_commits":false,"optimistic":false,"total_time_ms":157328}
>>
>> trunk + async writes (and snappy):
>>
>> {"db":"load_test","total":10,"batch":500,"concurrency":10,"rounds":10,"delayed_commits":false,"optimistic":false,"total_time_ms":121518}
>>
>> ## 2.5Kb documents, 10 concurrent writers, batches of 500 docs
>>
>> trunk before snappy was added:
>>
>> {"db":"load_test","total":10,"batch":500,"concurrency":10,"rounds":10,"delayed_commits":false,"optimistic":false,"total_time_ms":507098}
>>
>> trunk:
>>
>> {"db":"load_test","total":10,"batch":500,"concurrency":10,"rounds":10,"delayed_commits":false,"optimistic":false,"total_time_ms":230391}
>>
>> trunk + async writes (and snappy):
>>
>> {"db":"load_test","total":10,"batch":500,"concurrency":10,"rounds":10,"delayed_com
