[jira] [Commented] (COUCHDB-1606) Replicator leaves plaintext password in logs
[ https://issues.apache.org/jira/browse/COUCHDB-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13500143#comment-13500143 ] Bob Dionne commented on COUCHDB-1606: - I'll take a look at this one Replicator leaves plaintext password in logs Key: COUCHDB-1606 URL: https://issues.apache.org/jira/browse/COUCHDB-1606 Project: CouchDB Issue Type: Bug Affects Versions: 1.2 Reporter: Nathan Vander Wilt Assignee: Bob Dionne Attachments: pwd log.txt While reviewing logs, I noticed that a password had been recorded in the logs as part of a replicator error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (COUCHDB-1606) Replicator leaves plaintext password in logs
[ https://issues.apache.org/jira/browse/COUCHDB-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Dionne reassigned COUCHDB-1606: --- Assignee: Bob Dionne Replicator leaves plaintext password in logs Key: COUCHDB-1606 URL: https://issues.apache.org/jira/browse/COUCHDB-1606 Project: CouchDB Issue Type: Bug Affects Versions: 1.2 Reporter: Nathan Vander Wilt Assignee: Bob Dionne Attachments: pwd log.txt While reviewing logs, I noticed that a password had been recorded in the logs as part of a replicator error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (COUCHDB-1583) improve the flow control in ibrowse streams
Bob Dionne created COUCHDB-1583: --- Summary: improve the flow control in ibrowse streams Key: COUCHDB-1583 URL: https://issues.apache.org/jira/browse/COUCHDB-1583 Project: CouchDB Issue Type: Improvement Components: Replication Reporter: Bob Dionne Assignee: Adam Kocoloski Fix For: 1.3 The flow control in ibrowse's async response streams is tricky. We call stream_next to pull more data off the socket, but it seems that ibrowse will sometimes split that data into multiple messages. If we call stream_next for each message we process we end up with an overflowing mailbox. This commit[1] changes the consumer so that it clears out the mailbox before calling stream_next. [1] 8ccf696f833a0f0a453d -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (COUCHDB-1557) upgrade some code to use BIFs
[ https://issues.apache.org/jira/browse/COUCHDB-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Dionne resolved COUCHDB-1557. - Resolution: Fixed Fix Version/s: 1.3 fixed on master upgrade some code to use BIFs - Key: COUCHDB-1557 URL: https://issues.apache.org/jira/browse/COUCHDB-1557 Project: CouchDB Issue Type: Improvement Components: HTTP Interface, Replication Reporter: Bob Dionne Assignee: Bob Dionne Priority: Minor Fix For: 1.3 Attachments: 0001-Upgrade-code-to-use-BIFs-for-improved-performance.patch couch_httpd:find_in_binary can now make use of the binary module introduced in R14 -- eprof runs of the replication can show good improvements with this patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1557) upgrade some code to use BIFs
[ https://issues.apache.org/jira/browse/COUCHDB-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13474038#comment-13474038 ] Bob Dionne commented on COUCHDB-1557: - I've also made similar changes to ibrowse[1], for which I can submit a patch once it's been decided how upstream projects will be handled in the new build [1] https://github.com/cloudant/ibrowse/commit/cc1f8e84a669b37 upgrade some code to use BIFs - Key: COUCHDB-1557 URL: https://issues.apache.org/jira/browse/COUCHDB-1557 Project: CouchDB Issue Type: Improvement Components: HTTP Interface, Replication Reporter: Bob Dionne Priority: Minor Attachments: 0001-Upgrade-code-to-use-BIFs-for-improved-performance.patch couch_httpd:find_in_binary can now make use of the binary module introduced in R14 -- eprof runs of the replication can show good improvements with this patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (COUCHDB-1557) upgrade some code to use BIFs
Bob Dionne created COUCHDB-1557: --- Summary: upgrade some code to use BIFs Key: COUCHDB-1557 URL: https://issues.apache.org/jira/browse/COUCHDB-1557 Project: CouchDB Issue Type: Improvement Components: HTTP Interface, Replication Reporter: Bob Dionne Priority: Minor couch_httpd:find_in_binary can now make use of the binary module introduced in R14 -- eprof runs of the replication can show good improvements with this patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-1557) upgrade some code to use BIFs
[ https://issues.apache.org/jira/browse/COUCHDB-1557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Dionne updated COUCHDB-1557: Attachment: 0001-Upgrade-code-to-use-BIFs-for-improved-performance.patch upgrade some code to use BIFs - Key: COUCHDB-1557 URL: https://issues.apache.org/jira/browse/COUCHDB-1557 Project: CouchDB Issue Type: Improvement Components: HTTP Interface, Replication Reporter: Bob Dionne Priority: Minor Attachments: 0001-Upgrade-code-to-use-BIFs-for-improved-performance.patch couch_httpd:find_in_binary can now make use of the binary module introduced in R14 -- eprof runs of the replication can show good improvements with this patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1288) More efficient builtin filters _doc_ids and _design
[ https://issues.apache.org/jira/browse/COUCHDB-1288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13107757#comment-13107757 ] Bob Dionne commented on COUCHDB-1288: - Filipe, I started reviewing this and it looks good so far. There's an edge case we ran into the other day that @davisp and @kocolosk ran down. When you have `feed=continuous` and a hearbeat and a filter function that fail enough, the heartbeat timeout never triggers and no changes are sent. It's easy to reproduce, you can see how it's handled in fabric[1]. I can probably add it to this patch or open a second ticket if you prefer. Also, as an aside the `couch_changes:get_changes_timeout` is slightly awkward in the way heartbeat is handled. It appears to allow `heartbeat=true` and in that case defaults to the timeout in the config. That certainly does not agree with the documented semantics. Cheers, Bob [1] https://github.com/cloudant/fabric/commit/f9eea28e62496afcb More efficient builtin filters _doc_ids and _design --- Key: COUCHDB-1288 URL: https://issues.apache.org/jira/browse/COUCHDB-1288 Project: CouchDB Issue Type: Improvement Reporter: Filipe Manana Attachments: couchdb_1288_2.patch, couchdb_1288_3.patch We have the _doc_ids and _design _changes filter as of CouchDB 1.1.0. While they meet the expectations of applications/users, they're far from efficient for large databases. Basically the implementation folds the entire seq btree and then filters values by the document's ID, causing too much IO and busting caches. This makes replication by doc IDs not so efficient as it could be. The proposed patch avoids this by doing direct lookups in the ID btree, for _doc_ids, and ranged fold for _design. If there are no objections, I would apply to branch 1.2.x besides -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (COUCHDB-1289) heartbeats skipped when continuous changes feed filter function produces no results
heartbeats skipped when continuous changes feed filter function produces no results --- Key: COUCHDB-1289 URL: https://issues.apache.org/jira/browse/COUCHDB-1289 Project: CouchDB Issue Type: Bug Components: Database Core Reporter: Bob Dionne Priority: Minor if the changes feed has a filter function that produces no results, db_updated messages will still be sent and the heartbeat timeout will never be reached. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (COUCHDB-1289) heartbeats skipped when continuous changes feed filter function produces no results
[ https://issues.apache.org/jira/browse/COUCHDB-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Dionne reassigned COUCHDB-1289: --- Assignee: Bob Dionne heartbeats skipped when continuous changes feed filter function produces no results --- Key: COUCHDB-1289 URL: https://issues.apache.org/jira/browse/COUCHDB-1289 Project: CouchDB Issue Type: Bug Components: Database Core Reporter: Bob Dionne Assignee: Bob Dionne Priority: Minor if the changes feed has a filter function that produces no results, db_updated messages will still be sent and the heartbeat timeout will never be reached. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-911) Repeating a doc._id in a _bulk_docs request results in erroneous Document conflict error
[
https://issues.apache.org/jira/browse/COUCHDB-911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bob Dionne updated COUCHDB-911:
---
Attachment: (was: jira-911-patch2.diff)
Repeating a doc._id in a _bulk_docs request results in erroneous Document
conflict error
--
Key: COUCHDB-911
URL: https://issues.apache.org/jira/browse/COUCHDB-911
Project: CouchDB
Issue Type: Bug
Components: HTTP Interface
Affects Versions: 1.0
Environment: Cloudant BigCouch EC2 node
Reporter: Jay Nelson
Priority: Minor
Original Estimate: 48h
Remaining Estimate: 48h
Repeating an _id in a _bulk_docs post data file results in both entries
being reported as document conflict errors. The first occurrence actual
inserts into the database, and only the second occurrence should report a
conflict.
curl -d '{ docs: [ {_id:foo}, {_id,foo} ] }' -H
'Content-Type:application/json' -X POST
http://appadvice.cloudant.com/foo/_bulk_docs
[{id:foo,error:conflict,reason:Document update
conflict.},{id:foo,error:conflict,reason:Document update
conflict.}]
But the database shows that one new document was actually inserted.
Only the second occurrence should report conflict. The first occurrence
should report the _rev property of the newly inserted doc.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-911) Repeating a doc._id in a _bulk_docs request results in erroneous Document conflict error
[
https://issues.apache.org/jira/browse/COUCHDB-911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bob Dionne updated COUCHDB-911:
---
Attachment: jira-911.patch
whitespace.patch
Here's a patch that adds a reference to each doc in order to distinguish the
dups. It's a much more substantial change but arguably safer. All the tests
pass.
Repeating a doc._id in a _bulk_docs request results in erroneous Document
conflict error
--
Key: COUCHDB-911
URL: https://issues.apache.org/jira/browse/COUCHDB-911
Project: CouchDB
Issue Type: Bug
Components: HTTP Interface
Affects Versions: 1.0
Environment: Cloudant BigCouch EC2 node
Reporter: Jay Nelson
Priority: Minor
Attachments: jira-911.patch, whitespace.patch
Original Estimate: 48h
Remaining Estimate: 48h
Repeating an _id in a _bulk_docs post data file results in both entries
being reported as document conflict errors. The first occurrence actual
inserts into the database, and only the second occurrence should report a
conflict.
curl -d '{ docs: [ {_id:foo}, {_id,foo} ] }' -H
'Content-Type:application/json' -X POST
http://appadvice.cloudant.com/foo/_bulk_docs
[{id:foo,error:conflict,reason:Document update
conflict.},{id:foo,error:conflict,reason:Document update
conflict.}]
But the database shows that one new document was actually inserted.
Only the second occurrence should report conflict. The first occurrence
should report the _rev property of the newly inserted doc.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1270) Rewrite the view engine
[ https://issues.apache.org/jira/browse/COUCHDB-1270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13099600#comment-13099600 ] Bob Dionne commented on COUCHDB-1270: - +1 I think the compact and compact_swap functions make the API somewhat specific, but a no-op implementation is easy. Implementers will certainly need to know the atoms that the `get(Field,State)` function is looking for. All the tests run with the latest version. Well done Paul! Rewrite the view engine --- Key: COUCHDB-1270 URL: https://issues.apache.org/jira/browse/COUCHDB-1270 Project: CouchDB Issue Type: Improvement Components: JavaScript View Server Reporter: Paul Joseph Davis Attachments: 0001-Minor-changes-for-new-indexing-engine.patch, 0002-Create-the-couch_index-application.patch, 0003-Create-the-couch_mrview-application.patch, 0004-Remove-the-old-view-engine.patch The view engine has been creaky and cluttered. As shown by GeoCouch, adding new indexers basically involves copying the entire view engine and hacking the parts that are different. In short, the opposite of good engineering. Over the last couple weeks I've refactored the view engine and reimplemented the map/reduce view engine. These changes are 100% internal and no external behavior has changed. Performance is just a tiny bit better than trunk. I did do some playing trying to improve view update times and there are some dances we could do, but for the time being I wanted to keep the same general architecture for updates so that the changes are minimal. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-911) Repeating a doc._id in a _bulk_docs request results in erroneous Document conflict error
[
https://issues.apache.org/jira/browse/COUCHDB-911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bob Dionne updated COUCHDB-911:
---
Attachment: (was: jira-911-patch.diff)
Repeating a doc._id in a _bulk_docs request results in erroneous Document
conflict error
--
Key: COUCHDB-911
URL: https://issues.apache.org/jira/browse/COUCHDB-911
Project: CouchDB
Issue Type: Bug
Components: HTTP Interface
Affects Versions: 1.0
Environment: Cloudant BigCouch EC2 node
Reporter: Jay Nelson
Priority: Minor
Attachments: jira-911-patch2.diff
Original Estimate: 48h
Remaining Estimate: 48h
Repeating an _id in a _bulk_docs post data file results in both entries
being reported as document conflict errors. The first occurrence actual
inserts into the database, and only the second occurrence should report a
conflict.
curl -d '{ docs: [ {_id:foo}, {_id,foo} ] }' -H
'Content-Type:application/json' -X POST
http://appadvice.cloudant.com/foo/_bulk_docs
[{id:foo,error:conflict,reason:Document update
conflict.},{id:foo,error:conflict,reason:Document update
conflict.}]
But the database shows that one new document was actually inserted.
Only the second occurrence should report conflict. The first occurrence
should report the _rev property of the newly inserted doc.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-911) Repeating a doc._id in a _bulk_docs request results in erroneous Document conflict error
[
https://issues.apache.org/jira/browse/COUCHDB-911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bob Dionne updated COUCHDB-911:
---
Attachment: jira-911-patch2.diff
Here's a somewhat cleaner version
Repeating a doc._id in a _bulk_docs request results in erroneous Document
conflict error
--
Key: COUCHDB-911
URL: https://issues.apache.org/jira/browse/COUCHDB-911
Project: CouchDB
Issue Type: Bug
Components: HTTP Interface
Affects Versions: 1.0
Environment: Cloudant BigCouch EC2 node
Reporter: Jay Nelson
Priority: Minor
Attachments: jira-911-patch2.diff
Original Estimate: 48h
Remaining Estimate: 48h
Repeating an _id in a _bulk_docs post data file results in both entries
being reported as document conflict errors. The first occurrence actual
inserts into the database, and only the second occurrence should report a
conflict.
curl -d '{ docs: [ {_id:foo}, {_id,foo} ] }' -H
'Content-Type:application/json' -X POST
http://appadvice.cloudant.com/foo/_bulk_docs
[{id:foo,error:conflict,reason:Document update
conflict.},{id:foo,error:conflict,reason:Document update
conflict.}]
But the database shows that one new document was actually inserted.
Only the second occurrence should report conflict. The first occurrence
should report the _rev property of the newly inserted doc.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-911) Repeating a doc._id in a _bulk_docs request results in erroneous Document conflict error
[
https://issues.apache.org/jira/browse/COUCHDB-911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bob Dionne updated COUCHDB-911:
---
Attachment: jira-911-patch.diff
Here's a JS test and a draft of a patch.
Repeating a doc._id in a _bulk_docs request results in erroneous Document
conflict error
--
Key: COUCHDB-911
URL: https://issues.apache.org/jira/browse/COUCHDB-911
Project: CouchDB
Issue Type: Bug
Components: HTTP Interface
Affects Versions: 1.0
Environment: Cloudant BigCouch EC2 node
Reporter: Jay Nelson
Priority: Minor
Attachments: jira-911-patch.diff
Original Estimate: 48h
Remaining Estimate: 48h
Repeating an _id in a _bulk_docs post data file results in both entries
being reported as document conflict errors. The first occurrence actual
inserts into the database, and only the second occurrence should report a
conflict.
curl -d '{ docs: [ {_id:foo}, {_id,foo} ] }' -H
'Content-Type:application/json' -X POST
http://appadvice.cloudant.com/foo/_bulk_docs
[{id:foo,error:conflict,reason:Document update
conflict.},{id:foo,error:conflict,reason:Document update
conflict.}]
But the database shows that one new document was actually inserted.
Only the second occurrence should report conflict. The first occurrence
should report the _rev property of the newly inserted doc.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-1256) Incremental requests to _changes can skip revisions
[
https://issues.apache.org/jira/browse/COUCHDB-1256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bob Dionne updated COUCHDB-1256:
Attachment: jira-1256-test.diff
here's a failing JS test - the suggested patch fixes it
Incremental requests to _changes can skip revisions
---
Key: COUCHDB-1256
URL: https://issues.apache.org/jira/browse/COUCHDB-1256
Project: CouchDB
Issue Type: Bug
Components: Replication
Affects Versions: 0.10, 0.10.1, 0.10.2, 0.11.1, 0.11.2, 1.0, 1.0.1, 1.0.2,
1.1, 1.0.3
Environment: confirmed on Apache CouchDB 1.1.0, bug appears to be
present in 1.0.3 and trunk
Reporter: Adam Kocoloski
Assignee: Adam Kocoloski
Priority: Blocker
Fix For: 1.0.4, 1.1.1, 1.2
Attachments: jira-1256-test.diff
Requests to _changes with style=all_docssince=N (requests made by the
replicator) are liable to suppress revisions of a document. The following
sequence of curl commands demonstrates the bug:
curl -X PUT localhost:5985/revseq
{ok:true}
curl -X PUT -Hcontent-type:application/json localhost:5985/revseq/foo -d
'{a:123}'
{ok:true,id:foo,rev:1-0dc33db52a43872b6f3371cef7de0277}
curl -X PUT -Hcontent-type:application/json localhost:5985/revseq/bar -d
'{a:456}'
{ok:true,id:bar,rev:1-cc609831f0ca66e8cd3d4c1e0d98108a}
% stick a conflict revision in foo
curl -X PUT -Hcontent-type:application/json
localhost:5985/revseq/foo?new_edits=false -d
'{_rev:1-cc609831f0ca66e8cd3d4c1e0d98108a, a:123}'
{ok:true,id:foo,rev:1-cc609831f0ca66e8cd3d4c1e0d98108a}
% request without since= gives the expected result
curl -Hcontent-type:application/json
localhost:5985/revseq/_changes?style=all_docs
{results:[
{seq:2,id:bar,changes:[{rev:1-cc609831f0ca66e8cd3d4c1e0d98108a}]},
{seq:3,id:foo,changes:[{rev:1-cc609831f0ca66e8cd3d4c1e0d98108a},{rev:1-0dc33db52a43872b6f3371cef7de0277}]}
],
last_seq:3}
% request starting from since=2 suppresses revision
1-0dc33db52a43872b6f3371cef7de0277 of foo
macbook:~ (master) $ curl
localhost:5985/revseq/_changes?style=all_docs\since=2
{results:[
{seq:3,id:foo,changes:[{rev:1-cc609831f0ca66e8cd3d4c1e0d98108a}]}
],
last_seq:3}
I believe the fix is something like this (though we could refactor further
because Style is unused):
diff --git a/src/couchdb/couch_db.erl b/src/couchdb/couch_db.erl
index e8705be..65aeca3 100644
--- a/src/couchdb/couch_db.erl
+++ b/src/couchdb/couch_db.erl
@@ -1029,19 +1029,7 @@ changes_since(Db, Style, StartSeq, Fun, Acc) -
changes_since(Db, Style, StartSeq, Fun, [], Acc).
changes_since(Db, Style, StartSeq, Fun, Options, Acc) -
-Wrapper = fun(DocInfo, _Offset, Acc2) -
-#doc_info{revs=Revs} = DocInfo,
-DocInfo2 =
-case Style of
-main_only -
-DocInfo;
-all_docs -
-% remove revs before the seq
-DocInfo#doc_info{revs=[RevInfo ||
-#rev_info{seq=RevSeq}=RevInfo - Revs, StartSeq
RevSeq]}
-end,
-Fun(DocInfo2, Acc2)
-end,
+Wrapper = fun(DocInfo, _Offset, Acc2) - Fun(DocInfo, Acc2) end,
{ok, _LastReduction, AccOut} = couch_btree:fold(by_seq_btree(Db),
Wrapper, Acc, [{start_key, StartSeq + 1}] ++ Options),
{ok, AccOut}.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-911) Repeating a doc._id in a _bulk_docs request results in erroneous Document conflict error
[
https://issues.apache.org/jira/browse/COUCHDB-911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13088183#comment-13088183
]
Bob Dionne commented on COUCHDB-911:
I think that's true, you might have a dup in the bulk docs request and both are
rejected because a third with the same id was sent from another client. All
this patch does is match the docs that are sent to couch_db_updater with any
conflict messages that are returned. If you send docs A,B,C,D, and E they will
be processed in this order. I think this is true, even though other clients
might send updates for B and C and they will be grouped in. But the client that
sent A,B,C,D, and E will see any conflicts returned in that order.
Two docs with the same id might have different bodies, .ie. aren't really dups.
It's important to indicate which is rejected and which committed. Currently for
both this case and the case of dups we save the second (because we reverse the
order) and report both rejected, which leaves the database in an inconsistent
state.
Repeating a doc._id in a _bulk_docs request results in erroneous Document
conflict error
--
Key: COUCHDB-911
URL: https://issues.apache.org/jira/browse/COUCHDB-911
Project: CouchDB
Issue Type: Bug
Components: HTTP Interface
Affects Versions: 1.0
Environment: Cloudant BigCouch EC2 node
Reporter: Jay Nelson
Priority: Minor
Original Estimate: 48h
Remaining Estimate: 48h
Repeating an _id in a _bulk_docs post data file results in both entries
being reported as document conflict errors. The first occurrence actual
inserts into the database, and only the second occurrence should report a
conflict.
curl -d '{ docs: [ {_id:foo}, {_id,foo} ] }' -H
'Content-Type:application/json' -X POST
http://appadvice.cloudant.com/foo/_bulk_docs
[{id:foo,error:conflict,reason:Document update
conflict.},{id:foo,error:conflict,reason:Document update
conflict.}]
But the database shows that one new document was actually inserted.
Only the second occurrence should report conflict. The first occurrence
should report the _rev property of the newly inserted doc.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-911) Repeating a doc._id in a _bulk_docs request results in erroneous Document conflict error
[
https://issues.apache.org/jira/browse/COUCHDB-911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13087671#comment-13087671
]
Bob Dionne commented on COUCHDB-911:
so actually when there are two docs of the same id in _bulk_docs, it's the
second that persists and the first that generates a conflict in
merge_rev_trees, because group_alike_docs[1] reverses the order of the docs in
a bucket. To preserve the order we need another reverse here. If we fixed this,
we could hack the ResultsDict[2], as we're processing the docs, and return only
the second doc as a conflict. But this would only solve the bulk_docs case.
There's also the issue of couch_db_updater:collect_updates [3] that appears to
allow more grouped docs to be added for other clients? This is ugly, perhaps it
needs to be disabled for the bulk_docs case. The alternative would be to add
some bits to the handle_info({update_docs call to track which docs turn out
to have conflicts.
It's not pretty
[1] https://github.com/bdionne/couchdb/blob/master/src/couchdb/couch_db.erl#L435
[2] https://github.com/bdionne/couchdb/blob/master/src/couchdb/couch_db.erl#L759
[3]
https://github.com/bdionne/couchdb/blob/master/src/couchdb/couch_db_updater.erl#L274
Repeating a doc._id in a _bulk_docs request results in erroneous Document
conflict error
--
Key: COUCHDB-911
URL: https://issues.apache.org/jira/browse/COUCHDB-911
Project: CouchDB
Issue Type: Bug
Components: HTTP Interface
Affects Versions: 1.0
Environment: Cloudant BigCouch EC2 node
Reporter: Jay Nelson
Priority: Minor
Original Estimate: 48h
Remaining Estimate: 48h
Repeating an _id in a _bulk_docs post data file results in both entries
being reported as document conflict errors. The first occurrence actual
inserts into the database, and only the second occurrence should report a
conflict.
curl -d '{ docs: [ {_id:foo}, {_id,foo} ] }' -H
'Content-Type:application/json' -X POST
http://appadvice.cloudant.com/foo/_bulk_docs
[{id:foo,error:conflict,reason:Document update
conflict.},{id:foo,error:conflict,reason:Document update
conflict.}]
But the database shows that one new document was actually inserted.
Only the second occurrence should report conflict. The first occurrence
should report the _rev property of the newly inserted doc.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-911) Repeating a doc._id in a _bulk_docs request results in erroneous Document conflict error
[
https://issues.apache.org/jira/browse/COUCHDB-911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13088101#comment-13088101
]
Bob Dionne commented on COUCHDB-911:
No it really doesn't matter. One could argue that since the list of buckets is
reversed then the buckets should also be reversed to preserve the order, or
neither should.
So here's a solution[1] that works that does depend on update_docs being called
with the docs in the correct order. It needs a bit of refactoring but the idea
itself is simple. It removes the dictionary since dict:from_list throws away
the dups, and works thru the list of docs backwards, looking them up in the
results list from the commits. If conflicts are present they are matched to the
correct docs. As they are used they are removed from the commits list.
It's not pretty but I should be able to simplify, the only alternative I could
think of involves passing some additional bits in to the elephant grave yard
that is couch_db_updater.
[1] https://github.com/cloudant/bigcouch/commit/3fbb4fd1caa0d998
Repeating a doc._id in a _bulk_docs request results in erroneous Document
conflict error
--
Key: COUCHDB-911
URL: https://issues.apache.org/jira/browse/COUCHDB-911
Project: CouchDB
Issue Type: Bug
Components: HTTP Interface
Affects Versions: 1.0
Environment: Cloudant BigCouch EC2 node
Reporter: Jay Nelson
Priority: Minor
Original Estimate: 48h
Remaining Estimate: 48h
Repeating an _id in a _bulk_docs post data file results in both entries
being reported as document conflict errors. The first occurrence actual
inserts into the database, and only the second occurrence should report a
conflict.
curl -d '{ docs: [ {_id:foo}, {_id,foo} ] }' -H
'Content-Type:application/json' -X POST
http://appadvice.cloudant.com/foo/_bulk_docs
[{id:foo,error:conflict,reason:Document update
conflict.},{id:foo,error:conflict,reason:Document update
conflict.}]
But the database shows that one new document was actually inserted.
Only the second occurrence should report conflict. The first occurrence
should report the _rev property of the newly inserted doc.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1228) Key range error apparently ignores view collation
[
https://issues.apache.org/jira/browse/COUCHDB-1228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13080394#comment-13080394
]
Bob Dionne commented on COUCHDB-1228:
-
Looks like the problem is in couch_httpd_view:warn_on_empty_key_range, it makes
use of couch_view:less_json as the default instead of taking the collation
option into account.
Shouldn't be too hard to fix in the view case but I'm not sure how to handle
the case of _all_docs
Key range error apparently ignores view collation
-
Key: COUCHDB-1228
URL: https://issues.apache.org/jira/browse/COUCHDB-1228
Project: CouchDB
Issue Type: Bug
Components: HTTP Interface
Affects Versions: 1.1
Environment: Debian
Reporter: Victor Nicollet
Fix For: 1.1.1
I have created a view (no reduce function) with options:{collation:raw}
and emit documents with keys A, C and b. Running a request on that view
with no parameters, I get as expected all three documents in order A, C
and b as required by the raw collation (instead of A, b, C for the
default ICU collation).
However, when I run a request with start key B and end key a, I expect
the C document to be returned alone (as B C a) but couchDB
responds:
{ error: query_parse_error, reason: No rows can match your key range,
reverse your start_key and end_key or set descending=true }
This error would make sense if I had been using the default ICU collation,
where B a, but with the raw collation the reverse (B a) is true.
It looks as if the key order warning does not take the view collation into
account.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1163) Document returned by id, but cannot be found by rev
[
https://issues.apache.org/jira/browse/COUCHDB-1163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13034662#comment-13034662
]
Bob Dionne commented on COUCHDB-1163:
-
Clare,
Can you possibly say more about the scenario? Have there been lots of
replications back and forth? Were the deletes done thru bulk edits? I'm trying
to create a small reproducible test.
Bob
Document returned by id, but cannot be found by rev
---
Key: COUCHDB-1163
URL: https://issues.apache.org/jira/browse/COUCHDB-1163
Project: CouchDB
Issue Type: Bug
Components: Database Core
Affects Versions: 1.0.1, 1.0.2
Reporter: James Howe
Priority: Critical
Attachments: Couch logging for jira issue
Somehow, our cluster has developed the following problem on a handful of
documents. Will post reproduction steps if we find them. All properties have
been redacted. All the documents this affects also have attachments, if that
is significant. Once a document is in this situation, it causes conflict
detection, replication and include_docs to behave incorrectly or outright
fail.
GET /database/4cdee83a118ea1cf3050b1d006144d46 returns
{
_id: 4cdee83a118ea1cf3050b1d006144d46,
_rev: 10-df4bf65a6104ea240f100c30d3cb245d,
foo: bar
}
GET /database/4cdee83a118ea1cf3050b1d006144d46?open_revs=all returns
[
{
ok: {
_id: 4cdee83a118ea1cf3050b1d006144d46,
_rev: 10-df4bf65a6104ea240f100c30d3cb245d,
foo: bar
}
},
{
ok: {
_id: 4cdee83a118ea1cf3050b1d006144d46,
_rev: 8-eea5e36daee12acd79a127abf36f7720,
_deleted: true
}
},
{
ok: {
_id: 4cdee83a118ea1cf3050b1d006144d46,
_rev: 9-2cead1e4c813a4f0d10a9bc4aa28bfda,
_deleted: true
}
},
{
ok: {
_id: 4cdee83a118ea1cf3050b1d006144d46,
_rev: 7-c3b44f004660caa496804409089b53d9,
_deleted: true
}
},
{
ok: {
_id: 4cdee83a118ea1cf3050b1d006144d46,
_rev: 6-52e978041bb324d19e01a2ac5a243702,
_deleted: true
}
},
{
ok: {
_id: 4cdee83a118ea1cf3050b1d006144d46,
_rev: 5-761bf28c6989f0fde41bdd5732c33159,
_deleted: true
}
},
{
ok: {
_id: 4cdee83a118ea1cf3050b1d006144d46,
_rev: 4-abb005cf4b2d2dd12880a33af1e7066e,
_deleted: true
}
},
{
ok: {
_id: 4cdee83a118ea1cf3050b1d006144d46,
_rev: 3-233e4624e620ec1c8b66f21a051832f8,
_deleted: true
}
},
{
ok: {
_id: 4cdee83a118ea1cf3050b1d006144d46,
_rev: 10-55f0cdf9dd95ed230b733a2c826c842c,
_deleted: true
}
},
{
ok: {
_id: 4cdee83a118ea1cf3050b1d006144d46,
_rev: 11-264c9d6c249ba2fc9b13df35cb447fd7,
_deleted: true
}
},
{
ok: {
_id: 4cdee83a118ea1cf3050b1d006144d46,
_rev: 9-2cead1e4c813a4f0d10a9bc4aa28bfda,
_deleted: true
}
},
{
ok: {
_id: 4cdee83a118ea1cf3050b1d006144d46,
_rev: 2-9f2df19059d9a460a12740a63a4d95e9,
_deleted: true
}
}
]
GET
/database/4cdee83a118ea1cf3050b1d006144d46?rev=10-df4bf65a6104ea240f100c30d3cb245d
returns
{
error: not_found,
reason: missing
}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1163) Document returned by id, but cannot be found by rev
[
https://issues.apache.org/jira/browse/COUCHDB-1163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13034683#comment-13034683
]
Bob Dionne commented on COUCHDB-1163:
-
Thanks Clare, this helps a lot, I suspect it's the combination of both and the
presence of attachments. I'm hopeful to generate a reproducible case
Document returned by id, but cannot be found by rev
---
Key: COUCHDB-1163
URL: https://issues.apache.org/jira/browse/COUCHDB-1163
Project: CouchDB
Issue Type: Bug
Components: Database Core
Affects Versions: 1.0.1, 1.0.2
Reporter: James Howe
Priority: Critical
Attachments: Couch logging for jira issue
Somehow, our cluster has developed the following problem on a handful of
documents. Will post reproduction steps if we find them. All properties have
been redacted. All the documents this affects also have attachments, if that
is significant. Once a document is in this situation, it causes conflict
detection, replication and include_docs to behave incorrectly or outright
fail.
GET /database/4cdee83a118ea1cf3050b1d006144d46 returns
{
_id: 4cdee83a118ea1cf3050b1d006144d46,
_rev: 10-df4bf65a6104ea240f100c30d3cb245d,
foo: bar
}
GET /database/4cdee83a118ea1cf3050b1d006144d46?open_revs=all returns
[
{
ok: {
_id: 4cdee83a118ea1cf3050b1d006144d46,
_rev: 10-df4bf65a6104ea240f100c30d3cb245d,
foo: bar
}
},
{
ok: {
_id: 4cdee83a118ea1cf3050b1d006144d46,
_rev: 8-eea5e36daee12acd79a127abf36f7720,
_deleted: true
}
},
{
ok: {
_id: 4cdee83a118ea1cf3050b1d006144d46,
_rev: 9-2cead1e4c813a4f0d10a9bc4aa28bfda,
_deleted: true
}
},
{
ok: {
_id: 4cdee83a118ea1cf3050b1d006144d46,
_rev: 7-c3b44f004660caa496804409089b53d9,
_deleted: true
}
},
{
ok: {
_id: 4cdee83a118ea1cf3050b1d006144d46,
_rev: 6-52e978041bb324d19e01a2ac5a243702,
_deleted: true
}
},
{
ok: {
_id: 4cdee83a118ea1cf3050b1d006144d46,
_rev: 5-761bf28c6989f0fde41bdd5732c33159,
_deleted: true
}
},
{
ok: {
_id: 4cdee83a118ea1cf3050b1d006144d46,
_rev: 4-abb005cf4b2d2dd12880a33af1e7066e,
_deleted: true
}
},
{
ok: {
_id: 4cdee83a118ea1cf3050b1d006144d46,
_rev: 3-233e4624e620ec1c8b66f21a051832f8,
_deleted: true
}
},
{
ok: {
_id: 4cdee83a118ea1cf3050b1d006144d46,
_rev: 10-55f0cdf9dd95ed230b733a2c826c842c,
_deleted: true
}
},
{
ok: {
_id: 4cdee83a118ea1cf3050b1d006144d46,
_rev: 11-264c9d6c249ba2fc9b13df35cb447fd7,
_deleted: true
}
},
{
ok: {
_id: 4cdee83a118ea1cf3050b1d006144d46,
_rev: 9-2cead1e4c813a4f0d10a9bc4aa28bfda,
_deleted: true
}
},
{
ok: {
_id: 4cdee83a118ea1cf3050b1d006144d46,
_rev: 2-9f2df19059d9a460a12740a63a4d95e9,
_deleted: true
}
}
]
GET
/database/4cdee83a118ea1cf3050b1d006144d46?rev=10-df4bf65a6104ea240f100c30d3cb245d
returns
{
error: not_found,
reason: missing
}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1004) list_to_existing_atom is too restrictive as used by couch_rep
[ https://issues.apache.org/jira/browse/COUCHDB-1004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13020772#comment-13020772 ] Bob Dionne commented on COUCHDB-1004: - It would nice, and helpful downstream. I think Adam's proposed trivial change is the simplest, use couch_util:to_existing_atom/1 list_to_existing_atom is too restrictive as used by couch_rep - Key: COUCHDB-1004 URL: https://issues.apache.org/jira/browse/COUCHDB-1004 Project: CouchDB Issue Type: Bug Components: Replication Environment: erlang Reporter: Bob Dionne Priority: Minor Fix For: 1.0.3, 1.1, 1.2 Attachments: COUCHDB-1004.patch We'd like to additional information to db_info in BigCouch, such as the Q and N constants for a given database. This causes replication to fail when replicating from BigCouch to CouchDB due to the use of list_to_existing_atom in couch_rep:dbinfo(... The claim is that list_to_atom pollutes the atoms table, however superficial testing indicates this is not the case, list_to_atom when called repeatedly seems to work fine. If this is true then consider reverting list_to_existing_atom back to list_to_atom. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (COUCHDB-462) track conflict count in db_info (was built-in conflicts view)
[
https://issues.apache.org/jira/browse/COUCHDB-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12997225#comment-12997225
]
Bob Dionne commented on COUCHDB-462:
that last sentence should read:
of how couch_key_tree is used might result.
track conflict count in db_info (was built-in conflicts view)
-
Key: COUCHDB-462
URL: https://issues.apache.org/jira/browse/COUCHDB-462
Project: CouchDB
Issue Type: Improvement
Components: HTTP Interface
Reporter: Adam Kocoloski
Fix For: 1.2
Attachments: 462-jan-2.patch, COUCHDB-462-adam-updated.patch,
COUCHDB-462-jan.patch, conflicts_in_db_info.diff, conflicts_in_db_info2.diff,
conflicts_view.diff, whitespace.diff
This patch adds a built-in _conflicts view indexed by document ID that looks
like
GET /dbname/_conflicts
{rows:[
{id:foo, rev:1-1aa8851c9bb2777e11ba56e0bf768649,
conflicts:[1-bdc15320c0850d4ee90ff43d1d298d5d]}
]}
GET /dbname/_conflicts?deleted=true
{rows:[
{id:bar, rev:5-dd31186f5aa11ebd47eb664fb342f1b1,
conflicts:[5-a0efbb1990c961a078dc5308d03b7044],
deleted_conflicts:[3-bdc15320c0850d4ee90ff43d1d298d5d,2-cce334eeeb02d04870e37dac6d33198a]},
{id:baz, rev:2-eec205a9d413992850a6e32678485900, deleted:true,
deleted_conflicts:[2-10009b36e28478b213e04e71c1e08beb]}
]}
As the HTTPd and view layers are a bit outside my specialty I figured I
should ask for a Review before Commit.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (COUCHDB-462) track conflict count in db_info (was built-in conflicts view)
[
https://issues.apache.org/jira/browse/COUCHDB-462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bob Dionne updated COUCHDB-462:
---
Comment: was deleted
(was: that last sentence should read:
of how couch_key_tree is used might result.
)
track conflict count in db_info (was built-in conflicts view)
-
Key: COUCHDB-462
URL: https://issues.apache.org/jira/browse/COUCHDB-462
Project: CouchDB
Issue Type: Improvement
Components: HTTP Interface
Reporter: Adam Kocoloski
Fix For: 1.2
Attachments: 462-jan-2.patch, COUCHDB-462-adam-updated.patch,
COUCHDB-462-jan.patch, conflicts_in_db_info.diff, conflicts_in_db_info2.diff,
conflicts_view.diff, whitespace.diff
This patch adds a built-in _conflicts view indexed by document ID that looks
like
GET /dbname/_conflicts
{rows:[
{id:foo, rev:1-1aa8851c9bb2777e11ba56e0bf768649,
conflicts:[1-bdc15320c0850d4ee90ff43d1d298d5d]}
]}
GET /dbname/_conflicts?deleted=true
{rows:[
{id:bar, rev:5-dd31186f5aa11ebd47eb664fb342f1b1,
conflicts:[5-a0efbb1990c961a078dc5308d03b7044],
deleted_conflicts:[3-bdc15320c0850d4ee90ff43d1d298d5d,2-cce334eeeb02d04870e37dac6d33198a]},
{id:baz, rev:2-eec205a9d413992850a6e32678485900, deleted:true,
deleted_conflicts:[2-10009b36e28478b213e04e71c1e08beb]}
]}
As the HTTPd and view layers are a bit outside my specialty I figured I
should ask for a Review before Commit.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (COUCHDB-462) track conflict count in db_info (was built-in conflicts view)
[
https://issues.apache.org/jira/browse/COUCHDB-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12997217#comment-12997217
]
Bob Dionne commented on COUCHDB-462:
I started reworking couch_key_tree[1] as suggested, to try and tease out some
fold style functions. This should probably be done in COUCHDB-988 but I wanted
to use has_conflicts as one of the tests. I'm able to refactor 4 or 5 so far
but it required two types of fold. I'm calling them foldl and foldr for now (no
relation to lists:foldl and lists:foldr).
So far the results are mixed in that I haven't reduced the amount of code. I'm
sticking to just reworking the functions in couch_key_tree internally and not
changing how they interact with the rest of the code. It could turn out that
some rework
of how couch_key_tree is might result in a simpler couch_key_tree.
[1] https://github.com/bdionne/couchdb/commits/462
track conflict count in db_info (was built-in conflicts view)
-
Key: COUCHDB-462
URL: https://issues.apache.org/jira/browse/COUCHDB-462
Project: CouchDB
Issue Type: Improvement
Components: HTTP Interface
Reporter: Adam Kocoloski
Fix For: 1.2
Attachments: 462-jan-2.patch, COUCHDB-462-adam-updated.patch,
COUCHDB-462-jan.patch, conflicts_in_db_info.diff, conflicts_in_db_info2.diff,
conflicts_view.diff, whitespace.diff
This patch adds a built-in _conflicts view indexed by document ID that looks
like
GET /dbname/_conflicts
{rows:[
{id:foo, rev:1-1aa8851c9bb2777e11ba56e0bf768649,
conflicts:[1-bdc15320c0850d4ee90ff43d1d298d5d]}
]}
GET /dbname/_conflicts?deleted=true
{rows:[
{id:bar, rev:5-dd31186f5aa11ebd47eb664fb342f1b1,
conflicts:[5-a0efbb1990c961a078dc5308d03b7044],
deleted_conflicts:[3-bdc15320c0850d4ee90ff43d1d298d5d,2-cce334eeeb02d04870e37dac6d33198a]},
{id:baz, rev:2-eec205a9d413992850a6e32678485900, deleted:true,
deleted_conflicts:[2-10009b36e28478b213e04e71c1e08beb]}
]}
As the HTTPd and view layers are a bit outside my specialty I figured I
should ask for a Review before Commit.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (COUCHDB-462) track conflict count in db_info (was built-in conflicts view)
[
https://issues.apache.org/jira/browse/COUCHDB-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12993940#comment-12993940
]
Bob Dionne commented on COUCHDB-462:
Looking over the couch_key_tree functions we see a problem with map_leafs. It
takes a function that's expected to have two arguments. However in the one
place it's called, couch_db_updater:handle_call({purge_docs... it looks like
it's being passed a function of one argument.
The one likely test that might exercise this path, purge.js, does not. Is this
dead code or a potential bug? This code has been there for a while
track conflict count in db_info (was built-in conflicts view)
-
Key: COUCHDB-462
URL: https://issues.apache.org/jira/browse/COUCHDB-462
Project: CouchDB
Issue Type: Improvement
Components: HTTP Interface
Reporter: Adam Kocoloski
Fix For: 1.2
Attachments: 462-jan-2.patch, COUCHDB-462-adam-updated.patch,
COUCHDB-462-jan.patch, conflicts_in_db_info.diff, conflicts_in_db_info2.diff,
conflicts_view.diff, whitespace.diff
This patch adds a built-in _conflicts view indexed by document ID that looks
like
GET /dbname/_conflicts
{rows:[
{id:foo, rev:1-1aa8851c9bb2777e11ba56e0bf768649,
conflicts:[1-bdc15320c0850d4ee90ff43d1d298d5d]}
]}
GET /dbname/_conflicts?deleted=true
{rows:[
{id:bar, rev:5-dd31186f5aa11ebd47eb664fb342f1b1,
conflicts:[5-a0efbb1990c961a078dc5308d03b7044],
deleted_conflicts:[3-bdc15320c0850d4ee90ff43d1d298d5d,2-cce334eeeb02d04870e37dac6d33198a]},
{id:baz, rev:2-eec205a9d413992850a6e32678485900, deleted:true,
deleted_conflicts:[2-10009b36e28478b213e04e71c1e08beb]}
]}
As the HTTPd and view layers are a bit outside my specialty I figured I
should ask for a Review before Commit.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (COUCHDB-1065) couch_db_updater calls couch_key_tree incorrectly
[ https://issues.apache.org/jira/browse/COUCHDB-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Dionne updated COUCHDB-1065: Summary: couch_db_updater calls couch_key_tree incorrectly (was: couch_db_updater calls couch_key_tree:) couch_db_updater calls couch_key_tree incorrectly - Key: COUCHDB-1065 URL: https://issues.apache.org/jira/browse/COUCHDB-1065 Project: CouchDB Issue Type: Bug Components: Database Core Reporter: Bob Dionne couch_db_updater at line 123 calls couch_key_tree:map_leafs( fun(RevInfo) -. incorrectly. The function needs to have two arguments. This is readily seen by creating two small dbs with a doc in each having the same id, replicating one to the other and then calling _purge -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (COUCHDB-1065) couch_db_updater calls couch_key_tree:
couch_db_updater calls couch_key_tree: -- Key: COUCHDB-1065 URL: https://issues.apache.org/jira/browse/COUCHDB-1065 Project: CouchDB Issue Type: Bug Components: Database Core Reporter: Bob Dionne couch_db_updater at line 123 calls couch_key_tree:map_leafs( fun(RevInfo) -. incorrectly. The function needs to have two arguments. This is readily seen by creating two small dbs with a doc in each having the same id, replicating one to the other and then calling _purge -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (COUCHDB-462) track conflict count in db_info (was built-in conflicts view)
[
https://issues.apache.org/jira/browse/COUCHDB-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12990524#comment-12990524
]
Bob Dionne commented on COUCHDB-462:
Thanks Adam,
Here's a fix[1] for both issues you mention, and a few more etaps to test it.
Looking over couch_key_tree I agree with you and Randall that many funs have
similar shapes and it could likely use a fold. I'll look at that next.
I'm not sure about using throw in Erlang. I know in Java where try/catch is
enforced I've used it for precisely this case where you want out of a recursion
based on some condition. Erlang has a different approach to failure. I'll look
into it. What' s really needed in these cases is Scheme's
call-with-current-continuation.
Cheers,
Bob
[1] https://github.com/bdionne/couchdb/commit/593bcde1b4.patch
track conflict count in db_info (was built-in conflicts view)
-
Key: COUCHDB-462
URL: https://issues.apache.org/jira/browse/COUCHDB-462
Project: CouchDB
Issue Type: Improvement
Components: HTTP Interface
Reporter: Adam Kocoloski
Fix For: 1.2
Attachments: 462-jan-2.patch, COUCHDB-462-adam-updated.patch,
COUCHDB-462-jan.patch, conflicts_in_db_info.diff, conflicts_in_db_info2.diff,
conflicts_view.diff, whitespace.diff
This patch adds a built-in _conflicts view indexed by document ID that looks
like
GET /dbname/_conflicts
{rows:[
{id:foo, rev:1-1aa8851c9bb2777e11ba56e0bf768649,
conflicts:[1-bdc15320c0850d4ee90ff43d1d298d5d]}
]}
GET /dbname/_conflicts?deleted=true
{rows:[
{id:bar, rev:5-dd31186f5aa11ebd47eb664fb342f1b1,
conflicts:[5-a0efbb1990c961a078dc5308d03b7044],
deleted_conflicts:[3-bdc15320c0850d4ee90ff43d1d298d5d,2-cce334eeeb02d04870e37dac6d33198a]},
{id:baz, rev:2-eec205a9d413992850a6e32678485900, deleted:true,
deleted_conflicts:[2-10009b36e28478b213e04e71c1e08beb]}
]}
As the HTTPd and view layers are a bit outside my specialty I figured I
should ask for a Review before Commit.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (COUCHDB-462) track conflict count in db_info (was built-in conflicts view)
[
https://issues.apache.org/jira/browse/COUCHDB-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12988878#comment-12988878
]
Bob Dionne commented on COUCHDB-462:
Here's a new version that only walks the revision tree until conflicts are
detected.
[1] https://github.com/bdionne/couchdb/tree/couchdb-462
track conflict count in db_info (was built-in conflicts view)
-
Key: COUCHDB-462
URL: https://issues.apache.org/jira/browse/COUCHDB-462
Project: CouchDB
Issue Type: Improvement
Components: HTTP Interface
Reporter: Adam Kocoloski
Fix For: 1.2
Attachments: 462-jan-2.patch, COUCHDB-462-adam-updated.patch,
COUCHDB-462-jan.patch, conflicts_in_db_info.diff, conflicts_in_db_info2.diff,
conflicts_view.diff, whitespace.diff
This patch adds a built-in _conflicts view indexed by document ID that looks
like
GET /dbname/_conflicts
{rows:[
{id:foo, rev:1-1aa8851c9bb2777e11ba56e0bf768649,
conflicts:[1-bdc15320c0850d4ee90ff43d1d298d5d]}
]}
GET /dbname/_conflicts?deleted=true
{rows:[
{id:bar, rev:5-dd31186f5aa11ebd47eb664fb342f1b1,
conflicts:[5-a0efbb1990c961a078dc5308d03b7044],
deleted_conflicts:[3-bdc15320c0850d4ee90ff43d1d298d5d,2-cce334eeeb02d04870e37dac6d33198a]},
{id:baz, rev:2-eec205a9d413992850a6e32678485900, deleted:true,
deleted_conflicts:[2-10009b36e28478b213e04e71c1e08beb]}
]}
As the HTTPd and view layers are a bit outside my specialty I figured I
should ask for a Review before Commit.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (COUCHDB-462) track conflict count in db_info (was built-in conflicts view)
[
https://issues.apache.org/jira/browse/COUCHDB-462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bob Dionne updated COUCHDB-462:
---
Attachment: conflicts_in_db_info.diff
This is patch that adds a conflct_count to db_info as commented on above. This
reflects Paul's suggestions and feedback. I also fixed up etaps that this
breaks. I don't think this change would materially impact performance but I've
not tested that.
This clearly will be very useful in bigcouch for monitoring dbs and so forth.
If folks want to include it I'll be happy to write a test case for it
track conflict count in db_info (was built-in conflicts view)
-
Key: COUCHDB-462
URL: https://issues.apache.org/jira/browse/COUCHDB-462
Project: CouchDB
Issue Type: Improvement
Components: HTTP Interface
Reporter: Adam Kocoloski
Fix For: 1.2
Attachments: 462-jan-2.patch, conflicts_in_db_info.diff,
conflicts_view.diff, COUCHDB-462-adam-updated.patch, COUCHDB-462-jan.patch
This patch adds a built-in _conflicts view indexed by document ID that looks
like
GET /dbname/_conflicts
{rows:[
{id:foo, rev:1-1aa8851c9bb2777e11ba56e0bf768649,
conflicts:[1-bdc15320c0850d4ee90ff43d1d298d5d]}
]}
GET /dbname/_conflicts?deleted=true
{rows:[
{id:bar, rev:5-dd31186f5aa11ebd47eb664fb342f1b1,
conflicts:[5-a0efbb1990c961a078dc5308d03b7044],
deleted_conflicts:[3-bdc15320c0850d4ee90ff43d1d298d5d,2-cce334eeeb02d04870e37dac6d33198a]},
{id:baz, rev:2-eec205a9d413992850a6e32678485900, deleted:true,
deleted_conflicts:[2-10009b36e28478b213e04e71c1e08beb]}
]}
As the HTTPd and view layers are a bit outside my specialty I figured I
should ask for a Review before Commit.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-462) track conflict count in db_info (was built-in conflicts view)
[
https://issues.apache.org/jira/browse/COUCHDB-462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bob Dionne updated COUCHDB-462:
---
Attachment: whitespace.diff
conflicts_in_db_info2.diff
Hi Adam,
this diff depends on changes made in 902 and 988 also. I rebased trunk and
created another.
I also broke out the white space fixes into a separate diff. These already
existed in couchdb. I added them before as I assumed we wanted to fix them.
Cheers,
Bob
track conflict count in db_info (was built-in conflicts view)
-
Key: COUCHDB-462
URL: https://issues.apache.org/jira/browse/COUCHDB-462
Project: CouchDB
Issue Type: Improvement
Components: HTTP Interface
Reporter: Adam Kocoloski
Fix For: 1.2
Attachments: 462-jan-2.patch, conflicts_in_db_info.diff,
conflicts_in_db_info2.diff, conflicts_view.diff,
COUCHDB-462-adam-updated.patch, COUCHDB-462-jan.patch, whitespace.diff
This patch adds a built-in _conflicts view indexed by document ID that looks
like
GET /dbname/_conflicts
{rows:[
{id:foo, rev:1-1aa8851c9bb2777e11ba56e0bf768649,
conflicts:[1-bdc15320c0850d4ee90ff43d1d298d5d]}
]}
GET /dbname/_conflicts?deleted=true
{rows:[
{id:bar, rev:5-dd31186f5aa11ebd47eb664fb342f1b1,
conflicts:[5-a0efbb1990c961a078dc5308d03b7044],
deleted_conflicts:[3-bdc15320c0850d4ee90ff43d1d298d5d,2-cce334eeeb02d04870e37dac6d33198a]},
{id:baz, rev:2-eec205a9d413992850a6e32678485900, deleted:true,
deleted_conflicts:[2-10009b36e28478b213e04e71c1e08beb]}
]}
As the HTTPd and view layers are a bit outside my specialty I figured I
should ask for a Review before Commit.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-462) track conflict count in db_info (was built-in conflicts view)
[
https://issues.apache.org/jira/browse/COUCHDB-462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12986552#action_12986552
]
Bob Dionne commented on COUCHDB-462:
Paul,
good suggestion. Here's a version that does something like that.
Cheers,
Bob
[1] https://github.com/bdionne/couchdb/commit/9c5eb734bb1515
track conflict count in db_info (was built-in conflicts view)
-
Key: COUCHDB-462
URL: https://issues.apache.org/jira/browse/COUCHDB-462
Project: CouchDB
Issue Type: Improvement
Components: HTTP Interface
Reporter: Adam Kocoloski
Fix For: 1.2
Attachments: 462-jan-2.patch, conflicts_view.diff,
COUCHDB-462-adam-updated.patch, COUCHDB-462-jan.patch
This patch adds a built-in _conflicts view indexed by document ID that looks
like
GET /dbname/_conflicts
{rows:[
{id:foo, rev:1-1aa8851c9bb2777e11ba56e0bf768649,
conflicts:[1-bdc15320c0850d4ee90ff43d1d298d5d]}
]}
GET /dbname/_conflicts?deleted=true
{rows:[
{id:bar, rev:5-dd31186f5aa11ebd47eb664fb342f1b1,
conflicts:[5-a0efbb1990c961a078dc5308d03b7044],
deleted_conflicts:[3-bdc15320c0850d4ee90ff43d1d298d5d,2-cce334eeeb02d04870e37dac6d33198a]},
{id:baz, rev:2-eec205a9d413992850a6e32678485900, deleted:true,
deleted_conflicts:[2-10009b36e28478b213e04e71c1e08beb]}
]}
As the HTTPd and view layers are a bit outside my specialty I figured I
should ask for a Review before Commit.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-902) Attachments that have recovered from conflict do not accept attachments.
[ https://issues.apache.org/jira/browse/COUCHDB-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Dionne updated COUCHDB-902: --- Attachment: 902.patch This is the fix proposed by Adam, that we agreed fixes the underlying issue couch_key_tree Attachments that have recovered from conflict do not accept attachments. Key: COUCHDB-902 URL: https://issues.apache.org/jira/browse/COUCHDB-902 Project: CouchDB Issue Type: Bug Components: Database Core Environment: trunk Reporter: Paul Joseph Davis Priority: Critical Fix For: 1.0.2, 1.1 Attachments: 0001-Fix-COUCHDB-902.patch, 902.patch, couchdb-902-test-case.py Apparently if a document has been in a conflict, they will reject requests to add an attachment with a conflict error. I've tracked this down to couch_db_updater.erl line 501, but I'm not too familiar with this part of the code so I figured I'd fill out a ticket in case anyone else can go through this more quickly than me. Sure would be nice if I could attach a file when I create an issue... -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-902) Attachments that have recovered from conflict do not accept attachments.
[ https://issues.apache.org/jira/browse/COUCHDB-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12986066#action_12986066 ] Bob Dionne commented on COUCHDB-902: Klaus, I have two more patches coming, one for COUCHDB-988 that includes new versions of the kt etaps, including the one for this fail. Cheers, Bob Attachments that have recovered from conflict do not accept attachments. Key: COUCHDB-902 URL: https://issues.apache.org/jira/browse/COUCHDB-902 Project: CouchDB Issue Type: Bug Components: Database Core Environment: trunk Reporter: Paul Joseph Davis Priority: Critical Fix For: 1.0.2, 1.1 Attachments: 0001-Fix-COUCHDB-902.patch, 902.patch, couchdb-902-test-case.py Apparently if a document has been in a conflict, they will reject requests to add an attachment with a conflict error. I've tracked this down to couch_db_updater.erl line 501, but I'm not too familiar with this part of the code so I figured I'd fill out a ticket in case anyone else can go through this more quickly than me. Sure would be nice if I could attach a file when I create an issue... -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-902) Attachments that have recovered from conflict do not accept attachments.
[ https://issues.apache.org/jira/browse/COUCHDB-902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Dionne updated COUCHDB-902: --- Attachment: 988.patch more refactoring of couch_key_tree, new etaps, test for couchdb-902 Attachments that have recovered from conflict do not accept attachments. Key: COUCHDB-902 URL: https://issues.apache.org/jira/browse/COUCHDB-902 Project: CouchDB Issue Type: Bug Components: Database Core Environment: trunk Reporter: Paul Joseph Davis Priority: Critical Fix For: 1.0.2, 1.1 Attachments: 0001-Fix-COUCHDB-902.patch, 902.patch, 988.patch, couchdb-902-test-case.py Apparently if a document has been in a conflict, they will reject requests to add an attachment with a conflict error. I've tracked this down to couch_db_updater.erl line 501, but I'm not too familiar with this part of the code so I figured I'd fill out a ticket in case anyone else can go through this more quickly than me. Sure would be nice if I could attach a file when I create an issue... -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-988) Rewrite couch_key_tree.erl
[
https://issues.apache.org/jira/browse/COUCHDB-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bob Dionne updated COUCHDB-988:
---
Attachment: 988.patch
more refactoring of couch_key_tree, new etaps, test for couchdb-902
Rewrite couch_key_tree.erl
--
Key: COUCHDB-988
URL: https://issues.apache.org/jira/browse/COUCHDB-988
Project: CouchDB
Issue Type: Improvement
Components: Database Core
Reporter: Paul Joseph Davis
Priority: Minor
Attachments: 988.patch
The current key tree module is a fairly complicated piece of code with enough
subtlety to fill three romance novels. This ticket is a proposal to rewrite
the current module with an algorithm that will be more easy to reason about.
Here I'll write a brief explanation of the current algorithms, and then a
short proposal for a simpler algorithm.
A key tree is used to represent the edit history of a document. Each node of
the tree represents a particular version of the document. Relations between
nodes represent the order that these edits were applied. For instance, a
simple tree with a single linear path A-B-C means that the edit C was based
on the version B which was in turn based on A. In a world without replication
(and no ability to disable MVCC checks), all histories would be forced to be
linear lists of edits due to constraints imposed by MVCC (ie, new edits must
be based on the current version). However, we have replication, so we must
deal with not so easy cases.
Consider a document in state A. This doc is replicated to a second node. We
then edit the document one each node leaving it in two different states, B
and C. We now have two key trees, A-B and A-C. When we go to replicate a
second time, the key tree must combine these two trees which gives us
A-(B|C). For the astute reader, this is how conflicts are introduced. In
terms of the key tree, we say that we have two leaves (B and C) that are not
deleted. Hence, conflict. To remove a conflict, one of the edits (B or C) can
be deleted, which results in, A-(B|C-D) where D is an edit that is
specially marked with a deleted=true flag.
Also, of note which will help us down the line, remember that there's another
completely different possible mode of operation. Given two couchdb instances,
we could *create* two different docs with the same id. Now we have two
documents with edit histories E and F. Now after replication we have edit
history (E|D) but really, this isn't a tree. Its two roots to two different
trees. Our algorithms still work, we just have to consider multiple trees
when we apply them. Remember this after this next aside.
Hopefully from that brief description of simple operations its fairly
intuitive how we can end up with arbitrarily deep and branching trees due to
distributed edits. Of note here is that this operation parallels the nature
of Git's commit model. Each commit sha1 depends on the parent sha1 creating a
sort of inverted merkle tree.
At this point I'll mention a quick point on the implementation of these
trees. In Erlang, they are implemented as a list of nested tuples with the
roots being the most shallow node. For instance, A-B-C could be represented
as roughly, [{A, [{B, [{C}]}]}] which in english: a single element list of a
two element tuple, where the first element is the key, and the second element
is a single element list of a two tupleturtles.
The reason I make this note on implementation is that it should be apparent
that the depth of this data structure is going to be linearly dependent on
the edit history of a document. As we found out quickly, Erlang also
apparently has some core C functions that are written recursively. As
everyone knows about recursion in C, there is a point where one more turtle
is one too many. Or, in other words, we run out of room on the stack to
allocate more frames.
At this point we had a couple options. Patch Erlang (we did, it was rejected
IIRC). Change our representation (could work, but would break some of the
algorithmic assumptions, especially with how Erlang's SSA works). Or, add
pruning to the revision history. This is what revision stemming is.
Revision stemming in a nutshell means Keep only the last N edits. But then
its a tree, so its actually, keep the last N edits per leaf. The tradeoff
here is that stemming may introduce a conflict by accident. For instance,
Given A-B on one node, replicate to a second, edit it so that its
A-B-C-D. Then applying stemming with an N of 2, we end up with A-B on
node 1, and C-D on node 2. After replication, this turns into (A-B|C-D)
ie, we weren't able to see that C was a descendant of B. This is generally
considered an acceptable tradeoff. You just need to make sure you replicate
more often
[jira] Commented: (COUCHDB-902) Attachments that have recovered from conflict do not accept attachments.
[ https://issues.apache.org/jira/browse/COUCHDB-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12986080#action_12986080 ] Bob Dionne commented on COUCHDB-902: Note: 988.patch is also attached to couchdb-988 where it belongs. It's attached here by mistake but JIRA doesn't seems to allow delete. Attachments that have recovered from conflict do not accept attachments. Key: COUCHDB-902 URL: https://issues.apache.org/jira/browse/COUCHDB-902 Project: CouchDB Issue Type: Bug Components: Database Core Environment: trunk Reporter: Paul Joseph Davis Priority: Critical Fix For: 1.0.2, 1.1 Attachments: 0001-Fix-COUCHDB-902.patch, 902.patch, 988.patch, couchdb-902-test-case.py Apparently if a document has been in a conflict, they will reject requests to add an attachment with a conflict error. I've tracked this down to couch_db_updater.erl line 501, but I'm not too familiar with this part of the code so I figured I'd fill out a ticket in case anyone else can go through this more quickly than me. Sure would be nice if I could attach a file when I create an issue... -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-902) Attachments that have recovered from conflict do not accept attachments.
[ https://issues.apache.org/jira/browse/COUCHDB-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12981461#action_12981461 ] Bob Dionne commented on COUCHDB-902: Klaus et. al., I think we found the crux of the matter. As you suspected, the merge_simple clause involving A B reports conflicts even when no new conflicts are present and the merge is successful. Adding an attachment to an existing conflict situation just exposes the issue. Kocolosk found a nice simple solution[1] that runs all the tests including the new etap we added. See what you think and if you agree we can turn it into a patch. I'm going to add some more tests there while I'm at it. Cheers, Bob [1] https://github.com/bdionne/couchdb/commit/5e424d118615ca14ee5 Attachments that have recovered from conflict do not accept attachments. Key: COUCHDB-902 URL: https://issues.apache.org/jira/browse/COUCHDB-902 Project: CouchDB Issue Type: Bug Components: Database Core Environment: trunk Reporter: Paul Joseph Davis Priority: Critical Attachments: 0001-Fix-COUCHDB-902.patch, couchdb-902-test-case.py Apparently if a document has been in a conflict, they will reject requests to add an attachment with a conflict error. I've tracked this down to couch_db_updater.erl line 501, but I'm not too familiar with this part of the code so I figured I'd fill out a ticket in case anyone else can go through this more quickly than me. Sure would be nice if I could attach a file when I create an issue... -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-1004) list_to_existing_atom is too restrictive as used by couch_rep
[ https://issues.apache.org/jira/browse/COUCHDB-1004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12978343#action_12978343 ] Bob Dionne commented on COUCHDB-1004: - Thanks Filipe, this solves our bigcouch issue and I'd be +1 on having it included in 1.0.2, Bob list_to_existing_atom is too restrictive as used by couch_rep - Key: COUCHDB-1004 URL: https://issues.apache.org/jira/browse/COUCHDB-1004 Project: CouchDB Issue Type: Bug Components: Replication Environment: erlang Reporter: Bob Dionne Priority: Minor Attachments: COUCHDB-1004.patch We'd like to additional information to db_info in BigCouch, such as the Q and N constants for a given database. This causes replication to fail when replicating from BigCouch to CouchDB due to the use of list_to_existing_atom in couch_rep:dbinfo(... The claim is that list_to_atom pollutes the atoms table, however superficial testing indicates this is not the case, list_to_atom when called repeatedly seems to work fine. If this is true then consider reverting list_to_existing_atom back to list_to_atom. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-1004) list_to_existing_atom is too restrictive as used by couch_rep
list_to_existing_atom is too restrictive as used by couch_rep - Key: COUCHDB-1004 URL: https://issues.apache.org/jira/browse/COUCHDB-1004 Project: CouchDB Issue Type: Bug Components: Replication Environment: erlang Reporter: Bob Dionne Priority: Minor We'd like to additional information to db_info in BigCouch, such as the Q and N constants for a given database. This causes replication to fail when replicating from BigCouch to CouchDB due to the use of list_to_existing_atom in couch_rep:dbinfo(... The claim is that list_to_atom pollutes the atoms table, however superficial testing indicates this is not the case, list_to_atom when called repeatedly seems to work fine. If this is true then consider reverting list_to_existing_atom back to list_to_atom. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-968) Duplicated IDs in _all_docs
[
https://issues.apache.org/jira/browse/COUCHDB-968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12965199#action_12965199
]
Bob Dionne commented on COUCHDB-968:
I see that there are possibly a few aspects to this but I think the core issue
is that couchdb simply does not handle exceeding the revs_limit gracefully.
Some more data points:
1. In my test I've set revs_limit=3, have added a single doc and updated it 10
times. The changes[1] run before compaction shows that the revs seem to be out
of sync with the seq no. Perhaps that's ok but with one doc which is the only
one updated they should be in sync? Also 3 of the changes are missing.
2. timing is a factor. I put in a 3 sec sleep between updates and all is hunky
dory. So the problem seems to occur when the replication from db2-db1 finds
that a new update has occurred on db1.
Interestingly one can fix it completely by reversing the direction in the stem
function, but it breaks almost everything else :) I don't quite grok what the
stemming is intended to do. In particular under what conditions will the Tree
exceed the Limit (which in all paths is always revs_limit) and need to be
stemmed? Since it's called immediately after merge_rev_trees, which has a
comment about checking that a previous revision is a leaf node, the real
culprit might be merge_rev_trees as you surmised.
Nice to see you made it safely back to civilization :)
[1] https://gist.github.com/721483
Duplicated IDs in _all_docs
---
Key: COUCHDB-968
URL: https://issues.apache.org/jira/browse/COUCHDB-968
Project: CouchDB
Issue Type: Bug
Components: Database Core
Affects Versions: 0.10.1, 0.10.2, 0.11.1, 0.11.2, 1.0, 1.0.1, 1.0.2
Environment: Ubuntu 10.04.
Reporter: Sebastian Cohnen
Priority: Blocker
We have a database, which is causing serious trouble with compaction and
replication (huge memory and cpu usage, often causing couchdb to crash b/c
all system memory is exhausted). Yesterday we discovered that db/_all_docs is
reporting duplicated IDs (see [1]). Until a few minutes ago we thought that
there are only few duplicates but today I took a closer look and I found 10
IDs which sum up to a total of 922 duplicates. Some of them have only 1
duplicate, others have hundreds.
Some facts about the database in question:
* ~13k documents, with 3-5k revs each
* all duplicated documents are in conflict (with 1 up to 14 conflicts)
* compaction is run on a daily bases
* several thousands updates per hour
* multi-master setup with pull replication from each other
* delayed_commits=false on all nodes
* used couchdb versions 1.0.0 and 1.0.x (*)
Unfortunately the database's contents are confidential and I'm not allowed to
publish it.
[1]: Part of http://localhost:5984/DBNAME/_all_docs
...
{id:9997,key:9997,value:{rev:6096-603c68c1fa90ac3f56cf53771337ac9f}},
{id:,key:,value:{rev:6097-3c873ccf6875ff3c4e2c6fa264c6a180}},
{id:,key:,value:{rev:6097-3c873ccf6875ff3c4e2c6fa264c6a180}},
...
[*]
There were two (old) servers (1.0.0) in production (already having the
replication and compaction issues). Then two servers (1.0.x) were added and
replication was set up to bring them in sync with the old production servers
since the two new servers were meant to replace the old ones (to update
node.js application code among other things).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-968) Duplicated IDs in _all_docs
[
https://issues.apache.org/jira/browse/COUCHDB-968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12964535#action_12964535
]
Bob Dionne commented on COUCHDB-968:
I tried to narrow it further and this seems like the minimum case. compaction
triggers it. It works fine if the replication is only in one direction or if
you don't exceed the revs limit.
Duplicated IDs in _all_docs
---
Key: COUCHDB-968
URL: https://issues.apache.org/jira/browse/COUCHDB-968
Project: CouchDB
Issue Type: Bug
Components: Database Core
Affects Versions: 1.0, 1.0.1, 1.0.2
Environment: Ubuntu 10.04.
Reporter: Sebastian Cohnen
Priority: Blocker
We have a database, which is causing serious trouble with compaction and
replication (huge memory and cpu usage, often causing couchdb to crash b/c
all system memory is exhausted). Yesterday we discovered that db/_all_docs is
reporting duplicated IDs (see [1]). Until a few minutes ago we thought that
there are only few duplicates but today I took a closer look and I found 10
IDs which sum up to a total of 922 duplicates. Some of them have only 1
duplicate, others have hundreds.
Some facts about the database in question:
* ~13k documents, with 3-5k revs each
* all duplicated documents are in conflict (with 1 up to 14 conflicts)
* compaction is run on a daily bases
* several thousands updates per hour
* multi-master setup with pull replication from each other
* delayed_commits=false on all nodes
* used couchdb versions 1.0.0 and 1.0.x (*)
Unfortunately the database's contents are confidential and I'm not allowed to
publish it.
[1]: Part of http://localhost:5984/DBNAME/_all_docs
...
{id:9997,key:9997,value:{rev:6096-603c68c1fa90ac3f56cf53771337ac9f}},
{id:,key:,value:{rev:6097-3c873ccf6875ff3c4e2c6fa264c6a180}},
{id:,key:,value:{rev:6097-3c873ccf6875ff3c4e2c6fa264c6a180}},
...
[*]
There were two (old) servers (1.0.0) in production (already having the
replication and compaction issues). Then two servers (1.0.x) were added and
replication was set up to bring them in sync with the old production servers
since the two new servers were meant to replace the old ones (to update
node.js application code among other things).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-968) Duplicated IDs in _all_docs
[
https://issues.apache.org/jira/browse/COUCHDB-968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12964457#action_12964457
]
Bob Dionne commented on COUCHDB-968:
My first case also involved compaction. Here's a simpler case:
1. create db1 and db2
2. start continuous replication between them in both directions
3. add a doc to db1
4. update the same doc 1005 times (where 1000 is the max revs, it doesn't fail
at 1001 as you would suspect)
5. compact db1
It will have 3 dups of doc1 in it.
This is against trunk with the default config. #4 seems to be a good clue that
it involves the rev limit, replication and compaction
Duplicated IDs in _all_docs
---
Key: COUCHDB-968
URL: https://issues.apache.org/jira/browse/COUCHDB-968
Project: CouchDB
Issue Type: Bug
Components: Database Core
Affects Versions: 1.0, 1.0.1, 1.0.2
Environment: Ubuntu 10.04.
Reporter: Sebastian Cohnen
We have a database, which is causing serious trouble with compaction and
replication (huge memory and cpu usage, often causing couchdb to crash b/c
all system memory is exhausted). Yesterday we discovered that db/_all_docs is
reporting duplicated IDs (see [1]). Until a few minutes ago we thought that
there are only few duplicates but today I took a closer look and I found 10
IDs which sum up to a total of 922 duplicates. Some of them have only 1
duplicate, others have hundreds.
Some facts about the database in question:
* ~13k documents, with 3-5k revs each
* all duplicated documents are in conflict (with 1 up to 14 conflicts)
* compaction is run on a daily bases
* several thousands updates per hour
* multi-master setup with pull replication from each other
* delayed_commits=false on all nodes
* used couchdb versions 1.0.0 and 1.0.x (*)
Unfortunately the database's contents are confidential and I'm not allowed to
publish it.
[1]: Part of http://localhost:5984/DBNAME/_all_docs
...
{id:9997,key:9997,value:{rev:6096-603c68c1fa90ac3f56cf53771337ac9f}},
{id:,key:,value:{rev:6097-3c873ccf6875ff3c4e2c6fa264c6a180}},
{id:,key:,value:{rev:6097-3c873ccf6875ff3c4e2c6fa264c6a180}},
...
[*]
There were two (old) servers (1.0.0) in production (already having the
replication and compaction issues). Then two servers (1.0.x) were added and
replication was set up to bring them in sync with the old production servers
since the two new servers were meant to replace the old ones (to update
node.js application code among other things).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-903) semantics of COPY call does not agree with the WIKI
semantics of COPY call does not agree with the WIKI
---
Key: COUCHDB-903
URL: https://issues.apache.org/jira/browse/COUCHDB-903
Project: CouchDB
Issue Type: Bug
Components: Database Core
Affects Versions: 1.0.1
Environment: emacs
Reporter: Bob Dionne
Priority: Minor
according to the WIKI, the copy command:
http://wiki.apache.org/couchdb/HTTP_Document_API#COPY
should return:
{ok:true,id:some_other_doc,rev:355068078}
It's missing the ok:true portion. Here's a patch, this is against 1.0.1 but
I'm sure it's also in trunk. The patch also includes a failing test. The patch
function also affects bulk save and there's a change in couch.js to check this.
jquey.couch.js might also need some tweaking
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-903) semantics of COPY call does not agree with the WIKI
[
https://issues.apache.org/jira/browse/COUCHDB-903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bob Dionne updated COUCHDB-903:
---
Attachment: doc_copy.patch
fix
semantics of COPY call does not agree with the WIKI
---
Key: COUCHDB-903
URL: https://issues.apache.org/jira/browse/COUCHDB-903
Project: CouchDB
Issue Type: Bug
Components: Database Core
Affects Versions: 1.0.1
Environment: emacs
Reporter: Bob Dionne
Priority: Minor
Attachments: doc_copy.patch
according to the WIKI, the copy command:
http://wiki.apache.org/couchdb/HTTP_Document_API#COPY
should return:
{ok:true,id:some_other_doc,rev:355068078}
It's missing the ok:true portion. Here's a patch, this is against 1.0.1
but I'm sure it's also in trunk. The patch also includes a failing test. The
patch function also affects bulk save and there's a change in couch.js to
check this. jquey.couch.js might also need some tweaking
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-862) replicator.js fails to run in standalone js tests
replicator.js fails to run in standalone js tests - Key: COUCHDB-862 URL: https://issues.apache.org/jira/browse/COUCHDB-862 Project: CouchDB Issue Type: Bug Components: Test Suite Environment: OS X Reporter: Bob Dionne Priority: Trivial replicator.js uses some jQuery functions such as jQuery.each and inArray, which causes it to fail when tests are run outside the browser -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-862) replicator.js fails to run in standalone js tests
[ https://issues.apache.org/jira/browse/COUCHDB-862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Dionne updated COUCHDB-862: --- Attachment: replicator.js.patch fixes replicator.js in standalone mode replicator.js fails to run in standalone js tests - Key: COUCHDB-862 URL: https://issues.apache.org/jira/browse/COUCHDB-862 Project: CouchDB Issue Type: Bug Components: Test Suite Environment: OS X Reporter: Bob Dionne Priority: Trivial Attachments: replicator.js.patch replicator.js uses some jQuery functions such as jQuery.each and inArray, which causes it to fail when tests are run outside the browser -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-862) replicator.js fails to run in standalone js tests
[ https://issues.apache.org/jira/browse/COUCHDB-862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Dionne updated COUCHDB-862: --- Attachment: replication.js.patch clean up sloppy JS replicator.js fails to run in standalone js tests - Key: COUCHDB-862 URL: https://issues.apache.org/jira/browse/COUCHDB-862 Project: CouchDB Issue Type: Bug Components: Test Suite Environment: OS X Reporter: Bob Dionne Priority: Trivial Attachments: replication.js.patch, replicator.js.patch replicator.js uses some jQuery functions such as jQuery.each and inArray, which causes it to fail when tests are run outside the browser -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-581) dead code in couch_db.erl
[ https://issues.apache.org/jira/browse/COUCHDB-581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Dionne updated COUCHDB-581: --- Attachment: couch_db.erl.patch removes unused function dead code in couch_db.erl - Key: COUCHDB-581 URL: https://issues.apache.org/jira/browse/COUCHDB-581 Project: CouchDB Issue Type: Improvement Components: Database Core Environment: OS X 10.6 Reporter: Bob Dionne Priority: Trivial Fix For: 0.11 Attachments: couch_db.erl.patch couch_db:register_update_notifier seems to be no longer used, according to distel:who_calls removing it has not impact on any tests, etaps or Futon -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-581) dead code in couch_db.erl
dead code in couch_db.erl - Key: COUCHDB-581 URL: https://issues.apache.org/jira/browse/COUCHDB-581 Project: CouchDB Issue Type: Improvement Components: Database Core Environment: OS X 10.6 Reporter: Bob Dionne Priority: Trivial Fix For: 0.11 Attachments: couch_db.erl.patch couch_db:register_update_notifier seems to be no longer used, according to distel:who_calls removing it has not impact on any tests, etaps or Futon -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-570) CouchDB code improvements
[
https://issues.apache.org/jira/browse/COUCHDB-570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12778967#action_12778967
]
Bob Dionne commented on COUCHDB-570:
Interesting, I'd like to read up on this tool also.
couch_btree:chunkify(Bt...) has a variable, Bt, that is never used and can be
discarded. How hard would it be for tidier to detect that?
CouchDB code improvements
-
Key: COUCHDB-570
URL: https://issues.apache.org/jira/browse/COUCHDB-570
Project: CouchDB
Issue Type: Question
Components: Database Core
Affects Versions: 0.11
Reporter: Kostis Sagonas
Priority: Minor
Fix For: 0.11
Attachments: DIFFS
Finally I've found the time today to sign up to jira (username: kostis).
I have a whole bunch of changes to the CouchDB code which were produced by
'tidier' (Jan Lehnard knows about this).
I am wondering what's the best way of contributing them? Create an issue and
attach a patch/diff file? (I've done so)
Or maybe you prefer some other way?
Also, I've noticed that function couch_httpd:find_partial/4 returns either
'not_found' or {partial, N, K} but the place where this result is used
(couch_httpd:673 or so) checks for matching against 'not_found' or {partial,
Skip}. Somebody needs to fix that.
Best,
Kostis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-504) HTTP/1.1 keep alive fails with AB
[
https://issues.apache.org/jira/browse/COUCHDB-504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bob Dionne updated COUCHDB-504:
---
Attachment: couch_httpd_hack.patch
this patch to send_json puts Connection : Keep-Alive into the response
header, if it exists in the request. This isn't likely the best solution but it
solves the problem and all the tests run.
HTTP/1.1 keep alive fails with AB
-
Key: COUCHDB-504
URL: https://issues.apache.org/jira/browse/COUCHDB-504
Project: CouchDB
Issue Type: Bug
Components: HTTP Interface
Affects Versions: 0.10
Environment: Trunk
Reporter: Paul Joseph Davis
Assignee: Paul Joseph Davis
Priority: Blocker
Attachments: couch_httpd_hack.patch
Reported on dev@
On Tue, Sep 15, 2009 at 3:33 PM, thomas hallaran thalla...@gmail.com wrote:
Hi all,
We are using couchdb in production and have come across what appears to be a
bug in the HTTP implementation. Apparently http keep alive requests are not
correctly supported by couchdb.
This problem has been verified for both couchdb .10 a running on ubuntu
with erlang 13.b1 and couchdb .91 running on osx. We have also reproduced
it with multiple http clients, the apache hc java client, curl and ab.
To Reproduce: ab -v 4 -k -n1
http://localhost:5984/{COUCHDB_GET_ENDPOINT}http://localhost:5984/%7BCOUCHDB_GET_ENDPOINT%7Dtimes
out. If you add more requests for ab to do (-n4 or something) you will
see that rather than reusing the connection, each keepalive request times
out and then the connection is reopened and the request made.
And I've verified:
$ svn up
#snipped
Updated to revision 815457.
$ ./bootstrap ./configure make dev
#snipped
$ ./utils/run
Apache CouchDB 0.11.0a815457 (LogLevel=info) is starting.
Apache CouchDB has started. Time to relax.
# In second terminal
$ curl -X PUT http://127.0.0.1:5984/test_fu
{ok:true}
$ curl -X PUT -d '{stuff: here}' http://127.0.0.1:5984/test_fu/beer
{ok:true,id:beer,rev:1-9782357dd8b8d18d94843686e510ec92}
$ ab -v 4 -k -n1 http://127.0.0.1:5984/test_fu/beer
This is ApacheBench, Version 2.3 $Revision: 655654 $
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 127.0.0.1 (be patient)...INFO: POST header ==
---
GET /test_fu/beer HTTP/1.0
Connection: Keep-Alive
Host: 127.0.0.1:5984
User-Agent: ApacheBench/2.3
Accept: */*
---
LOG: header received:
HTTP/1.0 200 OK
Server: CouchDB/0.11.0a815457 (Erlang OTP/R13B)
Etag: 1-9782357dd8b8d18d94843686e510ec92
Date: Tue, 15 Sep 2009 19:44:36 GMT
Content-Type: text/plain;charset=utf-8
Content-Length: 74
Cache-Control: must-revalidate
{_id:beer,_rev:1-9782357dd8b8d18d94843686e510ec92,stuff:here}
LOG: Response code = 200
apr_poll: The timeout specified has expired (70007)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-503) update etap to reflect new code in couch_btree
[ https://issues.apache.org/jira/browse/COUCHDB-503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Dionne updated COUCHDB-503: --- Attachment: 021-btree-reductions.t.patch update etap to reflect new code in couch_btree -- Key: COUCHDB-503 URL: https://issues.apache.org/jira/browse/COUCHDB-503 Project: CouchDB Issue Type: Test Components: Test Suite Reporter: Bob Dionne Priority: Trivial Attachments: 021-btree-reductions.t.patch updated etap/021-btree-reductions.t to reflect new code in couch_btree -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-493) tweaked etaps to support new startup
tweaked etaps to support new startup Key: COUCHDB-493 URL: https://issues.apache.org/jira/browse/COUCHDB-493 Project: CouchDB Issue Type: Bug Components: Test Suite Reporter: Bob Dionne Priority: Minor couch_server:start(... has gone away, changed etaps to use couch_server_sup:start_link -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-477) Add database uuid's
[ https://issues.apache.org/jira/browse/COUCHDB-477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Dionne updated COUCHDB-477: --- Comment: was deleted (was: I tested this all against a clean checkout of trunk and it looks good. The new algorithm is faster on inserts[1] but interestingly for single inserts is makes for a larger db pre-compact. After compaction the db is smaller by a factor of 3 I thought we leave the new sequentially random as the default? Paul, is the concern just what's advertised to users? Algorithms based on the system clock I think can be problematic as they assume all machines have the correct time. [1] http://gist.github.com/170982) Add database uuid's --- Key: COUCHDB-477 URL: https://issues.apache.org/jira/browse/COUCHDB-477 Project: CouchDB Issue Type: New Feature Reporter: Robert Newson Attachments: 0001-add-uuid-to-database-on-creation-return-it-in-db_in.patch Add a uuid to db_header to distinguish different databases that have the same name (for example, by deleting and creating the same named database). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-465) Produce sequential, but unique, document id's
[ https://issues.apache.org/jira/browse/COUCHDB-465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12745438#action_12745438 ] Bob Dionne commented on COUCHDB-465: Robert, sorry for the confusion, there are 4 tests there, tests 2 and 4 use the old random and the new sequential random codes respectively. So I'm comparing: precompact: 50020852 postcompact: 6099042 to precompact: 52517614 postcompact: 2506850 That's why I used the phrase sequentially random. Sequential in these tests (1 and 3) means 1,2,3 I agree about the batching, the diffs are almost noise with a batch size of 1000, which is what the compactor uses. Cheers, Bob Produce sequential, but unique, document id's - Key: COUCHDB-465 URL: https://issues.apache.org/jira/browse/COUCHDB-465 Project: CouchDB Issue Type: Improvement Reporter: Robert Newson Attachments: couch_uuids.patch, uuid_generator.patch Currently, if the client does not specify an id (POST'ing a single document or using _bulk_docs) a random 16 byte value is created. This kind of key is particularly brutal on b+tree updates and the append-only nature of couchdb files. Attached is a patch to change this to a two-part identifier. The first part is a random 12 byte value and the remainder is a counter. The random prefix is rerandomized when the counter reaches its maximum. The rollover in the patch is at 16 million but can obviously be changed. The upshot is that the b+tree is updated in a better fashion, which should lead to performance benefits. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-465) Produce sequential, but unique, document id's
[ https://issues.apache.org/jira/browse/COUCHDB-465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Dionne updated COUCHDB-465: --- Attachment: 041-uuid-gen.t 041-uuid-gen-utc.ini 041-uuid-gen-seq.ini etaps to test the new couch_uuids work of rnewson and davisp. Produce sequential, but unique, document id's - Key: COUCHDB-465 URL: https://issues.apache.org/jira/browse/COUCHDB-465 Project: CouchDB Issue Type: Improvement Reporter: Robert Newson Attachments: 041-uuid-gen-seq.ini, 041-uuid-gen-utc.ini, 041-uuid-gen.t, couch_uuids.patch, uuid_generator.patch Currently, if the client does not specify an id (POST'ing a single document or using _bulk_docs) a random 16 byte value is created. This kind of key is particularly brutal on b+tree updates and the append-only nature of couchdb files. Attached is a patch to change this to a two-part identifier. The first part is a random 12 byte value and the remainder is a counter. The random prefix is rerandomized when the counter reaches its maximum. The rollover in the patch is at 16 million but can obviously be changed. The upshot is that the b+tree is updated in a better fashion, which should lead to performance benefits. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-396) Fixing weirdness in couch_stats_aggregator.erl
[ https://issues.apache.org/jira/browse/COUCHDB-396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12744154#action_12744154 ] Bob Dionne commented on COUCHDB-396: we tested using conditional compilation to disable stats collection. It made no measurable difference in either the Futon test suite or any local load tests. Just updating ets tables appears to be quite cheap. Fixing weirdness in couch_stats_aggregator.erl -- Key: COUCHDB-396 URL: https://issues.apache.org/jira/browse/COUCHDB-396 Project: CouchDB Issue Type: Improvement Components: Database Core, HTTP Interface Affects Versions: 0.10 Environment: trunk Reporter: Paul Joseph Davis Assignee: Paul Joseph Davis Fix For: 0.10 Attachments: couch_stats.patch, couchdb_stats_aggregator.patch Looking at adding unit tests to the couchdb_stats_aggregator module the other day I realized it was doing some odd calculations. This is a fairly non-trivial patch so I figured that I'd put in JIRA and get feed back before applying. This patch does everything the old version does afaict, but I'll be adding tests before I consider it complete. List of major changes: * The old behavior for stats was to integrate incoming values for a time period and then reset the values and start integrating again. That seemed a bit odd so I rewrote things to keep the average and standard deviation for the last N seconds with approximately 1 sample per second. * Changed request timing calculations [note below] * Sample periods are configurable in the .ini file. Sample periods of 0 are a special case and integrate all values from couchdb boot up. * Sample descriptions are in the configuration files now. * You can request different time periods for the root stats end point. * Added a sum to the list of statistics * Simplified some of the external API The biggest change is in how time for requests are calculated. AFAICT, the old way was accumulating request timings in the stats collector and just adding new values as clock ticks went by as everything else does which makes sense in the case of resetting counters every time period. In the new way I'm keeping a list of the samples in the last time period and when I get a clock tick part of the update is to remove the samples that have passed out of the time period. For a variable like request_time this would lead to unbounded storage. The new method is calculating the average time of all requests in a single clock tick (1s). One thing this loses is when you start having lots of variability in a single clock tick. Ie, your average request time is 100ms, but 10% of your requests are taking 500ms. I've read of people doing the averaging trick but also storing quantile information as well [1]. There are also algorithms for doing single pass quantile estimation and the like so its possible to do those things in O(N) time. The issue with quantiles is that it'd start breaking the logic of how the collector and aggregators are setup. As it is now, there's basically a one event - one stat constraint. For the time being I went without quartiles to minimize the impact of the patch. This code will also be on github [3] as I add patches. [1] http://code.flickr.com/blog/2008/10/27/counting-timing/ [2] http://www.slamb.org/svn/repos/trunk/projects/loadtest/benchtools/stats.py (See the QuantileEstimator class) [3] http://github.com/davisp/couchdb/tree/stats-patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-423) couch_file misses an edge case
couch_file misses an edge case -- Key: COUCHDB-423 URL: https://issues.apache.org/jira/browse/COUCHDB-423 Project: CouchDB Issue Type: Bug Components: Database Core Reporter: Bob Dionne Priority: Minor pread_binary reads the first 20 bytes. This causes a simple append_binary followed by pread_binary to fail, as in 010-file-basics.t test case. This case may never occur because of couchdb's file format -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-423) couch_file misses an edge case
[ https://issues.apache.org/jira/browse/COUCHDB-423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Dionne updated COUCHDB-423: --- Attachment: couch_file.diff attached patch fixes the issue. couch_file misses an edge case -- Key: COUCHDB-423 URL: https://issues.apache.org/jira/browse/COUCHDB-423 Project: CouchDB Issue Type: Bug Components: Database Core Reporter: Bob Dionne Priority: Minor Attachments: couch_file.diff pread_binary reads the first 20 bytes. This causes a simple append_binary followed by pread_binary to fail, as in 010-file-basics.t test case. This case may never occur because of couchdb's file format -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-424) fixed minor issues in etap tests dues to code evolution
[ https://issues.apache.org/jira/browse/COUCHDB-424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Dionne updated COUCHDB-424: --- Attachment: etaps.diff patch for three etap tests fixed minor issues in etap tests dues to code evolution --- Key: COUCHDB-424 URL: https://issues.apache.org/jira/browse/COUCHDB-424 Project: CouchDB Issue Type: Improvement Components: Test Suite Reporter: Bob Dionne Attachments: etaps.diff fixed etap tests to accomodate changes in couch_config, couch_stream, etc.. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-424) fixed minor issues in etap tests dues to code evolution
[ https://issues.apache.org/jira/browse/COUCHDB-424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Dionne updated COUCHDB-424: --- Attachment: etaps.diff this fixes the rest of the etaps and the permissions on the replication test fixed minor issues in etap tests dues to code evolution --- Key: COUCHDB-424 URL: https://issues.apache.org/jira/browse/COUCHDB-424 Project: CouchDB Issue Type: Improvement Components: Test Suite Reporter: Bob Dionne Attachments: etaps.diff, etaps.diff fixed etap tests to accomodate changes in couch_config, couch_stream, etc.. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-381) it would be nice to have an option in bin/couchdb to pass -sname node_name to erlang at startup
it would be nice to have an option in bin/couchdb to pass -sname node_name to erlang at startup - Key: COUCHDB-381 URL: https://issues.apache.org/jira/browse/COUCHDB-381 Project: CouchDB Issue Type: Improvement Components: Build System Affects Versions: 0.10 Reporter: Bob Dionne Priority: Minor attached is a patch to bin/couchdb.tpl.in that accomplishes this -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-381) it would be nice to have an option in bin/couchdb to pass -sname node_name to erlang at startup
[ https://issues.apache.org/jira/browse/COUCHDB-381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Dionne updated COUCHDB-381: --- Attachment: couchdb.tpl.in.patch adds -sname node_name to erlang shell using a -S option it would be nice to have an option in bin/couchdb to pass -sname node_name to erlang at startup - Key: COUCHDB-381 URL: https://issues.apache.org/jira/browse/COUCHDB-381 Project: CouchDB Issue Type: Improvement Components: Build System Affects Versions: 0.10 Reporter: Bob Dionne Priority: Minor Attachments: couchdb.tpl.in.patch attached is a patch to bin/couchdb.tpl.in that accomplishes this -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-214) Futon doesn't handle slashes in doc store names
Futon doesn't handle slashes in doc store names --- Key: COUCHDB-214 URL: https://issues.apache.org/jira/browse/COUCHDB-214 Project: CouchDB Issue Type: Bug Components: Administration Console Environment: OS X, Safari, Firefox Reporter: Bob Dionne Priority: Minor create a doc store with names foo/bar and then save a custom query for it -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-215) errors when creating and deleting multiple databases
errors when creating and deleting multiple databases Key: COUCHDB-215 URL: https://issues.apache.org/jira/browse/COUCHDB-215 Project: CouchDB Issue Type: Bug Components: Database Core Environment: OS X 10.4 Erlang latest, CouchDB trunk Reporter: Bob Dionne creating multiple databases and then deleting them causes couchdb to start throwing exceptions ( http://gist.github.com/49063 ) A cursory debugging session indicates that should_close in couch_file never returns true, the monitors list always contains the pid of the next process. Repeated use of the server in this way eventually makes it unusable. The following JS, http://gist.github.com/48465, will also exhibit the issue using the Futon tests. It can appear with as few as 60 dbs depending on the client and level of concurrency The database names do have slashes in them but this seems irrelevant on the face of it. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-212) tests fail running standalone with runner
[ https://issues.apache.org/jira/browse/COUCHDB-212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Dionne updated COUCHDB-212: --- Attachment: couch_js.c.diff PUT requests now require file size be sent to couchdb for the new streaming stuff tests fail running standalone with runner - Key: COUCHDB-212 URL: https://issues.apache.org/jira/browse/COUCHDB-212 Project: CouchDB Issue Type: Improvement Components: Infrastructure Environment: OS X Reporter: Bob Dionne Priority: Trivial Attachments: couch_js.c.diff, couch_js.c.diff some of the urls in the tests are encoded to escape slashes. This causes the curl library to return 301 error codes and dbs fail to open, etc.. tweaked couch_js.c to add CURLOPT_FOLLOWLOCATION -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-212) tests fail running standalone with runner
tests fail running standalone with runner - Key: COUCHDB-212 URL: https://issues.apache.org/jira/browse/COUCHDB-212 Project: CouchDB Issue Type: Improvement Components: Infrastructure Environment: OS X Reporter: Bob Dionne Priority: Trivial some of the urls in the tests are encoded to escape slashes. This causes the curl library to return 301 error codes and dbs fail to open, etc.. tweaked couch_js.c to add CURLOPT_FOLLOWLOCATION -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-212) tests fail running standalone with runner
[ https://issues.apache.org/jira/browse/COUCHDB-212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Dionne updated COUCHDB-212: --- Attachment: couch_js.c.diff added followlocation option to handle 301s tests fail running standalone with runner - Key: COUCHDB-212 URL: https://issues.apache.org/jira/browse/COUCHDB-212 Project: CouchDB Issue Type: Improvement Components: Infrastructure Environment: OS X Reporter: Bob Dionne Priority: Trivial Attachments: couch_js.c.diff some of the urls in the tests are encoded to escape slashes. This causes the curl library to return 301 error codes and dbs fail to open, etc.. tweaked couch_js.c to add CURLOPT_FOLLOWLOCATION -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (COUCHDB-203) added total run time to the tests
added total run time to the tests - Key: COUCHDB-203 URL: https://issues.apache.org/jira/browse/COUCHDB-203 Project: CouchDB Issue Type: Improvement Components: Administration Console Reporter: Bob Dionne Priority: Trivial added running total to footer. It provides a quick indication whether the test are faster overall or not -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
