[ https://issues.apache.org/jira/browse/COUCHDB-968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966571#action_12966571 ]
Adam Kocoloski commented on COUCHDB-968: ---------------------------------------- Thinking about how to repair DBs that have these dupes in them. The following patch should cover the 80% case, but to be really bulletproof I think the compaction also needs to run in Retry = true mode (i.e. the mode that causes the compactor to check the .compact file for old data about a document before saving). I mused in IRC about exposing the Retry flag to the end user, but maybe it doesn't make sense to leak those implementation details for a single bug. diff --git a/src/couchdb/couch_db_updater.erl b/src/couchdb/couch_db_updater.erl index e5c6019..caa46af 100644 --- a/src/couchdb/couch_db_updater.erl +++ b/src/couchdb/couch_db_updater.erl @@ -775,7 +775,10 @@ copy_rev_tree_attachments(SrcDb, DestFd, Tree) -> end, Tree). -copy_docs(Db, #db{fd=DestFd}=NewDb, InfoBySeq, Retry) -> +copy_docs(Db, #db{fd=DestFd}=NewDb, InfoBySeq0, Retry) -> + % COUCHDB-968, make sure we prune duplicates during compaction + InfoBySeq = lists:usort(fun(#doc_info{id=A}, #doc_info{id=B}) -> A =< B end, + InfoBySeq0), Ids = [Id || #doc_info{id=Id} <- InfoBySeq], LookupResults = couch_btree:lookup(Db#db.fulldocinfo_by_id_btree, Ids), > Duplicated IDs in _all_docs > --------------------------- > > Key: COUCHDB-968 > URL: https://issues.apache.org/jira/browse/COUCHDB-968 > Project: CouchDB > Issue Type: Bug > Components: Database Core > Affects Versions: 0.10.1, 0.10.2, 0.11.1, 0.11.2, 1.0, 1.0.1, 1.0.2 > Environment: any > Reporter: Sebastian Cohnen > Priority: Blocker > > We have a database, which is causing serious trouble with compaction and > replication (huge memory and cpu usage, often causing couchdb to crash b/c > all system memory is exhausted). Yesterday we discovered that db/_all_docs is > reporting duplicated IDs (see [1]). Until a few minutes ago we thought that > there are only few duplicates but today I took a closer look and I found 10 > IDs which sum up to a total of 922 duplicates. Some of them have only 1 > duplicate, others have hundreds. > Some facts about the database in question: > * ~13k documents, with 3-5k revs each > * all duplicated documents are in conflict (with 1 up to 14 conflicts) > * compaction is run on a daily bases > * several thousands updates per hour > * multi-master setup with pull replication from each other > * delayed_commits=false on all nodes > * used couchdb versions 1.0.0 and 1.0.x (*) > Unfortunately the database's contents are confidential and I'm not allowed to > publish it. > [1]: Part of http://localhost:5984/DBNAME/_all_docs > ... > {"id":"9997","key":"9997","value":{"rev":"6096-603c68c1fa90ac3f56cf53771337ac9f"}}, > {"id":"9999","key":"9999","value":{"rev":"6097-3c873ccf6875ff3c4e2c6fa264c6a180"}}, > {"id":"9999","key":"9999","value":{"rev":"6097-3c873ccf6875ff3c4e2c6fa264c6a180"}}, > ... > [*] > There were two (old) servers (1.0.0) in production (already having the > replication and compaction issues). Then two servers (1.0.x) were added and > replication was set up to bring them in sync with the old production servers > since the two new servers were meant to replace the old ones (to update > node.js application code among other things). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.