[
https://issues.apache.org/jira/browse/COUCHDB-487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748362#action_12748362
]
Adam Kocoloski commented on COUCHDB-487:
----------------------------------------
I don't think we want to be that aggressive about blocking incoming writes.
This algorithm would cause a large DB to be unavailable for writes for several
minutes, even if the write load is not so large as to prevent compaction from
finishing without blocking. The reason is that each compaction iteration works
with a single MVCC snapshot; any updates to the DB that occur during that
iteration are compacted in the next pass.
I've seen several cases where compaction finishes in 3-4 passes, with the first
one taking several hours, the next one several minutes, the next one several
seconds, and then it's done.
I think if we're going to block writes we should only do it when we have some
expectation that the non-blocking approach isn't working; e.g. we could track
(update_seq - compact_update_seq) and if that difference grows between
iterations, block until compaction is finished.
Thanks for kicking off this ticket with some code, though!
> Pause write requests to allow compaction to complete
> ----------------------------------------------------
>
> Key: COUCHDB-487
> URL: https://issues.apache.org/jira/browse/COUCHDB-487
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Reporter: Robert Newson
> Attachments: block_writes_after_end_of_compaction.patch
>
>
> Continuous writes will currently indefinitely prevent the compaction process
> from flipping over to the .compact file.
> Here's a small patch that adds a new db flag called 'write_available' which
> becomes false the first time compaction completes but fails to flip over
> because of concurrent writes. Subsequent calls to update_docs then sleep for
> 1/2 a second and throw retry.
> I ran 'ab -p json -n 1000000 http://localhost:5984/db1' in one window.
> Without this patch, I get a long sequence of;
> [info] [<0.64.0>] Compaction file still behind main file (update seq=1848.
> compact update seq=1845). Retrying.
> and compaction never completes.
> With the patch, I get this;
> [info] [<0.369.0>] 127.0.0.1 - - 'POST' /db1 201
> [info] [<0.65.0>] Compaction file still behind main file (update seq=2140.
> compact update seq=2092). Retrying.
> [info] [<0.65.0>] Blocking writes to complete compaction.
> [info] [<0.65.0>] Compaction for db "db1" completed.
> [info] [<0.370.0>] 127.0.0.1 - - 'POST' /db1 201
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
