[
https://issues.apache.org/jira/browse/CASSANDRA-8639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15034214#comment-15034214
]
Ariel Weisberg commented on CASSANDRA-8639:
-------------------------------------------
bq. This isn't related to the ticket but maybe we should fix it as well; I
don't see anyplace we wait for the replay futures to complete before we finish
recover().
Both 2.1 code and your patch will exit early before the futures have all
finished. It looks like the old version only waited when there were more than
max outstanding mutations. Which is also wrong and racy. We should always wait
for the queue to drain completely before the method exits.
Are you talking about
{{[CommitLogReplayer.blockForWrites()|https://github.com/apache/cassandra/blob/cassandra-2.1.3/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L98]}}
which is invoked from
{{[CommitLog.recover()|https://github.com/apache/cassandra/blob/cassandra-2.1.3/src/java/org/apache/cassandra/db/commitlog/CommitLog.java#L145]}}?
I think that is already covered. I can refactor it if you want and have
{{CommitLogReplayer.recover()}} return until everything is done anyways. Seems
like a safe change unless it turns out to deadlock some weird test code
somewhere.
bq. I'm not sure why futures was changed to a deque. looks like you only use
queue methods, but maybe I missed it?
It's just a habit for this kind of asynchronous result queue. Muscle memory
writes a loop that prioritizes consuming result futures over submitting new
work in order to minimize latency, working set size, and temporal cache
locality. To do that you need to be able to {{poll()}} a queue and {{List}}
doesn't let you do that.
The other issue with doing 1k and then draining 1k is that there is a period
where the number of tasks in flight is small because the producer is waiting
for the stragglers from the last 1k to arrive before issuing new work. This can
starve consumers. This version keeps a targeted number of pending
mutations/mutation bytes in flight at any given time.
bq. The only other thing I noticed was in the test you should validate the data
test data is not found after you clear the CF in-case the replay isn't working.
[Does clearing the data before
replay|https://github.com/apache/cassandra/compare/cassandra-2.1...aweisberg:CASSANDRA-8639-2.1?expand=1#diff-92ffe896212dc94b91ad86349f0647abR141]
and [then checking for it afterwards accomplish
that?|https://github.com/apache/cassandra/compare/cassandra-2.1...aweisberg:CASSANDRA-8639-2.1?expand=1#diff-92ffe896212dc94b91ad86349f0647abR183]
Unless you think that {{clearUnsafe()}} might not work it seems sufficient.
bq. You also have a 2.1 utest failure related to CL not sure if that's related.
org.apache.cassandra.cql3.DropKeyspaceCommitLogRecycleTest.testRecycle
[That is a pretty flakey
test.|http://cassci.datastax.com/view/cassandra-2.1/job/cassandra-2.1_testall/271/testReport/org.apache.cassandra.cql3/DropKeyspaceCommitLogRecycleTest/testRecycle/history/]
bq. And one dtest failure in 2.1 commitlog_test.TestCommitLog.test_bad_crc
[Looks like another failing/unreliable
test.|http://cassci.datastax.com/view/cassandra-2.1/job/cassandra-2.1_dtest/361/#showFailuresLink]
> Can OOM on CL replay with dense mutations
> -----------------------------------------
>
> Key: CASSANDRA-8639
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8639
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Reporter: T Jake Luciani
> Assignee: Ariel Weisberg
> Priority: Minor
> Fix For: 2.1.x
>
>
> If you write dense mutations with many clustering keys, the replay of the CL
> can quickly overwhelm a node on startup. This looks to be caused by the fact
> we only ensure there are 1000 mutations in flight at a time. but those
> mutations could have thousands of cells in them.
> A better approach would be to limit the CL replay to the amount of memory in
> flight using cell.unsharedHeapSize()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
