[
https://issues.apache.org/jira/browse/CASSANDRA-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15765213#comment-15765213
]
Paulo Motta edited comment on CASSANDRA-13055 at 12/20/16 9:19 PM:
-------------------------------------------------------------------
It seems this was mostly caused by multiple streaming sessions being executed
simultaneously for the same repair parent repair session due to {{RepairJob}}
releasing the executor after [validations were
finished|https://github.com/apache/cassandra/blob/732af7de7f4b7865c00dfa0d85e4dbf4ee9900e2/src/java/org/apache/cassandra/repair/RepairJob.java#L160],
which means the next repair session starts while the previous is still
syncing. In this case, instead of [waiting for
validations|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/repair/RepairJob.java#L160]
we can probably wait for sync results instead before releasing the
{{RepairJob}}, what will prevent multiple repair sessions to run simultaneously.
Another thing is that CASSANDRA-7795 added {{StreamReceiveTask}} executor's
limited by {{FBUtilities.getAvailableProcessors()}}, but this limit was removed
by CASSANDRA-6455 without clear reason, so I believe it could be a merge error.
Do you confirm [~yukim] or is there any particular reason for removing this
executor's limit?
If the limit removal was unintentional, I propose restoring it to
{{FBUtilities.getAvailableProcessors()}} and reduce it's priority, while
possibly making it settable via JMX or system property for advanced users which
want more control, but I think it shouldn't be necessary in most cases
specially after the additional repair job synchronization added on the
{{RepairJob}}.
* Please note that this is not a limit for concurrent streaming tasks, but only
to the final completion phase of streaming, where sstables are added to the
tracker which can be CPU intensive, but should not dominate streaming time in
general.
** Nevertherless it would be nice to check the impact of this for bootstrap, I
believe it won't have any significant impact but it's good to test/verify.
was (Author: pauloricardomg):
It seems this was mostly caused by multiple streaming sessions being executed
simultaneously for the same repair parent repair session due to {{RepairJob}}
releasing the executor after [validations were
finished|https://github.com/apache/cassandra/blob/732af7de7f4b7865c00dfa0d85e4dbf4ee9900e2/src/java/org/apache/cassandra/repair/RepairJob.java#L160],
which means the next repair session starts while the previous is still
syncing. This should be alleviated significantly on 3.0.11 by CASSANDRA-12901,
which waits for a previous {{RepairSession}} to finish (sync streaming
included) before starting the repair session for the next vnode. Can you test
with that version to see if it improves things?
Another thing is that CASSANDRA-7795 added {{StreamReceiveTask}} executor's
limited by {{FBUtilities.getAvailableProcessors()}}, but this limit was removed
by CASSANDRA-6455 without clear reason, so I believe it could be a merge error.
Do you confirm [~yukim] or is there any particular reason for removing this
executor's limit?
If the limit removal was unintentional, I propose restoring it to
{{FBUtilities.getAvailableProcessors()}} and reduce it's priority, while
possibly making it settable via JMX or system property for advanced users which
want more control, but I think it shouldn't be necessary in most cases
specially after the additional repair job synchronization added by
CASSANDRA-12901.
* Please note that this is not a limit for concurrent streaming tasks, but only
to the final completion phase of streaming, where sstables are added to the
tracker which can be CPU intensive, but should not dominate streaming time in
general.
** Nevertherless it would be nice to check the impact of this for bootstrap, I
believe it won't have any significant impact but it's good to test/verify.
> DoS by StreamReceiveTask, during incremental repair
> ---------------------------------------------------
>
> Key: CASSANDRA-13055
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13055
> Project: Cassandra
> Issue Type: Bug
> Reporter: Tom van der Woerdt
> Attachments: untitled 2.txt
>
>
> There's no limit on how many StreamReceiveTask there can be, and during an
> incremental repair on a vnode cluster with high replication factors, this can
> lead to thousands of conccurent StreamReceiveTask threads, effectively DoSing
> the node.
> I just found one of my nodes with 1000+ loadavg, caused by 1363 concurrent
> StreamReceiveTask threads.
> That sucks :)
> I think :
> * Cassandra shouldn't allow more than X concurrent StreamReceiveTask threads
> * StreamReceiveTask threads should be at a lower priority, like compaction
> threads
> Alternative ideas welcome as well, of course.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
