[
https://issues.apache.org/jira/browse/CASSANDRA-14554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16712803#comment-16712803
]
Benedict commented on CASSANDRA-14554:
--------------------------------------
||3.0||3.11||trunk||
|[patch|https://github.com/belliottsmith/cassandra/tree/14554-3.0]|[patch|https://github.com/belliottsmith/cassandra/tree/14554-3.11]|[patch|https://github.com/belliottsmith/cassandra/tree/14554-trunk]|
|[CI|https://circleci.com/workflow-run/3e615cf7-a985-448b-9ba0-6d52f9b87eec]|[CI|https://circleci.com/workflow-run/a74915a5-ea21-49ed-ab0a-389964b606f4]|[CI|https://circleci.com/workflow-run/64915295-ab06-4946-b736-85b799765003]|
AFAICT the CI failures are related to CASSANDRA-14921. There was a [brief
single unit test
failure|https://circleci.com/gh/belliottsmith/cassandra/1032#tests/containers/2]
for one run, but probably environment or timing related.
This patch is simply a slightly modified and ported version of Stefania's patch
above. As discussed, this doesn't necessarily solve the problem perfectly, but
it is close enough that it's worth applying now and worrying about that later.
[~bdeggleston]: I would appreciate it if you could take a quick look at
[this|https://github.com/belliottsmith/cassandra/commit/fd54c420da81ee6ebfa1d29f45b8edc4922c8bdb#diff-374d64d7ac810fe7be021a2ef356c071R216],
as it's not clear to me why there is a separate synchronised
{{finishTransaction}} method. Was there anticipated to be some kind of
potential deadlock?
> LifecycleTransaction encounters ConcurrentModificationException when used in
> multi-threaded context
> ---------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-14554
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14554
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Reporter: Dinesh Joshi
> Assignee: Dinesh Joshi
> Priority: Major
> Fix For: 4.0, 3.0.x, 3.11.x
>
>
> When LifecycleTransaction is used in a multi-threaded context, we encounter
> this exception -
> {quote}java.util.ConcurrentModificationException: null
> at
> java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
> at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:742)
> at java.lang.Iterable.forEach(Iterable.java:74)
> at
> org.apache.cassandra.db.lifecycle.LogReplicaSet.maybeCreateReplica(LogReplicaSet.java:78)
> at org.apache.cassandra.db.lifecycle.LogFile.makeRecord(LogFile.java:320)
> at org.apache.cassandra.db.lifecycle.LogFile.add(LogFile.java:285)
> at
> org.apache.cassandra.db.lifecycle.LogTransaction.trackNew(LogTransaction.java:136)
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.trackNew(LifecycleTransaction.java:529)
> {quote}
> During streaming we create a reference to a {{LifeCycleTransaction}} and
> share it between threads -
> [https://github.com/apache/cassandra/blob/5cc68a87359dd02412bdb70a52dfcd718d44a5ba/src/java/org/apache/cassandra/db/streaming/CassandraStreamReader.java#L156]
> This is used in a multi-threaded context inside {{CassandraIncomingFile}}
> which is an {{IncomingStreamMessage}}. This is being deserialized in parallel.
> {{LifecycleTransaction}} is not meant to be used in a multi-threaded context
> and this leads to streaming failures due to object sharing. On trunk, this
> object is shared across all threads that transfer sstables in parallel for
> the given {{TableId}} in a {{StreamSession}}. There are two options to solve
> this - make {{LifecycleTransaction}} and the associated objects thread safe,
> scope the transaction to a single {{CassandraIncomingFile}}. The consequences
> of the latter option is that if we experience streaming failure we may have
> redundant SSTables on disk. This is ok as compaction should clean this up. A
> third option is we synchronize access in the streaming infrastructure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
