[
https://issues.apache.org/jira/browse/CASSANDRA-13608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099356#comment-16099356
]
Kurt Greaves commented on CASSANDRA-13608:
------------------------------------------
I'll have a stab at reproducing and a fix.
> Connection closed/reopened during join causes Cassandra stream to close
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-13608
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13608
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: Cassandra 3.10. Windows Server 2016, 32GB ram, 2TB hard
> disk, RAID10 with 4 spindles, 8 Cores
> Reporter: Tania S Engel
> Assignee: Kurt Greaves
> Attachments: Cassandra 3.10 Join with lots GC collection leads to
> socket closure and join hang.mht, Cassandra 3.10 Join with lots GC collection
> leads to socket closure and join hang.pdf, Cassandra 3.10 Join with lots GC
> collection leads to socket closure and join hang.txt
>
>
> We start a JOIN bootstrap. Primary seed node streams to the replica. The
> replica requires some GC cleanup and experiences frequent pauses including a
> 12 second old gen cleanup following a memTable flush. Both replica and
> primary show _MessagingService IOException: An existing connection was
> forcibly closed by the remote host_. The replica MessagingService-Outgoing
> reestablishes the connection immediately but the primary
> StreamKeepAliveExecutor throws a _java.RuntimeException: Outgoing stream
> handler has been closed_. >From that point forward, the replica stays in JOIN
> mode, sending keeping alive to the primary. The primary receives the keep
> alive, but does not send its own and it repeatedly fails to send a hints file
> to the replica. It seems this limping condition would continue indefinitely,
> but stops as we stop the replica Cassandra. If we restart the replica
> Cassandra the JOIN picks up again but fails with _java.io.IOException:
> Corrupt value length 355151036 encountered, as it exceeds the maximum of
> 268435456, which is set via max_value_size_in_mb in cassandra.yaml_. We have
> not increased this value as we do not have values that large in our data so
> we presume it is indeed corrupt and moving past it would not be a good idea.
> Please see the attachment for details.
> {code}
> ERROR [BatchlogTasks:1] 2017-06-13 20:24:13,953 CassandraDaemon.java:229 -
> Exception in thread Thread[BatchlogTasks:1,5,main]
> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
> C:\Cassandra\data\data\system\batches-919a4bc57a333573b03e13fc3f68b465\mc-2-big-Data.db
> at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:395)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:257)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:133)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:69)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:67)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:26)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:96)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:755)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:446)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.cql3.UntypedResultSet$FromPager$1.computeNext(UntypedResultSet.java:193)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.cql3.UntypedResultSet$FromPager$1.computeNext(UntypedResultSet.java:179)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.batchlog.BatchlogManager.processBatchlogEntries(BatchlogManager.java:233)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.batchlog.BatchlogManager.replayFailedBatches(BatchlogManager.java:209)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_66]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_66]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_66]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_66]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_66]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_66]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> [apache-cassandra-3.10.jar:3.10]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66]
> Caused by: java.io.IOException: Corrupt value length 355151036 encountered,
> as it exceeds the maximum of 268435456, which is set via max_value_size_in_mb
> in cassandra.yaml
> at
> org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:428)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:245)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.readComplexColumn(UnfilteredSerializer.java:636)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.lambda$deserializeRowBody$1(UnfilteredSerializer.java:577)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.utils.btree.BTree.applyForwards(BTree.java:1222)
> ~[apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.utils.btree.BTree.apply(BTree.java:1177)
> ~[apache-cassandra-3.10.jar:3.10]
> at org.apache.cassandra.db.Columns.apply(Columns.java:377)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:571)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.readNext(UnfilteredDeserializer.java:211)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:144)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:168)
> ~[apache-cassandra-3.10.jar:3.10]
> at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:382)
> ~[apache-cassandra-3.10.jar:3.10]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
