[ https://issues.apache.org/jira/browse/CASSANDRA-12113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15357350#comment-15357350 ]
Paulo Motta commented on CASSANDRA-12113: ----------------------------------------- [~serhatd] Could you double check logs of this and other nodes for streaming errors around the time of that exception? In particular you should search for the string {{Streaming error occurred}} in the logs. > Cassandra 3.5.0 Repair Error > ---------------------------- > > Key: CASSANDRA-12113 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12113 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: Production > Reporter: Serhat Rıfat Demircan > Assignee: Paulo Motta > > I got the following error while repairing nodes with the "nodetool repair" > command. Error occured on 2 nodes in the cluster which have 9 nodes. > > Interesting thing is corrupted sstable is no more exists one of 2 nodes. > Copied existing one to test cluster and restored table from that sstable. No > error occured on test cluster. > {noformat} > ERROR [StreamReceiveTask:6] 2016-06-16 02:56:47,480 > StreamReceiveTask.java:215 - Error applying streamed data: > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: > /var/lib/cassandra/data/keyspace/table/ma-1518-big-Data.db > at > org.apache.cassandra.utils.Throwables.maybeFail(Throwables.java:50) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:372) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.index.SecondaryIndexManager.buildIndexesBlocking(SecondaryIndexManager.java:375) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.index.SecondaryIndexManager.buildAllIndexesBlocking(SecondaryIndexManager.java:262) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:182) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_91] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [na:1.8.0_91] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_91] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_91] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91] > Caused by: java.util.concurrent.ExecutionException: > org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: > /var/lib/cassandra/data/keyspace/table/ma-1518-big-Data.db > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > [na:1.8.0_91] > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > [na:1.8.0_91] > at > org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:365) > ~[apache-cassandra-3.5.0.jar:3.5.0] > ... 8 common frames omitted > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: /var/lib/cassandra/data/keyspace/table/ma-1518-big-Data.db > at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:367) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:229) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:93) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:25) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:112) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:549) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.db.Keyspace.indexPartition(Keyspace.java:570) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:70) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.db.compaction.CompactionManager$12.run(CompactionManager.java:1472) > ~[apache-cassandra-3.5.0.jar:3.5.0] > ... 5 common frames omitted > Caused by: java.io.IOException: Error building row with data deserialized > from /var/lib/cassandra/data/keyspace/table/ma-1518-big-Data.db - chunk > length 65536, data length 85398996. > at > org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:446) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.readNext(UnfilteredDeserializer.java:211) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:125) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:149) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:354) > ~[apache-cassandra-3.5.0.jar:3.5.0] > ... 23 common frames omitted > Caused by: java.lang.AssertionError: Memory was freed > at > org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at org.apache.cassandra.io.util.Memory.getLong(Memory.java:260) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:235) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:170) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:144) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.utils.vint.VIntCoding.readUnsignedVInt(VIntCoding.java:63) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.io.util.RebufferingInputStream.readUnsignedVInt(RebufferingInputStream.java:209) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.db.SerializationHeader.readLocalDeletionTime(SerializationHeader.java:210) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:224) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.db.rows.UnfilteredSerializer.readSimpleColumn(UnfilteredSerializer.java:455) > ~[apache-cassandra-3.5.0.jar:3.5.0] > at > org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:433) > ~[apache-cassandra-3.5.0.jar:3.5.0] > ... 27 common frames omitted > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)