[
https://issues.apache.org/jira/browse/CASSANDRA-9686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14624650#comment-14624650
]
Stefania commented on CASSANDRA-9686:
-------------------------------------
That's the expected behavior, see the discussion above regarding carrying on
with corrupt system tables. It's a bug that _stop_ allows the transports and
gossip to be started after we have encountered corrupt sstable files. If it
happened after startup, _stop_ would leave the process running but with no
gossip or transports, so that you cannot do anything with it other than
analyzing the status via JMX. During startup, gossip or the transports aren't
started yet, so they are not stopped and later they start. We decided the
easiest way to fix this was to stop the process if we encounter corrupt
sstables at startup.
You can use the _ignore_ policy to carry on, but you really should fix corrupt
sstable files, or the system may become unstable depending on which sstable
files are corrupt.
If standalone scrub (sstablescrub) recovers the files (it couldn't with yours)
then it would move them as well. Perhaps you could open a ticket to enhance
sstablescrub to always move files, even if they cannot be recovered. So I
suggest you try standalone scrub and open a ticket with a request to move the
files, perhaps via a command line argument? The idea would be: cassandra fails
fast (it won't start), you run standalone scrub and restart. Better like this
than ignoring corrupt sstable files IMO.
> FSReadError and LEAK DETECTED after upgrading
> ---------------------------------------------
>
> Key: CASSANDRA-9686
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9686
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Windows-7-32 bit, 3.2GB RAM, Java 1.7.0_55
> Reporter: Andreas Schnitzerling
> Assignee: Stefania
> Fix For: 2.1.x, 2.2.x
>
> Attachments: cassandra.bat, cassandra.yaml,
> compactions_in_progress.zip, sstable_activity.zip, system.log
>
>
> After upgrading one of 15 nodes from 2.1.7 to 2.2.0-rc1 I get FSReadError and
> LEAK DETECTED on start. Deleting the listed files, the failure goes away.
> {code:title=system.log}
> ERROR [SSTableBatchOpen:1] 2015-06-29 14:38:34,554
> DebuggableThreadPoolExecutor.java:242 - Error in ThreadPoolExecutor
> org.apache.cassandra.io.FSReadError: java.io.IOException: Compressed file
> with 0 chunks encountered: java.io.DataInputStream@1c42271
> at
> org.apache.cassandra.io.compress.CompressionMetadata.readChunkOffsets(CompressionMetadata.java:178)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:117)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:86)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:142)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:101)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:178)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.load(SSTableReader.java:681)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.load(SSTableReader.java:644)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:443)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:350)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:480)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> ~[na:1.7.0_55]
> at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_55]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> [na:1.7.0_55]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> [na:1.7.0_55]
> at java.lang.Thread.run(Unknown Source) [na:1.7.0_55]
> Caused by: java.io.IOException: Compressed file with 0 chunks encountered:
> java.io.DataInputStream@1c42271
> at
> org.apache.cassandra.io.compress.CompressionMetadata.readChunkOffsets(CompressionMetadata.java:174)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> ... 15 common frames omitted
> ERROR [Reference-Reaper:1] 2015-06-29 14:38:34,734 Ref.java:189 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@3e547f) to class
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1926439:D:\Programme\Cassandra\data\data\system\compactions_in_progress\system-compactions_in_progress-ka-6866
> was not released before the reference was garbage collected
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
