You have a/some corrupt SSTables. 2.1.9 is doing strict checking at startup and reacting based on "disk_failure_policy" per the stack trace.
For details, see: https://issues.apache.org/jira/browse/CASSANDRA-9686 Either way, you are going to have to run nodetool scrub. I'm not sure if it's better to do this from 2.1.8 or from 2.1.9 with "disk_failure_policy: ignore" It feels like that option got overloaded a bit strangely with the changes in CASSANDRA-9686 and I have not yet tried it with it's new meaning. On Tue, Sep 15, 2015 at 5:26 AM, George Sigletos <sigle...@textkernel.nl> wrote: > Hello, > > I tried to upgrade two of our clusters from 2.1.8 to 2.1.9. In some, but > not all nodes, I got errors about corrupt sstables when restarting. I > downgraded back to 2.1.8 for now. > > Has anybody else faced the same problem? Should sstablescrub fix the > problem? I ddin't tried that yet. > > Kind regards, > George > > ERROR [SSTableBatchOpen:3] 2015-09-14 10:16:03,296 FileUtils.java:447 - > Exiting forcefully due to file system exception on startup, disk failure > policy "stop" > org.apache.cassandra.io.sstable.CorruptSSTableException: > java.io.EOFException > at > org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:131) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:752) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:703) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:387) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:534) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at java.util.concurrent.Executors$RunnableAdapter.call(Unknown > Source) [na:1.7.0_75] > at java.util.concurrent.FutureTask.run(Unknown Source) > [na:1.7.0_75] > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > Source) [na:1.7.0_75] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) [na:1.7.0_75] > at java.lang.Thread.run(Unknown Source) [na:1.7.0_75] > Caused by: java.io.EOFException: null > at java.io.DataInputStream.readUnsignedShort(Unknown Source) > ~[na:1.7.0_75] > at java.io.DataInputStream.readUTF(Unknown Source) ~[na:1.7.0_75] > at java.io.DataInputStream.readUTF(Unknown Source) ~[na:1.7.0_75] > at > org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:106) > ~[apache-cassandra-2.1.9.jar:2.1.9] > -- ----------------- Nate McCall Austin, TX @zznate Co-Founder & Sr. Technical Consultant Apache Cassandra Consulting http://www.thelastpickle.com
