Re: Repair corrupt SSTable from power outage?
I'm running 2.1.9 On Fri, Oct 2, 2015 at 8:40 AM, Tyler Hobbs wrote: > What version of Cassandra are you running? It sounds like the disk > failure policy is incorrectly being applied to scrub, which kind of defeats > the purpose of it. I'd recommend opening a ticket ( > https://issues.apache.org/jira/browse/CASSANDRA) with the information you > posted here. > > On Fri, Oct 2, 2015 at 3:01 AM, George Sigletos > wrote: > >> I'm also facing problems regarding corrupt sstables and also couldn't run >> sstablescrub successfully. >> >> I restarted my nodes with disk failure policy "best_effort", then I run >> the "nodetool scrub " >> >> Once done I removed the corrupt tables manually and started repair >> >> >> On Thu, Oct 1, 2015 at 7:27 PM, John Anderson wrote: >> >>> I have a 25 node cluster and we lost power on one of the racks last >>> night and now 6 of our nodes will not start up and we are getting the >>> following error: >>> >>> INFO [main] 2015-10-01 10:19:22,111 CassandraDaemon.java:122 - JMX is >>> enabled to receive remote connections on port: 7199 >>> INFO [main] 2015-10-01 10:19:22,124 CacheService.java:111 - >>> Initializing key cache with capacity of 100 MBs. >>> INFO [main] 2015-10-01 10:19:22,129 CacheService.java:133 - >>> Initializing row cache with capacity of 0 MBs >>> INFO [main] 2015-10-01 10:19:22,133 CacheService.java:150 - >>> Initializing counter cache with capacity of 50 MBs >>> INFO [main] 2015-10-01 10:19:22,135 CacheService.java:161 - Scheduling >>> counter cache save to every 7200 seconds (going to save all keys). >>> INFO [main] 2015-10-01 10:19:22,211 ColumnFamilyStore.java:363 - >>> Initializing system.sstable_activity >>> INFO [SSTableBatchOpen:1] 2015-10-01 10:19:22,639 >>> SSTableReader.java:478 - Opening >>> /mnt/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-45 >>> (805 bytes) >>> ERROR [SSTableBatchOpen:1] 2015-10-01 10:19:22,657 FileUtils.java:447 - >>> Exiting forcefully due to file system exception on startup, disk failure >>> policy "stop" >>> org.apache.cassandra.io.sstable.CorruptSSTableException: >>> java.io.EOFException >>> at >>> org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:131) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:752) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:703) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:387) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:534) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>> [na:1.8.0_60] >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> [na:1.8.0_60] >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>> [na:1.8.0_60] >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>> [na:1.8.0_60] >>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] >>> Caused by: java.io.EOFException: null >>> at >>> java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340) >>> ~[na:1.8.0_60] >>> at java.io.DataInputStream.readUTF(DataInputStream.java:589) >>> ~[na:1.8.0_60] >>> at java.io.DataInputStream.readUTF(DataInputStream.java:564) >>> ~[na:1.8.0_60] >>> at >>> org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:106) >>> ~[apache-cassandra-2.1.9.jar:2.1.9] >>> ... 14 common frames omitted >>> >>> >>> I found some people recommending scrubbing the sstable so I attempted >>> that and got the following error: >>> >>> bin/sstablescrub system sstable_activity -v >>> >>> >>> ERROR 17:26:03 Exiting forcefully due to file system exception on >>> startup, disk failure policy "stop" >>> org.apache.cassandra.io.sstable.CorruptSSTableException: >>> java.io.EOFE
Re: Repair corrupt SSTable from power outage?
What version of Cassandra are you running? It sounds like the disk failure policy is incorrectly being applied to scrub, which kind of defeats the purpose of it. I'd recommend opening a ticket ( https://issues.apache.org/jira/browse/CASSANDRA) with the information you posted here. On Fri, Oct 2, 2015 at 3:01 AM, George Sigletos wrote: > I'm also facing problems regarding corrupt sstables and also couldn't run > sstablescrub successfully. > > I restarted my nodes with disk failure policy "best_effort", then I run > the "nodetool scrub " > > Once done I removed the corrupt tables manually and started repair > > > On Thu, Oct 1, 2015 at 7:27 PM, John Anderson wrote: > >> I have a 25 node cluster and we lost power on one of the racks last night >> and now 6 of our nodes will not start up and we are getting the following >> error: >> >> INFO [main] 2015-10-01 10:19:22,111 CassandraDaemon.java:122 - JMX is >> enabled to receive remote connections on port: 7199 >> INFO [main] 2015-10-01 10:19:22,124 CacheService.java:111 - Initializing >> key cache with capacity of 100 MBs. >> INFO [main] 2015-10-01 10:19:22,129 CacheService.java:133 - Initializing >> row cache with capacity of 0 MBs >> INFO [main] 2015-10-01 10:19:22,133 CacheService.java:150 - Initializing >> counter cache with capacity of 50 MBs >> INFO [main] 2015-10-01 10:19:22,135 CacheService.java:161 - Scheduling >> counter cache save to every 7200 seconds (going to save all keys). >> INFO [main] 2015-10-01 10:19:22,211 ColumnFamilyStore.java:363 - >> Initializing system.sstable_activity >> INFO [SSTableBatchOpen:1] 2015-10-01 10:19:22,639 SSTableReader.java:478 >> - Opening >> /mnt/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-45 >> (805 bytes) >> ERROR [SSTableBatchOpen:1] 2015-10-01 10:19:22,657 FileUtils.java:447 - >> Exiting forcefully due to file system exception on startup, disk failure >> policy "stop" >> org.apache.cassandra.io.sstable.CorruptSSTableException: >> java.io.EOFException >> at >> org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:131) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:752) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:703) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:387) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:534) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> [na:1.8.0_60] >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> [na:1.8.0_60] >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> [na:1.8.0_60] >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> [na:1.8.0_60] >> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] >> Caused by: java.io.EOFException: null >> at >> java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340) >> ~[na:1.8.0_60] >> at java.io.DataInputStream.readUTF(DataInputStream.java:589) >> ~[na:1.8.0_60] >> at java.io.DataInputStream.readUTF(DataInputStream.java:564) >> ~[na:1.8.0_60] >> at >> org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:106) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> ... 14 common frames omitted >> >> >> I found some people recommending scrubbing the sstable so I attempted >> that and got the following error: >> >> bin/sstablescrub system sstable_activity -v >> >> >> ERROR 17:26:03 Exiting forcefully due to file system exception on >> startup, disk failure policy "stop" >> org.apache.cassandra.io.sstable.CorruptSSTableException: >> java.io.EOFException >> at >> org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:131) >> ~[apache-cassandra-2.1.9.jar:2.1.9] >> at >> org.apache.cassandra.io.
Re: Repair corrupt SSTable from power outage?
I'm also facing problems regarding corrupt sstables and also couldn't run sstablescrub successfully. I restarted my nodes with disk failure policy "best_effort", then I run the "nodetool scrub " Once done I removed the corrupt tables manually and started repair On Thu, Oct 1, 2015 at 7:27 PM, John Anderson wrote: > I have a 25 node cluster and we lost power on one of the racks last night > and now 6 of our nodes will not start up and we are getting the following > error: > > INFO [main] 2015-10-01 10:19:22,111 CassandraDaemon.java:122 - JMX is > enabled to receive remote connections on port: 7199 > INFO [main] 2015-10-01 10:19:22,124 CacheService.java:111 - Initializing > key cache with capacity of 100 MBs. > INFO [main] 2015-10-01 10:19:22,129 CacheService.java:133 - Initializing > row cache with capacity of 0 MBs > INFO [main] 2015-10-01 10:19:22,133 CacheService.java:150 - Initializing > counter cache with capacity of 50 MBs > INFO [main] 2015-10-01 10:19:22,135 CacheService.java:161 - Scheduling > counter cache save to every 7200 seconds (going to save all keys). > INFO [main] 2015-10-01 10:19:22,211 ColumnFamilyStore.java:363 - > Initializing system.sstable_activity > INFO [SSTableBatchOpen:1] 2015-10-01 10:19:22,639 SSTableReader.java:478 > - Opening > /mnt/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-45 > (805 bytes) > ERROR [SSTableBatchOpen:1] 2015-10-01 10:19:22,657 FileUtils.java:447 - > Exiting forcefully due to file system exception on startup, disk failure > policy "stop" > org.apache.cassandra.io.sstable.CorruptSSTableException: > java.io.EOFException > at > org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:131) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:752) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:703) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:387) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:534) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_60] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [na:1.8.0_60] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_60] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_60] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] > Caused by: java.io.EOFException: null > at > java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340) > ~[na:1.8.0_60] > at java.io.DataInputStream.readUTF(DataInputStream.java:589) > ~[na:1.8.0_60] > at java.io.DataInputStream.readUTF(DataInputStream.java:564) > ~[na:1.8.0_60] > at > org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:106) > ~[apache-cassandra-2.1.9.jar:2.1.9] > ... 14 common frames omitted > > > I found some people recommending scrubbing the sstable so I attempted that > and got the following error: > > bin/sstablescrub system sstable_activity -v > > > ERROR 17:26:03 Exiting forcefully due to file system exception on startup, > disk failure policy "stop" > org.apache.cassandra.io.sstable.CorruptSSTableException: > java.io.EOFException > at > org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:131) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.
Repair corrupt SSTable from power outage?
I have a 25 node cluster and we lost power on one of the racks last night and now 6 of our nodes will not start up and we are getting the following error: INFO [main] 2015-10-01 10:19:22,111 CassandraDaemon.java:122 - JMX is enabled to receive remote connections on port: 7199 INFO [main] 2015-10-01 10:19:22,124 CacheService.java:111 - Initializing key cache with capacity of 100 MBs. INFO [main] 2015-10-01 10:19:22,129 CacheService.java:133 - Initializing row cache with capacity of 0 MBs INFO [main] 2015-10-01 10:19:22,133 CacheService.java:150 - Initializing counter cache with capacity of 50 MBs INFO [main] 2015-10-01 10:19:22,135 CacheService.java:161 - Scheduling counter cache save to every 7200 seconds (going to save all keys). INFO [main] 2015-10-01 10:19:22,211 ColumnFamilyStore.java:363 - Initializing system.sstable_activity INFO [SSTableBatchOpen:1] 2015-10-01 10:19:22,639 SSTableReader.java:478 - Opening /mnt/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-45 (805 bytes) ERROR [SSTableBatchOpen:1] 2015-10-01 10:19:22,657 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException at org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:131) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:752) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:703) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:387) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:534) ~[apache-cassandra-2.1.9.jar:2.1.9] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_60] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_60] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_60] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] Caused by: java.io.EOFException: null at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340) ~[na:1.8.0_60] at java.io.DataInputStream.readUTF(DataInputStream.java:589) ~[na:1.8.0_60] at java.io.DataInputStream.readUTF(DataInputStream.java:564) ~[na:1.8.0_60] at org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:106) ~[apache-cassandra-2.1.9.jar:2.1.9] ... 14 common frames omitted I found some people recommending scrubbing the sstable so I attempted that and got the following error: bin/sstablescrub system sstable_activity -v ERROR 17:26:03 Exiting forcefully due to file system exception on startup, disk failure policy "stop" org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException at org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:131) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:752) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:703) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491) ~[apache-cassandra-2.1.9.jar:2.1.9] at org.apache.cassandra.io.sstable.SSTable
