[jira] [Comment Edited] (CASSANDRA-11624) Scrub does not seem to work on previously marked corrupted SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-11624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15251392#comment-15251392 ] Michael Fong edited comment on CASSANDRA-11624 at 4/21/16 6:57 AM: --- Looking at the source code of Cassandra-2.0.17, it seems the cause might come from the following logic: >From org.apache.cassandra.db.compaction.CompactionManager … public void performScrub(ColumnFamilyStore cfStore, final boolean skipCorrupted, final boolean checkData) throws InterruptedException, ExecutionException { performAllSSTableOperation(cfStore, new AllSSTablesOperation() { … private void performAllSSTableOperation(final ColumnFamilyStore cfs, final AllSSTablesOperation operation) throws InterruptedException, ExecutionException { final Iterable sstables = cfs.markAllCompacting(); … org.apache.cassandra.db. ColumnFamilyStore … public Iterable markAllCompacting() { Callable> callable = new Callable>() { public Iterable call() throws Exception { assert data.getCompacting().isEmpty() : data.getCompacting(); Iterable sstables = Lists.newArrayList(AbstractCompactionStrategy.filterSuspectSSTables(getSSTables())); <-filter out all previously marked suspected SSTables if (Iterables.isEmpty(sstables)) return null; … -- It seems scrub will not perform if SSTable was marked as corrupted (in blacklist) previously; however, would this defeat the original purpose of scrub operation? was (Author: michael.fong): Looking at the source code of Cassandra-2.0.17, it seems the cause might come from the following logic: >From org.apache.cassandra.db.compaction.CompactionManager … public void performScrub(ColumnFamilyStore cfStore, final boolean skipCorrupted, final boolean checkData) throws InterruptedException, ExecutionException { performAllSSTableOperation(cfStore, new AllSSTablesOperation() { … private void performAllSSTableOperation(final ColumnFamilyStore cfs, final AllSSTablesOperation operation) throws InterruptedException, ExecutionException { final Iterable sstables = cfs.markAllCompacting(); … org.apache.cassandra.db. ColumnFamilyStore … public Iterable markAllCompacting() { Callable> callable = new Callable>() { public Iterable call() throws Exception { assert data.getCompacting().isEmpty() : data.getCompacting(); Iterable sstables = Lists.newArrayList(AbstractCompactionStrategy.filterSuspectSSTables(getSSTables())); <-filter out all previously marked suspected SSTables if (Iterables.isEmpty(sstables)) return null; … -- It seems scrub will not perform if SSTable was marked as corrupted previously; however, would this defeat the original purpose of scrub operation? > Scrub does not seem to work on previously marked corrupted SSTables > --- > > Key: CASSANDRA-11624 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11624 > Project: Cassandra > Issue Type: Bug >Reporter: Michael Fong > > We ran into a scenario that scrub does not seem to work on a previously > marked-as-corrupted SSTable. > Here is the log snippet related to the corrupted SSTable and scrub-attempt : > ERROR [ReadStage:174] 2016-03-17 04:14:39,658 CassandraDaemon.java (line 258) > Exception in thread Thread[ReadStage:174,5,main] > java.lang.RuntimeException: > org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: > mmap segment underflow; remaining is 10197 but 30062 requested for > /data/ng/db/data/wsg/dpStatusRealTime/wsg-dpStatusRealTime-ic-2-Data.db > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2022) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: > java.io.IOException: mmap segment underflow; remaining is 10197 but 30062 > requested for > /data/ng/db/data/wsg/dpStatusRealTime/wsg-dpStatusRealTime-ic-2-Data.db > at > org.apache.cassandra.db.columniterator.IndexedSliceReader.(IndexedSliceReader.java:97) > at > org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65) > at > org.apache.cassandra.db.columniterator.SSTableSliceIterator.(SSTableSliceIterator.java:42) > at > org.apache.cassandra.db.filter.Slice
[jira] [Comment Edited] (CASSANDRA-11624) Scrub does not seem to work on previously marked corrupted SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-11624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15251392#comment-15251392 ] Michael Fong edited comment on CASSANDRA-11624 at 4/21/16 6:48 AM: --- Looking at the source code of Cassandra-2.0.17, it seems the cause might come from the following logic: >From org.apache.cassandra.db.compaction.CompactionManager … public void performScrub(ColumnFamilyStore cfStore, final boolean skipCorrupted, final boolean checkData) throws InterruptedException, ExecutionException { performAllSSTableOperation(cfStore, new AllSSTablesOperation() { … private void performAllSSTableOperation(final ColumnFamilyStore cfs, final AllSSTablesOperation operation) throws InterruptedException, ExecutionException { final Iterable sstables = cfs.markAllCompacting(); … org.apache.cassandra.db. ColumnFamilyStore … public Iterable markAllCompacting() { Callable> callable = new Callable>() { public Iterable call() throws Exception { assert data.getCompacting().isEmpty() : data.getCompacting(); Iterable sstables = Lists.newArrayList(AbstractCompactionStrategy.filterSuspectSSTables(getSSTables())); <-filter out all previously marked suspected SSTables if (Iterables.isEmpty(sstables)) return null; … -- It seems scrub will not perform if SSTable was marked as corrupted previously; however, would this defeat the original purpose of scrub operation? was (Author: michael.fong): Looking at the source code of Cassandra-2.0.17, it seems the cause might come from the following logic: >From org.apache.cassandra.db.compaction.CompactionManager … public void performScrub(ColumnFamilyStore cfStore, final boolean skipCorrupted, final boolean checkData) throws InterruptedException, ExecutionException { performAllSSTableOperation(cfStore, new AllSSTablesOperation() { … private void performAllSSTableOperation(final ColumnFamilyStore cfs, final AllSSTablesOperation operation) throws InterruptedException, ExecutionException { final Iterable sstables = cfs.markAllCompacting(); … org.apache.cassandra.db. ColumnFamilyStore … public Iterable markAllCompacting() { Callable> callable = new Callable>() { public Iterable call() throws Exception { assert data.getCompacting().isEmpty() : data.getCompacting(); Iterable sstables = Lists.newArrayList(AbstractCompactionStrategy.filterSuspectSSTables(getSSTables())); <-filter out all previously marked suspected SSTables if (Iterables.isEmpty(sstables)) return null; … -- It seems scrub will not perform if SSTable was marked as corrupted previously; however, would this defeat the original perform of scrub operation? > Scrub does not seem to work on previously marked corrupted SSTables > --- > > Key: CASSANDRA-11624 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11624 > Project: Cassandra > Issue Type: Bug >Reporter: Michael Fong > > We ran into a scenario that scrub does not seem to work on a previously > marked-as-corrupted SSTable. > Here is the log snippet related to the corrupted SSTable and scrub-attempt : > ERROR [ReadStage:174] 2016-03-17 04:14:39,658 CassandraDaemon.java (line 258) > Exception in thread Thread[ReadStage:174,5,main] > java.lang.RuntimeException: > org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: > mmap segment underflow; remaining is 10197 but 30062 requested for > /data/ng/db/data/wsg/dpStatusRealTime/wsg-dpStatusRealTime-ic-2-Data.db > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2022) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: > java.io.IOException: mmap segment underflow; remaining is 10197 but 30062 > requested for > /data/ng/db/data/wsg/dpStatusRealTime/wsg-dpStatusRealTime-ic-2-Data.db > at > org.apache.cassandra.db.columniterator.IndexedSliceReader.(IndexedSliceReader.java:97) > at > org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65) > at > org.apache.cassandra.db.columniterator.SSTableSliceIterator.(SSTableSliceIterator.java:42) > at > org.apache.cassandra.db.filter.SliceQueryFilter.getSS
[jira] [Comment Edited] (CASSANDRA-11624) Scrub does not seem to work on previously marked corrupted SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-11624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15251392#comment-15251392 ] Michael Fong edited comment on CASSANDRA-11624 at 4/21/16 6:48 AM: --- Looking at the source code of Cassandra-2.0.17, it seems the cause might come from the following logic: >From org.apache.cassandra.db.compaction.CompactionManager … public void performScrub(ColumnFamilyStore cfStore, final boolean skipCorrupted, final boolean checkData) throws InterruptedException, ExecutionException { performAllSSTableOperation(cfStore, new AllSSTablesOperation() { … private void performAllSSTableOperation(final ColumnFamilyStore cfs, final AllSSTablesOperation operation) throws InterruptedException, ExecutionException { final Iterable sstables = cfs.markAllCompacting(); … org.apache.cassandra.db. ColumnFamilyStore … public Iterable markAllCompacting() { Callable> callable = new Callable>() { public Iterable call() throws Exception { assert data.getCompacting().isEmpty() : data.getCompacting(); Iterable sstables = Lists.newArrayList(AbstractCompactionStrategy.filterSuspectSSTables(getSSTables())); <-filter out all previously marked suspected SSTables if (Iterables.isEmpty(sstables)) return null; … -- It seems scrub will not perform if SSTable was marked as corrupted previously; however, would this defeat the original perform of scrub operation? was (Author: michael.fong): Looking at the source code of Cassandra-2.0.17, it seems the cause might come from the following logic: >From org.apache.cassandra.db.compaction.CompactionManager … public void performScrub(ColumnFamilyStore cfStore, final boolean skipCorrupted, final boolean checkData) throws InterruptedException, ExecutionException { performAllSSTableOperation(cfStore, new AllSSTablesOperation() { … private void performAllSSTableOperation(final ColumnFamilyStore cfs, final AllSSTablesOperation operation) throws InterruptedException, ExecutionException { final Iterable sstables = cfs.markAllCompacting(); … org.apache.cassandra.db. ColumnFamilyStore … public Iterable markAllCompacting() { Callable> callable = new Callable>() { public Iterable call() throws Exception { assert data.getCompacting().isEmpty() : data.getCompacting(); Iterable sstables = Lists.newArrayList(AbstractCompactionStrategy.filterSuspectSSTables(getSSTables())); <-filter out all previously marked suspected SSTables if (Iterables.isEmpty(sstables)) return null; … > Scrub does not seem to work on previously marked corrupted SSTables > --- > > Key: CASSANDRA-11624 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11624 > Project: Cassandra > Issue Type: Bug >Reporter: Michael Fong > > We ran into a scenario that scrub does not seem to work on a previously > marked-as-corrupted SSTable. > Here is the log snippet related to the corrupted SSTable and scrub-attempt : > ERROR [ReadStage:174] 2016-03-17 04:14:39,658 CassandraDaemon.java (line 258) > Exception in thread Thread[ReadStage:174,5,main] > java.lang.RuntimeException: > org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: > mmap segment underflow; remaining is 10197 but 30062 requested for > /data/ng/db/data/wsg/dpStatusRealTime/wsg-dpStatusRealTime-ic-2-Data.db > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2022) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: > java.io.IOException: mmap segment underflow; remaining is 10197 but 30062 > requested for > /data/ng/db/data/wsg/dpStatusRealTime/wsg-dpStatusRealTime-ic-2-Data.db > at > org.apache.cassandra.db.columniterator.IndexedSliceReader.(IndexedSliceReader.java:97) > at > org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65) > at > org.apache.cassandra.db.columniterator.SSTableSliceIterator.(SSTableSliceIterator.java:42) > at > org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:238) > at > org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62) > at >