[jira] [Commented] (CASSANDRA-12582) Removing static column results in ReadFailure due to CorruptSSTableException
[ https://issues.apache.org/jira/browse/CASSANDRA-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15561020#comment-15561020 ] Stefania commented on CASSANDRA-12582: -- Thanks for the review, committed to 3.0 as 2f0e365dc495336332e174b30e2d15e56fc344e2 and merged into 3.X and trunk. > Removing static column results in ReadFailure due to CorruptSSTableException > > > Key: CASSANDRA-12582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12582 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths > Environment: Cassandra 3.0.8 >Reporter: Evan Prothro >Assignee: Stefania >Priority: Critical > Labels: compaction, corruption, drop, read, static > Fix For: 3.0.10, 3.10 > > Attachments: 12582.cdl, 12582_reproduce.sh > > > We ran into an issue on production where reads began to fail for certain > queries, depending on the range within the relation for those queries. > Cassandra system log showed an unhandled {{CorruptSSTableException}} > exception. > CQL read failure: > {code} > ReadFailure: code=1300 [Replica(s) failed to execute read] message="Operation > failed - received 0 responses and 1 failures" info={'failures': 1, > 'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'} > {code} > Cassandra exception: > {code} > WARN [SharedPool-Worker-2] 2016-08-31 12:49:27,979 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-2,5,main]: {} > java.lang.RuntimeException: > org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2453) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_72] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) > [apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.0.8.jar:3.0.8] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:343) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:65) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:66) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:62) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:96) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:295) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:134) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:127) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:123) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:289) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1796) >
[jira] [Commented] (CASSANDRA-12582) Removing static column results in ReadFailure due to CorruptSSTableException
[ https://issues.apache.org/jira/browse/CASSANDRA-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1431#comment-1431 ] Carl Yeksigian commented on CASSANDRA-12582: +1 > Removing static column results in ReadFailure due to CorruptSSTableException > > > Key: CASSANDRA-12582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12582 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths > Environment: Cassandra 3.0.8 >Reporter: Evan Prothro >Assignee: Stefania >Priority: Critical > Labels: compaction, corruption, drop, read, static > Fix For: 3.0.x, 3.x > > Attachments: 12582.cdl, 12582_reproduce.sh > > > We ran into an issue on production where reads began to fail for certain > queries, depending on the range within the relation for those queries. > Cassandra system log showed an unhandled {{CorruptSSTableException}} > exception. > CQL read failure: > {code} > ReadFailure: code=1300 [Replica(s) failed to execute read] message="Operation > failed - received 0 responses and 1 failures" info={'failures': 1, > 'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'} > {code} > Cassandra exception: > {code} > WARN [SharedPool-Worker-2] 2016-08-31 12:49:27,979 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-2,5,main]: {} > java.lang.RuntimeException: > org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2453) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_72] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) > [apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.0.8.jar:3.0.8] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:343) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:65) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:66) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:62) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:96) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:295) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:134) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:127) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:123) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:289) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1796) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2449)
[jira] [Commented] (CASSANDRA-12582) Removing static column results in ReadFailure due to CorruptSSTableException
[ https://issues.apache.org/jira/browse/CASSANDRA-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15554114#comment-15554114 ] Stefania commented on CASSANDRA-12582: -- Rebased on the latest branches, the 3.0 patch applies cleanly upwards: ||3.0||3.X||trunk|| |[patch|https://github.com/stef1927/cassandra/commits/12582-3.0]|[patch|https://github.com/stef1927/cassandra/commits/12582-3.X]|[patch|https://github.com/stef1927/cassandra/commits/12582]| |[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12582-3.0-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12582-3.X-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12582-testall/]| |[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12582-3.0-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12582-3.X-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12582-dtest/]| Since there were no conflicts when rebasing, I did not relaunch the CI jobs which were clean. > Removing static column results in ReadFailure due to CorruptSSTableException > > > Key: CASSANDRA-12582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12582 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths > Environment: Cassandra 3.0.8 >Reporter: Evan Prothro >Assignee: Stefania >Priority: Critical > Labels: compaction, corruption, drop, read, static > Fix For: 3.0.x, 3.x > > Attachments: 12582.cdl, 12582_reproduce.sh > > > We ran into an issue on production where reads began to fail for certain > queries, depending on the range within the relation for those queries. > Cassandra system log showed an unhandled {{CorruptSSTableException}} > exception. > CQL read failure: > {code} > ReadFailure: code=1300 [Replica(s) failed to execute read] message="Operation > failed - received 0 responses and 1 failures" info={'failures': 1, > 'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'} > {code} > Cassandra exception: > {code} > WARN [SharedPool-Worker-2] 2016-08-31 12:49:27,979 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-2,5,main]: {} > java.lang.RuntimeException: > org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2453) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_72] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) > [apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.0.8.jar:3.0.8] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:343) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:65) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:66) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:62) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:96) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:295) >
[jira] [Commented] (CASSANDRA-12582) Removing static column results in ReadFailure due to CorruptSSTableException
[ https://issues.apache.org/jira/browse/CASSANDRA-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15521804#comment-15521804 ] Stefania commented on CASSANDRA-12582: -- Thanks for the explanation [~slebresne], I created CASSANDRA-12705 for adding a field to SystemKeyspace.DroppedColumns. I totally missed that the schema mutations are pushed to other nodes and would therefore cause exceptions during a rolling upgrade. I've checked the CI results and they are all clean except for testall on trunk, but the failures there also occur on trunk. Do you think you could be the reviewer since you've already looked at the change anyway? I also converted the reproduction script into a dtest, pull request [here|https://github.com/riptano/cassandra-dtest/pull/1342]. -- [~eprothro]: it's great to hear that the patch fixes your real-world use case as well, thank you so much for this update and, once again, for the reproduction script. > Removing static column results in ReadFailure due to CorruptSSTableException > > > Key: CASSANDRA-12582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12582 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths > Environment: Cassandra 3.0.8 >Reporter: Evan Prothro >Assignee: Stefania >Priority: Critical > Labels: compaction, corruption, drop, read, static > Fix For: 3.0.x, 3.x > > Attachments: 12582.cdl, 12582_reproduce.sh > > > We ran into an issue on production where reads began to fail for certain > queries, depending on the range within the relation for those queries. > Cassandra system log showed an unhandled {{CorruptSSTableException}} > exception. > CQL read failure: > {code} > ReadFailure: code=1300 [Replica(s) failed to execute read] message="Operation > failed - received 0 responses and 1 failures" info={'failures': 1, > 'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'} > {code} > Cassandra exception: > {code} > WARN [SharedPool-Worker-2] 2016-08-31 12:49:27,979 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-2,5,main]: {} > java.lang.RuntimeException: > org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2453) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_72] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) > [apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.0.8.jar:3.0.8] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:343) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:65) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:66) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:62) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:96) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:295) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:134) >
[jira] [Commented] (CASSANDRA-12582) Removing static column results in ReadFailure due to CorruptSSTableException
[ https://issues.apache.org/jira/browse/CASSANDRA-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516892#comment-15516892 ] Evan Prothro commented on CASSANDRA-12582: -- [~Stefania] confirmed that your patch fixes the issue for me in both the reproduction script, as well as for our real-world use case. Thank you! > Removing static column results in ReadFailure due to CorruptSSTableException > > > Key: CASSANDRA-12582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12582 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths > Environment: Cassandra 3.0.8 >Reporter: Evan Prothro >Assignee: Stefania >Priority: Critical > Labels: compaction, corruption, drop, read, static > Fix For: 3.0.x, 3.x > > Attachments: 12582.cdl, 12582_reproduce.sh > > > We ran into an issue on production where reads began to fail for certain > queries, depending on the range within the relation for those queries. > Cassandra system log showed an unhandled {{CorruptSSTableException}} > exception. > CQL read failure: > {code} > ReadFailure: code=1300 [Replica(s) failed to execute read] message="Operation > failed - received 0 responses and 1 failures" info={'failures': 1, > 'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'} > {code} > Cassandra exception: > {code} > WARN [SharedPool-Worker-2] 2016-08-31 12:49:27,979 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-2,5,main]: {} > java.lang.RuntimeException: > org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2453) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_72] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) > [apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.0.8.jar:3.0.8] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:343) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:65) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:66) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:62) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:96) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:295) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:134) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:127) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:123) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:289) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at >
[jira] [Commented] (CASSANDRA-12582) Removing static column results in ReadFailure due to CorruptSSTableException
[ https://issues.apache.org/jira/browse/CASSANDRA-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516024#comment-15516024 ] Sylvain Lebresne commented on CASSANDRA-12582: -- bq. If we add a field to SystemKeyspace.DroppedColumns, and we are careful to read it as an optional field that may be missing, given that the table has local replication Don't get fooled by the keyspace replication strategy, schema tables *are* replicated, just by their own manual mechanism (see {{MigrationManager}} for the gory details). Adding a column to a schema table exposes us to the same problem than in CASSANDRA-12236 (and it's follow-up, CASSANDRA-12697). And while for the {{cdc}} column we have been able to use the {{cdc_enabled}} to "solve" this problem (basically saying, "keep cdc_enabled to false until fully upgraded"), we don't have such trick here. Don't get me wrong, we could probably come up with something along the line of CASSANDRA-12236 if we really wanted to, but there is a fair chance it won't be very user friendly, and it's a tad involved anyway. So let's say that it's a lot easier to make the change in 4.0, and that even if we think it's important enough to try it before that, we might still want to defer that change to a follow-up ticket just so we get your work-around in in the meantime. > Removing static column results in ReadFailure due to CorruptSSTableException > > > Key: CASSANDRA-12582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12582 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths > Environment: Cassandra 3.0.8 >Reporter: Evan Prothro >Assignee: Stefania >Priority: Critical > Labels: compaction, corruption, drop, read, static > Fix For: 3.0.x, 3.x > > Attachments: 12582.cdl, 12582_reproduce.sh > > > We ran into an issue on production where reads began to fail for certain > queries, depending on the range within the relation for those queries. > Cassandra system log showed an unhandled {{CorruptSSTableException}} > exception. > CQL read failure: > {code} > ReadFailure: code=1300 [Replica(s) failed to execute read] message="Operation > failed - received 0 responses and 1 failures" info={'failures': 1, > 'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'} > {code} > Cassandra exception: > {code} > WARN [SharedPool-Worker-2] 2016-08-31 12:49:27,979 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-2,5,main]: {} > java.lang.RuntimeException: > org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2453) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_72] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) > [apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.0.8.jar:3.0.8] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:343) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:65) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:66) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:62) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at >
[jira] [Commented] (CASSANDRA-12582) Removing static column results in ReadFailure due to CorruptSSTableException
[ https://issues.apache.org/jira/browse/CASSANDRA-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515835#comment-15515835 ] Stefania commented on CASSANDRA-12582: -- Thank you for your comment [~slebresne]. Do we need to wait until 4.0? If we add a field to {{SystemKeyspace.DroppedColumns}}, and we are careful to read it as an optional field that may be missing, given that the table has local replication, isn't this enough to cover all columns that will be dropped from this point onwards? For columns that have already been dropped, there's nothing we can do, even with a full schema migration we have lost the information of whether the column was static or not. Am I missing something? Noted regarding storing the full kind, and I agree. > Removing static column results in ReadFailure due to CorruptSSTableException > > > Key: CASSANDRA-12582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12582 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths > Environment: Cassandra 3.0.8 >Reporter: Evan Prothro >Assignee: Stefania >Priority: Critical > Labels: compaction, corruption, drop, read, static > Fix For: 3.0.x, 3.x > > Attachments: 12582.cdl, 12582_reproduce.sh > > > We ran into an issue on production where reads began to fail for certain > queries, depending on the range within the relation for those queries. > Cassandra system log showed an unhandled {{CorruptSSTableException}} > exception. > CQL read failure: > {code} > ReadFailure: code=1300 [Replica(s) failed to execute read] message="Operation > failed - received 0 responses and 1 failures" info={'failures': 1, > 'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'} > {code} > Cassandra exception: > {code} > WARN [SharedPool-Worker-2] 2016-08-31 12:49:27,979 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-2,5,main]: {} > java.lang.RuntimeException: > org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2453) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_72] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) > [apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.0.8.jar:3.0.8] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:343) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:65) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:66) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:62) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:96) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:295) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:134) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:127) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at >
[jira] [Commented] (CASSANDRA-12582) Removing static column results in ReadFailure due to CorruptSSTableException
[ https://issues.apache.org/jira/browse/CASSANDRA-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515762#comment-15515762 ] Stefania commented on CASSANDRA-12582: -- It's worth noting that the exception I reproduced is actually different that the one in the ticket description. It seems the exception in the description is failing to read a static row whilst the exception I reproduced is failing to read a regular row (a clustering to be exact), I hope the workaround in the serialization header covers both, but I cannot reproduce the exact exception in the description. {code} java.lang.RuntimeException: org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /home/stefi/git/cstar/cassandra/data/data/issue12582/apples_by_tree-3980ba90809c11e6b62e6be1d44ebd9b/mc-1-big-Data.db at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2470) ~[main/:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_101] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[main/:na] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) [main/:na] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [main/:na] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /home/stefi/git/cstar/cassandra/data/data/issue12582/apples_by_tree-3980ba90809c11e6b62e6be1d44ebd9b/mc-1-big-Data.db at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:353) ~[main/:na] at org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:219) ~[main/:na] at org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32) ~[main/:na] at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) ~[main/:na] at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) ~[main/:na] at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[main/:na] at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) ~[main/:na] at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) ~[main/:na] at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[main/:na] at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) ~[main/:na] at org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:58) ~[main/:na] at org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:65) ~[main/:na] at org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24) ~[main/:na] at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:96) ~[main/:na] at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:295) ~[main/:na] at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:145) ~[main/:na] at org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:138) ~[main/:na] at org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:134) ~[main/:na] at org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76) ~[main/:na] at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:320) ~[main/:na] at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1796) ~[main/:na] at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2466) ~[main/:na] ... 5 common frames omitted Caused by: java.io.IOException: Corrupt flags value for clustering prefix (isStatic flag set): 160 at org.apache.cassandra.db.ClusteringPrefix$Deserializer.prepare(ClusteringPrefix.java:422) ~[main/:na] at org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.prepareNext(UnfilteredDeserializer.java:172) ~[main/:na] at org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.hasNext(UnfilteredDeserializer.java:153) ~[main/:na] at
[jira] [Commented] (CASSANDRA-12582) Removing static column results in ReadFailure due to CorruptSSTableException
[ https://issues.apache.org/jira/browse/CASSANDRA-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515688#comment-15515688 ] Sylvain Lebresne commented on CASSANDRA-12582: -- bq. I wonder if we should also add {{ColumnDefinition.Kind}} to {{SystemKeyspace.DroppedColumns}} We definitively should imo. Though that imply a schema change and that's kind of problematic at the moment until 4.0, so we might have to do so in a separate ticket and rely on work-around like you implemented until then. bq. Currently only regular or static columns can be dropped, so technically we could also only add a boolean I can't really see us ever allowing the dropping of partition key or clustering columns (it sounds way too painful to support in the storage engine to be worth it), but I don't think we have much to lose by saving the full kind so it's probably fine to be overly cautious. > Removing static column results in ReadFailure due to CorruptSSTableException > > > Key: CASSANDRA-12582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12582 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths > Environment: Cassandra 3.0.8 >Reporter: Evan Prothro >Assignee: Stefania >Priority: Critical > Labels: compaction, corruption, drop, read, static > Fix For: 3.0.x, 3.x > > Attachments: 12582.cdl, 12582_reproduce.sh > > > We ran into an issue on production where reads began to fail for certain > queries, depending on the range within the relation for those queries. > Cassandra system log showed an unhandled {{CorruptSSTableException}} > exception. > CQL read failure: > {code} > ReadFailure: code=1300 [Replica(s) failed to execute read] message="Operation > failed - received 0 responses and 1 failures" info={'failures': 1, > 'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'} > {code} > Cassandra exception: > {code} > WARN [SharedPool-Worker-2] 2016-08-31 12:49:27,979 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-2,5,main]: {} > java.lang.RuntimeException: > org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2453) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_72] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) > [apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.0.8.jar:3.0.8] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:343) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:65) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:66) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:62) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:96) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:295) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:134) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at >
[jira] [Commented] (CASSANDRA-12582) Removing static column results in ReadFailure due to CorruptSSTableException
[ https://issues.apache.org/jira/browse/CASSANDRA-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515526#comment-15515526 ] Stefania commented on CASSANDRA-12582: -- This patch should fix the problem: ||3.0||trunk|| |[patch|https://github.com/stef1927/cassandra/commits/12582-3.0]|[patch|https://github.com/stef1927/cassandra/commits/12582]| |[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12582-3.0-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12582-testall/]| |[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12582-3.0-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-12582-dtest/]| Luckily the serialization header stores static and regular columns separately, so I was able to pass this information to {{CFMetadata}} in order to create a fake static column but I wonder if we may have issues in other parts of the code, where we create a regular column def instead of a static one. Basically all callers of [{{CFMetadata.getDroppedColumnDefinition}}|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/config/CFMetaData.java#L672] are at risk. So I wonder if we should also add {{ColumnDefinition.Kind}} to {{SystemKeyspace.DroppedColumns}}. Currently only regular or static columns can be dropped, so technically we could also only add a boolean. WDYT [~iamaleksey]? > Removing static column results in ReadFailure due to CorruptSSTableException > > > Key: CASSANDRA-12582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12582 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths > Environment: Cassandra 3.0.8 >Reporter: Evan Prothro >Assignee: Stefania >Priority: Critical > Labels: compaction, corruption, drop, read, static > Fix For: 3.0.x, 3.x > > Attachments: 12582.cdl, 12582_reproduce.sh > > > We ran into an issue on production where reads began to fail for certain > queries, depending on the range within the relation for those queries. > Cassandra system log showed an unhandled {{CorruptSSTableException}} > exception. > CQL read failure: > {code} > ReadFailure: code=1300 [Replica(s) failed to execute read] message="Operation > failed - received 0 responses and 1 failures" info={'failures': 1, > 'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'} > {code} > Cassandra exception: > {code} > WARN [SharedPool-Worker-2] 2016-08-31 12:49:27,979 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-2,5,main]: {} > java.lang.RuntimeException: > org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2453) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_72] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) > [apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.0.8.jar:3.0.8] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:343) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:65) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:66) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:62) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at >
[jira] [Commented] (CASSANDRA-12582) Removing static column results in ReadFailure due to CorruptSSTableException
[ https://issues.apache.org/jira/browse/CASSANDRA-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512745#comment-15512745 ] Stefania commented on CASSANDRA-12582: -- The reason of the corruption is that the sstable iterators try to read a static column as a regular column. This is due to the fact that the serialization header is missing the dropped static column. When the header is deserialized, it relies on {{CFMetadata}} to provide a fake dropped column. The problem is that {{CFMetadata}} always assumes that dropped columns are regular, see [here|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/config/CFMetaData.java#L682]. Then when the iterators read the static column, they rely on the header to make the decision of whether a static column is present or not, [here|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/columniterator/AbstractSSTableIterator.java#L169]. As a consequence, they don't attempt to skip the static column and they try to read it as a regular column later on. > Removing static column results in ReadFailure due to CorruptSSTableException > > > Key: CASSANDRA-12582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12582 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths > Environment: Cassandra 3.0.8 >Reporter: Evan Prothro >Assignee: Stefania >Priority: Critical > Labels: compaction, corruption, drop, read, static > Fix For: 3.0.x, 3.x > > Attachments: 12582.cdl, 12582_reproduce.sh > > > We ran into an issue on production where reads began to fail for certain > queries, depending on the range within the relation for those queries. > Cassandra system log showed an unhandled {{CorruptSSTableException}} > exception. > CQL read failure: > {code} > ReadFailure: code=1300 [Replica(s) failed to execute read] message="Operation > failed - received 0 responses and 1 failures" info={'failures': 1, > 'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'} > {code} > Cassandra exception: > {code} > WARN [SharedPool-Worker-2] 2016-08-31 12:49:27,979 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-2,5,main]: {} > java.lang.RuntimeException: > org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2453) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_72] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) > [apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.0.8.jar:3.0.8] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:343) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:65) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:66) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:62) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:96) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:295) >
[jira] [Commented] (CASSANDRA-12582) Removing static column results in ReadFailure due to CorruptSSTableException
[ https://issues.apache.org/jira/browse/CASSANDRA-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512573#comment-15512573 ] Stefania commented on CASSANDRA-12582: -- Reproduced without problems. > Removing static column results in ReadFailure due to CorruptSSTableException > > > Key: CASSANDRA-12582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12582 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths > Environment: Cassandra 3.0.8 >Reporter: Evan Prothro >Assignee: Stefania >Priority: Critical > Labels: compaction, corruption, drop, read, static > Fix For: 3.0.x, 3.x > > Attachments: 12582.cdl, 12582_reproduce.sh > > > We ran into an issue on production where reads began to fail for certain > queries, depending on the range within the relation for those queries. > Cassandra system log showed an unhandled {{CorruptSSTableException}} > exception. > CQL read failure: > {code} > ReadFailure: code=1300 [Replica(s) failed to execute read] message="Operation > failed - received 0 responses and 1 failures" info={'failures': 1, > 'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'} > {code} > Cassandra exception: > {code} > WARN [SharedPool-Worker-2] 2016-08-31 12:49:27,979 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-2,5,main]: {} > java.lang.RuntimeException: > org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2453) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_72] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) > [apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.0.8.jar:3.0.8] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:343) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:65) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:66) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:62) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:96) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:295) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:134) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:127) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:123) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:289) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1796) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at >
[jira] [Commented] (CASSANDRA-12582) Removing static column results in ReadFailure due to CorruptSSTableException
[ https://issues.apache.org/jira/browse/CASSANDRA-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512556#comment-15512556 ] Stefania commented on CASSANDRA-12582: -- I'll have a go at reproducing this and see if I can understand what is going on. Thanks for providing such detailed information. > Removing static column results in ReadFailure due to CorruptSSTableException > > > Key: CASSANDRA-12582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12582 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths > Environment: Cassandra 3.0.8 >Reporter: Evan Prothro >Assignee: Stefania >Priority: Critical > Labels: compaction, corruption, drop, read, static > Fix For: 3.0.x, 3.x > > Attachments: 12582.cdl, 12582_reproduce.sh > > > We ran into an issue on production where reads began to fail for certain > queries, depending on the range within the relation for those queries. > Cassandra system log showed an unhandled {{CorruptSSTableException}} > exception. > CQL read failure: > {code} > ReadFailure: code=1300 [Replica(s) failed to execute read] message="Operation > failed - received 0 responses and 1 failures" info={'failures': 1, > 'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'} > {code} > Cassandra exception: > {code} > WARN [SharedPool-Worker-2] 2016-08-31 12:49:27,979 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-2,5,main]: {} > java.lang.RuntimeException: > org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2453) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_72] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) > [apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.0.8.jar:3.0.8] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:343) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:65) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:66) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:62) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:96) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:295) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:134) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:127) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:123) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:289) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1796) >
[jira] [Commented] (CASSANDRA-12582) Removing static column results in ReadFailure due to CorruptSSTableException
[ https://issues.apache.org/jira/browse/CASSANDRA-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506660#comment-15506660 ] T Jake Luciani commented on CASSANDRA-12582: [~eprothro] The versions get bumped when I do a release. We periodically do releases and 3.0.9 was long overdue (See changelog). I'm not sure who would be best to look into this issue but I'll ask if someone can help. thanks for the repro script! > Removing static column results in ReadFailure due to CorruptSSTableException > > > Key: CASSANDRA-12582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12582 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 3.0.8 >Reporter: Evan Prothro > Labels: compaction, corruption, drop, read, static > Fix For: 3.0.10 > > Attachments: 12582.cdl, 12582_reproduce.sh > > > We ran into an issue on production where reads began to fail for certain > queries, depending on the range within the relation for those queries. > Cassandra system log showed an unhandled {{CorruptSSTableException}} > exception. > CQL read failure: > {code} > ReadFailure: code=1300 [Replica(s) failed to execute read] message="Operation > failed - received 0 responses and 1 failures" info={'failures': 1, > 'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'} > {code} > Cassandra exception: > {code} > WARN [SharedPool-Worker-2] 2016-08-31 12:49:27,979 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-2,5,main]: {} > java.lang.RuntimeException: > org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2453) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_72] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) > [apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.0.8.jar:3.0.8] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:343) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:65) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:66) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:62) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:96) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:295) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:134) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:127) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:123) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:289) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1796) >
[jira] [Commented] (CASSANDRA-12582) Removing static column results in ReadFailure due to CorruptSSTableException
[ https://issues.apache.org/jira/browse/CASSANDRA-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506622#comment-15506622 ] Evan Prothro commented on CASSANDRA-12582: -- Just saw that 3.0.9 was released, so i guess ignore that question. Bummer that this didn't make it in. Is there a way to know when versions are planning on being released? Seems like theres a level of organizational conversation going on that I'm not aware of. > Removing static column results in ReadFailure due to CorruptSSTableException > > > Key: CASSANDRA-12582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12582 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 3.0.8 >Reporter: Evan Prothro > Labels: compaction, corruption, drop, read, static > Fix For: 3.0.10 > > Attachments: 12582.cdl, 12582_reproduce.sh > > > We ran into an issue on production where reads began to fail for certain > queries, depending on the range within the relation for those queries. > Cassandra system log showed an unhandled {{CorruptSSTableException}} > exception. > CQL read failure: > {code} > ReadFailure: code=1300 [Replica(s) failed to execute read] message="Operation > failed - received 0 responses and 1 failures" info={'failures': 1, > 'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'} > {code} > Cassandra exception: > {code} > WARN [SharedPool-Worker-2] 2016-08-31 12:49:27,979 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-2,5,main]: {} > java.lang.RuntimeException: > org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2453) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_72] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) > [apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.0.8.jar:3.0.8] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:343) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:65) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:66) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:62) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:96) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:295) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:134) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:127) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:123) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:289) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at >
[jira] [Commented] (CASSANDRA-12582) Removing static column results in ReadFailure due to CorruptSSTableException
[ https://issues.apache.org/jira/browse/CASSANDRA-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506606#comment-15506606 ] Evan Prothro commented on CASSANDRA-12582: -- [~tjake] I don't want to speak out of turn, but it seems to me that this not getting attention in 3.0.9 will leave that build without a complete fix to the known issues dealing with static columns (other related issues linked above). Is there any way this can get attention in 3.0.9? > Removing static column results in ReadFailure due to CorruptSSTableException > > > Key: CASSANDRA-12582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12582 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 3.0.8 >Reporter: Evan Prothro > Labels: compaction, corruption, drop, read, static > Fix For: 3.0.10 > > Attachments: 12582.cdl, 12582_reproduce.sh > > > We ran into an issue on production where reads began to fail for certain > queries, depending on the range within the relation for those queries. > Cassandra system log showed an unhandled {{CorruptSSTableException}} > exception. > CQL read failure: > {code} > ReadFailure: code=1300 [Replica(s) failed to execute read] message="Operation > failed - received 0 responses and 1 failures" info={'failures': 1, > 'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'} > {code} > Cassandra exception: > {code} > WARN [SharedPool-Worker-2] 2016-08-31 12:49:27,979 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-2,5,main]: {} > java.lang.RuntimeException: > org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2453) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_72] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) > [apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.0.8.jar:3.0.8] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:343) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:65) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:66) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:62) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:96) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:295) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:134) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:127) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:123) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:289) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at >
[jira] [Commented] (CASSANDRA-12582) Removing static column results in ReadFailure due to CorruptSSTableException
[ https://issues.apache.org/jira/browse/CASSANDRA-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15471888#comment-15471888 ] Evan Prothro commented on CASSANDRA-12582: -- If there is anything else needed to reproduce or understand this issue, please let us know. We're eager to help however we can to try to get this fixed for 3.0.9 to complete all the fixes related to issues with static columns in this issue, CASSANDRA-11988 and CASSANDRA-12336. > Removing static column results in ReadFailure due to CorruptSSTableException > > > Key: CASSANDRA-12582 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12582 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 3.0.8 >Reporter: Evan Prothro > Labels: compaction, corruption, drop, read, static > Fix For: 3.0.9 > > Attachments: 12582.cdl, 12582_reproduce.sh > > > We ran into an issue on production where reads began to fail for certain > queries, depending on the range within the relation for those queries. > Cassandra system log showed an unhandled {{CorruptSSTableException}} > exception. > CQL read failure: > {code} > ReadFailure: code=1300 [Replica(s) failed to execute read] message="Operation > failed - received 0 responses and 1 failures" info={'failures': 1, > 'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'} > {code} > Cassandra exception: > {code} > WARN [SharedPool-Worker-2] 2016-08-31 12:49:27,979 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-2,5,main]: {} > java.lang.RuntimeException: > org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2453) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_72] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) > [apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.0.8.jar:3.0.8] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: > Corrupted: > /usr/local/apache-cassandra-3.0.8/data/data/issue309/apples_by_tree-006748a06fa311e6a7f8ef8b642e977b/mb-1-big-Data.db > at > org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:343) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:65) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:66) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:62) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:96) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:295) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:134) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:127) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:123) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at > org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:289) > ~[apache-cassandra-3.0.8.jar:3.0.8] > at >