[ https://issues.apache.org/jira/browse/CASSANDRA-12152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kurt Greaves updated CASSANDRA-12152: ------------------------------------- Component/s: Materialized Views > Unknown exception caught while attempting to update MaterializedView: > AssertionError: Flags = 128 > ------------------------------------------------------------------------------------------------- > > Key: CASSANDRA-12152 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12152 > Project: Cassandra > Issue Type: Bug > Components: Materialized Views > Reporter: Nilson Pontello > > I have a single DC with 3 cassandra nodes. After a restart today, none of > them were capable of processing the commitlog while starting up. The > exception doesn't contains enough information about what is going on, please > check bellow: > {code} > ERROR [SharedPool-Worker-21] 2016-07-08 12:42:12,866 Keyspace.java:521 - > Unknown exception caught while attempting to update MaterializedView! > data_monitor.user_timeline > java.lang.AssertionError: Flags = 128 > at > org.apache.cassandra.db.ClusteringPrefix$Deserializer.prepare(ClusteringPrefix.java:421) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.prepareNext(UnfilteredDeserializer.java:172) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.hasNext(UnfilteredDeserializer.java:153) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.handlePreSliceData(SSTableIterator.java:96) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:141) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:354) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:229) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:93) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:25) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:70) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.SinglePartitionReadCommand.withSSTablesIterated(SinglePartitionReadCommand.java:637) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:586) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:463) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:325) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:366) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.ReadCommand.executeInternal(ReadCommand.java:397) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.service.pager.AbstractQueryPager.fetchPageInternal(AbstractQueryPager.java:77) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.service.pager.SinglePartitionPager.fetchPageInternal(SinglePartitionPager.java:34) > ~[apache-cassandra-3.5.jar:3.5] > at org.apache.cassandra.db.view.View.readLocalRows(View.java:565) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.view.View.getTemporalRowSet(View.java:621) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.view.ViewManager.pushViewReplicaUpdates(ViewManager.java:136) > ~[apache-cassandra-3.5.jar:3.5] > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:516) > [apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.Keyspace.applyFromCommitLog(Keyspace.java:409) > [apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.commitlog.CommitLogReplayer$MutationInitiator$1.runMayThrow(CommitLogReplayer.java:157) > [apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > [apache-cassandra-3.5.jar:3.5] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_91] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > [apache-cassandra-3.5.jar:3.5] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.5.jar:3.5] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91] > WARN [SharedPool-Worker-21] 2016-07-08 12:42:12,867 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-21,5,main]: {} > java.lang.AssertionError: Flags = 128 > at > org.apache.cassandra.db.ClusteringPrefix$Deserializer.prepare(ClusteringPrefix.java:421) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.prepareNext(UnfilteredDeserializer.java:172) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.hasNext(UnfilteredDeserializer.java:153) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.handlePreSliceData(SSTableIterator.java:96) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:141) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:354) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:229) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:93) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:25) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:70) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.SinglePartitionReadCommand.withSSTablesIterated(SinglePartitionReadCommand.java:637) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:586) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:463) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:325) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:366) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.ReadCommand.executeInternal(ReadCommand.java:397) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.service.pager.AbstractQueryPager.fetchPageInternal(AbstractQueryPager.java:77) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.service.pager.SinglePartitionPager.fetchPageInternal(SinglePartitionPager.java:34) > ~[apache-cassandra-3.5.jar:3.5] > at org.apache.cassandra.db.view.View.readLocalRows(View.java:565) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.view.View.getTemporalRowSet(View.java:621) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.view.ViewManager.pushViewReplicaUpdates(ViewManager.java:136) > ~[apache-cassandra-3.5.jar:3.5] > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:516) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.Keyspace.applyFromCommitLog(Keyspace.java:409) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.commitlog.CommitLogReplayer$MutationInitiator$1.runMayThrow(CommitLogReplayer.java:157) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > ~[apache-cassandra-3.5.jar:3.5] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_91] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > ~[apache-cassandra-3.5.jar:3.5] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.5.jar:3.5] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91] > ERROR [main] 2016-07-08 12:42:12,875 CassandraDaemon.java:697 - Exception > encountered during startup > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.lang.AssertionError: Flags = 128 > at > org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:384) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.replayMutation(CommitLogReplayer.java:592) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:521) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:407) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:236) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:192) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:172) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:283) > [apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551) > [apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:680) > [apache-cassandra-3.5.jar:3.5] > Caused by: java.util.concurrent.ExecutionException: > java.lang.AssertionError: Flags = 128 > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.get(AbstractLocalAwareExecutorService.java:200) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:380) > ~[apache-cassandra-3.5.jar:3.5] > ... 9 common frames omitted > Caused by: java.lang.AssertionError: Flags = 128 > at > org.apache.cassandra.db.ClusteringPrefix$Deserializer.prepare(ClusteringPrefix.java:421) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.prepareNext(UnfilteredDeserializer.java:172) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.hasNext(UnfilteredDeserializer.java:153) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.handlePreSliceData(SSTableIterator.java:96) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:141) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:354) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:229) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:93) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:25) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:419) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:279) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:70) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.SinglePartitionReadCommand.withSSTablesIterated(SinglePartitionReadCommand.java:637) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:586) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:463) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:325) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:366) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.ReadCommand.executeInternal(ReadCommand.java:397) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.service.pager.AbstractQueryPager.fetchPageInternal(AbstractQueryPager.java:77) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.service.pager.SinglePartitionPager.fetchPageInternal(SinglePartitionPager.java:34) > ~[apache-cassandra-3.5.jar:3.5] > at org.apache.cassandra.db.view.View.readLocalRows(View.java:565) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.view.View.getTemporalRowSet(View.java:621) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.view.ViewManager.pushViewReplicaUpdates(ViewManager.java:136) > ~[apache-cassandra-3.5.jar:3.5] > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:516) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.Keyspace.applyFromCommitLog(Keyspace.java:409) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.db.commitlog.CommitLogReplayer$MutationInitiator$1.runMayThrow(CommitLogReplayer.java:157) > ~[apache-cassandra-3.5.jar:3.5] > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > ~[apache-cassandra-3.5.jar:3.5] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_91] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > ~[apache-cassandra-3.5.jar:3.5] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > ~[apache-cassandra-3.5.jar:3.5] > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_91] > {code} > I have played with *commit_failure_policy* hoping that this could be ignored > but it wasn't. I don't mind loosing this materialized view, I can create it > over again if needed. > I am opening this bug because the error message is not informative and I have > no clues of what should be done in order to put my cluster live again... > Also, my system administrator did stop cassandra without running the bellow > commands: > {code} > nodetool disablethrift > nodetool disablegossip > nodetool drain > {code} > I am not sure if this could have affected the pending commitlogs somehow. > Any help will be appreciated -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org