Re: Cassandra failure during read query at consistency QUORUM (2 responses were required but only 0 replica responded, 2 failed)
Why does it prevent quorum? How to fix it? > On 28 Oct 2016, at 16:02, Edward Capriolo wrote: > > This looks like another case of an assert bubbling through try catch that > don't catch assert > > On Fri, Oct 28, 2016 at 6:30 AM, Denis Mikhaylov wrote: > Hi! > > We’re running Cassandra 3.9 > > On the application side I see failed reads with this exception > com.datastax.driver.core.exceptions.ReadFailureException: Cassandra failure > during read query at consistency QUORUM (2 responses were required but only 0 > replica responded, 2 failed) > > On the server side we see: > > WARN [SharedPool-Worker-3] 2016-10-28 13:28:22,965 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-3,5, > main]: {} > java.lang.AssertionError: null > at org.apache.cassandra.db.rows.BTreeRow.getCell(BTreeRow.java:212) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.db.SinglePartitionReadCommand.canRemoveRow(SinglePartitionReadCommand.java:899) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.db.SinglePartitionReadCommand.reduceFilter(SinglePartitionReadCommand.java:863) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndSSTablesInTimestampOrder(SinglePartitionReadCommand.java:748) > ~[apache-cassan > dra-3.7.jar:3.7] > at > org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:519) > ~[apache-cassandra-3.7.jar: > 3.7] > at > org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:496) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:358) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:366) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:48) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_102] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > ~[apache-cassandra- > 3.7.jar:3.7] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) > [apache > -cassandra-3.7.jar:3.7] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.7.jar:3.7] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102] > > It’s only affect single table. Sadly both on test (3.9) and production (3.7) > deployments of cassandra. > > What could be the problem? Please help. >
Re: Cassandra failure during read query at consistency QUORUM (2 responses were required but only 0 replica responded, 2 failed)
This looks like another case of an assert bubbling through try catch that don't catch assert On Fri, Oct 28, 2016 at 6:30 AM, Denis Mikhaylov wrote: > Hi! > > We’re running Cassandra 3.9 > > On the application side I see failed reads with this exception > com.datastax.driver.core.exceptions.ReadFailureException: Cassandra > failure during read query at consistency QUORUM (2 responses were required > but only 0 replica responded, 2 failed) > > On the server side we see: > > WARN [SharedPool-Worker-3] 2016-10-28 13:28:22,965 > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread > Thread[SharedPool-Worker-3,5, > main]: {} > java.lang.AssertionError: null > at org.apache.cassandra.db.rows.BTreeRow.getCell(BTreeRow.java:212) > ~[apache-cassandra-3.7.jar:3.7] > at org.apache.cassandra.db.SinglePartitionReadCommand. > canRemoveRow(SinglePartitionReadCommand.java:899) > ~[apache-cassandra-3.7.jar:3.7] > at org.apache.cassandra.db.SinglePartitionReadCommand. > reduceFilter(SinglePartitionReadCommand.java:863) > ~[apache-cassandra-3.7.jar:3.7] > at org.apache.cassandra.db.SinglePartitionReadCommand. > queryMemtableAndSSTablesInTimestampOrder(SinglePartitionReadCommand.java:748) > ~[apache-cassan > dra-3.7.jar:3.7] > at org.apache.cassandra.db.SinglePartitionReadCommand. > queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:519) > ~[apache-cassandra-3.7.jar: > 3.7] > at org.apache.cassandra.db.SinglePartitionReadCommand. > queryMemtableAndDisk(SinglePartitionReadCommand.java:496) > ~[apache-cassandra-3.7.jar:3.7] > at org.apache.cassandra.db.SinglePartitionReadCommand. > queryStorage(SinglePartitionReadCommand.java:358) > ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:366) > ~[apache-cassandra-3.7.jar:3.7] > at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb( > ReadCommandVerbHandler.java:48) ~[apache-cassandra-3.7.jar:3.7] > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) > ~[apache-cassandra-3.7.jar:3.7] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[na:1.8.0_102] > at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorServ > ice$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > ~[apache-cassandra- > 3.7.jar:3.7] > at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorServ > ice$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) > [apache > -cassandra-3.7.jar:3.7] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-3.7.jar:3.7] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102] > > It’s only affect single table. Sadly both on test (3.9) and production > (3.7) deployments of cassandra. > > What could be the problem? Please help.