[ https://issues.apache.org/jira/browse/CASSANDRA-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16878488#comment-16878488 ]
Sam Tunnicliffe edited comment on CASSANDRA-15199 at 7/4/19 9:32 AM: --------------------------------------------------------------------- Just for reference, the stacktrace from the Kong issue. Looks like the {{PagingState}} contains a null partition key, so potentially an issue with the lua client, but I seem to recall seeing some similar stacktraces when looking into paging issues in mixed version (2.1/3.0) clusters during upgrades. {code} WARN [ReadStage-6] 2019-05-31 01:08:07,971 ReadCommand.java:569 - Read 1000 live rows and 1003 tombstone cells for query SELECT * FROM kong_dev.routes WHERE partition = routes LIMIT 1000 (see tombstone_warn_threshold) ERROR [Native-Transport-Requests-4] 2019-05-31 01:08:07,997 QueryMessage.java:129 - Unexpected error during query java.lang.NullPointerException: null at org.apache.cassandra.dht.Murmur3Partitioner.getHash(Murmur3Partitioner.java:230) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.dht.Murmur3Partitioner.decorateKey(Murmur3Partitioner.java:66) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.config.CFMetaData.decorateKey(CFMetaData.java:666) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.service.pager.PartitionRangeQueryPager.<init>(PartitionRangeQueryPager.java:44) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.db.PartitionRangeReadCommand.getPager(PartitionRangeReadCommand.java:269) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.statements.SelectStatement.getPager(SelectStatement.java:474) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:287) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:117) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:225) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:256) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:241) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:566) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.11.4.jar:3.11.4] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) [netty-all-4.0.44.Final.jar:4.0.44.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114) [apache-cassandra-3.11.4.jar:3.11.4] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181] ERROR [Native-Transport-Requests-4] 2019-05-31 01:08:07,997 ErrorMessage.java:384 - Unexpected exception during request java.lang.NullPointerException: null at org.apache.cassandra.dht.Murmur3Partitioner.getHash(Murmur3Partitioner.java:230) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.dht.Murmur3Partitioner.decorateKey(Murmur3Partitioner.java:66) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.config.CFMetaData.decorateKey(CFMetaData.java:666) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.service.pager.PartitionRangeQueryPager.<init>(PartitionRangeQueryPager.java:44) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.db.PartitionRangeReadCommand.getPager(PartitionRangeReadCommand.java:269) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.statements.SelectStatement.getPager(SelectStatement.java:474) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:287) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:117) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:225) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:256) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:241) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:566) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.11.4.jar:3.11.4] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) [netty-all-4.0.44.Final.jar:4.0.44.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114) [apache-cassandra-3.11.4.jar:3.11.4] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181] {code} was (Author: beobal): Just for reference, the stacktrace from the Kong issue. Looks like the {PagingState} contains a null partition key, so potentially an issue with the lua client, but I seem to recall seeing some similar stacktraces when looking into paging issues in mixed version (2.1/3.0) clusters during upgrades. {code} WARN [ReadStage-6] 2019-05-31 01:08:07,971 ReadCommand.java:569 - Read 1000 live rows and 1003 tombstone cells for query SELECT * FROM kong_dev.routes WHERE partition = routes LIMIT 1000 (see tombstone_warn_threshold) ERROR [Native-Transport-Requests-4] 2019-05-31 01:08:07,997 QueryMessage.java:129 - Unexpected error during query java.lang.NullPointerException: null at org.apache.cassandra.dht.Murmur3Partitioner.getHash(Murmur3Partitioner.java:230) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.dht.Murmur3Partitioner.decorateKey(Murmur3Partitioner.java:66) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.config.CFMetaData.decorateKey(CFMetaData.java:666) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.service.pager.PartitionRangeQueryPager.<init>(PartitionRangeQueryPager.java:44) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.db.PartitionRangeReadCommand.getPager(PartitionRangeReadCommand.java:269) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.statements.SelectStatement.getPager(SelectStatement.java:474) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:287) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:117) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:225) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:256) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:241) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:566) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.11.4.jar:3.11.4] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) [netty-all-4.0.44.Final.jar:4.0.44.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114) [apache-cassandra-3.11.4.jar:3.11.4] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181] ERROR [Native-Transport-Requests-4] 2019-05-31 01:08:07,997 ErrorMessage.java:384 - Unexpected exception during request java.lang.NullPointerException: null at org.apache.cassandra.dht.Murmur3Partitioner.getHash(Murmur3Partitioner.java:230) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.dht.Murmur3Partitioner.decorateKey(Murmur3Partitioner.java:66) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.config.CFMetaData.decorateKey(CFMetaData.java:666) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.service.pager.PartitionRangeQueryPager.<init>(PartitionRangeQueryPager.java:44) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.db.PartitionRangeReadCommand.getPager(PartitionRangeReadCommand.java:269) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.statements.SelectStatement.getPager(SelectStatement.java:474) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:287) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:117) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:225) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:256) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:241) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:566) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.11.4.jar:3.11.4] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.44.Final.jar:4.0.44.Final] at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) [netty-all-4.0.44.Final.jar:4.0.44.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114) [apache-cassandra-3.11.4.jar:3.11.4] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181] {code} > Cassandra throwing occasional NPE 3.11.x > ---------------------------------------- > > Key: CASSANDRA-15199 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15199 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Client > Reporter: Jeremy Justus > Priority: Normal > > Hey folks, decided to raise an official Jira(never done one of these before) > about an issue we have found between Kong API Gateway leveraging Cassandra as > our db. > We run a C* cluster in 2 close DCs, 6 C* nodes total, 3 in each DC. Data > replicated across all nodes. We have found C* sometimes throws NPEs based on > the calls made by the lua-cassandra driver Kong > leverages([https://github.com/thibaultcha/lua-cassandra]). Very specifically > it seems to occur when attempting to do paging across multiple C* nodes. When > persistently paging to a single C* node we can't reproduce NPEs in C*. > The exact Error C* throws with its stack-trace can be seen here: > [https://github.com/Kong/kong/issues/4194#issuecomment-497572751] > And again when we tried to upgrade from 3.11.2 to 3.11.4 in hopes it was > already resolved: > [https://github.com/Kong/kong/issues/4194#issuecomment-497590235] > Same error same line numbers, so code must be same in this portion. > > Sample of our C* Config: > [https://github.com/Kong/kong/issues/4194#issuecomment-497595766] > We discussed this in the ASF Slack flow as well: > ASF Slack discussion > ([https://the-asf.slack.com/archives/CJZLTM05A/p1559321422028200] ) > Some of the more important technical comments I saw people posting here: > [https://github.com/Kong/kong/issues/4194#issuecomment-497858824] > > I am not a C* DBA so I don't have an exact repro for you other than stating > what the client application was attempting to do(Paging across multiple C* > nodes within a DC) when we could see the failures. Any Apache C* folk think > they see the issue or could drop me C* JAR with extra debugging print > statements I could run in dev to help feed you more info? Or if you see the > problem and can one shot a fix so no more NPE and Cassandra responds > appropriately to the client with some sort of error message around what was > wrong that would be insightful. > > Thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org