[jira] [Created] (CASSANDRA-15172) AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4
Shalom created CASSANDRA-15172: -- Summary: AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4 Key: CASSANDRA-15172 URL: https://issues.apache.org/jira/browse/CASSANDRA-15172 Project: Cassandra Issue Type: Bug Reporter: Shalom Hi All, This is the first time I open an issue, so apologies if I'm not following the rules properly. After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a lot of AbstractLocalAwareExecutorService exceptions. This happened right after the node successfully started up with the new 3.11.4 binaries. INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog; proceeding INFO [main] 2019-06-05 04:41:38,036 NativeTransportService.java:70 - Netty using native Epoll event loop INFO [main] 2019-06-05 04:41:38,117 Server.java:155 - Using Netty Version: [netty-buffer=netty-buffer-4.0.44.Final.452812a, netty-codec=netty-codec-4.0.44.Final.452812a, netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a, netty-codec-http=netty-codec-http-4.0.44.Final.452812a, netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a, netty-common=netty-common-4.0.44.Final.452812a, netty-handler=netty-handler-4.0.44.Final.452812a, netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb, netty-transport=netty-transport-4.0.44.Final.452812a, netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a, netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a, netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a, netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a] INFO [main] 2019-06-05 04:41:38,118 Server.java:156 - Starting listening for CQL clients on /0.0.0.0:9042 (unencrypted)... INFO [main] 2019-06-05 04:41:38,179 CassandraDaemon.java:556 - Not starting RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool (enablethrift) to start it INFO [Native-Transport-Requests-21] 2019-06-05 04:41:39,145 AuthCache.java:161 - (Re)initializing PermissionsCache (validity period/update interval/max entries) (2000/2000/1000) INFO [OptionalTasks:1] 2019-06-05 04:41:39,729 CassandraAuthorizer.java:409 - Converting legacy permissions data INFO [HANDSHAKE-/10.10.10.8] 2019-06-05 04:41:39,808 OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.8 INFO [HANDSHAKE-/10.10.10.9] 2019-06-05 04:41:39,808 OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.9 INFO [HANDSHAKE-dc1_02/10.10.10.6] 2019-06-05 04:41:39,809 OutboundTcpConnection.java:561 - Handshaking version with dc1_02/10.10.10.6 WARN [ReadStage-2] 2019-06-05 04:41:39,857 AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread Thread[ReadStage-2,5,main]: {} java.lang.ArrayIndexOutOfBoundsException: 1 at org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55) at org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545) at org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522) at org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565) at org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446) at org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352) at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171) at org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77) at org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802) at org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953) at org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929) at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134) at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114) at java.lang.Thread.run(Thread.java:745) After several of the above warnings, the following warning appeared as well: WARN [ReadStage-9] 2019-06-05 04:42:04,369 AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread Thread[ReadStage-9,5,main]: {} java.lang.ArrayIndexOutOfBoundsException: null WA
[jira] [Commented] (CASSANDRA-15172) AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4
[ https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16888040#comment-16888040 ] Shalom commented on CASSANDRA-15172: Trying to push this up a bit because I still want to upgrade to 3.11.4 but I fear that this issue may recur. If someone has any idea on what happened here or how to mitigate it, that'd be awesome! Thanks! > AbstractLocalAwareExecutorService Exception During Upgrade to 3.11.4 > > > Key: CASSANDRA-15172 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15172 > Project: Cassandra > Issue Type: Bug >Reporter: Shalom >Priority: Normal > > Hi All, > This is the first time I open an issue, so apologies if I'm not following the > rules properly. > > After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a > lot of AbstractLocalAwareExecutorService exceptions. This happened right > after the node successfully started up with the new 3.11.4 binaries. > INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog; > proceeding > INFO [main] 2019-06-05 04:41:38,036 NativeTransportService.java:70 - Netty > using native Epoll event loop > INFO [main] 2019-06-05 04:41:38,117 Server.java:155 - Using Netty Version: > [netty-buffer=netty-buffer-4.0.44.Final.452812a, > netty-codec=netty-codec-4.0.44.Final.452812a, > netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a, > netty-codec-http=netty-codec-http-4.0.44.Final.452812a, > netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a, > netty-common=netty-common-4.0.44.Final.452812a, > netty-handler=netty-handler-4.0.44.Final.452812a, > netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb, > netty-transport=netty-transport-4.0.44.Final.452812a, > netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a, > netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a, > netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a, > netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a] > INFO [main] 2019-06-05 04:41:38,118 Server.java:156 - Starting listening for > CQL clients on /0.0.0.0:9042 (unencrypted)... > INFO [main] 2019-06-05 04:41:38,179 CassandraDaemon.java:556 - Not starting > RPC server as requested. Use JMX (StorageService->startRPCServer()) or > nodetool (enablethrift) to start it > INFO [Native-Transport-Requests-21] 2019-06-05 04:41:39,145 > AuthCache.java:161 - (Re)initializing PermissionsCache (validity > period/update interval/max entries) (2000/2000/1000) > INFO [OptionalTasks:1] 2019-06-05 04:41:39,729 CassandraAuthorizer.java:409 > - Converting legacy permissions data > INFO [HANDSHAKE-/10.10.10.8] 2019-06-05 04:41:39,808 > OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.8 > INFO [HANDSHAKE-/10.10.10.9] 2019-06-05 04:41:39,808 > OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.9 > INFO [HANDSHAKE-dc1_02/10.10.10.6] 2019-06-05 04:41:39,809 > OutboundTcpConnection.java:561 - Handshaking version with dc1_02/10.10.10.6 > WARN [ReadStage-2] 2019-06-05 04:41:39,857 > AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread > Thread[ReadStage-2,5,main]: {} > java.lang.ArrayIndexOutOfBoundsException: 1 > at > org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55) > at > org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545) > at > org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522) > at > org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565) > at > org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446) > at > org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352) > at > org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171) > at > org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77) > at > org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802) > at > org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953) > at > org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929) > at > org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at > org.apache.cassandra.concurrent.AbstractLocalAwareE
[jira] [Commented] (CASSANDRA-15172) LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16915197#comment-16915197 ] Shalom commented on CASSANDRA-15172: Hi [~benedict] Sorry for the late reply... Could you also confirm if you utilise thrift, or CQL schema? It's possible this is a compatibility issue specific to thrift. I do see there are a few thrift counter tables that were created with the WITH COMPACT STORAGE attribute. I also see that [~ferozshaik...@gmail.com] had seen WARN [ReadStage-4] 2019-08-06 02:57:57,408 AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread Thread[ReadStage-4,5,main]: {} java.lang.NullPointerException: null What I mainly saw was AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread Thread[ReadStage-9,5,main]: {} java.lang.ArrayIndexOutOfBoundsException: null Does the fix fit the same scenario as Feroz's? Could my issue be thrift related? Sorry again for the late reply. > LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException > > > Key: CASSANDRA-15172 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15172 > Project: Cassandra > Issue Type: Bug > Components: Local/Other >Reporter: Shalom >Assignee: Benedict >Priority: Normal > Fix For: 3.0.19, 3.11.5 > > > Hi All, > This is the first time I open an issue, so apologies if I'm not following the > rules properly. > > After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a > lot of AbstractLocalAwareExecutorService exceptions. This happened right > after the node successfully started up with the new 3.11.4 binaries. > {noformat} > INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog; > proceeding > INFO [main] 2019-06-05 04:41:38,036 NativeTransportService.java:70 - Netty > using native Epoll event loop > INFO [main] 2019-06-05 04:41:38,117 Server.java:155 - Using Netty Version: > [netty-buffer=netty-buffer-4.0.44.Final.452812a, > netty-codec=netty-codec-4.0.44.Final.452812a, > netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a, > netty-codec-http=netty-codec-http-4.0.44.Final.452812a, > netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a, > netty-common=netty-common-4.0.44.Final.452812a, > netty-handler=netty-handler-4.0.44.Final.452812a, > netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb, > netty-transport=netty-transport-4.0.44.Final.452812a, > netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a, > netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a, > netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a, > netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a] > INFO [main] 2019-06-05 04:41:38,118 Server.java:156 - Starting listening for > CQL clients on /0.0.0.0:9042 (unencrypted)... > INFO [main] 2019-06-05 04:41:38,179 CassandraDaemon.java:556 - Not starting > RPC server as requested. Use JMX (StorageService->startRPCServer()) or > nodetool (enablethrift) to start it > INFO [Native-Transport-Requests-21] 2019-06-05 04:41:39,145 > AuthCache.java:161 - (Re)initializing PermissionsCache (validity > period/update interval/max entries) (2000/2000/1000) > INFO [OptionalTasks:1] 2019-06-05 04:41:39,729 CassandraAuthorizer.java:409 > - Converting legacy permissions data > INFO [HANDSHAKE-/10.10.10.8] 2019-06-05 04:41:39,808 > OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.8 > INFO [HANDSHAKE-/10.10.10.9] 2019-06-05 04:41:39,808 > OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.9 > INFO [HANDSHAKE-dc1_02/10.10.10.6] 2019-06-05 04:41:39,809 > OutboundTcpConnection.java:561 - Handshaking version with dc1_02/10.10.10.6 > WARN [ReadStage-2] 2019-06-05 04:41:39,857 > AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread > Thread[ReadStage-2,5,main]: {} > java.lang.ArrayIndexOutOfBoundsException: 1 > at > org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55) > at > org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545) > at > org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522) > at > org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565) > at > org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446) > at > org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352) > at > org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171) > at > org.apache.cassa
[jira] [Commented] (CASSANDRA-15172) LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16916910#comment-16916910 ] Shalom commented on CASSANDRA-15172: Hi [~benedict] , I'm not sure if my comments/tickets are somehow not getting filed, but I hope not :) Is the above error (previous comment) related to this bug or is it a different one? Thanks! > LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException > > > Key: CASSANDRA-15172 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15172 > Project: Cassandra > Issue Type: Bug > Components: Local/Other >Reporter: Shalom >Assignee: Benedict >Priority: Normal > Fix For: 3.0.19, 3.11.5 > > > Hi All, > This is the first time I open an issue, so apologies if I'm not following the > rules properly. > > After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a > lot of AbstractLocalAwareExecutorService exceptions. This happened right > after the node successfully started up with the new 3.11.4 binaries. > {noformat} > INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog; > proceeding > INFO [main] 2019-06-05 04:41:38,036 NativeTransportService.java:70 - Netty > using native Epoll event loop > INFO [main] 2019-06-05 04:41:38,117 Server.java:155 - Using Netty Version: > [netty-buffer=netty-buffer-4.0.44.Final.452812a, > netty-codec=netty-codec-4.0.44.Final.452812a, > netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a, > netty-codec-http=netty-codec-http-4.0.44.Final.452812a, > netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a, > netty-common=netty-common-4.0.44.Final.452812a, > netty-handler=netty-handler-4.0.44.Final.452812a, > netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb, > netty-transport=netty-transport-4.0.44.Final.452812a, > netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a, > netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a, > netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a, > netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a] > INFO [main] 2019-06-05 04:41:38,118 Server.java:156 - Starting listening for > CQL clients on /0.0.0.0:9042 (unencrypted)... > INFO [main] 2019-06-05 04:41:38,179 CassandraDaemon.java:556 - Not starting > RPC server as requested. Use JMX (StorageService->startRPCServer()) or > nodetool (enablethrift) to start it > INFO [Native-Transport-Requests-21] 2019-06-05 04:41:39,145 > AuthCache.java:161 - (Re)initializing PermissionsCache (validity > period/update interval/max entries) (2000/2000/1000) > INFO [OptionalTasks:1] 2019-06-05 04:41:39,729 CassandraAuthorizer.java:409 > - Converting legacy permissions data > INFO [HANDSHAKE-/10.10.10.8] 2019-06-05 04:41:39,808 > OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.8 > INFO [HANDSHAKE-/10.10.10.9] 2019-06-05 04:41:39,808 > OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.9 > INFO [HANDSHAKE-dc1_02/10.10.10.6] 2019-06-05 04:41:39,809 > OutboundTcpConnection.java:561 - Handshaking version with dc1_02/10.10.10.6 > WARN [ReadStage-2] 2019-06-05 04:41:39,857 > AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread > Thread[ReadStage-2,5,main]: {} > java.lang.ArrayIndexOutOfBoundsException: 1 > at > org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55) > at > org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545) > at > org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522) > at > org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565) > at > org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446) > at > org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352) > at > org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171) > at > org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77) > at > org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802) > at > org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953) > at > org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929) > at > org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) > at > java.util.concurrent.Executors$Runnab
[jira] [Commented] (CASSANDRA-15172) LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16916953#comment-16916953 ] Shalom commented on CASSANDRA-15172: Thanks for clarifying [~michaelsembwever]! > LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException > > > Key: CASSANDRA-15172 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15172 > Project: Cassandra > Issue Type: Bug > Components: Local/Other >Reporter: Shalom >Assignee: Benedict >Priority: Normal > Fix For: 3.0.19, 3.11.5 > > > Hi All, > This is the first time I open an issue, so apologies if I'm not following the > rules properly. > > After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a > lot of AbstractLocalAwareExecutorService exceptions. This happened right > after the node successfully started up with the new 3.11.4 binaries. > {noformat} > INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog; > proceeding > INFO [main] 2019-06-05 04:41:38,036 NativeTransportService.java:70 - Netty > using native Epoll event loop > INFO [main] 2019-06-05 04:41:38,117 Server.java:155 - Using Netty Version: > [netty-buffer=netty-buffer-4.0.44.Final.452812a, > netty-codec=netty-codec-4.0.44.Final.452812a, > netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a, > netty-codec-http=netty-codec-http-4.0.44.Final.452812a, > netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a, > netty-common=netty-common-4.0.44.Final.452812a, > netty-handler=netty-handler-4.0.44.Final.452812a, > netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb, > netty-transport=netty-transport-4.0.44.Final.452812a, > netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a, > netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a, > netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a, > netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a] > INFO [main] 2019-06-05 04:41:38,118 Server.java:156 - Starting listening for > CQL clients on /0.0.0.0:9042 (unencrypted)... > INFO [main] 2019-06-05 04:41:38,179 CassandraDaemon.java:556 - Not starting > RPC server as requested. Use JMX (StorageService->startRPCServer()) or > nodetool (enablethrift) to start it > INFO [Native-Transport-Requests-21] 2019-06-05 04:41:39,145 > AuthCache.java:161 - (Re)initializing PermissionsCache (validity > period/update interval/max entries) (2000/2000/1000) > INFO [OptionalTasks:1] 2019-06-05 04:41:39,729 CassandraAuthorizer.java:409 > - Converting legacy permissions data > INFO [HANDSHAKE-/10.10.10.8] 2019-06-05 04:41:39,808 > OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.8 > INFO [HANDSHAKE-/10.10.10.9] 2019-06-05 04:41:39,808 > OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.9 > INFO [HANDSHAKE-dc1_02/10.10.10.6] 2019-06-05 04:41:39,809 > OutboundTcpConnection.java:561 - Handshaking version with dc1_02/10.10.10.6 > WARN [ReadStage-2] 2019-06-05 04:41:39,857 > AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread > Thread[ReadStage-2,5,main]: {} > java.lang.ArrayIndexOutOfBoundsException: 1 > at > org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55) > at > org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545) > at > org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522) > at > org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565) > at > org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446) > at > org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352) > at > org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171) > at > org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77) > at > org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802) > at > org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953) > at > org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929) > at > org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorS
[jira] [Commented] (CASSANDRA-15172) LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16918545#comment-16918545 ] Shalom commented on CASSANDRA-15172: Thanks a lot for further clarifying [~benedict]. (I hope you enjoyed your vacation :) ) Just to set my mind straight, the issue is when there are mixed versions in the cluster, so if I upgraded all binaries to 3.11, it won't recur even if I haven't upgraded the SSTables yet. Is my assumption correct? > LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException > > > Key: CASSANDRA-15172 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15172 > Project: Cassandra > Issue Type: Bug > Components: Local/Other >Reporter: Shalom >Assignee: Benedict >Priority: Normal > Fix For: 3.0.19, 3.11.5 > > > Hi All, > This is the first time I open an issue, so apologies if I'm not following the > rules properly. > > After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a > lot of AbstractLocalAwareExecutorService exceptions. This happened right > after the node successfully started up with the new 3.11.4 binaries. > {noformat} > INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog; > proceeding > INFO [main] 2019-06-05 04:41:38,036 NativeTransportService.java:70 - Netty > using native Epoll event loop > INFO [main] 2019-06-05 04:41:38,117 Server.java:155 - Using Netty Version: > [netty-buffer=netty-buffer-4.0.44.Final.452812a, > netty-codec=netty-codec-4.0.44.Final.452812a, > netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a, > netty-codec-http=netty-codec-http-4.0.44.Final.452812a, > netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a, > netty-common=netty-common-4.0.44.Final.452812a, > netty-handler=netty-handler-4.0.44.Final.452812a, > netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb, > netty-transport=netty-transport-4.0.44.Final.452812a, > netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a, > netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a, > netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a, > netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a] > INFO [main] 2019-06-05 04:41:38,118 Server.java:156 - Starting listening for > CQL clients on /0.0.0.0:9042 (unencrypted)... > INFO [main] 2019-06-05 04:41:38,179 CassandraDaemon.java:556 - Not starting > RPC server as requested. Use JMX (StorageService->startRPCServer()) or > nodetool (enablethrift) to start it > INFO [Native-Transport-Requests-21] 2019-06-05 04:41:39,145 > AuthCache.java:161 - (Re)initializing PermissionsCache (validity > period/update interval/max entries) (2000/2000/1000) > INFO [OptionalTasks:1] 2019-06-05 04:41:39,729 CassandraAuthorizer.java:409 > - Converting legacy permissions data > INFO [HANDSHAKE-/10.10.10.8] 2019-06-05 04:41:39,808 > OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.8 > INFO [HANDSHAKE-/10.10.10.9] 2019-06-05 04:41:39,808 > OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.9 > INFO [HANDSHAKE-dc1_02/10.10.10.6] 2019-06-05 04:41:39,809 > OutboundTcpConnection.java:561 - Handshaking version with dc1_02/10.10.10.6 > WARN [ReadStage-2] 2019-06-05 04:41:39,857 > AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread > Thread[ReadStage-2,5,main]: {} > java.lang.ArrayIndexOutOfBoundsException: 1 > at > org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55) > at > org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545) > at > org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522) > at > org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565) > at > org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446) > at > org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352) > at > org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171) > at > org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77) > at > org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802) > at > org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953) > at > org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929) > at > org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62) > at > org.apache.cassandra.net.Me
[jira] [Commented] (CASSANDRA-15172) LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16936164#comment-16936164 ] Shalom commented on CASSANDRA-15172: Hi [~benedict] I tried the patch. It didn't do the trick, however, I was able to fully reproduce the bug. tl;dr The bug does occur when running queries on range tombstones, but on top of that, those queries have to specifically be range queries. *+Steps to Reproduce: +* CREATE KEYSPACE ks1 WITH replication = \{'class': 'NetworkTopologyStrategy', 'DC1': '3'} AND durable_writes = true; +*TABLE:*+ CREATE TABLE ks1.table1 ( col1 text, col2 text, col3 text, col4 text, col5 text, col6 timestamp, data text, PRIMARY KEY ((col1, col2, col3), col4, col5, col6) ); Inserted ~4 million rows and created range tombstones by deleting ~1 million rows. +*Create Data*+ _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) VALUES ( '1', '11', '21', '1', 'a', 1231231230, 'data');_ _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) VALUES ( '1', '11', '21', '2', 'a', 1231231230, 'data');_ _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) VALUES ( '1', '11', '21', '3', 'a', 1231231230, 'data');_ _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) VALUES ( '1', '11', '21', '4', 'a', 1231231230, 'data');_ _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) VALUES ( '1', '11', '21', '5', 'a', 1231231230, 'data');_ +*Create Range Tombstones*+ delete from ks1.table1 where col1='1' and col2='11' and col3='21' and col4='1'; +*Query Live Rows (no tombstones)*+ _select * from ks1.table1 where col1='1' and col2='201' and col3='21' and col4='1' and col5='a' and *col6>1231231230*;_ No issues found, everything is running properly. +*Query Range Tombstones*+ _select * from ks1.table1 where col1='1' and col2='11' and col3='21' and col4='1' and col5='a' and *col6=1231231230*;_ No issues found, everything is running properly. +BUT when running range queries:+ _select * from ks1.table1 where col1='1' and col2='11' and col3='21' and col4='1' and col5='a' and *col6>1231231220;*_ WARN [ReadStage-1] 2019-09-23 14:17:10,281 AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread Thread[ReadStage-1,5,main]: {} java.lang.ArrayIndexOutOfBoundsException: 2 at org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55) at org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545) at org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522) at org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565) at org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446) at org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352) at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171) at org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77) at org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802) at org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953) at org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929) at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134) at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114) at java.lang.Thread.run(Thread.java:745) This WARN is constantly generated until I stop the range queries script. Hope this helps.. Thanks! > LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException > > > Key: CASSANDRA-15172 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15172 > Project: Cassandra > Issue Type: Bug > Components: Local/Other >Reporter: Shalom >Assignee: Benedict >Priority: Normal > Fix For: 3.0.19, 3.11.5 > > > Hi All, > This is the first time I open an issue, so apologies if I'm not following the > rules properly. > > After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a > lot of AbstractLocalAw
[jira] [Commented] (CASSANDRA-15172) LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16937526#comment-16937526 ] Shalom commented on CASSANDRA-15172: For users that happen to stumble upon this use case too, I'd like to emphasize that the bug occurs only during mixed versions. Once the cluster got fully upgraded (binaries only), the issue was gone. > LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException > > > Key: CASSANDRA-15172 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15172 > Project: Cassandra > Issue Type: Bug > Components: Local/Other >Reporter: Shalom >Assignee: Benedict >Priority: Normal > Fix For: 3.0.19, 3.11.5 > > > Hi All, > This is the first time I open an issue, so apologies if I'm not following the > rules properly. > > After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a > lot of AbstractLocalAwareExecutorService exceptions. This happened right > after the node successfully started up with the new 3.11.4 binaries. > {noformat} > INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog; > proceeding > INFO [main] 2019-06-05 04:41:38,036 NativeTransportService.java:70 - Netty > using native Epoll event loop > INFO [main] 2019-06-05 04:41:38,117 Server.java:155 - Using Netty Version: > [netty-buffer=netty-buffer-4.0.44.Final.452812a, > netty-codec=netty-codec-4.0.44.Final.452812a, > netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a, > netty-codec-http=netty-codec-http-4.0.44.Final.452812a, > netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a, > netty-common=netty-common-4.0.44.Final.452812a, > netty-handler=netty-handler-4.0.44.Final.452812a, > netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb, > netty-transport=netty-transport-4.0.44.Final.452812a, > netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a, > netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a, > netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a, > netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a] > INFO [main] 2019-06-05 04:41:38,118 Server.java:156 - Starting listening for > CQL clients on /0.0.0.0:9042 (unencrypted)... > INFO [main] 2019-06-05 04:41:38,179 CassandraDaemon.java:556 - Not starting > RPC server as requested. Use JMX (StorageService->startRPCServer()) or > nodetool (enablethrift) to start it > INFO [Native-Transport-Requests-21] 2019-06-05 04:41:39,145 > AuthCache.java:161 - (Re)initializing PermissionsCache (validity > period/update interval/max entries) (2000/2000/1000) > INFO [OptionalTasks:1] 2019-06-05 04:41:39,729 CassandraAuthorizer.java:409 > - Converting legacy permissions data > INFO [HANDSHAKE-/10.10.10.8] 2019-06-05 04:41:39,808 > OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.8 > INFO [HANDSHAKE-/10.10.10.9] 2019-06-05 04:41:39,808 > OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.9 > INFO [HANDSHAKE-dc1_02/10.10.10.6] 2019-06-05 04:41:39,809 > OutboundTcpConnection.java:561 - Handshaking version with dc1_02/10.10.10.6 > WARN [ReadStage-2] 2019-06-05 04:41:39,857 > AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread > Thread[ReadStage-2,5,main]: {} > java.lang.ArrayIndexOutOfBoundsException: 1 > at > org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55) > at > org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545) > at > org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522) > at > org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565) > at > org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446) > at > org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352) > at > org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171) > at > org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77) > at > org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802) > at > org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953) > at > org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929) > at > org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) > at > java.util.concurrent.Executors$Run
[jira] [Created] (CASSANDRA-15336) LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException When Running Range Queries
Shalom created CASSANDRA-15336: -- Summary: LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException When Running Range Queries Key: CASSANDRA-15336 URL: https://issues.apache.org/jira/browse/CASSANDRA-15336 Project: Cassandra Issue Type: Bug Reporter: Shalom Assignee: Benedict Hi All, This bug is similar to https://issues.apache.org/jira/browse/CASSANDRA-15172 but relates specifically to range queries running over range tombstones. *+Steps to Reproduce: +* CREATE KEYSPACE ks1 WITH replication = \{'class': 'NetworkTopologyStrategy', 'DC1': '3'} AND durable_writes = true; +*TABLE:*+ CREATE TABLE ks1.table1 ( col1 text, col2 text, col3 text, col4 text, col5 text, col6 timestamp, data text, PRIMARY KEY ((col1, col2, col3), col4, col5, col6) ); Inserted ~4 million rows and created range tombstones by deleting ~1 million rows. +*Create Data*+ _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) VALUES ( '1', '11', '21', '1', 'a', 1231231230, 'data');_ _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) VALUES ( '1', '11', '21', '2', 'a', 1231231230, 'data');_ _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) VALUES ( '1', '11', '21', '3', 'a', 1231231230, 'data');_ _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) VALUES ( '1', '11', '21', '4', 'a', 1231231230, 'data');_ _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) VALUES ( '1', '11', '21', '5', 'a', 1231231230, 'data');_ +*Create Range Tombstones*+ delete from ks1.table1 where col1='1' and col2='11' and col3='21' and col4='1'; +*Query Live Rows (no tombstones)*+ _select * from ks1.table1 where col1='1' and col2='201' and col3='21' and col4='1' and col5='a' and *col6>1231231230*;_ No issues found, everything is running properly. +*Query Range Tombstones*+ _select * from ks1.table1 where col1='1' and col2='11' and col3='21' and col4='1' and col5='a' and *col6=1231231230*;_ No issues found, everything is running properly. +BUT when running range queries:+ _select * from ks1.table1 where col1='1' and col2='11' and col3='21' and col4='1' and col5='a' and *col6>1231231220;*_ WARN [ReadStage-1] 2019-09-23 14:17:10,281 AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread Thread[ReadStage-1,5,main]: {} java.lang.ArrayIndexOutOfBoundsException: 2 at org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55) at org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545) at org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522) at org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565) at org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446) at org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352) at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171) at org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77) at org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802) at org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953) at org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929) at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134) at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114) at java.lang.Thread.run(Thread.java:745) This WARN is constantly generated until I stop the range queries script. Hope this helps.. Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15172) LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/CASSANDRA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16937903#comment-16937903 ] Shalom commented on CASSANDRA-15172: Hi [~benedict] Done and assigned to you! https://issues.apache.org/jira/browse/CASSANDRA-15336 Enjoy your holiday. (the more the merrier) :) > LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException > > > Key: CASSANDRA-15172 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15172 > Project: Cassandra > Issue Type: Bug > Components: Local/Other >Reporter: Shalom >Assignee: Benedict >Priority: Normal > Fix For: 3.0.19, 3.11.5 > > > Hi All, > This is the first time I open an issue, so apologies if I'm not following the > rules properly. > > After upgrading a node from version 2.1.21 to 3.11.4, we've started seeing a > lot of AbstractLocalAwareExecutorService exceptions. This happened right > after the node successfully started up with the new 3.11.4 binaries. > {noformat} > INFO [main] 2019-06-05 04:41:37,730 Gossiper.java:1715 - No gossip backlog; > proceeding > INFO [main] 2019-06-05 04:41:38,036 NativeTransportService.java:70 - Netty > using native Epoll event loop > INFO [main] 2019-06-05 04:41:38,117 Server.java:155 - Using Netty Version: > [netty-buffer=netty-buffer-4.0.44.Final.452812a, > netty-codec=netty-codec-4.0.44.Final.452812a, > netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a, > netty-codec-http=netty-codec-http-4.0.44.Final.452812a, > netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a, > netty-common=netty-common-4.0.44.Final.452812a, > netty-handler=netty-handler-4.0.44.Final.452812a, > netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb, > netty-transport=netty-transport-4.0.44.Final.452812a, > netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a, > netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a, > netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a, > netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a] > INFO [main] 2019-06-05 04:41:38,118 Server.java:156 - Starting listening for > CQL clients on /0.0.0.0:9042 (unencrypted)... > INFO [main] 2019-06-05 04:41:38,179 CassandraDaemon.java:556 - Not starting > RPC server as requested. Use JMX (StorageService->startRPCServer()) or > nodetool (enablethrift) to start it > INFO [Native-Transport-Requests-21] 2019-06-05 04:41:39,145 > AuthCache.java:161 - (Re)initializing PermissionsCache (validity > period/update interval/max entries) (2000/2000/1000) > INFO [OptionalTasks:1] 2019-06-05 04:41:39,729 CassandraAuthorizer.java:409 > - Converting legacy permissions data > INFO [HANDSHAKE-/10.10.10.8] 2019-06-05 04:41:39,808 > OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.8 > INFO [HANDSHAKE-/10.10.10.9] 2019-06-05 04:41:39,808 > OutboundTcpConnection.java:561 - Handshaking version with /10.10.10.9 > INFO [HANDSHAKE-dc1_02/10.10.10.6] 2019-06-05 04:41:39,809 > OutboundTcpConnection.java:561 - Handshaking version with dc1_02/10.10.10.6 > WARN [ReadStage-2] 2019-06-05 04:41:39,857 > AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread > Thread[ReadStage-2,5,main]: {} > java.lang.ArrayIndexOutOfBoundsException: 1 > at > org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55) > at > org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545) > at > org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522) > at > org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565) > at > org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446) > at > org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352) > at > org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171) > at > org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77) > at > org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802) > at > org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953) > at > org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929) > at > org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at > org.apac
[jira] [Commented] (CASSANDRA-15336) LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException When Running Range Queries
[ https://issues.apache.org/jira/browse/CASSANDRA-15336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16942902#comment-16942902 ] Shalom commented on CASSANDRA-15336: [~benedict] I'm running v2.1.21 and wish to upgrade to v3.11.4. I'm pretty sure I upgraded using the patch but I'll double check. One question though, I see that the branch you used is 3.0 ( https://github.com/belliottsmith/cassandra/blob/{color:#FF}*15172-3.0*{color}/src/java/org/apache/cassandra/db/LegacyLayout.java) Is this the correct branch or should it be a 3.11 branch? > LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException When Running > Range Queries > --- > > Key: CASSANDRA-15336 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15336 > Project: Cassandra > Issue Type: Bug >Reporter: Shalom >Assignee: Benedict Elliott Smith >Priority: Normal > > Hi All, > This bug is similar to https://issues.apache.org/jira/browse/CASSANDRA-15172 > but relates specifically to range queries running over range tombstones. > > > *+Steps to Reproduce: > +* > CREATE KEYSPACE ks1 WITH replication = \{'class': 'NetworkTopologyStrategy', > 'DC1': '3'} AND durable_writes = true; > +*TABLE:*+ > CREATE TABLE ks1.table1 ( > col1 text, > col2 text, > col3 text, > col4 text, > col5 text, > col6 timestamp, > data text, > PRIMARY KEY ((col1, col2, col3), col4, col5, col6) > ); > > Inserted ~4 million rows and created range tombstones by deleting ~1 million > rows. > > +*Create Data*+ > _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) > VALUES ( '1', '11', '21', '1', 'a', 1231231230, 'data');_ > _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) > VALUES ( '1', '11', '21', '2', 'a', 1231231230, 'data');_ > _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) > VALUES ( '1', '11', '21', '3', 'a', 1231231230, 'data');_ > _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) > VALUES ( '1', '11', '21', '4', 'a', 1231231230, 'data');_ > _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) > VALUES ( '1', '11', '21', '5', 'a', 1231231230, 'data');_ > > +*Create Range Tombstones*+ > delete from ks1.table1 where col1='1' and col2='11' and col3='21' and > col4='1'; > > +*Query Live Rows (no tombstones)*+ > _select * from ks1.table1 where col1='1' and col2='201' and col3='21' and > col4='1' and col5='a' and *col6>1231231230*;_ > No issues found, everything is running properly. > > +*Query Range Tombstones*+ > _select * from ks1.table1 where col1='1' and col2='11' and col3='21' and > col4='1' and col5='a' and *col6=1231231230*;_ > No issues found, everything is running properly. > > +BUT when running range queries:+ > _select * from ks1.table1 where col1='1' and col2='11' and col3='21' and > col4='1' and col5='a' and *col6>1231231220;*_ > WARN [ReadStage-1] 2019-09-23 14:17:10,281 > AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread > Thread[ReadStage-1,5,main]: {} > java.lang.ArrayIndexOutOfBoundsException: 2 > at > org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55) > at > org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545) > at > org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522) > at > org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565) > at > org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446) > at > org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352) > at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171) > at > org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77) > at > org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802) > at > org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953) > at > org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929) > at > org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorS
[jira] [Commented] (CASSANDRA-15336) LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException When Running Range Queries
[ https://issues.apache.org/jira/browse/CASSANDRA-15336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16943812#comment-16943812 ] Shalom commented on CASSANDRA-15336: Hi [~benedict] I tried to use the 3.11 branch and unfortunately was able to reproduce the bug. I've double and triple checked that the fix was there and that I used the correct branch (pasted it below). I'm a bit stressed about this matter because I have a planned upgrade this coming Sunday and will be super happy if it works flawlessly. I believe that once all nodes are upgraded though, the issue will be gone. Is my assumption correct? Thanks! ... for (int i = 0; i < this.size; i++) { LegacyBound start = starts[i]; LegacyBound end = ends[i]; CompositeType.Builder startBuilder = type.builder(); CompositeType.Builder endBuilder = type.builder(); *{color:#FF}for (int j = 0; j < start.bound.size(); j++){color}* *{color:#FF}startBuilder.add(start.bound.get(j));{color}* *{color:#FF}for (int j = 0; j < end.bound.size(); j++){color}* *{color:#FF}endBuilder.add(end.bound.get(j));{color}* if (start.collectionName != null) startBuilder.add(start.collectionName.name.bytes); if (end.collectionName != null) endBuilder.add(end.collectionName.name.bytes); size += ByteBufferUtil.serializedSizeWithShortLength(startBuilder.build()); size += ByteBufferUtil.serializedSizeWithShortLength(endBuilder.buildAsEndOfRange()); ... > LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException When Running > Range Queries > --- > > Key: CASSANDRA-15336 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15336 > Project: Cassandra > Issue Type: Bug >Reporter: Shalom >Assignee: Benedict Elliott Smith >Priority: Normal > > Hi All, > This bug is similar to https://issues.apache.org/jira/browse/CASSANDRA-15172 > but relates specifically to range queries running over range tombstones. > > > *+Steps to Reproduce: > +* > CREATE KEYSPACE ks1 WITH replication = \{'class': 'NetworkTopologyStrategy', > 'DC1': '3'} AND durable_writes = true; > +*TABLE:*+ > CREATE TABLE ks1.table1 ( > col1 text, > col2 text, > col3 text, > col4 text, > col5 text, > col6 timestamp, > data text, > PRIMARY KEY ((col1, col2, col3), col4, col5, col6) > ); > > Inserted ~4 million rows and created range tombstones by deleting ~1 million > rows. > > +*Create Data*+ > _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) > VALUES ( '1', '11', '21', '1', 'a', 1231231230, 'data');_ > _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) > VALUES ( '1', '11', '21', '2', 'a', 1231231230, 'data');_ > _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) > VALUES ( '1', '11', '21', '3', 'a', 1231231230, 'data');_ > _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) > VALUES ( '1', '11', '21', '4', 'a', 1231231230, 'data');_ > _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) > VALUES ( '1', '11', '21', '5', 'a', 1231231230, 'data');_ > > +*Create Range Tombstones*+ > delete from ks1.table1 where col1='1' and col2='11' and col3='21' and > col4='1'; > > +*Query Live Rows (no tombstones)*+ > _select * from ks1.table1 where col1='1' and col2='201' and col3='21' and > col4='1' and col5='a' and *col6>1231231230*;_ > No issues found, everything is running properly. > > +*Query Range Tombstones*+ > _select * from ks1.table1 where col1='1' and col2='11' and col3='21' and > col4='1' and col5='a' and *col6=1231231230*;_ > No issues found, everything is running properly. > > +BUT when running range queries:+ > _select * from ks1.table1 where col1='1' and col2='11' and col3='21' and > col4='1' and col5='a' and *col6>1231231220;*_ > WARN [ReadStage-1] 2019-09-23 14:17:10,281 > AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread > Thread[ReadStage-1,5,main]: {} > java.lang.ArrayIndexOutOfBoundsException: 2 > at > org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55) > at > org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545) > at > org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522) > at > org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565) > at > org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446) > at > org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352) > at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171) > at > org.apa
[jira] [Commented] (CASSANDRA-15336) LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException When Running Range Queries
[ https://issues.apache.org/jira/browse/CASSANDRA-15336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948205#comment-16948205 ] Shalom commented on CASSANDRA-15336: Hi All, I'll just update for anyone that might be experiencing the same issue, that after all the binaries were upgraded, the issue was gone as it came. Moreover, we did some preliminary actions to mitigate this issue by running major compactions on "hazardous" tables, thus dropping potentially affecting tombstones. There were still errors, but not as many as before. Thanks [~benedict] for your assistance on this matter! > LegacyLayout RangeTombstoneList throws IndexOutOfBoundsException When Running > Range Queries > --- > > Key: CASSANDRA-15336 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15336 > Project: Cassandra > Issue Type: Bug >Reporter: Shalom >Assignee: Benedict Elliott Smith >Priority: Normal > > Hi All, > This bug is similar to https://issues.apache.org/jira/browse/CASSANDRA-15172 > but relates specifically to range queries running over range tombstones. > > > *+Steps to Reproduce: > +* > CREATE KEYSPACE ks1 WITH replication = \{'class': 'NetworkTopologyStrategy', > 'DC1': '3'} AND durable_writes = true; > +*TABLE:*+ > CREATE TABLE ks1.table1 ( > col1 text, > col2 text, > col3 text, > col4 text, > col5 text, > col6 timestamp, > data text, > PRIMARY KEY ((col1, col2, col3), col4, col5, col6) > ); > > Inserted ~4 million rows and created range tombstones by deleting ~1 million > rows. > > +*Create Data*+ > _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) > VALUES ( '1', '11', '21', '1', 'a', 1231231230, 'data');_ > _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) > VALUES ( '1', '11', '21', '2', 'a', 1231231230, 'data');_ > _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) > VALUES ( '1', '11', '21', '3', 'a', 1231231230, 'data');_ > _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) > VALUES ( '1', '11', '21', '4', 'a', 1231231230, 'data');_ > _insert into ks1.table1 (col1, col2 , col3 , col4 , col5 , col6 , data ) > VALUES ( '1', '11', '21', '5', 'a', 1231231230, 'data');_ > > +*Create Range Tombstones*+ > delete from ks1.table1 where col1='1' and col2='11' and col3='21' and > col4='1'; > > +*Query Live Rows (no tombstones)*+ > _select * from ks1.table1 where col1='1' and col2='201' and col3='21' and > col4='1' and col5='a' and *col6>1231231230*;_ > No issues found, everything is running properly. > > +*Query Range Tombstones*+ > _select * from ks1.table1 where col1='1' and col2='11' and col3='21' and > col4='1' and col5='a' and *col6=1231231230*;_ > No issues found, everything is running properly. > > +BUT when running range queries:+ > _select * from ks1.table1 where col1='1' and col2='11' and col3='21' and > col4='1' and col5='a' and *col6>1231231220;*_ > WARN [ReadStage-1] 2019-09-23 14:17:10,281 > AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread > Thread[ReadStage-1,5,main]: {} > java.lang.ArrayIndexOutOfBoundsException: 2 > at > org.apache.cassandra.db.AbstractBufferClusteringPrefix.get(AbstractBufferClusteringPrefix.java:55) > at > org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSizeCompound(LegacyLayout.java:2545) > at > org.apache.cassandra.db.LegacyLayout$LegacyRangeTombstoneList.serializedSize(LegacyLayout.java:2522) > at > org.apache.cassandra.db.LegacyLayout.serializedSizeAsLegacyPartition(LegacyLayout.java:565) > at > org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:446) > at > org.apache.cassandra.db.ReadResponse$Serializer.serializedSize(ReadResponse.java:352) > at org.apache.cassandra.net.MessageOut.payloadSize(MessageOut.java:171) > at > org.apache.cassandra.net.OutboundTcpConnectionPool.getConnection(OutboundTcpConnectionPool.java:77) > at > org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:802) > at > org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:953) > at > org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:929) > at > org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:62) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessi