Re: Nodes show different number of tokens than initially
On Tue, Jan 30, 2018 at 5:13 AM, kurt greaves wrote: > Shouldn't happen. Can you send through nodetool ring output from one of > those nodes? Also, did the logs have anything to say about tokens when you > started the 3 seed nodes? > Hi Kurt, I cannot run nodetool ring anymore, since these test nodes are long gone. However I've grepped the logs and this is what I've found: Jan 25 08:57:18 ip-172-31-128-41 docker/cf3ea463915a[854]: INFO 08:57:18 Nodes /172.31.128.31 and /172.31.128.41 have the same token -9223372036854775808. Ignoring /172.31.128.31 Jan 25 08:57:18 ip-172-31-128-41 docker/cf3ea463915a[854]: INFO 08:57:18 Nodes /172.31.144.32 and /172.31.128.41 have the same token -8454757700450211158. Ignoring /172.31.144.32 Jan 25 08:58:30 ip-172-31-144-41 docker/48fba443d99f[852]: INFO 08:58:30 Nodes /172.31.128.41 and /172.31.128.31 have the same token -9223372036854775808. /172.31.128.41 is the new owner Jan 25 08:58:30 ip-172-31-144-41 docker/48fba443d99f[852]: INFO 08:58:30 Nodes /172.31.144.32 and /172.31.128.41 have the same token -8454757700450211158. Ignoring /172.31.144.32 Jan 25 08:59:45 ip-172-31-160-41 docker/cced70e132f2[849]: INFO 08:59:45 Nodes /172.31.128.41 and /172.31.128.31 have the same token -9223372036854775808. /172.31.128.41 is the new owner Jan 25 08:59:45 ip-172-31-160-41 docker/cced70e132f2[849]: INFO 08:59:45 Nodes /172.31.144.32 and /172.31.128.41 have the same token -8454757700450211158. Ignoring /172.31.144.32 Since we are allocating the tokens for seed nodes manually, it appears that the first seed node in the new ring (172.31.128.41) gets the same first token (-9223372036854775808) as the node in the old ring (172.31.128.31). The same goes for the 3rd token of the new seed node (-8454757700450211158). What is beyond me is why would that matter and why would token ownership change at all, while these nodes are in the *different virtual DCs*? To me this sounds like a paticularly nasty bug... -- Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176 127-59-707
Re: TWCS not deleting expired sstables
Likely a read repair caused old data to be brought into a newer SSTable. Try running sstableexpiredblockers to find out if there's a newer SSTable blocking that one from being dropped.
Re: Nodes show different number of tokens than initially
Shouldn't happen. Can you send through nodetool ring output from one of those nodes? Also, did the logs have anything to say about tokens when you started the 3 seed nodes?
Re: group by select queries
What consistency were you querying at? Can you retry with CONSISTENCY ALL?
Re: Nodetool hanging - Cassandra 3.10, Oracle Java 1.8.0_131
I’ve resolved the issue. These hosts were initially created in a different VLAN. There was a holdover entry in /etc/hosts pointing to the old IP Addr for this host. So, the return back to the client never happened because it was sending the connection to the old IP to a host that did not exist. Thanks for looking! James From: James Lovato Reply-To: "user@cassandra.apache.org" Date: Thursday, January 25, 2018 at 10:44 AM To: "user@cassandra.apache.org" Subject: Re: Nodetool hanging - Cassandra 3.10, Oracle Java 1.8.0_131 It starts up, I can telnet to the port. Also the log states so: ^C[root@dfprdbcdlpcs5 logs]# grep JMX cassandra.log INFO [main] 2018-01-25 10:52:54,176 JMXServerUtils.java:249 - Configured JMX server at: service:jmx:rmi://0.0.0.0/jndi/rmi://0.0.0.0:7199/jmxrmi INFO [main] 2018-01-25 10:52:54,379 StartupChecks.java:164 - JMX is enabled to receive remote connections on port: 7199 INFO [main] 2018-01-25 10:55:25,751 JMXServerUtils.java:249 - Configured JMX server at: service:jmx:rmi://0.0.0.0/jndi/rmi://0.0.0.0:7199/jmxrmi INFO [main] 2018-01-25 10:55:25,957 StartupChecks.java:164 - JMX is enabled to receive remote connections on port: 7199 From: Roger Brown Reply-To: "user@cassandra.apache.org" Date: Thursday, January 25, 2018 at 9:51 AM To: "user@cassandra.apache.org" Subject: Re: Nodetool hanging - Cassandra 3.10, Oracle Java 1.8.0_131 My first thought is restart cassandra and monitor its log to make sure it starts up. On Thu, Jan 25, 2018 at 9:15 AM Jeff Jirsa mailto:jji...@gmail.com>> wrote: Can you telnet to the 7199 port? -- Jeff Jirsa On Jan 25, 2018, at 8:00 AM, James Lovato mailto:james.lov...@iacapps.com>> wrote: I have a 15 node cluster, 5 in each of 3 dcs. One host just recently started hanging doing any `nodetool` requests. I’ve enabled remote JMX on all these hosts and it was working fine until this week. No other changes have been made. All the Cassandra-env.sh files are the same among the nodes, and only seeds and listen_address are different for Cassandra.yaml files. Nothing in debug log about why the Connection timed out. This is what I get running nodetool from the same host: [root@host5 conf]# nodetool status nodetool: Failed to connect to 'xx.xx.xx.xx:7199' - ConnectException: 'Connection timed out (Connection timed out)'. Any thoughts or how to log nodetool’s actions? Or see why it’s not accepting the request? Telnet to the port from localhost as well as another node work fine. Thanks, James
Re: Cassandra nodes are down
Something is invoking the shutdown hook (calling kill). May be your config management or similar. -- Jeff Jirsa > On Jan 29, 2018, at 9:38 AM, Mokkapati, Bhargav (Nokia - IN/Chennai) > wrote: > > Hi Team, > > I’m getting the below warnings. Please help me out to clear these issues. > > Apache Cassandra version : 3.0.13, 5 Node cluster > > INFO [main] 2018-01-29 16:58:19,487 NativeLibrary.java:167 - JNA mlockall > successful > WARN [main] 2018-01-29 16:58:19,488 StartupChecks.java:121 - jemalloc shared > library could not be preloaded to speed up memory allocations > INFO [main] 2018-01-29 16:58:19,488 StartupChecks.java:160 - JMX is enabled > to receive remote connections on port: 8002 > WARN [main] 2018-01-29 16:58:19,488 StartupChecks.java:178 - OpenJDK is not > recommended. Please upgrade to the newest Oracle Java release > INFO [main] 2018-01-29 16:58:19,490 SigarLibrary.java:44 - Initializing > SIGAR library > WARN [main] 2018-01-29 16:58:19,498 SigarLibrary.java:174 - Cassandra server > running in degraded mode. Is swap disabled? : true, Address space adequate? > : true, nofile limit adequate? : false, nproc limit adequate? : true > WARN [main] 2018-01-29 16:58:19,500 StartupChecks.java:246 - Maximum number > of memory map areas per process (vm.max_map_count) 65530 is too low, > recommended value: 1048575, you can change it with sysctl. > > WARN [main] 2018-01-29 17:05:07,844 SystemKeyspace.java:1042 - No host ID > found, created 2dc59352-e98e-4e77-a5f2-289697e467c7 (Note: This should happen > exactly once per node). > INFO [main] 2018-01-29 17:05:16,421 Server.java:160 - Starting listening for > CQL clients on /10.50.21.22:9042 (unencrypted)... > INFO [main] 2018-01-29 17:05:16,449 CassandraDaemon.java:488 - Not starting > RPC server as requested. Use JMX (StorageService->startRPCServer()) or > nodetool (enablethrift) to start it > INFO [OptionalTasks:1] 2018-01-29 17:05:18,443 CassandraRoleManager.java:350 > - Created default superuser role 'cassandra' > INFO [StorageServiceShutdownHook] 2018-01-29 17:09:55,737 > HintsService.java:212 - Paused hints dispatch > INFO [StorageServiceShutdownHook] 2018-01-29 17:09:55,740 Server.java:180 - > Stop listening for CQL clients > INFO [StorageServiceShutdownHook] 2018-01-29 17:09:55,740 Gossiper.java:1490 > - Announcing shutdown > INFO [StorageServiceShutdownHook] 2018-01-29 17:09:55,741 > StorageService.java:1991 - Node /10.50.21.22 state jump to shutdown > INFO [StorageServiceShutdownHook] 2018-01-29 17:09:57,743 > MessagingService.java:811 - Waiting for messaging service to quiesce > INFO [ACCEPT-/10.50.21.22] 2018-01-29 17:09:57,743 > MessagingService.java:1110 - MessagingService has terminated the accept() > thread > INFO [StorageServiceShutdownHook] 2018-01-29 17:09:57,797 > HintsService.java:212 - Paused hints dispatch > > Thanks, > Bhargav M.
Cassandra nodes are down
Hi Team, I'm getting the below warnings. Please help me out to clear these issues. Apache Cassandra version : 3.0.13, 5 Node cluster INFO [main] 2018-01-29 16:58:19,487 NativeLibrary.java:167 - JNA mlockall successful WARN [main] 2018-01-29 16:58:19,488 StartupChecks.java:121 - jemalloc shared library could not be preloaded to speed up memory allocations INFO [main] 2018-01-29 16:58:19,488 StartupChecks.java:160 - JMX is enabled to receive remote connections on port: 8002 WARN [main] 2018-01-29 16:58:19,488 StartupChecks.java:178 - OpenJDK is not recommended. Please upgrade to the newest Oracle Java release INFO [main] 2018-01-29 16:58:19,490 SigarLibrary.java:44 - Initializing SIGAR library WARN [main] 2018-01-29 16:58:19,498 SigarLibrary.java:174 - Cassandra server running in degraded mode. Is swap disabled? : true, Address space adequate? : true, nofile limit adequate? : false, nproc limit adequate? : true WARN [main] 2018-01-29 16:58:19,500 StartupChecks.java:246 - Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low, recommended value: 1048575, you can change it with sysctl. WARN [main] 2018-01-29 17:05:07,844 SystemKeyspace.java:1042 - No host ID found, created 2dc59352-e98e-4e77-a5f2-289697e467c7 (Note: This should happen exactly once per node). INFO [main] 2018-01-29 17:05:16,421 Server.java:160 - Starting listening for CQL clients on /10.50.21.22:9042 (unencrypted)... INFO [main] 2018-01-29 17:05:16,449 CassandraDaemon.java:488 - Not starting RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool (enablethrift) to start it INFO [OptionalTasks:1] 2018-01-29 17:05:18,443 CassandraRoleManager.java:350 - Created default superuser role 'cassandra' INFO [StorageServiceShutdownHook] 2018-01-29 17:09:55,737 HintsService.java:212 - Paused hints dispatch INFO [StorageServiceShutdownHook] 2018-01-29 17:09:55,740 Server.java:180 - Stop listening for CQL clients INFO [StorageServiceShutdownHook] 2018-01-29 17:09:55,740 Gossiper.java:1490 - Announcing shutdown INFO [StorageServiceShutdownHook] 2018-01-29 17:09:55,741 StorageService.java:1991 - Node /10.50.21.22 state jump to shutdown INFO [StorageServiceShutdownHook] 2018-01-29 17:09:57,743 MessagingService.java:811 - Waiting for messaging service to quiesce INFO [ACCEPT-/10.50.21.22] 2018-01-29 17:09:57,743 MessagingService.java:1110 - MessagingService has terminated the accept() thread INFO [StorageServiceShutdownHook] 2018-01-29 17:09:57,797 HintsService.java:212 - Paused hints dispatch Thanks, Bhargav M.
