Re: Bootstrap error - Cassandra 4.1.5

2024-08-15 Thread Bowen Song via user
ClosedChannelException is often the result of a network connectivity issue. It's worth check the OS logs and the configuration of network devices between the nodes. For example, has the conntrack table filled up? Does any router between the two nodes have short TCP session timeout? A crashed p

Re: Bootstrap error - Cassandra 4.1.5

2024-08-15 Thread Joe Obernberger
Thank you Bowen.  Adding more nodes, but the latest one keeps failing. On the node being added:ERROR [Stream-Deserializer-/192.168.115.187:7000-83045bad] 2024-08-15 21:08:08,315 CassandraEntireSSTableStreamReader.java:146 - [Stream 59fa6610-5b48-11ef-8ffc-a13bcc763c65] Error while reading ssta

Re: Bootstrap error - Cassandra 4.1.5

2024-08-15 Thread Bowen Song via user
You may need to look at the zipped log files if the streaming had been running for a while before failing. The error could have happened hours or days before the final failure. If your cluster is already experiencing performance issues (e.g. due to CPU bottleneck or GC pauses), it's highly lik

Re: Bootstrap error - Cassandra 4.1.5

2024-08-15 Thread Joe Obernberger
Thank you Bowen - yeah the only ERROR I see in /var/log/cassandra/debug.log is: ERROR [main] 2024-08-15 04:48:23,374 StorageService.java:2041 - Error while waiting on bootstrap to complete. Bootstrap will have to be restarted. java.util.concurrent.ExecutionException: org.apache.cassandra.strea

Re: Bootstrap error - Cassandra 4.1.5

2024-08-14 Thread Bowen Song via user
It looks like all your nodes are in the same DC and the same rack with 256 vnodes each. It's very hard (if not impossible) to add multiple nodes to the same DC concurrently and safely in this setup. You are better off adding one node at a time to this cluster. Try search for "ERROR" in the log

Bootstrap error - Cassandra 4.1.5

2024-08-14 Thread Joe Obernberger
Hi all - when adding a node to our existing 15 node cluster, I get: DEBUG [NonPeriodicTasks:1] 2024-08-14 20:34:10,383 StreamCoordinator.java:152 - Finished connecting all sessions WARN  [NonPeriodicTasks:1] 2024-08-14 20:34:10,385 StreamResultFuture.java:242 - [Stream #d7bf9f60-5a5e-11ef-aa7