Re: RC1 - joining cluster

2021-05-20 Thread Kane Wilson
Well, that certainly looks like you've hit potentially multiple bugs there. You could set auto_bootstrap to false, but you'll probably still want to run a repair afterwards (note this is not good advice for a production cluster). You should probably raise a JIRA with all the relevant info you've

Re: RC1 - joining cluster

2021-05-12 Thread Joe Obernberger
HI All - I tried adding another node with RC1, it streams for a while, but then I get: INFO  [NonPeriodicTasks:1] 2021-05-12 13:29:22,382 LogTransaction.java:240 - Unfinished transaction log, deleting /data/8/cassandra/data/doc/doc-839e5850ad9f11ebade9fdc1d34336d3/na-982-big-Data.db INFO 

Re: RC1 - joining cluster

2021-05-10 Thread Kane Wilson
Well, that sounds like a dangerous sequence of events, but should have worked in the end regardless. Probably next time give it a bit more time and keep an eye on netstats and compactionstats. raft.so - Cassandra consulting, support, and managed services On Mon, May 10, 2021 at 10:23 PM Joe

Re: RC1 - joining cluster

2021-05-10 Thread Joe Obernberger
Hi - I waited 3 hours.  It was syncing up data; I could see network traffic, but then it stopped.  I didn't check netstats, but I did check compactionstats and there were no pending tasks. I then set auto_bootstrap to false on both new machines and they joined.  Then ran a repair. -Joe

Re: RC1 - joining cluster

2021-05-09 Thread Kane Wilson
How long are you waiting for the node to join? Have you checked nodetool netstats and compactionstats to see if all streams/compactions are complete? raft.so - Cassandra consulting, support, and managed services On Sat, May 8, 2021 at 11:23 AM Joe Obernberger < joseph.obernber...@gmail.com>

Re: RC1 - joining cluster

2021-05-07 Thread Joe Obernberger
Whoops - had it in the wrong datacenter.  Same issue - new node is stuck in UJ, but I can start/stop OK with systemctl. Datacenter: datacenter1 === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving --  Address                    Load       

Re: RC1 - joining cluster

2021-05-07 Thread Joe Obernberger
When I try to halt the joining node with systemctl stop cassandra, it hangs.  I don't see it doing any network, disk, or CPU activity using tools like iotop, atop, and top. I ended up kill -9'ing the process.  I tried the same join on a different machine, and the same issue occurs.  It hangs

RC1 - joining cluster

2021-05-07 Thread Joe Obernberger
Have an existing 5 node RC1 cluster and trying to join two more nodes to it. The new node is stuck in the UJ status: Datacenter: datacenter1 === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving --  Address         Load        Tokens  Owns (effective)Â