Hi Habash, The reason of "Cannot replace_address /10.xx.xx.xxx.xx because it doesn't exist in gossip " error during replace is that the dead node gossip information could not survive when you did full cluster (rest of the nodes) restart.
I faced this issue before, you can check my experience of resolving this issue in below link https://github.com/laxmikant99/cassandra-single-node-disater-recovery-lessons regards, Laxmikant On Fri, Mar 15, 2019 at 5:21 AM Stefan Miklosovic < stefan.mikloso...@instaclustr.com> wrote: > It is just a C* in Docker Compose with static IP addresses as long as all > containers run. I am just killing Cassandra process and starting it again > in each container. > > On Fri, 15 Mar 2019 at 10:47, Jeff Jirsa <jji...@gmail.com> wrote: > >> Are your IPs changing as you restart the cluster? Kubernetes or Mesos or >> something where your data gets scheduled on different machines? If so, if >> it gets an IP that was previously in the cluster, it’ll stomp on the old >> entry in the gossiper maps >> >> >> >> -- >> Jeff Jirsa >> >> >> On Mar 14, 2019, at 3:42 PM, Fd Habash <fmhab...@gmail.com> wrote: >> >> I can conclusively say, none of these commands were run. However, I think >> this is the likely scenario … >> >> >> >> If you have a cluster of three nodes 1,2,3 … >> >> - If 3 shows as DN >> - Restart C* on 1 & 2 >> - Nodetool status should NOT show node 3 IP at all. >> >> >> >> Restarting the cluster while a node is down resets gossip state. >> >> >> >> There is a good chance this is what happened. >> >> >> >> Plausible? >> >> >> >> ---------------- >> Thank you >> >> >> >> *From: *Jeff Jirsa <jji...@gmail.com> >> *Sent: *Thursday, March 14, 2019 11:06 AM >> *To: *cassandra <user@cassandra.apache.org> >> *Subject: *Re: Cannot replace_address /10.xx.xx.xx because it doesn't >> exist ingossip >> >> >> >> Two things that wouldn't be a bug: >> >> >> >> You could have run removenode >> >> You could have run assassinate >> >> >> >> Also could be some new bug, but that's much less likely. >> >> >> >> >> >> On Thu, Mar 14, 2019 at 2:50 PM Fd Habash <fmhab...@gmail.com> wrote: >> >> I have a node which I know for certain was a cluster member last week. It >> showed in nodetool status as DN. When I attempted to replace it today, I >> got this message >> >> >> >> ERROR [main] 2019-03-14 14:40:49,208 CassandraDaemon.java:654 - Exception >> encountered during startup >> >> java.lang.RuntimeException: Cannot replace_address /10.xx.xx.xxx.xx >> because it doesn't exist in gossip >> >> at >> org.apache.cassandra.service.StorageService.prepareReplacementInfo(StorageService.java:449) >> ~[apache-cassandra-2.2.8.jar:2.2.8] >> >> >> >> >> >> DN 10.xx.xx.xx 388.43 KB 256 6.9% >> bdbd632a-bf5d-44d4-b220-f17f258c4701 1e >> >> >> >> Under what conditions does this happen? >> >> >> >> >> >> ---------------- >> Thank you >> >> >> >> >> >> > > -- > > > *Stefan Miklosovic**Senior Software Engineer* > > > M: +61459911436 > > <https://www.instaclustr.com> > > <https://www.facebook.com/instaclustr> <https://twitter.com/instaclustr> > <https://www.linkedin.com/company/instaclustr> > > Read our latest technical blog posts here > <https://www.instaclustr.com/blog/>. > > This email has been sent on behalf of Instaclustr Pty. Limited (Australia) > and Instaclustr Inc (USA). > > This email and any attachments may contain confidential and legally > privileged information. If you are not the intended recipient, do not copy > or disclose its content, but please reply to this email immediately and > highlight the error to the sender and then immediately delete the message. > > Instaclustr values your privacy. Our privacy policy can be found at > https://www.instaclustr.com/company/policies/privacy-policy > -- regards, Laxmikant Upadhyay
