Hi Sunil, I’m afraid the steps you’ve taken are not supported in Project Clearwater deployments. Both changing the ‘local_ip’ of a node, and removing nodes just by deleting the VMs.
On the first point, you need to be able to give your VMs permanent static IP addresses. On the second, by deleting the VMs in your cluster, your underlying etcd cluster has lost quorum. I would suggest http://clearwater.readthedocs.io/en/stable/Handling_Multiple_Failed_Nodes.htm as a starting point for recovering information from it. However, as your single remaining node will likely also have problems due to the local IP changing, you may simply want to redeploy from scratch. More in general, you seem to have hit a substantial number of issues in deploying Project Clearwater, which is both not what we want, and not what the experience of many other users seems to be. I would suggest taking a wider look over our provided documentation, and making sure your environment matches our expectations, and that you’re clear on our processes. This should make your next deployment a lot smoother. Cheers, and good luck, Adam From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On Behalf Of Sunil Kumar Sent: 19 April 2018 07:16 To: clearwater@lists.projectclearwater.org Subject: Re: [Project Clearwater] Unable to contact the etcd cluster Hi, the node with ip 10.224.61.109, 10.224.61.112 etc is no more there, I have deleted the node directly. It looks like they are still in the etcd cluster. Can you please tell me how to remove them [IST Apr 19 19:32:45] error : 'etcd_process' process is not running [IST Apr 19 19:32:45] info : 'etcd_process' trying to restart [IST Apr 19 19:32:45] info : 'etcd_process' restart: /bin/bash [IST Apr 19 19:33:15] error : 'etcd_process' failed to restart (exit status -1) -- /bin/bash: Program timed out -- zmq_msg_recv: Resource temporarily unavailable cat: /var/run/clearwater-etcd/clearwater-etcd.pid: No such file or directory cat: /var/run/clearwater-etcd/clearwater-etcd.pid: No such file or directory context deadline excee [IST Apr 19 19:33:25] error : 'etcd_process' process is not running [IST Apr 19 19:33:25] info : 'etcd_process' trying to restart [IST Apr 19 19:33:25] info : 'etcd_process' restart: /bin/bash [IST Apr 19 19:33:55] error : 'etcd_process' failed to restart (exit status -1) -- /bin/bash: Program timed out -- zmq_msg_recv: Resource temporarily unavailable client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 10.224.61.109:4000<http://10.224.61.109:4000>: getsockopt: no route to host ; error #1: dial tcp 10.224.61.47:4000<http://10.224.61.47:4000>: getsockopt: co [IST Apr 19 19:34:05] error : 'etcd_process' process is not running [IST Apr 19 19:34:05] info : 'etcd_process' trying to restart [IST Apr 19 19:34:05] info : 'etcd_process' restart: /bin/bash [IST Apr 19 19:34:36] error : 'etcd_process' failed to restart (exit status 2) -- /bin/bash: zmq_msg_recv: Resource temporarily unavailable context deadline exceeded On Thu, Apr 19, 2018 at 11:03 AM, Sunil Kumar <skgola1...@gmail.com<mailto:skgola1...@gmail.com>> wrote: Hi, Any body can help me on this. after ip lost, i update the ip in local_config and dns and restart the service. extra vm is deleted lik i had 3 sprout node so 2 are deleted. [vellum]ubuntu@vellum:~$ cw-config upload shared_config Unable to contact the etcd cluster. thanks sunil
_______________________________________________ Clearwater mailing list Clearwater@lists.projectclearwater.org http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org
