Hi Richard, I think my problem here is not the issue<https://github.com/Metaswitch/clearwater-etcd/issues/320> you mentioned in last mail. As my clearwater-etcd.log is quite different with that one. And my clearwater version is already the latest Onix, release 108. PS: I followed the manual install instruction and try the upgrade procedure but nothing had to be updated.
In my case, my ctcd-process looks like unable to use the socket to successfully start and communicate with each other. As the boot.log shows cat /var/log/boot.log ……….. ……….. zmq_msg_recv: Resource temporarily unavailable Configuring monit for only localhost access Error: dial tcp 192.168.2.205:4000: getsockopt: no route to host Rejoining cluster... Etcd failed to come up – exiting I think the root cause is about “zmq_msg_recv: Resource temporarily unavailable”. (this is something about socket error) Since I did the two test on two deployment. -------------------First deployment --------------- (1). Six new VM with Ubuntu official 14.04.02 64-bits server (2). Reboot the system when OS is finished (there is no zmq_msg_recv: Resource temporarily unavailable in the boot.log) (3.) Run the manual installation of clearwater then restart VMs. the zmq_msg_recv info shows! Etcd_process failed to start. (4.) Try to upgrade to the release Onix 108, but everything is newest. (5.) the eted-process is still unfunctional. ------------------Second deployment----------------- (1). In this deployment, I used my 6 VM images of Clearwater nodes that I setup in about 2015 Oct. And this deployment was running smoothly. (2). Run the upgrade procedure to the latest version Onix 108. (3). The zmq_msg_recv: Resource temporarily unavailable shows during the upgrade process. (4.) Reboot after the upgrade finished. (5.) The zmq_msg_recv: Resource temporarily unavailable shows during the boot step. And the etcd_process became unfunctional. I would like to know what’s version of your python? Seems like zmq_msg_recv is from a python binary. And how do I check my clearwater release version? And Install the new machine with certain version? I think the old version might just good for me. Thank you very much! Jace. From: Richard Whitehouse [mailto:richard.whiteho...@metaswitch.com] Sent: Friday, October 21, 2016 5:43 PM To: clearwater@lists.projectclearwater.org; 梁維恩 <jace.li...@itri.org.tw> Subject: RE: etcd_process execution failed on each node. Jace Liang, What version of Project Clearwater are you running? From your problem description, you might be hitting https://github.com/Metaswitch/clearwater-etcd/issues/320 which we’ve fixed in the latest release, Onix, release 108. This can cause the etcd cluster to lose quorum and thus prevent it starting up correctly. If this is the case, you’ve got two options: 1) You can delete your existing installation, and install release-108 instead. That’s probably the simplest solution, but you’ll lose all of your data. 2) Alternatively, you can upgrade your nodes to release-108, and then restore the cluster. As it’s lost quorum, you’ll need to follow the instructions for multiple node recovery, which are documented in the docs at http://clearwater.readthedocs.io/en/stable/Handling_Failed_Nodes.html#multiple-failed-nodes Hope this helps, Richard From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On Behalf Of jace.li...@itri.org.tw<mailto:jace.li...@itri.org.tw> Sent: 21 October 2016 03:23 To: clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org> Subject: [Project Clearwater] etcd_process execution failed on each node. Dear All, Recently I found out my 6 VMs of each node have trouble of executing etcd_process I think I did the config file right, because at first their etcd_process was running well and the “clearwater-etcdctl cluster health” shows all healthly. But somehow they suddenly I tried “monit restart etcd_process” but still failed. Here are some command result for more information. (on the node ellis) root@ellis1:/var/log# clearwater-etcdctl cluster-health cluster may be unhealthy: failed to list members Error: client: etcd cluster is unavailable or misconfigured error #0: dial tcp 192.168.2.206:4000: getsockopt: connection refused (.206 is homestead’s ip) cat /var/log/boot.log ……….. ……….. ……….. zmq_msg_recv: Resource temporarily unavailable Configuring monit for only localhost access Error: dial tcp 192.168.2.205:4000: getsockopt: no route to host Rejoining cluster... Etcd failed to come up - exiting root@ellis1:/var/log/clearwater-etcd# cat clearwater-etcd.log …………. …………. ………… 2016-10-21 10:11:46.686827 I | etcdmain: etcd Version: 2.2.5 2016-10-21 10:11:46.686888 I | etcdmain: Git SHA: bc9ddf2 2016-10-21 10:11:46.686895 I | etcdmain: Go Version: go1.5.3 2016-10-21 10:11:46.686902 I | etcdmain: Go OS/Arch: linux/amd64 2016-10-21 10:11:46.686913 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4 2016-10-21 10:11:46.686953 N | etcdmain: the server is already initialized as member before, starting as etcd member... 2016-10-21 10:11:46.687015 I | etcdmain: listening for peers on http://192.168.2.206:2380 2016-10-21 10:11:46.687039 I | etcdmain: listening for client requests on http://192.168.2.206:4000 2016-10-21 10:11:46.689639 I | etcdserver: recovered store from snapshot at index 10001 2016-10-21 10:11:46.689654 I | etcdserver: name = 192-168-2-206 2016-10-21 10:11:46.689660 I | etcdserver: data dir = /var/lib/clearwater-etcd/192.168.2.206 2016-10-21 10:11:46.689668 I | etcdserver: member dir = /var/lib/clearwater-etcd/192.168.2.206/member 2016-10-21 10:11:46.689674 I | etcdserver: heartbeat = 100ms 2016-10-21 10:11:46.689680 I | etcdserver: election = 1000ms 2016-10-21 10:11:46.689686 I | etcdserver: snapshot count = 10000 2016-10-21 10:11:46.689696 I | etcdserver: advertise client URLs = http://192.168.2.206:4000 2016-10-21 10:11:46.689717 I | etcdserver: loaded cluster information from store: <nil> 2016-10-21 10:11:46.726159 I | etcdserver: restarting member 4cb5fd19beaa1750 in cluster 877b90a46cdaaa83 at commit index 14044 2016-10-21 10:11:46.727646 I | raft: 4cb5fd19beaa1750 became follower at term 814 2016-10-21 10:11:46.727690 I | raft: newRaft 4cb5fd19beaa1750 [peers: [1226bb321c91a88e,4cb5fd19beaa1750,8ac8820f24de7303,a4a5d4f826d5740a], term: 814, commit: 14044, applied: 10001, lastindex: 14045, lastterm: 814] 2016-10-21 10:11:46.734589 I | rafthttp: the connection with 1226bb321c91a88e became active 2016-10-21 10:11:46.739284 E | rafthttp: failed to dial 8ac8820f24de7303 on stream Message (dial tcp 192.168.2.202:2380: getsockopt: connection refused) 2016-10-21 10:11:46.740156 E | rafthttp: failed to dial 8ac8820f24de7303 on stream MsgApp v2 (dial tcp 192.168.2.202:2380: getsockopt: connection refused) 2016-10-21 10:11:46.745962 I | etcdserver: starting server... [version: 2.2.5, cluster version: 2.2] 2016-10-21 10:11:46.747252 E | rafthttp: failed to dial a4a5d4f826d5740a on stream Message (dial tcp 192.168.2.203:2380: getsockopt: connection refused) 2016-10-21 10:11:46.747394 E | rafthttp: failed to dial a4a5d4f826d5740a on stream MsgApp v2 (dial tcp 192.168.2.203:2380: getsockopt: connection refused) 2016-10-21 10:11:46.756637 I | rafthttp: the connection with 1226bb321c91a88e became inactive 2016-10-21 10:11:46.756660 E | rafthttp: failed to read 1226bb321c91a88e on stream Message (net/http: request canceled) 2016-10-21 10:11:46.756687 N | etcdserver: removed member 1226bb321c91a88e from cluster 877b90a46cdaaa83 2016-10-21 10:11:46.756766 D | etcdserver: skipped updating attributes of removed member 1226bb321c91a88e 2016-10-21 10:11:46.756853 C | etcdserver: nodeToMember should never fail: raftAttributes key doesn't exist panic: nodeToMember should never fail: raftAttributes key doesn't exist Seems like the nodes cannot connect to each other, but I test it with ping, they still ping to each other. Can anyone give us some advice or solution? Thank you. -- 本信件可能包含工研院機密資訊,非指定之收件者,請勿使用或揭露本信件內容,並請銷毀此信件。 This email may contain confidential information. Please do not use or disclose it in any way and delete it if you are not the intended recipient. -- 本信件可能包含工研院機密資訊,非指定之收件者,請勿使用或揭露本信件內容,並請銷毀此信件。 This email may contain confidential information. Please do not use or disclose it in any way and delete it if you are not the intended recipient.
_______________________________________________ Clearwater mailing list Clearwater@lists.projectclearwater.org http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org