As a few more pointers of where to look, our HEAT templates currently generate shared config on the Ellis node, and upload this to the cluster as a whole. It would be worth looking at the install log on the ellis node to see if there was some error there. You may have the shared_config file present, and be able to upload that to the etcd cluster to bring the deployment to life. If that doesn't give you any joy, make sure that the template install process has completed successfully on each of your nodes. It seems odd that shared_config has not been generated, and so I'd want to sanity check that nothing else is going wrong. If all nodes are installed, but lacking config, you should be able to generate shared config based on the script in the ellis.yaml file. As Ying has said, once those things are sorted, you can look in the process logs to see why you are hitting errors. Almost all of our components log to /var/log/<process_name> on their node, so that's a great place to start debugging. You may also find our troubleshooting guide helpful: https://clearwater.readthedocs.io/en/stable/Troubleshooting_and_Recovery.html
Cheers, Adam From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On Behalf Of Ying Huang Sent: 15 December 2017 17:54 To: clearwater@lists.projectclearwater.org Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while running live tests Hi Kedar, Shared config shouldn't be empty. Looks like Sprout process has failed for some reason. It's worthwhile checking the sprout log under /var/sys/log/sprout/sprout_current or send it over to see what went wrong. Thanks, Ying From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On Behalf Of Kedar Ambekar Sent: 14 December 2017 14:07 To: clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org> Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while running live tests Hi, /etc/Clearwater/shared_config is empty on both the sprout instances post installation. Is this expected if I want to run sproutlets like ICSCF, SCSCF at least ? Here are couple of commands I tried. [sprout]ubuntu@0:~$ sudo monit summary Monit 5.18.1 uptime: 7h 12m Service Name Status Type node-0.sprout.me.cw-ngv.com Running System sprout_process Execution failed | Does... Process snmpd_process Running Process ntp_process Running Process nginx_process Running Process memcached_process Running Process clearwater_queue_manager_pro... Running Process etcd_process Running Process clearwater_diags_monitor_pro... Running Process clearwater_config_manager_pr... Running Process clearwater_cluster_manager_p... Running Process chronos_process Running Process astaire_process Running Process sprout_uptime Wait parent Program poll_sprout_sip Wait parent Program poll_sprout_http Wait parent Program nginx_ping Status ok Program nginx_uptime Status ok Program monit_uptime Status ok Program memcached_uptime Status ok Program poll_memcached Status ok Program clearwater_queue_manager_uptime Status ok Program etcd_uptime Status ok Program poll_etcd_cluster Status ok Program poll_etcd Status ok Program chronos_uptime Status ok Program poll_chronos Status ok Program astaire_uptime Status ok Program [sprout]ubuntu@0:~$ service sprout status * sprout is not running [sprout]ubuntu@0:~$ sudo service sprout start 14-12-2017 14:06:17.215 UTC Status utils.cpp:496: Switching to daemon mode From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On Behalf Of Kedar Ambekar Sent: Thursday, December 14, 2017 12:29 PM To: clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org> Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while running live tests Hi, I gave a try with release 127 ( repo URL http://repo.cw-ngv.com/archive/repo127 in env file). But issues seem to be very same. From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On Behalf Of Kedar Ambekar Sent: Wednesday, December 13, 2017 12:30 PM To: clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org> Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while running live tests Hi Adam, Mike, Clearwater heat templates on ONAP wiki are put around July 13. Clearwater release 127, which is released on May 30th , seems to have been used for testing these templates. If I want to give a try with release 127 installation via heat, how to do that ? I tried URL http://repo.cw-ngv.com/release-127/binary/Packages in browser, but got 404. From: Kedar Ambekar Sent: Wednesday, December 13, 2017 11:37 AM To: clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org> Subject: RE: 502 Bad Gateway and other errors while running live tests Hi Adam, Got the point. I had suspected the same. I have repo_url http://repo.cw-ngv.com/stable in heat's env file. Is it possible to use same release on which ONAP's heat templates are based, may be by changing this repo URL? I will have a look at current heat template on Clearwater GitHub but I think those won't be ONAP compatible. From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On Behalf Of Adam Lindley Sent: Tuesday, December 12, 2017 10:50 PM To: clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org> Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while running live tests Do you know which release you based the templates on? We should be able to take a look over our release notes between then and now, which should give a reasonable view of the changes, and then we can go from there. Equally, the commit history (https://github.com/Metaswitch/clearwater-heat/commits/stable) gives a fairly clear overview of what and why, with the pull requests adding a bit more detail. Though as we're likely to hit similar problems in future as we continue to develop Project Clearwater and our heat templates. Are the ONAP changes something we could be pulling in to the main code base? I'd be interested to know the extent of the work needed to keep them running From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On Behalf Of Mike Evans Sent: 12 December 2017 17:00 To: clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org> Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while running live tests Adam I put the ONAP templates together, so I'm happy to update them if they won't work with the latest Project Clearwater code. Is there anywhere I can look for an explanation of the changes? Mike From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On Behalf Of Adam Lindley Sent: 12 December 2017 16:41 To: clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org> Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while running live tests Hi Kedar, I've just had a look around for the templates on the ONAP wiki; I wasn't aware they had their own copies of them. I found https://wiki.onap.org/display/DW/Clearwater+vIMS+Onboarding+and+Instantiation . Is this what you've been working from? When deploying with the templates, have you been specifying a "repo_url" option in the arguments? The Clearwater HEAT templates included steps to install the Clearwater packages from our public stable repo by default, but the templates on the ONAP wiki are now somewhat out of date. If you haven't specified a repo containing packages from the same point in time the templates were effectively forked, you're going to be hitting conflicts between the code version, and the orchestration steps. I am not sure what differences there need to be for templates to run on ONAP, but I can point you towards our current set of heat templates: https://github.com/Metaswitch/clearwater-heat/tree/stable . I would suggest you try redeploying with these templates, as a fair number of things have changed in the code since the templates on the wiki were valid. If you hit any issues, do let us know. Cheers, Adam From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On Behalf Of Kedar Ambekar Sent: 12 December 2017 13:46 To: clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org> Subject: [Project Clearwater] 502 Bad Gateway and other errors while running live tests Hi, I have installed Clearwater with heat templates (available on ONAP wiki) directly on openstack. Appreciate help on below issues. 1. When I run Clearwater live tests, all tests fail with below error. RuntimeError thrown: - Account creation failed with HTTP code 502, body {"status": 502, "message": "Bad Gateway", "reason": "Upstream request failed", "detail": {"Upstream error": "599", "Upstream URL": "None"}, "error": true} 1. ps for bono process shows this. [bono]ubuntu@0:/etc/clearwater$ ps -ef |grep bono bono 6897 1 0 05:06 ? 00:00:11 /usr/share/clearwater/bin/bono --domain= --localhost=10.0.0.28,0.bono.me.cw-ngv.com --al ias=10.0.0.28,0.bono.me.cw-ngv.com, --pcscf=5060,5058 --webrtc-port=5062 --routing-proxy=icscf.,5052,50,600 --sas=0.0.0.0,b...@0.bono.me.cw-ngv.com --dns-server=127.0.0.1 --worker-threads=2 --analytics=/var/log/bono --log-file=/var/log/bono --log-level=2 --daemon --pidfile=/var/run/bono/bono.pid 1. In bono, I saw these logs. UTC Error sip_connection_pool.cpp:190: Failed to resolve icscf. to an IP address - Not found (PJ_ENOTFOUND) UTC Warning dnscachedresolver.cpp:828: Failed to retrieve record for icscf.: Domain name not found 1. To debug above issues, I ended up making these changes in /etc/init.d/bono and restarted bono service. DAEMON_ARGS="--domain=me.cw-ngv.com (instead of -domain=$home_domain) upstream_hostname=icscf.sprout.me.cw-ngv.com (instead of upstream_hostname=icscf.$sprout_hostname) 1. But now, I get below error in bono. Port 5052 is not in listen mode on sprout. UTC Warning pjsip: tcpc0x7f90cc07 TCP connect() error: Connection refused [code=120111] UTC Warning pjsip: tcpc0x7f90cc07 Unable to connect to <sprout ip>:5052 ============================================================================================================================ Disclaimer: This message and the information contained herein is proprietary and confidential and subject to the Tech Mahindra policy statement, you may review the policy at http://www.techmahindra.com/Disclaimer.html externally http://tim.techmahindra.com/tim/disclaimer.html internally within TechMahindra. ============================================================================================================================
_______________________________________________ Clearwater mailing list Clearwater@lists.projectclearwater.org http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org
