Hi Ying, After changing log level to 5 on sprout, we need to restart sprout service, right ? In my case, sprout service is not running.
I guess for this, I need to change the heat template to create user_settings file with log level 5. [sprout]ubuntu@0:/var/log/sprout$ service sprout status * sprout is not running [sprout]ubuntu@0:/var/log/sprout$ ps -ef |grep sprout ubuntu 8475 7053 0 04:53 pts/5 00:00:00 grep --color=auto sprout root 10876 1 0 Dec18 ? 00:04:44 /usr/share/clearwater/clearwater-queue-manager/env/bin/python /usr/share/clearwater/bin/clearwater-queue-manager --local-ip=10.0.0.26 --local-site=site1 --log-level=3 --log-directory=/var/log/clearwater-queue-manager --pidfile=/var/run/clearwater-queue-manager.pid --etcd-key=clearwater --node-type=sprout --wait-plugin-complete=Y root 11042 1 0 Dec18 ? 00:04:41 /usr/share/clearwater/clearwater-cluster-manager/env/bin/python /usr/share/clearwater/bin/clearwater-cluster-manager --mgmt-local-ip=10.0.0.26 --sig-local-ip=10.0.0.26 --local-site=site1 --remote-site= --remote-cassandra-seeds= --signaling-namespace= --uuid=535133a6-cb33-44b0-9a36-9da6e47ae280 --etcd-key=clearwater --etcd-cluster-key=sprout --cluster-manager-enabled=Y --log-level=3 --log-directory=/var/log/clearwater-cluster-manager --pidfile=/var/run/clearwater-cluster-manager.pid From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On Behalf Of Ying Huang Sent: Wednesday, December 20, 2017 12:15 AM To: clearwater@lists.projectclearwater.org Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while running live tests Hi Kedar, Looks like a complicated situation to me. As far as I can see, 1. You should use log_level=5 to get more detailed logging 2. No the shared config shouldn't be the empty default template, this is an error 3. What you said about IP sounds suspicious, as one of the log file says "Failed to resolve icscf. to an IP address" Must be frustrating to encounter errors everywhere. Shall we focus on the Sprout process and see why it fails. Would you please set log_level=5 in /etc/clearwater/user_settings on Sprout node and send a complete sprout_current.txt? Thanks, Ying From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On Behalf Of Kedar Ambekar Sent: 18 December 2017 14:00 To: clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org> Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while running live tests Hi, 6 continued. 6-a. bono errors attached for log locations mentioned in troubleshooting page. I tried stopping bono service. It got started by monit but same errors continue. 6-b. sprout errors attached for log locations mentioned in troubleshooting page. Sprout service is not running, so didn't try to stop to let me restart by monit. 6-c. /var/log/ellis/ellis-err.log is empty. /var/log/ellis/ellis_<time>.txt has these entries INFO web.py:1447: 200 GET /ping (0.0.0.0) 0.53ms. I tried stopping ellis service. It got started by monit but don't see anything changed. /var/log/monit.log errors attached. Appreciate any help here ! From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On Behalf Of Kedar Ambekar Sent: Monday, December 18, 2017 11:34 AM To: clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org> Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while running live tests Hi Adam, Ying, 1. When I said shared_config is empty on sprout instances, I wanted to say that no config statements in it. [sprout]ubuntu@1:/etc/clearwater$ more shared_config ##################################################################### # No Shared Config has been provided # Replace this file with the Shared Configuration for your deployment ##################################################################### 1. /var/log/sprout/sprout_current has these entries periodically. UTC Status utils.cpp:607: Log level set to 2 UTC Status main.cpp:1618: Access logging enabled to /var/log/sprout UTC Warning signalhandler.h:90: SIGNAL already hooked UTC Warning main.cpp:1674: SAS server option was invalid or not configured - SAS is disabled UTC Error main.cpp:1713: S/I-CSCF enabled with no Homestead server 1. So is it ok if only one node, that is going to push shared config to cluster (ellis in the heat based deployment), to have shared_config file with all the configuration and other nodes to have 'empty' file like above ? 1. PFA ellis's shared_config attached. I don't find any sproutlet related configuration there..OK ? I have attached tail of ellis's heat installation log. Looks ok to me. Couldn't find anything suspicious in remaining log file. 1. Btw, one thing I should have told this in the beginning - When I tried the heat templates as it is, I observed that DNS VM's IP was not being passed to all other components. So I ended up carving out DNS VM and instantiated that first and used its IP in env file. I could confirm that IP got used by heat instructions of other VMs and DNSMASQ on them got configured ok. nslookup for components' names also works ok. (may be we need to use wait condition in the heat template) 1. I am going through troubleshooting options and will update on that. From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On Behalf Of Adam Lindley Sent: Friday, December 15, 2017 11:51 PM To: clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org> Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while running live tests As a few more pointers of where to look, our HEAT templates currently generate shared config on the Ellis node, and upload this to the cluster as a whole. It would be worth looking at the install log on the ellis node to see if there was some error there. You may have the shared_config file present, and be able to upload that to the etcd cluster to bring the deployment to life. If that doesn't give you any joy, make sure that the template install process has completed successfully on each of your nodes. It seems odd that shared_config has not been generated, and so I'd want to sanity check that nothing else is going wrong. If all nodes are installed, but lacking config, you should be able to generate shared config based on the script in the ellis.yaml file. As Ying has said, once those things are sorted, you can look in the process logs to see why you are hitting errors. Almost all of our components log to /var/log/<process_name> on their node, so that's a great place to start debugging. You may also find our troubleshooting guide helpful: https://clearwater.readthedocs.io/en/stable/Troubleshooting_and_Recovery.html Cheers, Adam From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On Behalf Of Ying Huang Sent: 15 December 2017 17:54 To: clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org> Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while running live tests Hi Kedar, Shared config shouldn't be empty. Looks like Sprout process has failed for some reason. It's worthwhile checking the sprout log under /var/sys/log/sprout/sprout_current or send it over to see what went wrong. Thanks, Ying From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On Behalf Of Kedar Ambekar Sent: 14 December 2017 14:07 To: clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org> Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while running live tests Hi, /etc/Clearwater/shared_config is empty on both the sprout instances post installation. Is this expected if I want to run sproutlets like ICSCF, SCSCF at least ? Here are couple of commands I tried. [sprout]ubuntu@0:~$ sudo monit summary Monit 5.18.1 uptime: 7h 12m Service Name Status Type node-0.sprout.me.cw-ngv.com Running System sprout_process Execution failed | Does... Process snmpd_process Running Process ntp_process Running Process nginx_process Running Process memcached_process Running Process clearwater_queue_manager_pro... Running Process etcd_process Running Process clearwater_diags_monitor_pro... Running Process clearwater_config_manager_pr... Running Process clearwater_cluster_manager_p... Running Process chronos_process Running Process astaire_process Running Process sprout_uptime Wait parent Program poll_sprout_sip Wait parent Program poll_sprout_http Wait parent Program nginx_ping Status ok Program nginx_uptime Status ok Program monit_uptime Status ok Program memcached_uptime Status ok Program poll_memcached Status ok Program clearwater_queue_manager_uptime Status ok Program etcd_uptime Status ok Program poll_etcd_cluster Status ok Program poll_etcd Status ok Program chronos_uptime Status ok Program poll_chronos Status ok Program astaire_uptime Status ok Program [sprout]ubuntu@0:~$ service sprout status * sprout is not running [sprout]ubuntu@0:~$ sudo service sprout start 14-12-2017 14:06:17.215 UTC Status utils.cpp:496: Switching to daemon mode From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On Behalf Of Kedar Ambekar Sent: Thursday, December 14, 2017 12:29 PM To: clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org> Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while running live tests Hi, I gave a try with release 127 ( repo URL http://repo.cw-ngv.com/archive/repo127 in env file). But issues seem to be very same. From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On Behalf Of Kedar Ambekar Sent: Wednesday, December 13, 2017 12:30 PM To: clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org> Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while running live tests Hi Adam, Mike, Clearwater heat templates on ONAP wiki are put around July 13. Clearwater release 127, which is released on May 30th , seems to have been used for testing these templates. If I want to give a try with release 127 installation via heat, how to do that ? I tried URL http://repo.cw-ngv.com/release-127/binary/Packages in browser, but got 404. From: Kedar Ambekar Sent: Wednesday, December 13, 2017 11:37 AM To: clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org> Subject: RE: 502 Bad Gateway and other errors while running live tests Hi Adam, Got the point. I had suspected the same. I have repo_url http://repo.cw-ngv.com/stable in heat's env file. Is it possible to use same release on which ONAP's heat templates are based, may be by changing this repo URL? I will have a look at current heat template on Clearwater GitHub but I think those won't be ONAP compatible. From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On Behalf Of Adam Lindley Sent: Tuesday, December 12, 2017 10:50 PM To: clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org> Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while running live tests Do you know which release you based the templates on? We should be able to take a look over our release notes between then and now, which should give a reasonable view of the changes, and then we can go from there. Equally, the commit history (https://github.com/Metaswitch/clearwater-heat/commits/stable) gives a fairly clear overview of what and why, with the pull requests adding a bit more detail. Though as we're likely to hit similar problems in future as we continue to develop Project Clearwater and our heat templates. Are the ONAP changes something we could be pulling in to the main code base? I'd be interested to know the extent of the work needed to keep them running From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On Behalf Of Mike Evans Sent: 12 December 2017 17:00 To: clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org> Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while running live tests Adam I put the ONAP templates together, so I'm happy to update them if they won't work with the latest Project Clearwater code. Is there anywhere I can look for an explanation of the changes? Mike From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On Behalf Of Adam Lindley Sent: 12 December 2017 16:41 To: clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org> Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while running live tests Hi Kedar, I've just had a look around for the templates on the ONAP wiki; I wasn't aware they had their own copies of them. I found https://wiki.onap.org/display/DW/Clearwater+vIMS+Onboarding+and+Instantiation . Is this what you've been working from? When deploying with the templates, have you been specifying a "repo_url" option in the arguments? The Clearwater HEAT templates included steps to install the Clearwater packages from our public stable repo by default, but the templates on the ONAP wiki are now somewhat out of date. If you haven't specified a repo containing packages from the same point in time the templates were effectively forked, you're going to be hitting conflicts between the code version, and the orchestration steps. I am not sure what differences there need to be for templates to run on ONAP, but I can point you towards our current set of heat templates: https://github.com/Metaswitch/clearwater-heat/tree/stable . I would suggest you try redeploying with these templates, as a fair number of things have changed in the code since the templates on the wiki were valid. If you hit any issues, do let us know. Cheers, Adam From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On Behalf Of Kedar Ambekar Sent: 12 December 2017 13:46 To: clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org> Subject: [Project Clearwater] 502 Bad Gateway and other errors while running live tests Hi, I have installed Clearwater with heat templates (available on ONAP wiki) directly on openstack. Appreciate help on below issues. 1. When I run Clearwater live tests, all tests fail with below error. RuntimeError thrown: - Account creation failed with HTTP code 502, body {"status": 502, "message": "Bad Gateway", "reason": "Upstream request failed", "detail": {"Upstream error": "599", "Upstream URL": "None"}, "error": true} 1. ps for bono process shows this. [bono]ubuntu@0:/etc/clearwater$ ps -ef |grep bono bono 6897 1 0 05:06 ? 00:00:11 /usr/share/clearwater/bin/bono --domain= --localhost=10.0.0.28,0.bono.me.cw-ngv.com --al ias=10.0.0.28,0.bono.me.cw-ngv.com, --pcscf=5060,5058 --webrtc-port=5062 --routing-proxy=icscf.,5052,50,600 --sas=0.0.0.0,b...@0.bono.me.cw-ngv.com --dns-server=127.0.0.1 --worker-threads=2 --analytics=/var/log/bono --log-file=/var/log/bono --log-level=2 --daemon --pidfile=/var/run/bono/bono.pid 1. In bono, I saw these logs. UTC Error sip_connection_pool.cpp:190: Failed to resolve icscf. to an IP address - Not found (PJ_ENOTFOUND) UTC Warning dnscachedresolver.cpp:828: Failed to retrieve record for icscf.: Domain name not found 1. To debug above issues, I ended up making these changes in /etc/init.d/bono and restarted bono service. DAEMON_ARGS="--domain=me.cw-ngv.com (instead of -domain=$home_domain) upstream_hostname=icscf.sprout.me.cw-ngv.com (instead of upstream_hostname=icscf.$sprout_hostname) 1. But now, I get below error in bono. Port 5052 is not in listen mode on sprout. UTC Warning pjsip: tcpc0x7f90cc07 TCP connect() error: Connection refused [code=120111] UTC Warning pjsip: tcpc0x7f90cc07 Unable to connect to <sprout ip>:5052 ============================================================================================================================ Disclaimer: This message and the information contained herein is proprietary and confidential and subject to the Tech Mahindra policy statement, you may review the policy at http://www.techmahindra.com/Disclaimer.html externally http://tim.techmahindra.com/tim/disclaimer.html internally within TechMahindra. ============================================================================================================================
_______________________________________________ Clearwater mailing list Clearwater@lists.projectclearwater.org http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org