I had missed to notice this in ellis heat log. I think this is the culprit.
+ sudo
/usr/share/clearwater/clearwater-config-manager/scripts/upload_shared_config
sudo:
/usr/share/clearwater/clearwater-config-manager/scripts/upload_shared_config:
command not found
Anyway, I did below to upload shared_config. Commands seem to have changed with
latest release.
[ellis]ubuntu@ellis-0:/var/log$ cw-config download shared_config
shared_config downloaded to
/home/ubuntu/clearwater-config-manager/ubuntu/shared_config
File was same as ellis shared_config file. I had to modify it with a dummy
change of putting dots in a comment line as upload command was detecting no
change.
[ellis]ubuntu@ellis-0:/var/log$ cw-config upload shared_config
Config successfully validated
Configuration file change: user ubuntu has modified shared_config.
Lines removed:
"# Deployment definitions"
Lines added:
"# Deployment definitions.."
Please check the config changes and confirm that you wish to continue with the
config upload.
Do you want to continue? [yes/no] yes
shared_config successfully uploaded
[ellis]ubuntu@ellis-0:/var/log$ cw-check_restart_queue_state
Describing the current queue state for apply_config_ellis
Nodes currently queued:
Node ID: 10.0.0.14-ellis, Node status: processing
[ellis]ubuntu@ellis-0:/var/log$ cw-check_restart_queue_state
Describing the current queue state for apply_config_ellis
No nodes are currently queued
[sprout]ubuntu@0:~$ cw-check_restart_queue_state
Describing the current queue state for apply_config_sprout
No nodes are currently queued
After this, if I download shared_config on sprout, I get the 'empty' file only
in /home/ubuntu/clearwater-config-manager/ubuntu directory. So clearly sprout
is not receiving the shared_config from cluster. Same is the case with bono,
homestead etc.
From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On
Behalf Of Kedar Ambekar
Sent: Wednesday, December 20, 2017 5:48 PM
To: clearwater@lists.projectclearwater.org
Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while
running live tests
Hi Ying,
[ellis]ubuntu@ellis-0:/etc/clearwater$ sudo cw-upload config shared_confg
sudo: cw-upload: command not found
Then tried below command as per
http://clearwater.readthedocs.io/en/stable/Modifying_Clearwater_settings.html
[ellis]ubuntu@ellis-0:/etc/clearwater$ sudo cw-upload_shared_config
sudo: cw-upload_shared_config: command not found
Am I missing anything ?
From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On
Behalf Of Ying Huang
Sent: Wednesday, December 20, 2017 5:13 PM
To:
clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org>
Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while
running live tests
Hi Kebar,
Yes as you noticed, the scscf uri is wrong and sprout uri isn't getting set
either. This is because your nodes are missing shared_config. The shared_config
on your Ellis looks right, so just run "sudo cw-upload config shared_confg"
(detailed instruction here
http://clearwater.readthedocs.io/en/stable/Clearwater_Configuration_Options_Reference.html#shared-config)
to upload it to etcd cluster for other nodes to pick up. Then run
"cw-check_restart_queue_state" on different types of nodes several times to see
the nodes restarting one by one to pick up the changes. If something goes wrong
here, we know the shared_config isn't being picked up correctly.
Thanks,
Ying
From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On
Behalf Of Kedar Ambekar
Sent: 20 December 2017 11:27
To:
clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org>
Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while
running live tests
Hi,
I have attached sprout_current.txt, which has some debug statements . I think
that is giving some pointers.
My guess is /etc/clearwater/local_config (or some other appropriate config
file) should have variables set for hostnames which are referred while
composing these URLs.
e.g sip:scscf.;transport=TCP should be
sip:scscf.<hostname<sip:scscf.%3chostname> of scscf>;transport=TCP
I will try is this direction..
Debug utils.cpp:405: Malformed host/port
Debug utils.cpp:405: Malformed host/port
Info main.cpp:1265: Sprout hostname set to
Info main.cpp:1243: S-CSCF node URI set to
Info main.cpp:1851: scscf enabled on 5054
Info main.cpp:1851: scscf prefix set to scscf
Info main.cpp:1851: scscf uri set to sip:scscf.;transport=TCP
Info main.cpp:1851: icscf enabled on 5052
Info main.cpp:1851: icscf prefix set to icscf
Info main.cpp:1851: icscf uri set to sip:icscf.;transport=TCP
Info main.cpp:1851: memento prefix set to memento
Info main.cpp:1851: memento uri set to sip:memento.;transport=TCP
Info main.cpp:1851: mangelwurzel prefix set to mangelwurzel
Info main.cpp:1851: mangelwurzel uri set to sip:mangelwurzel.;transport=TCP
Info main.cpp:1851: gemini prefix set to gemini
Info main.cpp:1851: gemini uri set to sip:gemini.;transport=TCP
Info main.cpp:1851: cdiv prefix set to cdiv
Info main.cpp:1851: cdiv uri set to sip:cdiv.;transport=TCP
Info main.cpp:1851: mmtel enabled on 5055
Info main.cpp:1851: mmtel prefix set to mmtel
Info main.cpp:1851: mmtel uri set to sip:mmtel.;transport=TCP
Info main.cpp:1851: bgcf enabled on 5053
Info main.cpp:1851: bgcf prefix set to bgcf
Info main.cpp:1851: bgcf uri set to sip:bgcf.;transport=TCP
Warning main.cpp:1855: SAS server option was invalid or not configured - SAS is
disabled
Error main.cpp:1894: S/I-CSCF enabled with no Homestead server
From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On
Behalf Of Kedar Ambekar
Sent: Wednesday, December 20, 2017 10:30 AM
To:
clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org>
Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while
running live tests
Hi Ying,
After changing log level to 5 on sprout, we need to restart sprout service,
right ?
In my case, sprout service is not running.
I guess for this, I need to change the heat template to create user_settings
file with log level 5.
[sprout]ubuntu@0:/var/log/sprout$ service sprout status
* sprout is not running
[sprout]ubuntu@0:/var/log/sprout$ ps -ef |grep sprout
ubuntu 8475 7053 0 04:53 pts/5 00:00:00 grep --color=auto sprout
root 10876 1 0 Dec18 ? 00:04:44
/usr/share/clearwater/clearwater-queue-manager/env/bin/python
/usr/share/clearwater/bin/clearwater-queue-manager --local-ip=10.0.0.26
--local-site=site1 --log-level=3
--log-directory=/var/log/clearwater-queue-manager
--pidfile=/var/run/clearwater-queue-manager.pid --etcd-key=clearwater
--node-type=sprout --wait-plugin-complete=Y
root 11042 1 0 Dec18 ? 00:04:41
/usr/share/clearwater/clearwater-cluster-manager/env/bin/python
/usr/share/clearwater/bin/clearwater-cluster-manager --mgmt-local-ip=10.0.0.26
--sig-local-ip=10.0.0.26 --local-site=site1 --remote-site=
--remote-cassandra-seeds= --signaling-namespace=
--uuid=535133a6-cb33-44b0-9a36-9da6e47ae280 --etcd-key=clearwater
--etcd-cluster-key=sprout --cluster-manager-enabled=Y --log-level=3
--log-directory=/var/log/clearwater-cluster-manager
--pidfile=/var/run/clearwater-cluster-manager.pid
From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On
Behalf Of Ying Huang
Sent: Wednesday, December 20, 2017 12:15 AM
To:
clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org>
Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while
running live tests
Hi Kedar,
Looks like a complicated situation to me. As far as I can see,
1. You should use log_level=5 to get more detailed logging
2. No the shared config shouldn't be the empty default template, this is an
error
3. What you said about IP sounds suspicious, as one of the log file says
"Failed to resolve icscf. to an IP address"
Must be frustrating to encounter errors everywhere. Shall we focus on the
Sprout process and see why it fails. Would you please set log_level=5 in
/etc/clearwater/user_settings on Sprout node and send a complete
sprout_current.txt?
Thanks,
Ying
From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On
Behalf Of Kedar Ambekar
Sent: 18 December 2017 14:00
To:
clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org>
Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while
running live tests
Hi,
6 continued.
6-a. bono errors attached for log locations mentioned in troubleshooting page.
I tried stopping bono service. It got started by monit but same errors continue.
6-b. sprout errors attached for log locations mentioned in troubleshooting
page. Sprout service is not running, so didn't try to stop to let me restart by
monit.
6-c. /var/log/ellis/ellis-err.log is empty.
/var/log/ellis/ellis_<time>.txt has these entries INFO web.py:1447: 200
GET /ping (0.0.0.0) 0.53ms.
I tried stopping ellis service. It got started by monit but don't see
anything changed.
/var/log/monit.log errors attached.
Appreciate any help here !
From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On
Behalf Of Kedar Ambekar
Sent: Monday, December 18, 2017 11:34 AM
To:
clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org>
Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while
running live tests
Hi Adam, Ying,
1. When I said shared_config is empty on sprout instances, I wanted to say
that no config statements in it.
[sprout]ubuntu@1:/etc/clearwater$ more shared_config
#####################################################################
# No Shared Config has been provided
# Replace this file with the Shared Configuration for your deployment
#####################################################################
1. /var/log/sprout/sprout_current has these entries periodically.
UTC Status utils.cpp:607: Log level set to 2
UTC Status main.cpp:1618: Access logging enabled to /var/log/sprout
UTC Warning signalhandler.h:90: SIGNAL already hooked
UTC Warning main.cpp:1674: SAS server option was invalid or not configured -
SAS is disabled
UTC Error main.cpp:1713: S/I-CSCF enabled with no Homestead server
1. So is it ok if only one node, that is going to push shared config to
cluster (ellis in the heat based deployment), to have shared_config file with
all the configuration and other nodes to have 'empty' file like above ?
1. PFA ellis's shared_config attached. I don't find any sproutlet related
configuration there..OK ? I have attached tail of ellis's heat installation
log. Looks ok to me. Couldn't find anything suspicious in remaining log file.
1. Btw, one thing I should have told this in the beginning - When I tried
the heat templates as it is, I observed that DNS VM's IP was not being passed
to all other components. So I ended up carving out DNS VM and instantiated that
first and used its IP in env file. I could confirm that IP got used by heat
instructions of other VMs and DNSMASQ on them got configured ok. nslookup for
components' names also works ok. (may be we need to use wait condition in the
heat template)
1. I am going through troubleshooting options and will update on that.
From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On
Behalf Of Adam Lindley
Sent: Friday, December 15, 2017 11:51 PM
To:
clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org>
Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while
running live tests
As a few more pointers of where to look, our HEAT templates currently generate
shared config on the Ellis node, and upload this to the cluster as a whole. It
would be worth looking at the install log on the ellis node to see if there was
some error there. You may have the shared_config file present, and be able to
upload that to the etcd cluster to bring the deployment to life.
If that doesn't give you any joy, make sure that the template install process
has completed successfully on each of your nodes. It seems odd that
shared_config has not been generated, and so I'd want to sanity check that
nothing else is going wrong. If all nodes are installed, but lacking config,
you should be able to generate shared config based on the script in the
ellis.yaml file.
As Ying has said, once those things are sorted, you can look in the process
logs to see why you are hitting errors. Almost all of our components log to
/var/log/<process_name> on their node, so that's a great place to start
debugging. You may also find our troubleshooting guide helpful:
https://clearwater.readthedocs.io/en/stable/Troubleshooting_and_Recovery.html
Cheers,
Adam
From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On
Behalf Of Ying Huang
Sent: 15 December 2017 17:54
To:
clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org>
Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while
running live tests
Hi Kedar,
Shared config shouldn't be empty. Looks like Sprout process has failed for some
reason. It's worthwhile checking the sprout log under
/var/sys/log/sprout/sprout_current or send it over to see what went wrong.
Thanks,
Ying
From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On
Behalf Of Kedar Ambekar
Sent: 14 December 2017 14:07
To:
clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org>
Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while
running live tests
Hi,
/etc/Clearwater/shared_config is empty on both the sprout instances post
installation. Is this expected if I want to run sproutlets like ICSCF, SCSCF at
least ?
Here are couple of commands I tried.
[sprout]ubuntu@0:~$ sudo monit summary
Monit 5.18.1 uptime: 7h 12m
Service Name Status Type
node-0.sprout.me.cw-ngv.com Running System
sprout_process Execution failed | Does... Process
snmpd_process Running Process
ntp_process Running Process
nginx_process Running Process
memcached_process Running Process
clearwater_queue_manager_pro... Running Process
etcd_process Running Process
clearwater_diags_monitor_pro... Running Process
clearwater_config_manager_pr... Running Process
clearwater_cluster_manager_p... Running Process
chronos_process Running Process
astaire_process Running Process
sprout_uptime Wait parent Program
poll_sprout_sip Wait parent Program
poll_sprout_http Wait parent Program
nginx_ping Status ok Program
nginx_uptime Status ok Program
monit_uptime Status ok Program
memcached_uptime Status ok Program
poll_memcached Status ok Program
clearwater_queue_manager_uptime Status ok Program
etcd_uptime Status ok Program
poll_etcd_cluster Status ok Program
poll_etcd Status ok Program
chronos_uptime Status ok Program
poll_chronos Status ok Program
astaire_uptime Status ok Program
[sprout]ubuntu@0:~$ service sprout status
* sprout is not running
[sprout]ubuntu@0:~$ sudo service sprout start
14-12-2017 14:06:17.215 UTC Status utils.cpp:496: Switching to daemon mode
From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On
Behalf Of Kedar Ambekar
Sent: Thursday, December 14, 2017 12:29 PM
To:
clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org>
Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while
running live tests
Hi,
I gave a try with release 127 ( repo URL
http://repo.cw-ngv.com/archive/repo127 in env file). But issues seem to be very
same.
From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On
Behalf Of Kedar Ambekar
Sent: Wednesday, December 13, 2017 12:30 PM
To:
clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org>
Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while
running live tests
Hi Adam, Mike,
Clearwater heat templates on ONAP wiki are put around July 13. Clearwater
release 127, which is released on May 30th , seems to have been used for
testing these templates.
If I want to give a try with release 127 installation via heat, how to do that
? I tried URL http://repo.cw-ngv.com/release-127/binary/Packages in browser,
but got 404.
From: Kedar Ambekar
Sent: Wednesday, December 13, 2017 11:37 AM
To:
clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org>
Subject: RE: 502 Bad Gateway and other errors while running live tests
Hi Adam,
Got the point. I had suspected the same.
I have repo_url http://repo.cw-ngv.com/stable in heat's env file.
Is it possible to use same release on which ONAP's heat templates are based,
may be by changing this repo URL?
I will have a look at current heat template on Clearwater GitHub but I think
those won't be ONAP compatible.
From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On
Behalf Of Adam Lindley
Sent: Tuesday, December 12, 2017 10:50 PM
To:
clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org>
Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while
running live tests
Do you know which release you based the templates on?
We should be able to take a look over our release notes between then and now,
which should give a reasonable view of the changes, and then we can go from
there. Equally, the commit history
(https://github.com/Metaswitch/clearwater-heat/commits/stable) gives a fairly
clear overview of what and why, with the pull requests adding a bit more detail.
Though as we're likely to hit similar problems in future as we continue to
develop Project Clearwater and our heat templates. Are the ONAP changes
something we could be pulling in to the main code base? I'd be interested to
know the extent of the work needed to keep them running
From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On
Behalf Of Mike Evans
Sent: 12 December 2017 17:00
To:
clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org>
Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while
running live tests
Adam
I put the ONAP templates together, so I'm happy to update them if they won't
work with the latest Project Clearwater code. Is there anywhere I can look for
an explanation of the changes?
Mike
From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On
Behalf Of Adam Lindley
Sent: 12 December 2017 16:41
To:
clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org>
Subject: Re: [Project Clearwater] 502 Bad Gateway and other errors while
running live tests
Hi Kedar,
I've just had a look around for the templates on the ONAP wiki; I wasn't aware
they had their own copies of them.
I found
https://wiki.onap.org/display/DW/Clearwater+vIMS+Onboarding+and+Instantiation .
Is this what you've been working from?
When deploying with the templates, have you been specifying a "repo_url" option
in the arguments? The Clearwater HEAT templates included steps to install the
Clearwater packages from our public stable repo by default, but the templates
on the ONAP wiki are now somewhat out of date. If you haven't specified a repo
containing packages from the same point in time the templates were effectively
forked, you're going to be hitting conflicts between the code version, and the
orchestration steps.
I am not sure what differences there need to be for templates to run on ONAP,
but I can point you towards our current set of heat templates:
https://github.com/Metaswitch/clearwater-heat/tree/stable . I would suggest you
try redeploying with these templates, as a fair number of things have changed
in the code since the templates on the wiki were valid. If you hit any issues,
do let us know.
Cheers,
Adam
From: Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org] On
Behalf Of Kedar Ambekar
Sent: 12 December 2017 13:46
To:
clearwater@lists.projectclearwater.org<mailto:clearwater@lists.projectclearwater.org>
Subject: [Project Clearwater] 502 Bad Gateway and other errors while running
live tests
Hi,
I have installed Clearwater with heat templates (available on ONAP wiki)
directly on openstack. Appreciate help on below issues.
1. When I run Clearwater live tests, all tests fail with below error.
RuntimeError thrown:
- Account creation failed with HTTP code 502, body {"status": 502,
"message": "Bad Gateway", "reason": "Upstream request failed", "detail":
{"Upstream error": "599", "Upstream URL": "None"}, "error": true}
1. ps for bono process shows this.
[bono]ubuntu@0:/etc/clearwater$ ps -ef |grep bono
bono 6897 1 0 05:06 ? 00:00:11 /usr/share/clearwater/bin/bono --domain=
--localhost=10.0.0.28,0.bono.me.cw-ngv.com --al
ias=10.0.0.28,0.bono.me.cw-ngv.com, --pcscf=5060,5058 --webrtc-port=5062
--routing-proxy=icscf.,5052,50,600 --sas=0.0.0.0,b...@0.bono.me.cw-ngv.com
--dns-server=127.0.0.1 --worker-threads=2 --analytics=/var/log/bono
--log-file=/var/log/bono --log-level=2 --daemon --pidfile=/var/run/bono/bono.pid
1. In bono, I saw these logs.
UTC Error sip_connection_pool.cpp:190: Failed to resolve icscf. to an IP
address - Not found (PJ_ENOTFOUND)
UTC Warning dnscachedresolver.cpp:828: Failed to retrieve record for icscf.:
Domain name not found
1. To debug above issues, I ended up making these changes in
/etc/init.d/bono and restarted bono service.
DAEMON_ARGS="--domain=me.cw-ngv.com (instead of -domain=$home_domain)
upstream_hostname=icscf.sprout.me.cw-ngv.com (instead of
upstream_hostname=icscf.$sprout_hostname)
1. But now, I get below error in bono. Port 5052 is not in listen mode on
sprout.
UTC Warning pjsip: tcpc0x7f90cc07 TCP connect() error: Connection refused
[code=120111]
UTC Warning pjsip: tcpc0x7f90cc07 Unable to connect to <sprout ip>:5052
============================================================================================================================
Disclaimer: This message and the information contained herein is proprietary
and confidential and subject to the Tech Mahindra policy statement, you may
review the policy at http://www.techmahindra.com/Disclaimer.html externally
http://tim.techmahindra.com/tim/disclaimer.html internally within TechMahindra.
============================================================================================================================
_______________________________________________
Clearwater mailing list
Clearwater@lists.projectclearwater.org
http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org
