Hi James and Dan, I restarted sdnc pod to enable karaf log, after the restart sdnc healthcheck passed. I don’t know if you or someone else happened to make some change, or restart helped. Similarly after I restarted APPC pod, its healthcheck passed as well. This is Casablanca released version, we will make a note of it to see if it happens again.
Thanks for your help, -Yang From: [email protected] [mailto:[email protected]] On Behalf Of Yang Xu Sent: Wednesday, December 12, 2018 2:43 PM To: [email protected]; [email protected]; TIMONEY, DAN <[email protected]> Subject: Re: [onap-discuss] [sdnc][appc][integration] SDNC and APPC fail on healthcheck with 404 Hi James, Thanks for the reply. I don’t know much about SDNC cluster settings. On SB05, from Robot container I can ping sdnc-cluster.onap and also I can telenet sdnc.onap 8282 port. I am guessing, port 8181 is only used internally by SDNC domain to check heartbeat of each replica, so port 8181 doesn’t need to be exposed as a service port. Please let me if you need more info about SB05 testbed. Regards, -Yang From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of James MacNider Sent: Wednesday, December 12, 2018 1:30 PM To: Yang Xu (Yang, Fixed Network) <[email protected]<mailto:[email protected]>>; TIMONEY, DAN <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Subject: Re: [onap-discuss] [sdnc][appc][integration] SDNC and APPC fail on healthcheck with 404 It’s possible there’s a misconfiguration here, but I’ve not had the chance to investigate yet. One thing to keep in mind is that when we’re discussing “domains” in the context of a Kubernetes cluster, what you’re actually referring to is the name of a Kubernetes service that is used to load-balance across one or more pods. These names are tied to specific ports that the pods expose. In the case of SDN-C, these are the service names that are being exposed, along with their associated ports: $ kubectl get services -n onap NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE … controller-blueprints ClusterIP 10.43.14.62 <none> 8080/TCP 16m controller-blueprints-db ClusterIP None <none> 3306/TCP 16m … sdnc NodePort 10.43.161.171 <none> 8282:30202/TCP,8202:30208/TCP,8280:30246/TCP,8443:30267/TCP 16m sdnc-ansible-server ClusterIP 10.43.83.83 <none> 8000/TCP 16m sdnc-cluster ClusterIP None <none> 2550/TCP 16m sdnc-dbhost ClusterIP None <none> 3306/TCP 16m sdnc-dbhost-read ClusterIP 10.43.51.29 <none> 3306/TCP 16m sdnc-dgbuilder NodePort 10.43.50.6 <none> 3000:30203/TCP 16m sdnc-dmaap-listener ClusterIP None <none> <none> 16m sdnc-portal NodePort 10.43.185.129 <none> 8843:30201/TCP 16m sdnc-sdnctldb01 ClusterIP None <none> 3306/TCP 16m sdnc-sdnctldb02 ClusterIP None <none> 3306/TCP 16m sdnc-ueb-listener ClusterIP None <none> <none> 16m So if the cluster health check is using port 8181, this could a problem, as I don’t’ see any services exposing that port… From: Yang Xu (Yang, Fixed Network) <[email protected]<mailto:[email protected]>> Sent: Wednesday, December 12, 2018 12:43 PM To: TIMONEY, DAN <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; James MacNider <[email protected]<mailto:[email protected]>> Subject: RE: [onap-discuss] [sdnc][appc][integration] SDNC and APPC fail on healthcheck with 404 Dan, I see your point. From sdnc oom script, I see sdnc-cluster.onap is used for SDNC cluster health check and it uses port 8181. The issue I reported is from SDNC component health check, the port used is 8282. Maybe both sdnc-cluster.onap and sdnc.onap are valid. Thanks, -Yang From: TIMONEY, DAN [mailto:[email protected]] Sent: Wednesday, December 12, 2018 9:59 AM To: Yang Xu (Yang, Fixed Network) <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; MACNIDER, JAMES <[email protected]<mailto:[email protected]>> Subject: Re: [onap-discuss] [sdnc][appc][integration] SDNC and APPC fail on healthcheck with 404 Yang, The problem is that the scripts that setup clustering rely on the domain sdnc-cluster.onap being setup. I just rebuilt my WindRiver test ONAP environment last week, following the instructions in the OOM User Guide for installing with Rancher, and using the casablanca branch of OOM. In my environment, the pod names and domain name work as they should. If we're saying the domain name should be sdnc.onap instead of sdnc-cluster.onap, I'd be okay with making that change (which is really in OOM). However, the sdnc.onap domain also isn't working between pods (i.e. if I exec to dev-sdnc-sdnc-0, I can't get to dev-sdnc-sdnc-1.sdnc.onap either). Also, I notice in my local environment that I can ping dev-sdnc-1.sdnc-cluster.onap from dev-sdnc-0, but not dev-sdnc-1.sdnc.onap ... so I think the domain 'sdnc-cluster.onap' is correct, but for some reason not working. James – can you perhaps take a look at SB05? I’m not really familiar enough with the details of how that sdnc-cluster.onap domain is set up within K8S to be of too much more help here other than to point out it’s not working. Dan Dan Timoney Principal Technical Staff Member AT&T Email : [email protected]<mailto:[email protected]> Office : +1 (732) 420-3226 Mobile : +1 (201) 960-1211 200 S Laurel Ave, Rm E2-2A03 Middletown, NJ 08873 From: "Yang Xu (Yang, Fixed Network)" <[email protected]<mailto:[email protected]>> Date: Tuesday, December 11, 2018 at 11:43 PM To: "TIMONEY, DAN" <[email protected]<mailto:[email protected]>>, onap-discuss <[email protected]<mailto:[email protected]>> Subject: RE: [onap-discuss] [sdnc][appc][integration] SDNC and APPC fail on healthcheck with 404 Hi Dan, I really appreciate you looked into the issue. The healthcheck request is sent from ONAP Robot docker, and from Robot report log I can see the domain is sdnc.onap which can be resolved when testing with telnet from Robot. In fact, when I try to access SDNC API page with http://sdnc_external_ip:30202/apidoc/explorer/index.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__sdnc-5Fexternal-5Fip-3A30202_apidoc_explorer_index.html&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=Ms-DNhR23dO4sQFfYRm4ow&m=Br1l2AklrgZQToyKfpJ6ktBL8xbsbvxuEt3_u21uEfc&s=RquV8JPmf7KUk71vDRnba_rhntyrvyjRNckhBaC3TSc&e=>, it also fails with 404. The same url works for an ONAP deployment from last week. Regards, -Yang From: TIMONEY, DAN [mailto:[email protected]] Sent: Tuesday, December 11, 2018 6:27 PM To: [email protected]<mailto:[email protected]>; Yang Xu (Yang, Fixed Network) <[email protected]<mailto:[email protected]>> Subject: Re: [onap-discuss] [sdnc][appc][integration] SDNC and APPC fail on healthcheck with 404 Yang, Sorry, I meant to explain a little more ….. We use the sdnc-cluster.onap domain names in our akka.conf file to set up clustering. So, if that domain isn’t working – as seems to be the case in SB05 – the nodes won’t be able to cluster properly. That’s what’s causing that 404 error. So the root cause of this problem is that the sdnc-cluster.onap domain isn’t resolving properly. As I mentioned, that’s actually working for me in my windriver env, so I suspect the error is probably some environment error. Dan -- Dan Timoney SDN-CP Development ONAP Project Technical Lead : CCSDK and SDNC Please go to D2 ECOMP Release Planning Wiki<https://wiki.web.att.com/display/DERP/D2+ECOMP+Release+Planning+Home> for D2 ECOMP Project In-take, 2016 Release Planning, Change Management, and find key Release Planning Contact Information. From: "TIMONEY, DAN" <[email protected]<mailto:[email protected]>> Date: Tuesday, December 11, 2018 at 6:21 PM To: onap-discuss <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: [onap-discuss] [sdnc][appc][integration] SDNC and APPC fail on healthcheck with 404 Yang, I replied to this on RocketChat as well, but there’s some environmental issue going on in that environment. In my local k8s env, based on the latest OOM casablanca branch in gerrit, my pods are named dev-sdnc-0, dev-sdnc-1 and dev-sdnc-2, and K8S defines an ‘sdnc-cluster.onap’ domain where the nodes can refer to each other. In other words, if I exec into dev-sdnc-0, I can then telnet to dev-sdnc-1.sdnc-cluster.onap on port 2550 to confirm akka connectivity. In SB05, the sdnc node names have the ‘sdnc’ part doubled (e.g. dev-sdnc-sdnc-0), and the sdnc-cluster.onap domain is not working (i.e. telnet dev-sdnc-sdnc-1.sdnc-cluster.onap tells me name/service not known). I think this is some sort of environment issue, maybe with the environment.yaml file for that environment. In my local k8s env, I’m just using the out of the box OOM helm charts (only change I make is in onap/values.yaml to disable the components I don’t need). Dan -- Dan Timoney SDN-CP Development ONAP Project Technical Lead : CCSDK and SDNC Please go to D2 ECOMP Release Planning Wiki<https://wiki.web.att.com/display/DERP/D2+ECOMP+Release+Planning+Home> for D2 ECOMP Project In-take, 2016 Release Planning, Change Management, and find key Release Planning Contact Information. From: onap-discuss <[email protected]<mailto:[email protected]>> on behalf of Yang Xu <[email protected]<mailto:[email protected]>> Reply-To: onap-discuss <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Tuesday, December 11, 2018 at 1:31 PM To: onap-discuss <[email protected]<mailto:[email protected]>> Subject: [onap-discuss] [sdnc][appc][integration] SDNC and APPC fail on healthcheck with 404 Hi Dan and Taka, I deployed ONAP Casablanca on SB-05 in Windriver lab and found both SDNC and APPC failed their healthcheck with 404 error. SDNC and APPC pods seem ok and other ONAP modules passed their healthcheck. Please help take a look. Regards, -Yang This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14459): https://lists.onap.org/g/onap-discuss/message/14459 Mute This Topic: https://lists.onap.org/mt/28722186/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
