Have you checked the network for the project the additional DCAE VMs are created in? This error http://localhost:9998/resources/services/vm-docker-host-1 code:400 msg:REQUEST-FAILED-DEPLOY-4001W Unable to find network: oam_ecomp_B6pc makes me suspicious as it seems to find the private network oam_ecomp_B6pc. Double check the config in Openstack vs. /opt/config/openstack_private_network_name.txt on DCAE controller vs. /opt/app/dcae-controller/config.yaml (line 11 for me ;1:OPENSTACK-PRIVATE-NETWORK: oam_ecomp_<4-letter-random-value>). They all need to be the same. If not, you need to adapt the two files in the DCAE controller to the name in Openstack and give it a new try. If they all are the same .. .let us know :-)
Mit freundlichen Grü?en / Kind regards Josef Reisinger From: "Arun Arora (c)" <[email protected]> To: Josef Reisinger <[email protected]> Cc: "LUND, CARSTEN (CARSTEN)" <[email protected]>, "[email protected]" <[email protected]> Date: 21.07.2017 14:25 Subject: RE: [onap-discuss] DCAE controller-startup.sh fails and throws error Hi Josef, I was able to move past the Keystone issue and get the keystone token with V2.0. The ngrep tool helped, Thanks!! Now I am seeing errors with an HTTP request to localhost:9998 inside the DCAE-Controller docker. http://localhost:9998/resources/services/vm-docker-host-1 Attaching the relevant logs. Also, I tried using CURL for the same request and got 403 Forbidden. Heres the output. I am guessing 403 is received as I didn’t put and Auth Headers in the request, but not entirely sure of that: root@a1ec8e324a7c:/opt/app/dcae-controller-platform-server# curl -i http://localhost:9998/resources/services/vm-docker-host-1 HTTP/1.1 403 Forbidden Access-Control-Allow-Methods: POST, GET, OPTIONS, PUT Access-Control-Allow-Origin: * Access-Control-Allow-Headers: Origin, Content-Type, Authorization, X-ECOMP-RequestID, * Access-Control-Allow-Credentials: true Access-Control-Request-Headers: Origin, Content-Type, Authorization, X-ECOMP-RequestID, * Content-Length: 0 Server: Jetty(8.1.17.v20150415) Is the error in log familiar? Any pointers on why HTTP req is getting 400 Error? Thanks, Arun From: Josef Reisinger [mailto:[email protected]] Sent: Thursday, July 20, 2017 7:23 PM To: Arun Arora (c) <[email protected]> Cc: LUND, CARSTEN (CARSTEN) <[email protected]>; [email protected] Subject: RE: [onap-discuss] DCAE controller-startup.sh fails and throws error That looks strange, nevertheless, let's change gears now. I would assume that looking at the network traffic may unveil which part of the communication does not work. my view (and Carsten et al, please correct me if I am wrong) is that DCAE controller needs the endpoints of various services (Nova, Neutron, etc..) from Openstack. To do so, the controller needs to get a token from Openstack. See https://developer.openstack.org/api-ref/identity/v2/?expanded=authenticate-detail , look for POST to /v2.0/tokens. If you want to dig that deep, install ngrep (apt install ngrep). cd to /opt/dcae-startup-vm-controller, type "make down", maybe ./init.sh. Open a second terminal to the DCAE controller and type ngrep -W byline . "port 5000"| tee keystone.log This logs all traffic to port 5000 a bit better formatted as tcpdump(1) In the first window, type "make up" and after this docker logs -f dcaestartupvmcontroller_dcae-controller_1. At some point in time you should see activity in the second terminal. Wait until you see the well-known error message in terminal 1 and ctrl-C out of ngrep in terminal 2. You might want to analyse keystone.log. You should see the request sent to your OS controller and the response given back. If you wish, post the log file on this list (make sure you sensitive information is deleted). Btw, What about carrying the discussion to https://wiki.onap.org/questions to add to our common knowledge? Mit freundlichen Grü?en / Kind regards Josef Reisinger From: "Arun Arora (c)" <[email protected]> To: Josef Reisinger <[email protected]> Cc: "LUND, CARSTEN (CARSTEN)" <[email protected]>, " [email protected]" <[email protected]> Date: 20.07.2017 15:02 Subject: RE: [onap-discuss] DCAE controller-startup.sh fails and throws error Hi Josef, Do you mean reboot the controller node? BTW, I did the change in keystone_url.txt and re-executed dcae_install.sh after cleaning docker image, container and all downloaded files in /opt. Seen the error again, log are attached (controller-startup-exec3.txt) While re-running DCAE I realized one more change I did which was needed to start the container. However, I am not sure if the change is impacting the controller’s ability to reach the OpenStack controller or not. To show the change, I re-executed dcae_install.sh after deleting docker image, container and all downloaded files in /opt . Logs are attached (dcae_install-exec.txt) The error I got was: (Line 370) ERROR: Network "dcaestartupvmcontroller_default" needs to be recreated - options have changed make: *** [up] Error 1 To work-around the error I removed the ‘network’ section from vi /opt/dcae-startup-vm-controller/docker-compose.yml (Line 370) and executed dcae_vm_init.sh (Line 380) Attaching both docker-compose.yml files (docker-compose(modified).yml, docker-compose(unchanged).yml) Thanks, Arun From: Josef Reisinger [mailto:[email protected]] Sent: Thursday, July 20, 2017 5:48 PM To: Arun Arora (c) <[email protected]> Cc: LUND, CARSTEN (CARSTEN) <[email protected]>; [email protected] Subject: RE: [onap-discuss] DCAE controller-startup.sh fails and throws error Arun, your are lucky. The v2.0 version of the API is deprecated, but enabled. Changing keystone_url.txt and rebooting should help you to get a step forward. Mit freundlichen Grü?en / Kind regards Josef Reisinger From: "Arun Arora (c)" <[email protected]> To: "LUND, CARSTEN (CARSTEN)" <[email protected]>, Josef Reisinger <[email protected]> Cc: "[email protected]" <[email protected]> Date: 20.07.2017 13:53 Subject: RE: [onap-discuss] DCAE controller-startup.sh fails and throws error Thanks Josef, Carsten, Let me try changing the URL in keystone_url.txt file. For tenant ID and password, I have checked them and they are fine. Before making change to the keyston_url.txt I executed curl for keystone port and got the following result. FYI, I replaced the IP of controller with <Controller IP> string viouser@loadbalancer01:~$ curl http://<ControllerIP>:5000 curl: /opt/vmware/lib/vami/libcurl.so.4: no version information available (required by curl) {"versions": {"values": [{"status": "stable", "updated": "2016-04-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.6", "links": [{"href": "http://<Controller IP>:5000/v3/", "rel": "self"}]}, {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://<Controller IP>:5000/v2.0/", "rel": "self"}, {"href": " http://docs.openstack.org/";, "type": "text/html", "rel": "describedby"}]}]}} Thanks, Arun From: [email protected][ mailto:[email protected]] On Behalf Of LUND, CARSTEN (CARSTEN) Sent: Thursday, July 20, 2017 5:16 PM To: Josef Reisinger <[email protected]> Cc: [email protected] Subject: Re: [onap-discuss] DCAE controller-startup.sh fails and throws error You are right the URL looks wrong, but Openstack can be setup so many ways that I would not know what is right or wrong. Arun, try to work with CURL to get a request that works to this API when we can determine how the config.yaml need to look. Carsten From: Josef Reisinger [mailto:[email protected]] Sent: Thursday, July 20, 2017 7:41 AM To: LUND, CARSTEN (CARSTEN) <[email protected]> Cc: Arun Arora <[email protected]>; [email protected]; PLATANIA, MARCO (MARCO) <[email protected]> Subject: RE: [onap-discuss] DCAE controller-startup.sh fails and throws error Carsten, thanks for the pointer. Seems to me the same issue in the JIRA case... a wrong keystone URL? At least the one in this mail string and the one in JIRA are structure-wise different from mine, for which I can say it works :-) Mit freundlichen Grü?en / Kind regards Josef Reisinger When wisdom comes to call, there's nobody listening at all - Pendragon / Man Of Nomadic Traits Certified IT-Architect Telecommunications IBM Certified Telecommunications Industry ITA Global Telecommunications, Media and Entertainment Industry Lehrbeauftragter an der Hochschule Fresenius IBM Deutschland Godesberger Allee 127 53175 Bonn Beuel Phone: +49 151 1426 4559 Mobile: +49-(0) 151 1426 4559 E-Mail: [email protected] IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter Gesch?ftsführung: Martina Koederitz (Vorsitzende), Nicole Reimer, Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Stefan Lutz Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 From: "LUND, CARSTEN (CARSTEN)" <[email protected]> To: Josef Reisinger <[email protected]>, " [email protected]" <[email protected]> Cc: Arun Arora <[email protected]>, "PLATANIA, MARCO (MARCO)" <[email protected]> Date: 20.07.2017 13:15 Subject: RE: [onap-discuss] DCAE controller-startup.sh fails and throws error Josef, I should have added comment to this email thread too, but I added some comment to the JIRA https://jira.onap.org/browse/DCAEGEN2-69 In fact for Arun the REST calls to Openstack goes through but something is the Keystone API call is wrong, password or tenantId. Carsten From: Josef Reisinger [mailto:[email protected]] Sent: Thursday, July 20, 2017 4:44 AM To: [email protected] Cc: Arun Arora <[email protected]>; LUND, CARSTEN (CARSTEN) < [email protected]>; PLATANIA, MARCO (MARCO) <[email protected] > Subject: Re: [onap-discuss] DCAE controller-startup.sh fails and throws error Arun, I visited the docker logs of the docker container in the DCAE controller to compare them with yours. As far as I can see, I get the same "File not found" exceptions for example exception: OPENECOMP-DEMO-ZONE/cdap-instances/cdap-helloworld.yaml class java.nio.file.NoSuchFileException Same for the jar files error: cp: cannot stat '/opt/app/dcae-*-jars/lib/*.jar': No such file or directory cp: cannot stat '/opt/app/dcae-*-model/lib/*.jar': No such file or directory cp: cannot stat '/opt/app/dcae-*-manager/lib/*.jar': No such file or directory cp: cannot stat '/opt/app/dcae-*-adaptor/lib/*.jar': No such file or directory cp: cannot stat '/opt/app/ncomp-*-jars/lib/*.jar': No such file or directory cp: cannot stat '/opt/app/ncomp-*-model/lib/*.jar': No such file or directory cp: cannot stat '/opt/app/ncomp-*-manager/lib/*.jar': No such file or directory cp: cannot stat '/opt/app/ncomp-*-adaptor/lib/*.jar': No such file or directory and even the same for error: Loading runtime factory: org.openecomp.dcae.controller.service.servers.docker.DcaeDockerFactory I ignored these errors in the very beginning and well, DCAE works for me :-) This line bin/dcae-controller.sh deploy-user -l $ZONE -p ONAP -u $OPENSTACK_KEYNAME Caught: org.openecomp.ncomp.sirius.manager.Jetty8ClientException: HTTP Request Failed: URL: http://localhost:9998/resources/services/vm-postgresqlcode:400 msg:REQUEST-FAILED-UNDEPLOY-4001W REQUEST-FAILED-DELETESERVER-4001W !Absolute URI: null/servers is the first real error and remember to have seen this "null" in the URI once also.. but I didn't write thesolution down, so I need to make some assumptions here. I think it has to do with the communication with the openstack controller. What I remember is the usage of the default host name 'controller' for the controller host. Not a good choice btw, but we did it. To verify this is the correct path, could you post the content of /opt/config/keystone_url.txt on the dcae controller and the output of the command "openstack endpoint list" on the openstack controller? Or, likewise, check whether there is a host name in either of them? If there is a hostname, can it be resolved from the container in the DCAE controller, i.e. with the command docker exec -it dcaestartupvmcontroller_dcae-controller_1 ping controller Mit freundlichen Grü?en / Kind regards Josef Reisinger When wisdom comes to call, there's nobody listening at all - Pendragon / Man Of Nomadic Traits Certified IT-Architect Telecommunications IBM Certified Telecommunications Industry ITA Global Telecommunications, Media and Entertainment Industry Lehrbeauftragter an der Hochschule Fresenius IBM Deutschland Godesberger Allee 127 53175 Bonn Beuel Phone: +49 151 1426 4559 Mobile: +49-(0) 151 1426 4559 E-Mail: [email protected] IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter Gesch?ftsführung: Martina Koederitz (Vorsitzende), Nicole Reimer, Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Stefan Lutz Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 From: "Arun Arora (c)" <[email protected]> To: Arun Arora <[email protected]>, "LUND, CARSTEN (CARSTEN)" <[email protected]>, "PLATANIA, MARCO (MARCO)" < [email protected]> Cc: "[email protected]" <[email protected]> Date: 20.07.2017 09:43 Subject: Re: [onap-discuss] DCAE controller-startup.sh fails and throws error Sent by: [email protected] Forgot to attach the logs. Sending now. Thanks, Arun From: Arun Arora (c) Sent: Thursday, July 20, 2017 1:11:05 PM To: Arun Arora; LUND, CARSTEN (CARSTEN); PLATANIA, MARCO (MARCO) Cc: [email protected] Subject: RE: [onap-discuss] DCAE controller-startup.sh fails and throws error @Carsten, I redeployed and didn’t see any extra SPACE in the logs. However, it also didn’t help. Seeing the same issue. Also, I didn’t see any SPACE in config.yaml so no change was required as well. It might be logs saving issue where I saved the Linux terminal logs into Windows Notepad, can’t say for sure though. @Marco, tried the commands you suggested but no joy. Thanks, Arun From: Arun Arora [mailto:[email protected]] Sent: Thursday, July 20, 2017 12:59 AM To: Arun Arora (c) <[email protected]>; LUND, CARSTEN (CARSTEN) < [email protected]>; PLATANIA, MARCO (MARCO) <[email protected] > Cc: [email protected] Subject: Re: [onap-discuss] DCAE controller-startup.sh fails and throws error Thanks Marco. Will get back here soon with results. Regards, Arun From: PLATANIA, MARCO (MARCO) Sent: Thursday, July 20, 12:09 AM Subject: Re: [onap-discuss] DCAE controller-startup.sh fails and throws error To: Arun Arora (c), LUND, CARSTEN (CARSTEN) Cc: [email protected] Arun, You can modify the yaml file manually. Then you have to run “make down” to shutdown the entire environment and then “bash init.sh” and “make up” to startup again. Those files are in /opt/dcae-startup-vm-controller. No need to rebuild ONAP. Thanks, Marco From: <[email protected]> on behalf of "Arun Arora (c)" <[email protected]> Date: Wednesday, July 19, 2017 at 2:00 PM To: "LUND, CARSTEN (CARSTEN)" <[email protected]> Cc: "[email protected]" <[email protected]> Subject: Re: [onap-discuss] DCAE controller-startup.sh fails and throws error Okay, thanks for pointing that out. I think redeploying ONAP stack is not necessary? Will it be fine if I just change the Zone in config.yaml in DCAE and rerun dcae_install.shmanually? Also, do you think this is causing the problem of missing files and directories? Thanks, Arun On Jul 19, 2017, at 11:06 PM, "LUND, CARSTEN (CARSTEN)" < [email protected]> wrote: I noticed this from the log + export 'ZONE=NorthIndia ' + ZONE='NorthIndia ' Note the space after NorthIndia. Please remove from config.yaml and redeploy. Carsten From: [email protected][ mailto:[email protected]] On Behalf Of Arun Arora (c) Sent: Wednesday, July 19, 2017 9:29 AM To: MAJHI, TAPAN <[email protected]>; Josef Reisinger < [email protected]>; FREEMAN, BRIAN D <[email protected]>; [email protected] Subject: Re: [onap-discuss] DCAE controller-startup.sh fails and throws error Hi All, Regarding the issue I am facing for DCAE VMs deployment, the immediate blocker for me is required files and directories are not found (like cdap-instances, docker-instances, docker-instances). I am suspecting these directories are created in run-time as I searched them in the DCAA VM, Container and DCAE Demo Gittree but couldn’t find them. Attaching the execution logs for your reference. All errors for missing files, directories are mentioned as exceptions in the attached execution log. 1) I tried with rel-1.0and rel-1.1 but both gave same errors 2) Also tried with both 2-NIC and 1-NIC-FLOATING-IPS. No luck in this as well 3) Currently using 1-NIC-FLOATING-IPS 4) Verified config.yaml parameters with all configurations and modifications in https://jira.onap.org/browse/DCAEGEN2-69and https://jira.onap.org/browse/DCAEGEN2-7 5) Found some more config issues in the config.yaml and fixed 6) Finally, made sure the execution is going as per the sequence mentioned in https://wiki.onap.org/display/DW/DCAE +Controller+Development+Guide The problem happens with the following command execution done as part of ‘ make gen-config sync restart’ java -cp 'lib/*' org.openecomp.dcae.controller.operation.utils.GenControllerConfiguration $ZONE . GITLINK OPENECOMP-DEMO I am not able to find the directories/ files listed in the errors. Any pointers from anyone who have fixed these issues? @Tapan, @Joseph, @Brian, special mention to you for any pointers to resolve this issue, as you have finished the deployment (Congrats BTW J ) Thanks & Regards, Arun Arora PS: Wanted to mention again, I am deploying on VIO (VMware Integrated OpenStack) From: Kanagaraj Manickam [mailto:[email protected]] Sent: Tuesday, July 18, 2017 8:29 PM To: Arun Arora (c) <[email protected]>; [email protected] Subject: RE: DCAE controller-startup.sh fails and throws error We also faced this issue and created an JIRA ID: DCAEGEN2-69 And we tried to provided required conf and logs. Please let us know if additional details required. Thanks. Regards Kanagaraj M *************************************************************************************** 本邮件及其附件含有华为公司的保密信息，仅限于发送给上面地址中列出的个人或群 组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散 发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除 本邮件！ ************************************************************************************** *************************************************************************************** This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! *************************************************************************************** From: [email protected][ mailto:[email protected]] On Behalf Of Arun Arora (c) Sent: Tuesday, July 18, 2017 7:47 PM To: [email protected] Subject: [onap-discuss] DCAE controller-startup.sh fails and throws error Hi All, We are trying to deploy ONAP on VIO. Using the following YAML and env files: https://gerrit.onap.org/r/gitweb?p=demo.git;a=blob;f=heat/OpenECOMP /onap_openstack_float.yaml;h=659f94c93b9c5dfae02031ddcae82161ce927318;hb =refs/heads/master https://gerrit.onap.org/r/gitweb?p=demo.git;a=blob;f=heat/OpenECOMP /onap_openstack_float.yaml;h=659f94c93b9c5dfae02031ddcae82161ce927318;hb =refs/heads/master In the DCAE VM, the dcae-controller docker is running however, controller-startup.sh inside it fails with error. Seen the following and fixed 1) /etc/hosts has static IP addresses for each DCAE VM which does not match with the Private subnet IP addresses mentioned in ENV file. So changed the IP addresses in /etc/hosts 2) FYI, the floating IPs for DCAE VMs are created in the VIO and are also mentioned in the ENV file (Total 6 IPs including 1 for dcae-controller) After this, on executing controller-startup.sh, varied errors are seem and finally additional VMs are not created. The command execution gets stuck in a wait loop. Some of the errors are following. 1) destinationRoot=GITLINK/OPENECOMP-DEMO-NorthIndia/controller-gen exception: OPENECOMP-DEMO-NorthIndia/cdap-instances/cdap-tca-hi-lo.yaml class java.nio.file.NoSuchFileException exception: OPENECOMP-DEMO-NorthIndia/cdap-instances/cdap-helloworld.yaml class java.nio.file.NoSuchFileException exception: OPENECOMP-DEMO-NorthIndia/docker-instances/docker-common-event.yaml class java.nio.file.NoSuchFileException exception: OPENECOMP-DEMO-NorthIndia/docker-instances/docker-databus-controller.yaml class java.nio.file.NoSuchFileException exception: OPENECOMP-DEMO-NorthIndia/chef.yaml class java.nio.file.NoSuchFileException exception: OPENECOMP-DEMO-NorthIndia/vm-instances/vm-controller.yaml class java.nio.file.NoSuchFileException XXXXX: ./OPENECOMP-DEMO-NorthIndia/vm-config/vm-controller exception: OPENECOMP-DEMO-NorthIndia/vm-instances/vm-postgresql.yaml class java.nio.file.NoSuchFileException 2) Enter keystore password: keytool error: java.lang.Exception: Certificate not imported, alias <simpledemo-root-ca> already exists Complete Log file is attached. Attaching all logs files for reference. Pls point to any possible solution for this. Thanks & Regards, Arun Arora [attachment "controller-startup-exec2.txt" deleted by Josef Reisinger/Germany/IBM] _______________________________________________ onap-discuss mailing list [email protected] https://lists.onap.org/mailman/listinfo/onap-discuss [attachment "controller-startup-exec3.txt" deleted by Josef Reisinger/Germany/IBM] [attachment "dcae_install-exec.txt" deleted by Josef Reisinger/Germany/IBM] [attachment "docker-compose(modified).yml" deleted by Josef Reisinger/Germany/IBM] [attachment "docker-compose(unchanged).yml" deleted by Josef Reisinger/Germany/IBM] [attachment "controller-startup-exec4.txt" deleted by Josef Reisinger/Germany/IBM]
_______________________________________________ onap-discuss mailing list [email protected] https://lists.onap.org/mailman/listinfo/onap-discuss
