Team, All of DCAE is up via OOM (including the 7 CDAP nodes) Issue was: each tenant hits its floating IP allocation after 2.5 DCAE installs - we run out of IPs because they are not deleted. Fix: delete all unassociated IPs before bringing up OOM/DCAE - we cannot mix cloudify blueprint orchestration with manual openstack deletion - once in a blueprint - we need to remove everything orchestrated on top of HEAT using the cloudify manager - or do as the integration team does and clean the tenant before a deployment.
after deleting all floating IPs and rerunning the OOM deployment Time: 35 min from heat side dcae-boot install - 55 min total from one-click OOM install obrienbiometrics:lab_logging michaelobrien$ ssh [email protected] Last login: Fri Feb 9 16:50:48 2018 from 10.12.25.197 ubuntu@onap-oom-obrien:~$ kubectl -n onap-dcaegen2 exec -it heat-bootstrap-4010086101-fd5p2 bash root@heat-bootstrap:/# cd /opt/heat root@heat-bootstrap:/opt/heat# source DCAE-openrc-v3.sh root@heat-bootstrap:/opt/heat# openstack server list +--------------------------------------+---------------------+--------+----------------------------------------------------+--------------------------+------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+---------------------+--------+----------------------------------------------------+--------------------------+------------+ | 29990fcb-881f-457c-a386-aa32691d3beb | dcaepgvm00 | ACTIVE | oam_onap_3QKg=10.99.0.13, 10.12.6.144 | ubuntu-16-04-cloud-amd64 | m1.medium | | 7b4b63f3-c436-41a8-96dd-665baa94a698 | dcaecdap01 | ACTIVE | oam_onap_3QKg=10.99.0.19, 10.12.5.219 | ubuntu-16-04-cloud-amd64 | m1.large | | f4e6c499-8938-4e04-ab78-f0e753fe3cbb | dcaecdap00 | ACTIVE | oam_onap_3QKg=10.99.0.9, 10.12.6.69 | ubuntu-16-04-cloud-amd64 | m1.large | | 60ccff1f-e7c3-4ab4-b749-96aef7ee0b8c | dcaecdap04 | ACTIVE | oam_onap_3QKg=10.99.0.16, 10.12.5.106 | ubuntu-16-04-cloud-amd64 | m1.large | | df56d059-dc91-4122-a8de-d59ea14c5062 | dcaecdap05 | ACTIVE | oam_onap_3QKg=10.99.0.15, 10.12.6.131 | ubuntu-16-04-cloud-amd64 | m1.large | | 648ea7d3-c92f-4cd8-870f-31cb80eb7057 | dcaecdap02 | ACTIVE | oam_onap_3QKg=10.99.0.20, 10.12.6.128 | ubuntu-16-04-cloud-amd64 | m1.large | | c13fb83f-1011-44bb-bc6c-36627845a468 | dcaecdap06 | ACTIVE | oam_onap_3QKg=10.99.0.18, 10.12.6.134 | ubuntu-16-04-cloud-amd64 | m1.large | | 5ed7b172-1203-45a3-91e1-c97447ef201e | dcaecdap03 | ACTIVE | oam_onap_3QKg=10.99.0.6, 10.12.6.123 | ubuntu-16-04-cloud-amd64 | m1.large | | 80ada3ca-745e-42db-b67c-cdd83140e68e | dcaedoks00 | ACTIVE | oam_onap_3QKg=10.99.0.12, 10.12.6.173 | ubuntu-16-04-cloud-amd64 | m1.medium | | 5e9ef7af-abb3-4311-ae96-a2d27713f4c5 | dcaedokp00 | ACTIVE | oam_onap_3QKg=10.99.0.17, 10.12.6.168 | ubuntu-16-04-cloud-amd64 | m1.medium | | d84bbb08-f496-4762-8399-0aef2bb773c2 | dcaecnsl00 | ACTIVE | oam_onap_3QKg=10.99.0.7, 10.12.6.184 | ubuntu-16-04-cloud-amd64 | m1.medium | | 53f41bfc-9512-4a0f-b431-4461cd42839e | dcaecnsl01 | ACTIVE | oam_onap_3QKg=10.99.0.11, 10.12.6.188 | ubuntu-16-04-cloud-amd64 | m1.medium | | b6177cb2-5920-40b8-8f14-0c41b73b9f1b | dcaecnsl02 | ACTIVE | oam_onap_3QKg=10.99.0.4, 10.12.6.178 | ubuntu-16-04-cloud-amd64 | m1.medium | | 5e6fd14b-e75b-41f2-ad61-b690834df458 | dcaeorcl00 | ACTIVE | oam_onap_3QKg=10.99.0.8, 10.12.6.185 | CentOS-7 | m1.medium | | 5217dabb-abd7-4e57-972a-86efdd5252f5 | dcae-dcae-bootstrap | ACTIVE | oam_onap_3QKg=10.99.0.3, 10.12.6.183 | ubuntu-16-04-cloud-amd64 | m1.small | | 87569b68-cd4c-4a1f-9c6c-96ea7ce3d9b9 | onap-oom-obrien | ACTIVE | oam_onap_w37L=10.0.16.1, 10.12.6.124 | ubuntu-16-04-cloud-amd64 | m1.xxlarge | | d80f35ac-1257-47fc-828e-dddc3604d3c1 | oom-jenkins | ACTIVE | appc-multicloud-integration=10.10.5.14, 10.12.6.49 | | v1.xlarge | +--------------------------------------+---------------------+--------+----------------------------------------------------+--------------------------+------------+ root@heat-bootstrap:/opt/heat# Thanks to Alexis 2 days ago with his help I'll add these troubleshooting workarounds to the documentation in the wiki and RTD. http://onap.readthedocs.io/en/latest/submodules/dcaegen2.git/docs/sections/installation_heat.html https://wiki.onap.org/display/DW/vFWCL+instantiation%2C+testing%2C+and+debuging https://wiki.onap.org/display/DW/ONAP+on+Kubernetes+on+Rancher+in+OpenStack https://jira.onap.org/browse/LOG-167 https://jira.onap.org/browse/DOC-245 /michael -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Michael O'Brien Sent: Friday, February 9, 2018 11:37 To: OPNFV Tech <[email protected]>; [email protected] Subject: Re: [onap-discuss] [opnfv-tech-discuss] [Auto][ONAP] Are there any tips for installing "DCAE-only" on OPNFV/OpenStack Joe, Tina, Issue was lack of IP's - fixing (also I will increase the ram of the cloudify manager vm) - will document. HEAT side of an OOM deployment ubuntu@dcae-dcae-bootstrap:~$ sudo docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9de08c3eb9f6 nexus3.onap.org:10001/onap/org.onap.dcaegen2.deployments.bootstrap:v1.1.1 "/bin/sh -c 'exec ..." 10 hours ago Up 10 hours boot ubuntu@dcae-dcae-bootstrap:~$ sudo docker logs -f boot 2018-02-09 05:15:29 LOG <manager> [manager_resources_316df.create] WARNING: Ignoring validation errors. Validation Error: The provided host does not have enough memory to run Cloudify Manager (Current: 3790MB, Required: 3792MB). Validation Error: The Manager's Resources Package http://169.254.169.254/nosuchthing/cloudify-manager-resources_3.4.0-ga-b400.tar.gz is not accessible (HTTP Error: 404) 2018-02-09T05:34:24 CFY <cdap7> [fixedip_cdap00_e1701] Configuring node 2018-02-09T05:34:24 CFY <cdap7> [fixedip_cdap05_2d55b.create] Task failed 'neutron_plugin.port.create' -> No more IP addresses available on network dadac8e1-2f4b-4ed2-b289-26af8bbe44b0. [status_code=409] 2018-02-09T05:34:24 CFY <cdap7> [fixedip_cdap01_2c67f.create] Task failed 'neutron_plugin.port.create' -> No more IP addresses available on network dadac8e1-2f4b-4ed2-b289-26af8bbe44b0. [status_code=409] 2018-02-09T05:34:24 CFY <cdap7> 'install' workflow execution failed: RuntimeError: Workflow failed: Task failed 'neutron_plugin.port.create' -> No more IP addresses available on network dadac8e1-2f4b-4ed2-b289-26af8bbe44b0. [status_code=409] Execution of workflow install for deployment cdap7 failed. [error=Traceback (most recent call last): File "/tmp/pip-build-AednvY/cloudify-plugins-common/cloudify/dispatch.py", line 472, in _remote_workflow_child_thread File "/tmp/pip-build-AednvY/cloudify-plugins-common/cloudify/dispatch.py", line 504, in _execute_workflow_function -----Original Message----- From: Joe Kidder [mailto:[email protected]] Sent: Friday, February 9, 2018 06:57 To: Michael O'Brien <[email protected]> Cc: Tina Tsou <[email protected]>; OPNFV Tech <[email protected]>; [email protected] Subject: Re: [onap-discuss] [opnfv-tech-discuss] [Auto][ONAP] Are there any tips for installing "DCAE-only" on OPNFV/OpenStack Michael, Thanks for this email! I’m still parsing/consuming - the first link has much more background information/explanation than what I’d been looking at. Really great. Also thanks for taking the journey and leaving such good notes. Joe > On Feb 9, 2018, at 12:44 AM, Michael O'Brien <[email protected]> wrote: > > Tina, > Hi, this is Michael. I am bringing up DCAEGEN2 in the Intel/Windriver > openlab right now via OOM - on 25% of my time as I need it to exercise logs > in the closed loop components (dcae, policy, appc...). > > I have had the usual assistance from Alexis and understand the two > 25.2(everything except designate) and 25.5 (Designate) openstacks a lot more > now. > > My issues were misapplied tenant id's and the 3 keystone urls - I > read the DCAE readthedocs page in detail a couple times > http://onap.readthedocs.io/en/latest/submodules/dcaegen2.git/docs/sect > ions/installation_heat.html but still had some ambiguity on a couple > of the template values - writing this up as well under > https://jira.onap.org/browse/DOC-245 > > I also had an issue with running a 2nd oom to heat deployment because > of a hardcoded stack name being fixed (you will be ok for now with 1 > dcae setup per tenant - or the workaround) > https://jira.onap.org/browse/OOM-673 > > As of now I have passed heat-bootstrap (oom container in the onap-dcaegen2 > pod) creation of the zones in designate and the dcae-bootstrap vm via heat > template in the container. > Switching over to the heat side - the dcea-bootstrap vm has > orchestrated the cloudify vm (dcaeorcl00 - we don’t neet the > vm1-multi-service xxlarge anymore) - this takes 7 min Now we are out > of heat and running the cloudify tosca blueprint for the consul and > cdap nodes My last issue was a busted dcae key which caused a > cloudify-consul ssh issue The other DCAE nodes start to come up from > the dcaeorcl00 node > Dcaensls00/01/02 after 10 more min > Dcaedokp00 after 1 more min > Dcaedoks00 after 4 more min > Dcaecdap00/01/02/03/04/05/06 pending > Dcaepgvm00 pending > > Deleting a deployment - to get it to work you will need to wipe the entire > tenant because removing DCAE because of the hybrid HEAT/Cloudify > orchestration will require a quick script to delete the non-DCAE vms, delete > the recordsets from designate, delete the ports (a heat stack delete will > fail on a security group dependency), then delete the stack. After that > optionally delete your oom stack. > > Running details are on > https://jira.onap.org/browse/LOG-167 > > The original thread is on > https://lists.onap.org/pipermail/onap-discuss/2018-February/007884.htm > l > https://lists.onap.org/pipermail/onap-discuss/2018-February/007904.htm > l > > I will update Alexis's page tomorrow when I get the last of the cdap nodes > working - with some additional troubleshooting. > In the end a fully specified onap-parameters.yaml will work out of > the box on a clean openstack tenant with 180G > https://wiki.onap.org/display/DW/ONAP+on+Kubernetes+on+Rancher+in+Open > Stack#ONAPonKubernetesonRancherinOpenStack-Deployingthedcaegen2pod > > As aai rest calls for the region from the heat side to the aai side - work - > we should be ok for closed loop - once I rerun the vFW. > I'll send a mail when everything is up. > > The work you refer to is for Beijing (the DCAE work is not yet ported > there), if you use Amsterdam, DCAE will come up via OOM. > Later there is also full cloudify-manager containerization and a designate > port going on that should remove the heat dependency - this is pending. > > Running the heat stack directly (no OOM) does not work as well for me - I > cannot get any of the cloudify orchestrated VMs up - It does work in the lab > so it is a config issue on my part. For OOM I am getting further on the heat > side - just having an issue with the 7 cdap nodes. > > Thank you > /michael > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of Tina Tsou > Sent: Thursday, February 8, 2018 14:12 > To: Joe Kidder <[email protected]>; OPNFV Tech > <[email protected]> > Cc: [email protected] > Subject: Re: [onap-discuss] [opnfv-tech-discuss] [Auto][ONAP] Are > there any tips for installing "DCAE-only" on OPNFV/OpenStack > > Dear Joe, > > I cc ONAP list, hope to get a reply specifically. > > > Thank you, > Tina Tsou > Enterprise Architect > Arm > [email protected] > +1 (408)931-3833 > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of Joe > Kidder > Sent: Wednesday, February 7, 2018 12:47 PM > To: OPNFV Tech <[email protected]> > Subject: [opnfv-tech-discuss] [Auto][ONAP] Are there any tips for > installing "DCAE-only" on OPNFV/OpenStack > > Hi, > I’m looking for pointers for installing DCAE on an Openstack environment for > the Auto project. > We’re installing most of ONAP on kubernetes, but DCAE is not quite ready (a > couple of weeks out?) for deployment on k8s. So, we’d like to install DCAE > by itself (as much as possible) on an OPNFV bare metal pod’s open stack > environment. > > I’m not very experienced with ONAP. I have done a couple of ONAP > deployment attempts using the heat template, but am chasing a number > of small issues. My hope is that if I can limit the installation > scope to just DCAE, then it’s a little easier to focus;) > > In the meantime, I’ll continue working on full ONAP install on openstack > with the heat template. > > Thanks! > Joe Kidder > Auto Project contributor > _______________________________________________ > opnfv-tech-discuss mailing list > [email protected] > https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. > _______________________________________________ > onap-discuss mailing list > [email protected] > https://lists.onap.org/mailman/listinfo/onap-discuss > This message and the information contained herein is proprietary and > confidential and subject to the Amdocs policy statement, > > you may review at https://www.amdocs.com/about/email-disclaimer > <https://www.amdocs.com/about/email-disclaimer> This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer <https://www.amdocs.com/about/email-disclaimer> _______________________________________________ onap-discuss mailing list [email protected] https://lists.onap.org/mailman/listinfo/onap-discuss This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer <https://www.amdocs.com/about/email-disclaimer> _______________________________________________ opnfv-tech-discuss mailing list [email protected] https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
