Hi Feng I finally managed to bring up “os-nosdn-nofeature-noha” scenario using 4.0-20170620 version RPMs. Initially I was able to login to undercloud and overcloud and create nova instances and neutron networks within overcloud. However, after keeping that setup idle for a day or two, I noticed that I am not able to login to those VMs anymore. I still have all the VMs running.
[root@jumphost2 vagrant]# virsh list --all Id Name State ---------------------------------------------------- 1 undercloud running 4 baremetal1 running 5 baremetal0 running [root@jumphost2 vagrant]# sudo opnfv-util undercloud ssh: Could not resolve hostname : Name or service not known Seems like the reason for this error is there is no corresponding ARP entry to the MAC address of the undercloud/overcloud VMs in ARP table. For some reason, all the ARP entries on that hosts are in “incomplete” state (as shown below). Are u aware of any condition where we may hit this type of issue? Is there any way to recover from this state? [root@jumphost2 vagrant]# arp -an ? (192.168.122.45) at <incomplete> on virbr0 ? (192.168.122.42) at <incomplete> on virbr0 ? (192.168.122.35) at <incomplete> on virbr0 ? (192.168.122.24) at <incomplete> on virbr0 ? (192.168.122.17) at <incomplete> on virbr0 ? (192.168.122.14) at <incomplete> on virbr0 ? (192.168.122.7) at <incomplete> on virbr0 ? (192.168.122.252) at <incomplete> on virbr0 ? (192.168.122.245) at <incomplete> on virbr0 ? (192.168.122.242) at <incomplete> on virbr0 ? (192.168.122.235) at <incomplete> on virbr0 ? (192.168.122.224) at <incomplete> on virbr0 ? (192.168.122.217) at <incomplete> on virbr0 ? (192.168.122.214) at <incomplete> on virbr0 ? (192.168.122.207) at <incomplete> on virbr0 ? (192.168.122.196) at <incomplete> on virbr0 ? (192.168.121.1) at 52:54:00:48:67:32 [ether] on eth0 ? (192.168.122.49) at <incomplete> on virbr0 ? (192.168.122.46) at <incomplete> on virbr0 ? (192.168.122.39) at <incomplete> on virbr0 ? (192.168.122.28) at <incomplete> on virbr0 ? (192.168.122.21) at <incomplete> on virbr0 ? (192.168.122.18) at <incomplete> on virbr0 ? (192.168.122.11) at <incomplete> on virbr0 ? (192.168.122.249) at <incomplete> on virbr0 ? (192.168.122.246) at <incomplete> on virbr0 ? (192.168.122.239) at <incomplete> on virbr0 ? (192.168.122.228) at <incomplete> on virbr0 ? (192.168.122.221) at <incomplete> on virbr0 ? (192.168.122.218) at <incomplete> on virbr0 ? (192.168.122.211) at <incomplete> on virbr0 ? (192.168.122.200) at <incomplete> on virbr0 ? (192.168.122.193) at <incomplete> on virbr0 ? (192.168.122.190) at <incomplete> on virbr0 ? (192.168.122.50) at <incomplete> on virbr0 ? (192.168.122.43) at <incomplete> on virbr0 ? (192.168.122.32) at <incomplete> on virbr0 ? (192.168.122.25) at <incomplete> on virbr0 ? (192.168.122.22) at <incomplete> on virbr0 ? (192.168.122.15) at <incomplete> on virbr0 ? (192.168.122.4) at <incomplete> on virbr0 ? (192.168.122.253) at <incomplete> on virbr0 ? (192.168.122.250) at <incomplete> on virbr0 ? (192.168.122.243) at <incomplete> on virbr0 ? (192.168.122.232) at <incomplete> on virbr0 ? (192.168.122.225) at <incomplete> on virbr0 ? (192.168.122.222) at <incomplete> on virbr0 ? (192.168.122.215) at <incomplete> on virbr0 ? (192.168.122.204) at <incomplete> on virbr0 ? (192.168.122.197) at <incomplete> on virbr0 ? (192.168.122.194) at <incomplete> on virbr0 ? (192.168.122.47) at <incomplete> on virbr0 ? (192.168.122.36) at <incomplete> on virbr0 ? (192.168.122.29) at <incomplete> on virbr0 ? (192.168.122.26) at <incomplete> on virbr0 ? (192.168.122.19) at <incomplete> on virbr0 ? (192.168.122.8) at <incomplete> on virbr0 ? (192.168.122.254) at <incomplete> on virbr0 ? (192.168.122.247) at <incomplete> on virbr0 ? (192.168.122.236) at <incomplete> on virbr0 ? (192.168.122.229) at <incomplete> on virbr0 ? (192.168.122.226) at <incomplete> on virbr0 ? (192.168.122.219) at <incomplete> on virbr0 ? (192.168.122.208) at <incomplete> on virbr0 ? (192.168.122.201) at <incomplete> on virbr0 ? (192.168.122.198) at <incomplete> on virbr0 ? (192.168.122.191) at <incomplete> on virbr0 ? (192.168.122.51) at <incomplete> on virbr0 ? (192.168.122.40) at <incomplete> on virbr0 ? (192.168.122.33) at <incomplete> on virbr0 ? (192.168.122.30) at <incomplete> on virbr0 ? (192.168.122.23) at <incomplete> on virbr0 ? (192.168.122.12) at <incomplete> on virbr0 ? (192.168.122.5) at <incomplete> on virbr0 ? (192.168.122.2) at <incomplete> on virbr0 ? (192.168.122.251) at <incomplete> on virbr0 ? (192.168.122.240) at <incomplete> on virbr0 ? (192.168.122.233) at <incomplete> on virbr0 ? (192.168.122.230) at <incomplete> on virbr0 ? (192.168.122.223) at <incomplete> on virbr0 ? (192.168.122.212) at <incomplete> on virbr0 ? (192.168.122.205) at <incomplete> on virbr0 ? (192.168.122.202) at <incomplete> on virbr0 ? (192.168.122.195) at <incomplete> on virbr0 ? (192.168.122.44) at <incomplete> on virbr0 ? (192.168.122.37) at <incomplete> on virbr0 ? (192.168.122.34) at <incomplete> on virbr0 ? (192.168.122.27) at <incomplete> on virbr0 ? (192.168.122.16) at <incomplete> on virbr0 ? (192.168.122.9) at <incomplete> on virbr0 ? (192.168.122.6) at <incomplete> on virbr0 ? (192.168.122.244) at <incomplete> on virbr0 ? (192.168.122.237) at <incomplete> on virbr0 ? (192.168.122.234) at <incomplete> on virbr0 ? (192.168.122.227) at <incomplete> on virbr0 ? (192.168.122.216) at <incomplete> on virbr0 ? (192.168.122.209) at <incomplete> on virbr0 ? (192.168.122.206) at <incomplete> on virbr0 ? (192.168.122.199) at <incomplete> on virbr0 ? (192.168.122.48) at <incomplete> on virbr0 ? (192.168.122.41) at <incomplete> on virbr0 ? (192.168.122.38) at <incomplete> on virbr0 ? (192.168.122.31) at <incomplete> on virbr0 ? (192.168.122.20) at <incomplete> on virbr0 ? (192.168.122.13) at <incomplete> on virbr0 ? (192.168.122.10) at <incomplete> on virbr0 ? (192.168.122.3) at <incomplete> on virbr0 ? (192.168.122.248) at <incomplete> on virbr0 ? (192.168.122.241) at <incomplete> on virbr0 ? (192.168.122.238) at <incomplete> on virbr0 ? (192.168.122.231) at <incomplete> on virbr0 ? (192.168.122.220) at <incomplete> on virbr0 ? (192.168.122.213) at <incomplete> on virbr0 ? (192.168.122.210) at <incomplete> on virbr0 ? (192.168.122.203) at <incomplete> on virbr0 ? (192.168.122.192) at <incomplete> on virbr0 Thanks Srikanth From: Feng Pan [mailto:f...@redhat.com] Sent: Tuesday, June 20, 2017 4:29 PM To: Srikanth Vavilapalli <srikanth.vavilapa...@ericsson.com> Cc: opnfv-tech-discuss@lists.opnfv.org Subject: Re: [opnfv-tech-discuss] [APEX] Apex virtual deployment install fails with error Hi Srikanth, The failure you see is caused by a syntax error in ovs-dpdk-preconfig.yaml that we use to configure ovs-dpdk. I submitted a patch to fix this: https://gerrit.opnfv.org/gerrit/#/c/36277/ You could either wait for this fix to be merged or apply and build this patch yourself. Thanks Feng On Fri, Jun 16, 2017 at 1:37 AM, Srikanth Vavilapalli <srikanth.vavilapa...@ericsson.com<mailto:srikanth.vavilapa...@ericsson.com>> wrote: Hi Feng This time I tried with following danube daily rpms: wget http://artifacts.opnfv.org/apex/danube/opnfv-apex-undercloud-4.0-20170613.noarch.rpm wget http://artifacts.opnfv.org/apex/danube/opnfv-apex-common-4.0-20170613.noarch.rpm wget http://artifacts.opnfv.org/apex/danube/opnfv-apex-4.0-20170613.noarch.rpm I see the following errors with this install (attached full log). Seems some issues with OVS-DPDK deployment. Appreciate any pointers on what may be going wrong? Could this be because of any issue with my jumphost config? I am running this script on a 64GB RAM centos 7 VM. 2017-06-15 20:24:21Z [overcloud.Compute.0.ComputeExtraConfigPre]: CREATE_FAILED Error: resources.ComputeExtraConfigPre.resources.OvsDpdkSetup: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1 2017-06-15 20:24:21Z [overcloud.Compute.0]: CREATE_FAILED Resource CREATE failed: Error: resources.ComputeExtraConfigPre.resources.OvsDpdkSetup: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1 2017-06-15 20:24:22Z [overcloud.Compute.0]: CREATE_FAILED Error: resources[0].resources.ComputeExtraConfigPre.resources.OvsDpdkSetup: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1 2017-06-15 20:24:22Z [overcloud.Compute]: CREATE_FAILED Resource CREATE failed: Error: resources[0].resources.ComputeExtraConfigPre.resources.OvsDpdkSetup: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1 2017-06-15 20:24:23Z [overcloud.Compute]: CREATE_FAILED Error: resources.Compute.resources[0].resources.ComputeExtraConfigPre.resources.OvsDpdkSetup: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1 2017-06-15 20:24:23Z [overcloud]: CREATE_FAILED Resource CREATE failed: Error: resources.Compute.resources[0].resources.ComputeExtraConfigPre.resources.OvsDpdkSetup: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1 Stack overcloud CREATE_FAILED One other observation I can see all the three VMs (undercloud, baremetal0, baremetal1) are in running state. But I am unable to login to undercloud VM. [vagrant@jumphost2 ~]$ sudo virsh list --all Id Name State ---------------------------------------------------- 1 undercloud running 4 baremetal0 running 5 baremetal1 running [vagrant@jumphost2 ~]$ opnfv-util undercloud root error: failed to get domain 'undercloud' error: Domain not found: no domain with matching name 'undercloud' Usage: grep [OPTION]... PATTERN [FILE]... Try 'grep --help' for more information. ssh: Could not resolve hostname : Name or service not known Thanks Srikanth From: Feng Pan [mailto:f...@redhat.com<mailto:f...@redhat.com>] Sent: Wednesday, June 14, 2017 8:07 PM To: Srikanth Vavilapalli <srikanth.vavilapa...@ericsson.com<mailto:srikanth.vavilapa...@ericsson.com>> Cc: opnfv-tech-discuss@lists.opnfv.org<mailto:opnfv-tech-discuss@lists.opnfv.org> Subject: Re: [opnfv-tech-discuss] [APEX] Apex virtual deployment install fails with error Hi Srikanth, You are getting this error because your jump host doesn't have internet connectivity during deploy process. Those dependencies should have been added to the latest daily rpms, can you try using those instead of Danube 2.0 release? Thanks Feng On Wed, Jun 14, 2017 at 4:03 PM, Srikanth Vavilapalli <srikanth.vavilapa...@ericsson.com<mailto:srikanth.vavilapa...@ericsson.com>> wrote: Thanks Feng This time I tried with rpms from Danube branch on a fresh centos 7 VM but got different errors (below) while undercloud install… Plz find the attached full log. I installed the following rpms: sudo yum install https://repos.fedorapeople.org/repos/openstack/openstack-newton/rdo-release-newton-4.noarch.rpm sudo yum install epel-release sudo yum install http://artifacts.opnfv.org/apex/dependencies/python34-markupsafe-0.23-9.el7.centos.x86_64.rpm sudo yum install http://artifacts.opnfv.org/apex/dependencies/python3-ipmi-0.3.0-1.noarch.rpm sudo yum install http://artifacts.opnfv.org/apex/dependencies/python3-jinja2-2.8-5.el7.centos.noarch.rpm wget http://artifacts.opnfv.org/apex/danube/opnfv-apex-undercloud-4.0-danube-2.0.noarch.rpm wget http://artifacts.opnfv.org/apex/danube/opnfv-apex-common-4.0-danube.2.0.noarch.rpm wget http://artifacts.opnfv.org/apex/danube/opnfv-apex-4.0-danube.2.0.noarch.rpm sudo yum -y install opnfv-apex-undercloud-4.0-danube-2.0.noarch.rpm opnfv-apex-common-4.0-danube.2.0.noarch.rpm opnfv-apex-4.0-danube.2.0.noarch.rpm Errors in the log: 2017-06-14 19:41:27 - _[mNotice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]/returns: ERROR: (pymysql.err.OperationalError) (1045, u"Access denied for user 'heat'@'192.0.2.1' (using password: YES)")_[0m 2017-06-14 19:41:27 - _[1;31mError: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Failed to call refresh: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0]_[0m 2017-06-14 19:41:27 - _[1;31mError: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0]_[0m … + rc=6 + set -e + echo 'puppet apply exited with exit code 6' puppet apply exited with exit code 6 + '[' 6 '!=' 2 -a 6 '!=' 0 ']' + exit 6 [2017-06-14 19:41:34,877] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1] [2017-06-14 19:41:34,878] (os-refresh-config) [ERROR] Aborting... Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1213, in install _run_orc(instack_env) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1003, in _run_orc _run_live_command(args, instack_env, 'os-refresh-config') File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 519, in _run_live_command raise RuntimeError('%s failed. See log for details.' % name) RuntimeError: os-refresh-config failed. See log for details. Command 'instack-install-undercloud' returned non-zero exit status 1 Thanks Srikanth From: Feng Pan [mailto:f...@redhat.com<mailto:f...@redhat.com>] Sent: Tuesday, June 13, 2017 6:41 PM To: Srikanth Vavilapalli <srikanth.vavilapa...@ericsson.com<mailto:srikanth.vavilapa...@ericsson.com>> Cc: opnfv-tech-discuss@lists.opnfv.org<mailto:opnfv-tech-discuss@lists.opnfv.org> Subject: Re: [opnfv-tech-discuss] [APEX] Apex virtual deployment install fails with error Hi Srikanth, Please use rpms from Danube branch and not master branch, most scenarios are not yet enabled on master, since we moved from Newton to Ocata release of Openstack. Thanks Feng On Tue, Jun 13, 2017 at 5:50 PM, Srikanth Vavilapalli <srikanth.vavilapa...@ericsson.com<mailto:srikanth.vavilapa...@ericsson.com>> wrote: Hi I am trying to setup APEX virtual deployment on a centos7 VM by following the instructions from http://docs.opnfv.org/en/stable-danube/submodules/apex/docs/release/installation/virtualinstall.html. At a high level: 1. Created a VM with Centos 7 on an ubuntu server 2. Installed RDO RPM: https://www.rdoproject.org/repos/rdo-release.rpm 3. Installed APEX RPMs (version 5.0-20170608) from opnfv artifacts 4. sudo opnfv-deploy -v --virtual-computes 1 -n /etc/opnfv-apex/network_settings.yaml -d /etc/opnfv-apex/os-odl-bgpvpn-noha.yaml The opnfv-deploy script ran for nearly 90mins or so after which it returned the following error log (attached full log): Configuring undercloud and discovering nodes Waiting for messages on queue '18d292d8-e4c2-4bde-a6b0-88557f636f4e' with no timeout. Waiting for messages on queue '18d292d8-e4c2-4bde-a6b0-88557f636f4e' with no timeout. Started Mistral Workflow tripleo.baremetal.v1.register_or_update. Execution ID: 2f658b2a-29ea-4eb8-a877-86f19bebffa8 Successfully registered node UUID 121aaeca-ab9a-47cb-856b-d38521758e85 Successfully registered node UUID 0023532c-1548-44a9-9bc1-fcbf8be18513 Started Mistral Workflow tripleo.baremetal.v1.provide. Execution ID: 12606ed5-5320-4571-890d-89570c7512d0 Successfully set all nodes to available. Configuring nameserver on ctlplane network Executing overcloud deployment, this could run for an extended period without output. Error: The following files were not found: /usr/share/openstack-tripleo-heat-templates/environments/neutron-opendaylight-bgpvpn.yaml ERROR: The Stack (overcloud) could not be found. [vagrant@jumphost ~]$ Can anyone plz help me with this and point me if I am missing any step in the process? Thanks Srikanth _______________________________________________ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org<mailto:opnfv-tech-discuss@lists.opnfv.org> https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
_______________________________________________ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
