** Changed in: kolla Status: Incomplete => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1563029
Title: Nova instance fail to launch: "Unexpected API Error." Status in kolla: Invalid Status in OpenStack Compute (nova): Invalid Bug description: This is a Kolla installation of openstack. The docker images build well and some services work great; e.g cinder, glance and neutron. Nova seems to have a problem that is prevent provisioning instances. The manila-api logs below says: "Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1070" 1. The GIT commit level is: lsorrillo@ptest:~/git/kolla$ git log -1 commit 90321d0497f14bda370908610cfc44296a940523 Merge: 328a7e8 a789346 Author: Jenkins <jenk...@review.openstack.org> Date: Sun Mar 27 23:42:47 2016 +0000 Merge "Fix gate to use world writeable docker socket" lsorrillo@ptest:~/git/kolla$ RPM version: lsorrillo@ptest:~$ lsorrillo@ptest:~$ docker exec -ti nova_api bash bash-4.2$ bash-4.2$ bash-4.2$ bash-4.2$ rpm -qa | grep nova python-novaclient-3.3.1-0.20160303082753.d63800d.el7.centos.noarch openstack-nova-common-13.0.0.0b4-0.20160304162843.c5a45a2.el7.centos.noarch python-nova-13.0.0.0b4-0.20160304162843.c5a45a2.el7.centos.noarch openstack-nova-api-13.0.0.0b4-0.20160304162843.c5a45a2.el7.centos.noarch bash-4.2$ 2. The following error is seen in the nova-api.log: 2016-03-28 19:53:40.086 29 DEBUG oslo.messaging._drivers.impl_rabbit [req-88ffb6f7-1b1f-4332-b066-5b0b41827988 5bd4af0c54a14a789661d868a9fb1d84 9e4b40d958e741599adc497b0a9d955a - - -] Connecting to AMQP server on 192.168.0.121:5672 __init__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:535 2016-03-28 19:53:40.132 29 DEBUG oslo.messaging._drivers.impl_rabbit [req-88ffb6f7-1b1f-4332-b066-5b0b41827988 5bd4af0c54a14a789661d868a9fb1d84 9e4b40d958e741599adc497b0a9d955a - - -] Connected to AMQP server on 192.168.0.121:5672 via [amqp] client __init__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:562 2016-03-28 19:53:40.148 29 DEBUG oslo.messaging._drivers.pool [req-88ffb6f7-1b1f-4332-b066-5b0b41827988 5bd4af0c54a14a789661d868a9fb1d84 9e4b40d958e741599adc497b0a9d955a - - -] Pool creating new connection create /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/pool.py:109 2016-03-28 19:53:40.151 29 DEBUG oslo.messaging._drivers.impl_rabbit [req-88ffb6f7-1b1f-4332-b066-5b0b41827988 5bd4af0c54a14a789661d868a9fb1d84 9e4b40d958e741599adc497b0a9d955a - - -] Connecting to AMQP server on 192.168.0.121:5672 __init__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:535 2016-03-28 19:53:40.205 29 DEBUG oslo.messaging._drivers.impl_rabbit [req-88ffb6f7-1b1f-4332-b066-5b0b41827988 5bd4af0c54a14a789661d868a9fb1d84 9e4b40d958e741599adc497b0a9d955a - - -] Connected to AMQP server on 192.168.0.121:5672 via [amqp] client __init__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py:562 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions [req-88ffb6f7-1b1f-4332-b066-5b0b41827988 5bd4af0c54a14a789661d868a9fb1d84 9e4b40d958e741599adc497b0a9d955a - - -] Unexpected exception in API method 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 630, in create 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions **create_kwargs) 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 154, in inner 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions rv = f(*args, **kwargs) 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1556, in create 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions check_server_group_quota=check_server_group_quota) 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1139, in _create_instance 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions reservation_id, max_count) 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 834, in _validate_and_build_base_options 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions requested_networks, max_count) 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 448, in _check_requested_networks 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions max_count) 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 49, in wrapped 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions return func(self, context, *args, **kwargs) 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 399, in validate_networks 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions requested_networks) 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/network/rpcapi.py", line 212, in validate_networks 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions return self.client.call(ctxt, 'validate_networks', networks=networks) 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 413, in call 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions return self.prepare().call(ctxt, method, **kwargs) 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions retry=self.retry) 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions timeout=timeout, retry=retry) 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions retry=retry) 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions result = self._waiter.wait(msg_id, timeout) 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions message = self.waiters.get(msg_id, timeout=timeout) 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions 'to message ID %s' % msg_id) 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions MessagingTimeout: Timed out waiting for a reply to message ID 73fdd3b38da34bf7a19b08a086fb26f8 2016-03-28 19:54:40.216 29 ERROR nova.api.openstack.extensions 2016-03-28 19:54:40.229 29 INFO nova.api.openstack.wsgi [req-88ffb6f7-1b1f-4332-b066-5b0b41827988 5bd4af0c54a14a789661d868a9fb1d84 9e4b40d958e741599adc497b0a9d955a - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'oslo_messaging.exceptions.MessagingTimeout'> 2016-03-28 19:54:40.231 29 DEBUG nova.api.openstack.wsgi [req-88ffb6f7-1b1f-4332-b066-5b0b41827988 5bd4af0c54a14a789661d868a9fb1d84 9e4b40d958e741599adc497b0a9d955a - - -] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'oslo_messaging.exceptions.MessagingTimeout'> __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1070 2016-03-28 19:54:40.237 29 INFO nova.osapi_compute.wsgi.server [req-88ffb6f7-1b1f-4332-b066-5b0b41827988 5bd4af0c54a14a789661d868a9fb1d84 9e4b40d958e741599adc497b0a9d955a - - -] Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 483, in handle_one_response write(b''.join(towrite)) File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 426, in write _writelines(towrite) File "/usr/lib64/python2.7/socket.py", line 334, in writelines self.flush() File "/usr/lib64/python2.7/socket.py", line 303, in flush self._sock.sendall(view[write_offset:write_offset+buffer_size]) File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 377, in sendall tail = self.send(data, flags) File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 359, in send total_sent += fd.send(data[total_sent:], flags) error: [Errno 104] Connection reset by peer 2016-03-28 19:54:40.238 29 INFO nova.osapi_compute.wsgi.server [req- 88ffb6f7-1b1f-4332-b066-5b0b41827988 5bd4af0c54a14a789661d868a9fb1d84 9e4b40d958e741599adc497b0a9d955a - - -] 10.10.10.254,192.168.0.121 "POST /v2/9e4b40d958e741599adc497b0a9d955a/servers HTTP/1.1" status: 500 len: 0 time: 60.8730421 3. git clone https://git.openstack.org/openstack/kolla git checkout -b ltshome and make the following changes: lsorrillo@ptest:~/git/kolla$ git status # On branch ltshome # Changes not staged for commit: # (use "git add <file>..." to update what will be committed) # (use "git checkout -- <file>..." to discard changes in working directory) # # modified: ansible/group_vars/all.yml # modified: etc/kolla/globals.yml # modified: etc/kolla/passwords.yml # # Untracked files: # (use "git add <file>..." to include in what will be committed) # # etc/kolla/passwords.yml.orig no changes added to commit (use "git add" and/or "git commit -a") lsorrillo@ptest:~/git/kolla$ Here are diffs of the modified files above: lsorrillo@ptest:~/git/kolla$ git diff ansible/group_vars/all.yml diff --git a/ansible/group_vars/all.yml b/ansible/group_vars/all.yml index a0ab95a..6ec956e 100644 --- a/ansible/group_vars/all.yml +++ b/ansible/group_vars/all.yml @@ -176,7 +176,7 @@ enable_rabbitmq: "yes" # Additional optional OpenStack services are specified here enable_ceph: "no" enable_ceph_rgw: "no" -enable_cinder: "no" +enable_cinder: "yes" enable_heat: "yes" enable_horizon: "yes" enable_swift: "no" @@ -184,9 +184,9 @@ enable_murano: "no" enable_ironic: "no" enable_magnum: "no" enable_mistral: "no" -enable_central_logging: "no" +enable_central_logging: "yes" enable_mongodb: "no" -enable_manila: "no" +enable_manila: "yes" ironic_keystone_user: "ironic" neutron_keystone_user: "neutron" lsorrillo@ptest:~/git/kolla$ lsorrillo@ptest:~/git/kolla$ git diff etc/kolla/globals.yml diff --git a/etc/kolla/globals.yml b/etc/kolla/globals.yml index ec26126..2afea72 100644 --- a/etc/kolla/globals.yml +++ b/etc/kolla/globals.yml @@ -55,14 +55,14 @@ kolla_internal_vip_address: "10.10.10.254" # This interface is what all your api services will be bound to by default. # Additionally, all vxlan/tunnel and storage network traffic will go over this # interface by default. This interface must contain an IPv4 address. -network_interface: "eth0" +network_interface: "enp2s0f1" # These can be adjusted for even more customization. The default is the same as # the 'network_interface'. These interfaces must container an IPv4 address. -#kolla_external_vip_interface: "{{ network_interface }}" -#api_interface: "{{ network_interface }}" -#storage_interface: "{{ network_interface }}" -#tunnel_interface: "{{ network_interface }}" +kolla_external_vip_interface: "{{ network_interface }}" +api_interface: "{{ network_interface }}" +storage_interface: "{{ network_interface }}" +tunnel_interface: "{{ network_interface }}" # This is the raw interface given to neutron as its external network port. Even # though an IP address can exist on this interface, it will be unusable in most @@ -80,7 +80,7 @@ neutron_external_interface: "eth1" # To provide encryption and authentication on the kolla_external_vip_interface, # TLS can be enabled. When TLS is enabled, certificates must be provided to # allow clients to perform authentication. -#kolla_enable_tls_external: "no" +kolla_enable_tls_external: "no" #kolla_external_fqdn_cert: "{{ node_config_directory }}/certificates/haproxy.pem" @@ -89,13 +89,13 @@ neutron_external_interface: "eth1" #################### # Use these options to set the various log levels across all OpenStack projects # Valid options are [ True, False ] -#openstack_logging_debug: "False" +openstack_logging_debug: "True" # Valid options are [ novnc, spice ] #nova_console: "novnc" # OpenStack services can be enabled or disabled with these options -#enable_cinder: "no" +enable_cinder: "yes" #enable_heat: "yes" #enable_magnum: "no" #enable_swift: "no" @@ -118,7 +118,7 @@ neutron_external_interface: "eth1" ####################################### # Manila - Shared File Systems Options ####################################### -#manila_enable_dhss: "yes" +manila_enable_dhss: "yes" ####################################### lsorrillo@ptest:~/git/kolla$ git diff etc/kolla/passwords.yml xxx xxx To manage notifications about this bug go to: https://bugs.launchpad.net/kolla/+bug/1563029/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp