** Also affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1624494
Title: [master] metadata is not working on multi-node setup Status in devstack: New Status in neutron: New Bug description: Setup: 1. One controller 2. KVM and ESX nova compute 3. Two Network nodes running q-dhcp and q-meta Nodes: Ubuntu 14.04.3 amd64 Trusty On above setup, tried to run few metadata queries we see 500 internal server error. $ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 1.1.107.1 0.0.0.0 UG 0 0 0 eth0 1.1.107.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 169.254.169.254 1.1.107.1 255.255.255.255 UGH 0 0 0 eth0 $ curl http://169.254.169.254 500 Internal Server Error Remote metadata server experienced an internal server error. $ NOTE: Moved all services to single node(controller) and metadata queries seems to work fine. vmware@cntr11:~$ neutron agent-list +--------------------------------------+----------------+-----------+-------------------+-------+----------------+------------------------+ | id | agent_type | host | availability_zone | alive | admin_state_up | binary | +--------------------------------------+----------------+-----------+-------------------+-------+----------------+------------------------+ | 0168206e-6c13-40df-a0b8-7772220ab9cd | DHCP agent | network-1 | nova | :-) | True | neutron-dhcp-agent | | 08c8bb6e-0c7f-41e3-b134-81e2d5334aea | Metadata agent | network-1 | | :-) | True | neutron-metadata-agent | | 25e9fa6d-6acc-4b8e-bb0b-f2d3ac20981d | Metadata agent | network-2 | | :-) | True | neutron-metadata-agent | | 534d349c-8830-4648-814b-611a30f59287 | DHCP agent | network-2 | nova | :-) | True | neutron-dhcp-agent | +--------------------------------------+----------------+-----------+-------------------+-------+----------------+------------------------+ vmware@cntr11:~$ vmware@cntr11:~$ nova service-list +----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+ | Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+ | 7 | nova-conductor | cntr11 | internal | enabled | up | 2016-09-16T10:00:22.000000 | - | | 9 | nova-compute | esx-ubuntu-01 | nova | enabled | up | 2016-09-16T10:00:14.000000 | - | | 10 | nova-compute | kvm-3 | nova | enabled | up | 2016-09-16T10:00:23.000000 | - | | 11 | nova-compute | kvm-2 | nova | enabled | up | 2016-09-16T10:00:19.000000 | - | | 12 | nova-compute | kvm-1 | nova | enabled | up | 2016-09-16T10:00:19.000000 | - | | 13 | nova-scheduler | cntr11 | internal | enabled | up | 2016-09-16T10:00:15.000000 | - | | 14 | nova-consoleauth | cntr11 | internal | enabled | up | 2016-09-16T10:00:20.000000 | - | +----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+ vmware@cntr11:~$ Logs: 2016-09-13 13:31:50.713 14309 DEBUG eventlet.wsgi.server [-] (14309) accepted '' server /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py:868 2016-09-13 13:31:50.715 14309 DEBUG neutron.agent.metadata.agent [-] Request: GET / HTTP/1.0^M Accept-Encoding: gzip, deflate^M Content-Length: 0^M Content-Type: text/plain^M Host: 169.254.169.254^M User-Agent: Python-httplib2/0.9.2 (gzip)^M X-Forwarded-For: 1.1.107.3^M X-Neutron-Router-Id: bbe453a5-db77-4cd9-af02-31232a222f16 __call__ /opt/stack/neutron/neutron/agent/metadata/agent.py:86 2016-09-13 13:31:50.716 14309 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 50b283bdeab945cb93872300abd5b47c exchange 'neutron' topic 'q-plugin' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448 2016-09-13 13:31:50.807 14309 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 50b283bdeab945cb93872300abd5b47c __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:296 2016-09-13 13:31:50.808 14309 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 52646ead6313405fb688b80cbb4bfc73 exchange 'neutron' topic 'q-plugin' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448 2016-09-13 13:31:50.846 14309 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 52646ead6313405fb688b80cbb4bfc73 __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:296 2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent [-] Unexpected error. 2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent Traceback (most recent call last): 2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File "/opt/stack/neutron/neutron/agent/metadata/agent.py", line 90, in __call__ 2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent return self._proxy_request(instance_id, tenant_id, req) 2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File "/opt/stack/neutron/neutron/agent/metadata/agent.py", line 193, in _proxy_request 2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent body=req.body) 2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1609, in request 2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey) 2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1351, in _request 2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent (response, content) = self._conn_request(conn, request_uri, method, body, headers) 2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1307, in _conn_request 2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent response = conn.getresponse() 2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/httplib.py", line 1129, in getresponse 2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent response = self.response_class(*args, **kwds) 2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/httplib.py", line 390, in __init__ 2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent self.fp = sock.makefile('rb', 0) 2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 298, in makefile 2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent dupped = self.dup() 2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 289, in dup 2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent newsock = type(self)(sock, set_nonblocking=False) 2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent TypeError: __init__() got an unexpected keyword argument 'set_nonblocking' 2016-09-13 13:31:50.848 14309 ERROR neutron.agent.metadata.agent 2016-09-13 13:31:50.849 14309 INFO eventlet.wsgi.server [-] 1.1.107.3,<local> - - [13/Sep/2016 13:31:50] "GET / HTTP/1.1" 500 229 0.134582 2016-09-13 13:31:51.854 13766 DEBUG oslo_messaging._drivers.amqpdriver [-] CAST unique_id: 757d1802bce344a18d2ae1df8221058f exchange 'neutron' topic 'q-reports-plugin' _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448 To manage notifications about this bug go to: https://bugs.launchpad.net/devstack/+bug/1624494/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp