Hello everybody! I'm facing a quite confusing problem with my kilo neutron-metadata-agent on network node.
What's happening is that randomically, my metadata-agent stop working properly and causes instances to not be launched. The solution is to restart neutron-metadata-agent service, after a period of time, the error happens again. What the logs says: Accept-Encoding: gzip, deflate^M Content-Length: 0^M Content-Type: text/plain^M Host: 169.254.169.254^M User-Agent: Python-httplib2/0.8 (gzip)^M X-Forwarded-For: 10.4.3.211^M X-Neutron-Router-Id: 55ebf771-8a36-4185-90df-21413638c1fc __call__ /usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py:107 2016-03-23 07:00:45.571 31470 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "('neutron.agent.metadata.agent.MetadataProxyHandler._get_router_networks', '55ebf771-8a36-4185-90df-21413638c1fc')" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:391 2016-03-23 07:00:45.572 31470 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "('neutron.agent.metadata.agent.MetadataProxyHandler._get_router_networks', '55ebf771-8a36-4185-90df-21413638c1fc')" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:404 2016-03-23 07:00:45.573 31470 DEBUG neutronclient.client [-] *REQ: curl -i http://controller:35357/tokens <http://controller:35357/tokens> -X POST -H "User-Agent: python-neutronclient" -d '{"auth": {"tenantName": null, "passwordCredentials": {"username": null, "password": null}}}'* http_log_req /usr/lib/python2.7/dist-packages/neutronclient/common/utils.py:130 2016-03-23 07:00:45.579 31470 DEBUG neutronclient.client [-] RESP:404 {'content-length': '93', 'vary': 'X-Auth-Token', 'server': 'Apache/2.4.7 (Ubuntu)', 'date': 'Wed, 23 Mar 2016 10:00:45 GMT', 'content-type': 'application/json', 'x-distribution': 'Ubuntu'} {"error": {"message": "The resource could not be found.", "code": 404, "title": "Not Found"}} http_log_resp /usr/lib/python2.7/dist-packages/neutronclient/common/utils.py:139 2016-03-23 07:00:45.580 31470 ERROR neutron.agent.metadata.agent [-] Unexpected error. 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent Traceback (most recent call last): 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 109, in __call__ 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent instance_id, tenant_id = self._get_instance_and_tenant_id(req) 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 204, in _get_instance_and_tenant_id 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent ports = self._get_ports(remote_address, network_id, router_id) 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 192, in _get_ports 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent networks = self._get_router_networks(router_id) 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 101, in __call__ 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent return self._get_from_cache(target_self, *args, **kwargs) 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 79, in _get_from_cache 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent item = self.func(target_self, *args, **kwargs) 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 153, in _get_router_networks 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent internal_ports = self._get_ports_from_server(router_id=router_id) 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 135, in _get_ports_from_server 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent return self._get_ports_using_client(filters) 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 177, in _get_ports_using_client 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent ports = client.list_ports(**filters) 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 99, in with_params 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent ret = self.function(instance, *args, **kwargs) 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 497, in list_ports 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent **_params) 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 304, in list 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent for r in self._pagination(collection, path, **params): 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 317, in _pagination 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent res = self.get(path, params=params) 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 290, in get 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent headers=headers, params=params) 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 267, in retry_request 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent headers=headers, params=params) 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 197, in do_request 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent content_type=self.content_type()) 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 158, in do_request 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent self.authenticate_and_fetch_endpoint_url() 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 123, in authenticate_and_fetch_endpoint_url 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent self.authenticate() 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 237, in authenticate 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent self._authenticate_keystone() 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 218, in _authenticate_keystone 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent raise exceptions.Unauthorized(message=resp_body) 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent Unauthorized: {"error": {"message": "The resource could not be found.", "code": 404, "title": "Not Found"}} 2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent 2016-03-23 07:00:45.584 31470 INFO eventlet.wsgi.server [-] 10.4.3.211,<local> - - [23/Mar/2016 07:00:45] "GET /latest/meta-data/instance-type HTTP/1.1" 500 229 0.015149 2016-03-23 07:00:45.590 31469 INFO eventlet.wsgi.server [-] (31469) accepted '' 2016-03-23 07:00:45.592 31469 DEBUG neutron.agent.metadata.agent [-] Request: GET /latest/meta-data/ami-id HTTP/1.0^M I don't understand why python-neutronclient gets a token with null values since other services work normally and only metadata-agent throws this error. Thanks!!!
_______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators