Thanks Simon, That seems to be it, may find more as I test it but first pass adding the cache_url got by 64 instance test case to laumch without crashing the metadata service.
-Jon On Wed, Oct 22, 2014 at 3:25 AM, Simon Pasquier <spasqu...@mirantis.com> wrote: > Hello Jonathan, > Have you seen this discussion on the openstack-dev [1] that discusses the > bug 1361357 [2]? > I have no idea if it is related to your issue but FWIW a fix backport in the > latest Icehouse release introduced performance regression for the metadata > agent. > BR > Simon > > [1] > http://lists.openstack.org/pipermail/openstack-dev/2014-October/048916.html > [2] https://bugs.launchpad.net/cloud-archive/+bug/1361357 > > On Wed, Oct 22, 2014 at 2:33 AM, Jonathan Proulx <j...@jonproulx.com> wrote: >> >> Ah there's the log many instances of: >> >> 2014-10-21 19:50:15.527 12931 INFO neutron.wsgi [-] 10.10.167.98 - - >> [21/Oct/2014 19:50:15] "GET /openstack/2012-08-10 HTTP/1.1" 500 343 >> 120.411705 >> >> 2014-10-21 19:50:15.528 12931 ERROR >> neutron.agent.metadata.namespace_proxy [-] Unexpected error. >> 2014-10-21 19:50:15.528 12931 TRACE >> neutron.agent.metadata.namespace_proxy Traceback (most recent call >> last): >> 2014-10-21 19:50:15.528 12931 TRACE >> neutron.agent.metadata.namespace_proxy File >> >> "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/namespace_proxy.py", >> line 74, in __cal >> l__ >> 2014-10-21 19:50:15.528 12931 TRACE >> neutron.agent.metadata.namespace_proxy req.body) >> 2014-10-21 19:50:15.528 12931 TRACE >> neutron.agent.metadata.namespace_proxy File >> >> "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/namespace_proxy.py", >> line 105, in _pro >> xy_request >> 2014-10-21 19:50:15.528 12931 TRACE >> neutron.agent.metadata.namespace_proxy >> connection_type=UnixDomainHTTPConnection) >> 2014-10-21 19:50:15.528 12931 TRACE >> neutron.agent.metadata.namespace_proxy File >> "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1569, in >> request >> 2014-10-21 19:50:15.528 12931 TRACE >> neutron.agent.metadata.namespace_proxy (response, content) = >> self._request(conn, authority, uri, request_uri, method, body, >> headers, redi >> rections, cachekey) >> 2014-10-21 19:50:15.528 12931 TRACE >> neutron.agent.metadata.namespace_proxy File >> "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1316, in >> _request >> 2014-10-21 19:50:15.528 12931 TRACE >> neutron.agent.metadata.namespace_proxy (response, content) = >> self._conn_request(conn, request_uri, method, body, headers) >> 2014-10-21 19:50:15.528 12931 TRACE >> neutron.agent.metadata.namespace_proxy File >> "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1285, in >> _conn_request >> 2014-10-21 19:50:15.528 12931 TRACE >> neutron.agent.metadata.namespace_proxy response = >> conn.getresponse() >> 2014-10-21 19:50:15.528 12931 TRACE >> neutron.agent.metadata.namespace_proxy File >> "/usr/lib/python2.7/httplib.py", line 1045, in getresponse >> 2014-10-21 19:50:15.528 12931 TRACE >> neutron.agent.metadata.namespace_proxy response.begin() >> 2014-10-21 19:50:15.528 12931 TRACE >> neutron.agent.metadata.namespace_proxy File >> "/usr/lib/python2.7/httplib.py", line 409, in begin >> 2014-10-21 19:50:15.528 12931 TRACE >> neutron.agent.metadata.namespace_proxy version, status, reason = >> self._read_status() >> 2014-10-21 19:50:15.528 12931 TRACE >> neutron.agent.metadata.namespace_proxy File >> "/usr/lib/python2.7/httplib.py", line 373, in _read_status >> 2014-10-21 19:50:15.528 12931 TRACE >> neutron.agent.metadata.namespace_proxy raise BadStatusLine(line) >> 2014-10-21 19:50:15.528 12931 TRACE >> neutron.agent.metadata.namespace_proxy BadStatusLine: '' >> 2014-10-21 19:50:15.528 12931 TRACE neutron.agent.metadata.namespace_proxy >> >> On Tue, Oct 21, 2014 at 8:17 PM, Jonathan Proulx <j...@jonproulx.com> >> wrote: >> > running Icehouse + Neutron ML2/OVS and network names spaces. >> > >> > Was running well unitl recently, most recent change was switching to >> > Ceph RBD for ephemeral storage on the hypervisors (and glance). I >> > suspect this of being relevant because it makes the instances launch >> > much more quickly. >> > >> > I haven't classified the breaking point but launching 64 instances >> > deterministically breaks the metadata agent. >> > >> > The service seems to be running on the controller, but is not >> > listening in the network namespace. It seems to require restarting >> > both the dhcp-agent and the metadata agent to get it to go again. >> > >> > Even in debug mode I get no errors in the logs. >> > >> > Anyone seen this? >> > >> > -Jon >> >> _______________________________________________ >> OpenStack-operators mailing list >> OpenStack-operators@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > > _______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators