[Yahoo-eng-team] [Bug 1362766] Re: ConnectionFailed: Connection to neutron failed: 'HTTPSConnectionPool' object has no attribute 'insecure'
** Also affects: keystonemiddleware Importance: Undecided Status: New ** Summary changed: - ConnectionFailed: Connection to neutron failed: 'HTTPSConnectionPool' object has no attribute 'insecure' + ConnectionFailed: Connection to XX failed: 'HTTPSConnectionPool' object has no attribute 'insecure' -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1362766 Title: ConnectionFailed: Connection to XX failed: 'HTTPSConnectionPool' object has no attribute 'insecure' Status in OpenStack Identity (Keystone) Middleware: New Status in OpenStack Compute (Nova): New Status in Python client library for Glance: Fix Released Status in Python client library for Neutron: Incomplete Bug description: While compute manager was trying to authenticate with neutronclient, we see the following: 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager Traceback (most recent call last): 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager File /usr/lib/python2.7/site-packages/powervc_nova/compute/manager.py, line 672, in _populate_admin_context 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager nclient.authenticate() 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager File /usr/lib/python2.7/site-packages/neutronclient/client.py, line 231, in authenticate 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager self._authenticate_keystone() 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager File /usr/lib/python2.7/site-packages/neutronclient/client.py, line 209, in _authenticate_keystone 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager allow_redirects=True) 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager File /usr/lib/python2.7/site-packages/neutronclient/client.py, line 113, in _cs_request 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager raise exceptions.ConnectionFailed(reason=e) 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager ConnectionFailed: Connection to neutron failed: 'HTTPSConnectionPool' object has no attribute 'insecure' Setting a pdb breakpoint and stepping into the code, I see that the requests library is getting a connection object from a pool. The interesting thing is that the connection object is actually from glanceclient.common.https.HTTPSConnectionPool. It seems odd to me that neutronclient is using a connection object from glanceclient pool, but I do not know this requests code. Here is the stack just before failure: /usr/lib/python2.7/site-packages/neutronclient/client.py(234)authenticate() - self._authenticate_keystone() /usr/lib/python2.7/site-packages/neutronclient/client.py(212)_authenticate_keystone() - allow_redirects=True) /usr/lib/python2.7/site-packages/neutronclient/client.py(106)_cs_request() - resp, body = self.request(*args, **kargs) /usr/lib/python2.7/site-packages/neutronclient/client.py(151)request() - **kwargs) /usr/lib/python2.7/site-packages/requests/api.py(44)request() - return session.request(method=method, url=url, **kwargs) /usr/lib/python2.7/site-packages/requests/sessions.py(335)request() - resp = self.send(prep, **send_kwargs) /usr/lib/python2.7/site-packages/requests/sessions.py(438)send() - r = adapter.send(request, **kwargs) /usr/lib/python2.7/site-packages/requests/adapters.py(292)send() - timeout=timeout /usr/lib/python2.7/site-packages/urllib3/connectionpool.py(454)urlopen() - conn = self._get_conn(timeout=pool_timeout) /usr/lib/python2.7/site-packages/urllib3/connectionpool.py(272)_get_conn() - return conn or self._new_conn() /usr/lib/python2.7/site-packages/glanceclient/common/https.py(100)_new_conn() - return VerifiedHTTPSConnection(host=self.host, The code about to run there is this: class HTTPSConnectionPool(connectionpool.HTTPSConnectionPool): HTTPSConnectionPool will be instantiated when a new connection is requested to the HTTPSAdapter.This implementation overwrites the _new_conn method and returns an instances of glanceclient's VerifiedHTTPSConnection which handles no compression. ssl_compression is hard-coded to False because this will be used just when the user sets --no-ssl-compression. scheme = 'https' def _new_conn(self): self.num_connections += 1 return VerifiedHTTPSConnection(host=self.host, port=self.port, key_file=self.key_file, cert_file=self.cert_file, cacert=self.ca_certs, insecure=self.insecure,
[Yahoo-eng-team] [Bug 1362766] Re: ConnectionFailed: Connection to neutron failed: 'HTTPSConnectionPool' object has no attribute 'insecure'
I'm getting this in nova-api too, we don't use neutron. I get it when I do a nova list or nova show. Restarting nova-api fixed it for a while but then it comes back again. == /var/log/nova/nova-api.log == 2015-03-27 13:56:06.649 10962 WARNING keystonemiddleware.auth_token [-] Retrying on HTTP connection exception: 'HTTPSConnectionPool' object has no attribute 'insecure' 2015-03-27 13:56:07.153 10962 WARNING keystonemiddleware.auth_token [-] Retrying on HTTP connection exception: 'HTTPSConnectionPool' object has no attribute 'insecure' 2015-03-27 13:56:08.156 10962 WARNING keystonemiddleware.auth_token [-] Retrying on HTTP connection exception: 'HTTPSConnectionPool' object has no attribute 'insecure' 2015-03-27 13:56:10.159 10962 ERROR keystonemiddleware.auth_token [-] HTTP connection exception: 'HTTPSConnectionPool' object has no attribute 'insecure' 2015-03-27 13:56:10.161 10962 WARNING keystonemiddleware.auth_token [-] Authorization failed for token 2015-03-27 13:56:10.162 10962 INFO nova.osapi_compute.wsgi.server [-] 128.250.116.173,172.26.9.144 GET /v1.1/0bdf024c921848c4b74d9e69af9edf08/servers/detail HTTP/1.1 status: 401 len: 282 time: 3.5197592 ** Also affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1362766 Title: ConnectionFailed: Connection to neutron failed: 'HTTPSConnectionPool' object has no attribute 'insecure' Status in OpenStack Compute (Nova): New Status in Python client library for Glance: Fix Released Status in Python client library for Neutron: Incomplete Bug description: While compute manager was trying to authenticate with neutronclient, we see the following: 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager Traceback (most recent call last): 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager File /usr/lib/python2.7/site-packages/powervc_nova/compute/manager.py, line 672, in _populate_admin_context 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager nclient.authenticate() 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager File /usr/lib/python2.7/site-packages/neutronclient/client.py, line 231, in authenticate 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager self._authenticate_keystone() 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager File /usr/lib/python2.7/site-packages/neutronclient/client.py, line 209, in _authenticate_keystone 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager allow_redirects=True) 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager File /usr/lib/python2.7/site-packages/neutronclient/client.py, line 113, in _cs_request 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager raise exceptions.ConnectionFailed(reason=e) 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager ConnectionFailed: Connection to neutron failed: 'HTTPSConnectionPool' object has no attribute 'insecure' Setting a pdb breakpoint and stepping into the code, I see that the requests library is getting a connection object from a pool. The interesting thing is that the connection object is actually from glanceclient.common.https.HTTPSConnectionPool. It seems odd to me that neutronclient is using a connection object from glanceclient pool, but I do not know this requests code. Here is the stack just before failure: /usr/lib/python2.7/site-packages/neutronclient/client.py(234)authenticate() - self._authenticate_keystone() /usr/lib/python2.7/site-packages/neutronclient/client.py(212)_authenticate_keystone() - allow_redirects=True) /usr/lib/python2.7/site-packages/neutronclient/client.py(106)_cs_request() - resp, body = self.request(*args, **kargs) /usr/lib/python2.7/site-packages/neutronclient/client.py(151)request() - **kwargs) /usr/lib/python2.7/site-packages/requests/api.py(44)request() - return session.request(method=method, url=url, **kwargs) /usr/lib/python2.7/site-packages/requests/sessions.py(335)request() - resp = self.send(prep, **send_kwargs) /usr/lib/python2.7/site-packages/requests/sessions.py(438)send() - r = adapter.send(request, **kwargs) /usr/lib/python2.7/site-packages/requests/adapters.py(292)send() - timeout=timeout /usr/lib/python2.7/site-packages/urllib3/connectionpool.py(454)urlopen() - conn = self._get_conn(timeout=pool_timeout) /usr/lib/python2.7/site-packages/urllib3/connectionpool.py(272)_get_conn() - return conn or self._new_conn() /usr/lib/python2.7/site-packages/glanceclient/common/https.py(100)_new_conn() - return VerifiedHTTPSConnection(host=self.host, The code about to run there is this: class HTTPSConnectionPool(connectionpool.HTTPSConnectionPool):
[Yahoo-eng-team] [Bug 1362766] Re: ConnectionFailed: Connection to neutron failed: 'HTTPSConnectionPool' object has no attribute 'insecure'
https://review.openstack.org/#/c/110574/9 ** Also affects: glance Importance: Undecided Status: New ** Also affects: python-glanceclient Importance: Undecided Status: New ** No longer affects: glance ** Changed in: python-glanceclient Status: New = In Progress ** Changed in: python-glanceclient Assignee: (unassigned) = Flavio Percoco (flaper87) ** Changed in: python-glanceclient Importance: Undecided = Medium -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1362766 Title: ConnectionFailed: Connection to neutron failed: 'HTTPSConnectionPool' object has no attribute 'insecure' Status in Python client library for Glance: In Progress Status in Python client library for Neutron: Incomplete Bug description: While compute manager was trying to authenticate with neutronclient, we see the following: 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager Traceback (most recent call last): 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager File /usr/lib/python2.7/site-packages/powervc_nova/compute/manager.py, line 672, in _populate_admin_context 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager nclient.authenticate() 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager File /usr/lib/python2.7/site-packages/neutronclient/client.py, line 231, in authenticate 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager self._authenticate_keystone() 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager File /usr/lib/python2.7/site-packages/neutronclient/client.py, line 209, in _authenticate_keystone 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager allow_redirects=True) 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager File /usr/lib/python2.7/site-packages/neutronclient/client.py, line 113, in _cs_request 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager raise exceptions.ConnectionFailed(reason=e) 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager ConnectionFailed: Connection to neutron failed: 'HTTPSConnectionPool' object has no attribute 'insecure' Setting a pdb breakpoint and stepping into the code, I see that the requests library is getting a connection object from a pool. The interesting thing is that the connection object is actually from glanceclient.common.https.HTTPSConnectionPool. It seems odd to me that neutronclient is using a connection object from glanceclient pool, but I do not know this requests code. Here is the stack just before failure: /usr/lib/python2.7/site-packages/neutronclient/client.py(234)authenticate() - self._authenticate_keystone() /usr/lib/python2.7/site-packages/neutronclient/client.py(212)_authenticate_keystone() - allow_redirects=True) /usr/lib/python2.7/site-packages/neutronclient/client.py(106)_cs_request() - resp, body = self.request(*args, **kargs) /usr/lib/python2.7/site-packages/neutronclient/client.py(151)request() - **kwargs) /usr/lib/python2.7/site-packages/requests/api.py(44)request() - return session.request(method=method, url=url, **kwargs) /usr/lib/python2.7/site-packages/requests/sessions.py(335)request() - resp = self.send(prep, **send_kwargs) /usr/lib/python2.7/site-packages/requests/sessions.py(438)send() - r = adapter.send(request, **kwargs) /usr/lib/python2.7/site-packages/requests/adapters.py(292)send() - timeout=timeout /usr/lib/python2.7/site-packages/urllib3/connectionpool.py(454)urlopen() - conn = self._get_conn(timeout=pool_timeout) /usr/lib/python2.7/site-packages/urllib3/connectionpool.py(272)_get_conn() - return conn or self._new_conn() /usr/lib/python2.7/site-packages/glanceclient/common/https.py(100)_new_conn() - return VerifiedHTTPSConnection(host=self.host, The code about to run there is this: class HTTPSConnectionPool(connectionpool.HTTPSConnectionPool): HTTPSConnectionPool will be instantiated when a new connection is requested to the HTTPSAdapter.This implementation overwrites the _new_conn method and returns an instances of glanceclient's VerifiedHTTPSConnection which handles no compression. ssl_compression is hard-coded to False because this will be used just when the user sets --no-ssl-compression. scheme = 'https' def _new_conn(self): self.num_connections += 1 return VerifiedHTTPSConnection(host=self.host, port=self.port, key_file=self.key_file, cert_file=self.cert_file, cacert=self.ca_certs, insecure=self.insecure,
[Yahoo-eng-team] [Bug 1362766] Re: ConnectionFailed: Connection to neutron failed: 'HTTPSConnectionPool' object has no attribute 'insecure'
Not sure how neutron client depends on glance client. There is no such dependency in upstream python-neutronclient. Could you please clarify? ** No longer affects: neutron ** Changed in: python-neutronclient Status: New = Incomplete -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1362766 Title: ConnectionFailed: Connection to neutron failed: 'HTTPSConnectionPool' object has no attribute 'insecure' Status in Python client library for Neutron: Incomplete Bug description: While compute manager was trying to authenticate with neutronclient, we see the following: 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager Traceback (most recent call last): 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager File /usr/lib/python2.7/site-packages/powervc_nova/compute/manager.py, line 672, in _populate_admin_context 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager nclient.authenticate() 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager File /usr/lib/python2.7/site-packages/neutronclient/client.py, line 231, in authenticate 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager self._authenticate_keystone() 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager File /usr/lib/python2.7/site-packages/neutronclient/client.py, line 209, in _authenticate_keystone 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager allow_redirects=True) 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager File /usr/lib/python2.7/site-packages/neutronclient/client.py, line 113, in _cs_request 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager raise exceptions.ConnectionFailed(reason=e) 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager ConnectionFailed: Connection to neutron failed: 'HTTPSConnectionPool' object has no attribute 'insecure' Setting a pdb breakpoint and stepping into the code, I see that the requests library is getting a connection object from a pool. The interesting thing is that the connection object is actually from glanceclient.common.https.HTTPSConnectionPool. It seems odd to me that neutronclient is using a connection object from glanceclient pool, but I do not know this requests code. Here is the stack just before failure: /usr/lib/python2.7/site-packages/neutronclient/client.py(234)authenticate() - self._authenticate_keystone() /usr/lib/python2.7/site-packages/neutronclient/client.py(212)_authenticate_keystone() - allow_redirects=True) /usr/lib/python2.7/site-packages/neutronclient/client.py(106)_cs_request() - resp, body = self.request(*args, **kargs) /usr/lib/python2.7/site-packages/neutronclient/client.py(151)request() - **kwargs) /usr/lib/python2.7/site-packages/requests/api.py(44)request() - return session.request(method=method, url=url, **kwargs) /usr/lib/python2.7/site-packages/requests/sessions.py(335)request() - resp = self.send(prep, **send_kwargs) /usr/lib/python2.7/site-packages/requests/sessions.py(438)send() - r = adapter.send(request, **kwargs) /usr/lib/python2.7/site-packages/requests/adapters.py(292)send() - timeout=timeout /usr/lib/python2.7/site-packages/urllib3/connectionpool.py(454)urlopen() - conn = self._get_conn(timeout=pool_timeout) /usr/lib/python2.7/site-packages/urllib3/connectionpool.py(272)_get_conn() - return conn or self._new_conn() /usr/lib/python2.7/site-packages/glanceclient/common/https.py(100)_new_conn() - return VerifiedHTTPSConnection(host=self.host, The code about to run there is this: class HTTPSConnectionPool(connectionpool.HTTPSConnectionPool): HTTPSConnectionPool will be instantiated when a new connection is requested to the HTTPSAdapter.This implementation overwrites the _new_conn method and returns an instances of glanceclient's VerifiedHTTPSConnection which handles no compression. ssl_compression is hard-coded to False because this will be used just when the user sets --no-ssl-compression. scheme = 'https' def _new_conn(self): self.num_connections += 1 return VerifiedHTTPSConnection(host=self.host, port=self.port, key_file=self.key_file, cert_file=self.cert_file, cacert=self.ca_certs, insecure=self.insecure, ssl_compression=False) Note the self.insecure, which does not exist here. I see the following fairly recent change in the glanceclient.common.https code that added this: https://github.com/openstack/python-
[Yahoo-eng-team] [Bug 1362766] Re: ConnectionFailed: Connection to neutron failed: 'HTTPSConnectionPool' object has no attribute 'insecure'
** Also affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1362766 Title: ConnectionFailed: Connection to neutron failed: 'HTTPSConnectionPool' object has no attribute 'insecure' Status in OpenStack Neutron (virtual network service): New Status in Python client library for Neutron: New Bug description: While compute manager was trying to authenticate with neutronclient, we see the following: 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager Traceback (most recent call last): 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager File /usr/lib/python2.7/site-packages/powervc_nova/compute/manager.py, line 672, in _populate_admin_context 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager nclient.authenticate() 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager File /usr/lib/python2.7/site-packages/neutronclient/client.py, line 231, in authenticate 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager self._authenticate_keystone() 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager File /usr/lib/python2.7/site-packages/neutronclient/client.py, line 209, in _authenticate_keystone 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager allow_redirects=True) 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager File /usr/lib/python2.7/site-packages/neutronclient/client.py, line 113, in _cs_request 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager raise exceptions.ConnectionFailed(reason=e) 2014-08-28 05:03:33.052 29982 TRACE powervc_nova.compute.manager ConnectionFailed: Connection to neutron failed: 'HTTPSConnectionPool' object has no attribute 'insecure' Setting a pdb breakpoint and stepping into the code, I see that the requests library is getting a connection object from a pool. The interesting thing is that the connection object is actually from glanceclient.common.https.HTTPSConnectionPool. It seems odd to me that neutronclient is using a connection object from glanceclient pool, but I do not know this requests code. Here is the stack just before failure: /usr/lib/python2.7/site-packages/neutronclient/client.py(234)authenticate() - self._authenticate_keystone() /usr/lib/python2.7/site-packages/neutronclient/client.py(212)_authenticate_keystone() - allow_redirects=True) /usr/lib/python2.7/site-packages/neutronclient/client.py(106)_cs_request() - resp, body = self.request(*args, **kargs) /usr/lib/python2.7/site-packages/neutronclient/client.py(151)request() - **kwargs) /usr/lib/python2.7/site-packages/requests/api.py(44)request() - return session.request(method=method, url=url, **kwargs) /usr/lib/python2.7/site-packages/requests/sessions.py(335)request() - resp = self.send(prep, **send_kwargs) /usr/lib/python2.7/site-packages/requests/sessions.py(438)send() - r = adapter.send(request, **kwargs) /usr/lib/python2.7/site-packages/requests/adapters.py(292)send() - timeout=timeout /usr/lib/python2.7/site-packages/urllib3/connectionpool.py(454)urlopen() - conn = self._get_conn(timeout=pool_timeout) /usr/lib/python2.7/site-packages/urllib3/connectionpool.py(272)_get_conn() - return conn or self._new_conn() /usr/lib/python2.7/site-packages/glanceclient/common/https.py(100)_new_conn() - return VerifiedHTTPSConnection(host=self.host, The code about to run there is this: class HTTPSConnectionPool(connectionpool.HTTPSConnectionPool): HTTPSConnectionPool will be instantiated when a new connection is requested to the HTTPSAdapter.This implementation overwrites the _new_conn method and returns an instances of glanceclient's VerifiedHTTPSConnection which handles no compression. ssl_compression is hard-coded to False because this will be used just when the user sets --no-ssl-compression. scheme = 'https' def _new_conn(self): self.num_connections += 1 return VerifiedHTTPSConnection(host=self.host, port=self.port, key_file=self.key_file, cert_file=self.cert_file, cacert=self.ca_certs, insecure=self.insecure, ssl_compression=False) Note the self.insecure, which does not exist here. I see the following fairly recent change in the glanceclient.common.https code that added this: https://github.com/openstack/python- glanceclient/commit/dbb242b776908ca50ed8557ebfe7cfcd879366c8#diff- 524c1a0b226fa8e5fb4df6ff5574e153R104 I do not understand this
