On 7/14/2014 5:28 PM, Matt Riedemann wrote:
On 7/14/2014 5:18 PM, Ben Nemec wrote:
On 07/14/2014 04:21 PM, Matt Riedemann wrote:
On 7/14/2014 4:09 PM, Matt Riedemann wrote:
I opened bug 1341777 [1] against glance but it looks like it's due to
the default log level for requests.packages.urllib3.connectionpool in
oslo's log module.
The problem is this warning shows up nearly 420K times in 7 days in
Tempest runs:
WARNING urllib3.connectionpool [-] HttpConnectionPool is full,
discarding connection: 127.0.0.1
So either glance is doing something wrong, or that's logging too
high of
a level (I think it should be debug in this case). I'm not really sure
how to scope this down though, or figure out what is so damn chatty in
glance-api that is causing this. It doesn't seem to be causing test
failures, but the rate at which this is logged in glance-api is
surprising.
[1] https://bugs.launchpad.net/glance/+bug/1341777
I found this older thread [1] which led to this in oslo [2] but I'm not
really sure how to use it to make the connectionpool logging quieter in
glance, any guidance there? It looks like in Joe's change to nova for
oslo.messaging he just changed the value directly in the log module in
nova, something I thought was forbidden.
[1]
http://lists.openstack.org/pipermail/openstack-dev/2014-March/030763.html
[2] https://review.openstack.org/#/c/94001/
There was a change recently in incubator to address something related,
but since it's setting to WARN I don't think it would get rid of this
message:
https://github.com/openstack/oslo-incubator/commit/3310d8d2d3643da2fc249fdcad8f5000866c4389
It looks like Joe's change was a cherry-pick of the incubator change to
add oslo.messaging, so discouraged but not forbidden (and apparently
during feature freeze, which is understandable).
-Ben
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Yeah it sounds like either a problem in glance because they don't allow
configuring the max pool size so it defaults to 1, or it's an issue in
python-swiftclient and is being tracked in a different bug:
https://bugs.launchpad.net/python-swiftclient/+bug/1295812
It looks like the issue for the g-api logs was bug 1295812 in
python-swiftclient, around the time that moved to using python-requests.
I noticed last night that the n-cpu/c-vol logs started spiking with the
urllib3 connectionpool warning on 7/11 which is when python-glanceclient
started using requests, so I've changed bug 1341777 to a
python-glanceclient bug.
--
Thanks,
Matt Riedemann
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
