On 9/18/2014 5:49 AM, Sean Dague wrote:
On 09/17/2014 11:50 PM, Clark Boylan wrote:
On Wed, Sep 17, 2014, at 06:48 PM, Clark Boylan wrote:
On Wed, Sep 17, 2014, at 06:37 PM, Matt Riedemann wrote:


On 9/17/2014 7:59 PM, Ian Wienand wrote:
On 09/18/2014 09:49 AM, Clark Boylan wrote:
Recent sampling of test run times shows that our tempest jobs run
against clouds using PostgreSQL are significantly slower than jobs run
against clouds using MySQL.

FYI There is a possibly relevant review out for max_connections limits
[1], although it seems to have some issues with shmem usage

-i

[1] https://review.openstack.org/#/c/121952/

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


That's a backport of a fix from master where we were hitting fatal
errors due to too many DB connections which was brought on by the
changes to cinder and glance to run as many workers as there were CPUs
available.  So I don't think it probably plays here...

The errors pointed out in another part of the thread have been around
for awhile, I think they are due to negative tests where we're hitting
unique constraints because of the negative tests, so they are expected.

We should also note that the postgresql jobs run with the nova metadata
API service, I'm not sure how much of a factor that would have here.

Is there anything else unique about those jobs from the MySQL ones?

Good question. There are apparently other differences. The postgres job
runs Keystone under eventlet instead of via apache mod_wsgi. It also
sets FORCE_CONFIGDRIVE=False instead of always. And the final difference
I can find is the one you point out, nova api metadata service is run as
an independent thing.

Could these things be related? It would be relatively simple to push a
change or two to devstack-gate to test this but there are enough options
here that I probably won't do that until we think at least one of these
options is at fault.
I am starting to feel bad that I picked on PostgreSQL and completely
forgot that there were other items in play here. I went ahead and
uploaded [0] to run all devstack jobs without keystone wsgi services
(eventlet) and [1] to run all devstack job with keystone wsgi services
and the initial results are pretty telling.

It appears that keystone eventlet is the source of the slowness in this
job. With keystone eventlet all of the devstack jobs are slower and with
keystone wsgi all of the jobs are quicker. Probably need to collect a
bit more data but this doesn't look good for keystone eventlet.

Thank you Matt for pointing me in this direction.

[0] https://review.openstack.org/#/c/122299/
[1] https://review.openstack.org/#/c/122300/

Don't feel bad. :)

The point that Clark highlights here is a good one. There is an
assumption that once someone creates a job in infra, the magic elves are
responsible for it.

But there are no magic elves. So jobs like this need sponsors.

Maybe the right thing to do is not conflate this configuration and put
an eventlet version of the keystone job only on keystone (because the
keystone team was the one that proposed having a config like that, but
it's so far away from their project they aren't ever noticing when it's
regressing).

Same issue with the metadata server split. That's really only a thing
Nova cares about. It shouldn't impact anyone else.

        -Sean


Neutron cares about the nova metadata API service right?

--

Thanks,

Matt Riedemann


_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Reply via email to