As promised here are the fixes: https://review.openstack.org/#/q/Ifc17c27744dac5ad55e84752ca6f68169c2f5a86,n,z
Proposed to both master and liberty. On Wed, Jan 20, 2016 at 12:15 PM, Sean Dague <s...@dague.net> wrote: > On 01/20/2016 02:59 PM, Morgan Fainberg wrote: > > So this was due to a change in keystonemiddleware. We stopped doing > > in-memory caching of tokens per process, per worker by default [1]. > > There are a couple of reasons: > > > > 1) in-memory caching produced unreliable validation because some > > processed may have a cache, some may not > > 2) in-memory caching was unbounded memory wise per worker. > > > > I'll spin up a devstack change to enable memcache and use the memcache > > caching for keystonemiddleware today. This will benefit things in a > > couple ways > > > > * All services and all service's workers will share the offload of the > > validation, likely producing a real speedup even over the old in-memory > > caching > > * There will no longer be inconsistent validation offload/responses > > based upon which worker you happen to hit for a given service. > > > > I'll post to the ML here with the proposed change later today. > > > > [1] > > > https://github.com/openstack/keystonemiddleware/commit/f27d7f776e8556d976f75d07c99373455106de52 > > This seems like a pretty substantial performance impact. Was there a > reno associated with this? > > I think that we should still probably: > > * != the keystone middleware version, it's impacting the ability to land > fixes in the gate > * add devstack memcache code > * find some way to WARN if we are running without memcache config, so > people realize they are in a regressed state > * add back keystone middleware at that version > > -Sean > > > > > Cheers, > > --Morgan > > > > On Tue, Jan 19, 2016 at 10:57 PM, Armando M. <arma...@gmail.com > > <mailto:arma...@gmail.com>> wrote: > > > > > > > > On 19 January 2016 at 22:46, Kevin Benton <blak...@gmail.com > > <mailto:blak...@gmail.com>> wrote: > > > > Hi all, > > > > We noticed a major jump in the neutron tempest and API test run > > times recently in Neutron. After digging through logstash I > > found out that it first occurred on the requirements bump here: > > https://review.openstack.org/#/c/265697/ > > > > After locally testing each requirements change individually, I > > found that the keystonemiddleware change seems to be the > > culprit. It almost doubles the time it takes to fulfill simple > > port-list requests in Neutron. > > > > Armando pushed up a patch here to > > confirm: https://review.openstack.org/#/c/270024/ > > Once that's verified, we should probably put a cap on the > > middleware because it's causing the tests to run up close to > > their time limits. > > > > > > Kevin, > > > > As usual your analytical skills are to be praised. > > > > I wonder if anyone else is aware of the issue/s, because during the > > usual hunting I could see other projects being affected and showing > > abnormally high run times of the dsvm jobs. > > > > I am not sure that [1] is the right approach, but it should give us > > some data points if executed successfully. > > > > Cheers, > > Armando > > > > [1] https://review.openstack.org/#/c/270024/ > > > > > > -- > > Kevin Benton > > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: > > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > < > http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: > > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > < > http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > -- > Sean Dague > http://dague.net > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
