Re: [Openstack] [Keystone] performance issues after havana upgrade

2014-01-29 Thread Adam Young
ing work for the CERN upgrade and this looks like a very useful patch to be part of the standard Havana offering. Tim > -Original Message- > From: Jonathan Proulx [mailto:j...@jonproulx.com] > Sent: 12 January 2014 18:32 > To: Morgan Fainberg > Cc: openstack@lists.openstack.o

Re: [Openstack] [Keystone] performance issues after havana upgrade

2014-01-29 Thread Felix Lee
to be part of the standard Havana offering. Tim > -Original Message- > From: Jonathan Proulx [mailto:j...@jonproulx.com] > Sent: 12 January 2014 18:32 > To: Morgan Fainberg > Cc: openstack@lists.openstack.org > Subject: Re: [Openstack] [Keystone] performance issues after havana

Re: [Openstack] [Keystone] performance issues after havana upgrade

2014-01-13 Thread Morgan Fainberg
nproulx.com] > Sent: 12 January 2014 18:32 > To: Morgan Fainberg > Cc: openstack@lists.openstack.org > Subject: Re: [Openstack] [Keystone] performance issues after havana upgrade > > puzzling side effect? > > I just made a small change to neutron.conf (adjuste

Re: [Openstack] [Keystone] performance issues after havana upgrade

2014-01-12 Thread Ferreira, Rafael
a very >useful patch to be part of the standard Havana offering. > >Tim > >> -Original Message- >> From: Jonathan Proulx [mailto:j...@jonproulx.com] >> Sent: 12 January 2014 18:32 >> To: Morgan Fainberg >> Cc: openstack@lists.openstack.org >

Re: [Openstack] [Keystone] performance issues after havana upgrade

2014-01-12 Thread Tim Bell
anuary 2014 18:32 > To: Morgan Fainberg > Cc: openstack@lists.openstack.org > Subject: Re: [Openstack] [Keystone] performance issues after havana upgrade > > puzzling side effect? > > I just made a small change to neutron.conf (adjusted a default quota) and > restarte

Re: [Openstack] [Keystone] performance issues after havana upgrade

2014-01-12 Thread Jonathan Proulx
puzzling side effect? I just made a small change to neutron.conf (adjusted a default quota) and restarted neutron-server, now neutron (but not other services) is spweing: Invalid user token - rejecting request (quite possibly only from dashboard requests CLI seems to work). I've tried restartin

Re: [Openstack] [Keystone] performance issues after havana upgrade

2014-01-12 Thread Jonathan Proulx
On Sun, Jan 12, 2014 at 12:31 PM, Jonathan Proulx wrote: > puzzling side effect? > > I just made a small change to neutron.conf (adjusted a default quota) > and restarted neutron-server, now neutron (but not other services) is > spweing: > > Invalid user token - rejecting request Looking more clo

Re: [Openstack] [Keystone] performance issues after havana upgrade

2014-01-12 Thread Morgan Fainberg
Thanks for confirming this! It also validates my new logic going into icehouse (I might have had some ulterior motives here, or not so ulterior as the case may be). I'll make sure we resolve the test issues (unrelated to the patch) and get it into the Havana tree so you don't need to maintain it

Re: [Openstack] [Keystone] performance issues after havana upgrade

2014-01-11 Thread Jonathan Proulx
On Sat, Jan 11, 2014 at 10:57 PM, Morgan Fainberg wrote: > Sounds good! Just remember that prior to the fix I posted there, for each > token in the user’s index, it incurred a round-trip to memcached to validate > the token wasn’t expired. This change makes it so that there are > significantly l

Re: [Openstack] [Keystone] performance issues after havana upgrade

2014-01-11 Thread Morgan Fainberg
Sounds good!  Just remember that prior to the fix I posted there, for each token in the user’s index, it incurred a round-trip to memcached to validate the token wasn’t expired.  This change makes it so that there are significantly less trips from keystone to memcached. If this doesn’t 100% sol

Re: [Openstack] [Keystone] performance issues after havana upgrade

2014-01-11 Thread Jonathan Proulx
On Sat, Jan 11, 2014 at 8:24 PM, Morgan Fainberg wrote: > Hi Jon, > > I have published a patch set that I hope will help to address this issue: > https://review.openstack.org/#/c/66149/ . If you need this in another > format, please let me know. That's a fine format, also I love patches that onl

Re: [Openstack] [Keystone] performance issues after havana upgrade

2014-01-11 Thread Morgan Fainberg
Hi Jon, I have published a patch set that I hope will help to address this issue:  https://review.openstack.org/#/c/66149/ . If you need this in another format, please let me know. The only caveat is that you should expect the maximum number of tokens per-user to drop by as much as 50% due to s

Re: [Openstack] [Keystone] performance issues after havana upgrade

2014-01-11 Thread Jonathan Proulx
Note about 30min after flushing memcache things started gumming up again (maybe 15min long tokens instead of 60min?). Details in this response reflect the state of my world at about that point. Recent flush with things just starting to grind to a halt. On Sat, Jan 11, 2014 at 2:46 PM, Jay Pipes

Re: [Openstack] [Keystone] performance issues after havana upgrade

2014-01-11 Thread Jonathan Proulx
Hi Morgan, You may have just given me my weekend back, though don't want to be too optimistic just yet. I set the expiration to 3600, restarted keystone (eventlet) and sent memcached a flush_all. Very briefly everything worked well and at low load, saw accesses whizzing by in the keystone log an

Re: [Openstack] [Keystone] performance issues after havana upgrade

2014-01-11 Thread Jay Pipes
Hi Jonathan! I have not yet deployed Havana to production, however I'll add some comments and suggestions below that you may want to try in order to isolate root cause... On Sat, 2014-01-11 at 13:34 -0500, Jonathan Proulx wrote: > Hi All, > > recently upgraded my 1 controller 60 compute node Ubun

Re: [Openstack] [Keystone] performance issues after havana upgrade

2014-01-11 Thread Morgan Fainberg
Hi Johnathan, This might be related to your issue. I think there are two problems here.  The first problem has to do with limited page sizes in memcache.  If you have an insane number of tokens issued (and as you said neutron is making a ton of requests for new tokens), you can fill up the use

[Openstack] [Keystone] performance issues after havana upgrade

2014-01-11 Thread Jonathan Proulx
Hi All, recently upgraded my 1 controller 60 compute node Ubuntu 12.04(+cloud archive) system from Grizlzy to Havana. Now even before I let my users back to the API I'm barely able to do anything due to authentication time outs. I am using neutron which like to authenticate *a lot*, I'm not enti