Well, you might be in luck! Morgan Fainberg actually implemented an improvement that was apparently documented by Adam Young way back in March:
https://bugs.launchpad.net/keystone/+bug/1287757 There's a link to the stable/kilo backport in comment #2 - I'd be eager to hear how it performs for you! On Tue, Jul 21, 2015 at 5:58 PM, Matt Fischer <m...@mattfischer.com> wrote: > Dolph, > > Excuse the delayed reply, was waiting for a brilliant solution from > someone. Without one, personally I'd prefer the cronjob as it seems to be > the type of thing cron was designed for. That will be a painful change as > people now rely on this behavior so I don't know if its feasible. I will be > setting up monitoring for the revocation count and alerting me if it > crosses probably 500 or so. If the problem gets worse then I think a custom > no-op or sql driver is the next step. > > Thanks. > > > On Wed, Jul 15, 2015 at 4:00 PM, Dolph Mathews <dolph.math...@gmail.com> > wrote: > >> >> >> On Wed, Jul 15, 2015 at 4:51 PM, Matt Fischer <m...@mattfischer.com> >> wrote: >> >>> I'm having some issues with keystone revocation events. The bottom line >>> is that due to the way keystone handles the clean-up of these events[1], >>> having more than a few leads to: >>> >>> - bad performance, up to 2x slower token validation with about 600 >>> events based on my perf measurements. >>> - database deadlocks, which cause API calls to fail, more likely with >>> more events it seems >>> >>> I am seeing this behavior in code from trunk on June 11 using Fernet >>> tokens, but the token backend does not seem to make a difference. >>> >>> Here's what happens to the db in terms of deadlock: >>> 2015-07-15 21:25:41.082 31800 TRACE keystone.common.wsgi DBDeadlock: >>> (OperationalError) (1213, 'Deadlock found when trying to get lock; try >>> restarting transaction') 'DELETE FROM revocation_event WHERE >>> revocation_event.revoked_at < %s' (datetime.datetime(2015, 7, 15, 18, 55, >>> 41, 55186),) >>> >>> When this starts happening, I just go truncate the table, but this is >>> not ideal. If [1] is really true then the design is not great, it sounds >>> like keystone is doing a revocation event clean-up on every token >>> validation call. Reading and deleting/locking from my db cluster is not >>> something I want to do on every validate call. >>> >> >> Unfortunately, that's *exactly* what keystone is doing. Adam and I had a >> conversation about this problem in Vancouver which directly resulted in >> opening the bug referenced on the operator list: >> >> https://bugs.launchpad.net/keystone/+bug/1456797 >> >> Neither of us remembered the actual implemented behavior, which is what >> you've run into and Deepti verified in the bug's comments. >> >> >>> >>> So, can I turn of token revocation for now? I didn't see an obvious >>> no-op driver. >>> >> >> Not sure how, other than writing your own no-op driver, or perhaps an >> extended driver that doesn't try to clean the table on every read? >> >> >>> And in the long-run can this be fixed? I'd rather do almost anything >>> else, including writing a cronjob than what happens now. >>> >> >> If anyone has a better solution than the current one, that's also better >> than requiring a cron job on something like keystone-manage >> revocation_flush I'd love to hear it. >> >> >>> [1] - >>> http://lists.openstack.org/pipermail/openstack-operators/2015-June/007210.html >>> >>> >>> __________________________________________________________________________ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev