On Wed, Jul 15, 2015 at 4:51 PM, Matt Fischer <m...@mattfischer.com> wrote:
> I'm having some issues with keystone revocation events. The bottom line is > that due to the way keystone handles the clean-up of these events[1], > having more than a few leads to: > > - bad performance, up to 2x slower token validation with about 600 events > based on my perf measurements. > - database deadlocks, which cause API calls to fail, more likely with > more events it seems > > I am seeing this behavior in code from trunk on June 11 using Fernet > tokens, but the token backend does not seem to make a difference. > > Here's what happens to the db in terms of deadlock: > 2015-07-15 21:25:41.082 31800 TRACE keystone.common.wsgi DBDeadlock: > (OperationalError) (1213, 'Deadlock found when trying to get lock; try > restarting transaction') 'DELETE FROM revocation_event WHERE > revocation_event.revoked_at < %s' (datetime.datetime(2015, 7, 15, 18, 55, > 41, 55186),) > > When this starts happening, I just go truncate the table, but this is not > ideal. If [1] is really true then the design is not great, it sounds like > keystone is doing a revocation event clean-up on every token validation > call. Reading and deleting/locking from my db cluster is not something I > want to do on every validate call. > Unfortunately, that's *exactly* what keystone is doing. Adam and I had a conversation about this problem in Vancouver which directly resulted in opening the bug referenced on the operator list: https://bugs.launchpad.net/keystone/+bug/1456797 Neither of us remembered the actual implemented behavior, which is what you've run into and Deepti verified in the bug's comments. > > So, can I turn of token revocation for now? I didn't see an obvious no-op > driver. > Not sure how, other than writing your own no-op driver, or perhaps an extended driver that doesn't try to clean the table on every read? > And in the long-run can this be fixed? I'd rather do almost anything else, > including writing a cronjob than what happens now. > If anyone has a better solution than the current one, that's also better than requiring a cron job on something like keystone-manage revocation_flush I'd love to hear it. > [1] - > http://lists.openstack.org/pipermail/openstack-operators/2015-June/007210.html > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev