On Fri, Jun 9, 2017 at 9:57 AM, Mike Bayer <mba...@redhat.com> wrote:
> > > On 06/08/2017 01:34 PM, Lance Bragstad wrote: > >> After digging into etcd a bit, one place this might be help deployer >> experience would be the handling of fernet keys for token encryption in >> keystone. Currently, all keys used to encrypt and decrypt tokens are kept >> on disk for each keystone node in the deployment. While simple, it requires >> operators to perform rotation on a single node and then push, or sync, the >> new key set to the rest of the nodes. This must be done in lock step in >> order to prevent early token invalidation and inconsistent token responses. >> >> An alternative would be to keep the keys in etcd and make the fernet bits >> pluggable so that it's possible to read keys from disk or etcd (pending >> configuration). The advantage would be that operators could initiate key >> rotations from any keystone node in the deployment (or using etcd directly) >> and not have to worry about distributing the new key set. Since etcd >> associates metadata to the key-value pairs, we might be able to simplify >> the rotation strategy as well. >> > > Interesting, I had the mis-conception that "fernet" keys no longer > required any server-side storage (how is "kept-on-disk" now implemented?) . Currently - the keys used to encrypt and decrypt fernet tokens are stored as files on the keystone server. The repositories default location is in `/etc/keystone/fernet-keys`. The size of this repository is regulated by the rotation process we provide in keystone-manage tooling [0]. > We've had continuous issues with the pre-fernet Keystone tokens filling up > databases, even when operators were correctly expunging old tokens; some > environments just did so many requests that the keystone-token table still > blew up to where MySQL can no longer delete from it without producing a > too-large transaction for Galera. > Yep - we actually just fixed a bug related to this [1]. > > So after all the "finally fernet solves this problem" we propose, hey lets > put them *back* in the database :). That's great. But, lets please not > leave "cleaning out old tokens" as some kind of cron/worry-about-it-later > thing. that was a terrible architectural decision, with apologies to > whoever made it. if you're putting some kind of "we create an infinite, > rapidly growing, turns-to-garbage-in-30-seconds" kind of data in a > database, removing that data robustly and ASAP needs to be part of the > process. > > I should have clarified. The idea was to put the keys used to encrypt and decrypt the tokens in etcd so that synchronizing the repository across a cluster for keystone nodes is easier for operators (but not without other operator pain as Kevin pointed out). The tokens themselves will remain completely non-persistent. Fernet key creation is explicitly controlled by operators and isn't something that end users generate. [0] https://github.com/openstack/keystone/blob/c528539879e824b8e6d5654292a85ccbee6dcf89/keystone/conf/fernet_tokens.py#L44-L54 [1] https://launchpad.net/bugs/1649616 > > > > > >> On Thu, Jun 8, 2017 at 11:37 AM, Mike Bayer <mba...@redhat.com <mailto: >> mba...@redhat.com>> wrote: >> >> >> >> On 06/08/2017 12:47 AM, Joshua Harlow wrote: >> >> So just out of curiosity, but do people really even know what >> etcd is good for? I am thinking that there should be some >> guidance from folks in the community as to where etcd should be >> used and where it shouldn't (otherwise we just all end up in a >> mess). >> >> >> So far I've seen a proposal of etcd3 as a replacement for memcached >> in keystone, and a new dogpile connector was added to oslo.cache to >> handle referring to etcd3 as a cache backend. This is a really >> simplistic / minimal kind of use case for a key-store. >> >> But, keeping in mind I don't know anything about etcd3 other than >> "it's another key-store", it's the only database used by Kubernetes >> as a whole, which suggests it's doing a better job than Redis in >> terms of "durable". So I wouldn't be surprised if new / existing >> openstack applications express some gravitational pull towards using >> it as their own datastore as well. I'll be trying to hang onto >> the etcd3 track as much as possible so that if/when that happens I >> still have a job :). >> >> >> >> >> >> Perhaps a good idea to actually give examples of how it should >> be used, how it shouldn't be used, what it offers, what it >> doesn't... Or at least provide links for people to read up on >> this. >> >> Thoughts? >> >> Davanum Srinivas wrote: >> >> One clarification: Since >> https://pypi.python.org/pypi/etcd3gw >> <https://pypi.python.org/pypi/etcd3gw> just >> uses the HTTP API (/v3alpha) it will work under both >> eventlet and >> non-eventlet environments. >> >> Thanks, >> Dims >> >> >> On Wed, Jun 7, 2017 at 6:47 AM, Davanum >> Srinivas<dava...@gmail.com <mailto:dava...@gmail.com>> >> wrote: >> >> Team, >> >> Here's the update to the base services resolution from >> the TC: >> https://governance.openstack.o >> rg/tc/reference/base-services.html >> <https://governance.openstack. >> org/tc/reference/base-services.html> >> >> First request is to Distros, Packagers, Deployers, >> anyone who >> installs/configures OpenStack: >> Please make sure you have latest etcd 3.x available in >> your >> environment for Services to use, Fedora already does, we >> need help in >> making sure all distros and architectures are covered. >> >> Any project who want to use etcd v3 API via grpc, please >> use: >> https://pypi.python.org/pypi/etcd3 >> <https://pypi.python.org/pypi/etcd3> (works only for >> non-eventlet services) >> >> Those that depend on eventlet, please use the etcd3 >> v3alpha HTTP API using: >> https://pypi.python.org/pypi/etcd3gw >> <https://pypi.python.org/pypi/etcd3gw> >> >> If you use tooz, there are 2 driver choices for you: >> https://github.com/openstack/t >> ooz/blob/master/setup.cfg#L29 >> <https://github.com/openstack/ >> tooz/blob/master/setup.cfg#L29> >> https://github.com/openstack/t >> ooz/blob/master/setup.cfg#L30 >> <https://github.com/openstack/ >> tooz/blob/master/setup.cfg#L30> >> >> If you use oslo.cache, there is a driver for you: >> https://github.com/openstack/o >> slo.cache/blob/master/setup.cfg#L33 >> <https://github.com/openstack/ >> oslo.cache/blob/master/setup.cfg#L33> >> >> Devstack installs etcd3 by default and points cinder to >> it: >> http://git.openstack.org/cgit/ >> openstack-dev/devstack/tree/lib/etcd3 >> <http://git.openstack.org/cgit >> /openstack-dev/devstack/tree/lib/etcd3> >> http://git.openstack.org/cgit/ >> openstack-dev/devstack/tree/lib/cinder#n356 >> <http://git.openstack.org/cgit >> /openstack-dev/devstack/tree/lib/cinder#n356> >> >> >> Review in progress for keystone to use etcd3 for caching: >> https://review.openstack.org/#/c/469621/ >> <https://review.openstack.org/#/c/469621/> >> >> Doug is working on proposal(s) for oslo.config to store >> some >> configuration in etcd3: >> https://review.openstack.org/#/c/454897/ >> <https://review.openstack.org/#/c/454897/> >> >> So, feel free to turn on / test with etcd3 and report >> issues. >> >> Thanks, >> Dims >> >> -- Davanum Srinivas :: >> https://twitter.com/dims >> >> >> >> >> >> ____________________________________________________________ >> ______________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> <http://openstack-dev-requ...@lists.openstack.org?subject:un >> subscribe> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev> >> >> >> ____________________________________________________________ >> ______________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev> >> >> >> >> >> ____________________________________________________________ >> ______________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib >> e >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
