[openstack-dev] [keystone][fernet] Fernet tokens sync
Hello, As you know, keystone introduced non-persistent tokens in kilo -- Fernet tokens. These tokens use Fernet keys, that are rotated from time to time. A great description of key rotation and replication can be found on [0] and [1] (thanks, lbragstad). In HA setup there are multiple nodes with Keystone and that requires key replication. How do we do that with new Fernet tokens? Please keep in mind that the solution should be HA -- there should not be any "master" server, pushing keys to slave servers, because master server might go down. I can see some ways to do that. 1. Mount some distributed network file system to /etc/keystone/fernet-keys/ (the directory, where keys are) and leave syncronization and dealing with race conditions to it. This solution will not require any changes to existing code. Are there any mature filesystems for that? 2. Use a queue of staged keys. It would mean that a new staging key will be generated if there are no other staging keys in queue. Example: Suppose we have keystone setup on 2 servers. I. In the beginning they have keys 0, 1, 2. II. Rotation happens on keystone-1. 0 becomes 3, 1 is removed. Before generating 0, check that there are no keys in the queue. There are no keys in the queue, generate it and push to keystone-2's queue. III. Rotations happens on keystone-2. 0 becomes 3, 1 is removed. Before generating 0, check that there are no keys in the queue. There is a key from keystone-1, use it as new 0. Thanks to Alexander Makarov for the idea. How do we store this queue? Should we use some backend, rely on creation time or something else? This way requires changes to keystone code. 3. Store keys in backend completely and use well-known sync mechanisms. This would require some changes to keystone code too. -- Best regards, Boris Bobrov __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone][fernet] Fernet tokens sync
On Fri, Mar 27, 2015 at 10:14 AM, Boris Bobrov wrote: > As you know, keystone introduced non-persistent tokens in kilo -- Fernet > tokens. These tokens use Fernet keys, that are rotated from time to time. A > great description of key rotation and replication can be found on [0] and > [1] > (thanks, lbragstad). In HA setup there are multiple nodes with Keystone and > that requires key replication. How do we do that with new Fernet tokens? > > Please keep in mind that the solution should be HA -- there should not be > any > "master" server, pushing keys to slave servers, because master server > might go > down. > In my test environment I was using ansible to sync the keys across multiple nodes. Keystone should probably provide some guidance around this process, but I don't think it should deal with the actual syncing. I think that's better left to an installation's existing configuration management tools. -- David blog: http://www.traceback.org twitter: http://twitter.com/dstanek www: http://dstanek.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone][fernet] Fernet tokens sync
Do the keys all need to be changed at once in a cluster? If so that makes it difficult for puppet at least how we do puppet deployments. Also, David can you share your ansible script for this? On Fri, Mar 27, 2015 at 9:48 AM, David Stanek wrote: > > On Fri, Mar 27, 2015 at 10:14 AM, Boris Bobrov > wrote: > >> As you know, keystone introduced non-persistent tokens in kilo -- Fernet >> tokens. These tokens use Fernet keys, that are rotated from time to time. >> A >> great description of key rotation and replication can be found on [0] and >> [1] >> (thanks, lbragstad). In HA setup there are multiple nodes with Keystone >> and >> that requires key replication. How do we do that with new Fernet tokens? >> >> Please keep in mind that the solution should be HA -- there should not be >> any >> "master" server, pushing keys to slave servers, because master server >> might go >> down. >> > > In my test environment I was using ansible to sync the keys across > multiple nodes. Keystone should probably provide some guidance around this > process, but I don't think it should deal with the actual syncing. I think > that's better left to an installation's existing configuration management > tools. > > > -- > David > blog: http://www.traceback.org > twitter: http://twitter.com/dstanek > www: http://dstanek.com > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone][fernet] Fernet tokens sync
On Friday 27 March 2015 17:14:28 Boris Bobrov wrote: > Hello, > > As you know, keystone introduced non-persistent tokens in kilo -- Fernet > tokens. These tokens use Fernet keys, that are rotated from time to time. A > great description of key rotation and replication can be found on [0] and > [1] (thanks, lbragstad). In HA setup there are multiple nodes with > Keystone and that requires key replication. How do we do that with new > Fernet tokens? > > Please keep in mind that the solution should be HA -- there should not be > any "master" server, pushing keys to slave servers, because master server > might go down. > > [...] [0] and [1] in the mail are: [0]: http://lbragstad.com/?p=133 [1]: http://lbragstad.com/?p=156 After some discussion in #openstack-keystone it seems that token rotation should not be an often procedure and that 15 minutes in the blog post was just an example for the sake of simple math. -- Best regards, Boris Bobrov __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone][fernet] Fernet tokens sync
Matt, The idea is you have a staging key (next key) and you generate that, and sync it out. Once it is synced out you can rotate to it as needed. All keys on the server are valid for token validation. Only the "active" key is used for a given keystone to issue a token. Lance has some ansible stuff he put together for syncing the keys: https://github.com/lbragstad/revolver --Morgan Sent via mobile > On Mar 27, 2015, at 09:02, Matt Fischer wrote: > > Do the keys all need to be changed at once in a cluster? If so that makes it > difficult for puppet at least how we do puppet deployments. > > Also, David can you share your ansible script for this? > >> On Fri, Mar 27, 2015 at 9:48 AM, David Stanek wrote: >> >>> On Fri, Mar 27, 2015 at 10:14 AM, Boris Bobrov wrote: >>> As you know, keystone introduced non-persistent tokens in kilo -- Fernet >>> tokens. These tokens use Fernet keys, that are rotated from time to time. A >>> great description of key rotation and replication can be found on [0] and >>> [1] >>> (thanks, lbragstad). In HA setup there are multiple nodes with Keystone and >>> that requires key replication. How do we do that with new Fernet tokens? >>> >>> Please keep in mind that the solution should be HA -- there should not be >>> any >>> "master" server, pushing keys to slave servers, because master server might >>> go >>> down. >> >> In my test environment I was using ansible to sync the keys across multiple >> nodes. Keystone should probably provide some guidance around this process, >> but I don't think it should deal with the actual syncing. I think that's >> better left to an installation's existing configuration management tools. >> >> >> -- >> David >> blog: http://www.traceback.org >> twitter: http://twitter.com/dstanek >> www: http://dstanek.com >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone][fernet] Fernet tokens sync
On Fri, Mar 27, 2015 at 11:48:29AM -0400, David Stanek wrote: > On Fri, Mar 27, 2015 at 10:14 AM, Boris Bobrov wrote: > > > As you know, keystone introduced non-persistent tokens in kilo -- Fernet > > tokens. These tokens use Fernet keys, that are rotated from time to time. A > > great description of key rotation and replication can be found on [0] and > > [1] > > (thanks, lbragstad). In HA setup there are multiple nodes with Keystone and > > that requires key replication. How do we do that with new Fernet tokens? > > > > Please keep in mind that the solution should be HA -- there should not be > > any > > "master" server, pushing keys to slave servers, because master server > > might go > > down. > > > > In my test environment I was using ansible to sync the keys across multiple > nodes. Keystone should probably provide some guidance around this process, > but I don't think it should deal with the actual syncing. I think that's > better left to an installation's existing configuration management tools. Agreed. This is the same reason why I don't support building in replication functionality to Glance, either. There's lots of external tools that can do this kind of thing, from shared filesystems to BitTorrent, to using Ansible to orchestrate stuff... The best solution is one we don't have to write ourselves. Best, -jay __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
