Re: [openstack-dev] [tripleo] Fernet Key rotation
On 08/11/2016 06:25 AM, Steven Hardy wrote: On Wed, Aug 10, 2016 at 11:31:29AM -0400, Zane Bitter wrote: On 09/08/16 21:21, Adam Young wrote: On 08/09/2016 06:00 PM, Zane Bitter wrote: In either case a good mechanism might be to use a Heat Software Deployment via the Heat API directly (i.e. not as part of a stack) to push changes to the servers. (I say 'push' but it's more a case of making the data available for os-collect-config to grab it.) This is the part that interests me most. The rest, I'll code in python and we can call either from mistral or from Cron. What would a stack like this look like? Are there comparable examples? Basically use the "openstack software config create" command to upload a script and the "openstack software deployment create" command to deploy it to a server. I don't have an example I can point you at, but the data is in essentially the same format as the properties of the corresponding Heat resources.[1][2] Steve Baker would know if we have any more detailed docs. Actually we wrapped a mistral workflow and CLI interface around this for operator convenience, so you can just do: [stack@instack ~]$ cat run_ls.sh #!/bin/sh ls /tmp [stack@instack ~]$ openstack overcloud execute -s overcloud-controller-0 run_ls.sh This runs a mistral workflow that creates the heat software config and software deployment, waits for the deployment to complete, then returns the result. Wiring in a periodic mistral workflow which does the same should be possible, but tbh I've not yet looked into the deferred authentication method in that case (e.g I assume it uses trusts but I've not tried it yet). This is the mistral workflow, it could pretty easily be reused or adapted for the use-case described I think: https://github.com/openstack/tripleo-common/blob/master/workbooks/deployment.yaml Again, thanks for the stellar blooging, Steve. POC was posted earlier this month. http://adam.younglogic.com/2016/09/fernet-overcloud/ Packing up the tarball on the undercloud is the eay part. I would like to come up with a general approach for securely distributing keys/secrets from undercloud to overcloud. It might make sense to make use of Barbican for that in future release. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] Fernet Key rotation
On Wed, Aug 10, 2016 at 11:31:29AM -0400, Zane Bitter wrote: > On 09/08/16 21:21, Adam Young wrote: > > On 08/09/2016 06:00 PM, Zane Bitter wrote: > > > > > > In either case a good mechanism might be to use a Heat Software > > > Deployment via the Heat API directly (i.e. not as part of a stack) to > > > push changes to the servers. (I say 'push' but it's more a case of > > > making the data available for os-collect-config to grab it.) > > > > This is the part that interests me most. The rest, I'll code in python > > and we can call either from mistral or from Cron. What would a stack > > like this look like? Are there comparable examples? > > Basically use the "openstack software config create" command to upload a > script and the "openstack software deployment create" command to deploy it > to a server. I don't have an example I can point you at, but the data is in > essentially the same format as the properties of the corresponding Heat > resources.[1][2] Steve Baker would know if we have any more detailed docs. Actually we wrapped a mistral workflow and CLI interface around this for operator convenience, so you can just do: [stack@instack ~]$ cat run_ls.sh #!/bin/sh ls /tmp [stack@instack ~]$ openstack overcloud execute -s overcloud-controller-0 run_ls.sh This runs a mistral workflow that creates the heat software config and software deployment, waits for the deployment to complete, then returns the result. Wiring in a periodic mistral workflow which does the same should be possible, but tbh I've not yet looked into the deferred authentication method in that case (e.g I assume it uses trusts but I've not tried it yet). This is the mistral workflow, it could pretty easily be reused or adapted for the use-case described I think: https://github.com/openstack/tripleo-common/blob/master/workbooks/deployment.yaml Steve __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] Fernet Key rotation
On 09/08/16 21:21, Adam Young wrote: On 08/09/2016 06:00 PM, Zane Bitter wrote: In either case a good mechanism might be to use a Heat Software Deployment via the Heat API directly (i.e. not as part of a stack) to push changes to the servers. (I say 'push' but it's more a case of making the data available for os-collect-config to grab it.) This is the part that interests me most. The rest, I'll code in python and we can call either from mistral or from Cron. What would a stack like this look like? Are there comparable examples? Basically use the "openstack software config create" command to upload a script and the "openstack software deployment create" command to deploy it to a server. I don't have an example I can point you at, but the data is in essentially the same format as the properties of the corresponding Heat resources.[1][2] Steve Baker would know if we have any more detailed docs. - ZB [1] http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Heat::SoftwareConfig [2] http://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Heat::SoftwareDeployment __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] Fernet Key rotation
On 09/08/16 18:28, Fox, Kevin M wrote: It needs to work in a distributed way... What happens if the one node you have cron running on doesn't work for a while. Keystone breaks? IIUC it wouldn't break, but your keys wouldn't get rotated so you'd end up using the same key until such time as your machine running cron comes back again. Adam was suggesting once a month, which honestly ought to be enough time to replace the server with the cron job (which, to be clear, would also be the undercloud server). The bigger danger is probably in forgetting that something is supposed to be running it and never rotating the keys. (Maybe keystone should log a warning when the keys get too old, if it doesn't already.) If the undercloud deploys a timed workfow where the workflow can fail over from machine to machine, that would work. Indeed, but note that this depends on an HA undercloud, which isn't a thing yet in TripleO. (Mainly because deploying and maintaining an HA undercloud is as big of a problem - in fact it's the exact same problem - as deploying the overcloud.) You're correct however that the Mistral approach would get HA for free as soon as we have an HA undercloud, whereas the cron approach just presents another problem that has to be solved in order to get to an HA undercloud (i.e. how to make sure that exactly one machine runs the cron job). cheers, Zane. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] Fernet Key rotation
On 08/09/2016 05:11 PM, Adam Young wrote: The Fernet token format uses a symmetric key to sign tokens. In order to check the signature, these keys need to be synchronized across all of the Keystone servers. I don't want to pass around nake symmetric keys. The right way to do this is to put them into a PKCS 11 Envelope. Roughly, this: 1. Each server generates a keypair and sends the public key to the undercloud 2. undercloud generates a Fernet key 3. Undercloud puts the Fernet token into a PKCS11 document signed with the overcloud nodes public key 4. Undercloud posts the PKCS11 data to metadata Sorry, PKCS12. Not 11. 5. os-*config Node downloads and stores the proper PKCS11 data 6. Something unpackst the pkcs11 data and puts the key into the Fernet key store That last step needs to make use of the keystone-manage fernet_rotate command. How do we go about making this happen? The key rotations should be scheduled infrequently; let me throw out monthly as a starting point for the discussion, although that is probably way too frequent. How do we schedule this? Is this a new stack that depends on the Keystone role? __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] Fernet Key rotation
On 08/09/2016 09:21 PM, Adam Young wrote: On 08/09/2016 06:00 PM, Zane Bitter wrote: In either case a good mechanism might be to use a Heat Software Deployment via the Heat API directly (i.e. not as part of a stack) to push changes to the servers. (I say 'push' but it's more a case of making the data available for os-collect-config to grab it.) This is the part that interests me most. The rest, I'll code in python and we can call either from mistral or from Cron. What would a stack like this look like? Are there comparable examples? __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev So, another aspect to the problem is also that this needs to be done initially as part of the overcloud deployment. If we go Fernet, the keys need to be in place when the Keystone servers boot. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] Fernet Key rotation
On 08/09/2016 06:00 PM, Zane Bitter wrote: In either case a good mechanism might be to use a Heat Software Deployment via the Heat API directly (i.e. not as part of a stack) to push changes to the servers. (I say 'push' but it's more a case of making the data available for os-collect-config to grab it.) This is the part that interests me most. The rest, I'll code in python and we can call either from mistral or from Cron. What would a stack like this look like? Are there comparable examples? __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] Fernet Key rotation
It needs to work in a distributed way... What happens if the one node you have cron running on doesn't work for a while. Keystone breaks? If the undercloud deploys a timed workfow where the workflow can fail over from machine to machine, that would work. Thanks, Kevin From: Zane Bitter [zbit...@redhat.com] Sent: Tuesday, August 09, 2016 3:00 PM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [tripleo] Fernet Key rotation On 09/08/16 17:11, Adam Young wrote: > The Fernet token format uses a symmetric key to sign tokens. In order > to check the signature, these keys need to be synchronized across all of > the Keystone servers. > > > I don't want to pass around nake symmetric keys. The right way to do > this is to put them into a PKCS 11 Envelope. Roughly, this: > > > 1. Each server generates a keypair and sends the public key to the > undercloud > > 2. undercloud generates a Fernet key > > 3. Undercloud puts the Fernet token into a PKCS11 document signed with > the overcloud nodes public key > > 4. Undercloud posts the PKCS11 data to metadata > > 5. os-*config Node downloads and stores the proper PKCS11 data > > 6. Something unpackst the pkcs11 data and puts the key into the Fernet > key store > > That last step needs to make use of the keystone-manage fernet_rotate > command. > > > How do we go about making this happen? The key rotations should be > scheduled infrequently; let me throw out monthly as a starting point for > the discussion, although that is probably way too frequent. How do we > schedule this? Is this a new stack that depends on the Keystone role? This sounds like a classic example of a workflow. Two possibilities come to mind. The fun way: Implement this as a Mistral workflow. Deploy the workflow along with a timed trigger somewhere in the overcloud Heat templates - you can grab data from other resources to figure out e.g. which machines you need to push keys to. The Mistral service on the undercloud will take care of running the workflow for you (thanks to the timed trigger), and its presence in the templates will ensure it gets set up automatically. The boring way: Implement this as a shell script. Add a cron job on the undercloud to run the script. The cron service on the undercloud will take care of running the workflow for you, and adding it to the undercloud puppet manifests will ensure it gets set up automatically. In either case a good mechanism might be to use a Heat Software Deployment via the Heat API directly (i.e. not as part of a stack) to push changes to the servers. (I say 'push' but it's more a case of making the data available for os-collect-config to grab it.) The biggest drawback of the cron job is that it will need to have some way of obtaining credentials in order to push data onto the servers and also to query the overcloud stack to find out which servers to push to. Whereas the Mistral workflow runs as the undercloud (keystone) user who created the 'overcloud' stack and the server list can be supplied through the template. cheers, Zane. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] Fernet Key rotation
On 09/08/16 17:11, Adam Young wrote: The Fernet token format uses a symmetric key to sign tokens. In order to check the signature, these keys need to be synchronized across all of the Keystone servers. I don't want to pass around nake symmetric keys. The right way to do this is to put them into a PKCS 11 Envelope. Roughly, this: 1. Each server generates a keypair and sends the public key to the undercloud 2. undercloud generates a Fernet key 3. Undercloud puts the Fernet token into a PKCS11 document signed with the overcloud nodes public key 4. Undercloud posts the PKCS11 data to metadata 5. os-*config Node downloads and stores the proper PKCS11 data 6. Something unpackst the pkcs11 data and puts the key into the Fernet key store That last step needs to make use of the keystone-manage fernet_rotate command. How do we go about making this happen? The key rotations should be scheduled infrequently; let me throw out monthly as a starting point for the discussion, although that is probably way too frequent. How do we schedule this? Is this a new stack that depends on the Keystone role? This sounds like a classic example of a workflow. Two possibilities come to mind. The fun way: Implement this as a Mistral workflow. Deploy the workflow along with a timed trigger somewhere in the overcloud Heat templates - you can grab data from other resources to figure out e.g. which machines you need to push keys to. The Mistral service on the undercloud will take care of running the workflow for you (thanks to the timed trigger), and its presence in the templates will ensure it gets set up automatically. The boring way: Implement this as a shell script. Add a cron job on the undercloud to run the script. The cron service on the undercloud will take care of running the workflow for you, and adding it to the undercloud puppet manifests will ensure it gets set up automatically. In either case a good mechanism might be to use a Heat Software Deployment via the Heat API directly (i.e. not as part of a stack) to push changes to the servers. (I say 'push' but it's more a case of making the data available for os-collect-config to grab it.) The biggest drawback of the cron job is that it will need to have some way of obtaining credentials in order to push data onto the servers and also to query the overcloud stack to find out which servers to push to. Whereas the Mistral workflow runs as the undercloud (keystone) user who created the 'overcloud' stack and the server list can be supplied through the template. cheers, Zane. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [tripleo] Fernet Key rotation
The Fernet token format uses a symmetric key to sign tokens. In order to check the signature, these keys need to be synchronized across all of the Keystone servers. I don't want to pass around nake symmetric keys. The right way to do this is to put them into a PKCS 11 Envelope. Roughly, this: 1. Each server generates a keypair and sends the public key to the undercloud 2. undercloud generates a Fernet key 3. Undercloud puts the Fernet token into a PKCS11 document signed with the overcloud nodes public key 4. Undercloud posts the PKCS11 data to metadata 5. os-*config Node downloads and stores the proper PKCS11 data 6. Something unpackst the pkcs11 data and puts the key into the Fernet key store That last step needs to make use of the keystone-manage fernet_rotate command. How do we go about making this happen? The key rotations should be scheduled infrequently; let me throw out monthly as a starting point for the discussion, although that is probably way too frequent. How do we schedule this? Is this a new stack that depends on the Keystone role? __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
