Re: Secure deploy of keys
Just did a quick test. Seems feasible to use clevis w/ tpm2 to securely bind credentials to a machine. The idea is: - in case of new install there are no machine-specific files - secrets gets generated as usual - once the machine is up & running, use ssh to run a script to encrypt the needed secret files using machine's TPM and tranfer encrypted files to FAI - in case of reinstall, FAI transfers encrypted files to the machine and runs clevis decrypt to restore 'em That's just a rough idea. Any evident issues? Diego Il 16/01/2023 14:12, Diego Zuccato ha scritto: Tks for the answer. Sorry for seeing it late but it went in the spam folder :( I didn't know clevis/tang, but it's really interesting (maybe a bit overkill in my scenario). Diego Il 15/12/2022 18:53, Robert Markula ha scritto: Am 15.12.22 um 18:15 schrieb Toomas Tamm via linux-fai: This message was wrapped to be DMARC compliant. The actual message text is therefore in an attachment. Hi Toom, unforunately I can't quote you directly, but regarding a rogue attacker mimicking the MAC of an install client: You have to manually enable a FAI installation, otherwise the client cannot be installed: fai-chboot -c DEFAULT client.example.com Granted, with the right timing one could be faster with a rogue client than with the real client. But on the other hand, any client with access to the FAI NFS server can manually mount the NFSroot and obtain any secrets living on the NFS server via this method. So keeping a secret on the NFSroot is not a viable solution. But there are possibilities to work around that. What has been discussed: 1. the secret is created on the install client during installation and transfered to another system in a secure way, e.g. via SSH 2. the secret is pulled from a third-party solution, which is outside the scope of FAI (e.g. via Salt, Cfengine or any other configuration management software). Authenticated registration of the install client to the configuration management software of your choice is the weakest link here [1] 3. using public key encryption (GPG, PKI, SSH) [2] 4. using a zero-trust-like approach to secrets like clevis/tang [3] I have not looked into solutions like HashiCorp Vault, but maybe that can be cleverly integrated as well? Kind regards, Robert [1] https://www.mail-archive.com/linux-fai%40uni-koeln.de/msg07955.html [2] https://www.mail-archive.com/linux-fai%40uni-koeln.de/msg08003.html [3] https://www.mail-archive.com/linux-fai%40uni-koeln.de/msg08005.html -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Secure deploy of keys
Tks for the answer. Sorry for seeing it late but it went in the spam folder :( I didn't know clevis/tang, but it's really interesting (maybe a bit overkill in my scenario). Diego Il 15/12/2022 18:53, Robert Markula ha scritto: Am 15.12.22 um 18:15 schrieb Toomas Tamm via linux-fai: This message was wrapped to be DMARC compliant. The actual message text is therefore in an attachment. Hi Toom, unforunately I can't quote you directly, but regarding a rogue attacker mimicking the MAC of an install client: You have to manually enable a FAI installation, otherwise the client cannot be installed: fai-chboot -c DEFAULT client.example.com Granted, with the right timing one could be faster with a rogue client than with the real client. But on the other hand, any client with access to the FAI NFS server can manually mount the NFSroot and obtain any secrets living on the NFS server via this method. So keeping a secret on the NFSroot is not a viable solution. But there are possibilities to work around that. What has been discussed: 1. the secret is created on the install client during installation and transfered to another system in a secure way, e.g. via SSH 2. the secret is pulled from a third-party solution, which is outside the scope of FAI (e.g. via Salt, Cfengine or any other configuration management software). Authenticated registration of the install client to the configuration management software of your choice is the weakest link here [1] 3. using public key encryption (GPG, PKI, SSH) [2] 4. using a zero-trust-like approach to secrets like clevis/tang [3] I have not looked into solutions like HashiCorp Vault, but maybe that can be cleverly integrated as well? Kind regards, Robert [1] https://www.mail-archive.com/linux-fai%40uni-koeln.de/msg07955.html [2] https://www.mail-archive.com/linux-fai%40uni-koeln.de/msg08003.html [3] https://www.mail-archive.com/linux-fai%40uni-koeln.de/msg08005.html -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Secure deploy of keys
Il 15/12/2022 18:15, Toomas Tamm via linux-fai ha scritto: Some things that I can imagine that could mitigate such risks would be: - Inputting some secret on the physical machine during install (from the keyboard, USB stick, etc). This would defeat the idea of "fully automatic" install. That's a form of "root of trust". - Pre-loading a secret onto hardware (is this what you mean by using TPM?). Yes. TPM (Trusted Platform Module) is a piece of HW that handles crypto keys and should be hard to tamper. At least it would require unsupervised physical access to the interior of the machine for quite a long time. But once the attacker does have unsupervised physical access to the machine, it would be faster to just boot from USB key and extract the files. Unless TPM is also used for secure boot, but that's another can of worms. - Time-limiting the availability of secrets and/or some component of FAI. Most of us probably do not install clients every day, all day. That shouldn't be too hard. Just make secrets available only during install. Once the machine is installed it calls a hook to close the access to the secrets. - Monitoring of installation processes and flagging abnormal activities. This would not prevent successful attacks, but possible breaches could be patched up, eg keys replaced afterwards. This seems harder. -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Secure deploy of keys
Am 15.12.22 um 18:15 schrieb Toomas Tamm via linux-fai: This message was wrapped to be DMARC compliant. The actual message text is therefore in an attachment. Hi Toom, unforunately I can't quote you directly, but regarding a rogue attacker mimicking the MAC of an install client: You have to manually enable a FAI installation, otherwise the client cannot be installed: fai-chboot -c DEFAULT client.example.com Granted, with the right timing one could be faster with a rogue client than with the real client. But on the other hand, any client with access to the FAI NFS server can manually mount the NFSroot and obtain any secrets living on the NFS server via this method. So keeping a secret on the NFSroot is not a viable solution. But there are possibilities to work around that. What has been discussed: 1. the secret is created on the install client during installation and transfered to another system in a secure way, e.g. via SSH 2. the secret is pulled from a third-party solution, which is outside the scope of FAI (e.g. via Salt, Cfengine or any other configuration management software). Authenticated registration of the install client to the configuration management software of your choice is the weakest link here [1] 3. using public key encryption (GPG, PKI, SSH) [2] 4. using a zero-trust-like approach to secrets like clevis/tang [3] I have not looked into solutions like HashiCorp Vault, but maybe that can be cleverly integrated as well? Kind regards, Robert [1] https://www.mail-archive.com/linux-fai%40uni-koeln.de/msg07955.html [2] https://www.mail-archive.com/linux-fai%40uni-koeln.de/msg08003.html [3] https://www.mail-archive.com/linux-fai%40uni-koeln.de/msg08005.html
RE: Secure deploy of keys
Diese Nachricht wurde eingewickelt um DMARC-kompatibel zu sein. Die eigentliche Nachricht steht dadurch in einem Anhang. This message was wrapped to be DMARC compliant. The actual message text is therefore in an attachment.--- Begin Message --- Hello, In the case of public network and no access to the machine at time of installation, I think this is unsolvable in principle. Imagine the following scenario, say on a school on university network: an attacker (perhaps just a curious student) sets up a virtual machine with the MAC address and emulated hardware identical to a physical machine that would be installed by FAI. Then he disconnects the actual machine (say, in a computer classroom) and initiates a FAI install on his VM. Any exchange of secrets that would take place with the real machine now takes place with the student's VM. Afterwards, he takes the VM image home and can extract any secrets that were transferred to it during install. Some things that I can imagine that could mitigate such risks would be: - Inputting some secret on the physical machine during install (from the keyboard, USB stick, etc). This would defeat the idea of "fully automatic" install. - Pre-loading a secret onto hardware (is this what you mean by using TPM?). - Time-limiting the availability of secrets and/or some component of FAI. Most of us probably do not install clients every day, all day. - Monitoring of installation processes and flagging abnormal activities. This would not prevent successful attacks, but possible breaches could be patched up, eg keys replaced afterwards. BR, Toomas -Original Message- From: linux-fai On Behalf Of Diego Zuccato Sent: kolmapäev, 14. detsember 2022 07:40 To: linux-fai@uni-koeln.de Subject: Re: Secure deploy of keys Tks. Too bad I fear it's not applicable to my scenario. First because the network is public. Second because ssh is just one of the secrets I have to distribute (others are usually SaltStack key and Gluster certificate). I'm thinking that probably this is one of the few cases where a TPM is actually useful... GPG encrypted tarballs can be a good solution if there's a trusted person that can insert the password (or a tpm that can decrypt it) to complete the install... Diego Il 13/12/2022 20:44, Andrew Ruthven ha scritto: > Hey, > > On Tue, 2022-12-13 at 14:47 +0100, Diego Zuccato wrote: >> What's the recommended way to deploy (or re-deploy) security-sensitive >> objects (just to say one: private ssh key to avoid client warnings when >> redeploying a server)? > > For things like ssh host keys I have a command that we run which copies > them into the NFSROOT, and then a cron job that runs every minute that > removes "expired" files from the NFSROOT. Given our NFSROOT is on a > restricted network I feel that is sufficient. > > I know someone who had GPG encrypted tarballs, but that required > entering a passphrase during the build process. > > Another option for ssh which I am considering is using PKI for it. Then > servers and clients just need to trust a CA. > > Cheers, > Andrew > > -- > > Andrew Ruthven, Wellington, New Zealand > and...@etc.gen.nz | > Catalyst Cloud: | This space intentionally left blank > https://catalystcloud.nz | > -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786 --- End Message ---
Re: Secure deploy of keys
Am 13.12.22 um 14:47 schrieb Diego Zuccato:
Hello all.
What's the recommended way to deploy (or re-deploy) security-sensitive
objects (just to say one: private ssh key to avoid client warnings
when redeploying a server)?
You could use RedHat's Latchset framework [1] with Clevis (on the
client) and Tang (on the server).
It would roughly work as follows:
1. A secret is generated locally on the FAI client:
head -100 /dev/urandom | tr -dc 'a-z0-9' > secret.txt
2. This secret is encrypted locally using the Tang server:
clevis encrypt tang
'{"url":"http://tang.example.com","thp":"Jfmvz_ZjfbCgeFqAgWpTOIgEeRw4"}'
< secret.txt > secret_ENCRYPTED.txt
Where 'Jfmvz_ZjfbCgeFqAgWpTOIgEeRw4' is the publically known key of the
Tang server.
3. You can copy the 'secret_ENCRYPTED.txt' to any SSH server.
Alternatively you could place this file in FAI's log directory, which
gets conveniently copied to the FAI server at the end of the FAI
installation run. The secret would then live on the FAI server in an
encrpyted form.
4. From then on, the secret can be used by any kind of automated or
manual process. Let's assume, the admin (you) wants to read the secret.
You login to the FAI server, head to the FAI log dir and decrypt the
secret, again using the Tang server:
clevis decrypt < secret_ENCRYPTED.txt > secret.txt
With Clevis and Tang, you could even define the requirement to use e.g.
two different Tang servers in order to be able to decrypt the file. Or
you could require to use one Tang server and a password or even the TPM.
The framework is very flexible here.
But now for the really interesting part: you do not need (!) access to
the Tang server when encrypting files. So the FAI client and the FAI
server do not need access to the Tang server. You can encrypt files
offline by providing the so-called public 'server advertisement' of the
Tang server during encryption. So all clients can only encrypt, but
never decrypt data.
Your Salt server - or admin machine -, on the other hand, would have
access to the Tang server and thus be able to automatically decrypt your
secrets.
You can even use Clevis and Tang with LUKS in order to encrypt disks
upon creation and decrypt them automatically during bootup. So during
normal operation, fully encrypted systems are able to bootup without any
hands-on requirement as long as the Tang server is reachable. But any
intruder that steals your machines cannot decrypt the systems because he
has no access to your internal Tang server. Fallback to password-based
decryption is, of course, always possible.
[1] https://github.com/latchset/clevis
Re: Secure deploy of keys
Tks. Too bad I fear it's not applicable to my scenario. First because the network is public. Second because ssh is just one of the secrets I have to distribute (others are usually SaltStack key and Gluster certificate). I'm thinking that probably this is one of the few cases where a TPM is actually useful... GPG encrypted tarballs can be a good solution if there's a trusted person that can insert the password (or a tpm that can decrypt it) to complete the install... Diego Il 13/12/2022 20:44, Andrew Ruthven ha scritto: Hey, On Tue, 2022-12-13 at 14:47 +0100, Diego Zuccato wrote: What's the recommended way to deploy (or re-deploy) security-sensitive objects (just to say one: private ssh key to avoid client warnings when redeploying a server)? For things like ssh host keys I have a command that we run which copies them into the NFSROOT, and then a cron job that runs every minute that removes "expired" files from the NFSROOT. Given our NFSROOT is on a restricted network I feel that is sufficient. I know someone who had GPG encrypted tarballs, but that required entering a passphrase during the build process. Another option for ssh which I am considering is using PKI for it. Then servers and clients just need to trust a CA. Cheers, Andrew -- Andrew Ruthven, Wellington, New Zealand and...@etc.gen.nz | Catalyst Cloud: | This space intentionally left blank https://catalystcloud.nz | -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
Re: Secure deploy of keys
Hey, On Tue, 2022-12-13 at 14:47 +0100, Diego Zuccato wrote: > What's the recommended way to deploy (or re-deploy) security- > sensitive > objects (just to say one: private ssh key to avoid client warnings > when > redeploying a server)? For things like ssh host keys I have a command that we run which copies them into the NFSROOT, and then a cron job that runs every minute that removes "expired" files from the NFSROOT. Given our NFSROOT is on a restricted network I feel that is sufficient. I know someone who had GPG encrypted tarballs, but that required entering a passphrase during the build process. Another option for ssh which I am considering is using PKI for it. Then servers and clients just need to trust a CA. Cheers, Andrew -- Andrew Ruthven, Wellington, New Zealand and...@etc.gen.nz | Catalyst Cloud: | This space intentionally left blank https://catalystcloud.nz |
Re: Secure deploy of keys
Hi all, What's the recommended way to deploy (or re-deploy) security-sensitive objects (just to say one: private ssh key to avoid client warnings when redeploying a server)? One solution that comes to my mind is to generate a local GPG key and then authorize it for using a pass store (https://www.passwordstore.org/) before running a softupdate. This is not ideal, since there are no secrets available in the initial installation, though, but prevents leaking any sensitive data. Best, Max
Re: Secure deploy of keys
Hello, I would be very interested if you find any solutions. By design, the FAI config space has to be somewhere where it is accessible without access control (anonymous NFS or whatever), and everything within it obviously has to be readable. I guess you will need to find other solutions. As for the SSH keys, I am currently trying to publish SSH keys in DNS so clients can verify them. Haven't tested yet what happens when the client already has a (different) key in its known_hosts file, though. Bye, Andreas Am 13.12.22 um 14:47 schrieb Diego Zuccato: > Hello all. > > What's the recommended way to deploy (or re-deploy) security-sensitive > objects (just to say one: private ssh key to avoid client warnings when > redeploying a server)? > > TIA
Secure deploy of keys
Hello all. What's the recommended way to deploy (or re-deploy) security-sensitive objects (just to say one: private ssh key to avoid client warnings when redeploying a server)? TIA -- Diego Zuccato DIFA - Dip. di Fisica e Astronomia Servizi Informatici Alma Mater Studiorum - Università di Bologna V.le Berti-Pichat 6/2 - 40127 Bologna - Italy tel.: +39 051 20 95786
