Re: distributing SSH keys in a cluster environment
also sprach Blair Strang <[EMAIL PROTECTED]> [2004.10.30.0237 +0200]: > Based on a cursory look at how FAI works, if you're worried about > a 'laptop attack' -- i.e, an untrusted person with access to your > network media -- I think there are more problems than just SSH > keys. Well, you are too right, unfortunately. I am beginning to believe FAI really needs to be extended to allow for the use of security tokens on the clients (whatever that may be), and switch to getting the configuration space via WebDAV or the like. CVS is already supported, but CVS also adds an extra level of indirection, which may cause problems. The way to do it would be to use a token, such as a USB stick, or a manually keyed passphrase, which then allows (encrypted) access to the master server, from which the configuration space is obtained. After all, at the moment, /etc/fai is exported via NFS, and /etc/fai/class/DEFAULT.var contains the root password to be used on all the nodes. Uh oh. > [Unless I've misunderstood the threat model you're positing here] No, you have not. I was about to invest too much time into this key business though, when in fact, I was forcefully ignoring the fact that the whole thing is as insecure as . I wonder if it's possible to make a secure cluster environment with automatic installations. I guess I will have to go for the /scratch idea... -- Please do not send copies of list mail to me; I read the list! .''`. martin f. krafft <[EMAIL PROTECTED]> : :' :proud Debian developer, admin, user, and author `. `'` `- Debian - when you have better things to do than fixing a system Invalid/expired PGP subkeys? Use subkeys.pgp.net as keyserver! signature.asc Description: Digital signature
Re: distributing SSH keys in a cluster environment
On Fri, Oct 29, 2004 at 10:39:00PM +0200, martin f krafft wrote: > also sprach Mark Bucciarelli <[EMAIL PROTECTED]> [2004.10.29.1920 +0200]: > > what about some kind of cheap usb storage for each machine? > > Then I could just take the USB stick, put it onto my laptop, and > subvert the NFS home directories. If the USB hardware contains sensitive information, then don't put it on the outside of the box. After all, the USB port is just a connector and some wires that lead to the main board. Remove the screws that hold the connector in place and put the connector, with the USB stick, inside the box -- or, better yet, get yourself another connector, which you keep inside the box. Alternatively, you could add another IDE device with the keys on. That doesn't have to be a hard disk; Those soekris thingies which you can buy these days have a CompactFlash card slot with an IDE interface. I don't know whether that is something soekris-specific, but if it exists outside that stuff and isn't too expensive, you could do it that way. In any case, make sure the boxes are properly sealed ;-) -- EARTH smog | bricks AIR -- mud -- FIRE soda water | tequila WATER -- with thanks to fortune signature.asc Description: Digital signature
Re: distributing SSH keys in a cluster environment
also sprach Craig Sanders <[EMAIL PROTECTED]> [2004.10.30.0340 +0200]: > of course, you can be a bit looser with with keys if you're > confident that physical access to the machines AND to the network > segment they are on is properly restricted, AND you have firewall > or other access rules to prevent external machines from fetching > the key files. the switches are under the tables. :/ -- Please do not send copies of list mail to me; I read the list! .''`. martin f. krafft <[EMAIL PROTECTED]> : :' :proud Debian developer, admin, user, and author `. `'` `- Debian - when you have better things to do than fixing a system Invalid/expired PGP subkeys? Use subkeys.pgp.net as keyserver! signature.asc Description: Digital signature
Re: distributing SSH keys in a cluster environment
On Sat, Oct 30, 2004 at 12:37:31AM +0200, martin f krafft wrote: > also sprach Craig Sanders <[EMAIL PROTECTED]> [2004.10.30.0015 +0200]: > > 3. when a machine is being built or rebuilt, install the correct > > ssh keys in /etc/ssh. they can be fetched via password-protected > > http or https or ftp or even tftp, then decrypted and untarred. > > since they're encrypted you don't have to be completely paranoid > > about them - normal security precautions are adequate. > > well, the decryption requires a password, so the installation is not > unattended anymore. since we have a number of headless number > crunchers in the cluster, this is essential. you could do it without the encryption and pass-phrase (or write an expect type script but that would require putting the pass-phrase in plaintext in the script, which defeats the purpose of having a password), but then you'd have to be much more careful about access to the key files. > i am beginning to believe that i am looking for a solution where non > exists... you probably wont get it completely automated if you care about security of the ssh keys. mostly automated with some manual intervention is the best you can expect. of course, you can be a bit looser with with keys if you're confident that physical access to the machines AND to the network segment they are on is properly restricted, AND you have firewall or other access rules to prevent external machines from fetching the key files. craig -- craig sanders <[EMAIL PROTECTED]> (part time cyborg) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: distributing SSH keys in a cluster environment
Based on a cursory look at how FAI works, if you're worried about a 'laptop attack' -- i.e, an untrusted person with access to your network media -- I think there are more problems than just SSH keys. None of the tftp/dhcp/pxe stuff is really designed with security in mind. It seems to me that anyone could compromise an initial install by messing with the boot process. Noisy, but do-able. [Unless I've misunderstood the threat model you're positing here] From this point of view, I can see no reason not to just jigger a fixed host key for the initial install, followed by a keychange over SSH. Mark's suggestion also seemed good. Regards, Blair. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: distributing SSH keys in a cluster environment
martin f krafft said on Sat, Oct 30, 2004 at 01:35:33AM +0200: > FWIW, there is no cfengine host (yet). I am still somewhat taken > aback by its complexity. Just reinstalling the machines with FAI > seems simpler and cleaner. Yeah, I haven't gotten around to using it in production either. :) > Well, this is what I was thinking too. Use an unprivileged account > on the master to drop a sentinel, which makes the master distribute > the keys via SSH. That would work, except now the attacker simply > has to disable a machine and take over its IP, drop said sentinel, > and wait for the master to push the SSH keys. Yep. At some point, you're trusting your network. Only trusting your network at install time is better than trusting your network all of the time, I think. > We used systemimager for years and it drove us crazy as new hardware > was added and multiple people made changes, causing the images to > get out of sync, and multiple images to be created without people > knowing what they were. Yes, it's a policy issue, really... Now we > have an NFS/LDAP solution managed by FAI, which looks very promising > and flexible. We solved that problem at my site by only having one image per OS/architecture, all of which are the same package set. As you said, policy. We also update (using a wrapper around updateclient and cvsup) every night from cron. We're moving towards adding LDAP and Kerberos for user accounts (instead of disting the /etc/passwd stuff), but haven't gotten there yet. I'd like to replace some of the homegrown stuff with cfengine, but as you noticed, it's very complex. M pgpmJEBXsu1D8.pgp Description: PGP signature
Re: distributing SSH keys in a cluster environment
also sprach Mark Ferlatte <[EMAIL PROTECTED]> [2004.10.30.0059 +0200]: > Very little. I would use cfengine to push your ssh keys from your > cfengine host right after FAI. FWIW, there is no cfengine host (yet). I am still somewhat taken aback by its complexity. Just reinstalling the machines with FAI seems simpler and cleaner. > You could, I suppose allow the nodes to FAI, and generate new > keys, and have the master scp their correct keys out (ignoring the > temporary key) and kick sshd. Well, this is what I was thinking too. Use an unprivileged account on the master to drop a sentinel, which makes the master distribute the keys via SSH. That would work, except now the attacker simply has to disable a machine and take over its IP, drop said sentinel, and wait for the master to push the SSH keys. > However, I think this is your best shot for an unattended > installation where you care about the host keys. Yeah, possibly you are right. *This* would be the perfect use for a TPM in the nodes. > FYI: I use systemimager which is rsync based, so I just end up > putting the same ssh key on every sim node in the cluster. Since > I don't care if node42 is spoofing node21 or or not, this works > well for me. We used systemimager for years and it drove us crazy as new hardware was added and multiple people made changes, causing the images to get out of sync, and multiple images to be created without people knowing what they were. Yes, it's a policy issue, really... Now we have an NFS/LDAP solution managed by FAI, which looks very promising and flexible. -- Please do not send copies of list mail to me; I read the list! .''`. martin f. krafft <[EMAIL PROTECTED]> : :' :proud Debian developer, admin, user, and author `. `'` `- Debian - when you have better things to do than fixing a system Invalid/expired PGP subkeys? Use subkeys.pgp.net as keyserver! signature.asc Description: Digital signature
Re: distributing SSH keys in a cluster environment
Martin F Krafft said on Fri, Oct 29, 2004 at 07:03:02PM +0200: > As far as I can tell, there remains one problem: we use SSH > hostbased authentication between the nodes, and while I finally got > that to work, every machine gets a new host key on every > reinstallation, requiring the global database to be updated. Of > course, ssh-keyscan makes that easy, but people *will* forget to > call it, and I refuse to automate the process because there is > almost no intrusion detection going on, so that it would be trivial > to take a get access to the cluster with a laptop. As it stands, > I kept the attack vector small with respect to the data stored on > the cluster, physical security is good, and the whole thing is > behind a fascist firewall anyway. > > So what can I do about these SSH keys? Very little. I would use cfengine to push your ssh keys from your cfengine host right after FAI. You could, I suppose allow the nodes to FAI, and generate new keys, and have the master scp their correct keys out (ignoring the temporary key) and kick sshd. This would prevent transmitted your private keys in the clear, but wouldn't prevent someone from spoofing your newly installed cluster node. However, I think this is your best shot for an unattended installation where you care about the host keys. FYI: I use systemimager which is rsync based, so I just end up putting the same ssh key on every sim node in the cluster. Since I don't care if node42 is spoofing node21 or or not, this works well for me. M pgp833m6XyZUK.pgp Description: PGP signature
Re: distributing SSH keys in a cluster environment
also sprach Craig Sanders <[EMAIL PROTECTED]> [2004.10.30.0015 +0200]: > 3. when a machine is being built or rebuilt, install the correct > ssh keys in /etc/ssh. they can be fetched via password-protected > http or https or ftp or even tftp, then decrypted and untarred. > since they're encrypted you don't have to be completely paranoid > about them - normal security precautions are adequate. well, the decryption requires a password, so the installation is not unattended anymore. since we have a number of headless number crunchers in the cluster, this is essential. i am beginning to believe that i am looking for a solution where non exists... -- Please do not send copies of list mail to me; I read the list! .''`. martin f. krafft <[EMAIL PROTECTED]> : :' :proud Debian developer, admin, user, and author `. `'` `- Debian - when you have better things to do than fixing a system Invalid/expired PGP subkeys? Use subkeys.pgp.net as keyserver! signature.asc Description: Digital signature
Re: distributing SSH keys in a cluster environment
On Fri, Oct 29, 2004 at 07:03:02PM +0200, Martin F Krafft wrote: > As far as I can tell, there remains one problem: we use SSH hostbased > authentication between the nodes, and while I finally got that to > work, every machine gets a new host key on every reinstallation, > requiring the global database to be updated. Of course, ssh-keyscan > makes that easy, but people *will* forget to call it, and I refuse to > automate the process because there is almost no intrusion detection > going on, so that it would be trivial to take a get access to the > cluster with a laptop. As it stands, I kept the attack vector small > with respect to the data stored on the cluster, physical security is > good, and the whole thing is behind a fascist firewall anyway. > > So what can I do about these SSH keys? how about something like this: 1. each node should have gnupg installed, with a public and private key shared between all machines (with a fiendishly long pass-phrase, of course). this key set should be used ONLY for distributing the correct ssh keys to each machine. make a special account for it or specify the config file to use on the gpg command line when decrypting. 2. keep a copy of each node's ssh keys in individual .tar.gz files on the master/boot server machine. each tar.gz file should be encrypted by gnupg for the key above, and the filename should indicate the node's hostname or ip address or some other unique identifier that you can remember when you are building each node. 3. when a machine is being built or rebuilt, install the correct ssh keys in /etc/ssh. they can be fetched via password-protected http or https or ftp or even tftp, then decrypted and untarred. since they're encrypted you don't have to be completely paranoid about them - normal security precautions are adequate. this can be done before ssh is installed (in which case, the post-install script won't generate new keys), or it can be done after ssh is installed (in which case, sshd needs to be restarted after the keys are changed). craig -- craig sanders <[EMAIL PROTECTED]> (part time cyborg) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: distributing SSH keys in a cluster environment
On Fri, 29 Oct 2004 22:38:34 +0200, martin wrote in message <[EMAIL PROTECTED]>: > also sprach Arnt Karlsen <[EMAIL PROTECTED]> [2004.10.29.2054 +0200]: > > ..have each node scp those keys and whatever else you want from > > the boot server, say from each node's /etc/rc.local. _Combine_ some > > node hardware based ID schemes, say nics mac addresses, cpuid, etc. > > How do you suggest to combine a hardware based ID scheme with SSH? > Also, which hardware ID should be used, so that it's not forgeable? ..that depends on your hardware, nic mac addresses can be forged, cpuid can be forged etc. Now, list all your nodes hw info, and see if you can poll s.m.a.r.t'ly for disk partition uids or even md5sums off swap files or swap disks across boots, and you still wind up having to trust your nodes at some stage. Get creative! ;-) -- ..med vennlig hilsen = with Kind Regards from Arnt... ;-) ...with a number of polar bear hunters in his ancestry... Scenarios always come in sets of three: best case, worst case, and just in case. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: distributing SSH keys in a cluster environment
On Friday 29 October 2004 16:39, martin f krafft wrote: > also sprach Mark Bucciarelli <[EMAIL PROTECTED]> [2004.10.29.1920 +0200]: > > what about some kind of cheap usb storage for each machine? > > Then I could just take the USB stick, put it onto my laptop, and > subvert the NFS home directories. use superglue. ;) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: distributing SSH keys in a cluster environment
On Fri, 29 Oct 2004, martin f krafft wrote: > also sprach Mark Bucciarelli <[EMAIL PROTECTED]> [2004.10.29.1920 +0200]: > > what about some kind of cheap usb storage for each machine? > > Then I could just take the USB stick, put it onto my laptop, and > subvert the NFS home directories. Glue it in place, or have it on an internal USB port. Or have the machine room properly locked up in the first place, since anyone who can get to the machines physically has a good deal of a chance of subverting them even without the USB key. -- "One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie." -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: distributing SSH keys in a cluster environment
also sprach Mark Bucciarelli <[EMAIL PROTECTED]> [2004.10.29.1920 +0200]: > what about some kind of cheap usb storage for each machine? Then I could just take the USB stick, put it onto my laptop, and subvert the NFS home directories. -- Please do not send copies of list mail to me; I read the list! .''`. martin f. krafft <[EMAIL PROTECTED]> : :' :proud Debian developer, admin, user, and author `. `'` `- Debian - when you have better things to do than fixing a system Invalid/expired PGP subkeys? Use subkeys.pgp.net as keyserver! signature.asc Description: Digital signature
Re: distributing SSH keys in a cluster environment
also sprach Arnt Karlsen <[EMAIL PROTECTED]> [2004.10.29.2054 +0200]: > ..have each node scp those keys and whatever else you want from > the boot server, say from each node's /etc/rc.local. _Combine_ some > node hardware based ID schemes, say nics mac addresses, cpuid, etc. How do you suggest to combine a hardware based ID scheme with SSH? Also, which hardware ID should be used, so that it's not forgeable? -- Please do not send copies of list mail to me; I read the list! .''`. martin f. krafft <[EMAIL PROTECTED]> : :' :proud Debian developer, admin, user, and author `. `'` `- Debian - when you have better things to do than fixing a system Invalid/expired PGP subkeys? Use subkeys.pgp.net as keyserver! signature.asc Description: Digital signature
Re: distributing SSH keys in a cluster environment
On Fri, 29 Oct 2004 19:03:02 +0200, Martin wrote in message <[EMAIL PROTECTED]>: > Dear wizards, > > [I assume cluster stuff to be better here than -user. Please tell me > if you think otherwise] > > We have just converted our 40 node cluster to FAI and now it's > running shiny sarge at the press of the on button. Thanks to Thomas > Lange for a really incredible solution (FAI), and Mark Burgess for > cfengine2! > > As far as I can tell, there remains one problem: we use SSH > hostbased authentication between the nodes, and while I finally got > that to work, every machine gets a new host key on every > reinstallation, requiring the global database to be updated. Of > course, ssh-keyscan makes that easy, but people *will* forget to > call it, and I refuse to automate the process because there is > almost no intrusion detection going on, so that it would be trivial > to take a get access to the cluster with a laptop. As it stands, > I kept the attack vector small with respect to the data stored on > the cluster, physical security is good, and the whole thing is > behind a fascist firewall anyway. > > So what can I do about these SSH keys? ..have each node scp those keys and whatever else you want from the boot server, say from each node's /etc/rc.local. _Combine_ some node hardware based ID schemes, say nics mac addresses, cpuid, etc. -- ..med vennlig hilsen = with Kind Regards from Arnt... ;-) ...with a number of polar bear hunters in his ancestry... Scenarios always come in sets of three: best case, worst case, and just in case. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: distributing SSH keys in a cluster environment
On Friday 29 October 2004 13:03, Martin F Krafft wrote: > So these are the four possible ways I can think of, and not a single > one is satisfactory. i'm a wizard-wannabe, but i'll reply anyway. what about some kind of cheap usb storage for each machine? -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
