** Changed in: cloud-init (Ubuntu) Status: New => Fix Released ** Changed in: cloud-init (Ubuntu Trusty) Status: New => Confirmed
** Changed in: cloud-init (Ubuntu Wily) Status: New => Confirmed ** Changed in: cloud-init (Ubuntu Wily) Status: Confirmed => Won't Fix -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1553815 Title: host keys never restored following metadata api outage Status in cloud-init: Fix Committed Status in cloud-init package in Ubuntu: Fix Released Status in cloud-init source package in Trusty: Confirmed Status in cloud-init source package in Wily: Won't Fix Status in cloud-init source package in Xenial: Fix Released Bug description: We are running an Openstack cloud and have noticed some unexpected behaviour in our Ubuntu Trusty cloud instances created by Nova. We have observed that if a previously initialised instance (e.g. DataSourceOpenstack has already been run) is rebooted while the metadata api is not available (i.e. 169.254.169.264 is unreachable), cloud-init will retry a few times then switch to DataSourceNone and regenerate host keys. # Boot instance under normal conditions ubuntu@vm1:~$ readlink -f /var/lib/cloud/instance /var/lib/cloud/instances/cd535bc4-9c2f-4d31-8903-0ede59c7ef95 ubuntu@vm1:~$ grep "Generating public/private rsa key pair." /var/log/cloud-init-output.log Generating public/private rsa key pair. # Stop neutron metadata api service and reboot instance (observing that host keys were regenerated) ubuntu@vm1:~$ readlink -f /var/lib/cloud/instance /var/lib/cloud/instances/iid-datasource-none ubuntu@vm1:~$ grep "Generating public/private rsa key pair." /var/log/cloud-init-output.log Generating public/private rsa key pair. Generating public/private rsa key pair. So far so good since we expect this behaviour, but now we reboot this instance with the metadata api is once again reachable. Cloud-init rightly selects the original DataSourceOpenstack instance but it does nothing since it already ran once (and it is set to only run once). The problem here is that the original host keys are never restored so any client connecting to that instance will have no option to accept the new host keys along with MITM attack warning. ubuntu@vm1:~$ sudo reboot ... ubuntu@vm1:~$ readlink -f /var/lib/cloud/instance /var/lib/cloud/instances/cd535bc4-9c2f-4d31-8903-0ede59c7ef95 Surely we could find a way for cloud-init to know that if if the current DataSourceOpenstack uuid matches its previously run uuid, then it can check that the host keys are consistent with the original run. @smoser suggested in a side discussion that dmidecode info could perhaps be used since the Openstack instance uuid can be found there: ubuntu@vm1:~$ sudo dmidecode -t system # dmidecode 2.12 SMBIOS 2.8 present. Handle 0x0100, DMI type 1, 27 bytes System Information Manufacturer: OpenStack Foundation Product Name: OpenStack Nova Version: 13.0.0 Serial Number: ba5f7371-fd4c-a25e-132f-3dd1e5b92e93 UUID: CD535BC4-9C2F-4D31-8903-0EDE59C7EF95 Wake-up Type: Power Switch SKU Number: Not Specified Family: Virtual Machine Handle 0x2000, DMI type 32, 11 bytes System Boot Information Status: No errors detected If cloud-init kept a copy of previous host keys prior to regenerating them, it could presumably use this info to know when to safely restore the original host keys. Since it is not inconceivable for the metadata api to become unreachable for a brief period (perhpas during an upgrade), i think we really need to make cloud-init more tolerant of this circumstance. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1553815/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp