Public bug reported: Using CEPH RBD as our ephemeral drive led to an issue when upgrading from Kilo to Liberty. Our environment has "force_config_drive = True".
In Icehouse, Juno, and Kilo, this uses an ISO 9660 image created in /var/lib/nova/instances/$UUID/disk.config However, in Liberty, if using CEPH RBD for ephemeral, there is a switch to putting this in rbe like this: rbd:instances/${UUID}_disk.config While this works GREAT for new VMs, it is problematic with existing VMs as not all transition states were considered. In particular, if you do a nova stop $UUID followed by a nova start $UUID you will find your instance still in the stopped state. There is something in the start code that ASSUMES that the new rbd format will be in place (but it doesn't actually create it.) There is a work around if you find instances in that state, simply cold migrate them with nova migrate $UUID which redoes the config.drive plumbing and creates the rbd:instances/${UUID}_disk.config Our permanent work around has been to prepopulate the rbd via a script though getting this bug fixed would be much better. Liberty is a stable release and this is a loss of service type of bug so should get fixed. Not clear if this is also an issue (likely so) in Mitaka/Newton as we haven't got an environment yet to test it, but presumably with long running VMs from early config drive, it would also exist in Mitaka. Specifics: Liberty Nova nova:12.0.2-38-g7bc3355.13.1b76006 CEPH: 0.94.6-1trusty Host OS: Ubuntu Trusty ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1582684 Title: nova kilo liberty ceph configdrive upgrade Status in OpenStack Compute (nova): New Bug description: Using CEPH RBD as our ephemeral drive led to an issue when upgrading from Kilo to Liberty. Our environment has "force_config_drive = True". In Icehouse, Juno, and Kilo, this uses an ISO 9660 image created in /var/lib/nova/instances/$UUID/disk.config However, in Liberty, if using CEPH RBD for ephemeral, there is a switch to putting this in rbe like this: rbd:instances/${UUID}_disk.config While this works GREAT for new VMs, it is problematic with existing VMs as not all transition states were considered. In particular, if you do a nova stop $UUID followed by a nova start $UUID you will find your instance still in the stopped state. There is something in the start code that ASSUMES that the new rbd format will be in place (but it doesn't actually create it.) There is a work around if you find instances in that state, simply cold migrate them with nova migrate $UUID which redoes the config.drive plumbing and creates the rbd:instances/${UUID}_disk.config Our permanent work around has been to prepopulate the rbd via a script though getting this bug fixed would be much better. Liberty is a stable release and this is a loss of service type of bug so should get fixed. Not clear if this is also an issue (likely so) in Mitaka/Newton as we haven't got an environment yet to test it, but presumably with long running VMs from early config drive, it would also exist in Mitaka. Specifics: Liberty Nova nova:12.0.2-38-g7bc3355.13.1b76006 CEPH: 0.94.6-1trusty Host OS: Ubuntu Trusty To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1582684/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp