Public bug reported:
[Summary]
On Sep 12, cloud-tuned kernels promoted from 6.5->6.8 for 22.04. After this,
Jammy CVM instances began experiencing a kernel panic post-reboot. The direct
kernel panic regarding locating root part is seemingly a symptom of the actual
problem with the initramfs decompression:
[ 0.805183] Initramfs unpacking failed: invalid magic at start of
compressed archive
CVMs are are unique in the fact that they use a static pre-built
initramfs bundled with the kernel in a Unified Kernel Image.
The current working theory is a problem in the EFI stub produced at
build-time. The environments producing jammy's efi stub are jammy based
and noble's are noble-based. There are significant dependency and
environment changes between the two, which is potentially where the core
problem is. ubuntu-core-initramfs is involved with initramfs and the
assembly of the UKI, so these are the dependencies currently being
explored.
Some early testing was able to reproduce this in a local QEMU env, and
it was discovered that injecting noble's efi stub instead of the
produced jammy stub yielded positive results, but this is still being
explored. In tandem, a bisect of the earlier mentioned deps is also
occurring.
[Reproduction]
+ Launch a Jammy 22.04 CVM image (with or without FDE)
+ Install 6.8 packages in proposed
+ Reboot
[Additional Notes]
+ 6.8 was live in the jammy repos from 9/12 until 9/16. There are likely a
not-insignificant number of instances that either had installed the packages
via automation (UU or otherwise), or manually installed them. These instances
will fail on reboot unless the package is purged or a 6.8 package with a
superseding version number w/ a fix is pushed.
+ CVM instances use nullboot rather than grub. As a result, recovery
from the above situation is quite difficult. You would need to mount the
OS disk elsewhere and remove the jammy efi stub in order to trigger the
EFI fallback to the existing 6.5. This will allow non-fde to boot (on
6.5) after being re-attached to the original vm, but this damages the
measurement on FDE instances and would require the recovery key (CMK or
PMK) be input to continue. Afterwards you'd need to purge 6.8.
+ linux-azure* was pulled back, so 6.8 is currently not available for
install (outside of proposed), but the non-cvm jammy instances were
published w/ a 6.8 kernel. This won't inherently cause problems, but
would make upgrading an existing estate to match what the base images
not reflect difficult.
** Affects: linux-azure-fde (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-azure-fde in Ubuntu.
https://bugs.launchpad.net/bugs/2081311
Title:
Jammy Azure CVM instances fail to boot post 6.8 migration
Status in linux-azure-fde package in Ubuntu:
New
Bug description:
[Summary]
On Sep 12, cloud-tuned kernels promoted from 6.5->6.8 for 22.04. After this,
Jammy CVM instances began experiencing a kernel panic post-reboot. The direct
kernel panic regarding locating root part is seemingly a symptom of the actual
problem with the initramfs decompression:
[ 0.805183] Initramfs unpacking failed: invalid magic at start
of compressed archive
CVMs are are unique in the fact that they use a static pre-built
initramfs bundled with the kernel in a Unified Kernel Image.
The current working theory is a problem in the EFI stub produced at
build-time. The environments producing jammy's efi stub are jammy
based and noble's are noble-based. There are significant dependency
and environment changes between the two, which is potentially where
the core problem is. ubuntu-core-initramfs is involved with initramfs
and the assembly of the UKI, so these are the dependencies currently
being explored.
Some early testing was able to reproduce this in a local QEMU env, and
it was discovered that injecting noble's efi stub instead of the
produced jammy stub yielded positive results, but this is still being
explored. In tandem, a bisect of the earlier mentioned deps is also
occurring.
[Reproduction]
+ Launch a Jammy 22.04 CVM image (with or without FDE)
+ Install 6.8 packages in proposed
+ Reboot
[Additional Notes]
+ 6.8 was live in the jammy repos from 9/12 until 9/16. There are likely a
not-insignificant number of instances that either had installed the packages
via automation (UU or otherwise), or manually installed them. These instances
will fail on reboot unless the package is purged or a 6.8 package with a
superseding version number w/ a fix is pushed.
+ CVM instances use nullboot rather than grub. As a result, recovery
from the above situation is quite difficult. You would need to mount
the OS disk elsewhere and remove the jammy efi stub in order to
trigger the EFI fallback to the existing 6.5. This will allow non-fde
to boot (on 6.5) after being re-attached to the original vm, but this
damages the measurement on FDE instances and would require the
recovery key (CMK or PMK) be input to continue. Afterwards you'd need
to purge 6.8.
+ linux-azure* was pulled back, so 6.8 is currently not available for
install (outside of proposed), but the non-cvm jammy instances were
published w/ a 6.8 kernel. This won't inherently cause problems, but
would make upgrading an existing estate to match what the base images
not reflect difficult.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-azure-fde/+bug/2081311/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
