** Description changed:
[ Impact ]
On fast-booting AWS instances (e.g. Graviton 4 m8g/r8g), cloud-init-local
starts before the ENA driver finishes probing PCI.
DataSourceEc2 finds no interfaces, logs "The instance must have at least one
eligible NIC", and falls back to DataSourceNone without writing netplan
configuration. Instances boot without networking and SSH access.
[ Fix ]
PR #7065 added wait_for_candidate_nics() to DataSourceEc2, but restricted
polling to hpc7a.96xlarge.
- The fix expands polling to include Graviton instance families (m8g, r8g, c8g,
etc.) and makes wait_for_nic_timeout configurable via
datasource/Ec2/wait_for_nic_timeout.
+ The fix expands polling to include Graviton instance families (m8g, r8g, c8g,
c7g, m7g, r7g).
[ Test Plan ]
1. Delay ENA probe in modprobe:
echo "install ena /usr/bin/sleep 2; /sbin/modprobe --ignore-install ena" >
/etc/modprobe.d/ena-delay.conf
Run: cloud-init clean --logs && cloud-init init --local
Without fix: DataSourceEc2 exits False.
With fix: DataSourceEc2 waits for the interface and configures networking.
2. Run unit tests:
pytest tests/unittests/sources/test_ec2.py
[ Where problems could occur ]
- If an instance has no network interfaces attached, cloud-init-local will wait
up to the timeout before bailing out. The timeout is configurable via
datasource/Ec2/wait_for_nic_timeout.
+ If an instance has no network interfaces attached, cloud-init-local will wait
up to the default 60s timeout before bailing out.
[ Other Info ]
- Upstream PR: https://github.com/canonical/cloud-init/pull/7138
- Upstream issue: GH-6697
- Verification: 0 regressions on Ubuntu 26.04 and 26.10 test suites.
--- [ Original Report ]
1) Ubuntu Release: Description: Ubuntu 26.04 LTS Release: 26.04
2) Package Version: cloud-init: Installed: 26.1-0ubuntu2
3) What I expected to happen: Cloud-init should successfully detect the
Ec2 datasource and configure networking. Specifically, cloud-init-
local.service should have a dependency on network hardware enumeration
(udev) to ensure that the primary network interface (ena) is visible
before attempting to identify the cloud platform.
4) What happened instead: On modern, fast-booting hardware (specifically
Graviton 4 / m8g instances), or a minimalist OS image, or in scenarios
where the ena driver is slightly delayed, cloud-init-local executes
before the ENA driver has finished probing the PCI bus. Cloud-init sees
only the 'lo' interface, fails to identify the Ec2 datasource, and falls
back to DataSourceNone. This results in the instance booting without
network connectivity, causing systemd-networkd-wait-online to time out.
The cloud-init log explicitly shows the failure:
"DataSourceEc2.py[ERROR]: The instance must have at least one eligible
NIC"
The systemd unit file for cloud-init-local.service in this release is
missing any hardware or udev dependencies: [Unit] Description=Cloud-
init: Local Stage (pre-network) DefaultDependencies=no Wants=network-
pre.target After=hv_kvp_daemon.service Before=network-pre.target
Before=sysinit.target
STEPS TO REPRODUCE: The issue can be deterministically reproduced by
artificially delaying the ena driver initialization to simulate a fast
OS boot outrunning the hardware probe:
Launch a stock Ubuntu 26.04 m8g.medium instance.
Add a modprobe delay for the ena driver: echo "install ena /usr/bin/sleep 10;
/sbin/modprobe --ignore-install ena" > /etc/modprobe.d/ena-delay.conf
Clean cloud-init state and reboot: cloud-init clean --logs && reboot
On the subsequent boot, cloud-init will fail to detect the EC2 datasource
because the ENA driver will still be in its 10-second sleep when
cloud-init-local executes.
Thanks!
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2151279
Title:
cloud-init-local executes before ENA network driver completes PCI
probe, causing DataSourceEc2 failure
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/2151279/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
