Status changed to 'Confirmed' because the bug affects multiple users.
** Changed in: isc-dhcp (Ubuntu)
Status: New => Confirmed
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to isc-dhcp in Ubuntu.
https://bugs.launchpad.net/bugs/1934933
Title:
cloud-init dhclient apparmor denied with noexec on /var/tmp
Status in isc-dhcp package in Ubuntu:
Confirmed
Bug description:
Hello - we are seeing an issue on multiple Azure hosts where there is
a long delay during bootup. This appears to be related to an apparmor
issue with dhclient executed via cloud-init when /var is mounted
noexec. Because /var is noexec, the original dhclient is executed
rather than the copy in /var/tmp/cloud-init, which causes the AppArmor
profile to be applied.
This prevents the instance from being able to record the DHCP lease
information to /var/tmp/cloud-init/cloud-init-dhcp-*, which prevents
the instance from being able to obtain goalstate information, and with
cloud-init 21.2-3 or later, results in an extended delay during boot
due to a recent change in azure.py
(https://github.com/canonical/cloud-init/pull/842).
This issue does not occur in default Ubuntu installations (including
the Ubuntu 20.04 default Azure image), as the dhcp.py script in cloud-
init behaves differently, copying /usr/sbin/dhclient to
/var/tmp/cloud-init/cloud-init-dhcp-xxxxx/dhclient when /var allows
executables, and the apparmor profiles then do not apply to the copied
executable.
The syslog will show the following entry when the instance boots up:
cloud-init[820]: 2021-07-07 14:50:40,661 - dhcp.py[WARNING]: dhclient did not
produce expected files: dhcp.leases, dhclient.pid
The cloud-init.log file will show this entry when this issue is occurring.
Since the instance has no IP address at this stage of the boot process, an
unreachable network is to be expected:
azure.py[DEBUG]: Failed HTTP request with Azure endpoint
http://168.63.129.16/machine/?comp=goalstate during attempt 240 with exception:
HTTPConnectionPool(host='168.63.129.16', port=80): Max retries exceeded with
url: /machine/?comp=goalstate (Caused by
NewConnectionError('<urllib3.connection.HTTPConnection object at
0x7fd9e89ee190>: Failed to establish a new connection: [Errno 101] Network is
unreachable'))
With the timeouts in azure.py described above, the instance will not
boot for around 20 minutes until all 240 connection attempts are
completed.
This is logged in /var/log/audit/audit.log, showing that the dhclient process
executed from cloud-init is unable to write the dhclient.pid and dhcp.leases
files that are needed to continue the datasource retrieval process:
type=AVC msg=audit(1625678140.496:1898): apparmor="DENIED" operation="open"
profile="/{,usr/}sbin/dhclient" name="/proc/8537/task/8540/comm" pid=8537
comm="dhclient" requested_mask="wr" denied_mask="wr" fsuid=0 ouid=0
Adding the following file resolves the issue:
/etc/apparmor.d/local/sbin.dhclient
/var/tmp/cloud-init/cloud-init-dhcp-*/dhclient.pid lrw,
/var/tmp/cloud-init/cloud-init-dhcp-*/dhcp.leases lrw,
This allows dhclient executed via cloud-init to write the dhclient.pid
and dhcp.leases files to /var/tmp/cloud-init and the instance to boot
normally.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/isc-dhcp/+bug/1934933/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
