Thanks for the explanation, Dan! I was off down a wrong path, I
appreciate the correction.
I've just downloaded the Azure image from cloud-images.u.c and it
includes this in `/etc/netplan/90-hotplug-azure.yaml`:
# This netplan yaml is delivered in Azure cloud images to support
# attaching and detaching nics after the instance first boot.
# Cloud-init otherwise handles initial boot network configuration in
# /etc/netplan/50-cloud-init.yaml
network:
version: 2
ethernets:
ephemeral:
dhcp4: true
match:
driver: hv_netvsc
name: '!eth0'
optional: true
hotpluggedeth0:
dhcp4: true
match:
driver: hv_netvsc
name: 'eth0'
This file is not present in a booted system, because cloud-init removes
it during boot:
2020-11-09 18:12:09,306 - handlers.py[DEBUG]: start:
azure-ds/maybe_remove_ubuntu_network_config_scripts:
maybe_remove_ubuntu_network_config_scripts
2020-11-09 18:12:09,307 - DataSourceAzure.py[INFO]: Removing Ubuntu extended
network scripts because cloud-init updates Azure network configuration on the
following event: System boot.
2020-11-09 18:12:09,307 - util.py[DEBUG]: Attempting to remove
/etc/netplan/90-hotplug-azure.yaml
2020-11-09 18:12:09,307 - handlers.py[DEBUG]: finish:
azure-ds/maybe_remove_ubuntu_network_config_scripts: SUCCESS:
maybe_remove_ubuntu_network_config_scripts
It does this before the regular cloud-init network configuration is
written, or `netplan generate` is called:
2020-11-09 18:12:09,465 - util.py[DEBUG]: Writing to
/etc/netplan/50-cloud-init.yaml - wb: [644] 603 bytes
2020-11-09 18:12:09,466 - subp.py[DEBUG]: Running command ['netplan',
'generate'] with allowed return codes [0] (shell=False, capture=True)
cloud-init also runs a couple of udevadm commands right after `netplan
generate`:
2020-11-09 18:12:09,813 - subp.py[DEBUG]: Running command ['udevadm',
'test-builtin', 'net_setup_link', '/sys/class/net/eth0'] with allowed return
codes [0] (shell=False, capture=True)
2020-11-09 18:12:09,828 - subp.py[DEBUG]: Running command ['udevadm',
'test-builtin', 'net_setup_link', '/sys/class/net/lo'] with allowed return
codes [0] (shell=False, capture=True)
This all happens before systemd-networkd starts:
Nov 09 18:12:09.956027 focal-1604945439 systemd[1]: Starting Network
Service...
So: I'm not really sure what's going on here. I've tried restoring `90
-hotplug-azure.yaml` and removing `50-cloud-init.yaml`; that doesn't
cause the issue to reproduce on a subsequent boot.
One thing worth noting, that could lead to unexpected state: cloud-init
performs a DHCP on this interface (in order to be able to fetch the
network configuration it is going to apply). It does this in a sandbox
(i.e. it doesn't use system configuration for it), but potentially that
could mean that there's (kernel?) state for that interface which
{udev,network}d interpret in a way that leads to this issue?
** Changed in: cloud-init (Ubuntu)
Status: Incomplete => New
** Changed in: systemd (Ubuntu)
Status: New => Incomplete
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1902960
Title:
Upgrade from 245.4-4ubuntu3.3 to 245.4-4ubuntu3.2 appears to break DNS
resolution in some cases
Status in cloud-init package in Ubuntu:
New
Status in systemd package in Ubuntu:
Incomplete
Bug description:
The systemd upgrade 245.4-4ubuntu3.3 to 245.4-4ubuntu3.2 appears to
have broken DNS resolution across much of our Azure fleet earlier
today. We ended up mitigating this by forcing reboots on the
associated instances, no combination of networkctl reload,
reconfigure, systemctl daemon-reexec, systemctl daemon-reload, netplan
generate, netplan apply would get resolvectl to have a DNS server
again. The main symptom appears to have been systemd-networkd
believing it wasn't managing the eth0 interfaces:
ubuntu@machine-1:~$ sudo networkctl
IDX LINK TYPE OPERATIONAL SETUP
1 lo loopback carrier unmanaged
2 eth0 ether routable unmanaged
2
links listed.
Which eventually made them lose their DNS resolvers:
ubuntu@machine-1:~$ sudo resolvectl dns
Global:
Link 2 (eth0):
After rebooting, we see this behaving properly:
ubuntu@machine-1:~$ sudo networkctl list
IDX LINK TYPE OPERATIONAL SETUP
1 lo loopback carrier unmanaged
2 eth0 ether routable configured
2 links listed.
ubuntu@machine-1:~$ sudo resolvectl dns
Global:
Link 2 (eth0): 168.63.129.16
This appears to be specifically linked to the upgrade, i.e. we were able to
provoke the issue by upgrading the systemd package, so I suspect it's part of
the packaging in the upgrade process.
---
ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu27.10
Architecture: amd64
CasperMD5CheckResult: skip
DistroRelease: Ubuntu 20.04
Lspci-vt:
-[0000:00]-+-00.0 Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge
(AGP disabled)
+-07.0 Intel Corporation 82371AB/EB/MB PIIX4 ISA
+-07.1 Intel Corporation 82371AB/EB/MB PIIX4 IDE
+-07.3 Intel Corporation 82371AB/EB/MB PIIX4 ACPI
\-08.0 Microsoft Corporation Hyper-V virtual VGA
Lsusb: Error: command ['lsusb'] failed with exit code 1:
Lsusb-t:
Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1:
MachineType: Microsoft Corporation Virtual Machine
Package: systemd 245.4-4ubuntu3.3
PackageArchitecture: amd64
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
LANG=C.UTF-8
SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.4.0-1031-azure
root=PARTUUID=2e08bba3-68b4-4a16-af3b-47b73bd138a9 ro console=tty1
console=ttyS0 earlyprintk=ttyS0 panic=-1
ProcVersionSignature: Ubuntu 5.4.0-1031.32-azure 5.4.65
Tags: focal uec-images
Uname: Linux 5.4.0-1031-azure x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 12/07/2018
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 090008
dmi.board.name: Virtual Machine
dmi.board.vendor: Microsoft Corporation
dmi.board.version: 7.0
dmi.chassis.asset.tag: 7783-7084-3265-9085-8269-3286-77
dmi.chassis.type: 3
dmi.chassis.vendor: Microsoft Corporation
dmi.chassis.version: 7.0
dmi.modalias:
dmi:bvnAmericanMegatrendsInc.:bvr090008:bd12/07/2018:svnMicrosoftCorporation:pnVirtualMachine:pvr7.0:rvnMicrosoftCorporation:rnVirtualMachine:rvr7.0:cvnMicrosoftCorporation:ct3:cvr7.0:
dmi.product.name: Virtual Machine
dmi.product.uuid: 4412ad79-83fa-f845-b7c2-6f30dd4f1950
dmi.product.version: 7.0
dmi.sys.vendor: Microsoft Corporation
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1902960/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
