[Touch-packages] [Bug 2003250] Re: networkctl reload with bond devices causes slaves to go DOWN and UP, causing couple of seconds of network loss
I agree that the importance should be higher than "Low". This bug is also triggered every time a "netplan apply" is run, since netplan will always re-generate the systemd-networkd config files. VLAN interfaces are also torn down and recreated. This is highly problematic on critical networking hosts, such as firewalls, since any networking configuration change will trigger seconds of downtime, which can lead to VRRP failovers, etc... -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/2003250 Title: networkctl reload with bond devices causes slaves to go DOWN and UP, causing couple of seconds of network loss Status in systemd package in Ubuntu: Triaged Status in systemd source package in Jammy: Triaged Status in systemd source package in Kinetic: Triaged Bug description: We currently use Ubuntu 22.04.1 LTS including updates for our production cloud (switched from legacy Centos 7). Although we like the distribution we recently hit serious systemd buggy behavior described in [1] bugreport using packages [2]. Unfortunatelly the clouds we are running consist of openstack on top of kubernetes and we need to have complex network configuration including linux bond devices. Our observation is that every time we apply our configuration via CI/CD infrastructure using ansible and netplan (regardless whether there is actual network configuration change) we see approximatelly 8-16 seconds network interruptions and see bond interfaces going DOWN and then UP. We expect bond interfaces stay UP when there is no network configuration change. We went though couple of options how to solve the issue and the first one is to add such existing patch [3] into current systemd-249.11-0ubuntu3.6. Could you comment whether this kind of non-security patch is likely to land in 22.04.1 LTS soon. We are able to help to bring patch into systemd package community way if you suggest the steps. [1] https://github.com/systemd/systemd/issues/25067 [2] Packages root@controlplane-001:/etc/apt0# apt list | grep -E '^(systemd/|netplan.io)' netplan.io/jammy-updates,now 0.105-0ubuntu2~22.04.1 amd64 [installed,automatic] systemd/jammy-updates,now 249.11-0ubuntu3.6 amd64 [installed,automatic] [3] https://github.com/systemd/systemd/pull/25162 [4] # lsb_release -rd Description:Ubuntu 22.04.1 LTS Release:22.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2003250/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2003250] Re: networkctl reload with bond devices causes slaves to go DOWN and UP, causing couple of seconds of network loss
Seems its not "low importance" bug. 🤕 Updating systemd package can trigger such bug, and if you have a unattended-upgrade it's gonna cause a network flap on that host. Easy to reproduce it with: # apt install --reinstall systemd -- Distributor ID: Ubuntu Description:Ubuntu 22.04.2 LTS Release:22.04 Codename: jammy systemd 249 (249.11-0ubuntu3.9) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/2003250 Title: networkctl reload with bond devices causes slaves to go DOWN and UP, causing couple of seconds of network loss Status in systemd package in Ubuntu: Triaged Status in systemd source package in Jammy: Triaged Status in systemd source package in Kinetic: Triaged Bug description: We currently use Ubuntu 22.04.1 LTS including updates for our production cloud (switched from legacy Centos 7). Although we like the distribution we recently hit serious systemd buggy behavior described in [1] bugreport using packages [2]. Unfortunatelly the clouds we are running consist of openstack on top of kubernetes and we need to have complex network configuration including linux bond devices. Our observation is that every time we apply our configuration via CI/CD infrastructure using ansible and netplan (regardless whether there is actual network configuration change) we see approximatelly 8-16 seconds network interruptions and see bond interfaces going DOWN and then UP. We expect bond interfaces stay UP when there is no network configuration change. We went though couple of options how to solve the issue and the first one is to add such existing patch [3] into current systemd-249.11-0ubuntu3.6. Could you comment whether this kind of non-security patch is likely to land in 22.04.1 LTS soon. We are able to help to bring patch into systemd package community way if you suggest the steps. [1] https://github.com/systemd/systemd/issues/25067 [2] Packages root@controlplane-001:/etc/apt0# apt list | grep -E '^(systemd/|netplan.io)' netplan.io/jammy-updates,now 0.105-0ubuntu2~22.04.1 amd64 [installed,automatic] systemd/jammy-updates,now 249.11-0ubuntu3.6 amd64 [installed,automatic] [3] https://github.com/systemd/systemd/pull/25162 [4] # lsb_release -rd Description:Ubuntu 22.04.1 LTS Release:22.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2003250/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2003250] Re: networkctl reload with bond devices causes slaves to go DOWN and UP, causing couple of seconds of network loss
Same problem here but with strange behaviour with bond interface. All slaves was release from bond and after reloading bond stay without any slave links. dmesg logs bond-xe2: (slave eth4): Releasing backup interface bond-xe2: (slave eth5): Removing an active aggregator bond-xe2: (slave eth5): Releasing backup interface # cat /proc/net/bonding/bond-xe2 Ethernet Channel Bonding Driver: v5.19.0-35-generic Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer3+4 (1) MII Status: down MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Peer Notification Delay (ms): 0 802.3ad info LACP active: on LACP rate: fast Min links: 0 Aggregator selection policy (ad_select): stable System priority: 65535 System MAC address: da:74:dc:e2:48:b4 bond bond-xe2 has no active aggregator After running networkctl reload, bond enslave links and all start working correct. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/2003250 Title: networkctl reload with bond devices causes slaves to go DOWN and UP, causing couple of seconds of network loss Status in systemd package in Ubuntu: Triaged Status in systemd source package in Jammy: Triaged Status in systemd source package in Kinetic: Triaged Bug description: We currently use Ubuntu 22.04.1 LTS including updates for our production cloud (switched from legacy Centos 7). Although we like the distribution we recently hit serious systemd buggy behavior described in [1] bugreport using packages [2]. Unfortunatelly the clouds we are running consist of openstack on top of kubernetes and we need to have complex network configuration including linux bond devices. Our observation is that every time we apply our configuration via CI/CD infrastructure using ansible and netplan (regardless whether there is actual network configuration change) we see approximatelly 8-16 seconds network interruptions and see bond interfaces going DOWN and then UP. We expect bond interfaces stay UP when there is no network configuration change. We went though couple of options how to solve the issue and the first one is to add such existing patch [3] into current systemd-249.11-0ubuntu3.6. Could you comment whether this kind of non-security patch is likely to land in 22.04.1 LTS soon. We are able to help to bring patch into systemd package community way if you suggest the steps. [1] https://github.com/systemd/systemd/issues/25067 [2] Packages root@controlplane-001:/etc/apt0# apt list | grep -E '^(systemd/|netplan.io)' netplan.io/jammy-updates,now 0.105-0ubuntu2~22.04.1 amd64 [installed,automatic] systemd/jammy-updates,now 249.11-0ubuntu3.6 amd64 [installed,automatic] [3] https://github.com/systemd/systemd/pull/25162 [4] # lsb_release -rd Description:Ubuntu 22.04.1 LTS Release:22.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2003250/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2003250] Re: networkctl reload with bond devices causes slaves to go DOWN and UP, causing couple of seconds of network loss
Thank you very much for the status. On our side we improved the ansible logic configuring the networking via netplan the way ansible actually performs netplan apply **only if** there is netplan configuration change. This helps to minimize unexpected bond toggling, but still far from being solved. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/2003250 Title: networkctl reload with bond devices causes slaves to go DOWN and UP, causing couple of seconds of network loss Status in systemd package in Ubuntu: Triaged Status in systemd source package in Jammy: Triaged Status in systemd source package in Kinetic: Triaged Bug description: We currently use Ubuntu 22.04.1 LTS including updates for our production cloud (switched from legacy Centos 7). Although we like the distribution we recently hit serious systemd buggy behavior described in [1] bugreport using packages [2]. Unfortunatelly the clouds we are running consist of openstack on top of kubernetes and we need to have complex network configuration including linux bond devices. Our observation is that every time we apply our configuration via CI/CD infrastructure using ansible and netplan (regardless whether there is actual network configuration change) we see approximatelly 8-16 seconds network interruptions and see bond interfaces going DOWN and then UP. We expect bond interfaces stay UP when there is no network configuration change. We went though couple of options how to solve the issue and the first one is to add such existing patch [3] into current systemd-249.11-0ubuntu3.6. Could you comment whether this kind of non-security patch is likely to land in 22.04.1 LTS soon. We are able to help to bring patch into systemd package community way if you suggest the steps. [1] https://github.com/systemd/systemd/issues/25067 [2] Packages root@controlplane-001:/etc/apt0# apt list | grep -E '^(systemd/|netplan.io)' netplan.io/jammy-updates,now 0.105-0ubuntu2~22.04.1 amd64 [installed,automatic] systemd/jammy-updates,now 249.11-0ubuntu3.6 amd64 [installed,automatic] [3] https://github.com/systemd/systemd/pull/25162 [4] # lsb_release -rd Description:Ubuntu 22.04.1 LTS Release:22.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2003250/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2003250] Re: networkctl reload with bond devices causes slaves to go DOWN and UP, causing couple of seconds of network loss
Applying the patch from upstream had some unexpected problems, i.e. causing other interfaces not to come up when I would have expected them to. This needs further investigation -- maybe there is some missing logic from current upstream that makes the patch incorrect when backported. For now, this patch will not be backported. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/2003250 Title: networkctl reload with bond devices causes slaves to go DOWN and UP, causing couple of seconds of network loss Status in systemd package in Ubuntu: Triaged Status in systemd source package in Jammy: Triaged Status in systemd source package in Kinetic: Triaged Bug description: We currently use Ubuntu 22.04.1 LTS including updates for our production cloud (switched from legacy Centos 7). Although we like the distribution we recently hit serious systemd buggy behavior described in [1] bugreport using packages [2]. Unfortunatelly the clouds we are running consist of openstack on top of kubernetes and we need to have complex network configuration including linux bond devices. Our observation is that every time we apply our configuration via CI/CD infrastructure using ansible and netplan (regardless whether there is actual network configuration change) we see approximatelly 8-16 seconds network interruptions and see bond interfaces going DOWN and then UP. We expect bond interfaces stay UP when there is no network configuration change. We went though couple of options how to solve the issue and the first one is to add such existing patch [3] into current systemd-249.11-0ubuntu3.6. Could you comment whether this kind of non-security patch is likely to land in 22.04.1 LTS soon. We are able to help to bring patch into systemd package community way if you suggest the steps. [1] https://github.com/systemd/systemd/issues/25067 [2] Packages root@controlplane-001:/etc/apt0# apt list | grep -E '^(systemd/|netplan.io)' netplan.io/jammy-updates,now 0.105-0ubuntu2~22.04.1 amd64 [installed,automatic] systemd/jammy-updates,now 249.11-0ubuntu3.6 amd64 [installed,automatic] [3] https://github.com/systemd/systemd/pull/25162 [4] # lsb_release -rd Description:Ubuntu 22.04.1 LTS Release:22.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2003250/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2003250] Re: networkctl reload with bond devices causes slaves to go DOWN and UP, causing couple of seconds of network loss
I have confirmed that this bug affects Jammy and newer. The upstream patch looks straight-forward, so I will test a build with that patch included to see if it fixes the issue. ** Also affects: systemd (Ubuntu Jammy) Importance: Undecided Status: New ** Also affects: systemd (Ubuntu Kinetic) Importance: Undecided Status: New ** Changed in: systemd (Ubuntu Jammy) Status: New => Triaged ** Changed in: systemd (Ubuntu Kinetic) Status: New => Triaged ** Changed in: systemd (Ubuntu Jammy) Importance: Undecided => Low ** Changed in: systemd (Ubuntu Kinetic) Importance: Undecided => Low ** Changed in: systemd (Ubuntu) Status: New => Triaged ** Changed in: systemd (Ubuntu) Importance: Undecided => Low -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/2003250 Title: networkctl reload with bond devices causes slaves to go DOWN and UP, causing couple of seconds of network loss Status in systemd package in Ubuntu: Triaged Status in systemd source package in Jammy: Triaged Status in systemd source package in Kinetic: Triaged Bug description: We currently use Ubuntu 22.04.1 LTS including updates for our production cloud (switched from legacy Centos 7). Although we like the distribution we recently hit serious systemd buggy behavior described in [1] bugreport using packages [2]. Unfortunatelly the clouds we are running consist of openstack on top of kubernetes and we need to have complex network configuration including linux bond devices. Our observation is that every time we apply our configuration via CI/CD infrastructure using ansible and netplan (regardless whether there is actual network configuration change) we see approximatelly 8-16 seconds network interruptions and see bond interfaces going DOWN and then UP. We expect bond interfaces stay UP when there is no network configuration change. We went though couple of options how to solve the issue and the first one is to add such existing patch [3] into current systemd-249.11-0ubuntu3.6. Could you comment whether this kind of non-security patch is likely to land in 22.04.1 LTS soon. We are able to help to bring patch into systemd package community way if you suggest the steps. [1] https://github.com/systemd/systemd/issues/25067 [2] Packages root@controlplane-001:/etc/apt0# apt list | grep -E '^(systemd/|netplan.io)' netplan.io/jammy-updates,now 0.105-0ubuntu2~22.04.1 amd64 [installed,automatic] systemd/jammy-updates,now 249.11-0ubuntu3.6 amd64 [installed,automatic] [3] https://github.com/systemd/systemd/pull/25162 [4] # lsb_release -rd Description:Ubuntu 22.04.1 LTS Release:22.04 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2003250/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
