Hello Martijn, or anyone else affected, Accepted isc-dhcp into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/isc- dhcp/4.4.1-2.1ubuntu5.20.04.5 in a few hours, and then in the -proposed repository.
Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed- focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification- failed-focal. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: isc-dhcp (Ubuntu Focal) Status: In Progress => Fix Committed ** Tags added: verification-needed-focal -- You received this bug notification because you are a member of SE SRU ("STS") Sponsors, which is subscribed to the bug report. https://bugs.launchpad.net/bugs/1926139 Title: dhclient: thread concurrency race leads to DHCPOFFER packets not being received Status in bind9-libs package in Ubuntu: Won't Fix Status in isc-dhcp package in Ubuntu: Invalid Status in isc-dhcp source package in Focal: Fix Committed Status in isc-dhcp source package in Jammy: Fix Committed Bug description: [Impact] * Occasionally, during instance boot or machine start-up, dhclient will attempt to acquire a dhcp lease and fail, leaving the instance with no IP address and making it unreachable. * This happens about once every 100 reboots on bare metal, or affecting between ~0.3% to 2% of deployments on Azure (comment #2). * Azure uses dhclient called from cloud-init instead of systemd-networkd, and this is causing issues with larger deployments. * The logs of an affected dhclient produce the following: Listening on LPF/enp1s0/52:54:00:1c:d7:00 Sending on LPF/enp1s0/52:54:00:1c:d7:00 Sending on Socket/fallback DHCPDISCOVER on enp1s0 to 255.255.255.255 port 67 ... DHCPDISCOVER on enp1s0 to 255.255.255.255 port 67 ... ... (omitting 20 similar lines) ... DHCPDISCOVER on enp1s0 to 255.255.255.255 port 67 ... DHCPDISCOVER on enp1s0 to 255.255.255.255 port 67 ... DHCPDISCOVER on enp1s0 to 255.255.255.255 port 67 ... No DHCPOFFERS received. No working leases in persistent database - sleeping. * This only impacts Focal and Jammy, where bind9-libs are multi-threaded (Bionic/earlier and Kinetic/later are single-threaded). * The actual problem is dhclient containing a thread concurrency race condition, and when the race occurs, the read socket is incorrectly/prematurely unwatched because required structures are not yet consistent, thus dhclient does not read any DHCPOFFER replies. * Detailed analysis of the issue is in comment #17. [Fix] * Prevent the race condition by starting to watch the read socket after required structures are consistent. * The fix has been tested in Azure w/ 13500 instances, and no errors have been observed (previously: 0.4%). * Anyway, in case regressions are observed, the patch introduces 2 switches to revert to previous behavior, which can be applied per-process or system-wide: - DHCP_FD_FLAGS_POKE=0 environment variable - dhcp.fd_flags_poke=0 kernel cmdline option * (Previous approaches/discussions included reverting bind9-libs to single-threaded, but we concluded it would have more regression risk than the expected [some bits in comment #8, and some internal chat], and remove exported symbols (apparently unused, but). We also considered a mutex/spinlock approach, but later found a simpler way w/ isc lib; comment #13.) [Test Plan] * Synthetic reproducer with GDB to force the race condition, and DHCP server/client/noise injection is described in comment #9. * Test with the original package (problem occurs). * Test with the modified package (problem fixed). - Set DHCP_FD_FLAGS_POKE=0 (problem occurs). - Set dhcp.fd_flags_poke=0 (problem occurs). [Regression Potential] * 1) dhclient failing to acquire DHCP leases. * 2) dhcpd is also affected by code changes, thus failures to handle DHCP lease requests also have potential for regressions. * 3) the functional change added by the fix, if a regression were to occur, would likely be an issue only under some (unknown) race condition as well, thus expected to be rare. * Note: this potentially affects Focal/Jammy on Azure as a whole, per usage of dhclient in cloud-init instead of systemd-networkd. Azure provided extensive testing for all 3 approaches (mostly internal communications, and some bug comments), with ~13k instances. No issues were observed (previously: 0.4%). * Such testing scale seems to indicate that there are no regressions for dhclient to acquire DHCP leases (1), nor another race condition that hit the fix/new behavior (3). With that, apparently (2) should be OK too. * Also, so to mitigate the regression risk as much as possible, there's very detailed analysis provided here (comments #17, #18) and more information about the fix in its patch file's comment. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/bind9-libs/+bug/1926139/+subscriptions -- Mailing list: https://launchpad.net/~sts-sponsors Post to : sts-sponsors@lists.launchpad.net Unsubscribe : https://launchpad.net/~sts-sponsors More help : https://help.launchpad.net/ListHelp
