First I reverted isc-dhcp-server back to the original focal version, since I had an updated version from the PPA: $ sudo apt install isc-dhcp-server=4.4.1-2.1ubuntu5 isc-dhcp-common=4.4.1-2.1ubuntu5
Then I install the update packages: $ sudo apt update $ sudo apt install libdns-export1109/focal-proposed libirs-export161/focal-proposed libisc-export1105/focal-proposed $ dpkg --status libdns-export1109 libirs-export161 libisc-export1105 | grep Version Version: 1:9.11.16+dfsg-3~ubuntu1 Version: 1:9.11.16+dfsg-3~ubuntu1 Version: 1:9.11.16+dfsg-3~ubuntu1 Then I restarted dhcpd: sudo systemctl restart isc-dhcp-server It has been running for four hours on both systems. ** Tags removed: verification-needed verification-needed-focal ** Tags added: verification-done verification-done-focal -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to isc-dhcp in Ubuntu. https://bugs.launchpad.net/bugs/1872118 Title: [SRU] DHCP Cluster crashes after a few hours Status in DHCP: New Status in bind9-libs package in Ubuntu: Fix Released Status in isc-dhcp package in Ubuntu: Invalid Status in bind9-libs source package in Focal: Fix Committed Status in isc-dhcp source package in Focal: Invalid Status in bind9-libs source package in Groovy: Fix Released Status in isc-dhcp source package in Groovy: Invalid Bug description: [Description] isc-dhcp-server uses libisc-export (coming from bind9-libs package) for handling the socket event(s) when configured in peer mode (master/secondary). It's possible that a sequence of messages dispatched by the master that requires acknowledgment from its peers holds a socket in a pending to send state, a timer or a subsequent write request can be scheduled into this socket and the !sock->pending_send assertion will be raised when trying to write again at the time data hasn't been flushed entirely and the pending_send flag hasn't been reset to 0 state. If this race condition happens, the following stacktrace will be hit: (gdb) info threads Id Target Id Frame * 1 Thread 0x7fb4ddecb700 (LWP 3170) __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 2 Thread 0x7fb4dd6ca700 (LWP 3171) __lll_lock_wait (futex=futex@entry=0x7fb4de6d2028, private=0) at lowlevellock.c:52 3 Thread 0x7fb4de6cc700 (LWP 3169) futex_wake (private=<optimized out>, processes_to_wake=1, futex_word=<optimized out>) at ../sysdeps/nptl/futex-internal.h:364 4 Thread 0x7fb4de74f740 (LWP 3148) futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fb4de6cd0d0) at ../sysdeps/nptl/futex-internal.h:183 (gdb) frame 2 #2 0x00007fb4dec85985 in isc_assertion_failed (file=file@entry=0x7fb4decd8878 "../../../../lib/isc/unix/socket.c", line=line@entry=3361, type=type@entry=isc_assertiontype_insist, cond=cond@entry=0x7fb4decda033 "!sock->pending_send") at ../../../lib/isc/assertions.c:52 (gdb) bt #1 0x00007fb4deaa7859 in __GI_abort () at abort.c:79 #2 0x00007fb4dec85985 in isc_assertion_failed (file=file@entry=0x7fb4decd8878 "../../../../lib/isc/unix/socket.c", line=line@entry=3361, type=type@entry=isc_assertiontype_insist, cond=cond@entry=0x7fb4decda033 "!sock->pending_send") at ../../../lib/isc/assertions.c:52 #3 0x00007fb4decc17e1 in dispatch_send (sock=0x7fb4de6d4990) at ../../../../lib/isc/unix/socket.c:4041 #4 process_fd (writeable=<optimized out>, readable=<optimized out>, fd=11, manager=0x7fb4de6d0010) at ../../../../lib/isc/unix/socket.c:4054 #5 process_fds (writefds=<optimized out>, readfds=0x7fb4de6d1090, maxfd=13, manager=0x7fb4de6d0010) at ../../../../lib/isc/unix/socket.c:4211 #6 watcher (uap=0x7fb4de6d0010) at ../../../../lib/isc/unix/socket.c:4397 #7 0x00007fb4dea68609 in start_thread (arg=<optimized out>) at pthread_create.c:477 #8 0x00007fb4deba4103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 (gdb) frame 3 #3 0x00007fb4decc17e1 in dispatch_send (sock=0x7fb4de6d4990) at ../../../../lib/isc/unix/socket.c:4041 4041 in ../../../../lib/isc/unix/socket.c (gdb) p sock->pending_send $2 = 1 [TEST CASE] 1) Install isc-dhcp-server in 2 focal machine(s). 2) Configure peer/cluster mode as follows: Primary configuration: https://pastebin.ubuntu.com/p/XYj648MghK/ Secondary configuration: https://pastebin.ubuntu.com/p/PYkcshZCWK/ 2) Run dhcpd as follows in both machine(s) # dhcpd -f -d -4 -cf /etc/dhcp/dhcpd.conf --no-pid ens4 3) Leave the cluster running for a long (2h) period until the crash/race condition is reproduced. [REGRESSION POTENTIAL] * The fix will prevent the assertion to happen in the dispatch_send path, later versions of isch-dhcp upstream lack this logic and entirely removed the existence of this flag. Therefore, removing the need for this assertion at process_fd shouldn't be problematic. To manage notifications about this bug go to: https://bugs.launchpad.net/dhcp/+bug/1872118/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp