Hello Andrew, I just reviewed the core file that you provided. Thread 1 is the thread that panics on assertion because sock.pending_send is already set. This is the condition I prevented in the PPA, so *shouldn't* be hitting the frame 3
In my test systems I don't hit this condition, dispatch_send isn't called if pending_send is set. (gdb) thread 1 [Switching to thread 1 (Thread 0x7f39a41f5700 (LWP 18780))] #1 0x00007f39a4dd1859 in __GI_abort () at abort.c:79 79 in abort.c (gdb) bt #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #1 0x00007f39a4dd1859 in __GI_abort () at abort.c:79 #2 0x00007f39a4faf985 in isc_assertion_failed (file=<optimized out>, line=<optimized out>, type=<optimized out>, cond=<optimized out>) at ../../../lib/isc/assertions.c:52 #3 0x00007f39a4feb7e1 in dispatch_send (sock=0x7f39a4a03730) at ../../../../lib/isc/unix/socket.c:3380 #4 process_fd (writeable=<optimized out>, readable=<optimized out>, fd=0, manager=0x7f39a49fa010) at ../../../../lib/isc/unix/socket.c:4054 #5 process_fds (writefds=<optimized out>, readfds=0x16, maxfd=-1533038191, manager=0x7f39a49fa010) at ../../../../lib/isc/unix/socket.c:4211 #6 watcher (uap=0x7f39a49fa010) at ../../../../lib/isc/unix/socket.c:4397 [...] (gdb) frame 3 #3 0x00007f39a4feb7e1 in dispatch_send (sock=0x7f39a4a03730) at ../../../../lib/isc/unix/socket.c:3380 3380 ../../../../lib/isc/unix/socket.c: No such file or directory. (gdb) info locals iev = 0x0 ev = <optimized out> sender = 0x2 iev = <optimized out> ev = <optimized out> sender = <optimized out> (gdb) p sock $1 = (isc__socket_t *) 0x7f39a4a03730 (gdb) p sock.pending_send $2 = 1 Can you check your library links, etc? ubuntu@dhcpd1:~$ ldd /usr/sbin/dhcpd | grep export libirs-export.so.161 => /lib/x86_64-linux-gnu/libirs-export.so.161 (0x00007f5cb62e5000) libdns-export.so.1109 => /lib/x86_64-linux-gnu/libdns-export.so.1109 (0x00007f5cb60b0000) libisc-export.so.1105 => /lib/x86_64-linux-gnu/libisc-export.so.1105 (0x00007f5cb6039000) libisccfg-export.so.163 => /lib/x86_64-linux-gnu/libisccfg-export.so.163 (0x00007f5cb5df5000) ubuntu@dhcpd1:~$ dpkg -S /lib/x86_64-linux-gnu/libisc-export.so.1105 libisc-export1105:amd64: /lib/x86_64-linux-gnu/libisc-export.so.1105 ubuntu@dhcpd1:~$ apt-cache policy libisc-export1105 | grep -i ppa Installed: 1:9.11.16+dfsg-3~ppa1 Candidate: 1:9.11.16+dfsg-3~ppa1 *** 1:9.11.16+dfsg-3~ppa1 500 500 http://ppa.launchpad.net/niedbalski/1872188-dbg/ubuntu focal/main amd64 Packages -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to isc-dhcp in Ubuntu. https://bugs.launchpad.net/bugs/1872118 Title: DHCP Cluster crashes after a few hours Status in DHCP: New Status in bind9-libs package in Ubuntu: New Status in isc-dhcp package in Ubuntu: Confirmed Status in bind9-libs source package in Focal: New Status in isc-dhcp source package in Focal: New Status in bind9-libs source package in Groovy: New Status in isc-dhcp source package in Groovy: Confirmed Bug description: I have a pair of DHCP serevrs running in a cluster on ubuntu 20.04, All worked perfectly until recently, when they started stopping with code=killed, status=6/ABRT. This is being fixed by https://bugs.launchpad.net/bugs/1870729 However now one stops after a few hours with the following errors. One can stay on line but not both. Syslog shows Apr 10 17:20:15 dhcp-primary sh[6828]: ../../../../lib/isc/unix/socket.c:3361: INSIST(!sock->pending_send) failed, back trace Apr 10 17:20:15 dhcp-primary sh[6828]: #0 0x7fbe78702a4a in ?? Apr 10 17:20:15 dhcp-primary sh[6828]: #1 0x7fbe78702980 in ?? Apr 10 17:20:15 dhcp-primary sh[6828]: #2 0x7fbe7873e7e1 in ?? Apr 10 17:20:15 dhcp-primary sh[6828]: #3 0x7fbe784e5609 in ?? Apr 10 17:20:15 dhcp-primary sh[6828]: #4 0x7fbe78621103 in ?? nothing in kern.log apport.log shows ERROR: apport (pid 6850) Fri Apr 10 17:20:15 2020: called for pid 6828, signal 6, core limit 0, dump mode 2 ERROR: apport (pid 6850) Fri Apr 10 17:20:15 2020: not creating core for pid with dump mode of 2 ERROR: apport (pid 6850) Fri Apr 10 17:20:15 2020: executable: /usr/sbin/dhcpd (command line "dhcpd -user dhcpd -group dhcpd -f -4 -pf /run/dhcp-server/dhcpd.pid -cf /etc/dhcp/dhcpd.conf") ERROR: apport (pid 6850) Fri Apr 10 17:20:15 2020: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment ERROR: apport (pid 6850) Fri Apr 10 17:20:15 2020: wrote report /var/crash/_usr_sbin_dhcpd.0.crash /var/crash/_usr_sbin_dhcpd.0.crash shows ProblemType: Crash Architecture: amd64 CrashCounter: 1 Date: Fri Apr 10 17:20:15 2020 DistroRelease: Ubuntu 20.04 ExecutablePath: /usr/sbin/dhcpd ExecutableTimestamp: 1586210315 ProcCmdline: dhcpd -user dhcpd -group dhcpd -f -4 -pf /run/dhcp-server/dhcpd.pid -cf /etc/dhcp/dhcpd.conf ProcEnviron: Error: [Errno 13] Permission denied: 'environ' ProcMaps: Error: [Errno 13] Permission denied: 'maps' ProcStatus: Name: dhcpd Umask: 0022 State: D (disk sleep) Tgid: 6828 Ngid: 0 Pid: 6828 PPid: 1 TracerPid: 0 Uid: 113 113 113 113 Gid: 118 118 118 118 FDSize: 128 Groups: NStgid: 6828 NSpid: 6828 NSpgid: 6828 NSsid: 6828 VmPeak: 236244 kB VmSize: 170764 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 12064 kB VmRSS: 12064 kB RssAnon: 5940 kB RssFile: 6124 kB RssShmem: 0 kB VmData: 30792 kB VmStk: 132 kB VmExe: 592 kB VmLib: 5424 kB VmPTE: 76 kB VmSwap: 0 kB HugetlbPages: 0 kB CoreDumping: 1 THP_enabled: 1 Threads: 4 SigQ: 0/7609 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000000001000 SigCgt: 0000000180000000 CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: 0000003fffffffff CapAmb: 0000000000000000 NoNewPrivs: 0 Seccomp: 0 Speculation_Store_Bypass: thread vulnerable Cpus_allowed: 3 Cpus_allowed_list: 0-1 Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,0000 0000,00000000,00000000,00000001 Mems_allowed_list: 0 voluntary_ctxt_switches: 111 nonvoluntary_ctxt_switches: 144 Signal: 6 Uname: Linux 5.4.0-21-generic x86_64 UserGroups: To manage notifications about this bug go to: https://bugs.launchpad.net/dhcp/+bug/1872118/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp