Hi,
I have an amd64 system that I am using as a router/firewall for my home network running OpenBSD 6.6-stable with the latest syspatches. I have been running it without problems in an IPv4-only configuration. Recently, I decided I wanted to experiment with running a dual stack IPv4 and IPv6 network. I think I have successfully added IPv6 to my configuration which seemed to be functioning correctly to the best of my knowledge, but I have started to encounter kernel panics that crash to the ddb debugger after about 12 hours of uptime. I believe the panics might be specifically occurring when the DHCPv6 lease expires or renews on my external interface and the IPv6 routing table is updated. The panics go away when I reverted back to my IPv4-only configuration so I believe the panics are specific to IPv6. I have been troubleshooting and researching and for the past week or so and I thought I would check to see if some more experienced OpenBSD users might have some advice on how to proceed with troubleshooting. At the end of this email, I have included the relevant section from /var/run/dmesg.boot related to one instance of the panic. It includes the boot messages, panic message, trace, and the output of ps I ran in ddb after the panic. I did find a similar bug report in the mail archive for openbsd-bugs that looks like it was unfortunately not resolved: https://marc.info/?l=openbsd-bugs&m=152587044611044&w=2 Some more info about my configuration that I think may be relevant: My ISP uses DHCPv6 prefix delegation to distribute IPv6 addresses so I used dhcpcd from ports installed using pkg_add. I used rad for router advertisements on my internal interfaces and I am using SLAAC to auto-configure IPv6 addresses. My system has 6 Intel NICs, em[0-5], and I also have 4 VLANs vlan[0-3] on em5 that I am using for my wireless AP. em0 is the external interface connected to my cable modem. em[1-5] are my internal interfaces. em[1-3] are each connected to separate computers, em4 is unconnected, and em5 and vlan[0-3] are connected to my wireless AP. The only other observation I had that might be related to the panic is regarding the "ndp info overwritten" and "cannot forward src" lines in my dmesg. The "ndp info overwritten" lines correspond to interfaces that were connected to devices/hosts that powered on at the time of the panic while I think that the "cannot forward src" messages correspond to em2 and em3 which were connected to computers that were powered off at the time of the panic, but were previously powered on to test if they were receiving IPv6 addresses. I am wondering if the computers being powered off at the time of the DHCPv6 lease update is triggering the panic and I think I might test this by only enabling IPv6 on em5 and vlan[0-3] which are connected to my wireless AP is always on. I would be happy to provide any other relevant info that I may have omitted. Thanks for reading. Ben ###### OpenBSD 6.6 (GENERIC.MP) #3: Thu Nov 21 03:20:01 MST 2019 r...@syspatch-66-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENER IC.MP real mem = 4192743424 (3998MB) avail mem = 4052963328 (3865MB) mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x8d317000 (85 entries) bios0: vendor American Megatrends Inc. version "5.12" date 07/08/2019 bios0: Protectli FW6 acpi0 at bios0: ACPI 6.1 acpi0: sleep states S0 S5 acpi0: tables DSDT FACP APIC FPDT MCFG SSDT FIDT SSDT HPET SSDT SSDT UEFI SSDT LPIT WSMT SSDT SSDT SSDT SSDT DBGP DBG2 BGRT DMAR ASF! acpi0: wakeup devices PS2K(S0) PS2M(S0) RP09(S0) PXSX(S0) RP10(S0) PXSX(S0) RP11(S0) PXSX(S0) RP12(S0) PXSX(S0) RP13(S0) PXSX(S0) RP01(S0) PXSX(S0) RP02(S0) PXSX(S0) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Celeron(R) CPU 3865U @ 1.80GHz, 1696.58 MHz, 06-8e-09 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36, CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT, DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,M OVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3 DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,SMEP,ERMS,INVPCID,MPX,RDSEED,SMA P,CLFLUSHOPT,PT,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSA VEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 24MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1.1.1, IBE cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Celeron(R) CPU 3865U @ 1.80GHz, 1696.06 MHz, 06-8e-09 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36, CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT, DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,M OVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3 DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,SMEP,ERMS,INVPCID,MPX,RDSEED,SMA P,CLFLUSHOPT,PT,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSA VEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN cpu1: 256KB 64b/line 8-way L2 cache cpu1: smt 0, core 1, package 0 ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 120 pins acpimcfg0 at acpi0 acpimcfg0: addr 0xe0000000, bus 0-255 acpihpet0 at acpi0: 23999999 Hz acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus -1 (PEG0) acpiprt2 at acpi0: bus -1 (PEG1) acpiprt3 at acpi0: bus -1 (PEG2) acpiprt4 at acpi0: bus -1 (RP09) acpiprt5 at acpi0: bus -1 (RP10) acpiprt6 at acpi0: bus -1 (RP11) acpiprt7 at acpi0: bus -1 (RP12) acpiprt8 at acpi0: bus -1 (RP13) acpiprt9 at acpi0: bus 1 (RP01) acpiprt10 at acpi0: bus 2 (RP02) acpiprt11 at acpi0: bus 3 (RP03) acpiprt12 at acpi0: bus 4 (RP04) acpiprt13 at acpi0: bus 5 (RP05) acpiprt14 at acpi0: bus 6 (RP06) acpiprt15 at acpi0: bus -1 (RP07) acpiprt16 at acpi0: bus -1 (RP08) acpiprt17 at acpi0: bus -1 (RP17) acpiprt18 at acpi0: bus -1 (RP18) acpiprt19 at acpi0: bus -1 (RP19) acpiprt20 at acpi0: bus -1 (RP20) acpiprt21 at acpi0: bus -1 (RP21) acpiprt22 at acpi0: bus -1 (RP22) acpiprt23 at acpi0: bus -1 (RP23) acpiprt24 at acpi0: bus -1 (RP24) acpiprt25 at acpi0: bus -1 (RP14) acpiprt26 at acpi0: bus -1 (RP15) acpiprt27 at acpi0: bus -1 (RP16) acpiec0 at acpi0: not present acpicpu0 at acpi0: C3(200@1034 mwait.1@0x60), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS acpicpu1 at acpi0: C3(200@1034 mwait.1@0x60), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS acpipwrres0 at acpi0: WRST acpipwrres1 at acpi0: WRST acpipwrres2 at acpi0: WRST acpipwrres3 at acpi0: WRST acpipwrres4 at acpi0: WRST acpipwrres5 at acpi0: WRST acpipwrres6 at acpi0: WRST acpipwrres7 at acpi0: WRST acpipwrres8 at acpi0: WRST acpipwrres9 at acpi0: WRST acpipwrres10 at acpi0: WRST acpipwrres11 at acpi0: WRST acpipwrres12 at acpi0: WRST acpipwrres13 at acpi0: WRST acpipwrres14 at acpi0: WRST acpipwrres15 at acpi0: WRST acpipwrres16 at acpi0: WRST acpipwrres17 at acpi0: WRST acpipwrres18 at acpi0: WRST acpipwrres19 at acpi0: WRST acpipwrres20 at acpi0: FN00, resource for FAN0 acpipwrres21 at acpi0: FN01, resource for FAN1 acpipwrres22 at acpi0: FN02, resource for FAN2 acpipwrres23 at acpi0: FN03, resource for FAN3 acpipwrres24 at acpi0: FN04, resource for FAN4 acpitz0 at acpi0: critical temperature is 119 degC acpitz1 at acpi0: critical temperature is 119 degC acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001 acpicmos0 at acpi0 acpibtn0 at acpi0: SLPB "INT33A1" at acpi0 not configured acpibtn1 at acpi0: PWRB "PNP0C0B" at acpi0 not configured "PNP0C0B" at acpi0 not configured "PNP0C0B" at acpi0 not configured "PNP0C0B" at acpi0 not configured "PNP0C0B" at acpi0 not configured acpivideo0 at acpi0: GFX0 acpivout0 at acpivideo0: DD1F cpu0: using VERW MDS workaround (except on vmm entry) cpu0: Enhanced SpeedStep 1696 MHz: speeds: 1801, 1800, 1700, 1600, 1500, 1400, 1300, 1200, 1100, 1000, 900, 800, 700, 600, 500, 400 MHz pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel Core 7G Host" rev 0x02 inteldrm0 at pci0 dev 2 function 0 "Intel HD Graphics 610" rev 0x02 drm0 at inteldrm0 inteldrm0: msi xhci0 at pci0 dev 20 function 0 "Intel 100 Series xHCI" rev 0x21: msi, xHCI 1.0 usb0 at xhci0: USB revision 3.0 uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev 3.00/1.00 addr 1 "Intel 100 Series MEI" rev 0x21 at pci0 dev 22 function 0 not configured ahci0 at pci0 dev 23 function 0 "Intel 100 Series AHCI" rev 0x21: msi, AHCI 1.3.1 ahci0: port 0: 6.0Gb/s ahci0: PHY offline on port 1 scsibus1 at ahci0: 32 targets sd0 at scsibus1 targ 0 lun 0: <ATA, TS64GMSA230S, R081> naa.57c3548176158942 sd0: 61057MB, 512 bytes/sector, 125045424 sectors, thin ppb0 at pci0 dev 28 function 0 "Intel 100 Series PCIE" rev 0xf1: msi pci1 at ppb0 bus 1 em0 at pci1 dev 0 function 0 "Intel 82583V" rev 0x00: msi, address 00:e0:67:18:19:42 ppb1 at pci0 dev 28 function 1 "Intel 100 Series PCIE" rev 0xf1: msi pci2 at ppb1 bus 2 em1 at pci2 dev 0 function 0 "Intel 82583V" rev 0x00: msi, address 00:e0:67:18:19:43 ppb2 at pci0 dev 28 function 2 "Intel 100 Series PCIE" rev 0xf1: msi pci3 at ppb2 bus 3 em2 at pci3 dev 0 function 0 "Intel 82583V" rev 0x00: msi, address 00:e0:67:18:19:44 ppb3 at pci0 dev 28 function 3 "Intel 100 Series PCIE" rev 0xf1: msi pci4 at ppb3 bus 4 em3 at pci4 dev 0 function 0 "Intel 82583V" rev 0x00: msi, address 00:e0:67:18:19:45 ppb4 at pci0 dev 28 function 4 "Intel 100 Series PCIE" rev 0xf1: msi pci5 at ppb4 bus 5 em4 at pci5 dev 0 function 0 "Intel 82583V" rev 0x00: msi, address 00:e0:67:18:19:46 ppb5 at pci0 dev 28 function 5 "Intel 100 Series PCIE" rev 0xf1: msi pci6 at ppb5 bus 6 em5 at pci6 dev 0 function 0 "Intel 82583V" rev 0x00: msi, address 00:e0:67:18:19:47 pcib0 at pci0 dev 31 function 0 "Intel 200 Series LPC" rev 0x21 "Intel 100 Series PMC" rev 0x21 at pci0 dev 31 function 2 not configured ichiic0 at pci0 dev 31 function 4 "Intel 100 Series SMBus" rev 0x21: apic 2 int 16 iic0 at ichiic0 isa0 at pcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo com0: console pckbc0 at isa0 port 0x60/5 irq 1 irq 12 pckbd0 at pckbc0 (kbd slot) wskbd0 at pckbd0 mux 1 pcppi0 at isa0 port 0x61 spkr0 at pcppi0 it0 at isa0 port 0x2e/2: IT8772F rev 1, EC port 0xa40 vmm0 at mainbus0: VMX/EPT efifb at mainbus0 not configured vscsi0 at root scsibus2 at vscsi0: 256 targets softraid0 at root scsibus3 at softraid0: 256 targets root on sd0a (fc4381a8357fd1b0.a) swap on sd0b dump on sd0b WARNING: / was not properly unmounted inteldrm0: 1024x768, 32bpp wsdisplay0 at inteldrm0 mux 1 pckbd_enable: command error wsdisplay0: screen 0-5 added (std, vt100 emulation) ndp info overwritten for fe80:c::2e0:67ff:fe18:1947 by 00:e0:67:18:19:45 on vlan3 ndp info overwritten for fe80:6::2e0:67ff:fe18:1947 by 00:e0:67:18:19:45 on em5 ndp info overwritten for fe80:b::2e0:67ff:fe18:1947 by 00:e0:67:18:19:45 on vlan2 ndp info overwritten for fe80:a::2e0:67ff:fe18:1947 by 00:e0:67:18:19:45 on vlan1 ndp info overwritten for fe80:9::2e0:67ff:fe18:1947 by 00:e0:67:18:19:45 on vlan0 ndp info overwritten for fe80:2::2e0:67ff:fe18:1943 by 00:e0:67:18:19:45 on em1 cannot forward src fe80:a::5660:9ff:fedd:cca8, dst 2001:4860:4860::6464, nxt 17, rcvif 10, outif 1 cannot forward src fe80:a::5660:9ff:fedd:cca8, dst 2001:4860:4860::6464, nxt 17, rcvif 10, outif 1 panic: kernel diagnostic assertion "!ISSET(rt->rt_flags, RTF_LOCAL)" failed: file "/usr/src/sys/netinet6/nd6.c", line 727 Stopped at db_enter+0x10: popq %rbp TID PID UID PRFLAGS PFLAGS CPU COMMAND * 12172 56020 0 0x14000 0x40000200 0K softclock db_enter() at db_enter+0x10 panic() at panic+0x128 __assert(ffffffff81ca1bdf,ffffffff81cac63d,2d7,ffffffff81c877a4) at __assert+0x2b nd6_free(fffffd815eadb710) at nd6_free+0x12f nd6_llinfo_timer(fffffd815eadb710) at nd6_llinfo_timer+0x19e nd6_timer(ffffffff81ff7d50) at nd6_timer+0x64 softclock_thread(ffff8000ffffe000) at softclock_thread+0xfb end trace frame: 0x0, count: 8 https://www.openbsd.org/ddb.html describes the minimum info required in bug reports. Insufficient info makes it difficult to find and fix bugs. ddb{0}> ps PID TID PPID UID S FLAGS WAIT COMMAND 71101 370210 81160 94 3 0x100092 kqread rad 36663 233150 81160 94 3 0x100092 kqread rad 81160 34274 1 0 3 0x100080 kqread rad 8607 115278 1 0 3 0x100083 ttyin getty 92088 384510 1 0 3 0x100083 ttyin getty 99883 499812 1 0 3 0x100083 ttyin getty 26478 124491 1 0 3 0x100083 ttyin getty 69363 154114 1 0 3 0x100083 ttyin getty 86060 293146 1 0 3 0x100083 ttyin getty 37066 4797 1 0 3 0x100098 poll cron 54502 514469 1 0 3 0x80 kqread apmd 77136 522236 1 0 3 0x80 kqread dhcpcd 65621 515668 1 629 3 0x90 poll avahi-daemon 6344 170264 1 572 3 0x90 poll dbus-daemon 72501 157373 76033 95 3 0x100092 kqread smtpd 46629 436753 76033 103 3 0x100092 kqread smtpd 56768 98112 76033 95 3 0x100092 kqread smtpd 8204 106837 76033 95 3 0x100092 kqread smtpd 62327 125479 76033 95 3 0x100092 kqread smtpd 34358 470816 76033 95 3 0x100092 kqread smtpd 76033 7544 1 0 3 0x100080 kqread smtpd 61841 392491 1 77 3 0x100090 poll dhcpd 8345 285284 1 0 3 0x80 select sshd 66534 22989 1 0 3 0x100080 poll ntpd 6894 217071 30254 83 3 0x100092 poll ntpd 30254 338166 1 83 3 0x100092 poll ntpd 82791 382379 1 53 3 0x90 kqread unbound 57461 8270 54858 74 3 0x100092 bpf pflogd 54858 522464 1 0 3 0x80 netio pflogd 26929 242333 57577 73 3 0x100090 kqread syslogd 57577 493915 1 0 3 0x100082 netio syslogd 60733 317951 1 77 3 0x100090 poll dhclient 3305 473407 1 0 3 0x80 poll dhclient 51834 334199 35318 115 3 0x100092 kqread slaacd 47429 282699 35318 115 3 0x100092 kqread slaacd 35318 517851 1 0 3 0x100080 kqread slaacd 99085 420269 0 0 3 0x14280 schto i915/signal:6 33798 484525 0 0 3 0x14280 schto i915/signal:2 25861 170194 0 0 3 0x14280 schto i915/signal:1 76675 337056 0 0 3 0x14280 schto i915/signal:0 94463 253283 0 0 3 0x14200 bored i915-userptr-acq 89440 298121 0 0 3 0x14200 bored i915_modeset 43276 342976 0 0 3 0x14200 bored i915-dp 33409 16254 0 0 3 0x14200 bored i915 33089 228516 0 0 3 0x14200 pgzero zerothread 86710 349325 0 0 3 0x14200 aiodoned aiodoned 50212 84906 0 0 3 0x14200 syncer update 83345 246040 0 0 3 0x14200 cleaner cleaner 97789 396968 0 0 3 0x14200 reaper reaper 17359 153556 0 0 3 0x14200 pgdaemon pagedaemon 9948 234580 0 0 3 0x14200 bored crynlk 81030 332928 0 0 3 0x14200 bored crypto 51819 191268 0 0 3 0x14200 usbtsk usbtask 32163 214038 0 0 3 0x14200 usbatsk usbatsk 59871 370750 0 0 3 0x14200 bored drmtskl 80945 102084 0 0 3 0x14200 bored drmlwq 85907 159437 0 0 3 0x14200 bored drmlwq 3277 260402 0 0 3 0x14200 bored drmlwq 40689 82336 0 0 3 0x14200 bored drmubwq 71293 514738 0 0 3 0x14200 bored drmubwq 15998 134631 0 0 3 0x14200 bored drmubwq 71439 227948 0 0 3 0x14200 bored drmubwq 65395 170820 0 0 3 0x14200 bored drmwq 12666 420844 0 0 3 0x14200 bored drmwq 91628 417254 0 0 3 0x14200 bored drmwq 50050 283919 0 0 3 0x14200 bored drmwq 48545 214698 0 0 3 0x40014200 acpi0 acpi0 89352 316255 0 0 7 0x40014200 idle1 45452 317644 0 0 3 0x14200 bored sensors 42779 145809 0 0 3 0x14200 bored softnet 82778 378842 0 0 3 0x14200 bored systqmp 86243 366424 0 0 3 0x14200 bored systq *56020 12172 0 0 7 0x40014200 softclock 86139 86337 0 0 3 0x40014200 idle0 38007 388788 0 0 3 0x14200 bored smr 1 518657 0 0 3 0x82 wait init 0 0 -1 0 3 0x10200 scheduler swapper
