I'm moving this to tech as I think the issue is best discussed there but I might be wrong.
So, I did some slow progress on that hang. A recent snapshots now panics in ufs code with "ufs_rename: lost dir entry". That's the second one at ufs_vnops.c:1084. This means that the source of the rename was a directory and got either renamed or rm'ed concurrently. As a side note I didn't fully bisect it but I think it's rev 1.108 of uvm_vnode.c by anton@ that fixed the deadlock leaving us with the panic. Looking around I found that in 2001 dillon@fbsd removed the panic in [1] noting that those races did actually exist and were easy enough to trigger that it was a security problem (not sure about that solution to be honest as it hides the problem but w/e). Nowadays several os (linux and netbsd to name a few) choose the "easy" way and just use a mount-wide lock at a higher level (in sys_rename) in order to catch all those races in the locking mess that rename is. It was brought up recently in #dragonflybsd and it lead to the same fix in [2]. I've been meaning to try that approach on obsd but I lack the time so I'm dumping it all here in the hope that it will help someone! As a side note, the same test on a tmpfs mount point survives a full run. Here is the panic for reference: login: panic: ufs_rename: lost dir entry Stopped at db_enter+0x10: popq %rbp TID PID UID PRFLAGS PFLAGS CPU COMMAND 307330 34101 1000 0x3 0x4000000 0 a.out 302047 34101 1000 0x3 0x4000000 1 a.out *319683 34101 1000 0x3 0x4000000 2K a.out 104491 34684 74 0x100012 0x480 3 pflogd db_enter() at db_enter+0x10 panic(ffffffff81decc5f) at panic+0x12a ufs_rename(ffff800022b0afb8) at ufs_rename+0xcb6 VOP_RENAME(fffffd8118858d00,fffffd8118855a98,ffff800022b0b198,fffffd8118858410, fffffd811885e010,ffff800022b0b0c8) at VOP_RENAME+0x69 dorenameat(ffff800022aec5c0,ffffff9c,58cd3c8dd30,ffffff9c,58cd3c8dd10) at doren ameat+0x1f6 syscall(ffff800022b0b320) at syscall+0x389 Xsyscall() at Xsyscall+0x128 end of kernel end trace frame: 0x58cd3c8dcf0, count: 8 [1]: https://github.com/freebsd/freebsd/commit/dbebfe18a15ceac757fc126dd8caa59045ec9e47 [2]: https://gitweb.dragonflybsd.org/dragonfly.git/commit/ad1212685b9caac64c086a2363d15842dff21fd8 On 06 Oct 10:16, Mathieu - wrote: > >Synopsis: Concurrent rename/mkdir/rmdir on a hierarchy hangs the system > >Category: Kernel bug > >Environment: > System : OpenBSD 6.8 > Details : OpenBSD 6.8-current (GENERIC.MP) #7: Mon Oct 5 22:45:13 > CEST 2020 > > ptr@spear.local:/usr/src/sys/arch/amd64/compile/GENERIC.MP > > Architecture: OpenBSD.amd64 > Machine : amd64 > >Description: > Running dirconc.c [1] leads the system to hang under a minute, no ssh, no > serial it's a full freeze. Upon reboot the fs is obviously corrupted and needs > to go to single user mode to fix up. > > This is on a recent snapshot (dmesg indicates a compiled kernel cause I added > some debug code to no help). > > Tweaking a bit dirconc to lower the number of concurrent ops doesn't help. > Using > procs instead of threads also leads to the same hang. > > I intended to pinpoint it a bit better (that's why it's in a VM), but the hard > hang makes it really difficult and I lack the time and knowledge in that area > so I'm reporting it (I'll keep trying to debug it though and keep you > updated). > >How-To-Repeat: > Get dirconc.c [1], build it (cc dirconc.c -lpthread for the threaded > version), and > run dirconc against an empty directory. After a while the system will hang. > >Fix: > No known fix. > > [1]: https://www.netbsd.org/~riastradh/tmp/dirconc.c > > dmesg: > OpenBSD 6.8-current (GENERIC.MP) #7: Mon Oct 5 22:45:13 CEST 2020 > ptr@spear.local:/usr/src/sys/arch/amd64/compile/GENERIC.MP > real mem = 4278124544 (4079MB) > avail mem = 4133400576 (3941MB) > random: good seed from bootblocks > mpath0 at root > scsibus0 at mpath0: 256 targets > mainbus0 at root > bios0 at mainbus0: SMBIOS rev. 2.5 @ 0xe1000 (10 entries) > bios0: vendor innotek GmbH version "VirtualBox" date 12/01/2006 > bios0: innotek GmbH VirtualBox > acpi0 at bios0: ACPI 4.0 > acpi0: sleep states S0 S5 > acpi0: tables DSDT FACP APIC SSDT > acpi0: wakeup devices > acpitimer0 at acpi0: 3579545 Hz, 32 bits > acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat > cpu0 at mainbus0: apid 0 (boot processor) > cpu0: Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz, 2403.41 MHz, 06-4e-03 > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,RDRAND,NXE,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,AVX2,INVPCID,RDSEED,CLFLUSHOPT,MELTDOWN > cpu0: 256KB 64b/line 8-way L2 cache > cpu0: smt 0, core 0, package 0 > mtrr: CPU supports MTRRs but not enabled by BIOS > cpu0: apic clock running at 1000MHz > cpu1 at mainbus0: apid 1 (application processor) > cpu1: Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz, 2400.10 MHz, 06-4e-03 > cpu1: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,RDRAND,NXE,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,AVX2,INVPCID,RDSEED,CLFLUSHOPT,MELTDOWN > cpu1: 256KB 64b/line 8-way L2 cache > cpu1: smt 0, core 1, package 0 > cpu2 at mainbus0: apid 2 (application processor) > cpu2: Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz, 2400.09 MHz, 06-4e-03 > cpu2: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,RDRAND,NXE,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,AVX2,INVPCID,RDSEED,CLFLUSHOPT,MELTDOWN > cpu2: 256KB 64b/line 8-way L2 cache > cpu2: disabling user TSC (skew=161) > cpu2: smt 0, core 2, package 0 > cpu3 at mainbus0: apid 3 (application processor) > cpu3: Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz, 2400.09 MHz, 06-4e-03 > cpu3: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,RDRAND,NXE,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,AVX2,INVPCID,RDSEED,CLFLUSHOPT,MELTDOWN > cpu3: 256KB 64b/line 8-way L2 cache > cpu3: disabling user TSC (skew=282) > cpu3: smt 0, core 3, package 0 > ioapic0 at mainbus0: apid 4 pa 0xfec00000, version 20, 24 pins, remapped > acpiprt0 at acpi0: bus 0 (PCI0) > acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001 > acpibat0 at acpi0: BAT0 model "1" serial 0 type VBOX oem "innotek" > acpiac0 at acpi0: AC unit online > acpicpu0 at acpi0: C1(@1 halt!) > acpicpu1 at acpi0: C1(@1 halt!) > acpicpu2 at acpi0: C1(@1 halt!) > acpicpu3 at acpi0: C1(@1 halt!) > acpivideo0 at acpi0: GFX0 > cpu0: using Skylake AVX MDS workaround > pci0 at mainbus0 bus 0 > pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02 > pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00 > vga1 at pci0 dev 2 function 0 "InnoTek VirtualBox Graphics Adapter" rev 0x00 > wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) > wsdisplay0: screen 1-5 added (80x25, vt100 emulation) > em0 at pci0 dev 3 function 0 "Intel 82540EM" rev 0x02: apic 4 int 19, address > 08:00:27:ea:09:c8 > "InnoTek VirtualBox Guest Service" rev 0x00 at pci0 dev 4 function 0 not > configured > auich0 at pci0 dev 5 function 0 "Intel 82801AA AC97" rev 0x01: apic 4 int 21, > ICH > ac97: codec id 0x83847600 (SigmaTel STAC9700) > audio0 at auich0 > ohci0 at pci0 dev 6 function 0 "Apple Intrepid USB" rev 0x00: apic 4 int 22, > version 1.0 > piixpm0 at pci0 dev 7 function 0 "Intel 82371AB Power" rev 0x08: apic 4 int 23 > iic0 at piixpm0 > ehci0 at pci0 dev 11 function 0 "Intel 82801FB USB" rev 0x00: apic 4 int 19 > usb0 at ehci0: USB revision 2.0 > uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 > addr 1 > ahci0 at pci0 dev 13 function 0 "Intel 82801HBM AHCI" rev 0x02: apic 4 int > 21, AHCI 1.1 > ahci0: device on port 0 didn't come ready, TFD: 0x131<ERR> > ahci0: port 0: 3.0Gb/s > ahci0: device on port 1 didn't come ready, TFD: 0x171<ERR> > ahci0: port 1: 3.0Gb/s > scsibus1 at ahci0: 32 targets > cd0 at scsibus1 targ 0 lun 0: <VBOX, CD-ROM, 1.0> removable > sd0 at scsibus1 targ 1 lun 0: <ATA, VBOX HARDDISK, 1.0> > t10.ATA_VBOX_HARDDISK_VB954da61f-210850d3_ > sd0: 81920MB, 512 bytes/sector, 167772160 sectors > isa0 at pcib0 > isadma0 at isa0 > com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo > com0: probed fifo depth: 1 bytes > pckbc0 at isa0 port 0x60/5 irq 1 irq 12 > pckbd0 at pckbc0 (kbd slot) > wskbd0 at pckbd0: console keyboard, using wsdisplay0 > pms0 at pckbc0 (aux slot) > wsmouse0 at pms0 mux 0 > pcppi0 at isa0 port 0x61 > spkr0 at pcppi0 > usb1 at ohci0: USB revision 1.0 > uhub1 at usb1 configuration 1 interface 0 "Apple OHCI root hub" rev 1.00/1.00 > addr 1 > dt: 439 probes > vscsi0 at root > scsibus2 at vscsi0: 256 targets > softraid0 at root > scsibus3 at softraid0: 256 targets > root on sd0a (e08db1e8dc1d78ca.a) swap on sd0b dump on sd0b > > usbdevs: > Controller /dev/usb0: > addr 01: 8086:0000 Intel, EHCI root hub > high speed, self powered, config 1, rev 1.00 > driver: uhub0 > Controller /dev/usb1: > addr 01: 106b:0000 Apple, OHCI root hub > full speed, self powered, config 1, rev 1.00 > driver: uhub1 >