I'm moving this to tech as I think the issue is best discussed there but I might
be wrong.
So, I did some slow progress on that hang. A recent snapshots now panics in ufs
code with "ufs_rename: lost dir entry". That's the second one at
ufs_vnops.c:1084. This means that the source of the rename was a directory and
got either renamed or rm'ed concurrently.
As a side note I didn't fully bisect it but I think it's rev 1.108 of
uvm_vnode.c by anton@ that fixed the deadlock leaving us with the panic.
Looking around I found that in 2001 dillon@fbsd removed the panic in [1] noting
that those races did actually exist and were easy enough to trigger that it was
a security problem (not sure about that solution to be honest as it hides the
problem but w/e).
Nowadays several os (linux and netbsd to name a few) choose the "easy" way and
just use a mount-wide lock at a higher level (in sys_rename) in order to catch
all those races in the locking mess that rename is. It was brought up recently
in #dragonflybsd and it lead to the same fix in [2].
I've been meaning to try that approach on obsd but I lack the time so I'm
dumping it all here in the hope that it will help someone!
As a side note, the same test on a tmpfs mount point survives a full run.
Here is the panic for reference:
login: panic: ufs_rename: lost dir entry
Stopped at db_enter+0x10: popq %rbp
TID PID UID PRFLAGS PFLAGS CPU COMMAND
307330 34101 1000 0x3 0x4000000 0 a.out
302047 34101 1000 0x3 0x4000000 1 a.out
*319683 34101 1000 0x3 0x4000000 2K a.out
104491 34684 74 0x100012 0x480 3 pflogd
db_enter() at db_enter+0x10
panic(ffffffff81decc5f) at panic+0x12a
ufs_rename(ffff800022b0afb8) at ufs_rename+0xcb6
VOP_RENAME(fffffd8118858d00,fffffd8118855a98,ffff800022b0b198,fffffd8118858410,
fffffd811885e010,ffff800022b0b0c8) at VOP_RENAME+0x69
dorenameat(ffff800022aec5c0,ffffff9c,58cd3c8dd30,ffffff9c,58cd3c8dd10) at doren
ameat+0x1f6
syscall(ffff800022b0b320) at syscall+0x389
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x58cd3c8dcf0, count: 8
[1]:
https://github.com/freebsd/freebsd/commit/dbebfe18a15ceac757fc126dd8caa59045ec9e47
[2]:
https://gitweb.dragonflybsd.org/dragonfly.git/commit/ad1212685b9caac64c086a2363d15842dff21fd8
On 06 Oct 10:16, Mathieu - wrote:
> >Synopsis: Concurrent rename/mkdir/rmdir on a hierarchy hangs the system
> >Category: Kernel bug
> >Environment:
> System : OpenBSD 6.8
> Details : OpenBSD 6.8-current (GENERIC.MP) #7: Mon Oct 5 22:45:13
> CEST 2020
>
> ptr@spear.local:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>
> Architecture: OpenBSD.amd64
> Machine : amd64
> >Description:
> Running dirconc.c [1] leads the system to hang under a minute, no ssh, no
> serial it's a full freeze. Upon reboot the fs is obviously corrupted and needs
> to go to single user mode to fix up.
>
> This is on a recent snapshot (dmesg indicates a compiled kernel cause I added
> some debug code to no help).
>
> Tweaking a bit dirconc to lower the number of concurrent ops doesn't help.
> Using
> procs instead of threads also leads to the same hang.
>
> I intended to pinpoint it a bit better (that's why it's in a VM), but the hard
> hang makes it really difficult and I lack the time and knowledge in that area
> so I'm reporting it (I'll keep trying to debug it though and keep you
> updated).
> >How-To-Repeat:
> Get dirconc.c [1], build it (cc dirconc.c -lpthread for the threaded
> version), and
> run dirconc against an empty directory. After a while the system will hang.
> >Fix:
> No known fix.
>
> [1]: https://www.netbsd.org/~riastradh/tmp/dirconc.c
>
> dmesg:
> OpenBSD 6.8-current (GENERIC.MP) #7: Mon Oct 5 22:45:13 CEST 2020
> ptr@spear.local:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 4278124544 (4079MB)
> avail mem = 4133400576 (3941MB)
> random: good seed from bootblocks
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.5 @ 0xe1000 (10 entries)
> bios0: vendor innotek GmbH version "VirtualBox" date 12/01/2006
> bios0: innotek GmbH VirtualBox
> acpi0 at bios0: ACPI 4.0
> acpi0: sleep states S0 S5
> acpi0: tables DSDT FACP APIC SSDT
> acpi0: wakeup devices
> acpitimer0 at acpi0: 3579545 Hz, 32 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz, 2403.41 MHz, 06-4e-03
> cpu0:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,RDRAND,NXE,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,AVX2,INVPCID,RDSEED,CLFLUSHOPT,MELTDOWN
> cpu0: 256KB 64b/line 8-way L2 cache
> cpu0: smt 0, core 0, package 0
> mtrr: CPU supports MTRRs but not enabled by BIOS
> cpu0: apic clock running at 1000MHz
> cpu1 at mainbus0: apid 1 (application processor)
> cpu1: Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz, 2400.10 MHz, 06-4e-03
> cpu1:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,RDRAND,NXE,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,AVX2,INVPCID,RDSEED,CLFLUSHOPT,MELTDOWN
> cpu1: 256KB 64b/line 8-way L2 cache
> cpu1: smt 0, core 1, package 0
> cpu2 at mainbus0: apid 2 (application processor)
> cpu2: Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz, 2400.09 MHz, 06-4e-03
> cpu2:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,RDRAND,NXE,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,AVX2,INVPCID,RDSEED,CLFLUSHOPT,MELTDOWN
> cpu2: 256KB 64b/line 8-way L2 cache
> cpu2: disabling user TSC (skew=161)
> cpu2: smt 0, core 2, package 0
> cpu3 at mainbus0: apid 3 (application processor)
> cpu3: Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz, 2400.09 MHz, 06-4e-03
> cpu3:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,RDRAND,NXE,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,AVX2,INVPCID,RDSEED,CLFLUSHOPT,MELTDOWN
> cpu3: 256KB 64b/line 8-way L2 cache
> cpu3: disabling user TSC (skew=282)
> cpu3: smt 0, core 3, package 0
> ioapic0 at mainbus0: apid 4 pa 0xfec00000, version 20, 24 pins, remapped
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
> acpibat0 at acpi0: BAT0 model "1" serial 0 type VBOX oem "innotek"
> acpiac0 at acpi0: AC unit online
> acpicpu0 at acpi0: C1(@1 halt!)
> acpicpu1 at acpi0: C1(@1 halt!)
> acpicpu2 at acpi0: C1(@1 halt!)
> acpicpu3 at acpi0: C1(@1 halt!)
> acpivideo0 at acpi0: GFX0
> cpu0: using Skylake AVX MDS workaround
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
> pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
> vga1 at pci0 dev 2 function 0 "InnoTek VirtualBox Graphics Adapter" rev 0x00
> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> em0 at pci0 dev 3 function 0 "Intel 82540EM" rev 0x02: apic 4 int 19, address
> 08:00:27:ea:09:c8
> "InnoTek VirtualBox Guest Service" rev 0x00 at pci0 dev 4 function 0 not
> configured
> auich0 at pci0 dev 5 function 0 "Intel 82801AA AC97" rev 0x01: apic 4 int 21,
> ICH
> ac97: codec id 0x83847600 (SigmaTel STAC9700)
> audio0 at auich0
> ohci0 at pci0 dev 6 function 0 "Apple Intrepid USB" rev 0x00: apic 4 int 22,
> version 1.0
> piixpm0 at pci0 dev 7 function 0 "Intel 82371AB Power" rev 0x08: apic 4 int 23
> iic0 at piixpm0
> ehci0 at pci0 dev 11 function 0 "Intel 82801FB USB" rev 0x00: apic 4 int 19
> usb0 at ehci0: USB revision 2.0
> uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00
> addr 1
> ahci0 at pci0 dev 13 function 0 "Intel 82801HBM AHCI" rev 0x02: apic 4 int
> 21, AHCI 1.1
> ahci0: device on port 0 didn't come ready, TFD: 0x131<ERR>
> ahci0: port 0: 3.0Gb/s
> ahci0: device on port 1 didn't come ready, TFD: 0x171<ERR>
> ahci0: port 1: 3.0Gb/s
> scsibus1 at ahci0: 32 targets
> cd0 at scsibus1 targ 0 lun 0: <VBOX, CD-ROM, 1.0> removable
> sd0 at scsibus1 targ 1 lun 0: <ATA, VBOX HARDDISK, 1.0>
> t10.ATA_VBOX_HARDDISK_VB954da61f-210850d3_
> sd0: 81920MB, 512 bytes/sector, 167772160 sectors
> isa0 at pcib0
> isadma0 at isa0
> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> com0: probed fifo depth: 1 bytes
> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
> pckbd0 at pckbc0 (kbd slot)
> wskbd0 at pckbd0: console keyboard, using wsdisplay0
> pms0 at pckbc0 (aux slot)
> wsmouse0 at pms0 mux 0
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> usb1 at ohci0: USB revision 1.0
> uhub1 at usb1 configuration 1 interface 0 "Apple OHCI root hub" rev 1.00/1.00
> addr 1
> dt: 439 probes
> vscsi0 at root
> scsibus2 at vscsi0: 256 targets
> softraid0 at root
> scsibus3 at softraid0: 256 targets
> root on sd0a (e08db1e8dc1d78ca.a) swap on sd0b dump on sd0b
>
> usbdevs:
> Controller /dev/usb0:
> addr 01: 8086:0000 Intel, EHCI root hub
> high speed, self powered, config 1, rev 1.00
> driver: uhub0
> Controller /dev/usb1:
> addr 01: 106b:0000 Apple, OHCI root hub
> full speed, self powered, config 1, rev 1.00
> driver: uhub1
>
