Hi List. I've stumbled across a use-after-free condition in the NTFS filesystem driver of OpenBSD Current (HEAD, plus it's present in STABLE, and appears to go back a few releases - I repro'ed with 5.1). I have seen it triggered accidentally through normal use to cause a panic.
The bug revolves around the ntm_ntnodeq, a 16-entry LRU which is a member of the ntfsmount structure. Elements are added to this by ntfs_loadntnode, but are destroyed when removed from the LRU, without any reference checking - so another process may have its ntnodes freed unexectedly, causing the kernel to use memory which has been released. The following happens: 1) Process 'foo' starts loading some data from the disk. It loads an ntnode A, adds it to the ntnodeq (see ntfs_subr.c:353, ntfs_loadntnode - 'TAILQ_INSERT_HEAD(&ntmp->ntm_ntnodeq, ip, i_loaded);'), and then tsleep's waiting for a subsequent disk read to complete. 2) Process 'bar' reads sixteen (or more) ntnodes from disk. The ntnodeq gets cycled, and ntnode A, loaded by foo previously, is freed - see ntfs_subr.c:278, ntfs_loadntnode - 'ntfs_freentvattr(vap);') 3) Process 'foo' then attempts to use its ntnode A, which points to freed memory. The bad access isn't very obvious (although it does happen, as verified by hand via a debugger) and usually goes unnoticed without any crash or panic. If you zero out the memory before freeing it, it is much easier to reproduce. Edit ntfs_freentvattr, found in ntfs_subr.c: + memset(vap, 0, sizeof(struct ntvattr)); // (add this line) free(vap, M_NTFSNTVATTR, 0); // (this line already present) An easy way to repro the issue is to create ~100 small files, plus one heavily fragmented file. Reading the fragmented file will cause tsleep's between ntnode reads, while the small files can be read repeatedly in the background. Since there are only 100 of them, they should all fit in the cache and can be served quickly, causing them to exhaust the LRU. Here's a script that'll create such a disk layout. Run it in the ntfs mountpoint: ------ #!/bin/sh # Fill disk will small files i=0 while ( true ); do dd if=/dev/zero count=2 of=small_$i if [ $? -ne 0 ]; do break fi i=$(( $i + 1 )) done # Delete some non-sequential ones for toDel in 0 $i 100; do rm small_$i done # Create the big file dd if=/dev/zero of=/bigfile ------ You should be able to repro by: 0) Applying the memset above (or by being lucky) 1) Optionally reboot, which will make sure bigfile isn't cached and will up your chances of a successful repo 2) Mounting this FS 3) Repeatedly reading 100 of the small files, causing the LRU to churn: while ( true ); do cat /mnt/ntfs/small_?? > /dev/null; done & 4) Reading the large file. By the time the OS has finished reading the large file, the access of the small files will cause the first clusters of the large file to be freed. cat /mnt/ntfs/big > /dev/null The result will probably be a panic, since the memory area being used by the kernel is free'd. Manual analysis verifies that the memory being accessed is, indeed, freed. A sample panic (full logs below): ntfs_readntvattr_plain: POSSIBLE RUN ERROR ntfs_readattr_plain: ntfs_readntvattr_plain failed: o: 1179648, s: 65536 ntfs_readattr_plain: attrib: 0 - 0 panic: ntfs_ntrele: ino: 159 usecount: -1 Stopped at Debugger+0x9: leave RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS PANIC! DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION! ddb> trace Debugger() at Debugger+0x9 panic() at panic+0xfe ntfs_ntrele() at ntfs_ntrele+0x28 ntfs_ntvattrrele() at ntfs_ntvattrrele+0x11 ntfs_readattr() at ntfs_readattr+0x135 ntfs_read() at ntfs_read+0x69 VOP_READ() at VOP_READ+0x3f vn_read() at vn_read+0xa1 dofilereadv() at dofilereadv+0x1c4 sys_read() at sys_read+0x89 syscall() at syscall+0x192 --- syscall (number 3) --- end trace frame: 0x0, count: -11 My repros were done on a VMWare Workstation VM, configured to have 1 CPU core, 256MB RAM, and to have a 64bit guest. I have verified this issue on physical hardware also. I'm guessing that the call to ntfs_freentvattr is in error, and it should be a call to ntfs_ntput, but I'm unfamilliar with OpenBSD and don't know if this is correct (especially in regard to locking semantics). Hope this is helpful. Don't hesitate to contact me if there's any more info you need, or just to keep me updated. If you end up putting out an advisory, I'd be very grateful if you could credit me on it - it might help me find a job! - Aliz Hammond Full ddb log: rebooting... >> OpenBSD/amd64 BOOT 3.28 boot> booting hd0a:/bsd: entry point at 0x1000160 [7205c766, 34000004, 24448b12, 9060a304] [ using 932984 bytes of bsd ELF symbol table ] Copyright (c) 1982, 1986, 1989, 1991, 1993 The Regents of the University of California. All rights reserved. Copyright (c) 1995-2015 OpenBSD. All rights reserved. http://www.OpenBSD.org OpenBSD 5.8-beta (GENERIC) #0: Mon Jun 22 04:41:57 BST 2015 root@foo.localdomain:/usr/src/src/sys/arch/amd64/compile/GENERIC real mem = 251592704 (239MB) avail mem = 240250880 (229MB) mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xe0010 (98 entries) bios0: vendor Phoenix Technologies LTD version "6.00" date 12/31/2009 bios0: VMware, Inc. VMware Virtual Platform acpi0 at bios0: rev 2 acpi0: sleep states S0 S1 S4 S5 acpi0: tables DSDT FACP BOOT APIC MCFG SRAT acpi0: wakeup devices PCI0(S3) USB_(S1) P2P0(S3) S1F0(S3) S2F0(S3) S3F0(S3) S4F0(S3) S5F0(S3) S6F0(S3) S7F0(S3) S8F0(S3) S9F0(S3) Z00P(S3) Z00Q(S3) Z00R(S3) Z00S(S3) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU L5420 @ 2.50GHz, 2500.10 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,SSE3,SSSE3,CX16,SSE4.1,HV,NXE,LONG,LAHF,PERF,SENSOR cpu0: 6MB 64b/line 16-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 65MHz ioapic0 at mainbus0: apid 1 pa 0xfec00000, version 11, 24 pins acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255 acpiprt0 at acpi0: bus 0 (PCI0) acpicpu0 at acpi0 acpibat0 at acpi0: BAT1 not present acpibat1 at acpi0: BAT2 not present acpiac0 at acpi0: AC unit online acpibtn0 at acpi0: SLPB acpibtn1 at acpi0: LID_ vmt0 at mainbus0 pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel 82443BX AGP" rev 0x01 ppb0 at pci0 dev 1 function 0 "Intel 82443BX AGP" rev 0x01 pci1 at ppb0 bus 1 pcib0 at pci0 dev 7 function 0 "Intel 82371AB PIIX4 ISA" rev 0x08 pciide0 at pci0 dev 7 function 1 "Intel 82371AB IDE" rev 0x01: DMA, channel 0 configured to compatibility, channel 1 configured to compatibility pciide0: channel 0 disabled (no drives) atapiscsi0 at pciide0 channel 1 drive 0 scsibus1 at atapiscsi0: 2 targets cd0 at scsibus1 targ 0 lun 0: <NECVMWar, VMware IDE CDR10, 1.00> ATAPI 5/cdrom removable cd0(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 2 piixpm0 at pci0 dev 7 function 3 "Intel 82371AB Power" rev 0x08: SMBus disabled "VMware VMCI" rev 0x10 at pci0 dev 7 function 7 not configured vga1 at pci0 dev 15 function 0 "VMware SVGA II" rev 0x00 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) mpi0 at pci0 dev 16 function 0 "Symbios Logic 53c1030" rev 0x01: apic 1 int 17 mpi0: 0, firmware 1.3.41.32 scsibus2 at mpi0: 16 targets, initiator 7 sd0 at scsibus2 targ 0 lun 0: <VMware,, VMware Virtual S, 1.0> SCSI2 0/direct fixed sd0: 20480MB, 512 bytes/sector, 41943040 sectors mpi0: target 0 Sync at 160MHz width 16bit offset 127 QAS 1 DT 1 IU 1 ppb1 at pci0 dev 17 function 0 "VMware PCI" rev 0x02 pci2 at ppb1 bus 2 em0 at pci2 dev 0 function 0 "Intel 82545EM" rev 0x01: apic 1 int 18, address 00:0c:29:ef:91:73 ppb2 at pci0 dev 21 function 0 "VMware PCIE" rev 0x01 pci3 at ppb2 bus 3 ppb3 at pci0 dev 21 function 1 "VMware PCIE" rev 0x01 pci4 at ppb3 bus 4 ppb4 at pci0 dev 21 function 2 "VMware PCIE" rev 0x01 pci5 at ppb4 bus 5 ppb5 at pci0 dev 21 function 3 "VMware PCIE" rev 0x01 pci6 at ppb5 bus 6 ppb6 at pci0 dev 21 function 4 "VMware PCIE" rev 0x01 pci7 at ppb6 bus 7 ppb7 at pci0 dev 21 function 5 "VMware PCIE" rev 0x01 pci8 at ppb7 bus 8 ppb8 at pci0 dev 21 function 6 "VMware PCIE" rev 0x01 pci9 at ppb8 bus 9 ppb9 at pci0 dev 21 function 7 "VMware PCIE" rev 0x01 pci10 at ppb9 bus 10 ppb10 at pci0 dev 22 function 0 "VMware PCIE" rev 0x01 pci11 at ppb10 bus 11 ppb11 at pci0 dev 22 function 1 "VMware PCIE" rev 0x01 pci12 at ppb11 bus 12 ppb12 at pci0 dev 22 function 2 "VMware PCIE" rev 0x01 pci13 at ppb12 bus 13 ppb13 at pci0 dev 22 function 3 "VMware PCIE" rev 0x01 pci14 at ppb13 bus 14 ppb14 at pci0 dev 22 function 4 "VMware PCIE" rev 0x01 pci15 at ppb14 bus 15 ppb15 at pci0 dev 22 function 5 "VMware PCIE" rev 0x01 pci16 at ppb15 bus 16 ppb16 at pci0 dev 22 function 6 "VMware PCIE" rev 0x01 pci17 at ppb16 bus 17 ppb17 at pci0 dev 22 function 7 "VMware PCIE" rev 0x01 pci18 at ppb17 bus 18 ppb18 at pci0 dev 23 function 0 "VMware PCIE" rev 0x01 pci19 at ppb18 bus 19 ppb19 at pci0 dev 23 function 1 "VMware PCIE" rev 0x01 pci20 at ppb19 bus 20 ppb20 at pci0 dev 23 function 2 "VMware PCIE" rev 0x01 pci21 at ppb20 bus 21 ppb21 at pci0 dev 23 function 3 "VMware PCIE" rev 0x01 pci22 at ppb21 bus 22 ppb22 at pci0 dev 23 function 4 "VMware PCIE" rev 0x01 pci23 at ppb22 bus 23 ppb23 at pci0 dev 23 function 5 "VMware PCIE" rev 0x01 pci24 at ppb23 bus 24 ppb24 at pci0 dev 23 function 6 "VMware PCIE" rev 0x01 pci25 at ppb24 bus 25 ppb25 at pci0 dev 23 function 7 "VMware PCIE" rev 0x01 pci26 at ppb25 bus 26 ppb26 at pci0 dev 24 function 0 "VMware PCIE" rev 0x01 pci27 at ppb26 bus 27 ppb27 at pci0 dev 24 function 1 "VMware PCIE" rev 0x01 pci28 at ppb27 bus 28 ppb28 at pci0 dev 24 function 2 "VMware PCIE" rev 0x01 pci29 at ppb28 bus 29 ppb29 at pci0 dev 24 function 3 "VMware PCIE" rev 0x01 pci30 at ppb29 bus 30 ppb30 at pci0 dev 24 function 4 "VMware PCIE" rev 0x01 pci31 at ppb30 bus 31 ppb31 at pci0 dev 24 function 5 "VMware PCIE" rev 0x01 pci32 at ppb31 bus 32 ppb32 at pci0 dev 24 function 6 "VMware PCIE" rev 0x01 pci33 at ppb32 bus 33 ppb33 at pci0 dev 24 function 7 "VMware PCIE" rev 0x01 pci34 at ppb33 bus 34 isa0 at pcib0 isadma0 at isa0 fdc0 at isa0 port 0x3f0/6 irq 6 drq 2 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo com0: console com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo pckbc0 at isa0 port 0x60/5 irq 1 irq 12 pckbd0 at pckbc0 (kbd slot) wskbd0 at pckbd0: console keyboard, using wsdisplay0 pms0 at pckbc0 (aux slot) wsmouse0 at pms0 mux 0 pcppi0 at isa0 port 0x61 spkr0 at pcppi0 lpt0 at isa0 port 0x378/4 irq 7 vscsi0 at root scsibus3 at vscsi0: 256 targets softraid0 at root scsibus4 at softraid0: 256 targets root on sd0a (a5b771226ef737bb.a) swap on sd0b dump on sd0b WARNING: / was not properly unmounted Automatic boot in progress: starting file system checks. /dev/sd0a (a5b771226ef737bb.a): 1763 files, 84111 used, 289936 free (32 frags, 36238 blocks, 0.0% fragmentation) /dev/sd0a (a5b771226ef737bb.a): MARKING FILE SYSTEM CLEAN /dev/sd0k (a5b771226ef737bb.k): 10 files, 9 used, 3666686 free (14 frags, 458334 blocks, 0.0% fragmentation) /dev/sd0k (a5b771226ef737bb.k): MARKING FILE SYSTEM CLEAN /dev/sd0d (a5b771226ef737bb.d): 6 files, 5 used, 596570 free (42 frags, 74566 blocks, 0.0% fragmentation) /dev/sd0d (a5b771226ef737bb.d): MARKING FILE SYSTEM CLEAN /dev/sd0f (a5b771226ef737bb.f): 11744 files, 189302 used, 598985 free (209 frags, 74847 blocks, 0.0% fragmentation) /dev/sd0f (a5b771226ef737bb.f): MARKING FILE SYSTEM CLEAN /dev/sd0g (a5b771226ef737bb.g): 9183 files, 97521 used, 360782 free (1862 frags, 44865 blocks, 0.4% fragmentation) /dev/sd0g (a5b771226ef737bb.g): MARKING FILE SYSTEM CLEAN /dev/sd0h (a5b771226ef737bb.h): 111 files, 115 used, 1701804 free (68 frags, 212717 blocks, 0.0% fragmentation) /dev/sd0h (a5b771226ef737bb.h): MARKING FILE SYSTEM CLEAN /dev/sd0j (a5b771226ef737bb.j): 1 files, 1 used, 921678 free (14 frags, 115208 blocks, 0.0% fragmentation) /dev/sd0j (a5b771226ef737bb.j): MARKING FILE SYSTEM CLEAN /dev/sd0i (a5b771226ef737bb.i): 14268 files, 84608 used, 564431 free (183 frags, 70531 blocks, 0.0% fragmentation) /dev/sd0i (a5b771226ef737bb.i): MARKING FILE SYSTEM CLEAN /dev/sd0e (a5b771226ef737bb.e): 303 files, 3138 used, 908661 free (53 frags, 113576 blocks, 0.0% fragmentation) /dev/sd0e (a5b771226ef737bb.e): MARKING FILE SYSTEM CLEAN setting tty flags pf enabled machdep.allowaperture: 0 -> 2 starting network DHCPREQUEST on em0 to 255.255.255.255 DHCPACK from 192.168.174.254 (00:50:56:ea:e4:a7) bound to 192.168.174.139 -- renewal in 900 seconds. starting early daemons: syslogd pflogd(failed). starting RPC daemons:. savecore: no core dump checking quotas: done. clearing /tmp kern.securelevel: 0 -> 1 creating runtime link editor directory cache. preserving editor files. starting network daemons: smtpd sndiod. starting local daemons: cron. Mon Jun 22 13:51:45 BST 2015 ntfs_readntvattr_plain: POSSIBLE RUN ERROR ntfs_readattr_plain: ntfs_readntvattr_plain failed: o: 1179648, s: 65536 ntfs_readattr_plain: attrib: 0 - 0 panic: ntfs_ntrele: ino: 159 usecount: -1 Stopped at Debugger+0x9: leave RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS PANIC! DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION! ddb> trace Debugger() at Debugger+0x9 panic() at panic+0xfe ntfs_ntrele() at ntfs_ntrele+0x28 ntfs_ntvattrrele() at ntfs_ntvattrrele+0x11 ntfs_readattr() at ntfs_readattr+0x135 ntfs_read() at ntfs_read+0x69 VOP_READ() at VOP_READ+0x3f vn_read() at vn_read+0xa1 dofilereadv() at dofilereadv+0x1c4 sys_read() at sys_read+0x89 syscall() at syscall+0x192 --- syscall (number 3) --- end trace frame: 0x0, count: -11 0x9696706d3a: ddb> ps PID PPID PGRP UID S FLAGS WAIT COMMAND 23411 1151 1151 0 2 0x3 cat *16948 20938 16948 0 7 0x3 cat 1151 20938 1151 0 3 0x8b pause sh 15738 1 15738 0 3 0x83 ttyin getty 17533 1 17533 0 3 0x83 ttyin getty 1665 1 1665 0 3 0x83 ttyin getty 31938 1 31938 0 3 0x83 ttyin getty 20938 1 20938 0 3 0x8b pause ksh 30503 1 30503 0 3 0x80 poll cron 21467 1 21467 99 3 0x90 poll sndiod 11796 16973 16973 95 3 0x90 kqread smtpd 6 16973 16973 95 3 0x90 kqread smtpd 23807 16973 16973 95 3 0x90 kqread smtpd 12206 16973 16973 95 3 0x90 kqread smtpd 28488 16973 16973 95 3 0x90 kqread smtpd 30803 16973 16973 103 3 0x90 kqread smtpd 16973 1 16973 0 3 0x80 kqread smtpd 20314 3277 3277 73 2 0x90 syslogd 3277 1 3277 0 3 0x80 netio syslogd 31874 1 31874 77 3 0x90 poll dhclient 5710 1 5710 0 3 0x80 poll dhclient 12169 0 0 0 2 0x14200 zerothread 17046 0 0 0 3 0x14200 aiodoned aiodoned 7135 0 0 0 3 0x14200 syncer update 4702 0 0 0 3 0x14200 cleaner cleaner 26531 0 0 0 3 0x14200 reaper reaper 5615 0 0 0 3 0x14200 pgdaemon pagedaemon 786 0 0 0 3 0x14200 bored crypto 21707 0 0 0 3 0x14200 pftm pfpurge 13257 0 0 0 3 0x40014200 acpi0 acpi0 21965 0 0 0 3 0x14200 bored sensors 5400 0 0 0 3 0x14200 bored softnet 7731 0 0 0 3 0x14200 bored systqmp 27942 0 0 0 3 0x14200 bored systq 495 0 0 0 3 0x40014200 idle0 1 0 1 0 3 0x82 wait init 0 -1 0 0 3 0x10200 scheduler swapper
