On Sat, 19 Oct 2013, David Vasek wrote:
> On Thu, 17 Oct 2013, David Vasek wrote:
> > On Fri, 11 Oct 2013, Joel Sing wrote:
> >> On Thu, 10 Oct 2013, Manuel Giraud wrote:
> >>> Hi,
> >>>
> >>> I have a ntfs partition with rather large (about 3GB) files on it. When
> >>> I copy these files on a ffs partition they are corrupted. When I try to
> >>> checksum them directly from the ntfs partition the checksum is not
> >>> correct (compared to the same file on a fat32 partition copied with
> >>> Windows).
> >>>
> >>> I tried this (with same behaviour) on i386 5.3 release and on i386 last
> >>> week current. I'm willing to do some testing to fix this issue but
> >>> don't really know where to start.
> >>
> >> See if you can isolate the smallest possible reproducable test case. If
> >> you create a 3GB file with known content (e.g. the same byte repeated),
> >> does the
> >> same issue occur? If so, how small do you need to go before the problem
> >> goes
> >> away? Also, what operating system (and version) was used to write the
> >> files to the NTFS volume?
> >
> > Hello, I encountered the same issue. Anything over the 2 GB limit is
> > wrong. I mean, first exactly 2 GB of the file are read correctly,
> > following that I get wrong data till the end of the file. It is
> > reproducible with any file over 2 GB in size so far. Smells like int
> > somewhere... I get the same wrong data with any release since at least
> > 5.0, didn't test anything older, but I bet it is the same.
> >
> > The filesystem is a Windows XP NTFS system disk, 32-bit, the files were
> > copied there with explorer.exe.
>
> Some additional notes and findings:
>
> (1)
> The data I receive after first 2 GB are not part of the file, the data is
> from another file (from the same directory, if that fact could be
> important). The data is taken in uninterrupted sequence and the starting
> offset of that sequence is way less than 2 GB in the other file where the
> data belong.
>
> (2)
> While reading past 2 GB in larger blocks gives me just wrong data, reading
> in smaller blocks (2kB and less) gives me kernel panic in KASSERT
> immediately when I read past the 2 GB limit. It is 100% reproducible with
> any file larger than 2 GB so far.
Thanks for taking the time to dig into this further and provide some
reproducable test cases.
There were two problems - the first was an off_t (64-bit integer) to integer
conversion, which meant that attempting to read past a 2GB offset would have
become negative. The second issue was an unsigned 64-bit to unsigned 32-bit
truncation, which effectively wrapped the attribute data length at 4GB.
I've just committed fixes for both of these and I can now successfully
read/checksum a 6.5GB file on NTFS.
> # mount -r /dev/wd0i /mnt
>
> # ls -lo /mnt/DATA/ntfs_2gb_test.bin
> -rwxr-xr-x 1 root wheel - 3054813184 Oct 17 22:11
> /mnt/DATA/ntfs_2gb_test.bin
>
> # cat /mnt/DATA//ntfs_2gb_test.bin > /dev/null
>
> # dd if=/mnt/DATA/ntfs_2gb_test.bin bs=4k of=/dev/null
> 745804+0 records in
> 745804+0 records out
> 3054813184 bytes transferred in 108.518 secs (28150083 bytes/sec)
>
> # dd if=/mnt/DATA/ntfs_2gb_test.bin bs=2k count=1m of=/dev/null
> 1048576+0 records in
> 1048576+0 records out
> 2147483648 bytes transferred in 78.783 secs (27258052 bytes/sec)
>
> # dd if=/mnt/DATA/ntfs_2gb_test.bin bs=1k count=2m of=/dev/null
> 2097152+0 records in
> 2097152+0 records out
> 2147483648 bytes transferred in 81.210 secs (26443280 bytes/sec)
>
> # dd if=/mnt/DATA/ntfs_2gb_test.bin bs=4k skip=512k of=/dev/null
> 221516+0 records in
> 221516+0 records out
> 907329536 bytes transferred in 32.314 secs (28077667 bytes/sec)
>
> # dd if=/mnt/DATA/ntfs_2gb_test.bin bs=2k skip=1m of=/dev/null
> panic: kernel diagnostic assertion "cl == 1 && tocopy <= ntfs_cntob(1)"
> failed: file "../../../../ntfs/ntfs_subr.c", line 1556 Stopped at
> Debugger+0x4: popl %ebp
> RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS PANIC!
> DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION!
> ddb> trace
> Debugger(d08fdcbc,f544fb88,d08dc500,f544fb88,200) at Debugger+0x4
> panic(d08dc500,d085fc0e,d08dfe60,d08e00b0,614) at panic+0x5d
> __assert(d085fc0e,d08e00b0,614,d08dfe60,80000) at __assert+0x2e
> ntfs_readntvattr_plain(d1a2d200,d1a36200,d1a5bc00,80000800,0) at
> ntfs_readntvat tr_plain+0x2e6
> ntfs_readattr_plain(d1a2d200,d1a36200,80,0,80000800) at
> ntfs_readattr_plain+0x1 41
> ntfs_readattr(d1a2d200,d1a36200,80,0,80000800) at ntfs_readattr+0x156
> ntfs_read(f544fddc,d64e5140,d6522a60,f544fea0,0) at ntfs_read+0xa8
> VOP_READ(d6522a60,f544fea0,0,d6599000,d64e5140) at VOP_READ+0x35
> vn_read(d65290a8,d65290c4,f544fea0,d6599000,0) at vn_read+0xb5
> dofilereadv(d65365d4,3,d65290a8,f544ff08,1) at dofilereadv+0x13a
> sys_read(d65365d4,f544ff64,f544ff84,106,d653f100) at sys_read+0x89
> syscall() at syscall+0x227
> --- syscall (number 0) ---
> 0x2:
> ddb> ps
> PID PPID PGRP UID S FLAGS WAIT COMMAND
> *19967 9961 19967 0 7 0 dd
> 9961 1 9961 0 3 0x88 pause sh
> 14 0 0 0 3 0x100200 aiodoned aiodoned
> 13 0 0 0 3 0x100200 syncer update
> 12 0 0 0 3 0x100200 cleaner cleaner
> 11 0 0 0 3 0x100200 reaper reaper
> 10 0 0 0 3 0x100200 pgdaemon pagedaemon
> 9 0 0 0 3 0x100200 bored crypto
> 8 0 0 0 3 0x100200 pftm pfpurge
> 7 0 0 0 3 0x100200 usbtsk usbtask
> 6 0 0 0 3 0x100200 usbatsk usbatsk
> 5 0 0 0 3 0x100200 acpi0 acpi0
> 4 0 0 0 3 0x100200 bored syswq
> 3 0 0 0 3 0x40100200 idle0
> 2 0 0 0 3 0x100200 kmalloc kmthread
> 1 0 1 0 3 0x80 wait init
> 0 -1 0 0 3 0x200 scheduler swapper
> ddb> show registers
> ds 0x10
> es 0x10
> fs 0x20
> gs 0
> edi 0xd08dc500 addrmask+0x2920
> esi 0x100
> ebp 0xf544fb3c
> ebx 0xf544fb88
> edx 0x1
> ecx 0xd09e6f94 kprintf_mutex
> eax 0x1
> eip 0xd057b3f4 Debugger+0x4
> cs 0x8
> eflags 0x202
> esp 0xf544fb3c
> ss 0x10
> Debugger+0x4: popl %ebp
> ddb> show panic
> kernel diagnostic assertion "cl == 1 && tocopy <= ntfs_cntob(1)" failed:
> file " ../../../../ntfs/ntfs_subr.c", line 1556
> ddb>
>
>
> A few other important bits:
>
> OpenBSD 5.3 (GENERIC) #50: Tue Mar 12 18:35:23 MDT 2013
> dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC
>
> # fdisk wd0
> Disk: wd0 geometry: 20673/240/63 [312581808 Sectors]
> Offset: 0 Signature: 0xAA55
> Starting Ending LBA Info:
> #: id C H S - C H S [ start: size ]
> ---------------------------------------------------------------------------
>---- *0: 07 0 1 1 - 20671 239 63 [ 63: 312560577 ]
> NTFS 1: 00 0 0 0 - 0 0 0 [ 0: 0 ]
> unused 2: 00 0 0 0 - 0 0 0 [ 0: 0 ]
> unused 3: 00 0 0 0 - 0 0 0 [ 0: 0 ]
> unused
>
> # disklabel wd0
> # /dev/rwd0c:
> type: ESDI
> disk: ESDI/IDE disk
> label: Hitachi HTS54161
> duid: 0000000000000000
> flags:
> bytes/sector: 512
> sectors/track: 63
> tracks/cylinder: 240
> sectors/cylinder: 15120
> cylinders: 20673
> total sectors: 312581808
> boundstart: 0
> boundend: 312581808
> drivedata: 0
>
> 16 partitions:
> # size offset fstype [fsize bsize cpg]
> c: 312581808 0 unused
> i: 312560577 63 NTFS
>
>
> Regards,
> David
--
"Action without study is fatal. Study without action is futile."
-- Mary Ritter Beard
