Hi Jean-Pierre,
That one worked! I also tried it with 1200 and 12000 renamed files,
both with no errors.
Thanks for your info about ldd. There was indeed a
/x86_64-linux-gnu/libntfs-3g.so.833.0.0 (which I renamed just to be sure), but
I see that the test version is statically linked ("not a dynamic executable").
I also just discovered that ntfsresize -fiv does a filesystem consistency
check. Do you think it is good enough to trust to detect any issues, e.g.
after a power-failure? (This backup drive is likely only ever to be used with
Linux and rsync.)
Thanks,
-- Chris
PS. The use case for this drive is a daily Linux backup but also a
"grab-and-go" backup for fleeing a forest-fire where we will likely have to
rely on borrowed computers for a couple weeks. NTFS is the only
Linux-writable filesystem I found that can preserve POSIX ownerships and
permissions and is readable without any additional software on Mac/PC/Linux.
Thank you for your dedicated work on ntfs-3g!
On Tue Aug 4 2020, at 5:33 AM, Jean-Pierre André <[email protected]>
wrote:
> Hi,
>
> Chris Roehrig wrote on 8/4/20 2:46 AM:
>> Hi Jean-Pierre,
>> I tried it, but it made no difference: baddir still gives exactly the
>> same CHKDSK error.
>
> Well, I found out my previous tries do not fix all cases,
> as the behavior depends on the order of the renamings.
>
> Please find attached a third try (discard the previous
> variants before applying this one).
>
>> I'm not 100% sure if I'm using the patched version: To mount the
>> partition I'm running the patched ntfs-3g directly from its build directory:
>> ./ntfs-3g_ntfsprogs-2017.3.23AR.5/src/ntfs-3g /dev/sdb1 /mnt
>> Will that be completely stand-alone and isolated from Ubuntu's built-in
>> ntfs-3g or could it e.g. pull in an old AR.3 shared lib or something? I
>> couldn't find any existing ntfs-3g stuff in /lib or /usr/lib. I'm not
>> familiar with how FUSE stuff works.
>
> This a fifteen year old bug in a file unchanged for ten years,
> so the patch applies to any non-archaic version.
> The bug has been left unnoticed because it only occurs in very
> specific conditions, and it does not harm. It is also probably
> self-healing under some condition.
>
> You have apparently correctly applied the recipe for
> testing without installing (and without disturbing your
> installed configuration). However, if you do "make install"
> the generated code will not be put according to the rules
> for your distribution.
>
> You will have to declare the target directories as arguments
> to configure, something like (not tested, depends on your
> distribution).
>
> ./configure --bindir=/bin --sbindir=/sbin --libdir=/lib \
> --mandir=/usr/share/man
>
> You can determine where libntfs-3g is installed by executing
> ldd $(which ntfs-3g)
>
> Jean-Pierre
>
>> Thanks,
>> -- Chris
>>
>>
>> On Mon Aug 3 2020, at 12:16 AM, Jean-Pierre André
>> <[email protected]> wrote:
>>
>>> Hi,
>>>
>>> Please find attached a hopefully better variant of the fix.
>>>
>>> Jean-Pierre
>>>
>>> Jean-Pierre André wrote on 8/1/20 12:35 PM:
>>>> Hi,
>>>>
>>>> Thank you for your report including a test pointing out
>>>> the issue.
>>>>
>>>> Attached is a patch expected to fix it.
>>>>
>>>> Please test and report.
>>>>
>>>> Jean-Pierre
>>>>
>>>> Chris Roehrig wrote on 7/30/20 1:01 AM:
>>>>> I'm trying to get my Linux-based NTFS backup drive to pass a CHKDSK and
>>>>> came upon this curious situation where CHKDSK finds errors.
>>>>>
>>>>> It seems to be some issue with how ntfs-3g modifies a directory index
>>>>> when renaming many files.
>>>>>
>>>>> The CHKDSK error always seems to be of the form:
>>>>> Stage 2: Examining file name linkage ...
>>>>> The first free byte, 0xc0, and bytes available, 0x150, for root index
>>>>> $I30 in file 0x40 are not equal.
>>>>>
>>>>> I've attached a python script (mkbaddir.py) that creates two (apparently)
>>>>> identical directories, one of which reliably causes this CHKDSK error;
>>>>> the other doesn't.
>>>>>
>>>>> How to demonstrate:
>>>>> - Format an NTFS partition or thumbdrive using Windows or mkfs.ntfs.
>>>>> - Mount the partition on a Linux system.
>>>>> I used Mint 20 with ntfs-3g 2017.3.23AR.3 integrated FUSE 28 and
>>>>> python 3.8.2.
>>>>> - Chdir to the new NTFS partition and run the script:
>>>>> /tmp/mkbaddir.py # creates 'baddir' in current dir.
>>>>> /tmp/mkbaddir.py -G # creates 'gooddir' in current dir.
>>>>> diff -r baddir gooddir # no difference
>>>>> du -sB1 baddir gooddir # same size (128K)
>>>>> - Boot into Windows (10 v1903) and run (from a terminal) chkdsk X:
>>>>> (where X: is the NTFS drive).
>>>>> - This will say:
>>>>> "Errors found. CHKDSK cannot continue in read-only mode."
>>>>> - Delete baddir (I used cygwin's rm -rf), and run chkdsk X: again.
>>>>> - This will now have no errors.
>>>>>
>>>>> My guess at what's happening:
>>>>> The script creates a directory of 410 empty files and then renames them
>>>>> with slightly larger names, which as I understand leaves a bunch of
>>>>> unused nodes in the b-tree. The -G option just renames the 410 known
>>>>> files; without the -G option, it uses os.walk() to traverse the directory
>>>>> which I'm guessing leaves the b-tree in a slightly different state with
>>>>> even more unused nodes.
>>>>>
>>>>> The 410 was chosen by trial-and-error so that some internal threshhold is
>>>>> just exceeded by the baddir but not by the gooddir. With more than 410
>>>>> (using the -c option; say -c 500), both baddir and gooddir will cause
>>>>> CHKDSK errors.
>>>>>
>>>>> If I run the script on Windows/cygwin (Python 3.6.9) to create the
>>>>> folders, it does not give any CHKDSK errors even with many more files.
>>>>>
>>>>> So there seems to be some issue with how ntfs-3g modifies the b-tree when
>>>>> renaming many files that is causing CHKDSK to complain.
>>>>>
>>>>>
>>>>> I encountered this issue when trying to get my Linux-based NTFS backup
>>>>> drive to consistently pass a CHKDSK. I use a script to first rename
>>>>> POSIX names to valid windows names, replacing '?' with '@@3F', etc so I
>>>>> can reverse the renaming afterwards. I have some website mirror folders
>>>>> with many files of the form:
>>>>> details.asp?id=xxxxx&key=val
>>>>> which gave rise to this issue. (In the mkbaddir script I use only
>>>>> alphanumeric names to be clear this is not an illegal char issue).
>>>>>
>>>>>
>>> <shorten_index-v2.patch>
>>
>
> <shorten_index-v3.patch>
_______________________________________________
ntfs-3g-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ntfs-3g-devel
