du: fts and vfat
It appears that fts uses some stat data which doesn't work as expected
for fat32 filesystems.
|$ time nice ./src/du -sh /mnt/1/ 2>&1 |sed -e 2q
|1328
|./src/du: `/mnt/1/BACKUP 12-8-05/Fran\'s folder': No such file or directory
|real1m11.352s
|
|$ time nice ./src/du -sh /mnt/1/ 2>&1 |sed -e 2q
|1328
|./src/du: `/mnt/1/BACKUP 12-8-05/moved downloads': No such file or directory
|real1m17.916s
Here "1328" is my debug line in coreutils fts.c:
|if (p->fts_statp->st_dev != sb.st_dev
||| p->fts_statp->st_ino != sb.st_ino) {
|fprintf(stderr, "%d\n", __LINE__);
|__set_errno (ENOENT); /* disinformation */
|ret = -1;
|goto bail;
|}
Since the output changes on each invocation of the program, I'm of the
impression this is a bug in the filesystem driver (an apparent result
of generating data to fill the structure fields to appease the
traditional system calls when the filesystem doesn't natively use
those values).
The same problem occurs even when the FS is remounted readonly, but
disappears if the FTS_NOCHDIR option is set. Here's a diff fragment
of strace -v demonstrating the d_ino field differing between
invocations.
-getdents64(4, {{d_ino=645569, d_off=32, d_type=DT_DIR, d_reclen=24,
d_name="."} {d_ino=185378, d_off=64, d_type=DT_DIR, d_reclen=24, d_name=".."}
{d_ino=648015, d_off=32768, d_type=DT_DIR, d_reclen=24, d_name="Ram"}}, 32768)
= 72
+getdents64(4, {{d_ino=645569, d_off=32, d_type=DT_DIR, d_reclen=24,
d_name="."} {d_ino=185378, d_off=64, d_type=DT_DIR, d_reclen=24, d_name=".."}
{d_ino=648014, d_off=32768, d_type=DT_DIR, d_reclen=24, d_name="Ram"}}, 32768)
= 72
Justin
___
Bug-coreutils mailing list
Bug-coreutils@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: du: fts and vfat
Justin Pryzby <[EMAIL PROTECTED]> wrote: > It appears that fts uses some stat data which doesn't work as expected > for fat32 filesystems. > ... > Since the output changes on each invocation of the program, I'm of the > impression this is a bug in the filesystem driver (an apparent result > of generating data to fill the structure fields to appease the > traditional system calls when the filesystem doesn't natively use > those values). > > The same problem occurs even when the FS is remounted readonly, but > disappears if the FTS_NOCHDIR option is set. Here's a diff fragment > of strace -v demonstrating the d_ino field differing between > invocations. Thanks for the report. You're right. This is a problem in your environment, not with du/fts. If the driver changes the dev/inode of an open file, all bets are off. Why are you using FTS_NOCHDIR anyway? no openat support? What version of du are you using? For things like this, it's good to mention what kernel and libc version you're using -- for reference if/when others encounter the same problem later. ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: du: fts and vfat
On Sun, Nov 25, 2007 at 09:18:33AM +0100, Jim Meyering wrote: > Justin Pryzby <[EMAIL PROTECTED]> wrote: > > It appears that fts uses some stat data which doesn't work as expected > > for fat32 filesystems. > > > ... > > Since the output changes on each invocation of the program, I'm of the > > impression this is a bug in the filesystem driver (an apparent result > > of generating data to fill the structure fields to appease the > > traditional system calls when the filesystem doesn't natively use > > those values). > > > > The same problem occurs even when the FS is remounted readonly, but > > disappears if the FTS_NOCHDIR option is set. Here's a diff fragment > > of strace -v demonstrating the d_ino field differing between > > invocations. > > Thanks for the report. > > You're right. This is a problem in your environment, not with du/fts. > If the driver changes the dev/inode of an open file, all bets are off. > > Why are you using FTS_NOCHDIR anyway? no openat support? It was just for debugging. > What version of du are you using? 5.97-5.3 from debian etch. > For things like this, it's good to mention what kernel and libc > version you're using -- for reference if/when others encounter > the same problem later. linux-2.6.22-3-vserver-686, glibc-2.7-1 (although the problem was initially experienced with 2.6.X, I just reproduced it with this new version). However this time the problem occurred only on the first invocation. $ du -sh /mnt/1/music/jake/ du: `/mnt/1/music/jake/Music/Newly Ripped': No such file or directory du: `/mnt/1/music/jake/Music/Napster Directory': No such file or directory du: `/mnt/1/music/jake/New Folder': No such file or directory du: `/mnt/1/music/jake/Holiday CD 2006 takes': No such file or directory 177G/mnt/1/music/jake/ $ du -sh /mnt/1/music/jake/ 186G/mnt/1/music/jake/ I note that yesterday I added a 2nd 128MB ram chip to this machine. So my current hypothesis is that there's a hash table or such of inodes which was being overflowed. So this may just be an (expected?) consequence of accessing a large devices (with many files) on a machine with not much ram. Justin ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: du: fts and vfat
Justin Pryzby <[EMAIL PROTECTED]> wrote: > However this time the problem occurred only on the first invocation. > $ du -sh /mnt/1/music/jake/ > du: `/mnt/1/music/jake/Music/Newly Ripped': No such file or directory ... > 186G/mnt/1/music/jake/ > > I note that yesterday I added a 2nd 128MB ram chip to this machine. > So my current hypothesis is that there's a hash table or such of > inodes which was being overflowed. So this may just be an (expected?) > consequence of accessing a large devices (with many files) on a > machine with not much ram. Yes, it's expected, whenever you use a file system with imperfect-by-definition inode emulation. If you use coreutils-6.9 instead, this should be mitigated, since in the newer version of fts.c, the temporal locality of inode use and reuse is more cache friendly. ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: du: fts and vfat
Jim Meyering wrote: Yes, it's expected, whenever you use a file system with imperfect-by-definition inode emulation. AFAIR, the fat driver uses the starting cluster of the file as the inode number, so unless you defrag or something, it shouldn't change. ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: du: fts and vfat
Phillip Susi <[EMAIL PROTECTED]> wrote: > Jim Meyering wrote: >> Yes, it's expected, whenever you use a file system with >> imperfect-by-definition inode emulation. > > AFAIR, the fat driver uses the starting cluster of the file as the > inode number, so unless you defrag or something, it shouldn't change. Sorry, but no. If only it were so easy. The kernel maintains a limited-space cache for FAT inodes, so the inode of "." at the beginning of a traversal will not be the same as the inode reported for that same directory after du/fts has traversed a tree that's large enough. As I said, use a newer version of coreutils, and not in chdir-mode, and du works a lot better with FAT. ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: du: fts and vfat
Al Viro <[EMAIL PROTECTED]> wrote: > The real issue is that starting cluster won't work at all for empty > files. IOW, ftruncate() would change st_ino; no-go for all kinds of > obvious reasons. Now, one _could_ play games with separating inode > numbers of directories and non-directories, but that causes too much > PITA. What we have is a guaranteed "st_ino doesn't change for the > lifetime of in-core inode", without any promise of persistence beyond > that. Since way back - at least '99, IIRC. Here (in gnulib's fts.c), inodes are only ever compared for directories, so treating directories specially would help. While FAT may not be worth even a small PITA, some other types of file systems with similar limitations might be, e.g., FUSE-based ones. ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
Re: du: fts and vfat
On Mon, Nov 26, 2007 at 09:49:14PM +0100, Jim Meyering wrote: > Phillip Susi <[EMAIL PROTECTED]> wrote: > > Jim Meyering wrote: > >> Yes, it's expected, whenever you use a file system with > >> imperfect-by-definition inode emulation. > > > > AFAIR, the fat driver uses the starting cluster of the file as the > > inode number, so unless you defrag or something, it shouldn't change. > > Sorry, but no. > If only it were so easy. > The kernel maintains a limited-space cache for FAT inodes, > so the inode of "." at the beginning of a traversal will not > be the same as the inode reported for that same directory > after du/fts has traversed a tree that's large enough. The real issue is that starting cluster won't work at all for empty files. IOW, ftruncate() would change st_ino; no-go for all kinds of obvious reasons. Now, one _could_ play games with separating inode numbers of directories and non-directories, but that causes too much PITA. What we have is a guaranteed "st_ino doesn't change for the lifetime of in-core inode", without any promise of persistence beyond that. Since way back - at least '99, IIRC. ___ Bug-coreutils mailing list Bug-coreutils@gnu.org http://lists.gnu.org/mailman/listinfo/bug-coreutils
