Re: cygwin 1.7.6: find skipping over some directories on NTFS mount points
On Aug 19 18:11, Rolf Campbell wrote: On 2010-08-19 13:05, Corinna Vinschen wrote: For further testing purposes I have uploaded a new cygwin1.dll which a) adds debug output in readdir() which prints DOS attributes as well as evaluated d_type value for each readdir entry to strace, and b) which is heavily tweaked to try harder to get a useful d_type value without compromising speed. The patch is attached, for the records. Rolf, please try the following Cygwin DLL: [...] It still fails in the exact same way. I attached strace output. Thanks for the new strace. After some more experimenting I was finally able to reproduce the issue. The other problem you reported, about df(*), lead me onto the right track. I've checked my changes in to CVS. For testing I provided another test DLL: http://cygwin.de/cygwin-ug-177/new-cygwin1.dll.bz2 (md5sum compressed 3a354b276e1506548bf382620ef26e82) (md5sum uncompressed 605c25c83ba9cd32bcebe77b7dfec99c) Thanks, Corinna (*) http://cygwin.com/ml/cygwin/2010-08/msg00611.html -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Project Co-Leader cygwin AT cygwin DOT com Red Hat -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: cygwin 1.7.6: find skipping over some directories on NTFS mount points
On 2010-08-20 07:56, Corinna Vinschen wrote: Thanks for the new strace. After some more experimenting I was finally able to reproduce the issue. The other problem you reported, about df(*), lead me onto the right track. I've checked my changes in to CVS. For testing I provided another test DLL: I tried both df and find using the new test dll and everything worked perfectly. Thanks for the quick response. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: cygwin 1.7.6: find skipping over some directories on NTFS mount points
On Aug 18 18:50, Rolf Campbell wrote: I have an 2nd NTFS disk mounted in a directory in my primary NTFS disk. When I use 'find' (with no arguments), it only displays a small fraction of the files in the current directory. Using cygwin 1.7.5, it displayed about 100,000 files. Using cygwin 1.7.6, it only displays the content of the first top-level directory and then just stops. I tried out all the snapshots and narrowed it down to 20100618. It worked correctly in 20100614 and does not work correctly in 20100618. If I mount the disk as a normal drive and access it using /cygdrive, it displays all 100, files with both 1.7.5 and 1.7.6. That narrows down the problem to a specific change, but unfortunately I can not reproduce the problem. I assume, when you say you mounted the 2nd drive, you didn't mean a Cygwin mount but rather a Windows junction point, so your drive doesn't show up as D:, but as C:\DriveD in Windows, correct? At least, that's what I tested. The drive has 1577 files and directories in multiple levels, and `find' always prints all of them, regardless whether the drive is mounted as drive or as juntion point. I also tried a Cygwin mount additionally, with the same result. Here are a couple of questions. They are all related to the mount using a junction point. - When you call `ls -l' in the top-level dir of the junction point, are directories recognized as directories? How many links do the directories supposedly have (should be 1)? - Is it possible that this has something to do with permissions? acl vs. noacl. Your cygcheck output doesn't point that out, but who knows. - Eventually, under Cygwin 1.7.6, can you please send us the output of `ls -l' of the top-level dir, and call `strace -o find.trace find' in the top-level dir and send the strace output? Please don't try this with the *working* scenario, the strace of a find finding 10 files would be a bit... swollen. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Project Co-Leader cygwin AT cygwin DOT com Red Hat -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: cygwin 1.7.6: find skipping over some directories on NTFS mount points
On Aug 19 10:31, Corinna Vinschen wrote: On Aug 18 18:50, Rolf Campbell wrote: I have an 2nd NTFS disk mounted in a directory in my primary NTFS disk. When I use 'find' (with no arguments), it only displays a small fraction of the files in the current directory. Using cygwin 1.7.5, it displayed about 100,000 files. Using cygwin 1.7.6, it only displays the content of the first top-level directory and then just stops. I tried out all the snapshots and narrowed it down to 20100618. It worked correctly in 20100614 and does not work correctly in 20100618. If I mount the disk as a normal drive and access it using /cygdrive, it displays all 100, files with both 1.7.5 and 1.7.6. That narrows down the problem to a specific change, but unfortunately I can not reproduce the problem. I assume, when you say you mounted the 2nd drive, you didn't mean a Cygwin mount but rather a Windows junction point, so your drive doesn't show up as D:, but as C:\DriveD in Windows, correct? At least, that's what I tested. The drive has 1577 files and directories in multiple levels, and `find' always prints all of them, regardless whether the drive is mounted as drive or as juntion point. I also tried a Cygwin mount additionally, with the same result. Here are a couple of questions. They are all related to the mount using a junction point. - When you call `ls -l' in the top-level dir of the junction point, are directories recognized as directories? How many links do the directories supposedly have (should be 1)? - Is it possible that this has something to do with permissions? acl vs. noacl. Your cygcheck output doesn't point that out, but who knows. - Eventually, under Cygwin 1.7.6, can you please send us the output of `ls -l' of the top-level dir, and call `strace -o find.trace find' in the top-level dir and send the strace output? Please don't try this with the *working* scenario, the strace of a find finding 10 files would be a bit... swollen. Hmm, is it perhaps possible that this is a side effect of running some aggressive virus scanner which needs tweaking? It's very strange that it should make a difference, whether the drive is mounted as drive or as junction point. There's no difference from Cygwin's POV, and the content of the drive obviously doesn't change by this. Therefore... well, it's grasping for straws, but still... Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Project Co-Leader cygwin AT cygwin DOT com Red Hat -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: cygwin 1.7.6: find skipping over some directories on NTFS mount points
On Aug 19 09:50, Rolf Campbell wrote: NTFS Junction point: yes. I used the builtin windowns tool mountvol to mount the disk in an empty directory. It's technically mounted as C:\.timemachine\3. Output from ls -l [...] I do not set the CYGWIN environmental variable when running find. I noticed something of interest, if I run find, then it only descends into the ATI directory. After exhausting every file/dir under ATI, it terminates. But, if I run find -maxdepth 2, it shows all correct output (it correctly enters all top-level and 2nd level directories -- it does NOT stop after leaving ATI). Eric? Can you have a look here, please? This is weird. I checked the strace, and after ascending back from the ATI subdir into the toplevel dir successfully, find appears to exit just so, without any trace that it even *tries* to continue to scan further subdirs. And unfortunately there's no way to see why find thinks there's nothing to do anymore. Is there a way to debug find to find out why it exits? Also, ls -R works just fine (showing all 100,000 files). That's, at least, good news. Hmm, digging through Cygwin's readdir code, I have a vague idea. Eric, does find honor the struct dirent d_type flag? I'm wondering if d_type is erroneously set to DT_REG for some reason. If so, we could find this out by augmenting the debug output in the Cygwin DLL. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Project Co-Leader cygwin AT cygwin DOT com Red Hat -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: cygwin 1.7.6: find skipping over some directories on NTFS mount points
On 08/19/2010 08:43 AM, Corinna Vinschen wrote: Hmm, digging through Cygwin's readdir code, I have a vague idea. Eric, does find honor the struct dirent d_type flag? I'm wondering if d_type is erroneously set to DT_REG for some reason. If so, we could find this out by augmenting the debug output in the Cygwin DLL. find (but not oldfind) relies heavily on the d_type flag. If that flag is incorrect, it could explain why find gets lost. Could you repeat the experiment with 'oldfind' and see if that behaves better? -- Eric Blake ebl...@redhat.com+1-801-349-2682 Libvirt virtualization library http://libvirt.org signature.asc Description: OpenPGP digital signature
Re: cygwin 1.7.6: find skipping over some directories on NTFS mount points
On Aug 19 10:28, Eric Blake wrote:
On 08/19/2010 08:43 AM, Corinna Vinschen wrote:
Hmm, digging through Cygwin's readdir code, I have a vague idea.
Eric, does find honor the struct dirent d_type flag? I'm wondering
if d_type is erroneously set to DT_REG for some reason. If so, we
could find this out by augmenting the debug output in the Cygwin DLL.
find (but not oldfind) relies heavily on the d_type flag. If that flag
is incorrect, it could explain why find gets lost. Could you repeat the
experiment with 'oldfind' and see if that behaves better?
For further testing purposes I have uploaded a new cygwin1.dll which
a) adds debug output in readdir() which prints DOS attributes as well as
evaluated d_type value for each readdir entry to strace, and
b) which is heavily tweaked to try harder to get a useful d_type value
without compromising speed. The patch is attached, for the records.
Rolf, please try the following Cygwin DLL:
http://cygwin.de/cygwin-ug-177/new-cygwin1.dll.bz2
(md5sum compressed 17d66fdd070ce3c57ae735b814cfe527)
(md5sum uncompressed e30408e665195b351d62d755f3da82ed)
Bunzip the DLL, chmod +x it, and replace your current DLL with that
version. Then repeat the find. If it still fails, please send the
strace output again.
Thanks,
Corinna
Index: fhandler_disk_file.cc
===
RCS file: /cvs/src/src/winsup/cygwin/fhandler_disk_file.cc,v
retrieving revision 1.332
diff -u -p -r1.332 fhandler_disk_file.cc
--- fhandler_disk_file.cc 18 Aug 2010 10:10:14 - 1.332
+++ fhandler_disk_file.cc 19 Aug 2010 16:59:35 -
@@ -148,10 +148,15 @@ path_conv::isgood_inode (__ino64_t ino)
return hasgood_inode () (ino UINT32_MAX || !isremote () || fs_is_nfs
());
}
-static inline bool
-is_volume_mountpoint (POBJECT_ATTRIBUTES attr)
+/* Check reparse point for type. IO_REPARSE_TAG_MOUNT_POINT types are
+ either volume mount points, which are treated as directories, or they
+ are directory mount points, which are treated as symlinks.
+ IO_REPARSE_TAG_SYMLINK types are always symlinks. We don't know
+ anything about other reparse points, so they are treated as unknown. */
+static inline int
+check_reparse_point (POBJECT_ATTRIBUTES attr)
{
- bool ret = false;
+ DWORD ret = DT_UNKNOWN;
IO_STATUS_BLOCK io;
HANDLE reph;
UNICODE_STRING subst;
@@ -165,15 +170,24 @@ is_volume_mountpoint (POBJECT_ATTRIBUTES
alloca (MAXIMUM_REPARSE_DATA_BUFFER_SIZE);
if (NT_SUCCESS (NtFsControlFile (reph, NULL, NULL, NULL,
io, FSCTL_GET_REPARSE_POINT, NULL, 0,
- (LPVOID) rp, MAXIMUM_REPARSE_DATA_BUFFER_SIZE))
- rp-ReparseTag == IO_REPARSE_TAG_MOUNT_POINT
- (RtlInitCountedUnicodeString (subst,
- (WCHAR *)((char *)rp-MountPointReparseBuffer.PathBuffer
- + rp-MountPointReparseBuffer.SubstituteNameOffset),
- rp-MountPointReparseBuffer.SubstituteNameLength),
- RtlEqualUnicodePathPrefix (subst, ro_u_volume, TRUE)))
- ret = true;
- NtClose (reph);
+ (LPVOID) rp, MAXIMUM_REPARSE_DATA_BUFFER_SIZE)))
+ {
+ if (rp-ReparseTag == IO_REPARSE_TAG_MOUNT_POINT)
+ {
+ RtlInitCountedUnicodeString (subst,
+ (WCHAR *)((char *)rp-MountPointReparseBuffer.PathBuffer
+ + rp-MountPointReparseBuffer.SubstituteNameOffset),
+ rp-MountPointReparseBuffer.SubstituteNameLength);
+ /* Only volume mountpoints are treated as directories. */
+ if (RtlEqualUnicodePathPrefix (subst, ro_u_volume, TRUE))
+ ret = DT_DIR;
+ else
+ ret = DT_LNK;
+ }
+ else if (rp-ReparseTag == IO_REPARSE_TAG_SYMLINK)
+ ret = DT_LNK;
+ NtClose (reph);
+ }
}
return ret;
}
@@ -1826,17 +1840,16 @@ fhandler_disk_file::readdir_helper (DIR
dir-__flags = ~dirent_set_d_ino;
}
- /* Set d_type if type can be determined from file attributes.
- FILE_ATTRIBUTE_SYSTEM ommitted to leave DT_UNKNOWN for old symlinks.
- For new symlinks, d_type will be reset to DT_UNKNOWN below. */
+ /* Set d_type if type can be determined from file attributes. For .lnk
+ symlinks, d_type will be reset below. Reparse points can be NTFS
+ symlinks, even if they have the FILE_ATTRIBUTE_DIRECTORY flag set. */
if (attr
- !(attr ( ~FILE_ATTRIBUTE_VALID_FLAGS
- | FILE_ATTRIBUTE_SYSTEM
- | FILE_ATTRIBUTE_REPARSE_POINT)))
+ !(attr (~FILE_ATTRIBUTE_VALID_FLAGS | FILE_ATTRIBUTE_REPARSE_POINT)))
{
if (attr FILE_ATTRIBUTE_DIRECTORY)
de-d_type = DT_DIR;
- else
+ /* FILE_ATTRIBUTE_SYSTEM might denote system-bit type symlinks. */
+ else if (!(attr FILE_ATTRIBUTE_SYSTEM))
de-d_type =
Re: cygwin 1.7.6: find skipping over some directories on NTFS mount points
On 2010-08-19 12:28, Eric Blake wrote: On 08/19/2010 08:43 AM, Corinna Vinschen wrote: Hmm, digging through Cygwin's readdir code, I have a vague idea. Eric, does find honor the struct dirent d_type flag? I'm wondering if d_type is erroneously set to DT_REG for some reason. If so, we could find this out by augmenting the debug output in the Cygwin DLL. find (but not oldfind) relies heavily on the d_type flag. If that flag is incorrect, it could explain why find gets lost. Could you repeat the experiment with 'oldfind' and see if that behaves better? oldfind displays all 100,000 files. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: cygwin 1.7.6: find skipping over some directories on NTFS mount points
Greetings, Corinna Vinschen! I checked the strace, and after ascending back from the ATI subdir into the toplevel dir successfully, find appears to exit just so, without any trace that it even *tries* to continue to scan further subdirs. And unfortunately there's no way to see why find thinks there's nothing to do anymore. If ATI is the junction (reparse point, or however you call it) to a top-level directory on another partition, this behavior could be explained by exiting through the window: process enter the partition from the doors (junction), dig it, then trying to exit from real path, which is, indeed, at the root of partition. So the process finds itself at the top-level and gracefully dies considering work done. A wild guess, however. -- WBR, Andrey Repin (anrdae...@freemail.ru) 20.08.2010, 2:33 Sorry for my terrible english... -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: cygwin 1.7.6: find skipping over some directories on NTFS mount points
On 2010-08-19 18:37, Andrey Repin wrote: If ATI is the junction (reparse point, or however you call it) to a top-level directory on another partition, this behavior could be explained by exiting through the window: process enter the partition from the doors (junction), dig it, then trying to exit from real path, which is, indeed, at the root of partition. So the process finds itself at the top-level and gracefully dies considering work done. A wild guess, however. Unfortunitely, ATI is not the reparse point. The root of the mounted file system is the 3 directory where ATI sits. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
