Re: Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance
It's time to move this discussion out of cygwin-patches and into cygwin-developers. I'd appreciate it if people continued over there. cgf
Re: Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance
On 9/28/2010 9:10 AM, Christopher Faylor wrote: It isn't extremely surprising that Linux access speed for a filesystem in a simulated environment, which presumably does not go through multiple layers of DLLs, would be faster than Cygwin. I think it more likely that the HGFS driver doesn't try to preserve full POSIX semantics. There's plenty of precedent: vfat, iso9660... One could probably verify this faster by examining the driver's source code (http://open-vm-tools.sourceforge.net/) than by tracing syscalls. If that's the explanation, it points at a possible path forward. On Linux, these secondary filesystems aren't expected to provide full POSIX semantics, simply because they are secondary. No one cries very hard that you can't make symlinks on a FAT-formatted USB stick. Yet, there's probably no technical reason you couldn't get a POSIX-like system to run on a crippled filesystem. It's probably even been done lots of times before in the embedded world. Some of the PC Unix systems from the 80s and early 90s were pretty screwy in this way, too. Screwy doesn't prevent you from doing useful work, though. Would it not be useful to have a mode in Cygwin that purposely skips any POSIX semantics that it can't get for free by making the POSIX syscalls nothing more than thin wrappers around the nearest equivalent Win32 API? If you put it in this mode and it breaks, you get to keep both pieces. There are those who would happily accept the speed increase for loss of some functionality. I wouldn't, but some would. I'd bet a lot of the 3PPs are in that group, since they know their target environment very well.
Re: Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance
On Tue, Sep 28, 2010 at 08:56:18AM -0600, Warren Young wrote: >On 9/22/2010 7:44 AM, Christopher Faylor wrote: >> >> What is /mnt/hgfs/C in this case? How is it mounted? > >HGFS is the Host-Guest File System, a VMware technology that lets it >export host volumes to the guest in a high-speed way. > >If you used old versions of VMware Workstation for Linux, you may >remember that they used to ship a version of Samba to export Linux-side >filesystems to the Windows guest. Now they use their proprietary HGFS >technology instead. In addition to being smaller and faster than Samba, >it works with all supported host and guest combinations, and it removes >a dependency. > >I believe Yoni's point is that the > > Linux guest -> HGFS/VMware -> Windows native > >path apparently has less code in it than the > > Cygwin -> Windows native > >code path. It isn't extremely surprising that Linux access speed for a filesystem in a simulated environment, which presumably does not go through multiple layers of DLLs, would be faster than Cygwin. I wouldn't even be surprised to see that the old Samba implementation was faster. cgf
Re: Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance
On 9/22/2010 7:44 AM, Christopher Faylor wrote: What is /mnt/hgfs/C in this case? How is it mounted? HGFS is the Host-Guest File System, a VMware technology that lets it export host volumes to the guest in a high-speed way. If you used old versions of VMware Workstation for Linux, you may remember that they used to ship a version of Samba to export Linux-side filesystems to the Windows guest. Now they use their proprietary HGFS technology instead. In addition to being smaller and faster than Samba, it works with all supported host and guest combinations, and it removes a dependency. I believe Yoni's point is that the Linux guest -> HGFS/VMware -> Windows native path apparently has less code in it than the Cygwin -> Windows native code path.
Re: Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance
Hi, > What is /mnt/hgfs/C in this case? How is it mounted? .host:/ on /mnt/hgfs type vmhgfs (rw,ttl=5) I am now preparing an easy to use scandir/stat performance testing program that will perform it in various methods and printout comparison results in an easy to use manner. It will compare cygwin API (to allow us to see how cygwin1.dll changes affect performance), and various native Windows APIs - so we can see how much room there is for improvement. I hope to be able to post this program tomorrow. Yoni ~$ cat /etc/fstab # /etc/fstab: static file system information. # # proc/proc procdefaults0 0 /dev/sda1 / ext3errors=remount-ro 0 1 /dev/sda5 noneswapsw 0 0 /dev/hda/media/cdrom0 udf,iso9660 user,noauto 0 0 /dev/fd0/media/floppy0 autorw,user,noauto 0 0 # Beginning of the block added by the VMware software .host:/ /mnt/hgfs vmhgfs defaults,ttl=5 0 0 # End of the block added by the VMware software On 22/9/2010 3:44 PM, Christopher Faylor wrote: On Wed, Sep 22, 2010 at 07:50:14AM +0200, Yoni Londner wrote: Hi, I'm not exactly concerned about Linux being way faster accessing an NTFS drive. After all it's the OS itself and comes with it's own NTFS driver which obviously is streamlined for typical POSIX operations. I did not test& compare to using the Linux NTFS, rather I compared with Linux on VMWARE using the same Windows NTFS.SYS (via the same kernel32.dll APIs): Cygwin: "C:/cygwin/bin/ls.exe /bin" -> cygwin1.dll -> kernel32.dll -> NTOS kernel -> NTFS.SYS driver -> HD linux: "/bin/ls /mnt/hgfs/C/cygwin/bin" -> glibc -> linux kernel -> VMWARE hgfs driver -> vmware_player.exe (on Win32) -> kernel32.dll -> NTOS kernel -> NTFS.SYS driver -> HD As you can see the VMWARE path is much longer than Cygwin, and it passes the same APIs and NTFS.SYS driver, and yet it executes much faster. This helps us understand that there is a lot that still can be done in Cygwin's filesystem performance. What is /mnt/hgfs/C in this case? How is it mounted? cgf
Re: Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance
On Wed, Sep 22, 2010 at 07:50:14AM +0200, Yoni Londner wrote: >Hi, > > > I'm not exactly concerned about Linux being way faster accessing an NTFS > > drive. After all it's the OS itself and comes with it's own NTFS driver > > which obviously is streamlined for typical POSIX operations. > >I did not test & compare to using the Linux NTFS, rather I compared with >Linux on VMWARE using the same Windows NTFS.SYS (via the same >kernel32.dll APIs): > >Cygwin: "C:/cygwin/bin/ls.exe /bin" -> cygwin1.dll -> kernel32.dll -> >NTOS kernel -> NTFS.SYS driver -> HD > >linux: "/bin/ls /mnt/hgfs/C/cygwin/bin" -> glibc -> linux kernel -> >VMWARE hgfs driver -> vmware_player.exe (on Win32) -> kernel32.dll -> >NTOS kernel -> NTFS.SYS driver -> HD > >As you can see the VMWARE path is much longer than Cygwin, and it passes >the same APIs and NTFS.SYS driver, and yet it executes much faster. > >This helps us understand that there is a lot that still can be done in >Cygwin's filesystem performance. What is /mnt/hgfs/C in this case? How is it mounted? cgf
Re: Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance
On Sep 22 11:32, Corinna Vinschen wrote: > On Sep 22 07:45, Yoni Londner wrote: > > I checked out why, and found out that #1 and #2 don't modify the > > access time of the file, whereas #3 does. This already immediately > > I just checked this and I can't see that it does. If it would do > so, shouldn't the access time be different every time I call stat? > > $ stat foo | grep 'Access: [0-9]' > Access: 2010-09-09 16:27:20.769055700 +0200 > $ stat foo | grep 'Access: [0-9]' > Access: 2010-09-09 16:27:20.769055700 +0200 > $ stat foo | grep 'Access: [0-9]' > Access: 2010-09-09 16:27:20.769055700 +0200 > > I tried it on Windows XP SP3 and Windows 7. Did you test this on a "noacl" mount, or on a filesystem which doesn't keep permissions, like FAT? If so, then I know what happens. This is the executable test in fhandler_base::fstat_helper. It reads the first two bytes from the file to identify executables by their magic number. This is especially done to identify shell scripts by their "#!" magic, so that they are marked as executable in st_mode. You can switch this off by specifing the "exec" or "notexec" mount options. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Project Co-Leader cygwin AT cygwin DOT com Red Hat
Re: Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance
On Sep 22 07:45, Yoni Londner wrote:
> Hi,
>
> > There's also the problem of handling NFS shares. However, I just had an
> > idea how to speed up symlink_info::check without neglecting NFS shares.
> > This will take some time, though since it turns a lot of code upside
> > down. Stay tuned.
>
> This sounds great! Cygwin filesystem performance is a very important
> issue, and any improvement is more than welcome!
Unfortunately it didn't work out the way I anticipated. See below.
> > I don't understand how you think this should work. The filter expression
> > given to NtQueryDirectoryFile is either a constant string and has
> to match
> > the filename exactly, or it contains wildcards. This is documented
> > behaviour:
> http://msdn.microsoft.com/en-us/library/ff567047%28VS.85%29.aspx
> > So, "foo" works, "foo*" works, but a list like "foo foo.exe foo.lnk"
> > does not.
>
> There are two options for stat() and other places the need file info
> (such as check_symlink):
>
> 1) CreateFile(the_dir), then NtQueryDirectoryFile("foo*") and
> retrieve all the info (including the hardlink), filter out the
How are you going to do that? Pray tell me the NtQueryDirectoryFile
info class which returns the number of hardlinks to a file. To the
best of my knowledge there is none.
Then there are the FileIdBothDirectoryInformation and
FileIdFullDirectoryInformation info classes, which were a nice idea, but
they have a downside, as you can see in fhandler_disk_file::readdir.
First, it doesn't exist on pre-XP systems. Second, there are a couple
of filesystems out in the wild which don't support the call. Third,
worse, there are filesystems which implement the call wrongly. Forth,
there's MSFT NFS which does not return dangling symlinks in calls to
NtQueryDirectoryFile, unless the info class is FileNamesInformation.
Last but not least, hardlink count and permissions are still missing.
So, using FileIdBothDirectoryInformation is far from being a swiss army
knife.
> results in user-mode ("foo", "foo.exe", "foo.lnk"), and then call
> CloseHandle().
>
> 2) CreateFile(the_dir), NtQueryDirectoryFile("foo"),
> NtQueryDirectoryFile("foo.exe"), NtQueryDirectoryFile("foo.lnk"),
> CloseHandle(). The calls to NtQueryDirectoryFile() should be with
> RestartScan=1, so that the the_dir handle can be reused. Also
> ReturnSingleEntry=1 can be set to improve performance.
That was the method I tried. It doesn't work. This is documented
in the aforementioned link to the NtQueryDirectoryFile man page:
"The FileName is used as a search expression and is captured
on the very first call to ZwQueryDirectoryFile for a given handle.
^
Subsequent calls to ZwQueryDirectoryFile will use the search expression
set in the first call.
The FileName parameter passed to subsequent calls will be ignored."
^
You can't use RestartScan in conjunction with another filter expression
to restart the scan with another filter expression. Either you are
lucky and the filename exists without .exe or .lnk suffix so that it's
returned by the first call, or what you get is the status code
STATUS_NO_MORE_FILES in any subsequent call with changed filter
expression. So, if you use this method you would have to close and
reopen the directory handle for every iteration on the suffix. Which in
turn slows down the method enormously. The end result was slower than
the current implementation.
When I have time I will try the #1 solution as well, but it requires
some more changes in symlink_info and needs more time, so it's kind of
on the backburner for now. I can see how general file access can
profit from a faster symlink_info::check which does not open every
file, but that won't affect stat() in the first place, rather other
calls which don't need stat-like info.
> This is instead what is done today in cygwin:
> 3) CreateFile("foo"), NtQueryFileInformation(), CloseHandle() (and
> repeat this for "foo.exe" and "foo.lnk")
Well, it's more like
Repeat CreateFile("foo${SUFFIX}") until file is found, then call
NtQueryFileInformation(), CloseHandle() exactly once.
> I did some performance tests comparing #1 #2 and #3.
>
> I found out that #1 and #2 are both around 10x to 100x (!!!) times
> faster than #3.
I guess so, but you can also get such figures by just using `ls',
not `ls -l'. But afaics you're comparing apples with oranges
since you deliberately drop information. Where is the hardlink
count? Where are the POSIX permissions?
> I checked out why, and found out that #1 and #2 don't modify the
> access time of the file, whereas #3 does. This already immediately
I just checked this and I can't see that it does. If it would do
so, shouldn't the access time be different every time I call stat?
$ stat foo | grep 'Access: [0-9]'
Access: 2010-09-09 16:27:20.769055700 +0200
$ stat foo | grep 'Access: [0-9]'
Re: Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance
Hi, > I'm not exactly concerned about Linux being way faster accessing an NTFS > drive. After all it's the OS itself and comes with it's own NTFS driver > which obviously is streamlined for typical POSIX operations. I did not test & compare to using the Linux NTFS, rather I compared with Linux on VMWARE using the same Windows NTFS.SYS (via the same kernel32.dll APIs): Cygwin: "C:/cygwin/bin/ls.exe /bin" -> cygwin1.dll -> kernel32.dll -> NTOS kernel -> NTFS.SYS driver -> HD linux: "/bin/ls /mnt/hgfs/C/cygwin/bin" -> glibc -> linux kernel -> VMWARE hgfs driver -> vmware_player.exe (on Win32) -> kernel32.dll -> NTOS kernel -> NTFS.SYS driver -> HD As you can see the VMWARE path is much longer than Cygwin, and it passes the same APIs and NTFS.SYS driver, and yet it executes much faster. This helps us understand that there is a lot that still can be done in Cygwin's filesystem performance. Yoni On 14/9/2010 11:38 AM, Corinna Vinschen wrote: On Sep 13 13:45, Yoni Londner wrote: Hi, Abstract: I prepared a patch that improves Cygwin Filesystem performance by x4 on Cygwin 1.7.5 (1.7.5 vanilla 530ms --> 1.7.5 patched 120ms). I ported the patch to 1.7.7, did tests, and found out that 1.7.7 had a very serious 9x (!) performance degradation from 1.7.5 (1.7.5 vanilla 530ms --> 1.7.7 vanilla 3900ms --> 1.7.7 patched 3500ms), which does makes this patch useless until the performance degradation is fixed. The problem is, I can't reproduce such a degradation. If I run sometimg like `time ls -l /bin> /dev/null', the times are very slightly better with 1.7.7 than with 1.7.5 (without caching effect 1200ms vs. 1500ms, with caching effect 500ms vs. 620ms on average). Starting with 1.7.6, Cygwin reused handles from symlink_info::check in stat() and other syscalls. If there is such degradation under some circumstances, I need a reproducible scenario, or at least some strace output which shows at which point this happens. Apart from actual patches this should be discussed on the cygwin-developer list. First of all, even your results of 1200-1500ms (1st time) and 500-600ms (2nd time) is still way way way too long. On linux with an NTFS mount of C:/cygwin, this took<2ms! And even on Win32 CMD.EXE this same operation will take you less than 100ms. which is 5x to 10x faster. The main reason for the difference: the Windows CMD.EXE does not open file handles, which make the NTFS file system to actually go and read each file's first 16KB of contents (even though you did not ask for it!). I'm not exactly concerned about Linux being way faster accessing an NTFS drive. After all it's the OS itself and comes with it's own NTFS driver which obviously is streamlined for typical POSIX operations. And then there's Win32 which can go through a dir much faster as well, since it doesn't have to care for POSIX compatibility of the result, and the OS function calls coincidentally match what a cmd "dir" call needs. If you're looking for a fair comparision, why don't you look for Interix? I did, and what I see is pretty much the same thing we do in Cygwin. Actually, with the last Cygwin from CVS an ls -lR on a non-marginal directory tree is already faster than the same operation under Interix. That doesn't mean I won't look for more ways to enhance Cygwin's performance, but it won't be by adding CYGWIN environment switches or by neglecting correct information in stat. Corinna
Re: Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance
Hi,
> There's also the problem of handling NFS shares. However, I just had an
> idea how to speed up symlink_info::check without neglecting NFS shares.
> This will take some time, though since it turns a lot of code upside
> down. Stay tuned.
This sounds great! Cygwin filesystem performance is a very important
issue, and any improvement is more than welcome!
> I don't understand how you think this should work. The filter expression
> given to NtQueryDirectoryFile is either a constant string and has to
match
> the filename exactly, or it contains wildcards. This is documented
> behaviour:
http://msdn.microsoft.com/en-us/library/ff567047%28VS.85%29.aspx
> So, "foo" works, "foo*" works, but a list like "foo foo.exe foo.lnk"
> does not.
There are two options for stat() and other places the need file info
(such as check_symlink):
1) CreateFile(the_dir), then NtQueryDirectoryFile("foo*") and retrieve
all the info (including the hardlink), filter out the results in
user-mode ("foo", "foo.exe", "foo.lnk"), and then call CloseHandle().
2) CreateFile(the_dir), NtQueryDirectoryFile("foo"),
NtQueryDirectoryFile("foo.exe"), NtQueryDirectoryFile("foo.lnk"),
CloseHandle(). The calls to NtQueryDirectoryFile() should be with
RestartScan=1, so that the the_dir handle can be reused. Also
ReturnSingleEntry=1 can be set to improve performance.
This is instead what is done today in cygwin:
3) CreateFile("foo"), NtQueryFileInformation(), CloseHandle() (and
repeat this for "foo.exe" and "foo.lnk")
I did some performance tests comparing #1 #2 and #3.
I found out that #1 and #2 are both around 10x to 100x (!!!) times
faster than #3.
I checked out why, and found out that #1 and #2 don't modify the access
time of the file, whereas #3 does. This already immediately causes a
huge performance penalty (and it is also not according to the posix
standard: stat("foo") should not update atime of "foo").
Another reason is that the kernel NTFS driver performs automatically
read-ahead of the file, thus just stat("foo") (which calls
CreateFile("foo") in #3) causes the first 64k of "foo" to be read from
the disk - slowing down performance tremendously. Think of "ls /bin"
with 3500 files: NTFS reads the first 64K of all the 3500 files! no
wonder it takes so long...
And yet another reason why #3 is way slower than #1 and #2 is the
anti-viruses: Nearly all Windows users install an AV (or use Win7 MS
AV). These trap and monitor all CreateFile() to regular files (not to
directory files). Therefore CreateFile() to a regular file can take a
lot lot longer than CreateFile() to a directory.
I would suggest using #2 over #1, since its simpler code-wise, and I did
not see any serious performance difference between the two.
Yoni
On 14/9/2010 12:05 PM, Corinna Vinschen wrote:
On Sep 13 13:28, Yoni Londner wrote:
Hi,
However, isn't that kind of a chicken/egg situation? If you want to
reuse the content of the FILE_BOTH{_ID}_DIRECTORY_INFORMATION structure
from a previous call to readdir, you would have to call the
I am not talking about reusing info from a previous readdir.
Every single file cygwin tries to access, it does it in a loop,
trying afterwards to check for *.lnk file.
Using the directory query operations, it is possible to get this
info faster:
instead of getting file info for FOO and then for "FOO.lnk",
Cygwin can query the directory info for "FOO FOO.LNK" (for the file
requested, plus its possible symlink file).
I don't understand how you think this should work. The filter expression
given to NtQueryDirectoryFile is either a constant string and has to match
the filename exactly, or it contains wildcards. This is documented
behaviour: http://msdn.microsoft.com/en-us/library/ff567047%28VS.85%29.aspx
So, "foo" works, "foo*" works, but a list like "foo foo.exe foo.lnk"
does not.
There's also the problem of handling NFS shares. However, I just had an
idea how to speed up symlink_info::check without neglecting NFS shares.
This will take some time, though since it turns a lot of code upside
down. Stay tuned.
Corinna
Re: Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance
On Sep 13 13:28, Yoni Londner wrote:
> Hi,
>
> > However, isn't that kind of a chicken/egg situation? If you want to
> > reuse the content of the FILE_BOTH{_ID}_DIRECTORY_INFORMATION structure
> > from a previous call to readdir, you would have to call the
>
> I am not talking about reusing info from a previous readdir.
>
> Every single file cygwin tries to access, it does it in a loop,
> trying afterwards to check for *.lnk file.
>
> Using the directory query operations, it is possible to get this
> info faster:
> instead of getting file info for FOO and then for "FOO.lnk",
> Cygwin can query the directory info for "FOO FOO.LNK" (for the file
> requested, plus its possible symlink file).
I don't understand how you think this should work. The filter expression
given to NtQueryDirectoryFile is either a constant string and has to match
the filename exactly, or it contains wildcards. This is documented
behaviour: http://msdn.microsoft.com/en-us/library/ff567047%28VS.85%29.aspx
So, "foo" works, "foo*" works, but a list like "foo foo.exe foo.lnk"
does not.
There's also the problem of handling NFS shares. However, I just had an
idea how to speed up symlink_info::check without neglecting NFS shares.
This will take some time, though since it turns a lot of code upside
down. Stay tuned.
Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Project Co-Leader cygwin AT cygwin DOT com
Red Hat
Re: Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance
On Sep 13 13:45, Yoni Londner wrote: > Hi, > > >> Abstract: I prepared a patch that improves Cygwin Filesystem > >> performance by x4 on Cygwin 1.7.5 (1.7.5 vanilla 530ms --> 1.7.5 > >> patched 120ms). I ported the patch to 1.7.7, did tests, and found > >> out that 1.7.7 had a very serious 9x (!) performance degradation > >> from 1.7.5 (1.7.5 vanilla 530ms --> 1.7.7 vanilla 3900ms --> 1.7.7 > >> patched 3500ms), which does makes this patch useless until the > >> performance degradation is fixed. > > > > The problem is, I can't reproduce such a degradation. If I run sometimg > > like `time ls -l /bin> /dev/null', the times are very slightly better > > with 1.7.7 than with 1.7.5 (without caching effect 1200ms vs. 1500ms, > > with caching effect 500ms vs. 620ms on average). Starting with 1.7.6, > > Cygwin reused handles from symlink_info::check in stat() and other > > syscalls. If there is such degradation under some circumstances, I need > > a reproducible scenario, or at least some strace output which shows at > > which point this happens. Apart from actual patches this should be > > discussed on the cygwin-developer list. > > > > First of all, even your results of 1200-1500ms (1st time) and > 500-600ms (2nd time) is still way way way too long. On linux with an > NTFS mount of C:/cygwin, this took <2ms! > > And even on Win32 CMD.EXE this same operation will take you less > than 100ms. which is 5x to 10x faster. > > The main reason for the difference: the Windows CMD.EXE does not > open file handles, which make the NTFS file system to actually go > and read each file's first 16KB of contents (even though you did not > ask for it!). I'm not exactly concerned about Linux being way faster accessing an NTFS drive. After all it's the OS itself and comes with it's own NTFS driver which obviously is streamlined for typical POSIX operations. And then there's Win32 which can go through a dir much faster as well, since it doesn't have to care for POSIX compatibility of the result, and the OS function calls coincidentally match what a cmd "dir" call needs. If you're looking for a fair comparision, why don't you look for Interix? I did, and what I see is pretty much the same thing we do in Cygwin. Actually, with the last Cygwin from CVS an ls -lR on a non-marginal directory tree is already faster than the same operation under Interix. That doesn't mean I won't look for more ways to enhance Cygwin's performance, but it won't be by adding CYGWIN environment switches or by neglecting correct information in stat. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Project Co-Leader cygwin AT cygwin DOT com Red Hat
Re: Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance
Hi,
> However, isn't that kind of a chicken/egg situation? If you want to
> reuse the content of the FILE_BOTH{_ID}_DIRECTORY_INFORMATION structure
> from a previous call to readdir, you would have to call the
I am not talking about reusing info from a previous readdir.
Every single file cygwin tries to access, it does it in a loop, trying
afterwards to check for *.lnk file.
Using the directory query operations, it is possible to get this info
faster:
instead of getting file info for FOO and then for "FOO.lnk",
Cygwin can query the directory info for "FOO FOO.LNK" (for the file
requested, plus its possible symlink file).
Yoni
On 12/9/2010 1:41 PM, Corinna Vinschen wrote:
On Sep 12 10:49, Yoni Londner wrote:
Hi,
The caching-speed up is trivial:
We store the the FileFullDirectoryInformation fields, and if any of
them change - we re-read the file.
Its not (in practical life) possible to change a file without
causing a modification on
FileIndex/CreationTime/LastWriteTime/ChangeTime/EndOfFile/AllocationSize/FileAttributes/FileName/EaSize!
From the MSDN we see the info we can get on a
FileFullDirectoryInformation request:
We're already using FileBothDirectoryInformation and
FileBothIdDirectoryInformation in readdir anyway.
However, isn't that kind of a chicken/egg situation? If you want to
reuse the content of the FILE_BOTH{_ID}_DIRECTORY_INFORMATION structure
from a previous call to readdir, you would have to call the
corresponding NtQueryInformationFile call(s) to fetch the information
from the file for comparision purposes. When you fetched it anyway,
there's no reason anymore to compare them, since you can use what
you just fetched. Where's the advantage?
Corinna
Re: Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance
On Sep 12 13:41, Corinna Vinschen wrote:
> On Sep 12 10:49, Yoni Londner wrote:
> > Hi,
> >
> > The caching-speed up is trivial:
> > We store the the FileFullDirectoryInformation fields, and if any of
> > them change - we re-read the file.
> >
> > Its not (in practical life) possible to change a file without
> > causing a modification on
> > FileIndex/CreationTime/LastWriteTime/ChangeTime/EndOfFile/AllocationSize/FileAttributes/FileName/EaSize!
> >
> > From the MSDN we see the info we can get on a
> > FileFullDirectoryInformation request:
>
> We're already using FileBothDirectoryInformation and
> FileBothIdDirectoryInformation in readdir anyway.
>
> However, isn't that kind of a chicken/egg situation? If you want to
> reuse the content of the FILE_BOTH{_ID}_DIRECTORY_INFORMATION structure
> from a previous call to readdir, you would have to call the
> corresponding NtQueryInformationFile call(s) to fetch the information
> from the file for comparision purposes. When you fetched it anyway,
> there's no reason anymore to compare them, since you can use what
> you just fetched. Where's the advantage?
The patch I still have in the loop uses the FILE_NETWORK_OPEN_INFORMATION
structure throughout, starting in symlink_info::check. The content is
then stored in the path_conv structure and reused in
fhandler_disk_file::fstat_by_handle. If the content of that structure
would be used as fingerprint, then we could perform readdir/stat caching
without requiring fstatat. Something similar could be used to perform
ACL caching, so that a typical `ls -l foo' would not request the ACL
two or three times, but only once...
Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Project Co-Leader cygwin AT cygwin DOT com
Red Hat
Re: Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance
On Sep 12 10:49, Yoni Londner wrote:
> Hi,
>
> The caching-speed up is trivial:
> We store the the FileFullDirectoryInformation fields, and if any of
> them change - we re-read the file.
>
> Its not (in practical life) possible to change a file without
> causing a modification on
> FileIndex/CreationTime/LastWriteTime/ChangeTime/EndOfFile/AllocationSize/FileAttributes/FileName/EaSize!
>
> From the MSDN we see the info we can get on a
> FileFullDirectoryInformation request:
We're already using FileBothDirectoryInformation and
FileBothIdDirectoryInformation in readdir anyway.
However, isn't that kind of a chicken/egg situation? If you want to
reuse the content of the FILE_BOTH{_ID}_DIRECTORY_INFORMATION structure
from a previous call to readdir, you would have to call the
corresponding NtQueryInformationFile call(s) to fetch the information
from the file for comparision purposes. When you fetched it anyway,
there's no reason anymore to compare them, since you can use what
you just fetched. Where's the advantage?
Corinna
--
Corinna Vinschen Please, send mails regarding Cygwin to
Cygwin Project Co-Leader cygwin AT cygwin DOT com
Red Hat
Re: Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance
Hi, The caching-speed up is trivial: We store the the FileFullDirectoryInformation fields, and if any of them change - we re-read the file. Its not (in practical life) possible to change a file without causing a modification on FileIndex/CreationTime/LastWriteTime/ChangeTime/EndOfFile/AllocationSize/FileAttributes/FileName/EaSize! From the MSDN we see the info we can get on a FileFullDirectoryInformation request: http://msdn.microsoft.com/en-us/library/cc232068(PROT.10).aspx NextEntryOffset (4 bytes): A 32-bit unsigned integer that contains the byte offset from the beginning of this entry, at which the next FILE_FULL_DIR_INFORMATION entry is located, if multiple entries are present in a buffer. This member is zero if no other entries follow this one. An implementation MUST use this value to determine the location of the next entry (if multiple entries are present in a buffer). FileIndex (4 bytes): A 32-bit unsigned integer that contains the byte offset of the file within the parent directory. For file systems such as NTFS, in which the position of a file within the parent directory is not fixed and can be changed at any time to maintain sort order, this field SHOULD be set to 0, and MUST be ignored.<79> CreationTime (8 bytes): The time when the file was created; see section 2.1.1. This value MUST be greater than or equal to 0. LastAccessTime (8 bytes): The last time the file was accessed; see section 2.1.1. This value MUST be greater than or equal to 0. LastWriteTime (8 bytes): The last time information was written to the file; see section 2.1.1. This value MUST be greater than or equal to 0. ChangeTime (8 bytes): The last time the file was changed; see section 2.1.1. This value MUST be greater than or equal to 0. EndOfFile (8 bytes): A 64-bit signed integer that contains the absolute new end-of-file position as a byte offset from the start of the file. EndOfFile specifies the offset to the byte immediately following the last valid byte in the file. Because this value is zero-based, it actually refers to the first free byte in the file. That is, it is the offset from the beginning of the file at which new bytes appended to the file will be written. The value of this field MUST be greater than or equal to 0. AllocationSize (8 bytes): A 64-bit signed integer that contains the file allocation size, in bytes. The value of this field MUST be an integer multiple of the cluster size. FileAttributes (4 bytes): A 32-bit unsigned integer that contains the file attributes. For a list of valid file attributes, see section 2.6. FileNameLength (4 bytes): A 32-bit unsigned integer that specifies the length, in bytes, of the file name contained within the FileName member. EaSize (4 bytes): A 32-bit unsigned integer that contains the combined length, in bytes, of the extended attributes (EA) for the file. FileName (variable): A sequence of Unicode characters containing the file name. When working with this field, use FileNameLength to determine the length of the file name rather than assuming the presence of a trailing null delimiter. Dot directory names are valid for this field. For more details, see section 2.1.5.1. Yoni On 10/9/2010 9:39 PM, Corinna Vinschen wrote: On Sep 10 13:23, Christopher Faylor wrote: On Fri, Sep 10, 2010 at 05:08:40PM +0200, Corinna Vinschen wrote: What I'm still mulling over is a good idea to re-use the results of a former call to readdir in a stat call. One problem here is to make sure that a subsequent stat call is *really* accessing the same file as the former readdir returned. I've considered that before but you really can't cheaply determine that the file hasn't changed without going to the OS. And, then it isn't cheap. Yes, that's what it always comes down to. That's why I'm considering to restrict this speedup to fstatat. I wrote more about this on the cygwin-developers list. Corinna
Re: Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance
On Sep 10 13:23, Christopher Faylor wrote: > On Fri, Sep 10, 2010 at 05:08:40PM +0200, Corinna Vinschen wrote: > >What I'm still mulling over is a good idea to re-use the results of a > >former call to readdir in a stat call. One problem here is to make sure > >that a subsequent stat call is *really* accessing the same file as the > >former readdir returned. > > I've considered that before but you really can't cheaply determine that > the file hasn't changed without going to the OS. And, then it isn't cheap. Yes, that's what it always comes down to. That's why I'm considering to restrict this speedup to fstatat. I wrote more about this on the cygwin-developers list. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Project Co-Leader cygwin AT cygwin DOT com Red Hat
Re: Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance
On Fri, Sep 10, 2010 at 05:08:40PM +0200, Corinna Vinschen wrote: >What I'm still mulling over is a good idea to re-use the results of a >former call to readdir in a stat call. One problem here is to make sure >that a subsequent stat call is *really* accessing the same file as the >former readdir returned. I've considered that before but you really can't cheaply determine that the file hasn't changed without going to the OS. And, then it isn't cheap. cgf
Re: Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance
On Sep 6 15:24, Corinna Vinschen wrote: > > - GetVolumeInfo: The C:\ drive does not tend to be changed every > > millisecond! Therefore no reason for every filesystem syscall to > > call it. Caching this further increased the performance. > > Does your FS caching take volume mount points into account? We had some filesystem caching back in Cygwin 1.5 already, but the new code in symlink_info::check and fs_info::update was supposed to make it unnecessary. However, I created a new caching mechanism now, which requires only a single NtQueryVolumeInformationFile call and then uses the result for a hash. I also tested a method which uses the object name of the underlying NT device as key for the hash, but it's actually ~10% slower than using the content of the FILE_FS_VOLUME_INFORMATION structure returned by NtQueryVolumeInformationFile(..., FileFsVolumeInformation). That's a rather unexpected result, but that's Windows. > > - File security check: GetSecurityInfo() is implemented in Windows > > in usermode dll (Advapi32.dll). It calls at the end > > NtQuerySecurityObject(). So I implemented a faster version: > > zGetSecurityInfo() which does the same... just faster. > > Does your code preserve the inheritance entries which are available via > GetSecurityInfo? Note that I don't care at all for the GetSecurityInfo > API from a usage POV. I would prefer to use NtQuerySecurityObject > directly. However, the sole reason for using GetSecurityInfo is the > fact that NtQuerySecurityObject only returns the plain ACL as it's > stored for the file. It does not return the ACEs which are inherited > from the parent objects. These are only available via GetSecurityInfo, > or by checking the parent ACLs manually. This turned out to be total nonsense. The actual reason to use GetSecurityInfo was the fact that it's the only function which sets the INHERITED_ACE flag, as my comment in get_file_sd tells everybody who's willing to read. So I read that now, too. A bit of looking through the code shows that the INHERITED_ACE flag is *only* required if a file has just been created (see alloc_sd). So only the call to set_file_attribute in fhandler_base::open() really needs these flags. What I did now was to extend get_file_sd to call GetSecurityInfo only in this case, while every other call to get_file_sd will actually just call NtQuerySecurityObject, which is obviously faster because it doesn't have to check the parent ACL. Additionally I changed the allocation for the security_descriptor structure so that it's allocated big enough and only a single call to NtQuerySecurityObject is performed, rather than two. Other than that I have another patch in the loop which stores file information from symlink_info::check to reuse them in calls to stat, to drop another OS call. This needs a bit more testing, though. All in all the number of OS calls accessing "foo" in a single `ls -l foo' call has dropped from 18 to 12. `stat foo' needed 10 before, now it needs 8 or 7, depending on the fs caching state. `ls -lR' on a Samba drive now is almost 30% faster. What I'm still mulling over is a good idea to re-use the results of a former call to readdir in a stat call. One problem here is to make sure that a subsequent stat call is *really* accessing the same file as the former readdir returned. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Project Co-Leader cygwin AT cygwin DOT com Red Hat
Re: Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance
Thanks for the patch and for all of the work you put into it. On Mon, Sep 06, 2010 at 03:24:09PM +0200, Corinna Vinschen wrote: >The patch is also missing a ChangeLog entry. I only took a quick glance >over the patch itself. The code doesn't look correctly formatted in GNU >style. Also, using the diff -up flags would be helpful. And, this is the type of patch which would be better served if submitted in small chunks. You have multiple changes in your 1158 line patch and they don't seem to all be interrelated. Also, in addition to formatting concerns, you don't seem to have used comments very much. Corinna and I have been making a concerted effort to comment changes more thoroughly so it would be nice if your patch contained more of those. I didn't look at the patch very closely either since there are copyright issues but some of your conclusions don't seem right to me. I agree with Corinna's response. cgf
Re: Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance
Hi Yoni, On Sep 6 12:52, Yoni Londner wrote: > Hi, > > Abstract: I prepared a patch that improves Cygwin Filesystem > performance by x4 on Cygwin 1.7.5 (1.7.5 vanilla 530ms --> 1.7.5 > patched 120ms). I ported the patch to 1.7.7, did tests, and found > out that 1.7.7 had a very serious 9x (!) performance degradation > from 1.7.5 (1.7.5 vanilla 530ms --> 1.7.7 vanilla 3900ms --> 1.7.7 > patched 3500ms), which does makes this patch useless until the > performance degradation is fixed. The problem is, I can't reproduce such a degradation. If I run sometimg like `time ls -l /bin > /dev/null', the times are very slightly better with 1.7.7 than with 1.7.5 (without caching effect 1200ms vs. 1500ms, with caching effect 500ms vs. 620ms on average). Starting with 1.7.6, Cygwin reused handles from symlink_info::check in stat() and other syscalls. If there is such degradation under some circumstances, I need a reproducible scenario, or at least some strace output which shows at which point this happens. Apart from actual patches this should be discussed on the cygwin-developer list. > === > My patch: > First of all, your patch is very certainly significant in terms of code size. If you want to contribute code from it, we need a signed copyright assignment from you. See http://cygwin.com/contrib.html, the "Before you get started" section. The patch is also missing a ChangeLog entry. I only took a quick glance over the patch itself. The code doesn't look correctly formatted in GNU style. Also, using the diff -up flags would be helpful. > So I did some performace test on Cygwin 1.7.5, and found out the > biggest bottleneck were: > > - CreateFile()/CloseHandle() on syscalls that only need file node > info retrival, not file contents retrival (such as stat()). This can > be solved by calling Win32 that do not open the File handle itself, > rather query by name, or opening the directory handle (which is MUCH > faster). That's what I tried to speed up in 1.7.6 by keeping the handle from symlink_info::check for reuse in stat. > - repetitive Win32 Kernel calls on a single syscall (stat() would > call up to 5 CreateFile/CloseHandle pairs plus another additional 5 > to 10 Win32 APIs). > on stat() system calls, managed to improve the performance of ls > approx. by 4 times. This can be solved by caching: first time in a > syscall the API is called the result is stored, so the second time > the info is needed, it is re-used. I agree. We could use caching fileinfo more extensively, though carefully, at least for the time of a single syscall. > - ACLs: Windows is not a secure system. And its ACL model is strange > to say the least... Most cygwin users do not use the unix > permissions system on cygwin to achieve security. All they want is > that the unix tools will run on Windows. I don't agree. Windows is a secure system on the file level if users use the ACLs correctly. It's not exactly a strange model, it's just unnecessarily complicated for home users. However, in corporate environments it's utilized a lot. Besides, I made some performance tests myself before 1.7.6, and it turned out that the ACL checks for testing the POSIX permission bits are really not the problem. The problem was that ls(1) had a test which resulted in calling getacl() twice, just to print the '+' character attached to the POSIX permission bits in `ls -l' output. That should have gotten slightly better with the latest coreutils. Other than that, just use the "noacl" mount option if you don't like to be bothered with Windows ACLs or, FWIW, POSIX-like permissions. > - stat inode number and inode link count: getting the inode in > Windows requires a File handle or Directory handle (not possible to > get inode by file name). Very few applications actually need a REAL > inode number. So using the 'get file info by name' apis are used > (and of course there is an envirnoment if you really want real inode > numbers). Just use the "ihash" mount option if you don't like to use real inode numbers. However, you don't see hardlinks anymore. I don't like the idea to extend the CYGWIN environment variable with this functionality at all. I also don't like the idea that these settings are on by default. So by default you're trading correctness (in a POSIX sense) for speed. Speed is all well and nice, but it shouldn't be handled more important than correctness. There are certainly other ways to improve Cygwin's performance, which don't trade in correctness in the first place. We should more concentrate on intelligent caching of file info and handles. > - GetVolumeInfo: The C:\ drive does not tend to be changed every > millisecond! Therefore no reason for every filesystem syscall to > call it. Caching this further increased the performance. Does your FS caching take volume mount points into account? > - File security check: GetSecurityInfo() is implemente
