Re: incredibly slow file listing script on windoze 7 pro 4 core 64 bit
On 9/8/10, Eric Blake ebl...@russianhut.comie wrote: On 09/08/2010 09:24 AM, Larry Hall (Cygwin) wrote: To somewhat sooth your curiousity, Windows (or perhaps it's more accurate to say NTFS) ain't great with directories with a large number of files. I expect you would be less than impressed with the performance of of 'dir' in 'cmd.exe' in the same directory. That said, you're asking for allot more work than you may realize when doing the same thing in Cygwin. In order to I don't want to add more clutter with this contrived example but just to make a point, I just got a 500G WD USB disk and sent these things to their final resting place. I had to reformat it is as vfat as that seems to be the only thing that is RW everywhere. I ran the script on this newer debian install with vfat and USB disk and it is faster than 'doze and probably faster than old emachines because I now have 2.8ghz CPU LOL. fill in the information you ask for when you use the '-l' flag for 'ls', Cygwin needs to open and close the files, which takes a good chunk of time en masse. The same does not need to happen in Linux/UNIX-land. Additionally, the stat() interface is puny - it MUST take the time to fill out the complete struct, even if the caller only cares for part of the information. If the Linux kernel ever incorporates the pending xstat() kernel call[1], then cygwin adds support for it, and coreutils is taught to make good use of it, then operations like ls can be sped up by asking for only the portions of the stat data that they plan on actually using, letting cygwin skip the work of obtaining the rest of the stat information just to be thrown away by the caller. [1]version 6 of that kernel patch is still being debated; as recently as http://lkml.indiana.edu/hypermail/linux/kernel/1008.2/00274.html -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: incredibly slow file listing script on windoze 7 pro 4 core 64 bit
On 9/7/10, Larry Hall (Cygwin) reply-to-list-only...@xxx.com wrote: On 9/7/2010 12:05 PM, mike marchywka wrote: this takes a few minutes on old debian machine, taking much longer here on same data- about 19k file about 24Gb total size. Windoze finally has better perf stuff but still no help- one core at 25 pct all kernel time disk not exactly busy. All the time is in the ls loop not the find command. Now obbviously I expect the ls per file has to make a bunch of OS calls for each file but still even with cygwin layer seems a bit much. Thanks. The problem is that Cygwin has to open all the files to stat when you use the '-l' flag. Try without the flag to get the lower bound and see if that's better for you. If so, maybe '-s' will suit your purpose. Or even 'du'? Sure, I could find a better way to do it but I was just curious about performance diffs. I've got this expensive 'doze laptop with 4 cores and the latest OS and all and compared to this old emachines desktop with debian it seems a bit slow. Now maybe this is a bit of a contrived case, no one would really do this and afterall the new system may be optimized for real-life usages and of course cygwin can't speed up obligatory OS calls. Anyway yes du seems much faster and is quite reasonable :) FWIW. Thanks. -- -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: incredibly slow file listing script on windoze 7 pro 4 core 64 bit
On 9/8/2010 9:35 AM, mike marchywka wrote: On 9/7/10, Larry Hall (Cygwin)reply-to-list-only...@xxx.com wrote: On 9/7/2010 12:05 PM, mike marchywka wrote: this takes a few minutes on old debian machine, taking much longer here on same data- about 19k file about 24Gb total size. Windoze finally has better perf stuff but still no help- one core at 25 pct all kernel time disk not exactly busy. All the time is in the ls loop not the find command. Now obbviously I expect the ls per file has to make a bunch of OS calls for each file but still even with cygwin layer seems a bit much. Thanks. The problem is that Cygwin has to open all the files to stat when you use the '-l' flag. Try without the flag to get the lower bound and see if that's better for you. If so, maybe '-s' will suit your purpose. Or even 'du'? Sure, I could find a better way to do it but I was just curious about performance diffs. I've got this expensive 'doze laptop with 4 cores and the latest OS and all and compared to this old emachines desktop with debian it seems a bit slow. Now maybe this is a bit of a contrived case, no one would really do this and afterall the new system may be optimized for real-life usages and of course cygwin can't speed up obligatory OS calls. To somewhat sooth your curiousity, Windows (or perhaps it's more accurate to say NTFS) ain't great with directories with a large number of files. I expect you would be less than impressed with the performance of of 'dir' in 'cmd.exe' in the same directory. That said, you're asking for allot more work than you may realize when doing the same thing in Cygwin. In order to fill in the information you ask for when you use the '-l' flag for 'ls', Cygwin needs to open and close the files, which takes a good chunk of time en masse. The same does not need to happen in Linux/UNIX-land. So what you're looking at here, above and beyond the design decisions of Windows, is some limitations of the emulated environment. So you can now take some joy in the knowledge that you have a spanking new system to run Windows/Cygwin on rather than an old eMachines box. ;-) Anyway yes du seems much faster and is quite reasonable :) Glad it helps. -- Larry Hall http://www.rfk.com RFK Partners, Inc. (508) 893-9779 - RFK Office 216 Dalton Rd. (508) 893-9889 - FAX Holliston, MA 01746 _ A: Yes. Q: Are you sure? A: Because it reverses the logical flow of conversation. Q: Why is top posting annoying in email? -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: incredibly slow file listing script on windoze 7 pro 4 core 64 bit
On 09/08/2010 09:24 AM, Larry Hall (Cygwin) wrote: To somewhat sooth your curiousity, Windows (or perhaps it's more accurate to say NTFS) ain't great with directories with a large number of files. I expect you would be less than impressed with the performance of of 'dir' in 'cmd.exe' in the same directory. That said, you're asking for allot more work than you may realize when doing the same thing in Cygwin. In order to fill in the information you ask for when you use the '-l' flag for 'ls', Cygwin needs to open and close the files, which takes a good chunk of time en masse. The same does not need to happen in Linux/UNIX-land. Additionally, the stat() interface is puny - it MUST take the time to fill out the complete struct, even if the caller only cares for part of the information. If the Linux kernel ever incorporates the pending xstat() kernel call[1], then cygwin adds support for it, and coreutils is taught to make good use of it, then operations like ls can be sped up by asking for only the portions of the stat data that they plan on actually using, letting cygwin skip the work of obtaining the rest of the stat information just to be thrown away by the caller. [1]version 6 of that kernel patch is still being debated; as recently as http://lkml.indiana.edu/hypermail/linux/kernel/1008.2/00274.html -- Eric Blake ebl...@redhat.com+1-801-349-2682 Libvirt virtualization library http://libvirt.org -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
incredibly slow file listing script on windoze 7 pro 4 core 64 bit
this takes a few minutes on old debian machine, taking much longer
here on same data-
about 19k file about 24Gb total size.
Windoze finally has better perf stuff but still no help- one core at
25 pct all kernel
time disk not exactly busy. All the time is in the ls loop not the
find command.
Now obbviously I expect the ls per file has to make a bunch of OS calls for
each file but still even with cygwin layer seems a bit much.
Thanks.
$ more tots
foo=
#ls -al `find -type f| awk '{print \$0}' ` | awk '{tot=tot+\$5; }END{print tot}
'
#ls -al `find -type f ` | awk '{tot=tot+$5; }END{print tot}'
find -type f xxx
cat xxx | while read
do
ls -al $REPLY
done | awk '{tot=tot+$5; }END{print tot}'
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: incredibly slow file listing script on windoze 7 pro 4 core 64 bit
On 9/7/2010 12:05 PM, mike marchywka wrote: this takes a few minutes on old debian machine, taking much longer here on same data- about 19k file about 24Gb total size. Windoze finally has better perf stuff but still no help- one core at 25 pct all kernel time disk not exactly busy. All the time is in the ls loop not the find command. Now obbviously I expect the ls per file has to make a bunch of OS calls for each file but still even with cygwin layer seems a bit much. Thanks. The problem is that Cygwin has to open all the files to stat when you use the '-l' flag. Try without the flag to get the lower bound and see if that's better for you. If so, maybe '-s' will suit your purpose. Or even 'du'? -- Larry Hall http://www.rfk.com RFK Partners, Inc. (508) 893-9779 - RFK Office 216 Dalton Rd. (508) 893-9889 - FAX Holliston, MA 01746 _ A: Yes. Q: Are you sure? A: Because it reverses the logical flow of conversation. Q: Why is top posting annoying in email? -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
