One final time

[Christopher Wingert]

Just FYI, just to be clear how petty Mr. Faylor is:

I wrote my last email to the mailing list at 06/05/2010 23:16.

I unsubscribed to the cygwin@cygwin.com mailing list and received
success at 06/05/2010 23:26.

I received no further cygwin emails till 06/06/2010 01:26.

At that time, Mr Faylor added me back into the list specifically for
the purposes to get the last word in.

The message: http://www.cygwin.com/ml/cygwin/2010-06/msg00149.html

Again good luck with your project, but leave me alone!

Chris

--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Performance and stat()

[Christopher Wingert]

> I do think out loud with my "team".  You are not on it.

Agreed!  You would rather spend your time ridiculing any possible solution.

This is what lead to my initial reluctance to do any patch for Cygwin
software.

Good Luck with your inferior product.

Chris




--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Performance and stat()

[Christopher Wingert]

> Can't really parse that sentence.

Then load your English parser.


> I haven't detected any "picking on" but then I can't claim to have
> written the fhandler* code anymore Corinna has rewritten most of it.  I
> do know that if you want to be taken seriously you really need to send a
> concrete suggestion/patch.

I don't know the right answer to the problem.

BUT... I have shown how to do achieve similar results to the dll, but with
significantly less overhead.

So... the person who cared to improve his/her/its code would say,  "Well
we use NTOpenFile() because it does the blah blah extra functionality that
FindNextFile()/GetFileAttributes() do not."  Then we could look to other
Win32 APIs to try to achieve those results.

For example opening a file on Windows for the purposes of stat()ing a file
is dumb, considering the way most Windows Virus Scanners work.



> So far, it seems to me that you're basically thinking out loud and
> expecting us to fill in the blanks.  That's not an effective way to
> getting anything changed.

Yes, thinking out loud is an effective way for a team to work together. 
You should try it





--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Performance and stat()

[Christopher Wingert]

He is?  Holy crap, he is more helpful with his sarcasm and doubt than
anything else.  However, it does explains his tone, given that I am
picking on his code.

I see your reference to an acronym regarding top posting, but not a
policy.  In fact your reference is specifically listed under another
obscure acronym OLOCA, which does not reference any policy.

If you want to have a policy regarding top posting you should put it here:
http://www.cygwin.com/lists.html

FWIW, I think inline posting is very 90's.  Your point get lost.

Finally, I am serious and I am not looking for any further friends (I have
enough).  I am doing analysis and looking at code that shows how we can
make great software better.  If that's not welcome because of the format
of my emails then I'll stop.

Chris


> On 06/04/2010 03:14 PM, Christopher Wingert wrote:
>> Agreed, I would like to make a global change, however, unless I can talk
>> to the current maintainer of the fhandler* functions, it seems illogical
>> for me to change them (as I have about a week of cygwin dll experience).
>
> You ARE talking to the maintainer of the fhandler* functions - cgf knows
> what he's talking about, since he wrote the bulk of them.
>
> And QUIT top-posting - by violating list policy, you aren't winning any
> friends.  http://cygwin.com/acronyms/#TOFU
> Prove that you are serious about helping, by showing a modicum of honest
> effort to obey list policy.
>
> --
> Eric Blake   ebl...@redhat.com+1-801-349-2682
> Libvirt virtualization library http://libvirt.org
>
>



--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Performance and stat()

[Christopher Wingert]

See further down the thread, the right solution is to impact ALL cygwin
executables, but I don't have the experience in the dll to make those
changes.


> On 6/4/2010 2:20 PM, Christopher Faylor wrote:
>>> "But providing a variant of stat() along the lines of what you propose
>>> above is not practical for all the reasons already stated."
>> This is not something that I said.  That was actually Larry Hall.
>
> Heh.  Who needs him anyway!
>
> Just to clarify, this comment was in response to Chris Wingerts' assertion
> () that it would be
> worthwhile to provide some kind of switch to selectively disable the
> expensive parts of stat().  And my point was that this had already been
> discounted as a transparent way of addressing the performance problem
> because it would still be up to the user or application to determine when
> to make this trade-off
> ().
> This is the same conclusion Chris Wingert has now come to as well and
> stated
> in :
>
>All that being said, I think the best solution is not to optimize the
> dll
>stat(), but to do it at the executable level.  I see that Cygwin
> already
>has some level of patches at this level, it shouldn't be too difficult
> to
>support.
>
> So we're all back on the same page now. :-)
>
> --
> Larry Hall  http://www.rfk.com
> RFK Partners, Inc.  (508) 893-9779 - RFK Office
> 216 Dalton Rd.  (508) 893-9889 - FAX
> Holliston, MA 01746
>
> _
>
> A: Yes.
>> Q: Are you sure?
>>> A: Because it reverses the logical flow of conversation.
 Q: Why is top posting annoying in email?
>
> --
> Problem reports:   http://cygwin.com/problems.html
> FAQ:   http://cygwin.com/faq/
> Documentation: http://cygwin.com/docs.html
> Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
>
>



--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Performance and stat()

[Christopher Wingert]

I actually wrote my own rsync in AutoIt, but it is also limited.

However, I thought it would a nice addition to speed up Cygwin.  As I have
been using it for about 10 years and made my transition from Linux to
Windows so pleasant.


> On 4 June 2010 19:50, Eliot Moss wrote:
>> I don't think there's an objection here to
>> patching *rsync* specially in the cygwin
>> environment -- that would be between you
>> and the rsync port maintainer. The issue
>> is whether or not to make a more general
>> change to cygwin itself, and cgf is just
>> saying that that's hard to do.
>>
>> Conceivably we could come up with some
>> additional functions that cygwin ports
>> could use if they want to, but "out of
>> box" use of stat needs to replicate
>> full behavior since there's no way to
>> know what a given call of stat really
>> needs ...
>
> MKS has such a faster but limited version of stat:
>
> http://www.mkssoftware.com/docs/man3/_NutFastStat.3.asp
>
> Shame about the name.
>
> Andy
>
> --
> Problem reports:   http://cygwin.com/problems.html
> FAQ:   http://cygwin.com/faq/
> Documentation: http://cygwin.com/docs.html
> Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
>
>



--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Performance and stat()

[Christopher Wingert]

Agreed, I would like to make a global change, however, unless I can talk
to the current maintainer of the fhandler* functions, it seems illogical
for me to change them (as I have about a week of cygwin dll experience).

Also my interest in performance is limited to a very certain subset of
executables : bash, rsync, stat, and du.  As an example find already seems
to have good performance under cygwin.



> I don't think there's an objection here to
> patching *rsync* specially in the cygwin
> environment -- that would be between you
> and the rsync port maintainer. The issue
> is whether or not to make a more general
> change to cygwin itself, and cgf is just
> saying that that's hard to do.
>
> Conceivably we could come up with some
> additional functions that cygwin ports
> could use if they want to, but "out of
> box" use of stat needs to replicate
> full behavior since there's no way to
> know what a given call of stat really
> needs ...
>
> Eliot Moss
>
> --
> Problem reports:   http://cygwin.com/problems.html
> FAQ:   http://cygwin.com/faq/
> Documentation: http://cygwin.com/docs.html
> Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
>
>



--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Performance and stat()

[Christopher Wingert]

> [quit top-posting]

Now you are my mom too?


> That's where you're wrong.  Any patch you write that is technically
> sound and shows a measurable improvement will most likely be accepted.

Then you shouldn't have Cygwin's front line technical spokesman saying
things such as:

"If there was a way to make stat() faster why wouldn't it be in the source
code already?"

"Otherwise, I doubt that anyone outside of the cygwin developers
understands the stat() code well enough to come up with a patch."

"But providing a variant of stat()
along the lines of what you propose above is not practical for all the
reasons already stated."

"I guess it's possible that someone just doesn't want to go through the
pain of getting the patch accepted.  In that case, everyone enjoy your
private cygwin stat() patches."



When I threw the idea out initially I asked for input from the people that
have more experience in Cygwin than I do.  I have been seeing these
performance issues for years of using Cygwin.  However, in the past week
had some time to look at it.

FWIW, because of the initial resistance I have been shown by the "front
line developers."  I have already contemplated of my own Cygwin branch.

All that being said, I think the best solution is not to optimize the dll
stat(), but to do it at the executable level.  I see that Cygwin already
has some level of patches at this level, it shouldn't be too difficult to
support.


Chris



--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Performance and stat()

[Christopher Wingert]

Again, not production, this was to highlight a point and to enhance (I
would assume) the most common use case (file to file rsync).

Plenty of solutions for a good patch, complete the Windows version of
stat() or call the Cygwin version if the Window's version fails.

Since I don't have any confidence that the "Cygwin organization" will
accept any patch I write (per Faylor), I really have no incentive to do
the job right.

Chris



> On 6/4/2010 12:37 PM, Christopher Wingert wrote:
>> So here is an example of a performance gain by not using cygwin stat().
>> I
>> did this patch in about an hour (with the help of some git code), so I
>> wouldn't recommend it for any production use.
>>
>> On a dry run rsync from my local drive to my NAS (105GB, 34k files, 4k
>> directories).  The current release cygwin rsync did it in [36m:43s],
>> with
>> the patch (below) applied, it dropped to [04m:48s].
>
> But your patch only does a portion of the work that stat() is supposed to
> do.  I don't think there's any question that ignoring the need to fill in
> certain stat fields will save time.  But that's not really a workable
> alternative either, if that's what you're implying.
>
> --
> Larry Hall  http://www.rfk.com
> RFK Partners, Inc.  (508) 893-9779 - RFK Office
> 216 Dalton Rd.  (508) 893-9889 - FAX
> Holliston, MA 01746
>
> _
>
> A: Yes.
>> Q: Are you sure?
>>> A: Because it reverses the logical flow of conversation.
>>>> Q: Why is top posting annoying in email?
>
> --
> Problem reports:   http://cygwin.com/problems.html
> FAQ:   http://cygwin.com/faq/
> Documentation: http://cygwin.com/docs.html
> Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
>
>



--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Performance and stat()

[Christopher Wingert]

So here is an example of a performance gain by not using cygwin stat().  I
did this patch in about an hour (with the help of some git code), so I
wouldn't recommend it for any production use.

On a dry run rsync from my local drive to my NAS (105GB, 34k files, 4k
directories).  The current release cygwin rsync did it in [36m:43s], with
the patch (below) applied, it dropped to [04m:48s].


$ diff rsync-3.0.7-1/src/rsync-3.0.7/syscall.c
fast-rsync-3.0.7-1/src/rsync-3.0.7/syscall.c
31a32,37
> // Get rid of a define for WINBOOL
> #define __OBJC__
> #define WIN32_LEAN_AND_MEAN
> #include 
> #include 
>
242a249
>printf( "doing [stat]\n" );
249a257,277
> static inline time_t filetime_to_time_t(const FILETIME *ft)
> {
>   long long winTime = ((long long)ft->dwHighDateTime << 32) +
ft->dwLowDateTime;
>   winTime -= 1164447360LL; /* Windows to Unix Epoch conversion */
>   winTime /= 1000; /* Nano to seconds resolution */
>   return (time_t)winTime;
> }
>
> static inline void filetime_to_timespec(const FILETIME *ft, struct
timespec *ts)
> {
>long long winTime = ((long long)ft->dwHighDateTime << 32) +
ft->dwLowDateTime;
>winTime -= 1164447360LL; /* Windows to Unix Epoch conversion */
>ts->tv_sec = (time_t)(winTime/1000); /* 100-nanosecond interval
to seconds */
>ts->tv_nsec = (long)(winTime - ts->tv_sec*1000LL) * 100; /*
nanoseconds */
> }
>
>
>
> #define size_to_blocks(s) (((s)+511)/512)
>
>
251a280,344
> #if 1
>char path[ 1024 ];
>WIN32_FILE_ATTRIBUTE_DATA fdata;
>int fMode;
>
>cygwin_conv_path( CCP_POSIX_TO_WIN_A | CCP_ABSOLUTE, fname, path,
1024 );
>
>if ( GetFileAttributesEx( path, GetFileExInfoStandard, &fdata ) )
>{
>   // if ((fdata.dwFileAttributes & FILE_ATTRIBUTE_SYSTEM) &&
>   //!(fdata.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY))
>   //return -1;
>   // printf( "GetFileAttributesEx() OK\n" );
>
>   if (fdata.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY)
>  fMode |= S_IFDIR;
>   else
>  fMode |= S_IFREG;
>   if (!(fdata.dwFileAttributes & FILE_ATTRIBUTE_READONLY))
>  fMode |= S_IWRITE;
>
>   st->st_ino = 0;
>   st->st_gid = st->st_uid = 0;
>   st->st_nlink = 1;
>   st->st_mode = fMode;
>   st->st_size = ((_off64_t)fdata.nFileSizeHigh << 32) +
>  fdata.nFileSizeLow;
>   st->st_blocks = size_to_blocks(st->st_size);
>   st->st_dev = st->st_rdev = 0;
>   filetime_to_timespec(&fdata.ftLastAccessTime, &st->st_atim);
>   filetime_to_timespec(&fdata.ftLastWriteTime, &st->st_mtim);
>   filetime_to_timespec(&fdata.ftCreationTime, &st->st_ctim);
>   errno = 0;
>   return 0;
>}
>else
>{
>   // rsyserr(FERROR, errno, "GetFileAttributesEx() failed %s", path );
>   // printf( "GetFileAttributesEx() FAILED\n" );
>   errno = ENOENT;
>   return( -1 );
>}
>
>
>switch (GetLastError())
>{
>   case ERROR_ACCESS_DENIED:
>   case ERROR_SHARING_VIOLATION:
>   case ERROR_LOCK_VIOLATION:
>   case ERROR_SHARING_BUFFER_EXCEEDED:
>  errno = EACCES;
>  break;
>   case ERROR_BUFFER_OVERFLOW:
>  errno = ENAMETOOLONG;
>  break;
>   case ERROR_NOT_ENOUGH_MEMORY:
>  errno = ENOMEM;
>  break;
>   default:
>  break;
>}
>
>return( -1 );
> #endif
>
264a358
>// printf( "doing [fstat]\n" );



--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Performance and stat()

[Christopher Wingert]

> On Thu, Jun 03, 2010 at 05:32:46PM -0700, Christopher Wingert wrote:
> Yeah, that's what I thought you were doing.  Given that the timestamps
> don't indicate "elapsed time of function X", it's not always possible to
> figure out how long a function takes by subtracting.  Subtracting
> timestamps shows the delta between one time that someone thought to put
> an strace_printf in the code and another time that someone thought to
> put an strace_printf in the code.  There is no guarantee that there is
> an strace_printf at entry or exit of a function.

Yes, except that someone instrumenting the entry/exit points was me, and
the fact that they are not actually real syscalls and no other cygwin
processes are running says the timestamps are accurate for the purposes of
debugging.  Further the numbers match up with the overall timings I did
before the debugging.


> It is a shame that we weren't more standardized in our strace output so
> that kind of thing could be possible.

Agreed (for once), the instrumentation of the syscalls and the fact that
they are sprinkled over multiple files made things very difficult to
debug.


> However, for Cygwin, the web site says multiple times in multiple places
> that you shouldn't send private email and to use the mailing list.  So,
> other projects aside that really is how we do things here.

Understood.  I'll wait with bated breath for a knowledgeable developer to
speak up.



> Oh, and it isn't clear if you're implying that I'm a lackey but I'm
> really not.

H, I thought it was abundantly clear.





--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Performance and stat()

[Christopher Wingert]

> On Thu, Jun 03, 2010 at 10:35:55AM -0700, Christopher Wingert wrote:
>>Using strace I was able to look at some of the functions that are
>>enumerated by debugging calls.
>>
>>The trace below is done by ls.exe for each file (approximately 95k files
>> @
>>88 mSecs/file), approximately 40 mSecs are spent in lstat64() and another
>>47 mSecs are spent in getacl().
>
> You're undoubtedly misinterpreting the timestamps in strace.  They don't
> indicate the amount of time spent in anything.  They are just timestamps.

Undoubtedly, no.

I am doing basic subtraction based on the synchronous call made from the
ls.exe executable to the cygwin1.dll and the timestamp provided by strace.


> You may be missing how this project is run.  The current maintainers of
> everything read this mailing list.  You don't need to contact anyone
> personally.  Actually, this is typical of many open source projects.

Actually it is atypical for core developers to monitor a high volume
generic question list such as this one, at least from my experience on
other open source projects.  The core developers would leave it to some
nay-saying lackey.




--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Performance and stat()

[Christopher Wingert]

93203 [main] ls 3688 set_flags: flags: binary (0x2)
   21  693224 [main] ls 3688 mount_info::conv_to_win32_path: src_path
/cygdrive/r/dropbox/MS/Dup/original/Dup.csv, dst
R:\dropbox\MS\Dup\original\Dup.csv, flags 0x4022, rc 0
  402  693642 [main] ls 3688 symlink_info::check: 0xC04F =
NtCreateFile (\??\R:\dropbox\MS\Dup\original\Dup.csv)
  774  694416 [main] ls 3688 symlink_info::check: 0x0 = NtOpenFile (no-EA,
\??\R:\dropbox\MS\Dup\original\Dup.csv)
 1175  695591 [main] ls 3688 symlink_info::check: not a symlink
  414  696005 [main] ls 3688 symlink_info::check: 0 = symlink.check
(R:\dropbox\MS\Dup\original\Dup.csv, 0x22B440) (0x4022)
   31  696036 [main] ls 3688 path_conv::check:
this->path(R:\dropbox\MS\Dup\original\Dup.csv), has_acls(0)
   28  696064 [main] ls 3688 build_fh_pc: fh 0x612329D8
   25  696089 [main] ls 3688 acl_worker: 4 = acl
(/cygdrive/r/dropbox/MS/Dup/original/Dup.csv)
   23  696154 [main] ls 3688 normalize_posix_path: src
/cygdrive/r/dropbox/MS/Dup/original/Dup.csv
   28  696182 [main] ls 3688 normalize_posix_path:
/cygdrive/r/dropbox/MS/Dup/original/Dup.csv = normalize_posix_path
(/cygdrive/r/dropbox/MS/Dup/original/Dup.csv)
   16  696198 [main] ls 3688 mount_info::conv_to_win32_path:
conv_to_win32_path (/cygdrive/r/dropbox/MS/Dup/original/Dup.csv)
   16  696214 [main] ls 3688 mount_info::cygdrive_win32_path: src
'/cygdrive/r/dropbox/MS/Dup/original/Dup.csv', dst
'R:\dropbox\MS\Dup\original\Dup.csv'
   16  696230 [main] ls 3688 set_flags: flags: binary (0x2)
   14  696244 [main] ls 3688 mount_info::conv_to_win32_path: src_path
/cygdrive/r/dropbox/MS/Dup/original/Dup.csv, dst
R:\dropbox\MS\Dup\original\Dup.csv, flags 0x4022, rc 0
  349  696609 [main] ls 3688 symlink_info::check: 0xC04F =
NtCreateFile (\??\R:\dropbox\MS\Dup\original\Dup.csv)
  602  697211 [main] ls 3688 symlink_info::check: 0x0 = NtOpenFile (no-EA,
\??\R:\dropbox\MS\Dup\original\Dup.csv)
 1182  698393 [main] ls 3688 symlink_info::check: not a symlink
  410  698803 [main] ls 3688 symlink_info::check: 0 = symlink.check
(R:\dropbox\MS\Dup\original\Dup.csv, 0x22B440) (0x4022)
   30  698833 [main] ls 3688 path_conv::check:
this->path(R:\dropbox\MS\Dup\original\Dup.csv), has_acls(0)
   26  698859 [main] ls 3688 build_fh_pc: fh 0x612329D8
   27  698886 [main] ls 3688 fhandler_base::open:
(\??\R:\dropbox\MS\Dup\original\Dup.csv, 0x11)
  724  699610 [main] ls 3688 fhandler_base::set_flags: flags 0x11,
supplied_bin 0x1
   29  699639 [main] ls 3688 fhandler_base::set_flags: O_TEXT/O_BINARY set
in flags 0x1
   23  699662 [main] ls 3688 fhandler_base::set_flags: filemode set to binary
   24  699686 [main] ls 3688 fhandler_base::open: 0 = NtCreateFile (0x620,
20080, \??\R:\dropbox\MS\Dup\original\Dup.csv, io, NULL, 0, 7, 1, 4000,
NULL, 0)
   27  699713 [main] ls 3688 fhandler_base::open: 1 = fhandler_base::open
(\??\R:\dropbox\MS\Dup\original\Dup.csv, 0x11)
   34  699747 [main] ls 3688 fhandler_base::open_fs: 1 =
fhandler_disk_file::open (\??\R:\dropbox\MS\Dup\original\Dup.csv,
0x1)
39883  739630 [main] ls 3688 fhandler_base::fstat_helper: 0 = fstat
(\??\R:\dropbox\MS\Dup\original\Dup.csv, 0x22C670) st_atime=4BFFF67E
st_size=615923, st_mode=0x81A4, st_ino=-5315584508382449066, sizeof=96
   35  739665 [main] ls 3688 fhandler_base::close: closing
'/cygdrive/r/dropbox/MS/Dup/original/Dup.csv' handle 0x620
  968  740633 [main] ls 3688 acl_worker: 4 = acl
(/cygdrive/r/dropbox/MS/Dup/original/Dup.csv)


> On Wed, Jun 02, 2010 at 10:46:03AM -0700, Christopher Wingert wrote:
>>Thanks for the pointer, I just gave it a whirl, it actually didn't make
>>much of a difference.
>>
>>I am going to start looking into making a patch.
>
> Let me point out that the kind of slowdown that you are seeing is not
> something that I would consider acceptable.  So what I'd like to see is
> an analysis of *why* Cygwin is so slow rather than a band-aid which uses
> an environment variable or different api to work around a real problem.
>
> cgf
>
> --
> Problem reports:   http://cygwin.com/problems.html
> FAQ:   http://cygwin.com/faq/
> Documentation: http://cygwin.com/docs.html
> Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
>
>



--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Performance and stat()

[Christopher Wingert]

Thanks for the pointer, I just gave it a whirl, it actually didn't make
much of a difference.

I am going to start looking into making a patch.

Chris



> On Jun  1 14:42, Christopher Wingert wrote:
>> I think there are a lot of use cases where the extra information (ACL
>> information *I assume* is the majority of the problem) is unnecessary.
>> For most of the applications filename, size, and the three dates are all
>> that is necessary.  So cygwin stat is overkill.  So if I can tell the
>> emulation layer (via an environment flag) or the actually utility
>> (bash/ls/make/find/du) via a command line switch, I think I can save a
>> lot
>> of time waiting.
>>
>> Just to highlight how bad this problem is.  I have a network drive with
>> 681 sub directories and approximately 90k files.  A time comparison for
>> getting directory information as follows:
>>
>> *DOS "dir /s" takes 17 seconds.
>> *Cygwin "ls -lR" takes 5950 seconds (that's almost two hours).
>> *msls -lR takes 55 seconds.
>> *myls (see code below) takes 7 seconds.
>>
>> Each test was done twice and after a reboot to make sure there was no
>> caching involved.
>>
>> To be clear, Cygwin ls is 850X slower.
>
> Did you try to mount the network drive with the "noacl" mount option?
> That skips requesting the owner/group information.
>
>
> Corinna
>
> --
> Corinna Vinschen  Please, send mails regarding Cygwin to
> Cygwin Project Co-Leader  cygwin AT cygwin DOT com
> Red Hat
>
> --
> Problem reports:   http://cygwin.com/problems.html
> FAQ:   http://cygwin.com/faq/
> Documentation: http://cygwin.com/docs.html
> Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
>
>



--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Performance and stat()

[Christopher Wingert]

That's fine, can you propose something that is acceptable?

BTW, who does this patch need to pass muster with?  The only maintainer I
could find is Dave Korn.

Thanks,

Chris


> On 6/1/2010 5:42 PM, Christopher Wingert wrote:
>> I think there are a lot of use cases where the extra information (ACL
>> information *I assume* is the majority of the problem) is unnecessary.
>> For most of the applications filename, size, and the three dates are all
>> that is necessary.  So cygwin stat is overkill.  So if I can tell the
>> emulation layer (via an environment flag) or the actually utility
>> (bash/ls/make/find/du) via a command line switch, I think I can save a
>> lot
>> of time waiting.
>>
>> Just to highlight how bad this problem is.  I have a network drive with
>> 681 sub directories and approximately 90k files.  A time comparison for
>> getting directory information as follows:
>>
>> *DOS "dir /s" takes 17 seconds.
>> *Cygwin "ls -lR" takes 5950 seconds (that's almost two hours).
>> *msls -lR takes 55 seconds.
>> *myls (see code below) takes 7 seconds.
>>
>> Each test was done twice and after a reboot to make sure there was no
>> caching involved.
>>
>> To be clear, Cygwin ls is 850X slower.
>
> Thanks for this information and perhaps I'm wrong but I don't believe
> anyone in this thread thought that you were lying when you noted issues
> with the performance of stat(). ;-)  But providing a variant of stat()
> along the lines of what you propose above is not practical for all the
> reasons already stated.  I believe we would all like stat() to be
> quicker but we need something that solves the root of the problem and
> not partial, hidden solutions that are problematic to use.
>
> --
> Larry Hall  http://www.rfk.com
> RFK Partners, Inc.  (508) 893-9779 - RFK Office
> 216 Dalton Rd.  (508) 893-9889 - FAX
> Holliston, MA 01746
>
> _
>
> A: Yes.
>> Q: Are you sure?
>>> A: Because it reverses the logical flow of conversation.
>>>> Q: Why is top posting annoying in email?
>
> --
> Problem reports:   http://cygwin.com/problems.html
> FAQ:   http://cygwin.com/faq/
> Documentation: http://cygwin.com/docs.html
> Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
>
>



--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Performance and stat()

[Christopher Wingert]

I think there are a lot of use cases where the extra information (ACL
information *I assume* is the majority of the problem) is unnecessary. 
For most of the applications filename, size, and the three dates are all
that is necessary.  So cygwin stat is overkill.  So if I can tell the
emulation layer (via an environment flag) or the actually utility
(bash/ls/make/find/du) via a command line switch, I think I can save a lot
of time waiting.

Just to highlight how bad this problem is.  I have a network drive with
681 sub directories and approximately 90k files.  A time comparison for
getting directory information as follows:

*DOS "dir /s" takes 17 seconds.
*Cygwin "ls -lR" takes 5950 seconds (that's almost two hours).
*msls -lR takes 55 seconds.
*myls (see code below) takes 7 seconds.

Each test was done twice and after a reboot to make sure there was no
caching involved.

To be clear, Cygwin ls is 850X slower.

msls can be retrieved here http://utools.com/msls.htm

myls is as follows:

int main( int arc, char *argv[] )
{
   dodir( "p:\\dl" );
}

int dodir( char *dir )
{
   WIN32_FIND_DATA findData;
   HANDLE f;
   char spec[ 1024 ];
   char fname[ 1024 ];

   printf( "DIR %s\n", dir );
   sprintf( spec, "%s\\*.*", dir );

   f = FindFirstFile( spec, &findData );
   do
   {
  sprintf( fname, "%s\\%s", dir, findData.cFileName );
  if ( findData.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY )
  {
 if ( ( strcmp( findData.cFileName, "." ) != 0 ) && ( strcmp(
findData.cFileName, ".." ) != 0 ) )
 {
dodir( fname );
 }
  }
  else
  {
 printf( "%s %d\n", fname, findData.nFileSizeLow );
  }
   }
   while( FindNextFile( f, &findData ) );
   FindClose( f );
}




> Christopher Wingert schrieb:
>> I assume POSIX compatibility.  However, I bet there are cases where one
>> can sacrifice compatibility for performance (configurable with an
>> environment flag of course).
>>
>> See
>> http://marc.info/?l=git&m=122278284210941
>> for an example.
>
> This git do_stat is for only meant for a 50% implementation of relative
> paths known before, and therefore onyl useful to certain apps, but it
> can never be useful for the cygwin1.dll layer, because cygwin has to
> provide the POSIX compat. layer, and not 50% cut-throughs for apps which
> don't need the other 50%. ACL, mounts, symlinks, inode.
>
> A better chaching stat or an cygwin extension for relative deeper only
> would be possible, but a better caching stat would need more memory and
> sacrifice speed for the first stat.
> A fast relative stat is very unlikely to be #IFDEF's in some apps just
> for us. So it's more likely that those apps which might need it, come up
> with their own 50% less, but 50% faster bits, as git did.
>
>>> On Sun, May 30, 2010 at 08:54:10AM -0700, Christopher Wingert wrote:
>>>> I was looking into speeding up stat() performance.  More specifically
>>>> bash, ls, test, stat performance.  I've seen the subject come up
>>>> before.
>>>> Git recently implemented a native Win32 work around.  Are there any
>>>> cygwin
>>>> patches around?
>>>
>>> If there was a way to make stat() faster why wouldn't it be in the
>>> source
>>> code already?
> --
> Reini Urban
> http://phpwiki.org/  http://murbreak.at/
>
> --
> Problem reports:   http://cygwin.com/problems.html
> FAQ:   http://cygwin.com/faq/
> Documentation: http://cygwin.com/docs.html
> Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
>
>



--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Re: Cygwin Performance and stat()

[Christopher Wingert]

I assume POSIX compatibility.  However, I bet there are cases where one
can sacrifice compatibility for performance (configurable with an
environment flag of course).

See

http://marc.info/?l=git&m=122278284210941

for an example.


> On Sun, May 30, 2010 at 08:54:10AM -0700, Christopher Wingert wrote:

>>I was looking into speeding up stat() performance.  More specifically
>>bash, ls, test, stat performance.  I've seen the subject come up before.
>>Git recently implemented a native Win32 work around.  Are there any
>> cygwin
>>patches around?
>
> If there was a way to make stat() faster why wouldn't it be in the source
> code already?
>
> cgf
>
> --
> Problem reports:   http://cygwin.com/problems.html
> FAQ:   http://cygwin.com/faq/
> Documentation: http://cygwin.com/docs.html
> Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
>
>



--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Cygwin Performance and stat()

[Christopher Wingert]

I was looking into speeding up stat() performance.  More specifically
bash, ls, test, stat performance.  I've seen the subject come up before. 
Git recently implemented a native Win32 work around.  Are there any cygwin
patches around?

Thanks,

Chris


--
Problem reports:   http://cygwin.com/problems.html
FAQ:   http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple

Re: -mwindows and hour glass

[Christopher Wingert]

Ummm, Ok, thanks for the response, but the program is...


#include 

int main( int argc, char *argv[] )
{
   sleep( 5 );
}



# gcc t.c -o t -Wl,--subsystem,windows

Starting this from explorer results in a 5 second busy hour glass and (of
course), explorer is unresponsive until the program ends.

Thanks


-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Christopher Wingert wrote:
> I am trying to write a small program that does not pop up a window.  I
> found the -mwindow option.  It does not pop up a window, but while the
> program is running explorer shows the hour glass busy cursor until the
> application ends.  I am assuming that explorer is waiting for a window to
> appear.  Is there a way to signal explorer that my program is running?
>
> Thanks

no its about what resources you are accessing. system resources provoke
a (in my knowledge) an arrow+hourglass and file resources provoke an
hourglass. I know for a fact its about what you are doing.
- --
Just a Thought
Morgan Gangwere

For those who want my PGP key:
http://pengunassasin.is-a-geek.com/pgpKey.html

*** Wisdom for the day ***
* Dont rawquote - it gives   *
*  spammers free bait!   *
**
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGdXWbCF9T/dUsmAgRAu8PAJ9mAdXJb7HVSzFyOverk8MEyUrgaACg4X7g
w8za1PrSdxV7Q5MaQ5F0UOk=
=xTi+
-END PGP SIGNATURE-




--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/

-mwindows and hour glass

[Christopher Wingert]

I am trying to write a small program that does not pop up a window.  I
found the -mwindow option.  It does not pop up a window, but while the
program is running explorer shows the hour glass busy cursor until the
application ends.  I am assuming that explorer is waiting for a window to
appear.  Is there a way to signal explorer that my program is running?

Thanks


--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/

Building cygwin src dll help

[Christopher Wingert]

I'm building the cygwin source and having a bit of trouble with the
following.

/bin/sh ../../.././winsup/cygwin/speclib
/home/cwingert/cygwin-1.5.24-2/i686-pc-cygwin/winsup/cygwin/libpthread.a
"nm" "ar"
/home/cwingert/cygwin-1.5.24-2/i686-pc-cygwin/winsup/cygwin/libcygwin.a
pthread.o thread.o
 in archive0.o
 in archive00048.o
 in archive00432.o
 in archive00433.o
 in archive01182.o
 in archive01183.o
 in archive01184.o
 in archive01185.o
 in archive01186.o
 in archive01187.o
 in archive01188.o
 in archive01189.o
 in archive01190.o
 in archive01191.o
 in archive01192.o
 in archive01193.o
 in archive01194.o
 in archive01195.o
 in archive01196.o
 in archive01197.o
 in archive01198.o
 in archive01199.o
 in archive01200.o
 in archive01201.o
 in archive01202.o
 in archive01203.o
 in archive01204.o
 in archive01205.o
 in archive01206.o
 in archive01207.o
 in archive01208.o
 in archive01209.o
 in archive01210.o
 in archive01211.o
 in archive01212.o
 in archive01213.o
 in archive01214.o
 in archive01215.o
 in archive01216.o
 in archive01217.o
 in archive01218.o
 in archive01219.o
 in archive01220.o
 in archive01221.o
 in archive01222.o
 in archive01223.o
 in archive01224.o
 in archive01225.o
 in archive01226.o
 in archive01227.o
 in archive01228.o
 in archive01229.o
 in archive01230.o
 in archive01231.o
 in archive01232.o
 in archive01233.o
 in archive01234.o
 in archive01235.o
 in archive01236.o
 in archive01237.o
 in archive01238.o
 in archive01239.o
 in archive01240.o
 in archive01241.o
 in archive01242.o
 in archive01243.o
 in archive01244.o
 in archive01245.o
 in archive01246.o
 in archive01247.o
 in archive01248.o
 in archive01249.o
 in archive01250.o
 in archive01251.o
 in archive01252.o
 in archive01253.o
 in archive01254.o
 in archive01255.o
 in archive01256.o
 in archive01257.o
 in archive01258.o
 in archive01331.o
 in archive01332.o
 in archive01333.o
 in archive01334.o
 in archive01335.o
 in archive01336.o
 in archive01337.o
 in archive01338.o
 in archive01339.o
 in archive01625.o
ar: *.o: No such file or directory
Can't open
/home/cwingert/cygwin-1.5.24-2/i686-pc-cygwin/winsup/cygwin/libpthread.a:
No such file or directory.

Adding the -v option to the speclib command makes things work, but I don't
really understand what speclib is trying to do nor do I understand why its
not already there.

Thanks for any pointers.




--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/

Re: bash completion slow

[Christopher Wingert]

Hi Eric,

Thanks for the reply.

The triple stat() issue I am talking about is a thread/patch that I read
from the 2002 timeframe.  I don't think the patch got integrated back tho.
http://sourceware.org/ml/cygwin/2002-05/msg01600.html
There was another thread that I can't find right now.

Actually Linux readdir doesn't provide any clues about the file either, it
returns the dirent structure.

However, this really opens my eyes as to the problem :

> operations, the result of which is just discarded).  With directories in
> particular, part of the slowdown is that stat()'s st_nlink field is
> populated by counting how many files appear in that sub-directory, even
> though readline didn't care about that.

I had the same issue with a tab in /cygdrive/c/ (obviously due to the
nasty-ness of the Windows directory).  My directory of 300 subdirectories
has umpteen files in the subdirectories.

What about turning this subdirectory parsing off by feeding a -1 in
st_nlink in Cygwin?  How could I propose such a change for cygwin?

Thanks again


> According to Christopher Wingert on 3/25/2007 8:35 AM:
> > Hi
> >
> > I have a directory where there are about 300 subdirectories in it. 
When I
> > hit tab, not only is the delay to the "Display all 300 possibilities..."
> > is slow but the screw draw after answer y is almost like a 300 baud
modem.
> >  I've read on these lists about the triple stat issue with cygwin, is
this
> > the cause of this problem?  Any solution to this problem?

> What 'triple stat' issue are you referring to?  Part of the problem is
> that tab completion relies on determining file type (directory vs. regular
> file, etc.), but unlike Linux, cygwin readdir() does not convey file type
> (because it is a relatively expensive operation for some file types, such
> as symlinks vs. block and character devices, involving opening the file,
> which would penalize readdir for other cases), while stat() conveys more
> information than readline cares about (involving even MORE expensive
> operations, the result of which is just discarded).  With directories in
> particular, part of the slowdown is that stat()'s st_nlink field is
> populated by counting how many files appear in that sub-directory, even
> though readline didn't care about that.  And part of the problem is that
> Windows just STINKS at enumerating directory contents.  Overall, it means
> that where there are 300 subdirectories, and readline is stat()ing all 300
> of them, cygwin ends up using Windows slow traversal of all 300
> directories.

> If readdir were to provide d_type in struct dirent, even populating it
> only with DT_DIR vs. DT_UNKNOWN, then readline's tab completion and
> several of the coreutils would gain some speedups, merely by being able to
> distinguish directories from other files without having to stat()
> directories.  But SHTDI and write such a patch, and prove that it doesn't
> penalize other uses of readdir that could care less about file type.


--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/

bash completion slow

[Christopher Wingert]

Hi

I have a directory where there are about 300 subdirectories in it.  When I
hit tab, not only is the delay to the "Display all 300 possibilities..."
is slow but the screw draw after answer y is almost like a 300 baud modem.
 I've read on these lists about the triple stat issue with cygwin, is this
the cause of this problem?  Any solution to this problem?

I am running Win XP SP2.

Thanks


--
Unsubscribe info:  http://cygwin.com/ml/#unsubscribe-simple
Problem reports:   http://cygwin.com/problems.html
Documentation: http://cygwin.com/docs.html
FAQ:   http://cygwin.com/faq/