Re: Cygwin Performance and stat()

2010-06-07 Thread Corinna Vinschen
On Jun 5 08:43, Christopher Wingert wrote: So... the person who cared to improve his/her/its code would say, Well we use NTOpenFile() because it does the blah blah extra functionality that FindNextFile()/GetFileAttributes() do not. Then we could look to other Win32 APIs to try to achieve

Re: Cygwin Performance and stat()

2010-06-07 Thread Corinna Vinschen
On Jun 6 19:28, Christopher Faylor wrote: On Mon, Jun 07, 2010 at 12:12:36AM +0200, Matthias Andree wrote: Meaning that: even if I'm only a Cygwin user, and I'm sometimes disappointed by how slow it is, too, I'm sort of convinced there isn't a cheaper way to get all the required

Re: Cygwin Performance and stat()

2010-06-06 Thread Haojun Bao
Dave Korn dave.korn.cyg...@gmail.com writes: On 04/06/2010 18:33, Christopher Wingert wrote: [quit top-posting] Now you are my mom too? No, I am. Now quit playing with all your new friends and dinner! cheers, Your Mom OK. I'm a Chinese, and I'm laughing out loud with this

Re: Cygwin Performance and stat()

2010-06-06 Thread Matthias Andree
Am 06.06.2010, 01:16 Uhr, schrieb Christopher Wingert: I do think out loud with my team. You are not on it. Agreed! You would rather spend your time ridiculing any possible solution. If only there had been a solution, rather than a loose collection of names (I wouldn't even dare call

Re: Cygwin Performance and stat()

2010-06-06 Thread Christopher Faylor
On Mon, Jun 07, 2010 at 12:12:36AM +0200, Matthias Andree wrote: Meaning that: even if I'm only a Cygwin user, and I'm sometimes disappointed by how slow it is, too, I'm sort of convinced there isn't a cheaper way to get all the required information. I'm disappointed in Cygwin's slowness too.

Re: Cygwin Performance and stat()

2010-06-05 Thread Christopher Faylor
On Sat, Jun 05, 2010 at 01:24:29AM -0400, Christopher Faylor wrote: On Fri, Jun 04, 2010 at 02:37:01PM -0700, Christopher Wingert wrote: On Fri, Jun 04, 2010 at 03:16:43PM -0600, Eric Blake wrote: On 06/04/2010 03:14 PM, Christopher Wingert wrote: Agreed, I would like to make a global change,

Re: Cygwin Performance and stat()

2010-06-05 Thread Christopher Wingert
Can't really parse that sentence. Then load your English parser. I haven't detected any picking on but then I can't claim to have written the fhandler* code anymore Corinna has rewritten most of it. I do know that if you want to be taken seriously you really need to send a concrete

Re: Cygwin Performance and stat()

2010-06-05 Thread Christopher Faylor
On Sat, Jun 05, 2010 at 08:43:50AM -0700, Christopher Wingert wrote: On Sat, 5 Jun 2010 01:24:29 -0400 Christopher Faylor wrote: On 06/04/2010 03:14 PM, Christopher Wingert wrote: He is? Holy crap, he is more helpful with his sarcasm and doubt than anything else. Can't really parse that

Re: Cygwin Performance and stat()

2010-06-05 Thread Christopher Wingert
I do think out loud with my team. You are not on it. Agreed! You would rather spend your time ridiculing any possible solution. This is what lead to my initial reluctance to do any patch for Cygwin software. Good Luck with your inferior product. Chris -- Problem reports:

Re: Cygwin Performance and stat()

2010-06-05 Thread Dave Korn
On 04/06/2010 18:33, Christopher Wingert wrote: [quit top-posting] Now you are my mom too? No, I am. Now quit playing with all your new friends and get home for your dinner! cheers, Your Mom -- Problem reports: http://cygwin.com/problems.html FAQ:

Re: Cygwin Performance and stat()

2010-06-05 Thread Christopher Faylor
On Sat, Jun 05, 2010 at 04:16:42PM -0700, Christopher Wingert wrote: I do think out loud with my team. You are not on it. Agreed! You would rather spend your time ridiculing any possible solution. This is what lead to my initial reluctance to do any patch for Cygwin software. Good Luck with

Re: Cygwin Performance and stat()

2010-06-04 Thread Christopher Wingert
So here is an example of a performance gain by not using cygwin stat(). I did this patch in about an hour (with the help of some git code), so I wouldn't recommend it for any production use. On a dry run rsync from my local drive to my NAS (105GB, 34k files, 4k directories). The current release

Re: Cygwin Performance and stat()

2010-06-04 Thread Larry Hall (Cygwin)
On 6/4/2010 12:37 PM, Christopher Wingert wrote: So here is an example of a performance gain by not using cygwin stat(). I did this patch in about an hour (with the help of some git code), so I wouldn't recommend it for any production use. On a dry run rsync from my local drive to my NAS

Re: Cygwin Performance and stat()

2010-06-04 Thread Christopher Wingert
Again, not production, this was to highlight a point and to enhance (I would assume) the most common use case (file to file rsync). Plenty of solutions for a good patch, complete the Windows version of stat() or call the Cygwin version if the Window's version fails. Since I don't have any

Re: Cygwin Performance and stat()

2010-06-04 Thread Eric Blake
[quit top-posting] On 06/04/2010 11:07 AM, Christopher Wingert wrote: Again, not production, this was to highlight a point and to enhance (I would assume) the most common use case (file to file rsync). Plenty of solutions for a good patch, complete the Windows version of stat() or call the

Re: Cygwin Performance and stat()

2010-06-04 Thread Christopher Wingert
[quit top-posting] Now you are my mom too? That's where you're wrong. Any patch you write that is technically sound and shows a measurable improvement will most likely be accepted. Then you shouldn't have Cygwin's front line technical spokesman saying things such as: If there was a way to

Re: Cygwin Performance and stat()

2010-06-04 Thread Christopher Faylor
On Fri, Jun 04, 2010 at 10:07:58AM -0700, Christopher Wingert wrote: Again, not production, this was to highlight a point and to enhance (I would assume) the most common use case (file to file rsync). Plenty of solutions for a good patch, complete the Windows version of stat() or call the Cygwin

Re: Cygwin Performance and stat()

2010-06-04 Thread Christopher Faylor
On Fri, Jun 04, 2010 at 10:33:47AM -0700, Christopher Wingert wrote: Eric Blake wrote: [quit top-posting] Now you are my mom too? too? I don't recall any other responses from Eric to you. That's where you're wrong. Any patch you write that is technically sound and shows a measurable

Re: Cygwin Performance and stat()

2010-06-04 Thread Larry Hall (Cygwin)
On 6/4/2010 2:20 PM, Christopher Faylor wrote: But providing a variant of stat() along the lines of what you propose above is not practical for all the reasons already stated. This is not something that I said. That was actually Larry Hall. Heh. Who needs him anyway! Just to clarify, this

Re: Cygwin Performance and stat()

2010-06-04 Thread Eliot Moss
I don't think there's an objection here to patching *rsync* specially in the cygwin environment -- that would be between you and the rsync port maintainer. The issue is whether or not to make a more general change to cygwin itself, and cgf is just saying that that's hard to do. Conceivably we

Re: Cygwin Performance and stat()

2010-06-04 Thread Andy Koppe
On 4 June 2010 19:50, Eliot Moss wrote: I don't think there's an objection here to patching *rsync* specially in the cygwin environment -- that would be between you and the rsync port maintainer. The issue is whether or not to make a more general change to cygwin itself, and cgf is just

Re: Cygwin Performance and stat()

2010-06-04 Thread Christopher Wingert
Agreed, I would like to make a global change, however, unless I can talk to the current maintainer of the fhandler* functions, it seems illogical for me to change them (as I have about a week of cygwin dll experience). Also my interest in performance is limited to a very certain subset of

Re: Cygwin Performance and stat()

2010-06-04 Thread Christopher Wingert
I actually wrote my own rsync in AutoIt, but it is also limited. However, I thought it would a nice addition to speed up Cygwin. As I have been using it for about 10 years and made my transition from Linux to Windows so pleasant. On 4 June 2010 19:50, Eliot Moss wrote: I don't think there's

Re: Cygwin Performance and stat()

2010-06-04 Thread Christopher Wingert
See further down the thread, the right solution is to impact ALL cygwin executables, but I don't have the experience in the dll to make those changes. On 6/4/2010 2:20 PM, Christopher Faylor wrote: But providing a variant of stat() along the lines of what you propose above is not practical

Re: Cygwin Performance and stat()

2010-06-04 Thread Eric Blake
On 06/04/2010 03:14 PM, Christopher Wingert wrote: Agreed, I would like to make a global change, however, unless I can talk to the current maintainer of the fhandler* functions, it seems illogical for me to change them (as I have about a week of cygwin dll experience). You ARE talking to the

Re: Cygwin Performance and stat()

2010-06-04 Thread Christopher Wingert
He is? Holy crap, he is more helpful with his sarcasm and doubt than anything else. However, it does explains his tone, given that I am picking on his code. I see your reference to an acronym regarding top posting, but not a policy. In fact your reference is specifically listed under another

Re: Cygwin Performance and stat()

2010-06-04 Thread Christopher Faylor
On Fri, Jun 04, 2010 at 02:37:01PM -0700, Christopher Wingert wrote: On Fri, Jun 04, 2010 at 03:16:43PM -0600, Eric Blake wrote: On 06/04/2010 03:14 PM, Christopher Wingert wrote: Agreed, I would like to make a global change, however, unless I can talk to the current maintainer of the fhandler*

Re: Cygwin Performance and stat()

2010-06-03 Thread Christopher Wingert
Using strace I was able to look at some of the functions that are enumerated by debugging calls. The trace below is done by ls.exe for each file (approximately 95k files @ 88 mSecs/file), approximately 40 mSecs are spent in lstat64() and another 47 mSecs are spent in getacl(). It also *looks*

Re: Cygwin Performance and stat()

2010-06-03 Thread Christopher Faylor
On Thu, Jun 03, 2010 at 10:35:55AM -0700, Christopher Wingert wrote: Using strace I was able to look at some of the functions that are enumerated by debugging calls. The trace below is done by ls.exe for each file (approximately 95k files @ 88 mSecs/file), approximately 40 mSecs are spent in

Re: Cygwin Performance and stat()

2010-06-03 Thread Christopher Wingert
On Thu, Jun 03, 2010 at 10:35:55AM -0700, Christopher Wingert wrote: Using strace I was able to look at some of the functions that are enumerated by debugging calls. The trace below is done by ls.exe for each file (approximately 95k files @ 88 mSecs/file), approximately 40 mSecs are spent in

Re: Cygwin Performance and stat()

2010-06-03 Thread Christopher Faylor
On Thu, Jun 03, 2010 at 05:32:46PM -0700, Christopher Wingert wrote: On Thu, Jun 03, 2010 at 10:35:55AM -0700, Christopher Wingert wrote: Using strace I was able to look at some of the functions that are enumerated by debugging calls. The trace below is done by ls.exe for each file

Re: Cygwin Performance and stat()

2010-06-03 Thread Christopher Wingert
On Thu, Jun 03, 2010 at 05:32:46PM -0700, Christopher Wingert wrote: Yeah, that's what I thought you were doing. Given that the timestamps don't indicate elapsed time of function X, it's not always possible to figure out how long a function takes by subtracting. Subtracting timestamps shows

Re: Cygwin Performance and stat()

2010-06-03 Thread Christopher Faylor
On Thu, Jun 03, 2010 at 07:57:31PM -0700, Christopher Wingert wrote: On Thu, Jun 03, 2010 at 05:32:46PM -0700, Christopher Wingert wrote: Yeah, that's what I thought you were doing. Given that the timestamps don't indicate elapsed time of function X, it's not always possible to figure out how

Re: Cygwin Performance and stat()

2010-06-02 Thread Corinna Vinschen
On Jun 1 14:42, Christopher Wingert wrote: I think there are a lot of use cases where the extra information (ACL information *I assume* is the majority of the problem) is unnecessary. For most of the applications filename, size, and the three dates are all that is necessary. So cygwin stat

Re: Cygwin Performance and stat()

2010-06-02 Thread Christopher Wingert
Thanks for the pointer, I just gave it a whirl, it actually didn't make much of a difference. I am going to start looking into making a patch. Chris On Jun 1 14:42, Christopher Wingert wrote: I think there are a lot of use cases where the extra information (ACL information *I assume* is

Re: Cygwin Performance and stat()

2010-06-02 Thread Christopher Faylor
On Wed, Jun 02, 2010 at 10:46:03AM -0700, Christopher Wingert wrote: Thanks for the pointer, I just gave it a whirl, it actually didn't make much of a difference. I am going to start looking into making a patch. Let me point out that the kind of slowdown that you are seeing is not something that

Re: Cygwin Performance and stat()

2010-06-01 Thread Christopher Wingert
I think there are a lot of use cases where the extra information (ACL information *I assume* is the majority of the problem) is unnecessary. For most of the applications filename, size, and the three dates are all that is necessary. So cygwin stat is overkill. So if I can tell the emulation

Re: Cygwin Performance and stat()

2010-06-01 Thread Larry Hall (Cygwin)
On 6/1/2010 5:42 PM, Christopher Wingert wrote: I think there are a lot of use cases where the extra information (ACL information *I assume* is the majority of the problem) is unnecessary. For most of the applications filename, size, and the three dates are all that is necessary. So cygwin stat

Re: Cygwin Performance and stat()

2010-06-01 Thread Eliot Moss
On 6/1/2010 6:18 PM, Larry Hall (Cygwin) wrote: Thanks for this information and perhaps I'm wrong but I don't believe anyone in this thread thought that you were lying when you noted issues with the performance of stat(). ;-) But providing a variant of stat() along the lines of what you propose

Re: Cygwin Performance and stat()

2010-06-01 Thread Christopher Wingert
That's fine, can you propose something that is acceptable? BTW, who does this patch need to pass muster with? The only maintainer I could find is Dave Korn. Thanks, Chris On 6/1/2010 5:42 PM, Christopher Wingert wrote: I think there are a lot of use cases where the extra information (ACL

Re: Cygwin Performance and stat()

2010-06-01 Thread Larry Hall (Cygwin)
On 6/1/2010 8:06 PM, Christopher Wingert wrote: That's fine, can you propose something that is acceptable? Actually no, I'm no visionary here. It's not clear to me how to transparently determine what fields provided by stat() are used by a particular application. I suppose that it's possible

Re: Cygwin Performance and stat()

2010-05-31 Thread Reini Urban
Christopher Wingert schrieb: I assume POSIX compatibility. However, I bet there are cases where one can sacrifice compatibility for performance (configurable with an environment flag of course). See http://marc.info/?l=gitm=122278284210941 for an example. This git do_stat is for only meant

Cygwin Performance and stat()

2010-05-30 Thread Christopher Wingert
I was looking into speeding up stat() performance. More specifically bash, ls, test, stat performance. I've seen the subject come up before. Git recently implemented a native Win32 work around. Are there any cygwin patches around? Thanks, Chris -- Problem reports:

Re: Cygwin Performance and stat()

2010-05-30 Thread Christopher Faylor
On Sun, May 30, 2010 at 08:54:10AM -0700, Christopher Wingert wrote: I was looking into speeding up stat() performance. More specifically bash, ls, test, stat performance. I've seen the subject come up before. Git recently implemented a native Win32 work around. Are there any cygwin patches

Re: Cygwin Performance and stat()

2010-05-30 Thread Christopher Wingert
I assume POSIX compatibility. However, I bet there are cases where one can sacrifice compatibility for performance (configurable with an environment flag of course). See http://marc.info/?l=gitm=122278284210941 for an example. On Sun, May 30, 2010 at 08:54:10AM -0700, Christopher Wingert

Re: Cygwin Performance and stat()

2010-05-30 Thread Christopher Faylor
On Sun, May 30, 2010 at 12:51:31PM -0700, Christopher Wingert wrote: I assume POSIX compatibility. However, I bet there are cases where one can sacrifice compatibility for performance (configurable with an environment flag of course). See http://marc.info/?l=gitm=122278284210941 for an

Re: Cygwin Performance and stat()

2010-05-30 Thread NightStrike
On Sun, May 30, 2010 at 1:07 PM, Christopher Faylor cgf-use-the-mailinglist-ple...@cygwin.com wrote: On Sun, May 30, 2010 at 08:54:10AM -0700, Christopher Wingert wrote: I was looking into speeding up stat() performance.  More specifically bash, ls, test, stat performance.  I've seen the subject

Re: Cygwin Performance and stat()

2010-05-30 Thread Christopher Faylor
On Sun, May 30, 2010 at 05:03:46PM -0400, NightStrike wrote: On Sun, May 30, 2010 at 1:07 PM, Christopher Faylor wrote: On Sun, May 30, 2010 at 08:54:10AM -0700, Christopher Wingert wrote: I was looking into speeding up stat() performance. ?More specifically bash, ls, test, stat performance.

Re: Cygwin Performance and stat()

2010-05-30 Thread Christian Franke
Christopher Faylor wrote: On Sun, May 30, 2010 at 12:51:31PM -0700, Christopher Wingert wrote: I assume POSIX compatibility. However, I bet there are cases where one can sacrifice compatibility for performance (configurable with an environment flag of course). The problem is that

Re: Cygwin Performance and stat()

2010-05-30 Thread Dave Korn
On 30/05/2010 22:39, Christopher Faylor wrote: If someone is ingenuous enough to make an improvement it's hard to believe that they wouldn't be ingenuous enough to send a patch to cygwin-patches. No, it isn't. (I'm assuming you meant ingenious rather than ingenuous, because it doesn't make

Re: Cygwin Performance and stat()

2010-05-30 Thread Christopher Faylor
On Mon, May 31, 2010 at 02:19:52AM +0100, Dave Korn wrote: On Sun, May 30, 2010 at 05:03:46PM -0400, NightStrike wrote: There's always room for ingenuity and improvements, isn't there? On 30/05/2010 22:39, Christopher Faylor wrote: If someone is ingenuous enough to make an improvement it's hard