Re: Checkout really slow in Windows with lots of files in one directory

2011-02-03 Thread Nick
On Wed, 2011-02-02 at 07:52 -0500, Mark Phippard wrote: > On Wed, Feb 2, 2011 at 7:41 AM, Geoff Rowell wrote: > > On Wed, Feb 2, 2011 at 4:09 AM, Nick wrote: > >> On Tue, 2011-02-01 at 13:00 -0500, Mark Phippard wrote: > >> > >> On Wed, Jan 26, 2011 at 9:28 AM, Neil Bird wrote: > >> > >>> We ha

Re: Checkout really slow in Windows with lots of files in one directory

2011-02-03 Thread Neil Bird
Around about 02/02/11 15:26, Stefan Sperling typed ... Please submit a patch against the 1.6.x branch. I will handle it. Will do, thanks. -- [neil@fnx ~]# rm -f .signature [neil@fnx ~]# ls -l .signature ls: .signature: No such file or directory [neil@fnx ~]# exit

Re: Checkout really slow in Windows with lots of files in one directory

2011-02-02 Thread OBones
Stefan Sperling wrote: On Wed, Feb 02, 2011 at 04:12:07PM +0100, OBones wrote: Neil Bird wrote: I couldn't use the version from trunk/1.7 as it differs too much. I will try to submit the patch for someone's perusal at some point, but I couldn't properly test is as for some reason m

Re: Checkout really slow in Windows with lots of files in one directory

2011-02-02 Thread Stefan Sperling
On Wed, Feb 02, 2011 at 04:12:07PM +0100, OBones wrote: > Neil Bird wrote: > > I couldn't use the version from trunk/1.7 as it differs too > >much. I will try to submit the patch for someone's perusal at > >some point, but I couldn't properly test is as for some reason my > >build of svn out of 1

RE: Checkout really slow in Windows with lots of files in one directory

2011-02-02 Thread Bob Archer
> On Wed, Feb 2, 2011 at 7:41 AM, Geoff Rowell > wrote: > > On Wed, Feb 2, 2011 at 4:09 AM, Nick > wrote: > >> On Tue, 2011-02-01 at 13:00 -0500, Mark Phippard wrote: > >> > >> On Wed, Jan 26, 2011 at 9:28 AM, Neil Bird > wrote: > >> > >>>  We have a graphics-oriented code-base that's auto-gener

Re: Checkout really slow in Windows with lots of files in one directory

2011-02-02 Thread Stefan Sperling
On Wed, Feb 02, 2011 at 03:04:22PM +, Neil Bird wrote: > I have since compiled up the latest 1.6.x branch, and patched > svn_io_open_unique_file3() to not call svn_io_open_uniquely_named() > with 'tempfile.tmp', & instead I cut'n'paste > svn_io_open_uniquely_named(), replacing the 1-9 loo

Re: Checkout really slow in Windows with lots of files in one directory

2011-02-02 Thread OBones
Neil Bird wrote: I couldn't use the version from trunk/1.7 as it differs too much. I will try to submit the patch for someone's perusal at some point, but I couldn't properly test is as for some reason my build of svn out of 1.6.x svn (even before modifying it) fails 'make check': all the

Re: Checkout really slow in Windows with lots of files in one directory

2011-02-02 Thread Neil Bird
Around about 01/02/11 18:00, Mark Phippard typed ... I created a folder with 5001 files in it ... maybe that is not enough? I think the issue is to do with properties, so you'd have to make sure all those files have props. (we use autoprops for mime-type/eol-style/header-expansion). IIR

Re: Checkout really slow in Windows with lots of files in one directory

2011-02-02 Thread Mark Phippard
On Wed, Feb 2, 2011 at 7:41 AM, Geoff Rowell wrote: > On Wed, Feb 2, 2011 at 4:09 AM, Nick wrote: >> On Tue, 2011-02-01 at 13:00 -0500, Mark Phippard wrote: >> >> On Wed, Jan 26, 2011 at 9:28 AM, Neil Bird wrote: >> >>>  We have a graphics-oriented code-base that's auto-generated and has >5000 >

Re: Checkout really slow in Windows with lots of files in one directory

2011-02-02 Thread Geoff Rowell
On Wed, Feb 2, 2011 at 4:09 AM, Nick wrote: > On Tue, 2011-02-01 at 13:00 -0500, Mark Phippard wrote: > > On Wed, Jan 26, 2011 at 9:28 AM, Neil Bird wrote: > >>  We have a graphics-oriented code-base that's auto-generated and has >5000 >> source files in one directory.  While I can check this out

Re: Checkout really slow in Windows with lots of files in one directory

2011-02-02 Thread Nick
On Tue, 2011-02-01 at 13:00 -0500, Mark Phippard wrote: > On Wed, Jan 26, 2011 at 9:28 AM, Neil Bird wrote: > > > We have a graphics-oriented code-base that's auto-generated and has >5000 > > source files in one directory. While I can check this out OK on Linux, > > we're seeing an unusable sl

Re: Checkout really slow in Windows with lots of files in one directory

2011-02-01 Thread Mark Phippard
On Wed, Jan 26, 2011 at 9:28 AM, Neil Bird wrote: >  We have a graphics-oriented code-base that's auto-generated and has >5000 > source files in one directory.  While I can check this out OK on Linux, > we're seeing an unusable slow-down on Windows XP (NTFS), both using Tortoise > directly, and a

Re: Checkout really slow in Windows with lots of files in one directory

2011-02-01 Thread Neil Bird
Around about 01/02/11 15:36, Neil Bird typed ... I've had no response from the email I sent him privately, so I may try that myself, and maybe then raise a ticket (or whatever system they use) to ask for the backport if it works. My bad, I just found Thunderbird has mis-threaded his reply. H

Re: Checkout really slow in Windows with lots of files in one directory

2011-02-01 Thread Neil Bird
Around about 28/01/11 04:32, Daniel Shahaf typed ... (I don't recall why the fixes on trunk weren't backported, but I imagine the threads linked from this thread say why.) The threads (that I've see) all peter out, with the last event being some saying a colleague was about to compile 1.6 wi

Re: Checkout really slow in Windows with lots of files in one directory

2011-02-01 Thread Neil Bird
Around about 27/01/11 12:51, Nico Kadel-Garcia typed ... Are you *normally* out to a CIFS shared location, or to local NTFS disk? Not ... as such. No, this was particularly to test the client code/OS to try to pin things down for this issue. -- [neil@fnx ~]# rm -f .signature [neil@fnx ~]#

Re: Checkout really slow in Windows with lots of files in one directory

2011-01-28 Thread Paul Holden
On 27 Jan 2011 08:23, "Neil Bird" wrote: > > The thread petered out with someone trying the 1.7 svn_io_open_unique_file impl. in a 1.6 build, but I can't see any feedback from that (so I'm afraid I'm going to cheekily Cc: the guys who posted that that was being tried!). > Hi, Yes - that was me.

Re: Checkout really slow in Windows with lots of files in one directory

2011-01-27 Thread Daniel Shahaf
This will probably be backported to 1.6.(x+1) if someone tells dev@ what revisions should be backported, or provides patches that we can apply. (I don't recall why the fixes on trunk weren't backported, but I imagine the threads linked from this thread say why.) Neil Bird wrote on Thu, Jan 27, 2

Re: Checkout really slow in Windows with lots of files in one directory

2011-01-27 Thread Nico Kadel-Garcia
On Wed, Jan 26, 2011 at 9:28 AM, Neil Bird wrote: > >  We have a graphics-oriented code-base that's auto-generated and has >5000 > source files in one directory.  While I can check this out OK on Linux, > we're seeing an unusable slow-down on Windows XP (NTFS), both using Tortoise > directly, and

Re: Checkout really slow in Windows with lots of files in one directory

2011-01-27 Thread Neil Bird
Around about 27/01/11 01:40, Johan Corveleyn typed ... http://svn.haxx.se/dev/archive-2010-04/0180.shtml Maybe someone else can comment more on this, and why it wasn't backported to 1.6 (or is it?). Excellent, yes, that would seem to be it. I did notice the huge number of counted .tmp file

Re: Checkout really slow in Windows with lots of files in one directory

2011-01-26 Thread Johan Corveleyn
On Wed, Jan 26, 2011 at 7:32 PM, Daniel Shahaf wrote: > Johan Corveleyn wrote on Wed, Jan 26, 2011 at 17:15:40 +0100: >> If I have more time, I'll try to search the archives some more. > > If it helps your code/mail grepping, I believe you're talking about > svn_io_open_uniquely_named(). Thanks,

Re: Checkout really slow in Windows with lots of files in one directory

2011-01-26 Thread Andy Levy
On Wed, Jan 26, 2011 at 11:59, Neil Bird wrote: > Around about 26/01/11 14:43, Andy Levy typed ... >> >> It's known and oft-lamented. NTFS just doesn't handle this scenario >> well - it's probably one of the reasons FSFS sharding was introduced >> (I'm speculating a bit here). > >  This is stuff t

Re: Checkout really slow in Windows with lots of files in one directory

2011-01-26 Thread Daniel Shahaf
Johan Corveleyn wrote on Wed, Jan 26, 2011 at 17:15:40 +0100: > If I have more time, I'll try to search the archives some more. If it helps your code/mail grepping, I believe you're talking about svn_io_open_uniquely_named(). (I found it by doing `fgrep '.%' subversion/libsvn_subr/*c`.)

Re: Checkout really slow in Windows with lots of files in one directory

2011-01-26 Thread Thorsten Schöning
Guten Tag Neil Bird, am Mittwoch, 26. Januar 2011 um 18:02 schrieben Sie: >When we do the checkout via tortoise, tortoise sits at 100% CPU load, and > I don't see anything else fighting it for time. This could be the cache processing the events from the shell about updated and new files. I of

RE: Checkout really slow in Windows with lots of files in one directory

2011-01-26 Thread Echlin, Jamie
>We see exactly the same problem on two diff. Windows PCs, > but I'll give ImDisk a bash in the morning for the sake of > experimentation. Have you tried Process Explorer? Look at the stack trace of one of the events when it's started slowing down. Make sure the symbol server is configured.

Re: Checkout really slow in Windows with lots of files in one directory

2011-01-26 Thread Neil Bird
Around about 26/01/11 15:30, Echlin, Jamie typed ... Perhaps your Master File Table is fragmented... In which case a defrag might help. We see exactly the same problem on two diff. Windows PCs, but I'll give ImDisk a bash in the morning for the sake of experimentation. And I recently def

Re: Checkout really slow in Windows with lots of files in one directory

2011-01-26 Thread Neil Bird
Around about 26/01/11 15:24, Stefan Sperling typed ... Don't run virus scanners on working copies. If you want to check for viruses in the repository, use a designated working copy and trigger a scan of changed files from the post-commit hook. The virus scanner excludes this particular direct

Re: Checkout really slow in Windows with lots of files in one directory

2011-01-26 Thread Neil Bird
Around about 26/01/11 14:43, Andy Levy typed ... It's known and oft-lamented. NTFS just doesn't handle this scenario well - it's probably one of the reasons FSFS sharding was introduced (I'm speculating a bit here). This is stuff that's currently in SourceSafe, which doesn't exhibit any obvi

Re: Checkout really slow in Windows with lots of files in one directory

2011-01-26 Thread Neil Bird
Around about 26/01/11 14:41, Campbell Allan typed ... If the code is auto generated would it be possible to generate it for each build? Interesting question; I shall have to ask it. I *think*, in actual fact, that it was *originally* generated, but has since been tweaked manually or with

Re: Checkout really slow in Windows with lots of files in one directory

2011-01-26 Thread Johan Corveleyn
On Wed, Jan 26, 2011 at 3:41 PM, Campbell Allan wrote: > > On Wednesday 26 Jan 2011, Neil Bird wrote: >>    We have a graphics-oriented code-base that's auto-generated and has >> >5000 source files in one directory.  While I can check this out OK on >> Linux, we're seeing an unusable slow-down on

RE: Checkout really slow in Windows with lots of files in one directory

2011-01-26 Thread Echlin, Jamie
On second thoughts my previous mail is related to when you have a massive number of sub-dirs in your working copy. I didn't properly read you mail, sorry. Perhaps your Master File Table is fragmented... In which case a defrag might help. ==

RE: Checkout really slow in Windows with lots of files in one directory

2011-01-26 Thread Echlin, Jamie
>I know that there's a negative speed difference on NTFS, > and that 1.7's WC-NG might make this better, but this is > getting near-logarithmically slower. There's good information about NTFS wrt subversion here: http://superuser.com/questions/15192/bad-ntfs-performance I found FAT was 10-2

Re: Checkout really slow in Windows with lots of files in one directory

2011-01-26 Thread Stefan Sperling
On Wed, Jan 26, 2011 at 09:43:15AM -0500, Andy Levy wrote: > NTFS just doesn't handle this scenario > well - it's probably one of the reasons FSFS sharding was introduced > (I'm speculating a bit here). IIRC that is correct. Sharding was introduced to prevent long-running readdir() system calls. I

Re: Checkout really slow in Windows with lots of files in one directory

2011-01-26 Thread Andy Levy
On Wed, Jan 26, 2011 at 09:28, Neil Bird wrote: > >  We have a graphics-oriented code-base that's auto-generated and has >5000 > source files in one directory.  While I can check this out OK on Linux, > we're seeing an unusable slow-down on Windows XP (NTFS), both using Tortoise > directly, and as

Re: Checkout really slow in Windows with lots of files in one directory

2011-01-26 Thread Ryan Schmidt
On Jan 26, 2011, at 08:28, Neil Bird wrote: > We have a graphics-oriented code-base that's auto-generated and has >5000 > source files in one directory. While I can check this out OK on Linux, we're > seeing an unusable slow-down on Windows XP (NTFS), both using Tortoise > directly, and as a

Re: Checkout really slow in Windows with lots of files in one directory

2011-01-26 Thread Campbell Allan
On Wednesday 26 Jan 2011, Campbell Allan wrote: > On Wednesday 26 Jan 2011, Neil Bird wrote: > >We have a graphics-oriented code-base that's auto-generated and has > > > > >5000 source files in one directory. While I can check this out OK on > > > > Linux, we're seeing an unusable slow-down o

Re: Checkout really slow in Windows with lots of files in one directory

2011-01-26 Thread Campbell Allan
On Wednesday 26 Jan 2011, Neil Bird wrote: >We have a graphics-oriented code-base that's auto-generated and has > >5000 source files in one directory. While I can check this out OK on > Linux, we're seeing an unusable slow-down on Windows XP (NTFS), both using > Tortoise directly, and as a te

Checkout really slow in Windows with lots of files in one directory

2011-01-26 Thread Neil Bird
We have a graphics-oriented code-base that's auto-generated and has >5000 source files in one directory. While I can check this out OK on Linux, we're seeing an unusable slow-down on Windows XP (NTFS), both using Tortoise directly, and as a test on Linux with the Windows drive mapped over C