Re: How to speed up subversion
On May 24, 2010, at 2:05 PM, Les Mikesell wrote: On 5/24/2010 3:51 PM, David Brodbeck wrote: On May 21, 2010, at 8:23 AM, Hyrum K. Wright wrote: Actually, Subversion is a bit more intelligent about it, attempting to use modification times and sizes, before doing a byte-by-byte comparison. Even that can be slow if there are lots of files involved. Note that some filesystems have particularly bad performance if there are lots of files in a single directory, as well. This is especially true of NFS. In one extreme example, I've seen a simple ls take tens of minutes to produce any output in an NFS-mounted directory with ~1,000,000 files. But to be fair, note that ls sorts its output, so it can't output anything until it has read everything. True, but other operations were slow, too. The way readdir() works means that a lot of file operations inevitably involve a linear search of the directory. -- David Brodbeck System Administrator, Linguistics University of Washington
Re: How to speed up subversion
On 5/25/2010 11:44 AM, David Brodbeck wrote: On May 24, 2010, at 2:05 PM, Les Mikesell wrote: On 5/24/2010 3:51 PM, David Brodbeck wrote: On May 21, 2010, at 8:23 AM, Hyrum K. Wright wrote: Actually, Subversion is a bit more intelligent about it, attempting to use modification times and sizes, before doing a byte-by-byte comparison. Even that can be slow if there are lots of files involved. Note that some filesystems have particularly bad performance if there are lots of files in a single directory, as well. This is especially true of NFS. In one extreme example, I've seen a simple ls take tens of minutes to produce any output in an NFS-mounted directory with ~1,000,000 files. But to be fair, note that ls sorts its output, so it can't output anything until it has read everything. True, but other operations were slow, too. The way readdir() works means that a lot of file operations inevitably involve a linear search of the directory. And worse, creating a new file involves doing that search first to see if it is new, then finding a free slot or growing the directory and adding it all as an atomic operation, blocking any other process trying to do the same. But, that's why subdirectories were invented - you don't have to put everything on one big inefficient space. -- Les Mikesell lesmikes...@gmail.com
Re: How to speed up subversion
On May 25, 2010, at 9:59 AM, Les Mikesell wrote: On 5/25/2010 11:44 AM, David Brodbeck wrote: On May 24, 2010, at 2:05 PM, Les Mikesell wrote: On 5/24/2010 3:51 PM, David Brodbeck wrote: On May 21, 2010, at 8:23 AM, Hyrum K. Wright wrote: Actually, Subversion is a bit more intelligent about it, attempting to use modification times and sizes, before doing a byte-by-byte comparison. Even that can be slow if there are lots of files involved. Note that some filesystems have particularly bad performance if there are lots of files in a single directory, as well. This is especially true of NFS. In one extreme example, I've seen a simple ls take tens of minutes to produce any output in an NFS-mounted directory with ~1,000,000 files. But to be fair, note that ls sorts its output, so it can't output anything until it has read everything. True, but other operations were slow, too. The way readdir() works means that a lot of file operations inevitably involve a linear search of the directory. And worse, creating a new file involves doing that search first to see if it is new, then finding a free slot or growing the directory and adding it all as an atomic operation, blocking any other process trying to do the same. But, that's why subdirectories were invented - you don't have to put everything on one big inefficient space. Yup. Now I just need to convince my users of that. ;) -- David Brodbeck System Administrator, Linguistics University of Washington
RE: How to speed up subversion
Hi, -Original Message- From: Jeremy Conlin [mailto:jlcon...@gmail.com] Sent: Wednesday, April 21, 2010 5:48 PM To: users@subversion.apache.org Subject: How to speed up subversion I have a working copy/respository with many files that are several hundred MB each. Whenever I try to check the status of my working copy or do a commit, it can take a long time (~1 min) before I get a response. I don't have any externals and the number of total files in my repository is ~100. Has anyone else experienced this or knows how to speed things up? I'm running svn 1.6.5 on Mac OSX 10.6.2. I guess the speed of your hard disk is the issue. Commit and status both check if the files on the local hard drive have changed since the last update. They do this by comparing the originals in the .svn dirs to the files you are working on. When you have a lot of large files, that can take quite a long time (100 MB times 100 files are 10 GB that have to be read, twice!) I hope this points you in the right direction... Cheers, Ulli Please note: This e-mail may contain confidential information intended solely for the addressee. If you have received this e-mail in error, please do not disclose it to anyone, notify the sender promptly, and delete the message from your system. Thank you.
Re: How to speed up subversion
On Fri, May 21, 2010 at 9:39 AM, ullrich.j...@elektrobit.com wrote: Hi, -Original Message- From: Jeremy Conlin [mailto:jlcon...@gmail.com] Sent: Wednesday, April 21, 2010 5:48 PM To: users@subversion.apache.org Subject: How to speed up subversion I have a working copy/respository with many files that are several hundred MB each. Whenever I try to check the status of my working copy or do a commit, it can take a long time (~1 min) before I get a response. I don't have any externals and the number of total files in my repository is ~100. Has anyone else experienced this or knows how to speed things up? I'm running svn 1.6.5 on Mac OSX 10.6.2. I guess the speed of your hard disk is the issue. Commit and status both check if the files on the local hard drive have changed since the last update. They do this by comparing the originals in the .svn dirs to the files you are working on. When you have a lot of large files, that can take quite a long time (100 MB times 100 files are 10 GB that have to be read, twice!) I hope this points you in the right direction... Actually, Subversion is a bit more intelligent about it, attempting to use modification times and sizes, before doing a byte-by-byte comparison. -Hyrum
Re: How to speed up subversion
On 4/21/2010 10:48 AM, Jeremy Conlin wrote: I have a working copy/respository with many files that are several hundred MB each. Whenever I try to check the status of my working copy or do a commit, it can take a long time (~1 min) before I get a response. I don't have any externals and the number of total files in my repository is ~100. Has anyone else experienced this or knows how to speed things up? I'm running svn 1.6.5 on Mac OSX 10.6.2. What client-server protocol are you using? This could be a problem with DNS or the authentication method besides whatever svn has to do. -- Les Mikesell lesmikes...@gmail.com
Re: How to speed up subversion
On Wed, Apr 21, 2010 at 9:58 AM, Les Mikesell lesmikes...@gmail.com wrote: On 4/21/2010 10:48 AM, Jeremy Conlin wrote: I have a working copy/respository with many files that are several hundred MB each. Whenever I try to check the status of my working copy or do a commit, it can take a long time (~1 min) before I get a response. I don't have any externals and the number of total files in my repository is ~100. Has anyone else experienced this or knows how to speed things up? I'm running svn 1.6.5 on Mac OSX 10.6.2. What client-server protocol are you using? This could be a problem with DNS or the authentication method besides whatever svn has to do. I am using svn+ssh. I have several other repositories located on the same server. These repositories are smaller in the number of MB and they also have no speed issues as I have with the big one. Jeremy
