Re: SVN performance -URGENT
On Thu, Aug 1, 2013 at 10:18 AM, Somashekarappa, Anup (CWM-NR) anup.somashekara...@rbc.com wrote: Hello Bob, Thanks for your response. I tried in the same server where svn is hosted but there also it is taking too much of time I e it is taking 110 mins to checkout the 2200 Mbytes of data(in Windows it took 143 mins). CIFS is a very chatty protocol. May I assume that you're checkout out to a network mounted directory? In this case, you might try checkout out to a locally mounted directory, then simply copying the checkout out directory over to your CIFS mount. It's ugly, but it was much, much faster way back in Subversion 1.4.x days. Also, 2 GBytes over a translatlantic checkout is not really that bad. And remember that Subversion is writing all the files *twice*, once in .svn, and once in the actually checkout out copy, so you're effectively transmitting 2 GB and writing 4 GB. Directories with many thousands of files in one directory are *hell* on most operating systems, especially inefficiently configured network mounts. Check for directories that have tens our hundreds of thousands of files in one directory, and avoid that practice. You might also try splitting up the upstream repository, and using svn:extern to cross reference to other repositories. It can make for much smaller, safer checkouts. I have not tried the command line option.Could you please tell how to do it from windows machine? In windows when the checkout is completed I got a message saying 361 Mbytes(different repository) transferred in 102 mins,but it is showing as 1.13GB when I checked the folder size.Why there is so much of difference . See the contents of the .svn subdirectory, which maintains prisitine copies of all the files. Ideally for 361 Mbytes,the checkout should complete in few mins but it is taking 102 mins. :-( I cannot turn off the anti-virus part and check. If nothing else comes to light, consider keeping a pre-staged checkout in a central directory. Download a clean copy of *that*, and simply do svn update and/or svn switch to switch to the relevant tag or branch. Thanks, Anup
SVN performance -URGENT
Hello Team, We are using subversion 1.7 which is hosted in linux and apache is being used along with this. The linux is very powerful but we are facing a major issue during the SVN operation from the windows system. Windows system : Microsoft windows XP 2.85 GB of Ram tortoisesvn 1.7 Windows system is based in london and SVN server(linux system) is located in New york .When i checked the bandwidth using iperf from the windows system ,it is showing as 35.5MB . So when i do a checkout of repository ,it get message at last saying . 2200MBytes transfered at 143 mins.Which is very annoying. According to bandwidth etc etc it should finish the checkout in few mins. May i know where is the bottleneck? Thanks, Anup __ This email is intended only for the use of the individual(s) to whom it is addressed and may be privileged and confidential. Unauthorised use or disclosure is prohibited. If you receive this e-mail in error, please advise immediately and delete the original message. This message may have been altered without your or our knowledge and the sender does not accept any liability for any errors or omissions in the message. Emails are monitored by supervisory personnel in jurisdictions where monitoring is permitted. Such communications are retained and may be produced to regulatory authorities or others with legal rights to the information.
Re: SVN performance -URGENT
Guten Tag Somashekarappa, Anup (CWM-NR), am Donnerstag, 1. August 2013 um 13:51 schrieben Sie: May i know where is the bottleneck? Did you have a look at the CPU and I/O for storage and network on both client and server? Even Windows provides enough tools built-in to see if those resources are saturated, simply use the Resource Monitor. Mit freundlichen Grüßen, Thorsten Schöning -- Thorsten Schöning E-Mail:thorsten.schoen...@am-soft.de AM-SoFT IT-Systeme http://www.AM-SoFT.de/ Telefon...05151- 9468- 55 Fax...05151- 9468- 88 Mobil..0178-8 9468- 04 AM-SoFT GmbH IT-Systeme, Brandenburger Str. 7c, 31789 Hameln AG Hannover HRB 207 694 - Geschäftsführer: Andreas Muchow
SVN performance -URGENT
Hello Team, We are using subversion 1.7 which is hosted in linux and apache is being used along with this. The linux is very powerful but we are facing a major issue during the SVN operation from the windows system. Windows system : Microsoft windows XP 2.85 GB of Ram tortoisesvn 1.7 Windows system is based in london and SVN server(linux system) is located in New york .When i checked the bandwidth using iperf from the windows system ,it is showing as 35.5MB . So when i do a checkout of repository ,it get message at last saying . 2200MBytes transfered at 143 mins.Which is very annoying. According to bandwidth etc etc it should finish the checkout in few mins. May i know where is the bottleneck? Thanks, Anup __ This email is intended only for the use of the individual(s) to whom it is addressed and may be privileged and confidential. Unauthorised use or disclosure is prohibited. If you receive this e-mail in error, please advise immediately and delete the original message. This message may have been altered without your or our knowledge and the sender does not accept any liability for any errors or omissions in the message. Emails are monitored by supervisory personnel in jurisdictions where monitoring is permitted. Such communications are retained and may be produced to regulatory authorities or others with legal rights to the information.
RE: SVN performance -URGENT
Hello Team, We are using subversion 1.7 which is hosted in linux and apache is being used along with this. The linux is very powerful but we are facing a major issue during the SVN operation from the windows system. Windows system : Microsoft windows XP 2.85 GB of Ram tortoisesvn 1.7 Windows system is based in london and SVN server(linux system) is located in New york .When i checked the bandwidth using iperf from the windows system ,it is showing as 35.5MB . So when i do a checkout of repository ,it get message at last saying . 2200MBytes transfered at 143 mins.Which is very annoying. According to bandwidth etc etc it should finish the checkout in few mins. May i know where is the bottleneck? My first question would be, given the same machine being on the same LAN as the server, what is the checkout time? One of the biggest slowdowns on windows is Virus Software. Are you running any. Also, have you tested using the svn command line to do the checkout to see if it is any faster? Most virus software ignores command line apps. BOb
RE: SVN performance -URGENT
Hello Bob, Thanks for your response. I tried in the same server where svn is hosted but there also it is taking too much of time I e it is taking 110 mins to checkout the 2200 Mbytes of data(in Windows it took 143 mins). I have not tried the command line option.Could you please tell how to do it from windows machine? In windows when the checkout is completed I got a message saying 361 Mbytes(different repository) transferred in 102 mins,but it is showing as 1.13GB when I checked the folder size.Why there is so much of difference . Ideally for 361 Mbytes,the checkout should complete in few mins but it is taking 102 mins. :-( I cannot turn off the anti-virus part and check. Thanks, Anup -Original Message- From: Bob Archer [mailto:bob.arc...@amsi.com] Sent: 01 August 2013 15:03 To: Somashekarappa, Anup (CWM-NR); users@subversion.apache.org Subject: RE: SVN performance -URGENT Hello Team, We are using subversion 1.7 which is hosted in linux and apache is being used along with this. The linux is very powerful but we are facing a major issue during the SVN operation from the windows system. Windows system : Microsoft windows XP 2.85 GB of Ram tortoisesvn 1.7 Windows system is based in london and SVN server(linux system) is located in New york .When i checked the bandwidth using iperf from the windows system ,it is showing as 35.5MB . So when i do a checkout of repository ,it get message at last saying . 2200MBytes transfered at 143 mins.Which is very annoying. According to bandwidth etc etc it should finish the checkout in few mins. May i know where is the bottleneck? My first question would be, given the same machine being on the same LAN as the server, what is the checkout time? One of the biggest slowdowns on windows is Virus Software. Are you running any. Also, have you tested using the svn command line to do the checkout to see if it is any faster? Most virus software ignores command line apps. BOb __ This email is intended only for the use of the individual(s) to whom it is addressed and may be privileged and confidential. Unauthorised use or disclosure is prohibited. If you receive this e-mail in error, please advise immediately and delete the original message. This message may have been altered without your or our knowledge and the sender does not accept any liability for any errors or omissions in the message. Emails are monitored by supervisory personnel in jurisdictions where monitoring is permitted. Such communications are retained and may be produced to regulatory authorities or others with legal rights to the information.
RE: SVN performance -URGENT
I tried in the same server where svn is hosted but there also it is taking too much of time I e it is taking 110 mins to checkout the 2200 Mbytes of data(in Windows it took 143 mins). How many files are in the working copy? For example, millions of small files can take a significant amount of time to process. Is the server I/O limited in some way? Does it have connection throttling enabled? What protocol is being used? (http://, https:// svn://, svn+ssh://) Kevin R.
RE: SVN performance -URGENT
Bob Archer bob.arc...@amsi.com wrote on 08/01/2013 09:02:32 AM: We are using subversion 1.7 which is hosted in linux and apache isbeing used along with this. The linux is very powerful but we are facing a major issue during the SVN operation from the windows system. Windows system : Microsoft windows XP 2.85 GB of Ram tortoisesvn 1.7 Windows system is based in london and SVN server(linux system) is located in New york .When i checked the bandwidth using iperf from the windows system ,it is showing as 35.5MB . 35.5MB/s or 35.5Mb/s? The first one would be a 284Mb/s connection between London and NY... What is the latency between the client and server? Latency can be a big killer depending upon the authentication used. So when i do a checkout of repository ,it get message at last saying . 2200MBytes transfered at 143 mins.Which is very annoying. According to bandwidth etc etc it should finish the checkout in few mins. May i know where is the bottleneck? The total amount of data transferred shown by TortoiseSVN can sometimes be quite misleading. How big is the working copy that is created? Kevin R.
RE: SVN performance -URGENT
Please stop top posting. I tried in the same server where svn is hosted but there also it is taking too much of time I e it is taking 110 mins to checkout the 2200 Mbytes of data(in Windows it took 143 mins). I have not tried the command line option.Could you please tell how to do it from windows machine? If you installed the command line option when you installed tortoise svn from a command line you would cd to your development root and use the a command like... svn co http://myserver/svn/path workingcopyfoldername In windows when the checkout is completed I got a message saying 361 Mbytes(different repository) transferred in 102 mins,but it is showing as 1.13GB when I checked the folder size.Why there is so much of difference . Ideally for 361 Mbytes,the checkout should complete in few mins but it is taking 102 mins. :-( The working copy is generally 2x the size of the files in it, since the .svn folder contains the pristine copies of your working copy. This is how svn can do diffs against changed items In your working copy without hitting the server, and also how it can do reverts. I cannot turn off the anti-virus part and check. That's unfortunate. Are you able to at least tell it to ignore your development root folder. We have corporate virus software too, but I am allowed to exclude folders, so I exclude c:\users\bob\development. Virus Software can really mess up svn and cause perf issues. Hello Team, We are using subversion 1.7 which is hosted in linux and apache is being used along with this. The linux is very powerful but we are facing a major issue during the SVN operation from the windows system. Windows system : Microsoft windows XP 2.85 GB of Ram tortoisesvn 1.7 Windows system is based in london and SVN server(linux system) is located in New york .When i checked the bandwidth using iperf from the windows system ,it is showing as 35.5MB . So when i do a checkout of repository ,it get message at last saying . 2200MBytes transfered at 143 mins.Which is very annoying. According to bandwidth etc etc it should finish the checkout in few mins. May i know where is the bottleneck? My first question would be, given the same machine being on the same LAN as the server, what is the checkout time? One of the biggest slowdowns on windows is Virus Software. Are you running any. Also, have you tested using the svn command line to do the checkout to see if it is any faster? Most virus software ignores command line apps. BOb __ This email is intended only for the use of the individual(s) to whom it is addressed and may be privileged and confidential. Unauthorised use or disclosure is prohibited. If you receive this e-mail in error, please advise immediately and delete the original message. This message may have been altered without your or our knowledge and the sender does not accept any liability for any errors or omissions in the message. Emails are monitored by supervisory personnel in jurisdictions where monitoring is permitted. Such communications are retained and may be produced to regulatory authorities or others with legal rights to the information.
RE: SVN performance -URGENT
Bandwidth is 35.4 MBytes/sec from my system(London) to server(New york) when i checked with iperf tool. We are using LDAP AuthzLDAPAuthoritative off AuthType Basic AuthBasicProvider ldap AuthName Windows Credentials As per message after checkout in TortoiseSVN GUI = 368 Mbytes transfered. Actual folder size = 1.15 GB(1236706079 bytes) Number of files = 201,712 Folder = 21,707 Guess this inculdes the .svn folder as well. Thanks, Anup From: kmra...@rockwellcollins.com [mailto:kmra...@rockwellcollins.com] Sent: 01 August 2013 15:20 To: Bob Archer Cc: Somashekarappa, Anup (CWM-NR); users@subversion.apache.org Subject: RE: SVN performance -URGENT Bob Archer bob.arc...@amsi.com wrote on 08/01/2013 09:02:32 AM: We are using subversion 1.7 which is hosted in linux and apache isbeing used along with this. The linux is very powerful but we are facing a major issue during the SVN operation from the windows system. Windows system : Microsoft windows XP 2.85 GB of Ram tortoisesvn 1.7 Windows system is based in london and SVN server(linux system) is located in New york .When i checked the bandwidth using iperf from the windows system ,it is showing as 35.5MB . 35.5MB/s or 35.5Mb/s? The first one would be a 284Mb/s connection between London and NY... What is the latency between the client and server? Latency can be a big killer depending upon the authentication used. So when i do a checkout of repository ,it get message at last saying . 2200MBytes transfered at 143 mins.Which is very annoying. According to bandwidth etc etc it should finish the checkout in few mins. May i know where is the bottleneck? The total amount of data transferred shown by TortoiseSVN can sometimes be quite misleading. How big is the working copy that is created? Kevin R. __ This email is intended only for the use of the individual(s) to whom it is addressed and may be privileged and confidential. Unauthorised use or disclosure is prohibited. If you receive this e-mail in error, please advise immediately and delete the original message. This message may have been altered without your or our knowledge and the sender does not accept any liability for any errors or omissions in the message. Emails are monitored by supervisory personnel in jurisdictions where monitoring is permitted. Such communications are retained and may be produced to regulatory authorities or others with legal rights to the information.
RE: SVN performance -URGENT
On 1 Aug 2013 16:52, Somashekarappa, Anup (CWM-NR) anup.somashekara...@rbc.com wrote: Bandwidth is 35.4 MBytes/sec from my system(London) to server(New york) when i checked with iperf tool. We are using LDAP AuthzLDAPAuthoritative off AuthType Basic AuthBasicProvider ldap AuthName Windows Credentials As per message after checkout in TortoiseSVN GUI = 368 Mbytes transfered. Actual folder size = 1.15 GB(1236706079 bytes) Number of files = 201,712 Folder = 21,707 Guess this inculdes the .svn folder as well. With so many files and folders I suspect the working copy to be the bottleneck. Can you try the latest release, 1.8.1, as client to see if it is any better? You may also want to experiment with the new in 1.8 option of exclusive locking mode in the client. Check out the release notes for that. -- Johan
RE: SVN performance -URGENT
Bandwidth is 35.4 MBytes/sec from my system(London) to server(New york) when i checked with iperf tool. WOW! That is insanely fast. Are you sure its not 35 Mbps (Mega bits!!)? Google Fiber that everyone covets is 1 Gbps (Giga bit per second) or 1024 Mbps or 128 MBps. At 35 Mbps it would take a little less than 7 minutes to transfer. That is assuming constant speed. Can you copy the folder from the machine in London to the machine in New York in under 7 minutes? At 35 MBps it would take under a minute! We are using LDAP AuthzLDAPAuthoritative off AuthType Basic AuthBasicProvider ldap AuthName Windows Credentials As per message after checkout in TortoiseSVN GUI = 368 Mbytes transfered. Actual folder size = 1.15 GB(1236706079 bytes) Number of files = 201,712 Folder = 21,707 Guess this inculdes the .svn folder as well. Thanks, Anup I'm not sure why there would be that much overhead with SVN, I guess it does no caching. One the plus side, you only have to check out once. ;) Have you tried to check out some large public stuff? For example, check out the subversion repository and see how long that takes. Was the command line checkout time any different? From: kmra...@rockwellcollins.com [mailto:kmra...@rockwellcollins.com] Sent: 01 August 2013 15:20 To: Bob Archer Cc: Somashekarappa, Anup (CWM-NR); users@subversion.apache.org Subject: RE: SVN performance -URGENT Bob Archer bob.arc...@amsi.com wrote on 08/01/2013 09:02:32 AM: We are using subversion 1.7 which is hosted in linux and apache isbeing used along with this. The linux is very powerful but we are facing a major issue during the SVN operation from the windows system. Windows system : Microsoft windows XP 2.85 GB of Ram tortoisesvn 1.7 Windows system is based in london and SVN server(linux system) is located in New york .When i checked the bandwidth using iperf from the windows system ,it is showing as 35.5MB . 35.5MB/s or 35.5Mb/s? The first one would be a 284Mb/s connection between London and NY... What is the latency between the client and server? Latency can be a big killer depending upon the authentication used. So when i do a checkout of repository ,it get message at last saying . 2200MBytes transfered at 143 mins.Which is very annoying. According to bandwidth etc etc it should finish the checkout in few mins. May i know where is the bottleneck? The total amount of data transferred shown by TortoiseSVN can sometimes be quite misleading. How big is the working copy that is created? Kevin R. __ This email is intended only for the use of the individual(s) to whom it is addressed and may be privileged and confidential. Unauthorised use or disclosure is prohibited. If you receive this e-mail in error, please advise immediately and delete the original message. This message may have been altered without your or our knowledge and the sender does not accept any liability for any errors or omissions in the message. Emails are monitored by supervisory personnel in jurisdictions where monitoring is permitted. Such communications are retained and may be produced to regulatory authorities or others with legal rights to the information.
Re: SVN performance -URGENT
On 8/1/2013 10:52 AM, Somashekarappa, Anup (CWM-NR) wrote: Bandwidth is 35.4 MBytes/secfrom my system(London) to server(New york) when i checked with iperf tool. We are using LDAP AuthzLDAPAuthoritative off AuthType Basic AuthBasicProvider ldap AuthName Windows Credentials As per message after checkout in TortoiseSVNGUI = 368 Mbytes transfered. Actual folder size = 1.15 GB(1236706079 bytes) Number of files = 201,712 Folder = 21,707 Guess this inculdes the .svn folder as well. That's a fairly complex working copy with many files/folders. Given that you have 35Mbps (note the lower case B) of bandwidth, an ideal transfer should be somewhere in the 45-60 minute range for a fresh checkout of the entire thing. However, you're obviously bottlenecked somewhere. On the Linux server side, I suggest installing a tool called atop and monitoring things like how busy the disks are, how busy the CPU cores are and the network throughput. This will give you an idea of how hard the Linux server is working while sending out the data to the SVN client. For the windows client, you will need to look at the Performance Monitor (perfmon) and Task Manager to see if you are bottlenecking somewhere. Good counters to watch in perfmon are Physical Disk / % Disk Read Time, Physical Disk / % Disk Write Time, Network Interface / Bytes Sent/sec, Network Interface / Bytes Received/sec. My guesses at this point would be: - You're not using a SSD on the Windows client, so there is a lot of disk activity as SVN goes to create the working copy. So your disks are 100% busy and are your bottleneck. - You're CPU bottlenecked somewhere. Either server-side or client-side. - Maybe you need to consider using sparse working copies or only checking out a portion of the repository at a time. (Such as only bringing down your project's trunk folder.) - You'll need to do this checkout once to create the initial working copy, then keep the working copy around for a long time. Future svn update commands will then only transmit the changes over the wire instead of all of the content.
Re: SVN performance -URGENT
On Aug 1, 2013, at 09:26, Bob Archer wrote: The working copy is generally 2x the size of the files in it, since the .svn folder contains the pristine copies of your working copy. This is how svn can do diffs against changed items In your working copy without hitting the server, and also how it can do reverts. Also, I would hope the data is compressed when it's sent over the wire but is stored on disk uncompressed.