Update/Revert doesn't restore missed files

[Sergey Azarkevich]

I have working copy in which some files and directories missed: present on
server but absent locally. Update, revert doesn't restore them. Also it
doesn't look like sparse.
Problem was resolved via clean checkout, so I only report about it and not
look for solution.

--
C:\temp\svn-test\solution1\proj1>ls
App_Data  Bootstrapper.cs  proj1.csproj  Properties  Unity.config
 Web.config  Wse.config  classes


C:\temp\svn-test\solution1\proj1>svn ls
App_Data/
App_GlobalResources/
App_Themes/
Bootstrapper.cs
[... 8< ...]

C:\temp\svn-test\solution1\proj1>svn st



C:\temp\svn-test\solution1\proj1>svn up
Updating '.':
At revision 24799.

C:\temp\svn-test\solution1\proj1>svn revert . -R


C:\temp\svn-test\solution1\proj1>svn up . --force
Updating '.':
At revision 24799.

C:\temp\svn-test\solution1\proj1>svn up . --set-depth=infinity --force
Updating '.':
At revision 24799.

C:\temp\svn-test\solution1\proj1>svn up . --depth=infinity --force
Updating '.':
At revision 24799.


C:\temp\svn-test\solution1\proj1>svn info .
Path: .
Working Copy Root Path: C:\temp\svn-test
URL: https:///svn/GF/trunk/solution1/proj1
Relative URL: ^/trunk/solution1/proj1
Repository Root: https:///svn/GF
Repository UUID: 01c7c80d-d356-5740-a48b-b9b5f862b89e
Revision: 24799
Node Kind: directory
Schedule: normal
Last Changed Author: xxx
Last Changed Rev: 24488
Last Changed Date: 2013-07-19 11:27:37 +0300 (Пт, 19 июл 2013)


C:\temp\svn-test\solution1\proj1>svn info App_Themes
svn: warning: W155010: The node
'C:\temp\svn-test\solution1\proj1\App_Themes' was not found.

svn: E29: Could not display info for all targets because some targets
don't exist

C:\temp\svn-test\solution1\proj1>svn up App_Themes
Updating 'App_Themes':
AApp_Themes
AApp_Themes\default
Updated to revision 24799.

--

(some file names was changed)

Here you can see that App_GlobalResources, App_Themes (and many others)
absent locally. But eventually downloaded when it was specified explicitly.


I can't say, how it was reached, because it is not my working copy. I
observe next hustle actions:

0. Working copy was bind to already deleted branch.
Working copy has local modifications
Some solutions from wc was opened in Visual Studio (with addin
VisualSVN)
1. Try switch wc root to ^/trunk with TortoiseSVN. Abort in the middle.
Cleanup.
2. Try switch wc/solution1 to ^/trunk/solution1 with TortoiseSVN. (I recall
that svn start delete many files and folders, but not sure). Abort in the
middle. Cleanup.
3. Try exclude(sparse wc) some solutions from wc with TortoiseSVN. Not sure
if it was started at all. May be also aborted.

This was performed with TortoiseSVN linked with subversion:
svn, version 1.7.2 (r1207936)
   compiled Dec  6 2011, 16:02:36

4. Then I start analyze working copy. I upgrade WC to 1.8, problem
preserved. All tests above was executed with 1.8
svn, version 1.8.0 (r1490375)
   compiled Jun 18 2013, 13:38:49 on x86/x86_64-microsoft-windows6.1.7601

5. Do not digg in depth, but seems wc.db no not hold references to (for
example) App_Themes in table NODES

Ask me for details.

SVN performance -URGENT

[Somashekarappa, Anup (CWM-NR)]

Hello Team,

We are using subversion 1.7 which is hosted in linux and apache is being
used along with this.

The linux is very powerful but we are facing a major issue during the
SVN operation from the windows system.

Windows system : Microsoft windows XP 
2.85 GB of Ram
tortoisesvn 1.7

Windows system is based in london and SVN server(linux system) is
located in New york .When i checked the bandwidth using iperf from the
windows system ,it is showing as 35.5MB .

So when i do a checkout of repository ,it get message at last saying .
2200MBytes transfered at  143 mins.Which is very annoying.

According to bandwidth etc etc it should finish the checkout in few
mins.

May i know where is the bottleneck?

Thanks,
Anup
__

This email is intended only for the use of the individual(s) to whom it is 
addressed and may be privileged and confidential.
Unauthorised use or disclosure is prohibited. If you receive this e-mail in 
error, please advise immediately
and delete the original message. This message may have been altered without 
your or our knowledge
and the sender does not accept any liability for any errors or omissions in the 
message.

Emails are monitored by supervisory personnel in jurisdictions where monitoring 
is permitted. 
Such communications are retained and may be produced to regulatory authorities 
or others with legal rights to the information.

Re: SVN performance -URGENT

[Thorsten Schöning]

Guten Tag Somashekarappa, Anup (CWM-NR),
am Donnerstag, 1. August 2013 um 13:51 schrieben Sie:

> May i know where is the bottleneck?

Did you have a look at the CPU and I/O for storage and network on both
client and server? Even Windows provides enough tools built-in to see
if those resources are saturated, simply use the Resource Monitor.

Mit freundlichen Grüßen,

Thorsten Schöning

-- 
Thorsten Schöning   E-Mail:thorsten.schoen...@am-soft.de
AM-SoFT IT-Systeme  http://www.AM-SoFT.de/

Telefon...05151-  9468- 55
Fax...05151-  9468- 88
Mobil..0178-8 9468- 04

AM-SoFT GmbH IT-Systeme, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 207 694 - Geschäftsführer: Andreas Muchow

SVN performance -URGENT

[Somashekarappa, Anup (CWM-NR)]

> Hello Team,
> 
> We are using subversion 1.7 which is hosted in linux and apache is
> being used along with this.
> 
> The linux is very powerful but we are facing a major issue during the
> SVN operation from the windows system.
> 
> Windows system : Microsoft windows XP 
> 2.85 GB of Ram
> tortoisesvn 1.7
> 
> Windows system is based in london and SVN server(linux system) is
> located in New york .When i checked the bandwidth using iperf from the
> windows system ,it is showing as 35.5MB .
> 
> So when i do a checkout of repository ,it get message at last saying .
> 2200MBytes transfered at  143 mins.Which is very annoying.
> 
> According to bandwidth etc etc it should finish the checkout in few
> mins.
> 
> May i know where is the bottleneck?
> 
> Thanks,
> Anup
__

This email is intended only for the use of the individual(s) to whom it is 
addressed and may be privileged and confidential.
Unauthorised use or disclosure is prohibited. If you receive this e-mail in 
error, please advise immediately
and delete the original message. This message may have been altered without 
your or our knowledge
and the sender does not accept any liability for any errors or omissions in the 
message.

Emails are monitored by supervisory personnel in jurisdictions where monitoring 
is permitted. 
Such communications are retained and may be produced to regulatory authorities 
or others with legal rights to the information.

RE: SVN performance -URGENT

[Bob Archer]

> Hello Team,
> We are using subversion 1.7 which is hosted in linux and apache is being used
> along with this.
> The linux is very powerful but we are facing a major issue during the SVN
> operation from the windows system.
> Windows system : Microsoft windows XP
> 2.85 GB of Ram
> tortoisesvn 1.7
> Windows system is based in london and SVN server(linux system) is located in
> New york .When i checked the bandwidth using iperf from the windows system
> ,it is showing as 35.5MB .
> So when i do a checkout of repository ,it get message at last saying .
> 2200MBytes transfered at  143 mins.Which is very annoying.
> According to bandwidth etc etc it should finish the checkout in few mins.
> May i know where is the bottleneck?

My first question would be, given the same machine being on the same LAN as the 
server, what is the checkout time?

One of the biggest slowdowns on windows is Virus Software. Are you running any.

Also, have you tested using the svn command line to do the checkout to see if 
it is any faster? Most virus software ignores command line apps.

BOb

RE: SVN performance -URGENT

[Somashekarappa, Anup (CWM-NR)]

Hello Bob,

Thanks for your response.

I tried in the same server where svn is hosted but there also it is taking too 
much of time I e it is taking 110 mins to checkout the 2200 Mbytes of data(in 
Windows it took 143 mins).

I have not tried the command line option.Could you please tell how to do it 
from windows machine?

In windows when the checkout is completed I got a message saying 361 
Mbytes(different repository) transferred in 102 mins,but it is showing as 
1.13GB when I checked the folder size.Why there is so much of difference .

Ideally for 361 Mbytes,the checkout should complete in few mins but it is 
taking 102 mins. :-(

I cannot turn off the anti-virus part and check.

 

Thanks,
Anup



-Original Message-
From: Bob Archer [mailto:bob.arc...@amsi.com] 
Sent: 01 August 2013 15:03
To: Somashekarappa, Anup (CWM-NR); users@subversion.apache.org
Subject: RE: SVN performance -URGENT

> Hello Team,
> We are using subversion 1.7 which is hosted in linux and apache is 
> being used along with this.
> The linux is very powerful but we are facing a major issue during the 
> SVN operation from the windows system.
> Windows system : Microsoft windows XP
> 2.85 GB of Ram
> tortoisesvn 1.7
> Windows system is based in london and SVN server(linux system) is 
> located in New york .When i checked the bandwidth using iperf from the 
> windows system ,it is showing as 35.5MB .
> So when i do a checkout of repository ,it get message at last saying .
> 2200MBytes transfered at  143 mins.Which is very annoying.
> According to bandwidth etc etc it should finish the checkout in few mins.
> May i know where is the bottleneck?

My first question would be, given the same machine being on the same LAN as the 
server, what is the checkout time?

One of the biggest slowdowns on windows is Virus Software. Are you running any.

Also, have you tested using the svn command line to do the checkout to see if 
it is any faster? Most virus software ignores command line apps.

BOb



__

This email is intended only for the use of the individual(s) to whom it is 
addressed and may be privileged and confidential.
Unauthorised use or disclosure is prohibited. If you receive this e-mail in 
error, please advise immediately
and delete the original message. This message may have been altered without 
your or our knowledge
and the sender does not accept any liability for any errors or omissions in the 
message.

Emails are monitored by supervisory personnel in jurisdictions where monitoring 
is permitted. 
Such communications are retained and may be produced to regulatory authorities 
or others with legal rights to the information.

RE: SVN performance -URGENT

[kmradke]

> I tried in the same server where svn is hosted but there also it is 
> taking too much of time I e it is taking 110 mins to checkout the 
> 2200 Mbytes of data(in Windows it took 143 mins).

How many files are in the working copy?  For example, millions of small
files can take a significant amount of time to process.

Is the server I/O limited in some way?  Does it have connection
throttling enabled?  What protocol is being used?  (http://, https://
svn://, svn+ssh://)

Kevin R.

RE: SVN performance -URGENT

[kmradke]

Bob Archer  wrote on 08/01/2013 09:02:32 AM:
> > We are using subversion 1.7 which is hosted in linux and apache 
isbeing used
> > along with this.
> > The linux is very powerful but we are facing a major issue during the 
SVN
> > operation from the windows system.
> > Windows system : Microsoft windows XP
> > 2.85 GB of Ram
> > tortoisesvn 1.7
> > Windows system is based in london and SVN server(linux system) is 
located in
> > New york .When i checked the bandwidth using iperf from the windows 
system
> > ,it is showing as 35.5MB .

35.5MB/s or 35.5Mb/s?  The first one would be a 284Mb/s connection between 
London
and NY...

What is the latency between the client and server?  Latency can be a big
killer depending upon the authentication used.

> > So when i do a checkout of repository ,it get message at last saying .
> > 2200MBytes transfered at  143 mins.Which is very annoying.
> > According to bandwidth etc etc it should finish the checkout in few 
mins.
> > May i know where is the bottleneck?

The total amount of data transferred shown by TortoiseSVN can sometimes
be quite misleading.  How big is the working copy that is created?

Kevin R.

RE: SVN performance -URGENT

[Bob Archer]

Please stop top posting.

> I tried in the same server where svn is hosted but there also it is taking too
> much of time I e it is taking 110 mins to checkout the 2200 Mbytes of data(in
> Windows it took 143 mins).
> 
> I have not tried the command line option.Could you please tell how to do it
> from windows machine?

If you installed the command line option when you installed tortoise svn from a 
command line you would cd to your development root and use the a command like...

svn co http://myserver/svn/path workingcopyfoldername

> 
> In windows when the checkout is completed I got a message saying 361
> Mbytes(different repository) transferred in 102 mins,but it is showing as
> 1.13GB when I checked the folder size.Why there is so much of difference .
> 
> Ideally for 361 Mbytes,the checkout should complete in few mins but it is 
> taking
> 102 mins. :-(

The working copy is generally 2x the size of the files in it, since the .svn 
folder contains the pristine copies of your working copy. This is how svn can 
do diffs against changed items In your working copy without hitting the server, 
and also how it can do reverts.

> 
> I cannot turn off the anti-virus part and check.

That's unfortunate. Are you able to at least tell it to ignore your development 
root folder. We have corporate virus software too, but I am allowed to exclude 
folders, so I exclude c:\users\bob\development. Virus Software can really mess 
up svn and cause perf issues.

> 
> > Hello Team,
> > We are using subversion 1.7 which is hosted in linux and apache is
> > being used along with this.
> > The linux is very powerful but we are facing a major issue during the
> > SVN operation from the windows system.
> > Windows system : Microsoft windows XP
> > 2.85 GB of Ram
> > tortoisesvn 1.7
> > Windows system is based in london and SVN server(linux system) is
> > located in New york .When i checked the bandwidth using iperf from the
> > windows system ,it is showing as 35.5MB .
> > So when i do a checkout of repository ,it get message at last saying .
> > 2200MBytes transfered at  143 mins.Which is very annoying.
> > According to bandwidth etc etc it should finish the checkout in few mins.
> > May i know where is the bottleneck?
> 
> My first question would be, given the same machine being on the same LAN as
> the server, what is the checkout time?
> 
> One of the biggest slowdowns on windows is Virus Software. Are you running
> any.
> 
> Also, have you tested using the svn command line to do the checkout to see if 
> it
> is any faster? Most virus software ignores command line apps.
> 
> BOb
> 
> 
> 
> 
> __
> 
> This email is intended only for the use of the individual(s) to whom it is
> addressed and may be privileged and confidential.
> Unauthorised use or disclosure is prohibited. If you receive this e-mail in 
> error,
> please advise immediately and delete the original message. This message may
> have been altered without your or our knowledge and the sender does not
> accept any liability for any errors or omissions in the message.
> 
> Emails are monitored by supervisory personnel in jurisdictions where
> monitoring is permitted.
> Such communications are retained and may be produced to regulatory
> authorities or others with legal rights to the information.

RE: SVN performance -URGENT

[Somashekarappa, Anup (CWM-NR)]

Bandwidth is 35.4 MBytes/sec from my system(London)  to server(New york)
when i checked with iperf tool.
 
 
We are using LDAP 
 
AuthzLDAPAuthoritative off
AuthType Basic
AuthBasicProvider ldap
AuthName "Windows Credentials"
 
 
As per message after checkout in TortoiseSVN GUI = 368 Mbytes
transfered.

Actual folder size = 1.15 GB(1236706079 bytes)

Number of files = 201,712

Folder = 21,707

 

Guess this inculdes the .svn folder as well.

 

 

Thanks,

Anup



 
 
 



From: kmra...@rockwellcollins.com [mailto:kmra...@rockwellcollins.com] 
Sent: 01 August 2013 15:20
To: Bob Archer
Cc: Somashekarappa, Anup (CWM-NR); users@subversion.apache.org
Subject: RE: SVN performance -URGENT


Bob Archer  wrote on 08/01/2013 09:02:32 AM:
> > We are using subversion 1.7 which is hosted in linux and apache
isbeing used
> > along with this.
> > The linux is very powerful but we are facing a major issue during
the SVN
> > operation from the windows system.
> > Windows system : Microsoft windows XP
> > 2.85 GB of Ram
> > tortoisesvn 1.7
> > Windows system is based in london and SVN server(linux system) is
located in
> > New york .When i checked the bandwidth using iperf from the windows
system
> > ,it is showing as 35.5MB . 

35.5MB/s or 35.5Mb/s?  The first one would be a 284Mb/s connection
between London 
and NY... 

What is the latency between the client and server?  Latency can be a big

killer depending upon the authentication used. 

> > So when i do a checkout of repository ,it get message at last saying
.
> > 2200MBytes transfered at  143 mins.Which is very annoying.
> > According to bandwidth etc etc it should finish the checkout in few
mins.
> > May i know where is the bottleneck?

The total amount of data transferred shown by TortoiseSVN can sometimes 
be quite misleading.  How big is the working copy that is created? 

Kevin R.

__

This email is intended only for the use of the individual(s) to whom it is 
addressed and may be privileged and confidential.
Unauthorised use or disclosure is prohibited. If you receive this e-mail in 
error, please advise immediately
and delete the original message. This message may have been altered without 
your or our knowledge
and the sender does not accept any liability for any errors or omissions in the 
message.

Emails are monitored by supervisory personnel in jurisdictions where monitoring 
is permitted. 
Such communications are retained and may be produced to regulatory authorities 
or others with legal rights to the information.

RE: SVN performance -URGENT

[Johan Corveleyn]

On 1 Aug 2013 16:52, "Somashekarappa, Anup (CWM-NR)" <
anup.somashekara...@rbc.com> wrote:
>
> Bandwidth is 35.4 MBytes/sec from my system(London)  to server(New york)
when i checked with iperf tool.
>
>
> We are using LDAP
>
> AuthzLDAPAuthoritative off
> AuthType Basic
> AuthBasicProvider ldap
> AuthName "Windows Credentials"
>
>
>
> As per message after checkout in TortoiseSVN GUI = 368 Mbytes transfered.
>
> Actual folder size = 1.15 GB(1236706079 bytes)
>
> Number of files = 201,712
>
> Folder = 21,707
>
>
>
> Guess this inculdes the .svn folder as well.
>

With so many files and folders I suspect the working copy to be the
bottleneck. Can you try the latest release, 1.8.1, as client to see if it
is any better?

You may also want to experiment with the new in 1.8 option of exclusive
locking mode in the client. Check out the release notes for that.

-- 
Johan

Solved: Could not read status line: Connection reset by server

[Weeber, Burkhard]

After being plagued by this error on more and more windows servers,
the solution turned out to be simple.
Add
http-library = serf   (or neon, if you prefer that)
to the servers file.

It looks like svn uses a function name that might appear in other libraries 
also, thus pulling the wrong function with a different, but for svn unexpected, 
behavior.

Hope it helps

i.A. Burkhard Weeber

viastore systems GmbH
Germany




Follow us: Facebook, 
Twitter, 
Linkedin, 
Xing, 
youtube


Geschäftsführer/Managing Directors: Philipp Hahn-Woernle, Frank Apel
HRB 17335, Amtsgericht Stuttgart (Commercial Register District Court Stuttgart)
St.-Nr. 99064/06051, USt-IdNr./VAT Reg.No.: DE 203036780

RE: SVN performance -URGENT

[Bob Archer]

> Bandwidth is 35.4 MBytes/sec from my system(London)  to server(New york)
> when i checked with iperf tool.

WOW! That is insanely fast. Are you sure its not 35 Mbps (Mega bits!!)? Google 
Fiber that everyone covets is 1 Gbps (Giga bit per second) or 1024 Mbps or 128 
MBps.

At 35 Mbps it would take a little less than 7 minutes to transfer. That is 
assuming constant speed. Can you copy the folder from the machine in London to 
the machine in New York in under 7 minutes?

At 35 MBps it would take under a minute!

> We are using LDAP
> 
> AuthzLDAPAuthoritative off
> AuthType Basic
> AuthBasicProvider ldap
> AuthName "Windows Credentials"
> 
> 
> As per message after checkout in TortoiseSVN GUI = 368 Mbytes transfered.
> Actual folder size = 1.15 GB(1236706079 bytes) Number of files = 201,712
> Folder = 21,707
> 
> Guess this inculdes the .svn folder as well.
> 
> 
> Thanks,
> Anup

I'm not sure why there would be that much overhead with SVN, I guess it does no 
caching. One the plus side, you only have to check out once. ;)

Have you tried to check out some large public stuff? For example, check out the 
subversion repository and see how long that takes. 

Was the command line checkout time any different?

> 
> 
> 
> 
> 
> 
> From: kmra...@rockwellcollins.com [mailto:kmra...@rockwellcollins.com]
> Sent: 01 August 2013 15:20
> To: Bob Archer
> Cc: Somashekarappa, Anup (CWM-NR); users@subversion.apache.org
> Subject: RE: SVN performance -URGENT
> Bob Archer  wrote on 08/01/2013 09:02:32 AM:
> > > We are using subversion 1.7 which is hosted in linux and apache
> > > isbeing used along with this.
> > > The linux is very powerful but we are facing a major issue during
> > > the SVN operation from the windows system.
> > > Windows system : Microsoft windows XP
> > > 2.85 GB of Ram
> > > tortoisesvn 1.7
> > > Windows system is based in london and SVN server(linux system) is
> > > located in New york .When i checked the bandwidth using iperf from
> > > the windows system ,it is showing as 35.5MB .
> 
> 35.5MB/s or 35.5Mb/s?  The first one would be a 284Mb/s connection between
> London and NY...
> 
> What is the latency between the client and server?  Latency can be a big 
> killer
> depending upon the authentication used.
> 
> > > So when i do a checkout of repository ,it get message at last saying .
> > > 2200MBytes transfered at  143 mins.Which is very annoying.
> > > According to bandwidth etc etc it should finish the checkout in few mins.
> > > May i know where is the bottleneck?
> 
> The total amount of data transferred shown by TortoiseSVN can sometimes be
> quite misleading.  How big is the working copy that is created?
> 
> Kevin R.
> 
> __
> This email is intended only for the use of the individual(s) to whom it is
> addressed and may be privileged and confidential.
> Unauthorised use or disclosure is prohibited. If you receive this e-mail in 
> error,
> please advise immediately and delete the original message. This message may
> have been altered without your or our knowledge and the sender does not
> accept any liability for any errors or omissions in the message.
> Emails are monitored by supervisory personnel in jurisdictions where
> monitoring is permitted.
> Such communications are retained and may be produced to regulatory
> authorities or others with legal rights to the information.

Re: SVN performance -URGENT

[Thomas Harold]

On 8/1/2013 10:52 AM, Somashekarappa, Anup (CWM-NR) wrote:

Bandwidth is 35.4 MBytes/secfrom my system(London)  to server(New york)
when i checked with iperf tool.
We are using LDAP
AuthzLDAPAuthoritative off
AuthType Basic
AuthBasicProvider ldap
AuthName "Windows Credentials"

As per message after checkout in TortoiseSVNGUI = 368 Mbytes transfered.

Actual folder size = 1.15 GB(1236706079 bytes)

Number of files = 201,712

Folder = 21,707

Guess this inculdes the .svn folder as well.



That's a fairly complex working copy with many files/folders.  Given 
that you have 35Mbps (note the lower case B) of bandwidth, an ideal 
transfer should be somewhere in the 45-60 minute range for a fresh 
checkout of the entire thing.


However, you're obviously bottlenecked somewhere.

On the Linux server side, I suggest installing a tool called "atop" and 
monitoring things like how busy the disks are, how busy the CPU cores 
are and the network throughput.  This will give you an idea of how hard 
the Linux server is working while sending out the data to the SVN client.


For the windows client, you will need to look at the Performance Monitor 
(perfmon) and Task Manager to see if you are bottlenecking somewhere. 
Good counters to watch in perfmon are "Physical Disk / % Disk Read 
Time", "Physical Disk / % Disk Write Time", "Network Interface / Bytes 
Sent/sec", "Network Interface / Bytes Received/sec".


My guesses at this point would be:

- You're not using a SSD on the Windows client, so there is a lot of 
disk activity as SVN goes to create the working copy.  So your disks are 
100% busy and are your bottleneck.


- You're CPU bottlenecked somewhere.  Either server-side or client-side.

- Maybe you need to consider using sparse working copies or only 
checking out a portion of the repository at a time.  (Such as only 
bringing down your project's "trunk" folder.)


- You'll need to do this checkout once to create the initial working 
copy, then keep the working copy around for a long time.  Future "svn 
update" commands will then only transmit the changes over the wire 
instead of all of the content.

Re: SVN performance -URGENT

[Ryan Schmidt]

On Aug 1, 2013, at 09:26, Bob Archer wrote:

> The working copy is generally 2x the size of the files in it, since the .svn 
> folder contains the pristine copies of your working copy. This is how svn can 
> do diffs against changed items In your working copy without hitting the 
> server, and also how it can do reverts.

Also, I would hope the data is compressed when it's sent over the wire but is 
stored on disk uncompressed.