svnserve 1.8 + slow network + large svn:mergeinfo property

2014-03-18 Thread Matthews, David
We recently upgraded from subversion 1.6 to 1.8.5.
After upgrading we have discovered a major performance issue when accessing
some repositories from a remote site.

After much investigation I've discovered that the problem only becomes
obvious for repositories with large svn:mergeinfo properties.
I can reproduce the problem by creating a dummy repository as follows:

#!/bin/bash
svnadmin create test_svn
tempdir=$(mktemp -d)
svn co file://$PWD/test_svn $tempdir
for i in {1..400}; do
  svn mkdir --parents $tempdir/trunk/dir-$i
  for j in {1..1}; do
 echo Test file $j$tempdir/trunk/dir-$i/file-$j
 svn add $tempdir/trunk/dir-$i/file-$j
  done
done
for i in {1..1000}; do
  echo branches/branch-$i:1-2$tempdir/mergeinfo
done
svn ps svn:mergeinfo -F $tempdir/mergeinfo $tempdir/trunk
svn ci -m Setup $tempdir
rm -rf $tempdir

If I vary the number of directories created and the number of lines added
to the svn:mergeinfo property I get the following results doing an ls -R
on the trunk (accessing the repository via svnserve over the slow network
link):

mergeinfo lines: 1000, directories: 100, time: 23s
mergeinfo lines: 1000, directories: 200, time: 46s
mergeinfo lines: 1000, directories: 400, time: 92s
mergeinfo lines:  500, directories: 400, time: 45s
mergeinfo lines:  250, directories: 400, time: 23s
mergeinfo lines:  125, directories: 400, time: 13s
mergeinfo lines:   30, directories: 400, time:  4s
no mergeinfo,  directories: 400, time:  3s

The problem only occurs when accessing the repository from the remote site
(relatively slow, busy link). On the local network the command completes
within 1s.

The problem only occurs using svnserve. Accessing the same repository via
http seems OK (mergeinfo lines: 1000, directories: 400, time: 4s)

Increasing the number of files in each directory (e.g. from 1 to 20) makes
virtually no difference to the timings.

info -R commands are affected in a similar way.

As well as 1.8.5, I've tried using 1.8.0  1.8.8 on the server but it made
no difference.
However, using 1.7.9 on the server (accessing using a 1.8.5 client) I get:
mergeinfo lines: 1000, directories: 400, time: 3s
So, this problem seems to have been introduced with 1.8.

The problem doesn't appear to be affected by the client version.
I've tried accessing the repository using an old 1.4.3 client and get the
same performance issue as using a 1.8.5 client.

Does this make sense to anyone?
Why should the size of the svn:mergeinfo property affect ls -R commands?
I haven't managed to measure the network traffic yet but I assume that, for
some reason, svnserve is transmitting a lot of unnecessary information?

Thanks,
David Matthews

Re: svnserve 1.8 + slow network + large svn:mergeinfo property

2014-03-18 Thread Philip Martin
Matthews, David david.matth...@metoffice.gov.uk writes:

 Does this make sense to anyone?
 Why should the size of the svn:mergeinfo property affect ls -R commands?
 I haven't managed to measure the network traffic yet but I assume that, for
 some reason, svnserve is transmitting a lot of unnecessary information?

Yes.  1.8 introduced optional inherited properties and the server is not
properly detecting clients that don't ask for those properties.  Running
svn ls -R queries a lot of paths and server keeps sending the unwanted
inherited properties.

Fixed on trunk by r1578853.

-- 
Philip Martin | Subversion Committer
WANdisco // *Non-Stop Data*


RE: svnserve 1.8 + slow network + large svn:mergeinfo property

2014-03-18 Thread Matthews, David
 -Original Message-
 From: Philip Martin [mailto:philip.mar...@wandisco.com] 
 Sent: 18 March 2014 13:02
 To: Matthews, David
 Cc: 'users@subversion.apache.org'
 Subject: Re: svnserve 1.8 + slow network + large 
 svn:mergeinfo property
 
 Matthews, David david.matth...@metoffice.gov.uk writes:
 
  Does this make sense to anyone?
  Why should the size of the svn:mergeinfo property affect 
 ls -R commands?
  I haven't managed to measure the network traffic yet but I 
 assume that, for
  some reason, svnserve is transmitting a lot of unnecessary 
 information?
 
 Yes.  1.8 introduced optional inherited properties and the 
 server is not
 properly detecting clients that don't ask for those 
 properties.  Running
 svn ls -R queries a lot of paths and server keeps sending 
 the unwanted
 inherited properties.
 
 Fixed on trunk by r1578853.

Philip

Many thanks for looking into this so quickly.

I guess we'll need to wait for 1.8.9?
(we normally use the wandisco RPMs)
Unfortunately this is causing us a lot of pain.

David Matthews