Re: Expected speed of commit over HTTP?

[Paul Hammant]

Great insights, Stefan.

So 'cp' with that sync, timed as you outline is averaging 3x slower than
the cp as I had it before.

Thus, that same cp with sync tacked on, is 3x faster that the curl PUT over
HTTP.

That's within the place where any speedup spent on the handoff of between
apache modules would be $100K of programming for a 15% speedup - not worth
it. At not worth it because mine is an esoteric use case and (say) Goldman
don't have Svn in a high frequency trading pipeline.

Case closed?

Oh, Philip, *yes you were right* - well done for spotting my *cough*
deliberate copy/paste mistake. What I'm using is in fact:

dd if=/dev/zero bs=1M count=500 2>/dev/null | openssl enc -rc4-40 -pass
pass:weak  > six


- Paul

On Fri, Jul 7, 2017 at 2:39 PM, Stefan Fuhrmann  wrote:

> On 07.07.2017 01:10, Paul Hammant wrote:
>
> With autorevision set to 'on' and curl:
>
> Reverence speed for boot drive to USB3 spinning platter 4TB thing:
>
> paul@paul-HiBox:~$ time cp /home/paul/clientDir/seven
> /media/paul/sg4t/sevenb
> real 0m1.539s
>
>
> That isn't exactly accurate - you write to the OS file
> cache and not to disk (yet). Subversion instructs the
> OS to flush the relevant parts its cache to make sure
> committed data will survive power failure etc.
>
> For a fair comparison, you should do something like:
>
> $ sync
> $ time (cp from/path to/path && sync)
>
> FSFS actually calls sync ~7 times per commit (store
> the revprops, the newest head revision number, ...).
> In 1.10, svnadmin has an option to temporarily disable
> this for things like "svnadmin load".
>
> Maybe, we could turn this into a fsfs.conf option for
> sufficiently safe / replicated environments. Not sure
> how I feel about that.
>
> Create a new 501MB file on Svn Server:
>
>
> paul@paul-HiBox:~$ time curl -u paul:myPassword
> http://192.168.1.178/svn/svnRepo1/twoB --upload-file
> /home/paul/clientDir/seven
> 201 Created
> real 0m49.442s
>
> *I ran that a couple more times and it was up there at 50s*
>
>
> Data compression via deflate (zip algorithm) is typically
> 20MB/s on a slower CPU. That accounts for most of the
> difference you see in the "compression off".
>
> You might try using compression-level=1. It still gives you
> some space savings - particularly in the "stupidly redundant
> data" cases - but is 3 .. 4 times faster than the default.
>
> Needlessly overwrite 501MB file (file is unchanged) on Svn Server:
>
>
> paul@paul-HiBox:~$ time curl -u paul:myPassword
> http://192.168.1.178/svn/svnRepo1/twoB --upload-file
> /home/paul/clientDir/seven
> real 0m13.426s
>
>
> This reads the old data, decompresses it, verifies its
> checksum and calculates the checksum of the incoming
> data twice (MD5 and SHA1).
>
> On a slow-ish CPU, you get something like
>
> * 100 MB/s decompress
> * 250 MB/s MD5 checksum verification
> * 250 MB/s MD5 checksum new contents
> * 200 MB/s SHA1 checksum new contents
>
> Roughly 12s for 500MB. A quick CPU will be twice as fast.
>
>
> Change the compression-level=0
>
> paul@paul-HiBox:~$ sudo nano /media/paul/sg4t/svnParent/svn
> Repo1/db/fsfs.conf
>
> Create a new 501MB file on Svn Server:
>
>
> paul@paul-HiBox:~$ time curl -u paul:myPassword
> http://192.168.1.178/svn/svnRepo1/twoC --upload-file
> /home/paul/clientDir/seven
> 201 Created
> real 0m15.312s
>
> Yay - a modest speed boost!!!
>
>
> That would be about 4..5s for the checksum calculation
> and the remaining time for the disk write (80..90MB/s).
>
>
> Restart Apache - which I didn't do before:
>
>
> paul@paul-HiBox:~$ systemctl restart apache2
>
> Create a new 501MB file on Svn Server:
>
>
> paul@paul-HiBox:~$ time curl -u paul:myPassword
> http://192.168.1.178/svn/svnRepo1/twoD --upload-file
> /home/paul/clientDir/seven
> 201 Created
> real 0m14.925s
>
>
> Conclusion:
>
> With compression-level=5 (default), there's is a 1:33 cp to curl-PUT ratio.
> With compression-level=0, there's is a 1:10 cp to curl-PUT ratio.
>
>
> If you want to measure pure CPU / protocol overhead,
> put your repository on a RAM disk. Most machines
> should be able to handle a <1GB repo.
>
>
> Is there there are other alluring settings, such as...
>
> enable-rep-sharing = false
>
> This kicks in only after the new data has been written
> (i.e. all has come in) and *then* it is found to be redundant.
> So, this option may safe quite a bit of disk space but does
> virtually nothing to reduce the time taken.
>
> enable-dir-deltification = false
>
> That *can* actually help, iff you have deeply nested,
> small directories. In that case, it will reduce I/O during
> commit. The effect on read access can be all over the place.
>
> ... but they didn't yield an improvement.
>
> Thanks for all the replies, gang.
>
>
> If you need to set up a repository for large, binary data,
> use the svn: protocol, tune the repo as previously discussed,
> pick a fast server CPU and you should be able to reach up
> to 200MB/s over a single network 

Re: Expected speed of commit over HTTP?

[Stefan Fuhrmann]

On 07.07.2017 01:10, Paul Hammant wrote:

With autorevision set to 'on' and curl:

Reverence speed for boot drive to USB3 spinning platter 4TB thing:

paul@paul-HiBox:~$ time cp /home/paul/clientDir/seven
/media/paul/sg4t/sevenb
real0m1.539s



That isn't exactly accurate - you write to the OS file
cache and not to disk (yet). Subversion instructs the
OS to flush the relevant parts its cache to make sure
committed data will survive power failure etc.

For a fair comparison, you should do something like:

$ sync
$ time (cp from/path to/path && sync)

FSFS actually calls sync ~7 times per commit (store
the revprops, the newest head revision number, ...).
In 1.10, svnadmin has an option to temporarily disable
this for things like "svnadmin load".

Maybe, we could turn this into a fsfs.conf option for
sufficiently safe / replicated environments. Not sure
how I feel about that.


Create a new 501MB file on Svn Server:


paul@paul-HiBox:~$ time curl -u paul:myPassword
http://192.168.1.178/svn/svnRepo1/twoB
 --upload-file
/home/paul/clientDir/seven
201 Created
real0m49.442s

/I ran that a couple more times and it was up there at 50s/



Data compression via deflate (zip algorithm) is typically
20MB/s on a slower CPU. That accounts for most of the
difference you see in the "compression off".

You might try using compression-level=1. It still gives you
some space savings - particularly in the "stupidly redundant
data" cases - but is 3 .. 4 times faster than the default.


Needlessly overwrite 501MB file (file is unchanged) on Svn Server:


paul@paul-HiBox:~$ time curl -u paul:myPassword
http://192.168.1.178/svn/svnRepo1/twoB
 --upload-file
/home/paul/clientDir/seven
real0m13.426s



This reads the old data, decompresses it, verifies its
checksum and calculates the checksum of the incoming
data twice (MD5 and SHA1).

On a slow-ish CPU, you get something like

* 100 MB/s decompress
* 250 MB/s MD5 checksum verification
* 250 MB/s MD5 checksum new contents
* 200 MB/s SHA1 checksum new contents

Roughly 12s for 500MB. A quick CPU will be twice as fast.



Change the compression-level=0

paul@paul-HiBox:~$ sudo nano
/media/paul/sg4t/svnParent/svnRepo1/db/fsfs.conf

Create a new 501MB file on Svn Server:


paul@paul-HiBox:~$ time curl -u paul:myPassword
http://192.168.1.178/svn/svnRepo1/twoC
 --upload-file
/home/paul/clientDir/seven
201 Created
real0m15.312s

Yay - a modest speed boost!!!


That would be about 4..5s for the checksum calculation
and the remaining time for the disk write (80..90MB/s).



Restart Apache - which I didn't do before:


paul@paul-HiBox:~$ systemctl restart apache2

Create a new 501MB file on Svn Server:


paul@paul-HiBox:~$ time curl -u paul:myPassword
http://192.168.1.178/svn/svnRepo1/twoD
 --upload-file
/home/paul/clientDir/seven
201 Created
real0m14.925s


Conclusion:

With compression-level=5 (default), there's is a 1:33 cp to
curl-PUT ratio.
With compression-level=0, there's is a 1:10 cp to curl-PUT ratio.



If you want to measure pure CPU / protocol overhead,
put your repository on a RAM disk. Most machines
should be able to handle a <1GB repo.


Is there there are other alluring settings, such as...

enable-rep-sharing = false


This kicks in only after the new data has been written
(i.e. all has come in) and *then* it is found to be redundant.
So, this option may safe quite a bit of disk space but does
virtually nothing to reduce the time taken.


enable-dir-deltification = false


That *can* actually help, iff you have deeply nested,
small directories. In that case, it will reduce I/O during
commit. The effect on read access can be all over the place.


... but they didn't yield an improvement.

Thanks for all the replies, gang.


If you need to set up a repository for large, binary data,
use the svn: protocol, tune the repo as previously discussed,
pick a fast server CPU and you should be able to reach up
to 200MB/s over a single network connection in 1.9 with
'svn import'. 'svn commit' takes about twice as long but
that could be fixed eventually as well.

-- Stefan^2.

Re: Expected speed of commit over HTTP?

[Branko Čibej]

On 07.07.2017 12:10, Paul Hammant wrote:
> ~/.subversion/servers contains a line that's tunable, right:
>
> # http-compression = no
>
> And per comment, tack on:
>
> --config-option=servers:global:http-compression=no .
>
> I'll report back the results.
>
>
> How is Apache and its modules organized?  As separate processes - with
> TCP/IP between each?
> If yes, is that interconnect loopback Unix domain sockets?

No, Apache modules are loadable shared libs. Whether these are all in
the same process or not depends on which MPM module is used. Data
transfer between modules is in-memory, AFAIK.

-- Brane

Re: Expected speed of commit over HTTP?

[Johan Corveleyn]

On Fri, Jul 7, 2017 at 8:54 AM, Jacek Materna  wrote:
> Paul,
>
> Got back from Ops. We don't have anything "special" setup other than what
> already been mentioned. Overall the defining metrics for commit performance
> are: ssh vs https, rtt on the network path and "randomness" of the blobs
> going up.
>
> No magic params on our end. We have it tuned the max via the metrics above.
>
> Most of the real world use cases require the default settings so curl, etc
> isn't seen a lot in the wild.
>
> Overall Dav and http is a poor transport in general for SVN.
>
> Hope it helps.
>
> --
> Jacek Materna

It would be interesting to know whether the remaining 1:10 factor is
caused by the transport or by the back-end (with its server-side
deltification and FSFS transaction features). I don't know if it would
be possible for Paul to test the same setup with svn+ssh.

OTOH, that would probably not be a very good comparison, because with
svn+ssh you can't use SVNAutoVersioning and curl, you'd have to use a
normal svn client, so IIUC it's not possible to eliminate the
client-side deltification. Unless Philip or someone else knows another
trick for that :).

-- 
Johan

Re: Expected speed of commit over HTTP?

[Jacek Materna]

Paul,

Got back from Ops. We don't have anything "special" setup other than what 
already been mentioned. Overall the defining metrics for commit performance 
are: ssh vs https, rtt on the network path and "randomness" of the blobs going 
up. 

No magic params on our end. We have it tuned the max via the metrics above. 

Most of the real world use cases require the default settings so curl, etc 
isn't seen a lot in the wild. 

Overall Dav and http is a poor transport in general for SVN. 

Hope it helps. 

--
Jacek Materna

Assembla
CTO
+1 210 410 7661
+48 578 296 708
Sent from my Mobile Device

> On Jul 7, 2017, at 1:10 AM, Paul Hammant  wrote:
> 
> With autorevision set to 'on' and curl:
> 
> Reverence speed for boot drive to USB3 spinning platter 4TB thing:
> 
> paul@paul-HiBox:~$ time cp /home/paul/clientDir/seven /media/paul/sg4t/sevenb
> real  0m1.539s
> 
> Create a new 501MB file on Svn Server:
> 
> paul@paul-HiBox:~$ time curl -u paul:myPassword  
> http://192.168.1.178/svn/svnRepo1/twoB --upload-file 
> /home/paul/clientDir/seven
> 201 Created
> real  0m49.442s
> 
> I ran that a couple more times and it was up there at 50s
> 
> Needlessly overwrite 501MB file (file is unchanged) on Svn Server:
> 
> paul@paul-HiBox:~$ time curl -u paul:myPassword  
> http://192.168.1.178/svn/svnRepo1/twoB --upload-file 
> /home/paul/clientDir/seven
> real  0m13.426s
> 
> Change the compression-level=0
> 
> paul@paul-HiBox:~$ sudo nano /media/paul/sg4t/svnParent/svnRepo1/db/fsfs.conf 
> 
> Create a new 501MB file on Svn Server:
> 
> paul@paul-HiBox:~$ time curl -u paul:myPassword  
> http://192.168.1.178/svn/svnRepo1/twoC --upload-file 
> /home/paul/clientDir/seven
> 201 Created
> real  0m15.312s
> 
> Yay - a modest speed boost!!!
> 
> Restart Apache - which I didn't do before:
> 
> paul@paul-HiBox:~$ systemctl restart apache2
> 
> Create a new 501MB file on Svn Server:
> 
> paul@paul-HiBox:~$ time curl -u paul:myPassword  
> http://192.168.1.178/svn/svnRepo1/twoD --upload-file 
> /home/paul/clientDir/seven
> 201 Created
> real  0m14.925s
> 
> Conclusion:
> 
> With compression-level=5 (default), there's is a 1:33 cp to curl-PUT ratio.
> With compression-level=0, there's is a 1:10 cp to curl-PUT ratio.
> 
> Is there there are other alluring settings, such as...
> enable-rep-sharing = false
> enable-dir-deltification = false
> 
> ... but they didn't yield an improvement.
> 
> Thanks for all the replies, gang.
> 
> -  Paul

Re: Expected speed of commit over HTTP?

[Paul Hammant]

With autorevision set to 'on' and curl:

Reverence speed for boot drive to USB3 spinning platter 4TB thing:

paul@paul-HiBox:~$ time cp /home/paul/clientDir/seven
/media/paul/sg4t/sevenb
real 0m1.539s

Create a new 501MB file on Svn Server:


paul@paul-HiBox:~$ time curl -u paul:myPassword  http://192.168.1.178/svn/
svnRepo1/twoB --upload-file /home/paul/clientDir/seven
201 Created
real 0m49.442s

*I ran that a couple more times and it was up there at 50s*


Needlessly overwrite 501MB file (file is unchanged) on Svn Server:


paul@paul-HiBox:~$ time curl -u paul:myPassword  http://192.168.1.178/svn/
svnRepo1/twoB --upload-file /home/paul/clientDir/seven
real 0m13.426s

Change the compression-level=0

paul@paul-HiBox:~$ sudo nano /media/paul/sg4t/svnParent/
svnRepo1/db/fsfs.conf

Create a new 501MB file on Svn Server:


paul@paul-HiBox:~$ time curl -u paul:myPassword  http://192.168.1.178/svn/
svnRepo1/twoC --upload-file /home/paul/clientDir/seven
201 Created
real 0m15.312s

Yay - a modest speed boost!!!


Restart Apache - which I didn't do before:


paul@paul-HiBox:~$ systemctl restart apache2

Create a new 501MB file on Svn Server:


paul@paul-HiBox:~$ time curl -u paul:myPassword  http://192.168.1.178/svn/
svnRepo1/twoD --upload-file /home/paul/clientDir/seven
201 Created
real 0m14.925s


Conclusion:

With compression-level=5 (default), there's is a 1:33 cp to curl-PUT ratio.
With compression-level=0, there's is a 1:10 cp to curl-PUT ratio.


Is there there are other alluring settings, such as...

enable-rep-sharing = false
enable-dir-deltification = false

... but they didn't yield an improvement.

Thanks for all the replies, gang.

-  Paul

Re: Expected speed of commit over HTTP?

[Philip Martin]

Johan Corveleyn  writes:

>> On my machine:
>>
>>   dd:  3.3 sec
>>   svn commit: 37   sec
>>   svnmucc commit: 26   sec
>>  curl commit:  9.5 sec
> [ cc += Stefan Fuhrman, who might have some more ideas about this ]
>
> If your SVN server is version 1.9 with FSFS backend, your could try if
> the setting "compression-level" in the deltification section in
> $REPOS/db/fsfs.conf makes a difference:

Good idea!

With compression-level=0

   dd:  3.2 sec
   svn commit: 26   sec
   svnmucc commit: 16   sec
  curl commit:  3.9 sec

-- 
Philip

Re: Expected speed of commit over HTTP?

[Johan Corveleyn]

On Thu, Jul 6, 2017 at 4:10 PM, Philip Martin  wrote:
> Paul Hammant  writes:
>
>> I'm making each revision with..
>>
>>   dd if=/dev/zero bs=1M count=500 2>/dev/null >
>> path/to/file/under/versionControl.dat
>>
>> .. which is random enough to completely thwart delta analysis of the file.
>> That's slow enough in itself, but I'm only checking the time of the commit.
>
> The standard client always sends deltas even for cases like yours where
> the delta is not really an advantage. On receiving the delta the server
> has to first reconstruct the full text and then construct a second delta
> to store in the repository.
>
> You can make commits a bit faster by using svnmucc without a working
> copy, the svnmucc client always sends a delta against an empty file
> which is faster to calculate than the standard client delta against the
> previous file contents:
>
>   svnmucc put some/file URL
>
> You can make commits even faster by enabling SVNAutoversioning on the
> server and using curl as your client as then the client doesn't
> calculate delta at all:
>
>   curl -XPUT @some/file URL
>
> The curl commit will still involve calculating a delta on the server.
>
> On my machine:
>
>   dd:  3.3 sec
>   svn commit: 37   sec
>   svnmucc commit: 26   sec
>  curl commit:  9.5 sec

[ cc += Stefan Fuhrman, who might have some more ideas about this ]

If your SVN server is version 1.9 with FSFS backend, your could try if
the setting "compression-level" in the deltification section in
$REPOS/db/fsfs.conf makes a difference:

[[[
### After deltification, we compress the data through zlib to minimize on-
### disk size.  That can be an expensive and ineffective process.  This
### setting controls the usage of zlib in future revisions.
### Revisions with highly compressible data in them may shrink in size
### if the setting is increased but may take much longer to commit.  The
### time taken to uncompress that data again is widely independent of the
### compression level.
### Compression will be ineffective if the incoming content is already
### highly compressed.  In that case, disabling the compression entirely
### will speed up commits as well as reading the data.  Repositories with
### many small compressible files (source code) but also a high percentage
### of large incompressible ones (artwork) may benefit from compression
### levels lowered to e.g. 1.
### Valid values are 0 to 9 with 9 providing the highest compression ratio
### and 0 disabling it altogether.
### The default value is 5.
# compression-level = 5
]]]

There is also the mod_dav_svn config directive SVNCompressionLevel [1]
and the client-side option http-compression [2] which you can try ...
don't know if those are a significant factor in your case.

[1] http://svnbook.red-bean.com/en/1.7/svn.ref.mod_dav_svn.conf.html

[2] see your ~/.subversion/servers file, or you can specify it for a
single command with an option:
--config-option servers:global:http-compression=no

-- 
Johan

Re: Expected speed of commit over HTTP?

[Philip Martin]

Philip Martin  writes:

> Paul Hammant  writes:
>
>> I'm making each revision with..
>>
>>   dd if=/dev/zero bs=1M count=500 2>/dev/null >
>> path/to/file/under/versionControl.dat
>>
>> .. which is random enough to completely thwart delta analysis of the file.
>> That's slow enough in itself, but I'm only checking the time of the commit.
>
> The standard client always sends deltas even for cases like yours where

I'm assuming /dev/urandom as /dev/zero doesn't really make sense here.

-- 
Philip

Re: Expected speed of commit over HTTP?

[Philip Martin]

Paul Hammant  writes:

> I'm making each revision with..
>
>   dd if=/dev/zero bs=1M count=500 2>/dev/null >
> path/to/file/under/versionControl.dat
>
> .. which is random enough to completely thwart delta analysis of the file.
> That's slow enough in itself, but I'm only checking the time of the commit.

The standard client always sends deltas even for cases like yours where
the delta is not really an advantage. On receiving the delta the server
has to first reconstruct the full text and then construct a second delta
to store in the repository.

You can make commits a bit faster by using svnmucc without a working
copy, the svnmucc client always sends a delta against an empty file
which is faster to calculate than the standard client delta against the
previous file contents:

  svnmucc put some/file URL

You can make commits even faster by enabling SVNAutoversioning on the
server and using curl as your client as then the client doesn't
calculate delta at all:

  curl -XPUT @some/file URL

The curl commit will still involve calculating a delta on the server.

On my machine:

  dd:  3.3 sec 
  svn commit: 37   sec
  svnmucc commit: 26   sec
 curl commit:  9.5 sec

-- 
Philip

Re: Expected speed of commit over HTTP?

[Jacek Materna]

Let me go do some digging with the Ops team-

On Thu, Jul 6, 2017 at 2:31 PM, Paul Hammant  wrote:

> Jacek, Can you share any config settings for mod_dav that might be areas
> to experiment with ?
>
> And, yes, just because the file commit has no discernible deltas at all,
> doesn't mean that Svn doesn't attempt to make its own determination.  I
> wasn't using .bin suffixes as it happens but I'll bet there's something in
> amongst mimetypes and suffixes somewhere that allows skipping.
>
> -ph
>
> On Thu, Jul 6, 2017 at 8:07 AM, Jacek Materna  wrote:
>
>> We've say 2-3x speed decrease using HTTP vs SSH in our cloud. With SSH
>> and some tuning we are getting close to the IO rate on the end server of
>> any size. HTTP/mod_dav and RTT is usually the culprit in our cases, on a
>> LAN that issues collapses. Likely server side processing of the delta's?
>>
>> -jacek
>>
>> On Thu, Jul 6, 2017 at 1:54 PM, Paul Hammant  wrote:
>>
>>> For something that's 500MB in size (random binary data) I'm experiencing
>>> commits taking
>>> 10x longer than a straight copy to the drive the Svn repo is on.
>>>
>>> Both timings are on the same Ubuntu 17.04 machine, with the boot drive
>>> being the starting position of the 512MB file and a USB3 mounted 4TB
>>> seagate hard drive being the destination.
>>>
>>> My goal is to fill the 4TB drive with commits for the simple experience
>>> of that.
>>>
>>> How many places in the Apache2 --> mod_dav --> mod_dav_svn handoff does
>>> the 512MB temporarily manifest itself in a file system on the way to its
>>> ultimate destination?  Is 1/10th speed the expectation?  Sure, I get that
>>> 7bit/8bit shenanigans are a factor, but not that much right?
>>>
>>> - Paul
>>>
>>>
>>
>>
>> --
>>
>> Jacek Materna
>> CTO
>>
>> Assembla
>> +1 210 410 7661 <(210)%20410-7661>
>> +48 578 296 708 <+48%20578%20296%20708>
>>
>
>


-- 

Jacek Materna
CTO

Assembla
+1 210 410 7661
+48 578 296 708

Re: Expected speed of commit over HTTP?

[Paul Hammant]

Jacek, Can you share any config settings for mod_dav that might be areas to
experiment with ?

And, yes, just because the file commit has no discernible deltas at all,
doesn't mean that Svn doesn't attempt to make its own determination.  I
wasn't using .bin suffixes as it happens but I'll bet there's something in
amongst mimetypes and suffixes somewhere that allows skipping.

-ph

On Thu, Jul 6, 2017 at 8:07 AM, Jacek Materna  wrote:

> We've say 2-3x speed decrease using HTTP vs SSH in our cloud. With SSH and
> some tuning we are getting close to the IO rate on the end server of any
> size. HTTP/mod_dav and RTT is usually the culprit in our cases, on a LAN
> that issues collapses. Likely server side processing of the delta's?
>
> -jacek
>
> On Thu, Jul 6, 2017 at 1:54 PM, Paul Hammant  wrote:
>
>> For something that's 500MB in size (random binary data) I'm experiencing
>> commits taking
>> 10x longer than a straight copy to the drive the Svn repo is on.
>>
>> Both timings are on the same Ubuntu 17.04 machine, with the boot drive
>> being the starting position of the 512MB file and a USB3 mounted 4TB
>> seagate hard drive being the destination.
>>
>> My goal is to fill the 4TB drive with commits for the simple experience
>> of that.
>>
>> How many places in the Apache2 --> mod_dav --> mod_dav_svn handoff does
>> the 512MB temporarily manifest itself in a file system on the way to its
>> ultimate destination?  Is 1/10th speed the expectation?  Sure, I get that
>> 7bit/8bit shenanigans are a factor, but not that much right?
>>
>> - Paul
>>
>>
>
>
> --
>
> Jacek Materna
> CTO
>
> Assembla
> +1 210 410 7661 <(210)%20410-7661>
> +48 578 296 708 <+48%20578%20296%20708>
>

Re: Expected speed of commit over HTTP?

[Paul Hammant]

I'm making each revision with..

  dd if=/dev/zero bs=1M count=500 2>/dev/null >
path/to/file/under/versionControl.dat

.. which is random enough to completely thwart delta analysis of the file.
That's slow enough in itself, but I'm only checking the time of the commit.

- Paul

On Thu, Jul 6, 2017 at 8:03 AM, Branko Čibej  wrote:

> On 06.07.2017 13:54, Paul Hammant wrote:
> > For something that's 500MB in size (random binary data) I'm
> > experiencing commits taking
> > 10x longer than a straight copy to the drive the Svn repo is on.
> >
> > Both timings are on the same Ubuntu 17.04 machine, with the boot drive
> > being the starting position of the 512MB file and a USB3 mounted 4TB
> > seagate hard drive being the destination.
> >
> > My goal is to fill the 4TB drive with commits for the simple
> > experience of that.
> >
> > How many places in the Apache2 --> mod_dav --> mod_dav_svn handoff
> > does the 512MB temporarily manifest itself in a file system on the way
> > to its ultimate destination?  Is 1/10th speed the expectation?  Sure,
> > I get that 7bit/8bit shenanigans are a factor, but not that much right?
>
> 7 vs. 8bit shouldn't be an issue; data transfers are binary. If you're
> committing updates to the file, deltification could be part of the
> reason for the slowdown. Other than that ... performance measurements
> would help.
>
> -- Brane
>
>

Re: Expected speed of commit over HTTP?

[Jacek Materna]

We've say 2-3x speed decrease using HTTP vs SSH in our cloud. With SSH and
some tuning we are getting close to the IO rate on the end server of any
size. HTTP/mod_dav and RTT is usually the culprit in our cases, on a LAN
that issues collapses. Likely server side processing of the delta's?

-jacek

On Thu, Jul 6, 2017 at 1:54 PM, Paul Hammant  wrote:

> For something that's 500MB in size (random binary data) I'm experiencing
> commits taking
> 10x longer than a straight copy to the drive the Svn repo is on.
>
> Both timings are on the same Ubuntu 17.04 machine, with the boot drive
> being the starting position of the 512MB file and a USB3 mounted 4TB
> seagate hard drive being the destination.
>
> My goal is to fill the 4TB drive with commits for the simple experience of
> that.
>
> How many places in the Apache2 --> mod_dav --> mod_dav_svn handoff does
> the 512MB temporarily manifest itself in a file system on the way to its
> ultimate destination?  Is 1/10th speed the expectation?  Sure, I get that
> 7bit/8bit shenanigans are a factor, but not that much right?
>
> - Paul
>
>


-- 

Jacek Materna
CTO

Assembla
+1 210 410 7661
+48 578 296 708

Re: Expected speed of commit over HTTP?

[Branko Čibej]

On 06.07.2017 13:54, Paul Hammant wrote:
> For something that's 500MB in size (random binary data) I'm
> experiencing commits taking 
> 10x longer than a straight copy to the drive the Svn repo is on.  
>
> Both timings are on the same Ubuntu 17.04 machine, with the boot drive
> being the starting position of the 512MB file and a USB3 mounted 4TB
> seagate hard drive being the destination.
>
> My goal is to fill the 4TB drive with commits for the simple
> experience of that.
>
> How many places in the Apache2 --> mod_dav --> mod_dav_svn handoff
> does the 512MB temporarily manifest itself in a file system on the way
> to its ultimate destination?  Is 1/10th speed the expectation?  Sure,
> I get that 7bit/8bit shenanigans are a factor, but not that much right?

7 vs. 8bit shouldn't be an issue; data transfers are binary. If you're
committing updates to the file, deltification could be part of the
reason for the slowdown. Other than that ... performance measurements
would help.

-- Brane