Proposal: new fsfs.conf properties

[Paul Hammant]

1. compression-exempt-suffixes = mp3,mp4,jpeg

2. deltification-exempt-suffixes = mp3,mp4,jpeg

Regardless of the setting of 'compression-level', #1 above two mean certain
things can skip the compression attempt.  It must give up at a certain
point right?

Same for deltification re #2

I'm assuming debate happens now. Then y'all let me go off and diligently
file a Jira ticket for this feature request, or I slink away suitably
admonished...

- Paul

Re: Expected speed of commit over HTTP?

[Paul Hammant]

Great insights, Stefan.

So 'cp' with that sync, timed as you outline is averaging 3x slower than
the cp as I had it before.

Thus, that same cp with sync tacked on, is 3x faster that the curl PUT over
HTTP.

That's within the place where any speedup spent on the handoff of between
apache modules would be $100K of programming for a 15% speedup - not worth
it. At not worth it because mine is an esoteric use case and (say) Goldman
don't have Svn in a high frequency trading pipeline.

Case closed?

Oh, Philip, *yes you were right* - well done for spotting my *cough*
deliberate copy/paste mistake. What I'm using is in fact:

dd if=/dev/zero bs=1M count=500 2>/dev/null | openssl enc -rc4-40 -pass
pass:weak  > six


- Paul

On Fri, Jul 7, 2017 at 2:39 PM, Stefan Fuhrmann  wrote:

> On 07.07.2017 01:10, Paul Hammant wrote:
>
> With autorevision set to 'on' and curl:
>
> Reverence speed for boot drive to USB3 spinning platter 4TB thing:
>
> paul@paul-HiBox:~$ time cp /home/paul/clientDir/seven
> /media/paul/sg4t/sevenb
> real 0m1.539s
>
>
> That isn't exactly accurate - you write to the OS file
> cache and not to disk (yet). Subversion instructs the
> OS to flush the relevant parts its cache to make sure
> committed data will survive power failure etc.
>
> For a fair comparison, you should do something like:
>
> $ sync
> $ time (cp from/path to/path && sync)
>
> FSFS actually calls sync ~7 times per commit (store
> the revprops, the newest head revision number, ...).
> In 1.10, svnadmin has an option to temporarily disable
> this for things like "svnadmin load".
>
> Maybe, we could turn this into a fsfs.conf option for
> sufficiently safe / replicated environments. Not sure
> how I feel about that.
>
> Create a new 501MB file on Svn Server:
>
>
> paul@paul-HiBox:~$ time curl -u paul:myPassword
> http://192.168.1.178/svn/svnRepo1/twoB --upload-file
> /home/paul/clientDir/seven
> 201 Created
> real 0m49.442s
>
> *I ran that a couple more times and it was up there at 50s*
>
>
> Data compression via deflate (zip algorithm) is typically
> 20MB/s on a slower CPU. That accounts for most of the
> difference you see in the "compression off".
>
> You might try using compression-level=1. It still gives you
> some space savings - particularly in the "stupidly redundant
> data" cases - but is 3 .. 4 times faster than the default.
>
> Needlessly overwrite 501MB file (file is unchanged) on Svn Server:
>
>
> paul@paul-HiBox:~$ time curl -u paul:myPassword
> http://192.168.1.178/svn/svnRepo1/twoB --upload-file
> /home/paul/clientDir/seven
> real 0m13.426s
>
>
> This reads the old data, decompresses it, verifies its
> checksum and calculates the checksum of the incoming
> data twice (MD5 and SHA1).
>
> On a slow-ish CPU, you get something like
>
> * 100 MB/s decompress
> * 250 MB/s MD5 checksum verification
> * 250 MB/s MD5 checksum new contents
> * 200 MB/s SHA1 checksum new contents
>
> Roughly 12s for 500MB. A quick CPU will be twice as fast.
>
>
> Change the compression-level=0
>
> paul@paul-HiBox:~$ sudo nano /media/paul/sg4t/svnParent/svn
> Repo1/db/fsfs.conf
>
> Create a new 501MB file on Svn Server:
>
>
> paul@paul-HiBox:~$ time curl -u paul:myPassword
> http://192.168.1.178/svn/svnRepo1/twoC --upload-file
> /home/paul/clientDir/seven
> 201 Created
> real 0m15.312s
>
> Yay - a modest speed boost!!!
>
>
> That would be about 4..5s for the checksum calculation
> and the remaining time for the disk write (80..90MB/s).
>
>
> Restart Apache - which I didn't do before:
>
>
> paul@paul-HiBox:~$ systemctl restart apache2
>
> Create a new 501MB file on Svn Server:
>
>
> paul@paul-HiBox:~$ time curl -u paul:myPassword
> http://192.168.1.178/svn/svnRepo1/twoD --upload-file
> /home/paul/clientDir/seven
> 201 Created
> real 0m14.925s
>
>
> Conclusion:
>
> With compression-level=5 (default), there's is a 1:33 cp to curl-PUT ratio.
> With compression-level=0, there's is a 1:10 cp to curl-PUT ratio.
>
>
> If you want to measure pure CPU / protocol overhead,
> put your repository on a RAM disk. Most machines
> should be able to handle a <1GB repo.
>
>
> Is there there are other alluring settings, such as...
>
> enable-rep-sharing = false
>
> This kicks in only after the new data has been written
> (i.e. all has come in) and *then* it is found to be redundant.
> So, this option may safe quite a bit of disk space but does
> virtually nothing to reduce the time taken.
>
> enable-dir-deltification = false
>
> That *can* actually help, iff you have deeply nested,
> small directories. In that case, it will reduce I/O during
> commit. The effect on read access can be all over the place.
>
> ... but they didn't yield an improvement.
>
> Thanks for all the replies, gang.
>
>
> If you need to set up a repository for large, binary data,
> use the svn: protocol, tune the repo as previously discussed,
> pick a fast server CPU and you should be able to reach up
> to 200MB/s over a single network connection in 1.9 with
> 'svn import'. 

Re: Expected speed of commit over HTTP?

[Stefan Fuhrmann]

On 07.07.2017 01:10, Paul Hammant wrote:

With autorevision set to 'on' and curl:

Reverence speed for boot drive to USB3 spinning platter 4TB thing:

paul@paul-HiBox:~$ time cp /home/paul/clientDir/seven
/media/paul/sg4t/sevenb
real0m1.539s



That isn't exactly accurate - you write to the OS file
cache and not to disk (yet). Subversion instructs the
OS to flush the relevant parts its cache to make sure
committed data will survive power failure etc.

For a fair comparison, you should do something like:

$ sync
$ time (cp from/path to/path && sync)

FSFS actually calls sync ~7 times per commit (store
the revprops, the newest head revision number, ...).
In 1.10, svnadmin has an option to temporarily disable
this for things like "svnadmin load".

Maybe, we could turn this into a fsfs.conf option for
sufficiently safe / replicated environments. Not sure
how I feel about that.


Create a new 501MB file on Svn Server:


paul@paul-HiBox:~$ time curl -u paul:myPassword
http://192.168.1.178/svn/svnRepo1/twoB
 --upload-file
/home/paul/clientDir/seven
201 Created
real0m49.442s

/I ran that a couple more times and it was up there at 50s/



Data compression via deflate (zip algorithm) is typically
20MB/s on a slower CPU. That accounts for most of the
difference you see in the "compression off".

You might try using compression-level=1. It still gives you
some space savings - particularly in the "stupidly redundant
data" cases - but is 3 .. 4 times faster than the default.


Needlessly overwrite 501MB file (file is unchanged) on Svn Server:


paul@paul-HiBox:~$ time curl -u paul:myPassword
http://192.168.1.178/svn/svnRepo1/twoB
 --upload-file
/home/paul/clientDir/seven
real0m13.426s



This reads the old data, decompresses it, verifies its
checksum and calculates the checksum of the incoming
data twice (MD5 and SHA1).

On a slow-ish CPU, you get something like

* 100 MB/s decompress
* 250 MB/s MD5 checksum verification
* 250 MB/s MD5 checksum new contents
* 200 MB/s SHA1 checksum new contents

Roughly 12s for 500MB. A quick CPU will be twice as fast.



Change the compression-level=0

paul@paul-HiBox:~$ sudo nano
/media/paul/sg4t/svnParent/svnRepo1/db/fsfs.conf

Create a new 501MB file on Svn Server:


paul@paul-HiBox:~$ time curl -u paul:myPassword
http://192.168.1.178/svn/svnRepo1/twoC
 --upload-file
/home/paul/clientDir/seven
201 Created
real0m15.312s

Yay - a modest speed boost!!!


That would be about 4..5s for the checksum calculation
and the remaining time for the disk write (80..90MB/s).



Restart Apache - which I didn't do before:


paul@paul-HiBox:~$ systemctl restart apache2

Create a new 501MB file on Svn Server:


paul@paul-HiBox:~$ time curl -u paul:myPassword
http://192.168.1.178/svn/svnRepo1/twoD
 --upload-file
/home/paul/clientDir/seven
201 Created
real0m14.925s


Conclusion:

With compression-level=5 (default), there's is a 1:33 cp to
curl-PUT ratio.
With compression-level=0, there's is a 1:10 cp to curl-PUT ratio.



If you want to measure pure CPU / protocol overhead,
put your repository on a RAM disk. Most machines
should be able to handle a <1GB repo.


Is there there are other alluring settings, such as...

enable-rep-sharing = false


This kicks in only after the new data has been written
(i.e. all has come in) and *then* it is found to be redundant.
So, this option may safe quite a bit of disk space but does
virtually nothing to reduce the time taken.


enable-dir-deltification = false


That *can* actually help, iff you have deeply nested,
small directories. In that case, it will reduce I/O during
commit. The effect on read access can be all over the place.


... but they didn't yield an improvement.

Thanks for all the replies, gang.


If you need to set up a repository for large, binary data,
use the svn: protocol, tune the repo as previously discussed,
pick a fast server CPU and you should be able to reach up
to 200MB/s over a single network connection in 1.9 with
'svn import'. 'svn commit' takes about twice as long but
that could be fixed eventually as well.

-- Stefan^2.

Re: [PATCH] Tweak the SHA-1 FAQ entry

[Evgeny Kotkov]

Bert Huijben  writes:

> Except for s/ it / is / on this line, +1... Looks very good!

Jacek Materna  writes:

> Fair enough!
>
> Rest looks great.

I fixed the typo and committed the patch in https://svn.apache.org/r1801202


Regards,
Evgeny Kotkov

Re: svn commit: r1801108 - in /subversion/trunk/subversion/bindings/javahl/native: Array.cpp CreateJ.cpp Iterator.cpp OperationContext.cpp RemoteSession.cpp RevisionRangeList.cpp

[Stefan Fuhrmann]

On 06.07.2017 23:11, Philip Martin wrote:

Philip Martin  writes:


I've upgraded my JDK and it produced all these warnings.

Excellent to see these warnings being addressed!


There is a second class of warnings of the form:

   WARNING: JNI local refs: 57, exceeds capacity: 56

They all seem to be off by one. But that could well be
an artefact of the first one too many getting reported.

These are generated in two places: JNIUtil::putErrorsInTrace() and
Java_org_apache_subversion_javahl_util_PropLib_unparseExternals().

I'm not sure how best to fix these so I have a local hack in
JNIUtil::wrappedCreateClientException to remove most of them:

// Create a local frame for our references
-  env->PushLocalFrame(LOCAL_FRAME_SIZE);
+  env->PushLocalFrame(LOCAL_FRAME_SIZE + 100);

Maybe, it is the SVN_ERR__TRACING section in that method
that causes the overflow?

-- Stefan^2.

Re: [PATCH] Tweak the SHA-1 FAQ entry

[Jacek Materna]

Fair enough!

Rest looks great. 

--
Jacek Materna

Assembla
CTO
+1 210 410 7661
+48 578 296 708
Sent from my Mobile Device

> On Jul 7, 2017, at 5:47 PM, Evgeny Kotkov  wrote:
> 
> Jacek Materna  writes:
> 
>> Shouldn't vulnerability / shattered remain a keyword? I believe it was
>> a comment in the patch I submitted - just wondering about google-ness
>> of it... shattered has a lot of google juice.
> 
> I was thinking about turning this FAQ entry into a more or less generic
> statement about Subversion and SHA-1 collisions, rather than making it
> centered around one particular kind of collision (https://shattered.io/).
> 
> 
> Regards,
> Evgeny Kotkov

Re: [PATCH] Tweak the SHA-1 FAQ entry

[Evgeny Kotkov]

Jacek Materna  writes:

> Shouldn't vulnerability / shattered remain a keyword? I believe it was
> a comment in the patch I submitted - just wondering about google-ness
> of it... shattered has a lot of google juice.

I was thinking about turning this FAQ entry into a more or less generic
statement about Subversion and SHA-1 collisions, rather than making it
centered around one particular kind of collision (https://shattered.io/).


Regards,
Evgeny Kotkov

Re: [PATCH] Tweak the SHA-1 FAQ entry

[Jacek Materna]

Shouldn't vulnerability / shattered remain a keyword? I believe it was
a comment in the patch I submitted - just wondering about google-ness
of it... shattered has a lot of google juice.

collision vs. 'shattered / vulnerability'

Otherwise more complete verbiage is always good.

On Fri, Jul 7, 2017 at 4:46 PM, Evgeny Kotkov
 wrote:
>
> Hi all,
>
> I made an attempt to tweak the SHA-1 FAQ entry (which is available at
> https://subversion.apache.org/faq#shattered-sha1) to make it a bit more
> user-friendly.
>
> Please see the attached patch.  What do you think about making a change
> like this?
>
> For convenience, here is the final result as plain text:
>
> [[[
> How is Subversion affected by SHA-1 hash collisions?
>
> Publication of the first known SHA-1 collision by Google and CWI unveiled a
> couple of related issues in the Subversion's use of SHA-1. The Subversion's
> core does not rely on SHA-1 for content indexing, but it was being used for
> such purposes in the following supplementary features:
>
> - repository data deduplication feature, and
> - content deduplication feature in the working copy.
>
> Speaking of the repository data deduplication feature, this can result in
> inability to access files with colliding SHA-1 values or cause data loss for
> such files. To prevent different content with identical SHA-1 from being
> stored in a repository, upgrade to 1.9.6 or 1.8.18 which, by default,
> prevent storing data with such collisions. See our SHA-1 advisory for
> details.
>
> Until the upgrade to these new releases is available, Unix-based servers can
> use the pre-commit hook found here. As an aside, we welcome Windows developers
> to submit a pre-commit script for the Windows platform. More information on
> submission can be found here.
>
> The working copy uses SHA-1 for deduplication of the stored content, and for
> performance reasons a client will avoid fetching content with the same SHA-1
> checksum. The workaround for this issue is to prevent storage of the colliding
> objects in the first place, via upgrade to 1.9.6 or installation of the
> aforementioned pre-commit script.
>
> Storing content with SHA-1 collisions it not a supported use case. If you
> have content with colliding SHA-1 hash values, we suggest you transform it
> via gzip before committing it to avoid the collision altogether. Moreover,
> an upgrade to 1.9.6 to prevent future insertion of duplicates is highly
> recommended.
> ]]]
>
>
> Regards,
> Evgeny Kotkov




-- 

Jacek Materna
CTO

Assembla
+1 210 410 7661
+48 578 296 708

RE: [PATCH] Tweak the SHA-1 FAQ entry

[Bert Huijben]

> -Original Message-
> From: Evgeny Kotkov [mailto:evgeny.kot...@visualsvn.com]
> Sent: vrijdag 7 juli 2017 16:46
> To: Subversion Development 
> Subject: [PATCH] Tweak the SHA-1 FAQ entry
> 
> Hi all,
> 
> I made an attempt to tweak the SHA-1 FAQ entry (which is available at
> https://subversion.apache.org/faq#shattered-sha1) to make it a bit more
> user-friendly.
> 
> Please see the attached patch.  What do you think about making a change
> like this?
> 
> For convenience, here is the final result as plain text:
> 
> [[[
> How is Subversion affected by SHA-1 hash collisions?
> 
> Publication of the first known SHA-1 collision by Google and CWI unveiled a
> couple of related issues in the Subversion's use of SHA-1. The Subversion's
> core does not rely on SHA-1 for content indexing, but it was being used for
> such purposes in the following supplementary features:
> 
> - repository data deduplication feature, and
> - content deduplication feature in the working copy.
> 
> Speaking of the repository data deduplication feature, this can result in
> inability to access files with colliding SHA-1 values or cause data loss for
> such files. To prevent different content with identical SHA-1 from being
> stored in a repository, upgrade to 1.9.6 or 1.8.18 which, by default,
> prevent storing data with such collisions. See our SHA-1 advisory for
> details.
> 
> Until the upgrade to these new releases is available, Unix-based servers can
> use the pre-commit hook found here. As an aside, we welcome Windows
> developers
> to submit a pre-commit script for the Windows platform. More information
> on
> submission can be found here.
> 
> The working copy uses SHA-1 for deduplication of the stored content, and for
> performance reasons a client will avoid fetching content with the same SHA-1
> checksum. The workaround for this issue is to prevent storage of the colliding
> objects in the first place, via upgrade to 1.9.6 or installation of the
> aforementioned pre-commit script.
> 
> Storing content with SHA-1 collisions it not a supported use case. If you

Except for s/ it / is / on this line, +1... Looks very good!

Thanks,

Bert

> have content with colliding SHA-1 hash values, we suggest you transform it
> via gzip before committing it to avoid the collision altogether. Moreover,
> an upgrade to 1.9.6 to prevent future insertion of duplicates is highly
> recommended.
> ]]]
> 
> 
> Regards,
> Evgeny Kotkov

[PATCH] Tweak the SHA-1 FAQ entry

[Evgeny Kotkov]

Hi all,

I made an attempt to tweak the SHA-1 FAQ entry (which is available at
https://subversion.apache.org/faq#shattered-sha1) to make it a bit more
user-friendly.

Please see the attached patch.  What do you think about making a change
like this?

For convenience, here is the final result as plain text:

[[[
How is Subversion affected by SHA-1 hash collisions?

Publication of the first known SHA-1 collision by Google and CWI unveiled a
couple of related issues in the Subversion's use of SHA-1. The Subversion's
core does not rely on SHA-1 for content indexing, but it was being used for
such purposes in the following supplementary features:

- repository data deduplication feature, and
- content deduplication feature in the working copy.

Speaking of the repository data deduplication feature, this can result in
inability to access files with colliding SHA-1 values or cause data loss for
such files. To prevent different content with identical SHA-1 from being
stored in a repository, upgrade to 1.9.6 or 1.8.18 which, by default,
prevent storing data with such collisions. See our SHA-1 advisory for
details.

Until the upgrade to these new releases is available, Unix-based servers can
use the pre-commit hook found here. As an aside, we welcome Windows developers
to submit a pre-commit script for the Windows platform. More information on
submission can be found here.

The working copy uses SHA-1 for deduplication of the stored content, and for
performance reasons a client will avoid fetching content with the same SHA-1
checksum. The workaround for this issue is to prevent storage of the colliding
objects in the first place, via upgrade to 1.9.6 or installation of the
aforementioned pre-commit script.

Storing content with SHA-1 collisions it not a supported use case. If you
have content with colliding SHA-1 hash values, we suggest you transform it
via gzip before committing it to avoid the collision altogether. Moreover,
an upgrade to 1.9.6 to prevent future insertion of duplicates is highly
recommended.
]]]


Regards,
Evgeny Kotkov
Index: publish/faq.html
===
--- publish/faq.html(revision 1801149)
+++ publish/faq.html(working copy)
@@ -61,8 +61,8 @@ For older questions, see 
 How is Subversion affected by changes
 in Daylight Savings Time (DST)?
-How do I protect my repository from the SHA-1
-Shattered vulnerability?
+How is Subversion affected by SHA-1 hash
+collisions?
 
 
 How-to:
@@ -748,28 +748,30 @@ for DST.
 
 
-How do I protect my repository from the SHA-1 Shattered vulnerability?
+How is Subversion affected by SHA-1 hash collisions?
   ¶
 
 
-Subversion's use of SHA-1 in how it processes content is subject to hashing
-collisions as identified by https://shattered.io/";>Google. One of
-Subversions's key assumptions in processing content is that SHA-1 is unique 
for 
-all objects.
-Subversion has two main areas of vulnerability.
-
+Publication of the first known SHA-1 collision by https://shattered.io/";>
+Google and CWI unveiled a couple of related issues in the Subversion's use
+of SHA-1. The Subversion's core does not rely on SHA-1 for content indexing,
+but it was being used for such purposes in the following supplementary 
features:
+
 
-The FS backend (repository) uses SHA-1.
-The Working Copy/RA layers use SHA-1.
+repository data deduplication feature, and
+content deduplication feature in the working copy.
 
-
-To prevent different content with identical SHA-1 from being stored, upgrade
-to 1.9.6 or 1.8.18 which, by default, prevent storing data with such 
collisions.
-See our SHA1 advisory for details.
+Speaking of the repository data deduplication feature, this can result in
+inability to access files with colliding SHA-1 values or cause data loss for
+such files. To prevent different content with identical SHA-1 from being stored
+in a repository, upgrade to 1.9.6 or 1.8.18 which, by default, prevent storing
+data with such collisions. See our SHA-1
+advisory for details.
 
 
-Until these new releases are available, Unix-based servers can use the 
pre-commit hook found
+Until the upgrade to these new releases is available, Unix-based servers
+can use the pre-commit hook found
 https://svn.apache.org/repos/asf/subversion/trunk/tools/hook-scripts/reject-known-sha1-collisions.sh";>
 here.
 As an aside, we welcome Windows developers to submit a pre-commit 
@@ -777,19 +779,18 @@ script for the Windows platform. More information
 here.
 
 
-The working copy/RA layer uses SHA-1 for de-duplication of content stored in 
-the working copy, and for performance reasons clients using the HTTP protocol 
-will avoid fetching content with a SHA-1 checksum which has been fetched 
-previously. There is no known workaround for this vector except to prevent 
-storage of the colliding objects in the first place, via upgrade to 1.9.6 or 
-installation of the aforementioned pre-commit script.
+The working copy uses SHA-1 for dedupl

Re: svn commit: r1801108 - in /subversion/trunk/subversion/bindings/javahl/native: Array.cpp CreateJ.cpp Iterator.cpp OperationContext.cpp RemoteSession.cpp RevisionRangeList.cpp

[Philip Martin]

Branko Čibej  writes:

> On 06.07.2017 23:11, Philip Martin wrote:
>>// Create a local frame for our references
>> -  env->PushLocalFrame(LOCAL_FRAME_SIZE);
>> +  env->PushLocalFrame(LOCAL_FRAME_SIZE + 100);
>
> Mmph ... it kinda hurts to see how limited the JNI API is in this
> respect. We could either increase the LOCAL_FRAME_SIZE variable, or
> create a new frame within each iteration of the loop.

We might be able to fix the unparseExternals case by passing an env into
the functor, then we would be creating fewer refs.

putErrorsInTrace is harder because it is recursive and stores a ref at
each level. Would pushing/poping frames invalidate those refs?  Perhaps
converting the recursion to iteration might be the best solution.

-- 
Philip

[ANNOUNCE] Apache Subversion 1.8.18 released

[Stefan Sperling]

I'm happy to announce the release of Apache Subversion 1.8.18.
Please choose the mirror closest to you by visiting:

http://subversion.apache.org/download.cgi#supported-releases

This release fixes a bug where a repository would fail to reject two files
having identical SHA-1 checksums [1].  Further information is available
in the 1.8.x series release notes [2] and in a new FAQ entry [3].

[1] http://shattered.io/
[2] https://subversion.apache.org/docs/release-notes/1.8#shattered-sha1
[3] https://subversion.apache.org/faq#shattered-sha1

The SHA1 checksums are:

9a3745245c2eb9fefaca50a5788604cfae89a636 subversion-1.8.18.zip
22ab7f366fb1d373247c56696b2b754c5fd06368 subversion-1.8.18.tar.gz
026f42b53b8595b9b87c4d24441f7a121c27554c subversion-1.8.18.tar.bz2

SHA-512 checksums are available at:

https://www.apache.org/dist/subversion/subversion-1.8.18.tar.bz2.sha512
https://www.apache.org/dist/subversion/subversion-1.8.18.tar.gz.sha512
https://www.apache.org/dist/subversion/subversion-1.8.18.zip.sha512

PGP Signatures are available at:

http://www.apache.org/dist/subversion/subversion-1.8.18.tar.bz2.asc
http://www.apache.org/dist/subversion/subversion-1.8.18.tar.gz.asc
http://www.apache.org/dist/subversion/subversion-1.8.18.zip.asc

For this release, the following people have provided PGP signatures:

   Julian Foad [4096R/1FB064B84EECC493] with fingerprint:
6011 63CF 9D49 9FD7 18CF  582D 1FB0 64B8 4EEC C493
   Johan Corveleyn [4096R/B59CE6D6010C8AAD] with fingerprint:
8AA2 C10E EAAD 44F9 6972  7AEA B59C E6D6 010C 8AAD
   Bert Huijben [4096R/C4A6C625CCC8E1DF] with fingerprint:
3D1D C66D 6D2E 0B90 3952  8138 C4A6 C625 CCC8 E1DF
   Stefan Sperling [2048R/4F7DBAA99A59B973] with fingerprint:
8BC4 DAE0 C5A4 D65F 4044  0107 4F7D BAA9 9A59 B973
   Stefan Fuhrmann [4096R/99EC741B57921ACC] with fingerprint:
056F 8016 D9B8 7B1B DE41  7467 99EC 741B 5792 1ACC
   Stefan Hett (CODE SIGNING KEY) [4096R/376A3CFD110B1C95] with fingerprint:
7B8C A7F6 451A D89C 8ADC  077B 376A 3CFD 110B 1C95
   Philip Martin [2048R/76D788E1ED1A599C] with fingerprint:
A844 790F B574 3606 EE95  9207 76D7 88E1 ED1A 599C

Release notes for the 1.8.x release series may be found at:

http://subversion.apache.org/docs/release-notes/1.8.html

You can find the list of changes between 1.8.18 and earlier versions at:

http://svn.apache.org/repos/asf/subversion/tags/1.8.18/CHANGES

Questions, comments, and bug reports to us...@subversion.apache.org.

Thanks,
- The Subversion Team

Re: Expected speed of commit over HTTP?

[Branko Čibej]

On 07.07.2017 12:10, Paul Hammant wrote:
> ~/.subversion/servers contains a line that's tunable, right:
>
> # http-compression = no
>
> And per comment, tack on:
>
> --config-option=servers:global:http-compression=no .
>
> I'll report back the results.
>
>
> How is Apache and its modules organized?  As separate processes - with
> TCP/IP between each?
> If yes, is that interconnect loopback Unix domain sockets?

No, Apache modules are loadable shared libs. Whether these are all in
the same process or not depends on which MPM module is used. Data
transfer between modules is in-memory, AFAIK.

-- Brane

Re: svn commit: r1801108 - in /subversion/trunk/subversion/bindings/javahl/native: Array.cpp CreateJ.cpp Iterator.cpp OperationContext.cpp RemoteSession.cpp RevisionRangeList.cpp

[Branko Čibej]

On 06.07.2017 23:11, Philip Martin wrote:
> Philip Martin  writes:
>
>> I've upgraded my JDK and it produced all these warnings.

Adding the exception checks is correct. Rewriting everything to use the
jniwrapper abstraction would be even better, but that's a huge-ish heap
of work.

> There is a second class of warnings of the form:
>
>   WARNING: JNI local refs: 57, exceeds capacity: 56
>
> These are generated in two places: JNIUtil::putErrorsInTrace() and
> Java_org_apache_subversion_javahl_util_PropLib_unparseExternals().
>
> I'm not sure how best to fix these so I have a local hack in
> JNIUtil::wrappedCreateClientException to remove most of them:
>
>// Create a local frame for our references
> -  env->PushLocalFrame(LOCAL_FRAME_SIZE);
> +  env->PushLocalFrame(LOCAL_FRAME_SIZE + 100);

Mmph ... it kinda hurts to see how limited the JNI API is in this
respect. We could either increase the LOCAL_FRAME_SIZE variable, or
create a new frame within each iteration of the loop.

-- Brane

Re: Expected speed of commit over HTTP?

[Paul Hammant]

~/.subversion/servers contains a line that's tunable, right:

# http-compression = no

And per comment, tack on:

--config-option=servers:global:http-compression=no .

I'll report back the results.


How is Apache and its modules organized?  As separate processes - with
TCP/IP between each?
If yes, is that interconnect loopback Unix domain sockets?

Or is it the same process, with buffers via malloc, and a plugin
architecture mean a single stack between Apache and the chain of modules
into Svn?

Or is there a temp-file situation happening - modules put/pickup payloads
(or maybe large payloads only) from the root FS as part of the delegation?

Could there be swap file issues here, on the 4GB machine?  Some condition
that manifests itself with Apache and 500MB payloads in play, but not 'cp'
of the same resource?

Oh, thanks for checking with your people, Jack.

- Paul

Re: svn commit: r1801108 - in /subversion/trunk/subversion/bindings/javahl/native: Array.cpp CreateJ.cpp Iterator.cpp OperationContext.cpp RemoteSession.cpp RevisionRangeList.cpp

[Philip Martin]

"Bert Huijben"  writes:

> Should we backport these to 1.9.x?

I don't know, possibly.  See my other mail to dev for the questions I
have about these checks.

-- 
Philip

RE: svn commit: r1801108 - in /subversion/trunk/subversion/bindings/javahl/native: Array.cpp CreateJ.cpp Iterator.cpp OperationContext.cpp RemoteSession.cpp RevisionRangeList.cpp

[Bert Huijben]

> -Original Message-
> From: phi...@apache.org [mailto:phi...@apache.org]
> Sent: donderdag 6 juli 2017 22:56
> To: comm...@subversion.apache.org
> Subject: svn commit: r1801108 - in
> /subversion/trunk/subversion/bindings/javahl/native: Array.cpp CreateJ.cpp
> Iterator.cpp OperationContext.cpp RemoteSession.cpp
> RevisionRangeList.cpp
> 
> Author: philip
> Date: Thu Jul  6 20:56:14 2017
> New Revision: 1801108
> 
> URL: http://svn.apache.org/viewvc?rev=1801108&view=rev
> Log:
> Add exception checks to some of the JavaHL native code to avoid JVM
> warnings about JNI problems of the form:
> 
>   WARNING in native method: JNI call made without checking exceptions
> when required to
> 
> * subversion/bindings/javahl/native/Array.cpp
>   (Array::init): Add exception check.
> 
> * subversion/bindings/javahl/native/CreateJ.cpp
>   (CreateJ::Mergeinfo): Add exception check.
> 
> * subversion/bindings/javahl/native/Iterator.cpp
>   (init_iterator, Iterator::next): Add exception check.
> 
> * subversion/bindings/javahl/native/OperationContext.cpp
>   (OperationContext::closeTunnel): Add exception check.
> 
> * subversion/bindings/javahl/native/RemoteSession.cpp
>   (build_string_array, long_iterable_to_revnum_array,
>location_hash_to_map): Add exception check.
> 
> * subversion/bindings/javahl/native/RevisionRangeList.cpp
>   (RevisionRangeList::RevisionRangeList): Add exception check.

Should we backport these to 1.9.x?

Bert 

Re: Expected speed of commit over HTTP?

[Johan Corveleyn]

On Fri, Jul 7, 2017 at 8:54 AM, Jacek Materna  wrote:
> Paul,
>
> Got back from Ops. We don't have anything "special" setup other than what
> already been mentioned. Overall the defining metrics for commit performance
> are: ssh vs https, rtt on the network path and "randomness" of the blobs
> going up.
>
> No magic params on our end. We have it tuned the max via the metrics above.
>
> Most of the real world use cases require the default settings so curl, etc
> isn't seen a lot in the wild.
>
> Overall Dav and http is a poor transport in general for SVN.
>
> Hope it helps.
>
> --
> Jacek Materna

It would be interesting to know whether the remaining 1:10 factor is
caused by the transport or by the back-end (with its server-side
deltification and FSFS transaction features). I don't know if it would
be possible for Paul to test the same setup with svn+ssh.

OTOH, that would probably not be a very good comparison, because with
svn+ssh you can't use SVNAutoVersioning and curl, you'd have to use a
normal svn client, so IIUC it's not possible to eliminate the
client-side deltification. Unless Philip or someone else knows another
trick for that :).

-- 
Johan