Re: 1.10 release notes: protocol changes, LZ4, etc.

[Evgeny Kotkov]

Julian Foad  writes:

>> - On the server-side, SVNCompressionLevel 0 can be used to disable
>> compression altogether.  The special value of SVNCompressionLevel 1 forces
>> the use of LZ4 compression for clients that support it.  All other values
>> result in negotiating the use of zlib compression with the respective
>> compression level, unless the compression is disabled on the client.
>
> There's a small disagreement with the table there: all other values result
> in negotiating either zlib or LZ4.

Indeed.  Perhaps, we could rephrase it like this?

  [...] All other values result in preferring zlib compression with the
  respective compression level.  Note that the negotiated algorithm is still
  a subject to the client's configuration.  For example, even if the server
  is configured to prefer zlib compression over LZ4, a client may still
  negotiate the use of LZ4 compression when its http-compression option
  is set to auto.

>> - On the client-side, setting http-compression to either yes or no will
>> disable or enable compression that is then negotiated based on the
>> server's
>> configuration.  The default value of auto will result in preferring LZ4
>> compression over low latency networks and zlib compression otherwise.
>
> Can we link to the reference documentation for the client and server
> options, so the reader (me) can cross-check the definition and look for
> related settings?

At the time being, the documentation for these options is quite scarce and
doesn't dive into the details of which compression algorithm is negotiated
behind the scenes.  The reasoning behind this was to avoid overwhelming
the users with technical and implementation details and to have a ground
for future extension by not promising too much.  This makes me think that
linking to these docs from the release notes probably wouldn't be too useful:

  
https://svn.apache.org/viewvc/subversion/trunk/subversion/mod_dav_svn/mod_dav_svn.c?view=markup=1820778#l1392
  
https://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_subr/config_file.c?view=markup=1820778#l1151

For the purpose of cross-checking the description, the following locations in
mod_dav_svn and ra_serf may prove useful:

  mod_dav_svn/repos.c:negotiate_encoding_prefs()
  libsvn_ra_serf/util.c:svn_ra_serf__setup_svndiff_accept_encoding()
  libsvn_ra_serf/commit.c:negotiate_put_encoding()


Thanks,
Evgeny Kotkov

Re: 1.10 release notes: protocol changes, LZ4, etc.

[Julian Foad]

Evgeny Kotkov wrote:

Maybe we could start with adding a subsection about that describes using
LZ4 for http:// and svn:// to the users.  If needed, we could then include
the technical details about the negotiation into the "Client-Server Protocol
Changes" and cross-link these two sections.


Agreed. (And I think we should include protocol details -- at least a 
summary and references to where to find the details.)



For the http:// part, we could probably do something along the lines of
the attached patch.  Here is also the same content as plain text for
convenience:

[[[
LZ4 compression over the wire in http:// and svn:// connections

Deltas transferred between Subversion 1.10 clients and servers may be
compressed with LZ4.  The actual choice of the compression algorithm depends
on the used protocol, environment and its configuration — see below.

For http:// protocol, use of LZ4 compression depends on the values of
the server-side SVNCompressionLevel directive, client-side http-compression
configuration option and on the network capabilities.  LZ4 compression
generally offers much faster compression and decompression speeds, but
slightly worse compression ratio than zlib.  By default, it is only
preferred over low latency networks where the overhead associated with
transferring the additional amount of data is assumed to be negligible.

- On the server-side, SVNCompressionLevel 0 can be used to disable
compression altogether.  The special value of SVNCompressionLevel 1 forces
the use of LZ4 compression for clients that support it.  All other values
result in negotiating the use of zlib compression with the respective
compression level, unless the compression is disabled on the client.


There's a small disagreement with the table there: all other values 
result in negotiating either zlib or LZ4.



- On the client-side, setting http-compression to either yes or no will
disable or enable compression that is then negotiated based on the server's
configuration.  The default value of auto will result in preferring LZ4
compression over low latency networks and zlib compression otherwise.


Can we link to the reference documentation for the client and server 
options, so the reader (me) can cross-check the definition and look for 
related settings?



Below is the table explaining the used compression algorithm in each
combination of the client- and server-side configuration options:


That's helpful and clear.

Thanks. +1 to commit after resolving the "all other values" point above.

- Julian

Re: 1.10 release notes: protocol changes, LZ4, etc.

[Evgeny Kotkov]

Julian Foad  writes:

> I added a skeleton "protocol changes" section to the release notes:
>
>   http://subversion.a.o.local/docs/release-notes/1.10.html#protocol-changes
>
> Here is a starting point. Can anyone help fill in the details?

[...]

> Thoughts? Help?

Maybe we could start with adding a subsection about that describes using
LZ4 for http:// and svn:// to the users.  If needed, we could then include
the technical details about the negotiation into the "Client-Server Protocol
Changes" and cross-link these two sections.

For the http:// part, we could probably do something along the lines of
the attached patch.  Here is also the same content as plain text for
convenience:

[[[
LZ4 compression over the wire in http:// and svn:// connections

Deltas transferred between Subversion 1.10 clients and servers may be
compressed with LZ4.  The actual choice of the compression algorithm depends
on the used protocol, environment and its configuration — see below.

For http:// protocol, use of LZ4 compression depends on the values of
the server-side SVNCompressionLevel directive, client-side http-compression
configuration option and on the network capabilities.  LZ4 compression
generally offers much faster compression and decompression speeds, but
slightly worse compression ratio than zlib.  By default, it is only
preferred over low latency networks where the overhead associated with
transferring the additional amount of data is assumed to be negligible.

- On the server-side, SVNCompressionLevel 0 can be used to disable
compression altogether.  The special value of SVNCompressionLevel 1 forces
the use of LZ4 compression for clients that support it.  All other values
result in negotiating the use of zlib compression with the respective
compression level, unless the compression is disabled on the client.

- On the client-side, setting http-compression to either yes or no will
disable or enable compression that is then negotiated based on the server's
configuration.  The default value of auto will result in preferring LZ4
compression over low latency networks and zlib compression otherwise.

Below is the table explaining the used compression algorithm in each
combination of the client- and server-side configuration options:

[table]


]]]


Thanks,
Evgeny Kotkov
Index: docs/release-notes/1.10.html
===
--- docs/release-notes/1.10.html(revision 1825639)
+++ docs/release-notes/1.10.html(working copy)
@@ -136,7 +136,7 @@ and what impact these changes may have.
 Use SVN 1.8 and above clients only for best results.
   
 
-  LZ4 compression over the wire in http:// connections
+  LZ4 compression over the wire in http:// 
and svn:// connections
 
 1.10
 1.10
@@ -526,7 +526,9 @@ containing large files.
 
 
 LZ4 compression is now used by default for the on-disk data in repositories
-with filesystem format 8 (see below).
+with filesystem format 8.  Also, Subversion 1.10 adds support for automatic
+negotiation and use of LZ4 compression over the wire for http:// and svn://
+connections when it is supported by both endpoints.
 
 Note: this does not apply to the pre-release
 https://lists.apache.org/thread.html/dd78432b301f3f95567930c242c46b308c8a017fd754c0f388245915@%3Cannounce.subversion.apache.org%3E;>Subversion
 1.10.0-alpha3
@@ -559,7 +561,7 @@ format number of a repository.)
 
 
 Configuring the repository to use LZ4 compression
-  
 
 
@@ -583,6 +585,116 @@ cycle into a new format 8 repository created with
 
   
 
+
+LZ4 compression over the wire in http:// and svn:// connections
+  
+
+
+Deltas transferred between Subversion 1.10 clients and servers may be
+compressed with LZ4. The actual choice of the compression algorithm depends
+on the used protocol, environment and its configuration  see below.
+
+For http:// protocol, use of LZ4 compression depends on the values
+of the server-side SVNCompressionLevel directive, client-side
+http-compression configuration option and on the network
+capabilities.  LZ4 compression generally offers much faster compression
+and decompression speeds, but slightly worse compression ratio than zlib.
+By default, it is only preferred for low latency networks where the
+overhead associated with transferring the additional amount of data is
+assumed to be negligible.
+
+
+  
+On the server-side, SVNCompressionLevel 0
+can be used to disable compression altogether.  The special value of
+SVNCompressionLevel 1 forces the use of LZ4 compression for
+clients that support it.  All other values result in negotiating the
+use of zlib compression with the respective compression level, unless
+the compression is disabled on the client.
+  
+  
+On the client-side, setting http-compression to
+either yes or no will disable or enable compression
+that is then negotiated based on the server's configuration.
+The default value of