Re: Possibly dead link in community guide?

[Branko Čibej]

On 20.10.2017 05:44, Troy Curtis Jr wrote:
> Currently the link
> http://subversion.tigris.org/servlets/ReadMsg?list=dev=132746
> referenced in
> https://subversion.apache.org/docs/community-guide/roles.html#committers
> results in a Tomcat error.  It may just be a temporary downtime issue,
> but I'm not sure so I though I'd point it out.

The link works for me and ... I can't imagine what Tomcat would have to
do with it, since our site isn't served through Tomcat.

-- Brane

Possibly dead link in community guide?

[Troy Curtis Jr]

Currently the link
http://subversion.tigris.org/servlets/ReadMsg?list=dev=132746
referenced in
https://subversion.apache.org/docs/community-guide/roles.html#committers
results in a Tomcat error.  It may just be a temporary downtime issue, but
I'm not sure so I though I'd point it out.

Troy

Re: Workflow for editing the subversion website

[Johan Corveleyn]

On Thu, Oct 19, 2017 at 10:04 PM, Johan Corveleyn  wrote:
> On Thu, Oct 5, 2017 at 11:42 PM, Branko Čibej  wrote:
>> On 05.10.2017 22:36, Johan Corveleyn wrote:
>>> On Thu, Oct 5, 2017 at 12:30 PM, Daniel Shahaf  
>>> wrote:
 Devil's advocate hat on, and in light of Brane's sibling reply, let me
 describe how an svnmucc workflow might work.
>>> Thanks, but I prefer the merge workflow. It seems more natural to me,
>>> and I think it's more likely to be used by other svn users out there,
>>> in case they have such a workflow. So it seems like the more
>>> interesting dog food to me :-).
>>>
>>> I'm not very good at writing down an accurate procedure, but I still
>>> think it should be something like I wrote in my first mail in this
>>> thread:
>>>
 1) Commit to staging. Other devs get the commit mail via the
 notifications@ list.

 2) Wait for others to review (the commit mail is the notification that
 something needs to be reviewed). In case of large or sensitive
 changes, preferably send a mail to dev@ to draw extra attention.

 3) If any other committer says +1, go ahead and "promote" (merge) to
 the live site.

 4) If no response after 1 week? 3 days? ...? go ahead and promote to
 live site (lazy consensus).
>>> As Brane suggested, let's do everything in this direction (test on
>>> staging first, then merge to publish), except for security
>>> announcements.
>>>
>>> And as Daniel suggested, let's serve the staging site via
>>> https://subversion-staging.apache.org/ (I'd say we ask infra to set
>>> this up for us).
>>
>> Sounds like a plan.
>>
>> -- Brane
>
> Sorry, I dropped this on the floor trying to catch some other things.
>
> I'll try to pick up the pieces now :-) ...
>
> svn cp $URL/site/publish $URL/site/staging
> open infra ticket for serving /site/staging on subversion-staging.a.o
> put a short description of our desired workflow in /site/README

Done. The infra ticket is here:
https://issues.apache.org/jira/browse/INFRA-15323

The description of the workflow in site/README is more or less
copy-pasted from this mail thread. Can probably be improved ...
(anyone: feel free). While writing it down, I realized that this
"promote from staging to publish with complete merges" isn't
parallellizable (if two committers want to make two distinct changes,
and not merge all of it in one go to publish). Oh well, it's a start.

-- 
Johan

Subsyncit - file sync that uses Subversion as a backend.

[Paul Hammant]

http://subsyncit.com/

If you're prepared to make a few additions to httpd.conf you get running
with this Python3 technology to do file-sync using Subversion as the
backing store.  It doesn't require any client side Subversion install, but
does Python and some pips. At least up until I make an installer.

And I've yet to make a Window-tray or Mac-menu icon thingamy like every
other background app does. Meaning this is still beta quality, really - but
it does its business and doesn't eat all available CPU.

I'm hoping to integrate with various other applications/services. Trello,
Slack, etc. That would be for workflow-ish functionality.  Fred assigns
something to Wilma in Trello by changing the 'assigned to' aspect of a
card, and a few seconds later it appears on her C: drive. A spreadsheet say.

I've tested it with both Assembla's and RhodeCode's Subversion platforms.
I've also tested it with a bunch of my own Subversion deployments. One - a
$10 Pi Zero W with a $70 200GB SdCard in it. Another a EC2 deployment of
Subversion/Apache2.

Questions?

- Paul

Re: Workflow for editing the subversion website

[Johan Corveleyn]

On Thu, Oct 5, 2017 at 11:42 PM, Branko Čibej  wrote:
> On 05.10.2017 22:36, Johan Corveleyn wrote:
>> On Thu, Oct 5, 2017 at 12:30 PM, Daniel Shahaf  
>> wrote:
>>> Devil's advocate hat on, and in light of Brane's sibling reply, let me
>>> describe how an svnmucc workflow might work.
>> Thanks, but I prefer the merge workflow. It seems more natural to me,
>> and I think it's more likely to be used by other svn users out there,
>> in case they have such a workflow. So it seems like the more
>> interesting dog food to me :-).
>>
>> I'm not very good at writing down an accurate procedure, but I still
>> think it should be something like I wrote in my first mail in this
>> thread:
>>
>>> 1) Commit to staging. Other devs get the commit mail via the
>>> notifications@ list.
>>>
>>> 2) Wait for others to review (the commit mail is the notification that
>>> something needs to be reviewed). In case of large or sensitive
>>> changes, preferably send a mail to dev@ to draw extra attention.
>>>
>>> 3) If any other committer says +1, go ahead and "promote" (merge) to
>>> the live site.
>>>
>>> 4) If no response after 1 week? 3 days? ...? go ahead and promote to
>>> live site (lazy consensus).
>> As Brane suggested, let's do everything in this direction (test on
>> staging first, then merge to publish), except for security
>> announcements.
>>
>> And as Daniel suggested, let's serve the staging site via
>> https://subversion-staging.apache.org/ (I'd say we ask infra to set
>> this up for us).
>
> Sounds like a plan.
>
> -- Brane

Sorry, I dropped this on the floor trying to catch some other things.

I'll try to pick up the pieces now :-) ...

svn cp $URL/site/publish $URL/site/staging
open infra ticket for serving /site/staging on subversion-staging.a.o
put a short description of our desired workflow in /site/README

-- 
Johan

Re: Python 3 Bindings Query

[Daniel Shahaf]

Troy Curtis Jr wrote on Thu, Oct 19, 2017 at 04:05:57 +:
> The places where we'd expect this to actually need to happen are less on
> the Python API side (since it should be str in and out), but more on the
> binding's usage of the underlying Subversion API.  All the 'char *' will be
> considered 'bytes' by py3, and need to be decoded to str (unicode), before
> it is given to the Python API so that the user gets the expected str
> objects.

Ah!  So you're thinking of data going from libsvn_* to user Python code,
not the other way around.  Now it makes sense.  (I was thinking of user
data passed into to the bindings.)

In this case, I think the rule is: if it's NUL terminated, then it's in
UTF-8 (except for some isolated exceptions such as svn_cmdline_*); if
it's a counted-length string, then it's bytes.  For example, a property
hash is an apr_hash_t* mapping const char* to const svn_string_t*,
corresponding to the data model where property names are UTF-8 strings
and property values are opaque binary blobs.

This is supposed to be explicitly documented, by the way.  For example,
svn_path.h states
.
 * All incoming and outgoing paths are non-NULL and in UTF-8, unless
 * otherwise documented.
.
but, apparently, the newer svn_dirent_uri.h doesn't have such a statement.

> So in general I think going with utf8 is the way to go.  However, there a
> few places I plan on looking carefully at:
> 1. Anywhere raw data is "streamed in": I'm not sure if this exists in the
> API or not, I am assuming it does somewhere to manually feed data into the
> library to be used as content of a commit.

Functions that add data, at various layers, are:

- svn_client_add5()
- svn_delta_editor_t
- svn_repos_load_fs6()
- svn_fs_make_file()

Paths in the repository are always in UTF-8.  File contents are treated
as opaque binary blobs and are generally presented as streams (either
svn_stream_t or an svndiff/txdelta stream).

> 2. Filesystem paths: The general case is there are separate functions for
> dealing with "the filesystem encoding" whichever it happens to be [1].  So
> perhaps one concrete question is are paths provided to the API and
> elsewhere within Subversion assumed to be UTF8?

Subversion's functions generally take UTF-8, but there are exceptiosn,
such as *_canonicalize() and *_internal_style().  They're generally
implemented in terms of apr_* functions which expect a different
encoding (see e.g. svn_io_check_file()).

Cheers,

Daniel

Re: Slow ISVNClient.getChangelists on Linux/NFS share

[Thomas Singer]

On 2017-10-19 10:44, Johan Corveleyn wrote:

I have no idea what ISVNClient.getChangelists corresponds to compared
with the native 'svn' commandline. But one thing that might be
relevant: in the client-side configuration file 'config' there are two
properties that can have a big influence here:

(see %APPDATA%/Subversion/config on Windows, or ~/.subversion/config on Unix)
[[[
### Section for configuring working copies.
[working-copy]
### Set to a list of the names of specific clients that should use
### exclusive SQLite locking of working copies.  This increases the
### performance of the client but prevents concurrent access by
### other clients.  Third-party clients may also support this
### option.
### Possible values:
###   svn(the command line client)
# exclusive-locking-clients =
### Set to true to enable exclusive SQLite locking of working
### copies by all clients using the 1.8 APIs.  Enabling this may
### cause some clients to fail to work properly. This does not have
### to be set for exclusive-locking-clients to work.
# exclusive-locking = false
]]]

So with the above options you can make all clients (that use that
config file) respect "exclusive-locking" (with the exclusive-locking
flag), or just a subset of specific client applications (with the
exclusive-locking-clients property).

Perhaps your commandline 'svn' invocation uses the standard
~/.subversion/config file, which has the exclusive-locking flag set,
and your SmartSVN application might use another one (it can choose
another "config dir").


Thank you for this suggestion. I've tried with

exclusive-locking = true

but it makes no difference. I don't know what to enter for

exclusive-locking-clients =

How does SVN detect the client name when using JavaHL?

--
Best regards,
Thomas Singer
=
syntevo GmbH
http://www.syntevo.com
http://www.syntevo.com/blog

Re: Slow ISVNClient.getChangelists on Linux/NFS share

[Johan Corveleyn]

On Thu, Oct 19, 2017 at 8:30 AM, Thomas Singer
 wrote:
> On 18.10.2017 19:56, Branko Čibej wrote:
>>
>> On 18.10.2017 13:31, Thomas Singer wrote:
>>>
>>> Hi,
>>>
>>> When performing following steps on my old Linux test machine (with
>>> slow hard disk):
>>>
>>> - have a SVN working copy at /home/user/test
>>> - sudo apt install nfs-kernel-server
>>> - add following line to /etc/exports:
>>>/home/user/test *(rw,sync,no_root_squash)
>>> - start the NFS server:
>>>sudo systemctl start nfs-kernel-server.service
>>> - mount the NFS share:
>>>sudo mount localhost:/home/user/test /home/user/test.nfs
>>>
>>> and then open /home/user/test.nfs in SmartSVN 9.2 (using SVN 1.9
>>> JavaHL binaries), adding/removing a file is very slow. It boils down
>>> to the call ISVNClient.getChangelists which takes ~8s on the NFS share
>>> (/home/user/test.nfs). First, I thought, it would be caused by the
>>> native-Java overhead calling the call-back ~11,000 times for my
>>> working copy, but when using the working copy directly
>>> (/home/user/test), the method just takes <1s though the ~11,000 times
>>> call-back invocations are still there.
>>>
>>> My working copy has no local modifications, no untracked or ignored
>>> files, no changelists.
>>>
>>> Is it expected that this method (ISVNClient.getChangelists) is so slow
>>> on a NFS share even if there are no changelists?
>>
>>
>> I don't know if it's "expected" but I bet that NFS is killing SQLite
>> performance.
>>
>>
>> https://www.mail-archive.com/sqlite-users@mailinglists.sqlite.org/msg88989.html
>>
>> I'm not sure about the reason but the most likely answer, apart from
>> slow data rate and latency when compared to a local filesystem (which,
>> in your case on loopback, should not be an issue), is that the OS can't
>> really use a cache for files on NFS since it has no way to know whether
>> or not it's valid. With a lot of random-access reads and writes, that
>> can be a HUGE slowdown, as you found.
>>
>>
>> Also this:
>> https://sqlite.org/faq.html#q5
>>
>> In other words, Subversion working copies on NFS are, and have always
>> been, a bad idea; not only because of SQLite but also because
>> Subversion's code itself relies on atomic renames, which NFS does not
>> provide.
>>
>> -- Brane
>
>
> What SVN command (on command line) I should test to get a similar result as
> from ISVNClient.getChangelists? I've tried "git status" and it just needs
> <1s on the NFS working copy.

I assume you meant "svn status", not "git status".

I have no idea what ISVNClient.getChangelists corresponds to compared
with the native 'svn' commandline. But one thing that might be
relevant: in the client-side configuration file 'config' there are two
properties that can have a big influence here:

(see %APPDATA%/Subversion/config on Windows, or ~/.subversion/config on Unix)
[[[
### Section for configuring working copies.
[working-copy]
### Set to a list of the names of specific clients that should use
### exclusive SQLite locking of working copies.  This increases the
### performance of the client but prevents concurrent access by
### other clients.  Third-party clients may also support this
### option.
### Possible values:
###   svn(the command line client)
# exclusive-locking-clients =
### Set to true to enable exclusive SQLite locking of working
### copies by all clients using the 1.8 APIs.  Enabling this may
### cause some clients to fail to work properly. This does not have
### to be set for exclusive-locking-clients to work.
# exclusive-locking = false
]]]

So with the above options you can make all clients (that use that
config file) respect "exclusive-locking" (with the exclusive-locking
flag), or just a subset of specific client applications (with the
exclusive-locking-clients property).

Perhaps your commandline 'svn' invocation uses the standard
~/.subversion/config file, which has the exclusive-locking flag set,
and your SmartSVN application might use another one (it can choose
another "config dir").

-- 
Johan

Re: Slow ISVNClient.getChangelists on Linux/NFS share

[Branko Čibej]

On 19.10.2017 10:07, Thomas Singer wrote:
> On 2017-10-19 9:05, Branko Čibej wrote:
>> On 19.10.2017 08:30, Thomas Singer wrote:
>>> On 18.10.2017 19:56, Branko Čibej wrote:
 On 18.10.2017 13:31, Thomas Singer wrote:
> Hi,
>
> When performing following steps on my old Linux test machine (with
> slow hard disk):
>
> - have a SVN working copy at /home/user/test
> - sudo apt install nfs-kernel-server
> - add following line to /etc/exports:
>     /home/user/test *(rw,sync,no_root_squash)
> - start the NFS server:
>     sudo systemctl start nfs-kernel-server.service
> - mount the NFS share:
>     sudo mount localhost:/home/user/test /home/user/test.nfs
>
> and then open /home/user/test.nfs in SmartSVN 9.2 (using SVN 1.9
> JavaHL binaries), adding/removing a file is very slow. It boils down
> to the call ISVNClient.getChangelists which takes ~8s on the NFS
> share
> (/home/user/test.nfs). First, I thought, it would be caused by the
> native-Java overhead calling the call-back ~11,000 times for my
> working copy, but when using the working copy directly
> (/home/user/test), the method just takes <1s though the ~11,000 times
> call-back invocations are still there.
>
> My working copy has no local modifications, no untracked or ignored
> files, no changelists.
>
> Is it expected that this method (ISVNClient.getChangelists) is so
> slow
> on a NFS share even if there are no changelists?

 I don't know if it's "expected" but I bet that NFS is killing SQLite
 performance.

 https://www.mail-archive.com/sqlite-users@mailinglists.sqlite.org/msg88989.html



 I'm not sure about the reason but the most likely answer, apart from
 slow data rate and latency when compared to a local filesystem (which,
 in your case on loopback, should not be an issue), is that the OS
 can't
 really use a cache for files on NFS since it has no way to know
 whether
 or not it's valid. With a lot of random-access reads and writes, that
 can be a HUGE slowdown, as you found.


 Also this:
 https://sqlite.org/faq.html#q5

 In other words, Subversion working copies on NFS are, and have always
 been, a bad idea; not only because of SQLite but also because
 Subversion's code itself relies on atomic renames, which NFS does not
 provide.

 -- Brane
>>>
>>> What SVN command (on command line) I should test to get a similar
>>> result as from ISVNClient.getChangelists?
>>
>> If "adding/removing a file" is any indication, then "svn add" or "svn
>> remove" should be comparable.
>
> Sorry, this was misleading. Adding/removing a file cause a refresh of
> which ISVNClient.getChangelists takes the longest time. With SmartSVN
> Foundation ISVNClient.getChangelists is not invoked and hence much
> faster when refreshing after a command, e.g. add or remove.
>
> Which is the SVN command line equivalent of ISVNClient.getChangelists?

The only one I can find that actually uses the
svn_client_get_changelists() function is 'svn update --changelist', but
of course it also does an update, so the comparison is not entirely valid.

-- Brane

Re: Slow ISVNClient.getChangelists on Linux/NFS share

[Branko Čibej]

On 19.10.2017 08:30, Thomas Singer wrote:
> On 18.10.2017 19:56, Branko Čibej wrote:
>> On 18.10.2017 13:31, Thomas Singer wrote:
>>> Hi,
>>>
>>> When performing following steps on my old Linux test machine (with
>>> slow hard disk):
>>>
>>> - have a SVN working copy at /home/user/test
>>> - sudo apt install nfs-kernel-server
>>> - add following line to /etc/exports:
>>>    /home/user/test *(rw,sync,no_root_squash)
>>> - start the NFS server:
>>>    sudo systemctl start nfs-kernel-server.service
>>> - mount the NFS share:
>>>    sudo mount localhost:/home/user/test /home/user/test.nfs
>>>
>>> and then open /home/user/test.nfs in SmartSVN 9.2 (using SVN 1.9
>>> JavaHL binaries), adding/removing a file is very slow. It boils down
>>> to the call ISVNClient.getChangelists which takes ~8s on the NFS share
>>> (/home/user/test.nfs). First, I thought, it would be caused by the
>>> native-Java overhead calling the call-back ~11,000 times for my
>>> working copy, but when using the working copy directly
>>> (/home/user/test), the method just takes <1s though the ~11,000 times
>>> call-back invocations are still there.
>>>
>>> My working copy has no local modifications, no untracked or ignored
>>> files, no changelists.
>>>
>>> Is it expected that this method (ISVNClient.getChangelists) is so slow
>>> on a NFS share even if there are no changelists?
>>
>> I don't know if it's "expected" but I bet that NFS is killing SQLite
>> performance.
>>
>> https://www.mail-archive.com/sqlite-users@mailinglists.sqlite.org/msg88989.html
>>
>>
>> I'm not sure about the reason but the most likely answer, apart from
>> slow data rate and latency when compared to a local filesystem (which,
>> in your case on loopback, should not be an issue), is that the OS can't
>> really use a cache for files on NFS since it has no way to know whether
>> or not it's valid. With a lot of random-access reads and writes, that
>> can be a HUGE slowdown, as you found.
>>
>>
>> Also this:
>> https://sqlite.org/faq.html#q5
>>
>> In other words, Subversion working copies on NFS are, and have always
>> been, a bad idea; not only because of SQLite but also because
>> Subversion's code itself relies on atomic renames, which NFS does not
>> provide.
>>
>> -- Brane
>
> What SVN command (on command line) I should test to get a similar
> result as from ISVNClient.getChangelists?

If "adding/removing a file" is any indication, then "svn add" or "svn
remove" should be comparable.

-- Brane

Re: Slow ISVNClient.getChangelists on Linux/NFS share

[Thomas Singer]

On 18.10.2017 19:56, Branko Čibej wrote:

On 18.10.2017 13:31, Thomas Singer wrote:

Hi,

When performing following steps on my old Linux test machine (with
slow hard disk):

- have a SVN working copy at /home/user/test
- sudo apt install nfs-kernel-server
- add following line to /etc/exports:
   /home/user/test *(rw,sync,no_root_squash)
- start the NFS server:
   sudo systemctl start nfs-kernel-server.service
- mount the NFS share:
   sudo mount localhost:/home/user/test /home/user/test.nfs

and then open /home/user/test.nfs in SmartSVN 9.2 (using SVN 1.9
JavaHL binaries), adding/removing a file is very slow. It boils down
to the call ISVNClient.getChangelists which takes ~8s on the NFS share
(/home/user/test.nfs). First, I thought, it would be caused by the
native-Java overhead calling the call-back ~11,000 times for my
working copy, but when using the working copy directly
(/home/user/test), the method just takes <1s though the ~11,000 times
call-back invocations are still there.

My working copy has no local modifications, no untracked or ignored
files, no changelists.

Is it expected that this method (ISVNClient.getChangelists) is so slow
on a NFS share even if there are no changelists?


I don't know if it's "expected" but I bet that NFS is killing SQLite
performance.

https://www.mail-archive.com/sqlite-users@mailinglists.sqlite.org/msg88989.html

I'm not sure about the reason but the most likely answer, apart from
slow data rate and latency when compared to a local filesystem (which,
in your case on loopback, should not be an issue), is that the OS can't
really use a cache for files on NFS since it has no way to know whether
or not it's valid. With a lot of random-access reads and writes, that
can be a HUGE slowdown, as you found.


Also this:
https://sqlite.org/faq.html#q5

In other words, Subversion working copies on NFS are, and have always
been, a bad idea; not only because of SQLite but also because
Subversion's code itself relies on atomic renames, which NFS does not
provide.

-- Brane


What SVN command (on command line) I should test to get a similar result 
as from ISVNClient.getChangelists? I've tried "git status" and it just 
needs <1s on the NFS working copy.


--
Best regards,
Thomas Singer
=
syntevo GmbH
http://www.syntevo.com
http://www.syntevo.com/blog