Re: Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)

[Stefan Sperling]

On Fri, Mar 02, 2018 at 05:15:46PM +0300, Evgeny Kotkov wrote:
> Unless I am missing something, this might be worth considering before the
> 1.10 GA release.

Evgeny, you were entirely right about calling this out as a release blocker.
I am sorry for having suggested otherwise.

Re: Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)

[Branko Čibej]

On 02.03.2018 19:21, Stefan Sperling wrote:
> I am just questioning the usefulness of halting the presses and restarting
> the soak for another month for something that isn't a security / data
> corruption issue.

It's a potential DOS that only needs read access. Falls under security
by my definition.

-- Brane

Re: Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)

[Stefan Sperling]

On Fri, Mar 02, 2018 at 07:21:15PM +0100, Stefan Sperling wrote:
> I am just questioning the usefulness of halting the presses and restarting
> the soak for another month for something that isn't a security / data
> corruption issue. I anticipate that problems of similar severity to this
> one will still be discovered after we release 1.10.0, regardless of
> whether we release 1.10.0 at end of March or later.
> 
> Though maybe my idea of the impact of this bug is wrong?
> If this really makes some repositories entirely unusable with authz enabled
> then of course it should be considered a blocker. Is it this severe?

I have misread our flowchart at:
http://subversion.apache.org/docs/community-guide/svn-soak-management.png

It seems for this kind of issue we'd extend soak by just one week instead
of four? I wouldn't mind a one-week extension for this kind of bug fix.
One week seems reasonable.

Re: Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)

[Stefan Sperling]

On Fri, Mar 02, 2018 at 09:02:02PM +0300, Evgeny Kotkov wrote:
> Stefan Sperling  writes:
> 
> > I'd rather ship 1.10.0 at the prospected release date followed closely
> > by 1.10.1 to fix bugs such as these, than delay general access to 1.10.0
> > even further.
> 
> While I do not have significant objections against such plan, I find the
> idea of shipping a performance feature that causes a massive slowdown
> instead of an improvement somewhat controversial.  (In other words,
> I am -0 for that.)
> 
> > You may not have realized this, but I have been waiting for 1.10.0 to
> > happen for *over a year* https://svn.haxx.se/dev/archive-2017-01/0043.shtml
> > For all this time, I have wanted the conflict resolver to get real world
> > exposure because I need feedback from users out there to improve it.
> > I kept mostly quiet because I didn't want to push too hard for this
> > release all by myself because of the relatively high share of burden
> > this would imply. So I waited for activity from the community to make
> > it happen as a true collective effort.
> 
> Not too sure about how this is connected to the soak period and to the
> release process — speaking of which, I would say that your e-mail may
> discourage people from reporting issues during the soak period.

I am not trying to discourage people from reporting and fixing problems.
I am sorry if what I wrote could be interpreted in this way.

I am just questioning the usefulness of halting the presses and restarting
the soak for another month for something that isn't a security / data
corruption issue. I anticipate that problems of similar severity to this
one will still be discovered after we release 1.10.0, regardless of
whether we release 1.10.0 at end of March or later.

Though maybe my idea of the impact of this bug is wrong?
If this really makes some repositories entirely unusable with authz enabled
then of course it should be considered a blocker. Is it this severe?

Re: Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)

[Evgeny Kotkov]

Stefan Sperling  writes:

> I'd rather ship 1.10.0 at the prospected release date followed closely
> by 1.10.1 to fix bugs such as these, than delay general access to 1.10.0
> even further.

While I do not have significant objections against such plan, I find the
idea of shipping a performance feature that causes a massive slowdown
instead of an improvement somewhat controversial.  (In other words,
I am -0 for that.)

> You may not have realized this, but I have been waiting for 1.10.0 to
> happen for *over a year* https://svn.haxx.se/dev/archive-2017-01/0043.shtml
> For all this time, I have wanted the conflict resolver to get real world
> exposure because I need feedback from users out there to improve it.
> I kept mostly quiet because I didn't want to push too hard for this
> release all by myself because of the relatively high share of burden
> this would imply. So I waited for activity from the community to make
> it happen as a true collective effort.

Not too sure about how this is connected to the soak period and to the
release process — speaking of which, I would say that your e-mail may
discourage people from reporting issues during the soak period.

> If this one bug really bothers you enough to hold the planned release back
> it makes me wonder why you didn't push for a fix much earlier. We have had
> plenty of time.

I haven't been and am not pushing for a fix.  Rather than that, I have just
included the additional information about the problem with a comment that
it might be viable to look into before the GA release.

Moreover, I reported the issue at the very moment I found it with an edge-case
reproduction.  Once I was asked to bisect for a specific revision, I should
have probably stated that I won't have time to do that.  But I have been
thinking that I would be able to find some.  When I stumbled across it again,
I found the revision and the simple reproduction — but as it seems, this
hasn't been the most appropriate time for including these new details.

Putting all that aside, I wouldn't say that it is productive to discuss issues
in such way.  In my opinion, we should be doing it the other way around by
actually encouraging reports of various problems during the soak period.


Regards,
Evgeny Kotkov

Re: Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)

[Stefan Sperling]

On Fri, Mar 02, 2018 at 05:15:46PM +0300, Evgeny Kotkov wrote:
> Unless I am missing something, this might be worth considering before the
> 1.10 GA release.

Not about the actual bug, just a meta comment on the process:

I'd rather ship 1.10.0 at the prospected release date followed closely
by 1.10.1 to fix bugs such as these, than delay general access to 1.10.0
even further.

You may not have realized this, but I have been waiting for 1.10.0 to
happen for *over a year* https://svn.haxx.se/dev/archive-2017-01/0043.shtml
For all this time, I have wanted the conflict resolver to get real world
exposure because I need feedback from users out there to improve it.
I kept mostly quiet because I didn't want to push too hard for this
release all by myself because of the relatively high share of burden
this would imply. So I waited for activity from the community to make
it happen as a true collective effort.
I was glad when Julian volunteered to drive the process.

If this one bug really bothers you enough to hold the planned release back
it makes me wonder why you didn't push for a fix much earlier. We have had
plenty of time. I don't mean to disrespect the fact that you may not have
had time recently -- we are all constraint for time these days.
But I also believe we shouldn't panic over every bug that slips into this
.0 release. It is OK to ship 1.10.0 with known bugs that aren't very
serious security / data corruption issues. There's a section in the
release notes which lists known issues. I would prefer to document this
memory consumption problem there and patch it like any other regular bug.

Re: Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)

[Evgeny Kotkov]

Stefan Fuhrmann  writes:

> Would it be possible for you to bisect this to find the offending revision?
> My random guess would be that in the context of mod_dav_svn, we might use
> an unsuitable pool for authz caching.

While looking through the various 1.10-related topics, I remembered about
this issue.  I have been able to narrow it down in my environment to
https://svn.apache.org/r1778923 (Ensure that even long-lived DAV
connections use up-to-date authz.)

Perhaps, a simpler reproduction script would be to issue an 'svn log' for
a medium-sized repository.  In my environment, doing so for the trunk of
TortoiseSVN's repository with 25,000 revisions causes the httpd process
to consume up to a 1 GB of RAM while processing the request.  Overall,
the log takes around 11 seconds instead of 2, compared to 1.9.7.

Unless I am missing something, this might be worth considering before the
1.10 GA release.


Thanks,
Evgeny Kotkov

Re: Subversion 1.10 RC1?

[Julian Foad]

Julian Foad wrote:
> Julian Foad wrote on 2017-12-15:
>> The 1.10 release branch has been created.

Progress update:

* 1.9 compatibility testing is done. Philip determined that the mixed-
  version test failures are not release critical.

* We are aware of two blockers:

  - STATUS has r1820778 tagged as a blocker:
  https://issues.apache.org/jira/browse/SVN-4718

Make the reporting of commit capabilites such as SVNDIFF version and
PUT checksums depend on the master version when a master-slave proxy
version is configured.  This allows 1.10 to be a slave proxy [...]

  - The protocol changes related to enabling LZ4/svndiff2 need to be
documented in the release notes.

Mid-March is now the earliest possible date for 1.10 release.

- Julian

Re: Subversion 1.10 RC1?

[Julian Foad]

Julian Foad wrote on 2017-12-15:

The 1.10 release branch has been created.

An RC1 tarball still needs to be rolled. I plan to do that by the end of 
December.


Progress update: we're still working through pre-RC1 issues.

The very end of February is now the earliest possible date for 1.10 release.

The is on the basis that we will need at least a week or two to 
check/resolve the compatibility-with-1.9 findings that Philip reported 
yesterday, and then we have a 4 week minimum 'soak' time from producing 
a release candidate until it can be released. I am not aware of any 
other issues holding up the RC1 at this time.


- Julian

Re: Subversion 1.10 RC1?

[Julian Foad]

Julian Foad wrote:

Julian Foad wrote:
At the hackathon today we (me, Stefan Hett, Bert, Johan) have been 
talking about how to progress 1.10.


It seems we have general consensus to move to an RC1. A few things we'd 
like to fix but no real blockers -- nothing we can't address during the 
RC phase.


I volunteer to drive the 1.10 release. It will be good to learn the 
process as I haven't done it before.


The 1.10 release branch has been created.

An RC1 tarball still needs to be rolled. I plan to do that by the end of 
December.


- Julian

Re: Subversion 1.10 RC1?

[Branko Čibej]

On 07.12.2017 17:42, Doug Robinson wrote:
>
> On Wed, Dec 6, 2017 at 7:09 PM, Daniel Shahaf  > wrote:
>
> Evgeny Kotkov wrote on Wed, Dec 06, 2017 at 00:12:55 +0300:
> > Unfortunately, on Windows both `--search M*` and (quoted)
> `--search "M*"`
> > would expand the asterisk.  While this is not the default shell
> behavior,
> > currently it's enabled for svn and a couple of other binaries by
> linking
> > to setargv.obj.  In turn, this probably means the command-line
> client
> > users on Windows could get quite unexpected results when using the
> > `--search ARG` syntax.
>
> Isn't there some escaping syntax, «svn log --search "M^*"» or
> something along
> these lines?
>
>
> Apparently
> not: 
> https://stackoverflow.com/questions/47512829/is-it-possible-to-quote-escape-command-line-arguments-on-windows-while-linking-s

Yes, it's a pity. I _almost_ regret not doing the grunt work for glob
expansion on Windows back in the day. But now that I'm not developing on
Windows, I don't regret it at all. :D


-- Brane

Re: Subversion 1.10 RC1?

[Doug Robinson]

On Wed, Dec 6, 2017 at 7:09 PM, Daniel Shahaf 
wrote:

> Evgeny Kotkov wrote on Wed, Dec 06, 2017 at 00:12:55 +0300:
> > Unfortunately, on Windows both `--search M*` and (quoted) `--search "M*"`
> > would expand the asterisk.  While this is not the default shell behavior,
> > currently it's enabled for svn and a couple of other binaries by linking
> > to setargv.obj.  In turn, this probably means the command-line client
> > users on Windows could get quite unexpected results when using the
> > `--search ARG` syntax.
>
> Isn't there some escaping syntax, «svn log --search "M^*"» or something
> along
> these lines?
>

Apparently not:
https://stackoverflow.com/questions/47512829/is-it-possible-to-quote-escape-command-line-arguments-on-windows-while-linking-s

-- 
*DOUGLAS B ROBINSON* SENIOR PRODUCT MANAGER

T +1 925 396 1125
*E* doug.robin...@wandisco.com

-- 


World Leader in Active Data Replication™
*Find out more wandisco.com *

THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY AND MAY BE 
PRIVILEGED

If this message was misdirected, WANdisco, Inc. and its subsidiaries, 
("WANdisco") does not waive any confidentiality or privilege. If you are 
not the intended recipient, please notify us immediately and destroy the 
message without disclosing its contents to anyone. Any distribution, use or 
copying of this email or the information it contains by other than an 
intended recipient is unauthorized. The views and opinions expressed in 
this email message are the author's own and may not reflect the views and 
opinions of WANdisco, unless the author is authorized by WANdisco to 
express such views or opinions on its behalf. All email sent to or from 
this address is subject to electronic storage and review by WANdisco. 
Although WANdisco operates anti-virus programs, it does not accept 
responsibility for any damage whatsoever caused by viruses being passed.

Re: Subversion 1.10 RC1?

[Daniel Shahaf]

Evgeny Kotkov wrote on Wed, Dec 06, 2017 at 00:12:55 +0300:
> Unfortunately, on Windows both `--search M*` and (quoted) `--search "M*"`
> would expand the asterisk.  While this is not the default shell behavior,
> currently it's enabled for svn and a couple of other binaries by linking
> to setargv.obj.  In turn, this probably means the command-line client
> users on Windows could get quite unexpected results when using the
> `--search ARG` syntax.

Isn't there some escaping syntax, «svn log --search "M^*"» or something along
these lines?

Re: Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)

[Stefan Fuhrmann]

On 05.12.2017 22:05, Evgeny Kotkov wrote:

Julian Foad  writes:


After any issues raised in this discussion are resolved, we feel we should
go ahead and produce RC1 as soon as possible.


I think that I am seeing a 1.10 regression in terms of httpd's memory
usage during large imports.

In my environment, when I `svn import` an unpacked version of Boost
(https://www.boost.org/) that consists of around 60,000 files, I see that
the memory consumption by the server process rapidly grows up to 1.5 GB.
The environment is a Windows 8.1 x64 machine with httpd 2.4.29 configured
to use short-circuit authz and HTTPS.  Apparently, this behavior is specific
to 1.10, as I cannot reproduce it with 1.9 binaries.

  (I haven't investigated the issue any further at this time, and it might as
   well be reproducible in other configurations.)


Would it be possible for you to bisect this to
find the offending revision?  My random guess
would be that in the context of mod_dav_svn, we
might use an unsuitable pool for authz caching.

-- Stefan^2.

Re: Subversion 1.10 RC1?

[Johan Corveleyn]

Op 5 dec. 2017 22:13 schreef "Evgeny Kotkov" :

Stefan Fuhrmann  writes:

> There seems to be little that could be done here (suggestions welcome).
> The problem is that the asterisk is being expanded by the shell itself.
> I made SVN print its command line parameters and this is the result:
>
> $ ./subversion/svn/svn ls svn://localhost/kde --search M*
> 0: ./subversion/svn/svn
> 1: ls
> 2: svn://localhost/kde
> 3: --search
> 4: Makefile
> 5: Makefile.in
>
> That can be prevented by adding quotation marks:
>
> $ ./subversion/svn/svn ls svn://localhost/kde --search "M*"
> 0: ./subversion/svn/svn
> 1: ls
> 2: svn://localhost/kde
> 3: --search
> 4: M*

Unfortunately, on Windows both `--search M*` and (quoted) `--search "M*"`
would expand the asterisk.  While this is not the default shell behavior,
currently it's enabled for svn and a couple of other binaries by linking
to setargv.obj.  In turn, this probably means the command-line client
users on Windows could get quite unexpected results when using the
`--search ARG` syntax.

A potential cheap solution for this issue, I think, would be to make the
--search argument work as a substring to search for in filenames, instead
of using it as a pattern that the (whole) filename should match.  While
there are some cases where the latter approach gives more flexibility,
my guess would be that a substring search would work well in the majority
of scenarios.

(Also, as far as I recall, `log --search` currently searches for a
substring,
 so that would be consistent with it, and would probably avoid surprising
 the users by having a switch with the same name, but behaving differently.)


Thanks,
Evgeny Kotkov



(Sorry if quoting is broken above ... I blame Google)

We've discussed whether ls --search should always match substrings or exact
before, and the people participating in that thread were in favor of exact
matching:

https://svn.haxx.se/dev/archive-2017-09/0002.shtml


Consistency with log --search also came up, but wasn't a good argument for
most.

Of course, if this would be the only way to make this work on Windows, some
people might reconsider ...

-- 
Johan

Re: Subversion 1.10 RC1?

[Branko Čibej]

On 05.12.2017 22:12, Evgeny Kotkov wrote:
> Stefan Fuhrmann  writes:
>
>> There seems to be little that could be done here (suggestions welcome).
>> The problem is that the asterisk is being expanded by the shell itself.
>> I made SVN print its command line parameters and this is the result:
>>
>> $ ./subversion/svn/svn ls svn://localhost/kde --search M*
>> 0: ./subversion/svn/svn
>> 1: ls
>> 2: svn://localhost/kde
>> 3: --search
>> 4: Makefile
>> 5: Makefile.in
>>
>> That can be prevented by adding quotation marks:
>>
>> $ ./subversion/svn/svn ls svn://localhost/kde --search "M*"
>> 0: ./subversion/svn/svn
>> 1: ls
>> 2: svn://localhost/kde
>> 3: --search
>> 4: M*
> Unfortunately, on Windows both `--search M*` and (quoted) `--search "M*"`
> would expand the asterisk.  While this is not the default shell behavior,
> currently it's enabled for svn and a couple of other binaries by linking
> to setargv.obj.

We do that for the command-line client and in a few other places, too.
It was the easy way out for Windows.

It's unfortunate, really ... ideally, we'd remove setargv.obj and do the
wildcard expansion ourselves, but someone has to do the work; it's very
Windows-specific and there are probably nasty edge cases around argument
quoting.

-- Brane

Re: Subversion 1.10 RC1?

[Evgeny Kotkov]

Stefan Fuhrmann  writes:

> There seems to be little that could be done here (suggestions welcome).
> The problem is that the asterisk is being expanded by the shell itself.
> I made SVN print its command line parameters and this is the result:
>
> $ ./subversion/svn/svn ls svn://localhost/kde --search M*
> 0: ./subversion/svn/svn
> 1: ls
> 2: svn://localhost/kde
> 3: --search
> 4: Makefile
> 5: Makefile.in
>
> That can be prevented by adding quotation marks:
>
> $ ./subversion/svn/svn ls svn://localhost/kde --search "M*"
> 0: ./subversion/svn/svn
> 1: ls
> 2: svn://localhost/kde
> 3: --search
> 4: M*

Unfortunately, on Windows both `--search M*` and (quoted) `--search "M*"`
would expand the asterisk.  While this is not the default shell behavior,
currently it's enabled for svn and a couple of other binaries by linking
to setargv.obj.  In turn, this probably means the command-line client
users on Windows could get quite unexpected results when using the
`--search ARG` syntax.

A potential cheap solution for this issue, I think, would be to make the
--search argument work as a substring to search for in filenames, instead
of using it as a pattern that the (whole) filename should match.  While
there are some cases where the latter approach gives more flexibility,
my guess would be that a substring search would work well in the majority
of scenarios.

(Also, as far as I recall, `log --search` currently searches for a substring,
 so that would be consistent with it, and would probably avoid surprising
 the users by having a switch with the same name, but behaving differently.)


Thanks,
Evgeny Kotkov

Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)

[Evgeny Kotkov]

Julian Foad  writes:

> After any issues raised in this discussion are resolved, we feel we should
> go ahead and produce RC1 as soon as possible.

I think that I am seeing a 1.10 regression in terms of httpd's memory
usage during large imports.

In my environment, when I `svn import` an unpacked version of Boost
(https://www.boost.org/) that consists of around 60,000 files, I see that
the memory consumption by the server process rapidly grows up to 1.5 GB.
The environment is a Windows 8.1 x64 machine with httpd 2.4.29 configured
to use short-circuit authz and HTTPS.  Apparently, this behavior is specific
to 1.10, as I cannot reproduce it with 1.9 binaries.

 (I haven't investigated the issue any further at this time, and it might as
  well be reproducible in other configurations.)


Thanks,
Evgeny Kotkov

Re: Subversion 1.10 RC1?

[Stefan Sperling]

On Mon, Dec 04, 2017 at 12:51:26PM +0100, Stefan Fuhrmann wrote:
> Work on 'svn ls --search' has been completed now,
> release notes and friends have been updated.
> 
> -- Stefan^2.

Thank you!

Re: Subversion 1.10 RC1?

[Stefan Fuhrmann]

On 22.11.2017 14:21, Stefan Fuhrmann wrote:

On 22.11.2017 11:53, Julian Foad wrote:
At the hackathon today we (me, Stefan Hett, Bert, Johan) have been talking 
about how to progress 1.10.


We think all the features and changes are safe to release and are not going to 
get more testing until we produce a "release candidate". (For example, at that 
point Stefan will be able to justify taking time at his work to test the 
client in production scenarios.)

My idea for the hackathon was to add native 'svn list' support
to ra_serf. But that is not a release blocker.


Work on 'svn ls --search' has been completed now,
release notes and friends have been updated.

-- Stefan^2.

Re: Subversion 1.10 RC1?

[Stefan Fuhrmann]

On 22.11.2017 15:48, Evgeny Kotkov wrote:

The other two features that I remember, are:

* improved authz with support for wildcards

* server-side search with `svn ls --search`

Speaking of the `ls --search`, I think that there is an issue with the
command-line parsing.  Based on what I see, the --search argument may
be interpreted as the target for listing, but not as the search pattern:

 svn ls --search *

 svn: warning: apr_err=SVN_ERR_WC_PATH_NOT_FOUND
 svn: warning: W155010: The node 'C:\Project\unversioned' was not found.
 ..\..\..\subversion\svn\list-cmd.c:453: (apr_err=SVN_ERR_ILLEGAL_TARGET)
 svn: E29: Could not list all targets because some targets don't exist

 svn ls http://spbvo-ws09.ostyserver.net:8080/svn/master2 --search *
 svn: E155007: 'C:\AnotherProject' is not a working copy


There seems to be little that could be done here (suggestions welcome).
The problem is that the asterisk is being expanded by the shell itself.
I made SVN print its command line parameters and this is the result:

$ ./subversion/svn/svn ls svn://localhost/kde --search M*
0: ./subversion/svn/svn
1: ls
2: svn://localhost/kde
3: --search
4: Makefile
5: Makefile.in

That can be prevented by adding quotation marks:

$ ./subversion/svn/svn ls svn://localhost/kde --search "M*"
0: ./subversion/svn/svn
1: ls
2: svn://localhost/kde
3: --search
4: M*

So, I extended the user docstring accordingly in r1817053.

-- Stefan^2.

Re: Subversion 1.10 RC1?

[Stefan]

On 25/11/2017 22:14, Julian Foad wrote:
> Julian Foad wrote:
>> At the hackathon today we (me, Stefan Hett, Bert, Johan) have been
>> talking about how to progress 1.10.
>
> It seems we have general consensus to move to an RC1. A few things
> we'd like to fix but no real blockers -- nothing we can't address
> during the RC phase.
>
> I volunteer to drive the 1.10 release. It will be good to learn the
> process as I haven't done it before.
>
> (We also have some outstanding fixes in 1.9 and 1.8. Maybe I will
> drive those too. Doing all three in parallel would encourage me to
> keep the release process automation up to date.)
>
> - Julian

Great Julian, thanks for taking care about this.

Regards,
Stefan

Re: Subversion 1.10 RC1?

[Julian Foad]

Julian Foad wrote:
At the hackathon today we (me, Stefan Hett, Bert, Johan) have been 
talking about how to progress 1.10.


It seems we have general consensus to move to an RC1. A few things we'd 
like to fix but no real blockers -- nothing we can't address during the 
RC phase.


I volunteer to drive the 1.10 release. It will be good to learn the 
process as I haven't done it before.


(We also have some outstanding fixes in 1.9 and 1.8. Maybe I will drive 
those too. Doing all three in parallel would encourage me to keep the 
release process automation up to date.)


- Julian

Re: Subversion 1.10 RC1?

[Paul Hammant]

I think you're misunderstanding me. I noted this in the conversation above:
"One conflict option that's *not done yet* is one that merges a textual
change from a path ^/trunk/B to a path ^/branch/A, after a move A -> B on
trunk. I don't consider this a release blocker. We can always add this
option in a patch release."

I interpreted that comment to be about something merged/integrated that
should not go ahead just yet, and deserving of some consideration.

I felt I had a suggestion that may be helpful to allow the release of
something alpha/beta/RC-ish to go ahead even with something the team felt
wasn't ready for release.

To reiterate: I agree the feature is useful and I have nothing to say that
I would wish to have interpreted as a concern, was trying to help speed
things up not slow things down.



On Thu, Nov 23, 2017 at 4:04 PM, Stefan Sperling  wrote:

> On Thu, Nov 23, 2017 at 03:53:02PM -0500, Paul Hammant wrote:
> > >
> > >
> > > > Is that a regression versus v1.9 and before?
> > >
> > > Far from it.
> > > This discussion is about the new conflict resovler added in 1.10.
> > >
> > > We can and always could perform such merges by manually specifying
> > > the paths, i.e. running a merge from ^/trunk/B to branch/A.
> > >
> > > The goal of the resolver is to make it easier to perform such merges
> > > in the straightforward 'svn merge ^/trunk' style.
> > >
> >
> > OK, so at the risk of representing *the department of unsolicited
> advice*,
> > I don't know why you would not ship with as 'off' but able to be toggled
> > 'on' client-side with a new option
> > --with-experimental-1-10-conflict-enhancement-technology
> > (or a better name). The 0.1% of users that want to play with it before it
> > is ready can do.
> >
> > - Paul
>
> Why? There's a lot of benefits in this feature. It has been worked on
> sicnce May 2015 and is already pretty useful. There's a high-level
> description in the 1.10 release notes, have you seen that?
> http://subversion.apache.org/docs/release-notes/1.10.html#
> conflict-resolver
>
> You're a bringing your concerns up a little late :)
> But I'm not even sure why you would have any concerns about this.
>



-- 
Paul Hammant DevOps  Let me give your
enterprise a step by step plan to get out of the hell crazy branching
models (ClearCase maybe?) and into the world of high-throughput CD on
DevOps foundations.

Re: Subversion 1.10 RC1?

[Stefan Sperling]

On Thu, Nov 23, 2017 at 03:53:02PM -0500, Paul Hammant wrote:
> >
> >
> > > Is that a regression versus v1.9 and before?
> >
> > Far from it.
> > This discussion is about the new conflict resovler added in 1.10.
> >
> > We can and always could perform such merges by manually specifying
> > the paths, i.e. running a merge from ^/trunk/B to branch/A.
> >
> > The goal of the resolver is to make it easier to perform such merges
> > in the straightforward 'svn merge ^/trunk' style.
> >
> 
> OK, so at the risk of representing *the department of unsolicited advice*,
> I don't know why you would not ship with as 'off' but able to be toggled
> 'on' client-side with a new option
> --with-experimental-1-10-conflict-enhancement-technology
> (or a better name). The 0.1% of users that want to play with it before it
> is ready can do.
> 
> - Paul

Why? There's a lot of benefits in this feature. It has been worked on
sicnce May 2015 and is already pretty useful. There's a high-level
description in the 1.10 release notes, have you seen that?
http://subversion.apache.org/docs/release-notes/1.10.html#conflict-resolver

You're a bringing your concerns up a little late :)
But I'm not even sure why you would have any concerns about this.

Re: Subversion 1.10 RC1?

[Paul Hammant]

>
>
> > Is that a regression versus v1.9 and before?
>
> Far from it.
> This discussion is about the new conflict resovler added in 1.10.
>
> We can and always could perform such merges by manually specifying
> the paths, i.e. running a merge from ^/trunk/B to branch/A.
>
> The goal of the resolver is to make it easier to perform such merges
> in the straightforward 'svn merge ^/trunk' style.
>

OK, so at the risk of representing *the department of unsolicited advice*,
I don't know why you would not ship with as 'off' but able to be toggled
'on' client-side with a new option
--with-experimental-1-10-conflict-enhancement-technology
(or a better name). The 0.1% of users that want to play with it before it
is ready can do.

- Paul

Re: Subversion 1.10 RC1?

[Stefan Sperling]

On Thu, Nov 23, 2017 at 10:55:02AM -0500, Paul Hammant wrote:
> >
> >
> > One conflict option that's not done yet is one that merges a textual change
> > from a path ^/trunk/B to a path ^/branch/A, after a move A -> B on trunk.
> > I don't consider this a release blocker. We can always add this option in
> > a patch release.
> >
> >
> Is that a regression versus v1.9 and before?

Far from it.
This discussion is about the new conflict resovler added in 1.10.

We can and always could perform such merges by manually specifying
the paths, i.e. running a merge from ^/trunk/B to branch/A.

The goal of the resolver is to make it easier to perform such merges
in the straightforward 'svn merge ^/trunk' style.

Re: Subversion 1.10 RC1?

[Paul Hammant]

>
>
> One conflict option that's not done yet is one that merges a textual change
> from a path ^/trunk/B to a path ^/branch/A, after a move A -> B on trunk.
> I don't consider this a release blocker. We can always add this option in
> a patch release.
>
>
Is that a regression versus v1.9 and before?

- Paul

Re: Subversion 1.10 RC1?

[Daniel Shahaf]

Stefan Sperling wrote on Wed, Nov 22, 2017 at 13:41:11 +0100:
> But I probably won't have time to help with preparations.

Likewise.

Re: Subversion 1.10 RC1?

[Daniel Shahaf]

Julian Foad wrote on Wed, Nov 22, 2017 at 17:17:34 +0100:
> Re. shelving...
> 
> Branko Čibej wrote:
> > Unless you're absolutely certain that the format and semantics of the
> > CLI commands won't change, I do suggest adding an "experimental" warning
> > to the help text.
> 
> Done. Thanks.

I don't think saying "This command is not forwards compatible" in the help text
will prevent users from relying on it being forwards compatible; which, in
turn, will discourage us, come 1.11, to make incompatible changes to this
command.

Perhaps we should rename the command to, say, "xshelve" — like the "X-" prefix
of experimental email headers, but without a minus for ease of typing?  Then we
could have a convention, "any command whose name starts with 'x' is not
guaranteed to be forwards compatible", and we'd be able to make incompatible
changes (to "xshelve") without worrying about users asking for compatibility
despite the documentation.  In 1.11, if we wanted to make "shelve" stable, we
could even continue to accept "xshelve" as an alias, if the semantics of
xshelve@1.10 and shelve@1.11 are compatible.

Re: Subversion 1.10 RC1?

[Julian Foad]

Re. shelving...

Branko Čibej wrote:

Unless you're absolutely certain that the format and semantics of the
CLI commands won't change, I do suggest adding an "experimental" warning
to the help text.


Done. Thanks.

- Julian

Re: Subversion 1.10 RC1?

[Julian Foad]

Re. shelving...

Evgeny Kotkov wrote:

Julian Foad  writes:

   * shelving v1: is isolated -- doesn't affect anything else; is limited but
already useful; will be changed in the next release so APIs are marked
"SVN_EXPERIMENTAL"; changes shelved by this release could be detected and
'upgraded' by a future release; should the CLI commands be marked
"experimental" in the help, too (Johan thinks yes)?


One comment that I have is that marking the new commands experimental
might not prevent the users from using them on a regular basis (e.g., for
those who don't read help or use a GUI client like TSVN that might not
have the "experimental" labels).

Which, in turn, could mean that if the future format changes, we would
have to convert the data stored in the patch files to avoid a data loss
for such users.


I agree we need to take care of data stored by the user using this 
version. As the patches stored by this version are simple patch files 
compatible with 'svn patch', we would not need to be able to upgrade 
them to a new format. It would be possible to keep support for them in 
the existing format. In the worst case, the user can recover them 
manually and use 'svn patch'.



Also, out of curiosity, are there any current plans to support binary changes
for the shelves?

As far as I recall, there has been a mention of using diff --git for these
purposes, and I also saw a recent commit where you added a test for diff
--git, which might be related to this topic :)


Yes, absolutely, binary support is high on my priority list, and yes I'm 
considering git-diff format.


Thanks!

- Julian

Re: Subversion 1.10 RC1?

[Evgeny Kotkov]

Julian Foad  writes:

> At the hackathon today we (me, Stefan Hett, Bert, Johan) have been talking
> about how to progress 1.10.
>
> We think all the features and changes are safe to release and are not going
> to get more testing until we produce a "release candidate". (For example, at
> that point Stefan will be able to justify taking time at his work to test
> the client in production scenarios.)

I did a couple of quick tests for these new features; some of my findings
that we might want to address before 1.10 GA and additional comments are
below:

>   * conflict resolution: we understand it is already much better than 1.9;
> low risk if parts of it don't work quite right; designed so that
> improvements can be made in patch releases.

I noticed that if the resolution of a tree conflict leads to a text conflict,
the temporary files (base, mine, theirs) are not cleaned up upon revert
and remain as unversioned files in the working copy.

C  +NewFile.txt
> moved from OldFile.txt
?   NewFile.txt.2.tmp
?   NewFile.txt.3.tmp
?   NewFile.txt.tmp

>   * shelving v1: is isolated -- doesn't affect anything else; is limited but
> already useful; will be changed in the next release so APIs are marked
> "SVN_EXPERIMENTAL"; changes shelved by this release could be detected and
> 'upgraded' by a future release; should the CLI commands be marked
> "experimental" in the help, too (Johan thinks yes)?

One comment that I have is that marking the new commands experimental
might not prevent the users from using them on a regular basis (e.g., for
those who don't read help or use a GUI client like TSVN that might not
have the "experimental" labels).

Which, in turn, could mean that if the future format changes, we would
have to convert the data stored in the patch files to avoid a data loss
for such users.

Also, out of curiosity, are there any current plans to support binary changes
for the shelves?

As far as I recall, there has been a mention of using diff --git for these
purposes, and I also saw a recent commit where you added a test for diff
--git, which might be related to this topic :)


The other two features that I remember, are:

* improved authz with support for wildcards

* server-side search with `svn ls --search`

Speaking of the `ls --search`, I think that there is an issue with the
command-line parsing.  Based on what I see, the --search argument may
be interpreted as the target for listing, but not as the search pattern:

svn ls --search *

svn: warning: apr_err=SVN_ERR_WC_PATH_NOT_FOUND
svn: warning: W155010: The node 'C:\Project\unversioned' was not found.
..\..\..\subversion\svn\list-cmd.c:453: (apr_err=SVN_ERR_ILLEGAL_TARGET)
svn: E29: Could not list all targets because some targets don't exist

svn ls http://spbvo-ws09.ostyserver.net:8080/svn/master2 --search *
svn: E155007: 'C:\AnotherProject' is not a working copy


Thanks,
Evgeny Kotkov

Re: Subversion 1.10 RC1?

[Stefan Fuhrmann]

On 22.11.2017 11:53, Julian Foad wrote:
At the hackathon today we (me, Stefan Hett, Bert, Johan) have been 
talking about how to progress 1.10.


We think all the features and changes are safe to release and are not 
going to get more testing until we produce a "release candidate". (For 
example, at that point Stefan will be able to justify taking time at 
his work to test the client in production scenarios.)

My idea for the hackathon was to add native 'svn list' support
to ra_serf. But that is not a release blocker.


  * conflict resolution: we understand it is already much better than 
1.9; low risk if parts of it don't work quite right; designed so that 
improvements can be made in patch releases.

If we broke something, it is likely one of the more complicated
cases where the need for manual intervention is not all that
surprising to the user.


  * LZ4 compression: in some senses the risk of bugs here is higher, 
but it seems like it is high quality already; is there one remaining 
place where we should add LZ4 negotiation (one direction of svnserve 
protocol)?

ra_svn is certainly the ra layer that may benefit the most from LZ4.
Browsing through the code, I found a couple of TODOs that should
be simple to address this week.

I've got some ideas to enhance the protocol in 1.11 and that would
include using LZ4 for "mod_compress"-like full protocol compression.
However, those are non-trivial and are the one thing I like to discuss
with the people at the hackathon.


  * shelving v1: is isolated -- doesn't affect anything else; is 
limited but already useful; will be changed in the next release so 
APIs are marked "SVN_EXPERIMENTAL"; changes shelved by this release 
could be detected and 'upgraded' by a future release; should the CLI 
commands be marked "experimental" in the help, too (Johan thinks yes)?

+1 on letting people gain experience with shelving.
+1 on marking the feature "experimental" in the UI b/c we might not
want to fully commit ourselves to upgradability. That is a departure
from the policy of previous releases.


After any issues raised in this discussion are resolved, we feel we 
should go ahead and produce RC1 as soon as possible.

Assuming no blockers come up, I think coming weekend would be
a reasonable value for $asap.

-- Stefan^2.

Re: Subversion 1.10 RC1?

[Stefan Sperling]

On Wed, Nov 22, 2017 at 11:53:11AM +0100, Julian Foad wrote:
> At the hackathon today we (me, Stefan Hett, Bert, Johan) have been talking
> about how to progress 1.10.
> 
> We think all the features and changes are safe to release and are not going
> to get more testing until we produce a "release candidate". (For example, at
> that point Stefan will be able to justify taking time at his work to test
> the client in production scenarios.)
> 
>   * conflict resolution: we understand it is already much better than 1.9;
> low risk if parts of it don't work quite right; designed so that
> improvements can be made in patch releases.

One conflict option that's not done yet is one that merges a textual change
from a path ^/trunk/B to a path ^/branch/A, after a move A -> B on trunk.
I don't consider this a release blocker. We can always add this option in
a patch release.

I would welcome a 1.10.0-RC1 release very much. It feels long overdue.
But I probably won't have time to help with preparations.

Re: Subversion 1.10 RC1?

[Branko Čibej]

On 22.11.2017 11:53, Julian Foad wrote:
> At the hackathon today we (me, Stefan Hett, Bert, Johan) have been
> talking about how to progress 1.10.
>
> We think all the features and changes are safe to release and are not
> going to get more testing until we produce a "release candidate". (For
> example, at that point Stefan will be able to justify taking time at
> his work to test the client in production scenarios.)
>
>   * conflict resolution: we understand it is already much better than
> 1.9; low risk if parts of it don't work quite right; designed so that
> improvements can be made in patch releases.

Other than the new compiler warnings that keep popping up on trunk
related to the conflict resolution, I have no objections.

>   * LZ4 compression: in some senses the risk of bugs here is higher,
> but it seems like it is high quality already; is there one remaining
> place where we should add LZ4 negotiation (one direction of svnserve
> protocol)?

Would be nice to have LZ4 negotiation everywhere but not a blocker.

>   * shelving v1: is isolated -- doesn't affect anything else; is
> limited but already useful; will be changed in the next release so
> APIs are marked "SVN_EXPERIMENTAL"; changes shelved by this release
> could be detected and 'upgraded' by a future release; should the CLI
> commands be marked "experimental" in the help, too (Johan thinks yes)?

Unless you're absolutely certain that the format and semantics of the
CLI commands won't change, I do suggest adding an "experimental" warning
to the help text.

> After any issues raised in this discussion are resolved, we feel we
> should go ahead and produce RC1 as soon as possible.

+(1 − ε)

-- Brane

Subversion 1.10 RC1?

[Julian Foad]

At the hackathon today we (me, Stefan Hett, Bert, Johan) have been 
talking about how to progress 1.10.


We think all the features and changes are safe to release and are not 
going to get more testing until we produce a "release candidate". (For 
example, at that point Stefan will be able to justify taking time at his 
work to test the client in production scenarios.)


  * conflict resolution: we understand it is already much better than 
1.9; low risk if parts of it don't work quite right; designed so that 
improvements can be made in patch releases.


  * LZ4 compression: in some senses the risk of bugs here is higher, 
but it seems like it is high quality already; is there one remaining 
place where we should add LZ4 negotiation (one direction of svnserve 
protocol)?


  * shelving v1: is isolated -- doesn't affect anything else; is 
limited but already useful; will be changed in the next release so APIs 
are marked "SVN_EXPERIMENTAL"; changes shelved by this release could be 
detected and 'upgraded' by a future release; should the CLI commands be 
marked "experimental" in the help, too (Johan thinks yes)?


After any issues raised in this discussion are resolved, we feel we 
should go ahead and produce RC1 as soon as possible.


- Julian (& Stefan, Bert, Johan)