Re: Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)

[Stefan Sperling]

On Fri, Mar 02, 2018 at 05:15:46PM +0300, Evgeny Kotkov wrote:
> Unless I am missing something, this might be worth considering before the
> 1.10 GA release.

Evgeny, you were entirely right about calling this out as a release blocker.
I am sorry for having suggested otherwise.

Re: Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)

[Stefan Sperling]

On Fri, Mar 02, 2018 at 07:21:15PM +0100, Stefan Sperling wrote:
> I am just questioning the usefulness of halting the presses and restarting
> the soak for another month for something that isn't a security / data
> corruption issue. I anticipate that problems of similar severity to this
> one will still be discovered after we release 1.10.0, regardless of
> whether we release 1.10.0 at end of March or later.
> 
> Though maybe my idea of the impact of this bug is wrong?
> If this really makes some repositories entirely unusable with authz enabled
> then of course it should be considered a blocker. Is it this severe?

I have misread our flowchart at:
http://subversion.apache.org/docs/community-guide/svn-soak-management.png

It seems for this kind of issue we'd extend soak by just one week instead
of four? I wouldn't mind a one-week extension for this kind of bug fix.
One week seems reasonable.

Re: Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)

[Stefan Sperling]

On Fri, Mar 02, 2018 at 09:02:02PM +0300, Evgeny Kotkov wrote:
> Stefan Sperling  writes:
> 
> > I'd rather ship 1.10.0 at the prospected release date followed closely
> > by 1.10.1 to fix bugs such as these, than delay general access to 1.10.0
> > even further.
> 
> While I do not have significant objections against such plan, I find the
> idea of shipping a performance feature that causes a massive slowdown
> instead of an improvement somewhat controversial.  (In other words,
> I am -0 for that.)
> 
> > You may not have realized this, but I have been waiting for 1.10.0 to
> > happen for *over a year* https://svn.haxx.se/dev/archive-2017-01/0043.shtml
> > For all this time, I have wanted the conflict resolver to get real world
> > exposure because I need feedback from users out there to improve it.
> > I kept mostly quiet because I didn't want to push too hard for this
> > release all by myself because of the relatively high share of burden
> > this would imply. So I waited for activity from the community to make
> > it happen as a true collective effort.
> 
> Not too sure about how this is connected to the soak period and to the
> release process — speaking of which, I would say that your e-mail may
> discourage people from reporting issues during the soak period.

I am not trying to discourage people from reporting and fixing problems.
I am sorry if what I wrote could be interpreted in this way.

I am just questioning the usefulness of halting the presses and restarting
the soak for another month for something that isn't a security / data
corruption issue. I anticipate that problems of similar severity to this
one will still be discovered after we release 1.10.0, regardless of
whether we release 1.10.0 at end of March or later.

Though maybe my idea of the impact of this bug is wrong?
If this really makes some repositories entirely unusable with authz enabled
then of course it should be considered a blocker. Is it this severe?

Re: Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)

[Evgeny Kotkov]

Stefan Sperling  writes:

> I'd rather ship 1.10.0 at the prospected release date followed closely
> by 1.10.1 to fix bugs such as these, than delay general access to 1.10.0
> even further.

While I do not have significant objections against such plan, I find the
idea of shipping a performance feature that causes a massive slowdown
instead of an improvement somewhat controversial.  (In other words,
I am -0 for that.)

> You may not have realized this, but I have been waiting for 1.10.0 to
> happen for *over a year* https://svn.haxx.se/dev/archive-2017-01/0043.shtml
> For all this time, I have wanted the conflict resolver to get real world
> exposure because I need feedback from users out there to improve it.
> I kept mostly quiet because I didn't want to push too hard for this
> release all by myself because of the relatively high share of burden
> this would imply. So I waited for activity from the community to make
> it happen as a true collective effort.

Not too sure about how this is connected to the soak period and to the
release process — speaking of which, I would say that your e-mail may
discourage people from reporting issues during the soak period.

> If this one bug really bothers you enough to hold the planned release back
> it makes me wonder why you didn't push for a fix much earlier. We have had
> plenty of time.

I haven't been and am not pushing for a fix.  Rather than that, I have just
included the additional information about the problem with a comment that
it might be viable to look into before the GA release.

Moreover, I reported the issue at the very moment I found it with an edge-case
reproduction.  Once I was asked to bisect for a specific revision, I should
have probably stated that I won't have time to do that.  But I have been
thinking that I would be able to find some.  When I stumbled across it again,
I found the revision and the simple reproduction — but as it seems, this
hasn't been the most appropriate time for including these new details.

Putting all that aside, I wouldn't say that it is productive to discuss issues
in such way.  In my opinion, we should be doing it the other way around by
actually encouraging reports of various problems during the soak period.


Regards,
Evgeny Kotkov

Re: Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)

[Stefan Sperling]

On Fri, Mar 02, 2018 at 05:15:46PM +0300, Evgeny Kotkov wrote:
> Unless I am missing something, this might be worth considering before the
> 1.10 GA release.

Not about the actual bug, just a meta comment on the process:

I'd rather ship 1.10.0 at the prospected release date followed closely
by 1.10.1 to fix bugs such as these, than delay general access to 1.10.0
even further.

You may not have realized this, but I have been waiting for 1.10.0 to
happen for *over a year* https://svn.haxx.se/dev/archive-2017-01/0043.shtml
For all this time, I have wanted the conflict resolver to get real world
exposure because I need feedback from users out there to improve it.
I kept mostly quiet because I didn't want to push too hard for this
release all by myself because of the relatively high share of burden
this would imply. So I waited for activity from the community to make
it happen as a true collective effort.
I was glad when Julian volunteered to drive the process.

If this one bug really bothers you enough to hold the planned release back
it makes me wonder why you didn't push for a fix much earlier. We have had
plenty of time. I don't mean to disrespect the fact that you may not have
had time recently -- we are all constraint for time these days.
But I also believe we shouldn't panic over every bug that slips into this
.0 release. It is OK to ship 1.10.0 with known bugs that aren't very
serious security / data corruption issues. There's a section in the
release notes which lists known issues. I would prefer to document this
memory consumption problem there and patch it like any other regular bug.

Re: Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)

[Evgeny Kotkov]

Stefan Fuhrmann  writes:

> Would it be possible for you to bisect this to find the offending revision?
> My random guess would be that in the context of mod_dav_svn, we might use
> an unsuitable pool for authz caching.

While looking through the various 1.10-related topics, I remembered about
this issue.  I have been able to narrow it down in my environment to
https://svn.apache.org/r1778923 (Ensure that even long-lived DAV
connections use up-to-date authz.)

Perhaps, a simpler reproduction script would be to issue an 'svn log' for
a medium-sized repository.  In my environment, doing so for the trunk of
TortoiseSVN's repository with 25,000 revisions causes the httpd process
to consume up to a 1 GB of RAM while processing the request.  Overall,
the log takes around 11 seconds instead of 2, compared to 1.9.7.

Unless I am missing something, this might be worth considering before the
1.10 GA release.


Thanks,
Evgeny Kotkov

Re: Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)

[Stefan Fuhrmann]

On 05.12.2017 22:05, Evgeny Kotkov wrote:

Julian Foad  writes:


After any issues raised in this discussion are resolved, we feel we should
go ahead and produce RC1 as soon as possible.


I think that I am seeing a 1.10 regression in terms of httpd's memory
usage during large imports.

In my environment, when I `svn import` an unpacked version of Boost
(https://www.boost.org/) that consists of around 60,000 files, I see that
the memory consumption by the server process rapidly grows up to 1.5 GB.
The environment is a Windows 8.1 x64 machine with httpd 2.4.29 configured
to use short-circuit authz and HTTPS.  Apparently, this behavior is specific
to 1.10, as I cannot reproduce it with 1.9 binaries.

  (I haven't investigated the issue any further at this time, and it might as
   well be reproducible in other configurations.)


Would it be possible for you to bisect this to
find the offending revision?  My random guess
would be that in the context of mod_dav_svn, we
might use an unsuitable pool for authz caching.

-- Stefan^2.

Potential regression: high server-side memory consumption during import (was: Subversion 1.10 RC1?)

[Evgeny Kotkov]

Julian Foad  writes:

> After any issues raised in this discussion are resolved, we feel we should
> go ahead and produce RC1 as soon as possible.

I think that I am seeing a 1.10 regression in terms of httpd's memory
usage during large imports.

In my environment, when I `svn import` an unpacked version of Boost
(https://www.boost.org/) that consists of around 60,000 files, I see that
the memory consumption by the server process rapidly grows up to 1.5 GB.
The environment is a Windows 8.1 x64 machine with httpd 2.4.29 configured
to use short-circuit authz and HTTPS.  Apparently, this behavior is specific
to 1.10, as I cannot reproduce it with 1.9 binaries.

 (I haven't investigated the issue any further at this time, and it might as
  well be reproducible in other configurations.)


Thanks,
Evgeny Kotkov