https://bz.mercurial-scm.org/show_bug.cgi?id=6075
Bug ID: 6075
Summary: explicit GC on every request make repositories with
thousands of largesfiles very slow to clone
Product: Mercurial
Version: 4.9
Hardware: All
OS: Linux
Status: UNCONFIRMED
Severity: bug
Priority: wish
Component: hgweb
Assignee: bugzi...@mercurial-scm.org
Reporter: las...@unity3d.com
CC: mercurial-devel@mercurial-scm.org
Created attachment 2032
--> https://bz.mercurial-scm.org/attachment.cgi?id=2032&action=edit
demo output of lfpull from two hgwebs
in mercurial 4.5.3 this:
====
finally:
# There are known cycles in localrepository that prevent
# those objects (and tons of held references) from being
# collected through normal refcounting. We mitigate those
# leaks by performing an explicit GC on every request.
# TODO remove this once leaks are fixed.
# TODO only run this on requests that create localrepository
# instances instead of every request.
gc.collect()
===
was added to "def run_wsgi(self, req, res):" in mercurial/hgweb/hgwebdir_mod.py
and is still present in latest version. This is a problem for repositories with
thousands of largesfile, as each largefile equals a request on initial clone.
at Unity We e.g. have a repository with ~17000 largefiles, and albeit that is a
pourly provisioned repostory, it takes ~15 minutes to clone from hwgeb running
on mercurial 4.5.2, and 2-4 hours on mercurial 4.5.3 and later which doenst
play well with our CI.
I've made a docker image to debug this issue:
https://github.com/lasse-aagren/hgweb-gc-debug-docker.
If created as hgweb-demo:latest and run as:
docker run --rm -it hgweb-demo:latest /hgweb/lfpull-test.sh
it will spin up 2 hg serve instances, one on 4.5.2 and one on 4.5.3, on top a
demorepo with 1000 4k largefiles. It will then clone 3 consecutive time from
hgweb standalon on 4.5.2 and then 3 times from hgweb standalone on 4.5.3.
Result is attached, but the important points is, all 3 lfpulls from 4.5.2 takes
~60secs each.
With 4.5.3 the first lfpull of the 1000 files takes ~60sec, the next one takes
~120 secs and the third ~180 secs. And it just get worse from there.
If I use mercurial 4.9 with the "gc.collect()" replaced by a "pass", it
performs as with version 4.5.2 and before.
Is my oberservations correct, and can something be done to mitigate it?
Please dont hesitate to ask me for further information.
Best regards,
Lasse Aagren
--
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
