D5305: py3: fix couple of division operator to do integer divison
This revision was automatically updated to reflect the committed changes. Closed by commit rHGe0b485a76009: py3: fix couple of division operator to do integer divison (authored by pulkit, committed by ). REPOSITORY rHG Mercurial CHANGES SINCE LAST UPDATE https://phab.mercurial-scm.org/D5305?vs=12599=12604 REVISION DETAIL https://phab.mercurial-scm.org/D5305 AFFECTED FILES hgext/remotefilelog/basepack.py hgext/remotefilelog/historypack.py CHANGE DETAILS diff --git a/hgext/remotefilelog/historypack.py b/hgext/remotefilelog/historypack.py --- a/hgext/remotefilelog/historypack.py +++ b/hgext/remotefilelog/historypack.py @@ -252,7 +252,7 @@ return self._index[end:end + entrylen] else: while start < end - entrylen: -mid = start + (end - start) / 2 +mid = start + (end - start) // 2 mid = mid - ((mid - origstart) % entrylen) midnode = self._index[mid:mid + NODELENGTH] if midnode == node: diff --git a/hgext/remotefilelog/basepack.py b/hgext/remotefilelog/basepack.py --- a/hgext/remotefilelog/basepack.py +++ b/hgext/remotefilelog/basepack.py @@ -45,7 +45,7 @@ # bisect) with (8 step fanout scan + 1 step bisect) # 5 step bisect = log(2^16 / 8 / 255) # fanout # 10 step fanout scan = 2^16 / (2^16 / 8) # fanout space divided by entries -SMALLFANOUTCUTOFF = 2**16 / 8 +SMALLFANOUTCUTOFF = 2**16 // 8 # The amount of time to wait between checking for new packs. This prevents an # exception when data is moved to a new pack after the process has already To: pulkit, #hg-reviewers Cc: mercurial-devel ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D5301: store: append to fncache if there are only new files to write
This revision was automatically updated to reflect the committed changes.
Closed by commit rHG0728d87a8631: store: append to fncache if there are only
new files to write (authored by pulkit, committed by ).
REPOSITORY
rHG Mercurial
CHANGES SINCE LAST UPDATE
https://phab.mercurial-scm.org/D5301?vs=12595=12605
REVISION DETAIL
https://phab.mercurial-scm.org/D5301
AFFECTED FILES
mercurial/store.py
CHANGE DETAILS
diff --git a/mercurial/store.py b/mercurial/store.py
--- a/mercurial/store.py
+++ b/mercurial/store.py
@@ -451,6 +451,8 @@
self.vfs = vfs
self.entries = None
self._dirty = False
+# set of new additions to fncache
+self.addls = set()
def _load(self):
'''fill the entries from the fncache file'''
@@ -479,32 +481,45 @@
fp.write(encodedir('\n'.join(self.entries) + '\n'))
fp.close()
self._dirty = False
+if self.addls:
+# if we have just new entries, let's append them to the fncache
+tr.addbackup('fncache')
+fp = self.vfs('fncache', mode='ab', atomictemp=True)
+if self.addls:
+fp.write(encodedir('\n'.join(self.addls) + '\n'))
+fp.close()
+self.entries = None
+self.addls = set()
def add(self, fn):
if self.entries is None:
self._load()
if fn not in self.entries:
-self._dirty = True
-self.entries.add(fn)
+self.addls.add(fn)
def remove(self, fn):
if self.entries is None:
self._load()
+if fn in self.addls:
+self.addls.remove(fn)
+return
try:
self.entries.remove(fn)
self._dirty = True
except KeyError:
pass
def __contains__(self, fn):
+if fn in self.addls:
+return True
if self.entries is None:
self._load()
return fn in self.entries
def __iter__(self):
if self.entries is None:
self._load()
-return iter(self.entries)
+return iter(self.entries | self.addls)
class _fncachevfs(vfsmod.abstractvfs, vfsmod.proxyvfs):
def __init__(self, vfs, fnc, encode):
@@ -580,6 +595,7 @@
def invalidatecaches(self):
self.fncache.entries = None
+self.fncache.addls = set()
def markremoved(self, fn):
self.fncache.remove(fn)
To: pulkit, #hg-reviewers
Cc: mjpieters, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D5304: py3: use dict.items() instead of dict.iteritems()
This revision was automatically updated to reflect the committed changes. Closed by commit rHG5d1550b9a533: py3: use dict.items() instead of dict.iteritems() (authored by pulkit, committed by ). REPOSITORY rHG Mercurial CHANGES SINCE LAST UPDATE https://phab.mercurial-scm.org/D5304?vs=12598=12603 REVISION DETAIL https://phab.mercurial-scm.org/D5304 AFFECTED FILES tests/test-remotefilelog-histpack.py CHANGE DETAILS diff --git a/tests/test-remotefilelog-histpack.py b/tests/test-remotefilelog-histpack.py --- a/tests/test-remotefilelog-histpack.py +++ b/tests/test-remotefilelog-histpack.py @@ -161,11 +161,11 @@ pack = self.createPack(revisions) # Verify the pack contents -for (filename, node), (p1, p2, lastnode) in allentries.iteritems(): +for (filename, node), (p1, p2, lastnode) in allentries.items(): ancestors = pack.getancestors(filename, node) self.assertEquals(ancestorcounts[(filename, node)], len(ancestors)) -for anode, (ap1, ap2, alinknode, copyfrom) in ancestors.iteritems(): +for anode, (ap1, ap2, alinknode, copyfrom) in ancestors.items(): ep1, ep2, elinknode = allentries[(filename, anode)] self.assertEquals(ap1, ep1) self.assertEquals(ap2, ep2) To: pulkit, #hg-reviewers Cc: mercurial-devel ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D5303: py3: convert strings to bytes in tests/test-remotefilelog-histpack.py
This revision was automatically updated to reflect the committed changes.
Closed by commit rHG9446d5aa0f32: py3: convert strings to bytes in
tests/test-remotefilelog-histpack.py (authored by pulkit, committed by ).
CHANGED PRIOR TO COMMIT
https://phab.mercurial-scm.org/D5303?vs=12597=12602#toc
REPOSITORY
rHG Mercurial
CHANGES SINCE LAST UPDATE
https://phab.mercurial-scm.org/D5303?vs=12597=12602
REVISION DETAIL
https://phab.mercurial-scm.org/D5303
AFFECTED FILES
tests/test-remotefilelog-histpack.py
CHANGE DETAILS
diff --git a/tests/test-remotefilelog-histpack.py
b/tests/test-remotefilelog-histpack.py
--- a/tests/test-remotefilelog-histpack.py
+++ b/tests/test-remotefilelog-histpack.py
@@ -15,6 +15,7 @@
from mercurial.node import nullid
from mercurial import (
+pycompat,
ui as uimod,
)
# Load the local remotefilelog, not the system one
@@ -35,13 +36,14 @@
def makeTempDir(self):
tempdir = tempfile.mkdtemp()
self.tempdirs.append(tempdir)
-return tempdir
+return pycompat.fsencode(tempdir)
def getHash(self, content):
return hashlib.sha1(content).digest()
def getFakeHash(self):
-return ''.join(chr(random.randint(0, 255)) for _ in range(20))
+return b''.join(pycompat.bytechr(random.randint(0, 255))
+for _ in range(20))
def createPack(self, revisions=None):
"""Creates and returns a historypack containing the specified
revisions.
@@ -53,7 +55,7 @@
revisions = [("filename", self.getFakeHash(), nullid, nullid,
self.getFakeHash(), None)]
-packdir = self.makeTempDir()
+packdir = pycompat.fsencode(self.makeTempDir())
packer = historypack.mutablehistorypack(uimod.ui(), packdir,
version=2)
@@ -107,7 +109,7 @@
chain.
"""
revisions = []
-filename = "foo"
+filename = b"foo"
lastnode = nullid
for i in range(10):
node = self.getFakeHash()
@@ -136,7 +138,7 @@
revisions = []
random.seed(0)
for i in range(100):
-filename = "filename-%s" % i
+filename = b"filename-%d" % i
entries = []
p2 = nullid
linknode = nullid
@@ -172,7 +174,7 @@
def testGetNodeInfo(self):
revisions = []
-filename = "foo"
+filename = b"foo"
lastnode = nullid
for i in range(10):
node = self.getFakeHash()
@@ -193,7 +195,7 @@
"""Test the getmissing() api.
"""
revisions = []
-filename = "foo"
+filename = b"foo"
for i in range(10):
node = self.getFakeHash()
p1 = self.getFakeHash()
@@ -223,7 +225,7 @@
pack = self.createPack()
try:
-pack.add('filename', nullid, nullid, nullid, nullid, None)
+pack.add(b'filename', nullid, nullid, nullid, nullid, None)
self.assertTrue(False, "historypack.add should throw")
except RuntimeError:
pass
@@ -250,7 +252,7 @@
total = basepack.SMALLFANOUTCUTOFF + 1
revisions = []
for i in xrange(total):
-filename = "foo-%s" % i
+filename = b"foo-%d" % i
node = self.getFakeHash()
p1 = self.getFakeHash()
p2 = self.getFakeHash()
To: pulkit, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D5300: py3: replace str() with pycompat.bytestr() or ('%d' % int)
This revision was automatically updated to reflect the committed changes.
Closed by commit rHG001f27970b60: py3: replace str() with pycompat.bytestr() or
(%d % int) (authored by pulkit, committed by ).
REPOSITORY
rHG Mercurial
CHANGES SINCE LAST UPDATE
https://phab.mercurial-scm.org/D5300?vs=12594=12601
REVISION DETAIL
https://phab.mercurial-scm.org/D5300
AFFECTED FILES
hgext/fastannotate/formatter.py
CHANGE DETAILS
diff --git a/hgext/fastannotate/formatter.py b/hgext/fastannotate/formatter.py
--- a/hgext/fastannotate/formatter.py
+++ b/hgext/fastannotate/formatter.py
@@ -39,23 +39,26 @@
orig = hexfunc
hexfunc = lambda x: None if x is None else orig(x)
wnode = hexfunc(repo[None].p1().node()) + '+'
-wrev = str(repo[None].p1().rev())
+wrev = '%d' % repo[None].p1().rev()
wrevpad = ''
if not opts.get('changeset'): # only show + if changeset is hidden
wrev += '+'
wrevpad = ' '
-revenc = lambda x: wrev if x is None else str(x) + wrevpad
-csetenc = lambda x: wnode if x is None else str(x) + ' '
+revenc = lambda x: wrev if x is None else ('%d' % x) + wrevpad
+def csetenc(x):
+if x is None:
+return wnode
+return pycompat.bytestr(x) + ' '
else:
-revenc = csetenc = str
+revenc = csetenc = pycompat.bytestr
# opt name, separator, raw value (for json/plain), encoder (for plain)
opmap = [('user', ' ', lambda x: getctx(x).user(), ui.shortuser),
('number', ' ', lambda x: getctx(x).rev(), revenc),
('changeset', ' ', lambda x: hexfunc(x[0]), csetenc),
('date', ' ', lambda x: getctx(x).date(), datefunc),
- ('file', ' ', lambda x: x[2], str),
- ('line_number', ':', lambda x: x[1] + 1, str)]
+ ('file', ' ', lambda x: x[2], pycompat.bytestr),
+ ('line_number', ':', lambda x: x[1] + 1, pycompat.bytestr)]
fieldnamemap = {'number': 'rev', 'changeset': 'node'}
funcmap = [(get, sep, fieldnamemap.get(op, op), enc)
for op, sep, get, enc in opmap
To: pulkit, #hg-reviewers
Cc: yuja, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D5302: py3: alias xrange to range
durin42 added inline comments. INLINE COMMENTS > test-remotefilelog-histpack.py:29 > +if pycompat.ispy3: > +xrange = range > + why not just use pycompat.xrange directly instead of doing extra indirection? REPOSITORY rHG Mercurial REVISION DETAIL https://phab.mercurial-scm.org/D5302 To: pulkit, #hg-reviewers Cc: durin42, mjpieters, mercurial-devel ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D5306: remotefilelog: fix typo in docstring
durin42 created this revision. Herald added a subscriber: mercurial-devel. Herald added a reviewer: hg-reviewers. REPOSITORY rHG Mercurial REVISION DETAIL https://phab.mercurial-scm.org/D5306 AFFECTED FILES hgext/remotefilelog/__init__.py CHANGE DETAILS diff --git a/hgext/remotefilelog/__init__.py b/hgext/remotefilelog/__init__.py --- a/hgext/remotefilelog/__init__.py +++ b/hgext/remotefilelog/__init__.py @@ -67,7 +67,7 @@ particular, minimum number of packs files > gencountlimit. ``remotefilelog.history.generations`` list for specifying the lower bound of - each generation of the historhy pack files. For example, list [ + each generation of the history pack files. For example, list [ '100MB', '1MB'] or ['1MB', '100MB'] will lead to three generations: [ 0, 1MB), [1MB, 100MB) and [100MB, infinity). To: durin42, #hg-reviewers Cc: mercurial-devel ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 8 of 8 V2] perf: add a `--clear-caches` to `perfbranchmapupdate`
# HG changeset patch # User Boris Feld # Date 1542931777 -3600 # Fri Nov 23 01:09:37 2018 +0100 # Node ID d5e0cfcc409ed9f71a7bd85a711f810151ed8c5e # Parent 41772e57bfc92d38899b24f729d992c39dc32473 # EXP-Topic perf-branchmap # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r d5e0cfcc409e perf: add a `--clear-caches` to `perfbranchmapupdate` This flag will help to measure the time we spend loading various cache that support the branchmap update. Example for an 500 000 revisions repository: hg perfbranchmapupdate --base 'not tip' --target 'tip' ! wall 0.000860 comb 0.00 user 0.00 sys 0.00 (best of 336) hg perfbranchmapupdate --base 'not tip' --target 'tip' --clear-caches ! wall 0.029494 comb 0.03 user 0.03 sys 0.00 (best of 100) diff --git a/contrib/perf.py b/contrib/perf.py --- a/contrib/perf.py +++ b/contrib/perf.py @@ -2234,10 +2234,16 @@ def perfbranchmap(ui, repo, *filternames @command(b'perfbranchmapupdate', [ (b'', b'base', [], b'subset of revision to start from'), (b'', b'target', [], b'subset of revision to end with'), + (b'', b'clear-caches', False, b'clear cache between each runs') ] + formatteropts) def perfbranchmapupdate(ui, repo, base=(), target=(), **opts): """benchmark branchmap update from for revs to revs +if `--clear-caches` is passed, the following items will be reset before +each update: +* the changelog instance and associated indexes +* the rev-branch-cache instance + Examples: # update for the one last revision @@ -2250,6 +2256,7 @@ def perfbranchmapupdate(ui, repo, base=( from mercurial import repoview opts = _byteskwargs(opts) timer, fm = gettimer(ui, opts) +clearcaches = opts['clear_caches'] unfi = repo.unfiltered() x = [None] # used to pass data between closure @@ -2315,6 +2322,9 @@ def perfbranchmapupdate(ui, repo, base=( def setup(): x[0] = base.copy() +if clearcaches: +unfi._revbranchcache = None +clearchangelog(repo) def bench(): x[0].update(targetrepo, newrevs) ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 7 of 8 V2] perf: start from an existing branchmap if possible
# HG changeset patch
# User Boris Feld
# Date 1542834707 0
# Wed Nov 21 21:11:47 2018 +
# Node ID 41772e57bfc92d38899b24f729d992c39dc32473
# Parent 03c3522ed8a678695eba5a0dc7612236fbdfb47b
# EXP-Topic perf-branchmap
# Available At https://bitbucket.org/octobus/mercurial-devel/
# hg pull https://bitbucket.org/octobus/mercurial-devel/ -r
41772e57bfc9
perf: start from an existing branchmap if possible
If the --base set if a superset of one of the cached branchmap, we should use as
a starting point. This greatly help the overall runtime of
`hg perfbranchmapupdate`
For example, for a repository with about 500 000 revisions, using this trick
make the command runtime move from about 200 second to about 10 seconds. A 20x
gain.
diff --git a/contrib/perf.py b/contrib/perf.py
--- a/contrib/perf.py
+++ b/contrib/perf.py
@@ -2296,8 +2296,22 @@ def perfbranchmapupdate(ui, repo, base=(
baserepo = repo.filtered('__perf_branchmap_update_base')
targetrepo = repo.filtered('__perf_branchmap_update_target')
-base = branchmap.branchcache()
-base.update(baserepo, allbaserevs)
+# try to find an existing branchmap to reuse
+subsettable = getbranchmapsubsettable()
+candidatefilter = subsettable.get(None)
+while candidatefilter is not None:
+candidatebm = repo.filtered(candidatefilter).branchmap()
+if candidatebm.validfor(baserepo):
+filtered = repoview.filterrevs(repo, candidatefilter)
+missing = [r for r in allbaserevs if r in filtered]
+base = candidatebm.copy()
+base.update(baserepo, missing)
+break
+candidatefilter = subsettable.get(candidatefilter)
+else:
+# no suitable subset where found
+base = branchmap.branchcache()
+base.update(baserepo, allbaserevs)
def setup():
x[0] = base.copy()
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 6 of 8 V2] perf: rely on repoview for perfbranchmapupdate
# HG changeset patch
# User Boris Feld
# Date 1542832522 0
# Wed Nov 21 20:35:22 2018 +
# Node ID 03c3522ed8a678695eba5a0dc7612236fbdfb47b
# Parent 3a9ca368b2362f2304895a9a238abccdd4aee633
# EXP-Topic perf-branchmap
# Available At https://bitbucket.org/octobus/mercurial-devel/
# hg pull https://bitbucket.org/octobus/mercurial-devel/ -r
03c3522ed8a6
perf: rely on repoview for perfbranchmapupdate
Using 'repoview' matching the base and target subset make the benchmark more
realistic. It also unlocks optimization to make the command initialization
faster.
diff --git a/contrib/perf.py b/contrib/perf.py
--- a/contrib/perf.py
+++ b/contrib/perf.py
@@ -2247,8 +2247,10 @@ def perfbranchmapupdate(ui, repo, base=(
$ hg perfbranchmapupdate --base 'stable' --target 'default'
"""
from mercurial import branchmap
+from mercurial import repoview
opts = _byteskwargs(opts)
timer, fm = gettimer(ui, opts)
+unfi = repo.unfiltered()
x = [None] # used to pass data between closure
# we use a `list` here to avoid possible side effect from smartset
@@ -2271,21 +2273,43 @@ def perfbranchmapupdate(ui, repo, base=(
newrevs = list(alltargetrevs.difference(allbaserevs))
newrevs.sort()
+allrevs = frozenset(unfi.changelog.revs())
+basefilterrevs = frozenset(allrevs.difference(allbaserevs))
+targetfilterrevs = frozenset(allrevs.difference(alltargetrevs))
+
+def basefilter(repo, visibilityexceptions=None):
+return basefilterrevs
+
+def targetfilter(repo, visibilityexceptions=None):
+return targetfilterrevs
+
msg = 'benchmark of branchmap with %d revisions with %d new ones\n'
ui.status(msg % (len(allbaserevs), len(newrevs)))
+if targetfilterrevs:
+msg = '(%d revisions still filtered)\n'
+ui.status(msg % len(targetfilterrevs))
-if True:
+try:
+repoview.filtertable['__perf_branchmap_update_base'] = basefilter
+repoview.filtertable['__perf_branchmap_update_target'] = targetfilter
+
+baserepo = repo.filtered('__perf_branchmap_update_base')
+targetrepo = repo.filtered('__perf_branchmap_update_target')
+
base = branchmap.branchcache()
-base.update(repo, allbaserevs)
+base.update(baserepo, allbaserevs)
def setup():
x[0] = base.copy()
def bench():
-x[0].update(repo, newrevs)
+x[0].update(targetrepo, newrevs)
timer(bench, setup=setup)
fm.end()
+finally:
+repoview.filtertable.pop('__perf_branchmap_update_base', None)
+repoview.filtertable.pop('__perf_branchmap_update_target', None)
@command(b'perfbranchmapload', [
(b'f', b'filter', b'', b'Specify repoview filter'),
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 5 of 8 V2] perf: pre-indent some code in `perfbranchmapupdate`
# HG changeset patch # User Boris Feld # Date 1542837366 -3600 # Wed Nov 21 22:56:06 2018 +0100 # Node ID 3a9ca368b2362f2304895a9a238abccdd4aee633 # Parent dc3191e230daa615929d41cc410bfb16cfe658d9 # EXP-Topic perf-branchmap # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 3a9ca368b236 perf: pre-indent some code in `perfbranchmapupdate` This make the next patch easier to read. diff --git a/contrib/perf.py b/contrib/perf.py --- a/contrib/perf.py +++ b/contrib/perf.py @@ -2274,17 +2274,18 @@ def perfbranchmapupdate(ui, repo, base=( msg = 'benchmark of branchmap with %d revisions with %d new ones\n' ui.status(msg % (len(allbaserevs), len(newrevs))) -base = branchmap.branchcache() -base.update(repo, allbaserevs) - -def setup(): -x[0] = base.copy() +if True: +base = branchmap.branchcache() +base.update(repo, allbaserevs) -def bench(): -x[0].update(repo, newrevs) +def setup(): +x[0] = base.copy() -timer(bench, setup=setup) -fm.end() +def bench(): +x[0].update(repo, newrevs) + +timer(bench, setup=setup) +fm.end() @command(b'perfbranchmapload', [ (b'f', b'filter', b'', b'Specify repoview filter'), ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 4 of 8 V2] perf: add a `perfbranchmapupdate` command
# HG changeset patch
# User Boris Feld
# Date 1542801745 0
# Wed Nov 21 12:02:25 2018 +
# Node ID dc3191e230daa615929d41cc410bfb16cfe658d9
# Parent 0adc2c0a0792d4a0015ec0634487ed9e08fd0e7e
# EXP-Topic perf-branchmap
# Available At https://bitbucket.org/octobus/mercurial-devel/
# hg pull https://bitbucket.org/octobus/mercurial-devel/ -r
dc3191e230da
perf: add a `perfbranchmapupdate` command
This command benchmark the time necessary to update the branchmap between two
sets of revisions. This changeset introduce a first version, doing nothing fancy
regarding cache or other internal details.
diff --git a/contrib/perf.py b/contrib/perf.py
--- a/contrib/perf.py
+++ b/contrib/perf.py
@@ -2231,6 +2231,61 @@ def perfbranchmap(ui, repo, *filternames
branchcachewrite.restore()
fm.end()
+@command(b'perfbranchmapupdate', [
+ (b'', b'base', [], b'subset of revision to start from'),
+ (b'', b'target', [], b'subset of revision to end with'),
+] + formatteropts)
+def perfbranchmapupdate(ui, repo, base=(), target=(), **opts):
+"""benchmark branchmap update from for revs to revs
+
+Examples:
+
+ # update for the one last revision
+ $ hg perfbranchmapupdate --base 'not tip' --target 'tip'
+
+ $ update for change coming with a new branch
+ $ hg perfbranchmapupdate --base 'stable' --target 'default'
+"""
+from mercurial import branchmap
+opts = _byteskwargs(opts)
+timer, fm = gettimer(ui, opts)
+x = [None] # used to pass data between closure
+
+# we use a `list` here to avoid possible side effect from smartset
+baserevs = list(scmutil.revrange(repo, base))
+targetrevs = list(scmutil.revrange(repo, target))
+if not baserevs:
+raise error.Abort('no revisions selected for --base')
+if not targetrevs:
+raise error.Abort('no revisions selected for --target')
+
+# make sure the target branchmap also contains the one in the base
+targetrevs = list(set(baserevs) | set(targetrevs))
+targetrevs.sort()
+
+cl = repo.changelog
+allbaserevs = list(cl.ancestors(baserevs, inclusive=True))
+allbaserevs.sort()
+alltargetrevs = frozenset(cl.ancestors(targetrevs, inclusive=True))
+
+newrevs = list(alltargetrevs.difference(allbaserevs))
+newrevs.sort()
+
+msg = 'benchmark of branchmap with %d revisions with %d new ones\n'
+ui.status(msg % (len(allbaserevs), len(newrevs)))
+
+base = branchmap.branchcache()
+base.update(repo, allbaserevs)
+
+def setup():
+x[0] = base.copy()
+
+def bench():
+x[0].update(repo, newrevs)
+
+timer(bench, setup=setup)
+fm.end()
+
@command(b'perfbranchmapload', [
(b'f', b'filter', b'', b'Specify repoview filter'),
(b'', b'list', False, b'List brachmap filter caches'),
diff --git a/tests/test-contrib-perf.t b/tests/test-contrib-perf.t
--- a/tests/test-contrib-perf.t
+++ b/tests/test-contrib-perf.t
@@ -57,6 +57,9 @@ perfstatus
benchmark the update of a branchmap
perfbranchmapload
benchmark reading the branchmap
+ perfbranchmapupdate
+ benchmark branchmap update from for revs to
+ revs
perfbundleread
Benchmark reading of bundle files.
perfcca (no help text available)
@@ -144,6 +147,8 @@ perfstatus
$ hg perfbookmarks
$ hg perfbranchmap
$ hg perfbranchmapload
+ $ hg perfbranchmapupdate --base "not tip" --target "tip"
+ benchmark of branchmap with 3 revisions with 1 new ones
$ hg perfcca
$ hg perfchangegroupchangelog
$ hg perfchangegroupchangelog --cgversion 01
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 3 of 8 V2] perf: run 'setup' function during stub run
# HG changeset patch # User Boris Feld # Date 1542926924 -3600 # Thu Nov 22 23:48:44 2018 +0100 # Node ID 0adc2c0a0792d4a0015ec0634487ed9e08fd0e7e # Parent 0a3cc351d7189969b98c7f926b408837683f003b # EXP-Topic perf-branchmap # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 0adc2c0a0792 perf: run 'setup' function during stub run The benchmarked function might need the content of the setup to be run in order to function properly. diff --git a/contrib/perf.py b/contrib/perf.py --- a/contrib/perf.py +++ b/contrib/perf.py @@ -276,6 +276,8 @@ def gettimer(ui, opts=None): return functools.partial(_timer, fm, displayall=displayall), fm def stub_timer(fm, func, setup=None, title=None): +if setup is not None: +setup() func() @contextlib.contextmanager ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 2 of 8 V2] perf: fallback to subset if ondisk cache is missing in perfbranchmapload
# HG changeset patch # User Boris Feld # Date 1542935281 -3600 # Fri Nov 23 02:08:01 2018 +0100 # Node ID 0a3cc351d7189969b98c7f926b408837683f003b # Parent 578646b1e2b685f8d499e38ee210d3ec5847a550 # EXP-Topic perf-branchmap # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 0a3cc351d718 perf: fallback to subset if ondisk cache is missing in perfbranchmapload If there is no branchmap on disk for that filter, it means that the cache from some subset's filter is relevant for this one. We look for it instead of aborting. That way it is much simpler to run the command in an automated way. We can now add it to `test-contrib-perf.t`. diff --git a/contrib/perf.py b/contrib/perf.py --- a/contrib/perf.py +++ b/contrib/perf.py @@ -2247,17 +2247,24 @@ def perfbranchmapload(ui, repo, filter=b ui.status(b'%s - %s\n' % (filtername, util.bytecount(st.st_size))) return -if filter: +if not filter: +filter = None +subsettable = getbranchmapsubsettable() +if filter is None: +repo = repo.unfiltered() +else: repo = repoview.repoview(repo, filter) -else: -repo = repo.unfiltered() repo.branchmap() # make sure we have a relevant, up to date branchmap +currentfilter = filter # try once without timer, the filter may not be cached -if branchmap.read(repo) is None: -raise error.Abort(b'No branchmap cached for %s repo' - % (filter or b'unfiltered')) +while branchmap.read(repo) is None: +currentfilter = subsettable.get(currentfilter) +if currentfilter is None: +raise error.Abort(b'No branchmap cached for %s repo' + % (filter or b'unfiltered')) +repo = repo.filtered(currentfilter) timer, fm = gettimer(ui, opts) def setup(): if clearrevlogs: diff --git a/tests/test-contrib-perf.t b/tests/test-contrib-perf.t --- a/tests/test-contrib-perf.t +++ b/tests/test-contrib-perf.t @@ -143,6 +143,7 @@ perfstatus $ hg perfunidiff --alldata 1 $ hg perfbookmarks $ hg perfbranchmap + $ hg perfbranchmapload $ hg perfcca $ hg perfchangegroupchangelog $ hg perfchangegroupchangelog --cgversion 01 ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 1 of 8 V2] perf: prewarm the branchmap in perfbranchmapload
# HG changeset patch # User Boris Feld # Date 1542935471 -3600 # Fri Nov 23 02:11:11 2018 +0100 # Node ID 578646b1e2b685f8d499e38ee210d3ec5847a550 # Parent 5bcf264bb1a0b1f57f01b4d20f28d9db9a74f47b # EXP-Topic perf-branchmap # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 578646b1e2b6 perf: prewarm the branchmap in perfbranchmapload It is not very interesting to have the command randomly failing because the branchmap for the tested filter happens to be cold. So we make sure to have a valid up to date branchmap before going further. The data might still be missing from disk if a subset was equivalent. See next changeset for details and fix. diff --git a/contrib/perf.py b/contrib/perf.py --- a/contrib/perf.py +++ b/contrib/perf.py @@ -2251,6 +2251,9 @@ def perfbranchmapload(ui, repo, filter=b repo = repoview.repoview(repo, filter) else: repo = repo.unfiltered() + +repo.branchmap() # make sure we have a relevant, up to date branchmap + # try once without timer, the filter may not be cached if branchmap.read(repo) is None: raise error.Abort(b'No branchmap cached for %s repo' ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH] perf: add a `--timing` argument to `perfhelper-tracecopies`
# HG changeset patch
# User Boris Feld
# Date 1542997971 -3600
# Fri Nov 23 19:32:51 2018 +0100
# Node ID eb3bef1d74e864bf0436a355b0f4859def00ffc2
# Parent cfaf3843491bde30916b89180d5f67561a22da28
# EXP-Topic perf-copies
# Available At https://bitbucket.org/octobus/mercurial-devel/
# hg pull https://bitbucket.org/octobus/mercurial-devel/ -r
eb3bef1d74e8
perf: add a `--timing` argument to `perfhelper-tracecopies`
The new argument will help picking better pair for benchmarking. See
documentation for details.
diff --git a/contrib/perf.py b/contrib/perf.py
--- a/contrib/perf.py
+++ b/contrib/perf.py
@@ -1162,19 +1162,39 @@ def perftemplating(ui, repo, testedtempl
@command(b'perfhelper-tracecopies', formatteropts +
[
(b'r', b'revs', [], b'restrict search to these revisions'),
+ (b'', b'timing', False, b'provides extra data (costly)'),
])
def perfhelpertracecopies(ui, repo, revs=[], **opts):
"""find statistic about potential parameters for the `perftracecopies`
This command find source-destination pair relevant for copytracing testing.
It report value for some of the parameters that impact copy tracing time.
+
+If `--timing` is set, rename detection is run and the associated timing
+will be reported. The extra details comes at the cost of a slower command
+execution.
+
+Since the rename detection is only run once, other factors might easily
+affect the precision of the timing. However it should give a good
+approximation of which revision pairs are very costly.
"""
opts = _byteskwargs(opts)
fm = ui.formatter(b'perf', opts)
-header = '%12s %12s %12s %12s\n'
-output = ("%(source)12s %(destination)12s "
- "%(nbrevs)12d %(nbmissingfiles)12d\n")
-fm.plain(header % ("source", "destination", "nb-revs", "nb-files"))
+dotiming = opts[b'timing']
+
+if dotiming:
+header = '%12s %12s %12s %12s %12s %12s\n'
+output = ("%(source)12s %(destination)12s "
+ "%(nbrevs)12d %(nbmissingfiles)12d "
+ "%(nbrenamedfiles)12d %(time)18.5f\n")
+header_names = ("source", "destination", "nb-revs", "nb-files",
+"nb-renames", "time")
+fm.plain(header % header_names)
+else:
+header = '%12s %12s %12s %12s\n'
+output = ("%(source)12s %(destination)12s "
+ "%(nbrevs)12d %(nbmissingfiles)12d\n")
+fm.plain(header % ("source", "destination", "nb-revs", "nb-files"))
if not revs:
revs = ['all()']
@@ -1193,18 +1213,26 @@ def perfhelpertracecopies(ui, repo, revs
missing = copies._computeforwardmissing(base, parent)
if not missing:
continue
-fm.startitem()
data = {
b'source': base.hex(),
b'destination': parent.hex(),
b'nbrevs': len(repo.revs('%d::%d', b, p)),
b'nbmissingfiles': len(missing),
}
+if dotiming:
+begin = util.timer()
+renames = copies.pathcopies(base, parent)
+end = util.timer()
+# not very stable timing since we did only one run
+data['time'] = end - begin
+data['nbrenamedfiles'] = len(renames)
+fm.startitem()
fm.data(**data)
out = data.copy()
out['source'] = fm.hexfunc(base.node())
out['destination'] = fm.hexfunc(parent.node())
fm.plain(output % out)
+
fm.end()
@command(b'perfcca', formatteropts)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 5 of 5 FOLLOW-UP] revlog: update the documentation for `trim_endidx`
# HG changeset patch
# User Boris Feld
# Date 1543188192 -3600
# Mon Nov 26 00:23:12 2018 +0100
# Node ID 498fc4b8da0cf147315b63b4580ef9b6a36e3911
# Parent 7dc2fade8ee58c0554b6f25484a4481ac625ce84
# EXP-Topic follow-up-yuya
# Available At https://bitbucket.org/octobus/mercurial-devel/
# hg pull https://bitbucket.org/octobus/mercurial-devel/ -r
498fc4b8da0c
revlog: update the documentation for `trim_endidx`
The function role drifted since the function was commented.
diff --git a/mercurial/cext/revlog.c b/mercurial/cext/revlog.c
--- a/mercurial/cext/revlog.c
+++ b/mercurial/cext/revlog.c
@@ -1086,7 +1086,7 @@ index_segment_span(indexObject *self, Py
return (end_offset - start_offset) + (int64_t)end_size;
}
-/* returns revs[startidx:endidx] without empty trailing revs */
+/* returns endidx so that revs[startidx:endidx] has no empty trailing revs */
static Py_ssize_t trim_endidx(indexObject *self, const Py_ssize_t *revs,
Py_ssize_t startidx, Py_ssize_t endidx)
{
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 4 of 5 FOLLOW-UP] revlog: properly detect corrupted revlog in `index_get_length`
# HG changeset patch
# User Boris Feld
# Date 1543188069 -3600
# Mon Nov 26 00:21:09 2018 +0100
# Node ID 7dc2fade8ee58c0554b6f25484a4481ac625ce84
# Parent 9e936fdbd084acef9ca4edf998024896a30a0937
# EXP-Topic follow-up-yuya
# Available At https://bitbucket.org/octobus/mercurial-devel/
# hg pull https://bitbucket.org/octobus/mercurial-devel/ -r
7dc2fade8ee5
revlog: properly detect corrupted revlog in `index_get_length`
Pointed out by Yuya Nishihara.
diff --git a/mercurial/cext/revlog.c b/mercurial/cext/revlog.c
--- a/mercurial/cext/revlog.c
+++ b/mercurial/cext/revlog.c
@@ -242,7 +242,14 @@ static inline int index_get_length(index
return (int)ret;
} else {
const char *data = index_deref(self, rev);
- return (int)getbe32(data + 8);
+ int tmp = (int)getbe32(data + 8);
+ if (tmp < 0) {
+ PyErr_Format(PyExc_OverflowError,
+"revlog entry size out of bound (%d)",
+tmp);
+ return -1;
+ }
+ return tmp;
}
}
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 3 of 5 FOLLOW-UP] perf: rename `perfhelper-tracecopies` to `perfhelper-pathcopies`
# HG changeset patch # User Boris Feld # Date 1543187712 -3600 # Mon Nov 26 00:15:12 2018 +0100 # Node ID 9e936fdbd084acef9ca4edf998024896a30a0937 # Parent 47117f84f39cb5bc12212ba9c31212c9236feb85 # EXP-Topic follow-up-yuya # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 9e936fdbd084 perf: rename `perfhelper-tracecopies` to `perfhelper-pathcopies` The command it supports is called `perfpathcopies`. It seems better to align the names. diff --git a/contrib/perf.py b/contrib/perf.py --- a/contrib/perf.py +++ b/contrib/perf.py @@ -1160,11 +1160,11 @@ def perftemplating(ui, repo, testedtempl timer(format) fm.end() -@command(b'perfhelper-tracecopies', formatteropts + +@command(b'perfhelper-pathcopies', formatteropts + [ (b'r', b'revs', [], b'restrict search to these revisions'), ]) -def perfhelpertracecopies(ui, repo, revs=[], **opts): +def perfhelperpathcopies(ui, repo, revs=[], **opts): """find statistic about potential parameters for the `perftracecopies` This command find source-destination pair relevant for copytracing testing. diff --git a/tests/test-contrib-perf.t b/tests/test-contrib-perf.t --- a/tests/test-contrib-perf.t +++ b/tests/test-contrib-perf.t @@ -83,7 +83,7 @@ perfstatus perffncachewrite (no help text available) perfheads (no help text available) - perfhelper-tracecopies + perfhelper-pathcopies find statistic about potential parameters for the 'perftracecopies' perfindex (no help text available) ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 2 of 5 FOLLOW-UP] perf: add a docstring to `perfpathcopies`
# HG changeset patch # User Boris Feld # Date 1543187630 -3600 # Mon Nov 26 00:13:50 2018 +0100 # Node ID 47117f84f39cb5bc12212ba9c31212c9236feb85 # Parent 87443a7ac7f6a37ed59d6429c87118c64b7287f6 # EXP-Topic follow-up-yuya # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 47117f84f39c perf: add a docstring to `perfpathcopies` This will help people to find this command. diff --git a/contrib/perf.py b/contrib/perf.py --- a/contrib/perf.py +++ b/contrib/perf.py @@ -834,6 +834,7 @@ def perfmergecalculate(ui, repo, rev, ** @command(b'perfpathcopies', [], b"REV REV") def perfpathcopies(ui, repo, rev1, rev2, **opts): +"""benchmark the copy tracing logic""" opts = _byteskwargs(opts) timer, fm = gettimer(ui, opts) ctx1 = scmutil.revsingle(repo, rev1, rev1) diff --git a/tests/test-contrib-perf.t b/tests/test-contrib-perf.t --- a/tests/test-contrib-perf.t +++ b/tests/test-contrib-perf.t @@ -104,7 +104,7 @@ perfstatus (no help text available) perfparents (no help text available) perfpathcopies - (no help text available) + benchmark the copy tracing logic perfphasesbenchmark phasesets computation perfphasesremote benchmark time needed to analyse phases of the remote server ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 1 of 5 FOLLOW-UP] revlog: update the docstring of `ancestors` to match reality
# HG changeset patch # User Boris Feld # Date 1543187291 -3600 # Mon Nov 26 00:08:11 2018 +0100 # Node ID 87443a7ac7f6a37ed59d6429c87118c64b7287f6 # Parent cfaf3843491bde30916b89180d5f67561a22da28 # EXP-Topic follow-up-yuya # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 87443a7ac7f6 revlog: update the docstring of `ancestors` to match reality Code using this method expect the revision to be (reverse) sorted. As pointed by Yuya Nishihara, the docstring should reflect that. diff --git a/mercurial/revlog.py b/mercurial/revlog.py --- a/mercurial/revlog.py +++ b/mercurial/revlog.py @@ -768,7 +768,7 @@ class revlog(object): return chain, stopped def ancestors(self, revs, stoprev=0, inclusive=False): -"""Generate the ancestors of 'revs' in reverse topological order. +"""Generate the ancestors of 'revs' in reverse revision order. Does not generate revs lower than stoprev. See the documentation for ancestor.lazyancestors for more details.""" ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 5 of 5 V3] perf: disable revlogs clearing in `perftags` by default
# HG changeset patch # User Boris Feld # Date 1542735864 0 # Tue Nov 20 17:44:24 2018 + # Node ID 400476885997b972298b537b5d694ba9500f3f90 # Parent 4394597460a13bcd6fc2b4d88826f9399ac27de0 # EXP-Topic perf-tags # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 400476885997 perf: disable revlogs clearing in `perftags` by default This aligns things with what `perfbookmarks` does. I decided to disable the revlogs clearing by default to focus on the core logic by default, ignoring side effects. If we prefer to emphasize the side effect, we can instead keep this on in `perftags` and enable it by default in `perfbookmarks`. diff --git a/contrib/perf.py b/contrib/perf.py --- a/contrib/perf.py +++ b/contrib/perf.py @@ -538,7 +538,7 @@ def perfheads(ui, repo, **opts): @command(b'perftags', formatteropts+ [ -(b'', b'clear-revlogs', True, 'refresh changelog and manifest'), +(b'', b'clear-revlogs', False, 'refresh changelog and manifest'), ]) def perftags(ui, repo, **opts): import mercurial.changelog ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 4 of 5 V3] perf: add a `clear-revlogs` flag to `perfbookmarks`
# HG changeset patch # User Boris Feld # Date 1542711320 0 # Tue Nov 20 10:55:20 2018 + # Node ID 4394597460a13bcd6fc2b4d88826f9399ac27de0 # Parent 568c6a0abb650b9c0949e2231f0d8ff51cf40ece # EXP-Topic perf-tags # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 4394597460a1 perf: add a `clear-revlogs` flag to `perfbookmarks` This flag (off by default) makes it possible to enable the refresh of the changelog and revlog. This is useful to check for costly side effects of bookmark loading. Usually, these side effects are shared with other logics (eg: tags). example output in my mercurial repo (with 1 bookmark, so not a great example): $ hg perfbookmarks ! wall 0.44 $ hg perfbookmarks --clear-revlogs ! wall 0.001380 diff --git a/contrib/perf.py b/contrib/perf.py --- a/contrib/perf.py +++ b/contrib/perf.py @@ -583,13 +583,19 @@ def perfancestorset(ui, repo, revset, ** timer(d) fm.end() -@command(b'perfbookmarks', formatteropts) +@command(b'perfbookmarks', formatteropts + +[ +(b'', b'clear-revlogs', False, 'refresh changelog and manifest'), +]) def perfbookmarks(ui, repo, **opts): """benchmark parsing bookmarks from disk to memory""" opts = _byteskwargs(opts) timer, fm = gettimer(ui, opts) +clearrevlogs = opts['clear_revlogs'] def s(): +if clearrevlogs: +clearchangelog(repo) clearfilecache(repo, b'_bookmarks') def d(): repo._bookmarks ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 3 of 5 V3] tags: cache `repo.changelog` access when checking tags nodes
# HG changeset patch # User Boris Feld # Date 1542710295 0 # Tue Nov 20 10:38:15 2018 + # Node ID 568c6a0abb650b9c0949e2231f0d8ff51cf40ece # Parent e5c18b7087cdeac2a1213c3499e9a8215b252af6 # EXP-Topic perf-tags # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 568c6a0abb65 tags: cache `repo.changelog` access when checking tags nodes The tags reading process checks if the nodes referenced in tags exist. Caching the access to `repo.changelog` provides a large speedup for repositories with many tags. running `hg perftags` in a large private repository before: ! wall 0.393464 comb 0.39 user 0.33 sys 0.06 (median of 25) after: ! wall 0.267711 comb 0.27 user 0.21 sys 0.06 (median of 38) diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py --- a/mercurial/localrepo.py +++ b/mercurial/localrepo.py @@ -1416,10 +1416,11 @@ class localrepository(object): tags, tt = self._findtags() else: tags = self._tagscache.tags +rev = self.changelog.rev for k, v in tags.iteritems(): try: # ignore tags to unknown nodes -self.changelog.rev(v) +rev(v) t[k] = v except (error.LookupError, ValueError): pass ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 2 of 5 V3] perf: add a `clear-revlogs` flag to `perftags`
# HG changeset patch # User Boris Feld # Date 1542710780 0 # Tue Nov 20 10:46:20 2018 + # Node ID e5c18b7087cdeac2a1213c3499e9a8215b252af6 # Parent a2a4aabea58bef816eeb53e80ddaf0d438717c66 # EXP-Topic perf-tags # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r e5c18b7087cd perf: add a `clear-revlogs` flag to `perftags` This flag (on by default) makes it possible to disable the refresh of the changelog and revlog. This is useful to check for the time spent in the core tags logic without the associated side effects. Usually, these side effects are shared with other logics (eg: bookmarks). Example output in my Mercurial repository $ hg perftags ! wall 0.017919 comb 0.02 user 0.02 sys 0.00 (best of 141) $ hg perftags --no-clear-revlogs ! wall 0.012982 comb 0.01 user 0.01 sys 0.00 (best of 207) diff --git a/contrib/perf.py b/contrib/perf.py --- a/contrib/perf.py +++ b/contrib/perf.py @@ -536,7 +536,10 @@ def perfheads(ui, repo, **opts): timer(d) fm.end() -@command(b'perftags', formatteropts) +@command(b'perftags', formatteropts+ +[ +(b'', b'clear-revlogs', True, 'refresh changelog and manifest'), +]) def perftags(ui, repo, **opts): import mercurial.changelog import mercurial.manifest @@ -545,9 +548,11 @@ def perftags(ui, repo, **opts): timer, fm = gettimer(ui, opts) svfs = getsvfs(repo) repocleartagscache = repocleartagscachefunc(repo) +clearrevlogs = opts['clear_revlogs'] def s(): -clearchangelog(repo) -clearfilecache(repo.unfiltered(), 'manifest') +if clearrevlogs: +clearchangelog(repo) +clearfilecache(repo.unfiltered(), 'manifest') repocleartagscache() def t(): return len(repo.tags()) ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 1 of 5 V3] perf: stop creating new revlog by hand in perftags
# HG changeset patch # User Boris Feld # Date 1543149473 -3600 # Sun Nov 25 13:37:53 2018 +0100 # Node ID a2a4aabea58bef816eeb53e80ddaf0d438717c66 # Parent cfaf3843491bde30916b89180d5f67561a22da28 # EXP-Topic perf-tags # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r a2a4aabea58b perf: stop creating new revlog by hand in perftags It i better to let the repository logic create its own object. We now just clear the cache. New object will be automatically created from there. diff --git a/contrib/perf.py b/contrib/perf.py --- a/contrib/perf.py +++ b/contrib/perf.py @@ -546,10 +546,8 @@ def perftags(ui, repo, **opts): svfs = getsvfs(repo) repocleartagscache = repocleartagscachefunc(repo) def s(): -repo.changelog = mercurial.changelog.changelog(svfs) -rootmanifest = mercurial.manifest.manifestrevlog(svfs) -repo.manifestlog = mercurial.manifest.manifestlog(svfs, repo, - rootmanifest) +clearchangelog(repo) +clearfilecache(repo.unfiltered(), 'manifest') repocleartagscache() def t(): return len(repo.tags()) ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 6 of 6] vfs: also audit rename
# HG changeset patch # User Boris Feld # Date 1498961267 -7200 # Sun Jul 02 04:07:47 2017 +0200 # Node ID afdc73b20bd1faeec1b278ef0a2ab8d1dda71eb8 # Parent cc7d970d99eb242dbe2d8e792a9212aabd06911f # EXP-Topic vfs.audit-rename # Available At https://bitbucket.org/octobus/mercurial-devel/ # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r afdc73b20bd1 vfs: also audit rename Renaming through the vfs is not used in many places, and none of them seems to be at security risk. However it is still worthwhile to run the auditing on rename file to perform developer-warning level checks. diff --git a/mercurial/vfs.py b/mercurial/vfs.py --- a/mercurial/vfs.py +++ b/mercurial/vfs.py @@ -436,6 +436,10 @@ class vfs(abstractvfs): return fp +def rename(self, src, dst, *args, **kwargs): +self._auditpath(dst, 'w') +return super(vfs, self).rename(src, dst, *args, **kwargs) + def symlink(self, src, dst): self.audit(dst) linkname = self.join(dst) ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 5 of 6] vfs: makes all audited path relative
# HG changeset patch
# User Boris Feld
# Date 1498963863 -7200
# Sun Jul 02 04:51:03 2017 +0200
# Node ID cc7d970d99eb242dbe2d8e792a9212aabd06911f
# Parent 7f04e7e8eea72eb588d9460df7c0f2b72941b760
# EXP-Topic vfs.audit-rename
# Available At https://bitbucket.org/octobus/mercurial-devel/
# hg pull https://bitbucket.org/octobus/mercurial-devel/ -r
cc7d970d99eb
vfs: makes all audited path relative
Only auditing relative path helps the vfs warning logic.
diff --git a/mercurial/vfs.py b/mercurial/vfs.py
--- a/mercurial/vfs.py
+++ b/mercurial/vfs.py
@@ -339,6 +339,8 @@ class vfs(abstractvfs):
def _auditpath(self, path, mode):
if self._audit:
+if os.path.isabs(path) and path.startswith(self.base):
+path = os.path.relpath(path, self.base)
r = util.checkosfilename(path)
if r:
raise error.Abort("%s: %r" % (r, path))
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 4 of 6] vfs: extract the audit path logic into a submethod
# HG changeset patch
# User Boris Feld
# Date 1498961184 -7200
# Sun Jul 02 04:06:24 2017 +0200
# Node ID 7f04e7e8eea72eb588d9460df7c0f2b72941b760
# Parent 3d8d2de85c9afc06b7aad740aa1ad2c817d18dfa
# EXP-Topic vfs.audit-rename
# Available At https://bitbucket.org/octobus/mercurial-devel/
# hg pull https://bitbucket.org/octobus/mercurial-devel/ -r
7f04e7e8eea7
vfs: extract the audit path logic into a submethod
This will make it possible to apply it in more cases.
diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -1029,12 +1029,12 @@ class localrepository(object):
path = path[len(repo.path) + 1:]
if path.startswith('cache/'):
msg = 'accessing cache with vfs instead of cachevfs: "%s"'
-repo.ui.develwarn(msg % path, stacklevel=2, config="cache-vfs")
+repo.ui.develwarn(msg % path, stacklevel=3, config="cache-vfs")
if path.startswith('journal.') or path.startswith('undo.'):
# journal is covered by 'lock'
if repo._currentlock(repo._lockref) is None:
repo.ui.develwarn('write with no lock: "%s"' % path,
- stacklevel=2, config='check-locks')
+ stacklevel=3, config='check-locks')
elif repo._currentlock(repo._wlockref) is None:
# rest of vfs files are covered by 'wlock'
#
@@ -1043,7 +1043,7 @@ class localrepository(object):
if path.startswith(prefix):
return
repo.ui.develwarn('write with no wlock: "%s"' % path,
- stacklevel=2, config='check-locks')
+ stacklevel=3, config='check-locks')
return ret
return checkvfs
@@ -1062,7 +1062,7 @@ class localrepository(object):
path = path[len(repo.sharedpath) + 1:]
if repo._currentlock(repo._lockref) is None:
repo.ui.develwarn('write with no lock: "%s"' % path,
- stacklevel=3)
+ stacklevel=4)
return ret
return checksvfs
diff --git a/mercurial/vfs.py b/mercurial/vfs.py
--- a/mercurial/vfs.py
+++ b/mercurial/vfs.py
@@ -337,6 +337,13 @@ class vfs(abstractvfs):
return
os.chmod(name, self.createmode & 0o666)
+def _auditpath(self, path, mode):
+if self._audit:
+r = util.checkosfilename(path)
+if r:
+raise error.Abort("%s: %r" % (r, path))
+self.audit(path, mode=mode)
+
def __call__(self, path, mode="r", atomictemp=False, notindexed=False,
backgroundclose=False, checkambig=False, auditpath=True):
'''Open ``path`` file, which is relative to vfs root.
@@ -369,11 +376,7 @@ class vfs(abstractvfs):
cases (see also issue5418 and issue5584 for detail).
'''
if auditpath:
-if self._audit:
-r = util.checkosfilename(path)
-if r:
-raise error.Abort("%s: %r" % (r, path))
-self.audit(path, mode=mode)
+self._auditpath(path, mode)
f = self.join(path)
if "b" not in mode:
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 3 of 6] subrepo-git: use an official origvfs when appropriate
# HG changeset patch
# User Boris Feld
# Date 1542913288 -3600
# Thu Nov 22 20:01:28 2018 +0100
# Node ID 3d8d2de85c9afc06b7aad740aa1ad2c817d18dfa
# Parent a83de6825f6aac64efc059e6224225b073d441f4
# EXP-Topic vfs.audit-rename
# Available At https://bitbucket.org/octobus/mercurial-devel/
# hg pull https://bitbucket.org/octobus/mercurial-devel/ -r
3d8d2de85c9a
subrepo-git: use an official origvfs when appropriate
The origvfs has the auditor properly set and can move file without issue.
The current code is currently working without errors because rename are not
audited, yet.
diff --git a/mercurial/subrepo.py b/mercurial/subrepo.py
--- a/mercurial/subrepo.py
+++ b/mercurial/subrepo.py
@@ -1806,11 +1806,15 @@ class gitsubrepo(abstractsubrepo):
if not opts.get(r'no_backup'):
status = self.status(None)
names = status.modified
+origvfs = scmutil.getorigvfs(self.ui, self._subparent)
+if origvfs is None:
+origvfs = self.wvfs
for name in names:
bakname = scmutil.origpath(self.ui, self._subparent, name)
self.ui.note(_('saving current version of %s as %s\n') %
(name, bakname))
-self.wvfs.rename(name, bakname)
+name = self.wvfs.join(name)
+origvfs.rename(name, bakname)
if not opts.get(r'dry_run'):
self.get(substate, overwrite=True)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 1 of 6] vfs: treat 'undo.' file the same as 'journal.' file
# HG changeset patch
# User Boris Feld
# Date 1542908647 -3600
# Thu Nov 22 18:44:07 2018 +0100
# Node ID 5bcdd426a817c333ca30dac9a02ec100f962155e
# Parent dba590f27c7abacbd7e9b27f3e06822bb0b339cb
# EXP-Topic vfs.audit-rename
# Available At https://bitbucket.org/octobus/mercurial-devel/
# hg pull https://bitbucket.org/octobus/mercurial-devel/ -r
5bcdd426a817
vfs: treat 'undo.' file the same as 'journal.' file
They are the same kind of file, they are protected by the store lock, but
directly lives inside the '.hg' directory.
No warnings were ever raised about them because `vfs.rename` is not audited.
Something we are trying to change.
diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -1030,7 +1030,7 @@ class localrepository(object):
if path.startswith('cache/'):
msg = 'accessing cache with vfs instead of cachevfs: "%s"'
repo.ui.develwarn(msg % path, stacklevel=2, config="cache-vfs")
-if path.startswith('journal.'):
+if path.startswith('journal.') or path.startswith('undo.'):
# journal is covered by 'lock'
if repo._currentlock(repo._lockref) is None:
repo.ui.develwarn('write with no lock: "%s"' % path,
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
[PATCH 2 of 6] revert: extract origvfs logic in a sub-function
# HG changeset patch
# User Boris Feld
# Date 1542911165 -3600
# Thu Nov 22 19:26:05 2018 +0100
# Node ID a83de6825f6aac64efc059e6224225b073d441f4
# Parent 5bcdd426a817c333ca30dac9a02ec100f962155e
# EXP-Topic vfs.audit-rename
# Available At https://bitbucket.org/octobus/mercurial-devel/
# hg pull https://bitbucket.org/octobus/mercurial-devel/ -r
a83de6825f6a
revert: extract origvfs logic in a sub-function
The subrepo's "revert" logic could benefit from it.
diff --git a/mercurial/scmutil.py b/mercurial/scmutil.py
--- a/mercurial/scmutil.py
+++ b/mercurial/scmutil.py
@@ -814,21 +814,29 @@ def parsefollowlinespattern(repo, rev, p
raise error.ParseError(msg)
return files[0]
+def getorigvfs(ui, repo):
+"""return a vfs suitable to save 'orig' file
+
+return None if no special directory is configured"""
+origbackuppath = ui.config('ui', 'origbackuppath')
+if not origbackuppath:
+return None
+return vfs.vfs(repo.wvfs.join(origbackuppath))
+
def origpath(ui, repo, filepath):
'''customize where .orig files are created
Fetch user defined path from config file: [ui] origbackuppath =
Fall back to default (filepath with .orig suffix) if not specified
'''
-origbackuppath = ui.config('ui', 'origbackuppath')
-if not origbackuppath:
+origvfs = getorigvfs(ui, repo)
+if origvfs is None:
return filepath + ".orig"
# Convert filepath from an absolute path into a path inside the repo.
filepathfromroot = util.normpath(os.path.relpath(filepath,
start=repo.root))
-origvfs = vfs.vfs(repo.wjoin(origbackuppath))
origbackupdir = origvfs.dirname(filepathfromroot)
if not origvfs.isdir(origbackupdir) or origvfs.islink(origbackupdir):
ui.note(_('creating directory: %s\n') % origvfs.join(origbackupdir))
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D5305: py3: fix couple of division operator to do integer divison
pulkit created this revision. Herald added a subscriber: mercurial-devel. Herald added a reviewer: hg-reviewers. REPOSITORY rHG Mercurial REVISION DETAIL https://phab.mercurial-scm.org/D5305 AFFECTED FILES hgext/remotefilelog/basepack.py hgext/remotefilelog/historypack.py CHANGE DETAILS diff --git a/hgext/remotefilelog/historypack.py b/hgext/remotefilelog/historypack.py --- a/hgext/remotefilelog/historypack.py +++ b/hgext/remotefilelog/historypack.py @@ -252,7 +252,7 @@ return self._index[end:end + entrylen] else: while start < end - entrylen: -mid = start + (end - start) / 2 +mid = start + (end - start) // 2 mid = mid - ((mid - origstart) % entrylen) midnode = self._index[mid:mid + NODELENGTH] if midnode == node: diff --git a/hgext/remotefilelog/basepack.py b/hgext/remotefilelog/basepack.py --- a/hgext/remotefilelog/basepack.py +++ b/hgext/remotefilelog/basepack.py @@ -45,7 +45,7 @@ # bisect) with (8 step fanout scan + 1 step bisect) # 5 step bisect = log(2^16 / 8 / 255) # fanout # 10 step fanout scan = 2^16 / (2^16 / 8) # fanout space divided by entries -SMALLFANOUTCUTOFF = 2**16 / 8 +SMALLFANOUTCUTOFF = 2**16 // 8 # The amount of time to wait between checking for new packs. This prevents an # exception when data is moved to a new pack after the process has already To: pulkit, #hg-reviewers Cc: mercurial-devel ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D5304: py3: use dict.items() instead of dict.iteritems()
pulkit created this revision. Herald added a subscriber: mercurial-devel. Herald added a reviewer: hg-reviewers. REVISION SUMMARY dict.iteritems() does not exist on Python 3. REPOSITORY rHG Mercurial REVISION DETAIL https://phab.mercurial-scm.org/D5304 AFFECTED FILES tests/test-remotefilelog-histpack.py CHANGE DETAILS diff --git a/tests/test-remotefilelog-histpack.py b/tests/test-remotefilelog-histpack.py --- a/tests/test-remotefilelog-histpack.py +++ b/tests/test-remotefilelog-histpack.py @@ -164,11 +164,11 @@ pack = self.createPack(revisions) # Verify the pack contents -for (filename, node), (p1, p2, lastnode) in allentries.iteritems(): +for (filename, node), (p1, p2, lastnode) in allentries.items(): ancestors = pack.getancestors(filename, node) self.assertEquals(ancestorcounts[(filename, node)], len(ancestors)) -for anode, (ap1, ap2, alinknode, copyfrom) in ancestors.iteritems(): +for anode, (ap1, ap2, alinknode, copyfrom) in ancestors.items(): ep1, ep2, elinknode = allentries[(filename, anode)] self.assertEquals(ap1, ep1) self.assertEquals(ap2, ep2) To: pulkit, #hg-reviewers Cc: mercurial-devel ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D5303: py3: convert strings to bytes in tests/test-remotefilelog-histpack.py
pulkit created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.
REPOSITORY
rHG Mercurial
REVISION DETAIL
https://phab.mercurial-scm.org/D5303
AFFECTED FILES
tests/test-remotefilelog-histpack.py
CHANGE DETAILS
diff --git a/tests/test-remotefilelog-histpack.py
b/tests/test-remotefilelog-histpack.py
--- a/tests/test-remotefilelog-histpack.py
+++ b/tests/test-remotefilelog-histpack.py
@@ -39,13 +39,14 @@
def makeTempDir(self):
tempdir = tempfile.mkdtemp()
self.tempdirs.append(tempdir)
-return tempdir
+return pycompat.fsencode(tempdir)
def getHash(self, content):
return hashlib.sha1(content).digest()
def getFakeHash(self):
-return ''.join(chr(random.randint(0, 255)) for _ in range(20))
+return b''.join(pycompat.bytechr(random.randint(0, 255))
+for _ in range(20))
def createPack(self, revisions=None):
"""Creates and returns a historypack containing the specified
revisions.
@@ -57,7 +58,7 @@
revisions = [("filename", self.getFakeHash(), nullid, nullid,
self.getFakeHash(), None)]
-packdir = self.makeTempDir()
+packdir = pycompat.fsencode(self.makeTempDir())
packer = historypack.mutablehistorypack(uimod.ui(), packdir,
version=2)
@@ -111,7 +112,7 @@
chain.
"""
revisions = []
-filename = "foo"
+filename = b"foo"
lastnode = nullid
for i in range(10):
node = self.getFakeHash()
@@ -140,7 +141,7 @@
revisions = []
random.seed(0)
for i in range(100):
-filename = "filename-%s" % i
+filename = b"filename-%d" % i
entries = []
p2 = nullid
linknode = nullid
@@ -176,7 +177,7 @@
def testGetNodeInfo(self):
revisions = []
-filename = "foo"
+filename = b"foo"
lastnode = nullid
for i in range(10):
node = self.getFakeHash()
@@ -197,7 +198,7 @@
"""Test the getmissing() api.
"""
revisions = []
-filename = "foo"
+filename = b"foo"
for i in range(10):
node = self.getFakeHash()
p1 = self.getFakeHash()
@@ -227,7 +228,7 @@
pack = self.createPack()
try:
-pack.add('filename', nullid, nullid, nullid, nullid, None)
+pack.add(b'filename', nullid, nullid, nullid, nullid, None)
self.assertTrue(False, "historypack.add should throw")
except RuntimeError:
pass
@@ -254,7 +255,7 @@
total = basepack.SMALLFANOUTCUTOFF + 1
revisions = []
for i in xrange(total):
-filename = "foo-%s" % i
+filename = b"foo-%d" % i
node = self.getFakeHash()
p1 = self.getFakeHash()
p2 = self.getFakeHash()
To: pulkit, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D5302: py3: alias xrange to range
pulkit created this revision. Herald added subscribers: mercurial-devel, mjpieters. Herald added a reviewer: hg-reviewers. REVISION SUMMARY xrange does not exists on Python 3. REPOSITORY rHG Mercurial REVISION DETAIL https://phab.mercurial-scm.org/D5302 AFFECTED FILES tests/test-remotefilelog-histpack.py CHANGE DETAILS diff --git a/tests/test-remotefilelog-histpack.py b/tests/test-remotefilelog-histpack.py --- a/tests/test-remotefilelog-histpack.py +++ b/tests/test-remotefilelog-histpack.py @@ -15,6 +15,7 @@ from mercurial.node import nullid from mercurial import ( +pycompat, ui as uimod, ) # Load the local remotefilelog, not the system one @@ -24,6 +25,9 @@ historypack, ) +if pycompat.ispy3: +xrange = range + class histpacktests(unittest.TestCase): def setUp(self): self.tempdirs = [] To: pulkit, #hg-reviewers Cc: mjpieters, mercurial-devel ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D5301: store: append to fncache if there are only new files to write
pulkit created this revision.
Herald added subscribers: mercurial-devel, mjpieters.
Herald added a reviewer: hg-reviewers.
REVISION SUMMARY
Before this patch, if we have to add a new entry to fncache, we write the
whole
fncache again which slows things down on large fncache which have millions of
entries. Addition of a new entry is common operation while pulling new files
or
commiting a new file.
This patch adds a new fncache.addls set which keeps track of the additions
happening and store them. When we write the fncache, we will just read the
addls
set and append those entries at the end of fncache.
We make sure that the entries are new entries by loading the fncache and
making
sure entry does not exists there. In future if we can check if an entry is new
without loading the fncache, that will speed up things more.
Performance numbers for commiting a new file:
mercurial repo
before: 0.08784651756286621
after: 0.08474504947662354
mozilla-central
before: 1.83314049243927
after: 1.7054164409637451
netbeans
before: 0.7953150272369385
after: 0.7202838659286499
pypy
before: 0.17805707454681396
after: 0.13431048393249512
In our internal repo, the performance improvement is in seconds.
I have used octobus's ASV perf benchmark thing to get the above numbers. I
also
see some minute perf improvements related to creating a new commit without a
new
file, but I believe that's just some noise.
REPOSITORY
rHG Mercurial
REVISION DETAIL
https://phab.mercurial-scm.org/D5301
AFFECTED FILES
mercurial/store.py
CHANGE DETAILS
diff --git a/mercurial/store.py b/mercurial/store.py
--- a/mercurial/store.py
+++ b/mercurial/store.py
@@ -451,6 +451,8 @@
self.vfs = vfs
self.entries = None
self._dirty = False
+# set of new additions to fncache
+self.addls = set()
def _load(self):
'''fill the entries from the fncache file'''
@@ -479,32 +481,45 @@
fp.write(encodedir('\n'.join(self.entries) + '\n'))
fp.close()
self._dirty = False
+if self.addls:
+# if we have just new entries, let's append them to the fncache
+tr.addbackup('fncache')
+fp = self.vfs('fncache', mode='ab', atomictemp=True)
+if self.addls:
+fp.write(encodedir('\n'.join(self.addls) + '\n'))
+fp.close()
+self.entries = None
+self.addls = set()
def add(self, fn):
if self.entries is None:
self._load()
if fn not in self.entries:
-self._dirty = True
-self.entries.add(fn)
+self.addls.add(fn)
def remove(self, fn):
if self.entries is None:
self._load()
+if fn in self.addls:
+self.addls.remove(fn)
+return
try:
self.entries.remove(fn)
self._dirty = True
except KeyError:
pass
def __contains__(self, fn):
+if fn in self.addls:
+return True
if self.entries is None:
self._load()
return fn in self.entries
def __iter__(self):
if self.entries is None:
self._load()
-return iter(self.entries)
+return iter(self.entries | self.addls)
class _fncachevfs(vfsmod.abstractvfs, vfsmod.proxyvfs):
def __init__(self, vfs, fnc, encode):
@@ -580,6 +595,7 @@
def invalidatecaches(self):
self.fncache.entries = None
+self.fncache.addls = set()
def markremoved(self, fn):
self.fncache.remove(fn)
To: pulkit, #hg-reviewers
Cc: mjpieters, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D5300: py3: replace str() with pycompat.bytestr() or ('%d' % int)
pulkit updated this revision to Diff 12594.
REPOSITORY
rHG Mercurial
CHANGES SINCE LAST UPDATE
https://phab.mercurial-scm.org/D5300?vs=12591=12594
REVISION DETAIL
https://phab.mercurial-scm.org/D5300
AFFECTED FILES
hgext/fastannotate/formatter.py
CHANGE DETAILS
diff --git a/hgext/fastannotate/formatter.py b/hgext/fastannotate/formatter.py
--- a/hgext/fastannotate/formatter.py
+++ b/hgext/fastannotate/formatter.py
@@ -39,23 +39,26 @@
orig = hexfunc
hexfunc = lambda x: None if x is None else orig(x)
wnode = hexfunc(repo[None].p1().node()) + '+'
-wrev = str(repo[None].p1().rev())
+wrev = '%d' % repo[None].p1().rev()
wrevpad = ''
if not opts.get('changeset'): # only show + if changeset is hidden
wrev += '+'
wrevpad = ' '
-revenc = lambda x: wrev if x is None else str(x) + wrevpad
-csetenc = lambda x: wnode if x is None else str(x) + ' '
+revenc = lambda x: wrev if x is None else ('%d' % x) + wrevpad
+def csetenc(x):
+if x is None:
+return wnode
+return pycompat.bytestr(x) + ' '
else:
-revenc = csetenc = str
+revenc = csetenc = pycompat.bytestr
# opt name, separator, raw value (for json/plain), encoder (for plain)
opmap = [('user', ' ', lambda x: getctx(x).user(), ui.shortuser),
('number', ' ', lambda x: getctx(x).rev(), revenc),
('changeset', ' ', lambda x: hexfunc(x[0]), csetenc),
('date', ' ', lambda x: getctx(x).date(), datefunc),
- ('file', ' ', lambda x: x[2], str),
- ('line_number', ':', lambda x: x[1] + 1, str)]
+ ('file', ' ', lambda x: x[2], pycompat.bytestr),
+ ('line_number', ':', lambda x: x[1] + 1, pycompat.bytestr)]
fieldnamemap = {'number': 'rev', 'changeset': 'node'}
funcmap = [(get, sep, fieldnamemap.get(op, op), enc)
for op, sep, get, enc in opmap
To: pulkit, #hg-reviewers
Cc: yuja, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D5300: py3: replace str() with pycompat.bytestr() or ('%d' % int)
pulkit added a comment.
In https://phab.mercurial-scm.org/D5300#78881, @yuja wrote:
> > - revenc = lambda x: wrev if x is None else str(x) + wrevpad
> > - csetenc = lambda x: wnode if x is None else str(x) + ' ' +
revenc = lambda x: wrev if x is None else ('%d' % x) + wrevpad +
csetenc = lambda x: wnode if x is None else pycompat.bytestr(x) + ' '
>
> check-code complains that the line is too long. Maybe rewrite as a function?
Ah, looks like I forgot to run tests. Sorry about that.
REPOSITORY
rHG Mercurial
REVISION DETAIL
https://phab.mercurial-scm.org/D5300
To: pulkit, #hg-reviewers
Cc: yuja, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Re: [PATCH] tests: disable remotefilelog on Windows
On Sat, 24 Nov 2018 23:17:26 -0500, Matt Harbison wrote: > # HG changeset patch > # User Matt Harbison > # Date 1543086662 18000 > # Sat Nov 24 14:11:02 2018 -0500 > # Node ID 99e50fb3ffc6fa3d5e81427c69e5280c84254ab0 > # Parent 1f9de5636e5f7f4bfe2d3fb8c5dde543a1870161 > tests: disable remotefilelog on Windows > > I've spent a non trivial amount of time trying to eliminate the test errors, > but > it's looking like this is pretty dependent on Unix support. For example, > there > are attempts to delete open files, and uses of threads that report I/O > attempts > on closed files. (Maybe this is a race condition? Don't we usually use > processes as workers on Windows?) > > In any event, I don't want real new errors elsewhere to be masked by these > known > problems. > > For some reason $CACHEDIR is reported as missing in > test-remotefilelog-repack.t, > but it actually exists in the hgcloneshallow call inside > shallowutil.mkstickygroupdir(). By the time the process exits, it's gone. I > don't see it being removed by code that calls 'rmdir' or 'remove' in the > extension itself. Queued, thanks. Should we make remotefilelog.uisetup() abort if enabled on Windows? ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Re: [PATCH 06 of 13] perf: prewarm the branchmap in perfbranchmapload
On Fri, Nov 23, 2018 at 8:21 PM Boris FELD wrote: > On 23/11/2018 15:49, Pulkit Goyal wrote: > > > > On Fri, Nov 23, 2018 at 5:19 PM Boris Feld wrote: > >> # HG changeset patch >> # User Boris Feld >> # Date 1542935471 -3600 >> # Fri Nov 23 02:11:11 2018 +0100 >> # Node ID 9f543638d909768a0db0aa779d37817c4b8878ab >> # Parent e72da9d014ba91ee4f2fe620a9646404a64d7484 >> # EXP-Topic perf-branchmap >> # Available At https://bitbucket.org/octobus/mercurial-devel/ >> # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r >> 9f543638d909 >> perf: prewarm the branchmap in perfbranchmapload >> >> It is not very interesting to have the command randomly failing because >> the >> branchmap for the tested filter happens to be cold. So we make sure to >> have a >> valid up to date branchmap before going further. >> >> The data might still be missing from disk if a subset was equivalent. See >> next >> changeset for details and fix. >> >> diff --git a/contrib/perf.py b/contrib/perf.py >> --- a/contrib/perf.py >> +++ b/contrib/perf.py >> @@ -2203,6 +2203,9 @@ def perfbranchmapload(ui, repo, filter=b >> repo = repoview.repoview(repo, filter) >> else: >> repo = repo.unfiltered() >> + >> +repo.branchmap # make sure we have a relevant, up to date branchmap >> > > Do you want to call this function? According to my understanding it won't > load the branchmap or do something useful. > > Ho yeah, good catch. > Okay, appended '()' in flight and queued 6-8, many thanks! > > ___ > Mercurial-devel mailing > listMercurial-devel@mercurial-scm.orghttps://www.mercurial-scm.org/mailman/listinfo/mercurial-devel > > ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Re: [PATCH] localrepo: correct docstring of filectx()
On Sun, Nov 25, 2018 at 4:51 PM Yuya Nishihara wrote: > # HG changeset patch > # User Yuya Nishihara > # Date 1543153194 -32400 > # Sun Nov 25 22:39:54 2018 +0900 > # Node ID 3cba13b2e62284f3bd4ed3219864b7418a99f596 > # Parent cfaf3843491bde30916b89180d5f67561a22da28 > localrepo: correct docstring of filectx() > > The same reason as b6c2543e1dd8. It can't be any changeset specifiers but > revision number. > Queued this, many thanks! ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D5296: store: don't read the whole fncache in memory
pulkit added a comment.
In https://phab.mercurial-scm.org/D5296#78834, @yuja wrote:
> > diff --git a/mercurial/store.py b/mercurial/store.py
> >
> > - a/mercurial/store.py +++ b/mercurial/store.py @@ -461,13 +461,13 @@
> > 1. skip nonexistent file self.entries = set() return
> > - self.entries = set(decodedir(fp.read()).splitlines())
> > - if '' in self.entries:
> > - fp.seek(0)
> > - for n, line in enumerate(util.iterfile(fp)):
> > - if not line.rstrip('\n'):
> > - t = _('invalid entry in fncache, line %d') % (n + 1)
> > - raise error.Abort(t) +self.entries = set() +for n, line
in enumerate(util.iterfile(fp)): +entry = line.rstrip('\n') +
if not entry: +t = _('invalid entry in fncache, line %d')
% (n + 1) +raise error.Abort(t) +
self.entries.add(decodedir(entry))
>
> This goes the opposite direction to
https://phab.mercurial-scm.org/rHG9fca5b056c0a2f673aefa64f7ec7488bd9188d9d. I
guess the current code
> would be faster if the fncache is around ~10MB. Can you benchmark it?
Yeah the current code is much faster if fncache is not large.
REPOSITORY
rHG Mercurial
REVISION DETAIL
https://phab.mercurial-scm.org/D5296
To: pulkit, #hg-reviewers
Cc: yuja, mjpieters, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Re: [PATCH 4 of 4] annotate: remove dead code to not convert path to relative path
On Sat, Nov 24, 2018 at 4:40 AM Yuya Nishihara wrote: > # HG changeset patch > # User Yuya Nishihara > # Date 1543057555 -32400 > # Sat Nov 24 20:05:55 2018 +0900 > # Node ID 1aac31914fee54803cd5c206f0dddcc18f6120e2 > # Parent 3ba66c3809b43da8216aea05b0cef15bf3a53109 > annotate: remove dead code to not convert path to relative path > Queued, thanks. ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
