Re: Next plans on radixlink and hash-preserving obsstore

[Sean Farley]

Jun Wu  writes:

> hash-preserving obsstore,

I've commented on this before but I will try to be as explicit as
possible here.

I do not want to think about hash-preserving obsstore right now.

I do not want to think about it for the next year even.

I want obs-exchange to solved before hash-preserving.

I want all the UI/UX of evolve to be solve and in core before we even
*contemplate* (the over-engineered) hash-preserving.

I don't know how to be more clear with you while still being polite.


signature.asc
Description: PGP signature
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 4 of 4] [RFC] color: support different styling depending on color support (BC)

[Jun Wu]

I think the series is a good improvement and can be queued with some small
in-flight fixes.

Excerpts from Gregory Szorc's message of 2017-07-09 16:46:17 -0700:
> diff --git a/mercurial/color.py b/mercurial/color.py
> --- a/mercurial/color.py
> +++ b/mercurial/color.py
> @@ -135,13 +135,31 @@ except ImportError:
>  'diff.inserted': 'green',
>  'diff.tab': '',
>  'diff.trailingwhitespace': 'bold red_background',
> -'changeset.public' : '',
> -'changeset.draft' : '',
> -'changeset.secret' : '',
> +'changeset.public': {
> +8: '',
> +16: 'brightred',

This is mostly nitpicking, but red seems to be mostly about errors or
warnings. I'd vote "brightgreen" for public changesets. Or make public
brightyellow and draft non-bright yellow.

> [...]

Not only in colortable, it seems config support is also worthwhile, like:

  [color]
  changeset.public:8 = ...
  changeset.public:16 = ...

That could be a future patch.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 3 of 4] color: automatically define 16 and 256 colors if supported

[Jun Wu]

Excerpts from Gregory Szorc's message of 2017-07-09 16:46:16 -0700:
> [...]
> @@ -146,6 +179,43 @@ def _terminfocolors(ui):
>  for color, value in TERMINFO_COLOR_8.items():
>  colors[color] = (value, False)
>  
> +# Add 16 and 256 bit colors if supported and allowed by config policy.
> +#
> +# There's no reliable way to detect if the extra colors will actually 
> work.
> +# The terminfo database just reports what the current terminal is
> +# advertising. Some terminals support querying the value of a color via
> +# e.g. \e]4;%d;?\a. However, this isn't universally supported. So set
> +# colors based purely on terminfo and provide users a way to disable in
> +# case it doesn't work.
> +
> +# Will likely return -1 on failure.
> +termcolors = curses.tigetnum('colors')

Should we check if curses is None here? It seems "debugcolor" will call this
and could be problematic on Windows.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: Transient Windows test failures

[Matt Harbison]

On Mon, 19 Jun 2017 11:30:28 -0400, Yuya Nishihara  wrote:


On Sun, 18 Jun 2017 22:19:29 -0400, Augie Fackler wrote:


> On Jun 16, 2017, at 22:02, Matt Harbison   
wrote:

>
> On Fri, 16 Jun 2017 09:59:30 -0400, Augie Fackler   
wrote:

>
>> On Fri, Jun 16, 2017 at 12:18:18AM -0400, Matt Harbison wrote:
>>> So apparently, this is a symptom of not having %SystemRoot% in the
>>> environment when calling CreateProcess().
>>>
>>> https://bugs.python.org/issue13524
>>>  
https://jpassing.com/2009/12/28/the-hidden-danger-of-forgetting-to-specify-systemroot-in-a-custom-environment-block/

>>>
>>> I see that setup.py special cases this variable.  I did a search  
for 'env
>>> =', and it looks like hooks and pager start with empty  
environments, so they
>>> must not inherit this.  IDR if any recent changes were made that  
start with

>>> an empty environment.
>>>
>>> The thing I can't get my mind around is the hit and miss nature of  
the

>>> error, if this is really the problem.
>>
>> It sounds like it should be harmless to just always forward
>> %SystemRoot% - should we just do that?
>
> Seems reasonable, but run-tests._getenv() already does an  
os.environ.copy(), so it should be there?

>
> It does seem like a good idea to do it for hooks and other things  
executed, where the environment is built from scratch.  The question is  
where?  There's util.popen[2-4](), plus some direct calls to  
subprocess.Popen(), and an os.system().  I considered  
util.shellenviron(), but there are far fewer calls to this than places  
where processes are spawned.


(+CC foozy since he has Windows)

Is the problem only seen in tests? I don't think environment variables  
are

cleared in hg side.


I hit this problem again this weekend, after it was quiet for the past  
couple of weeks.  It looks like it might be an issue with nearly running  
low on memory.


When it happened this time, Windows popped up a dialog box saying memory  
was running low, offering to kill some programs.  I had TaskManager open,  
and saw the Performance > System > Commit (MB) line was running around  
5400/6076.  I closed thg, which exited thg.exe and hg.exe (listed at ~300  
MB each in the process list), and the issue stopped.  I was able to  
recreate it again after a day of quiet by opening up a bunch of tabs in  
FireFox, and pushing the memory usage around that threshold.  I tend to  
run the tests with -j9, and I've seen the first number bounce around  
between 4900 and 5700+ during these failures.  So I'm not sure what the  
exact problem threshold is, as tests start and exit.  Interestingly, the  
free memory number in Resource Monitor at the same time indicates only  
150MB-20MB free.


One of the "optimizations" of the SSD install software was to cap the page  
file, which is probably why I hadn't seen this until recently.  Kostia had  
mentioned to me that he was seeing errors saying “application failed to  
start 0xc142”, which I also saw (along with dialog box failures of  
various msys executables, like env.exe and grep.exe).  So maybe this is  
useful to others wanting to run Windows tests.  It seems unlikely that it  
would be seen in the wild (the page file usually isn't capped), and I  
doubt there's anything we can do about it anyway.

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 6 of 6] rebase: remove revprecursor and revpruned states

[Jun Wu]

This should be retitled to "remove nullmerge and revignored states".

(it seems Phabricator could be helpful in this situation - the title could
be changed without a resend)

Excerpts from Jun Wu's message of 2017-07-09 12:14:53 -0700:
> # HG changeset patch
> # User Jun Wu 
> # Date 1499571514 25200
> #  Sat Jul 08 20:38:34 2017 -0700
> # Node ID 22d54f1e4c09d8fb68ea897afcdd0b705db94b5a
> # Parent  49d1672e7567f2a16c5ce9afa64babe08891718f
> # Available At https://bitbucket.org/quark-zju/hg-draft 
> #  hg pull https://bitbucket.org/quark-zju/hg-draft  -r 
> 22d54f1e4c09
> rebase: remove revprecursor and revpruned states
> 
> They are no longer necessary to make rebase behavior correct. Therefore
> remove them to make the code cleaner and easier to reason about.
> 
> diff --git a/hgext/rebase.py b/hgext/rebase.py
> --- a/hgext/rebase.py
> +++ b/hgext/rebase.py
> @@ -61,10 +61,8 @@ templateopts = cmdutil.templateopts
>  # Indicates that a revision needs to be rebased
>  revtodo = -1
> -nullmerge = -2
> -revignored = -3
>  
>  # legacy revstates no longer needed in current code
> -# -4: revprecursor, -5: revpruned
> -legacystates = {'-4', '-5'}
> +# -2: nullmerge, -3: revignored, -4: revprecursor, -5: revpruned
> +legacystates = {'-2', '-3', '-4', '-5'}
>  
>  cmdtable = {}
> @@ -234,6 +232,4 @@ class rebaseruntime(object):
>  if newrev in legacystates:
>  continue
> -if newrev in (str(nullmerge), str(revignored)):
> -state[repo[oldrev].rev()] = int(newrev)
>  elif newrev == nullid:
>  state[repo[oldrev].rev()] = revtodo
> @@ -440,8 +436,4 @@ class rebaseruntime(object):
>  self.state[rev] = p1
>  ui.debug('next revision set to %s\n' % p1)
> -elif self.state[rev] == nullmerge:
> -pass
> -elif self.state[rev] == revignored:
> -pass
>  else:
>  ui.status(_('already rebased %s as %s\n') %
> @@ -464,5 +456,5 @@ class rebaseruntime(object):
>  for rebased in self.state:
>  if rebased not in self.skipped and\
> -   self.state[rebased] > nullmerge:
> +   self.state[rebased] >= revtodo:
>  commitmsg += '\n* %s' % repo[rebased].description()
>  editopt = True
> @@ -480,5 +472,5 @@ class rebaseruntime(object):
>  newrev = repo[newnode].rev()
>  for oldrev in self.state.iterkeys():
> -if self.state[oldrev] > nullmerge:
> +if self.state[oldrev] >= revtodo:
>  self.state[oldrev] = newrev
>  
> @@ -1231,5 +1223,4 @@ def buildstate(repo, dest, rebaseset, co
>  roots.sort()
>  state = dict.fromkeys(rebaseset, revtodo)
> -detachset = set()
>  emptyrebase = True
>  for root in roots:
> @@ -1253,45 +1244,4 @@ def buildstate(repo, dest, rebaseset, co
>  emptyrebase = False
>  repo.ui.debug('rebase onto %s starting from %s\n' % (dest, root))
> -# Rebase tries to turn  into a parent of  while
> -# preserving the number of parents of rebased changesets:
> -#
> -# - A changeset with a single parent will always be rebased as a
> -#   changeset with a single parent.
> -#
> -# - A merge will be rebased as merge unless its parents are both
> -#   ancestors of  or are themselves in the rebased set and
> -#   pruned while rebased.
> -#
> -# If one parent of  is an ancestor of , the rebased
> -# version of this parent will be . This is always true with
> -# --base option.
> -#
> -# Otherwise, we need to *replace* the original parents with
> -# . This "detaches" the rebased set from its former location
> -# and rebases it onto . Changes introduced by ancestors of
> -#  not common with  (the detachset, marked as
> -# nullmerge) are "removed" from the rebased changesets.
> -#
> -# - If  has a single parent, set it to .
> -#
> -# - If  is a merge, we cannot decide which parent to
> -#   replace, the rebase operation is not clearly defined.
> -#
> -# The table below sums up this behavior:
> -#
> -# 
> +--+--+-+
> -# |  | one parent   |  merge 
>  |
> -# 
> +--+--+-+
> -# | parent in| new parent is  | parents in :: 
> are |
> -# | :: |  | remapped to  
>  |
> -# 
> +--+--+-+
> -# | unrelated source | new parent is  

[PATCH 3 of 4] color: automatically define 16 and 256 colors if supported

[Gregory Szorc]

# HG changeset patch
# User Gregory Szorc 
# Date 1499641339 25200
#  Sun Jul 09 16:02:19 2017 -0700
# Node ID fa6223b9e2a0d9fbfa81329b83c0512417cee713
# Parent  98c54db9407a7e0ba94a632aafdbbec3fc76fa8b
color: automatically define 16 and 256 colors if supported

The "colors" terminfo capability returns the number of colors
supported by the terminal profile.

But with terminals, nothing is straightforward. The "colors"
capability only returns what the current terminal profile (likely
defined by $TERM) supports. The current terminal type or its profile
may be wrong (most likely the former). So, any consumer of this
capability needs to take this scenario into account.

This commit adds code for querying the "colors" terminfo capability
and for defining additional colors if 16 or 256 color support is
present. Because of the aforementioned limitations with capability
accuracy, a config option is introduced that allows the color count
to be limited. If the extra colors cause any problems, once can simply
set this option to restore existing 8 color behavior.

As part of this feature, a dictionary containing cherry-picked colors
from the 256 color spectrum was created. Choices are very subjective
and can and should be adjusted, as appropriate.

The help documentation for color support has been significantly
overhauled as part of this commit.

diff --git a/mercurial/color.py b/mercurial/color.py
--- a/mercurial/color.py
+++ b/mercurial/color.py
@@ -45,10 +45,43 @@ try:
 'white': curses.COLOR_WHITE,
 }
 
+# Bright versions of the built-in colors are colors 8-15, in the same 
order.
+TERMINFO_COLOR_16 = {
+'brightblack': 8,
+'brightred': 9,
+'brightgreen': 10,
+'brightyellow': 11,
+'brightblue': 12,
+'brightmagenta': 13,
+'brightcyan': 14,
+'brightwhite': 15,
+}
+
+# 6x6x6 color cube is 16-231. We define some common ones.
+TERMINFO_COLOR_256 = {
+'darkgreen': 28,
+'turquoise': 45,
+'lightgreen': 70,
+'bluegreen': 73,
+'deeppurple': 89,
+'purple': 129,
+'lightpurple': 177,
+'lightorange': 178,
+'lightyellow': 191,
+'pink': 201,
+'darkorange': 208,
+}
+
+# 232-255 is a continuous spectrum of grey.
+for i, value in enumerate(range(232, 256)):
+TERMINFO_COLOR_256['grey%02d' % (i + 1)] = value
+
 except ImportError:
 curses = None
 _baseterminfoparams = {}
 TERMINFO_COLOR_8 = {}
+TERMINFO_COLOR_16 = {}
+TERMINFO_COLOR_256 = {}
 
 _enabledbydefault = True
 
@@ -146,6 +179,43 @@ def _terminfocolors(ui):
 for color, value in TERMINFO_COLOR_8.items():
 colors[color] = (value, False)
 
+# Add 16 and 256 bit colors if supported and allowed by config policy.
+#
+# There's no reliable way to detect if the extra colors will actually work.
+# The terminfo database just reports what the current terminal is
+# advertising. Some terminals support querying the value of a color via
+# e.g. \e]4;%d;?\a. However, this isn't universally supported. So set
+# colors based purely on terminfo and provide users a way to disable in
+# case it doesn't work.
+
+# Will likely return -1 on failure.
+termcolors = curses.tigetnum('colors')
+if termcolors < 8:
+termcolors = 8
+
+# Default is to allow as many colors as the terminal supports.
+colorlimit = ui.configint('color', 'colorlimit', 256)
+
+if colorlimit not in (8, 16, 256):
+ui.warn(_('unsupported color.colorlimit value %d; use 8, 16, '
+  'or 256\n') % colorlimit)
+colorlimit = 8
+
+usecolors = min(termcolors, colorlimit)
+
+if usecolors >= 16:
+for color, value in TERMINFO_COLOR_16.items():
+colors[color] = (value, False)
+
+if usecolors >= 256:
+for color, value in TERMINFO_COLOR_256.items():
+colors[color] = (value, False)
+
+# TODO consider filtering color values by usecolors. This has 2
+# implications: 1) users can't force colors beyond the supported
+# color range (this arguably makes sense but is BC) 2) configstyles()
+# emits a warning if there is a reference to this discarded color
+# (it may be desirable to suppress that warning).
 for key, value in ui.configitems('color'):
 if key.startswith('color.'):
 colors[key[6:]] = (int(value), True)
@@ -186,6 +256,7 @@ def _terminfosetup(ui, mode):
 # noisy and use ui.debug().
 ui.debug("no terminfo entry for %s\n" % e)
 del ui._terminfoparams[key]
+
 if not curses.tigetstr('setaf') or not curses.tigetstr('setab'):
 # Only warn about missing terminfo entries if we explicitly asked for
 # terminfo mode.
diff --git a/mercurial/help/color.txt b/mercurial/help/color.txt
--- a/mercurial/help/color.txt
+++ b/mercurial/help/color.txt
@@ -131,19 

[PATCH 1 of 4] color: call curses.setupterm() earlier

[Gregory Szorc]

# HG changeset patch
# User Gregory Szorc 
# Date 1499628062 25200
#  Sun Jul 09 12:21:02 2017 -0700
# Node ID 129c9709980108beecfa1c02a52c679654de8b21
# Parent  4672db164c986da4442bd864cd044512d975c3f2
color: call curses.setupterm() earlier

It doesn't make sense to iterate over config options and set
ui._terminfoparams if curses.setupterm() will fail and we'll
just clear ui._terminfoparams. So move curses.setupterm()
before any ui._terminfoparams manipulation.

diff --git a/mercurial/color.py b/mercurial/color.py
--- a/mercurial/color.py
+++ b/mercurial/color.py
@@ -141,6 +141,13 @@ def _terminfosetup(ui, mode):
 # Otherwise, see what the config file says.
 if mode not in ('auto', 'terminfo'):
 return
+
+try:
+curses.setupterm()
+except curses.error:
+ui._terminfoparams.clear()
+return
+
 ui._terminfoparams.update(_baseterminfoparams)
 
 for key, val in ui.configitems('color'):
@@ -150,11 +157,6 @@ def _terminfosetup(ui, mode):
 elif key.startswith('terminfo.'):
 newval = (True, '', val.replace('\\E', '\x1b'))
 ui._terminfoparams[key[9:]] = newval
-try:
-curses.setupterm()
-except curses.error as e:
-ui._terminfoparams.clear()
-return
 
 for key, (b, e, c) in ui._terminfoparams.items():
 if not b:
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 4 of 4] [RFC] color: support different styling depending on color support (BC)

[Gregory Szorc]

# HG changeset patch
# User Gregory Szorc 
# Date 1499643195 25200
#  Sun Jul 09 16:33:15 2017 -0700
# Node ID 91d6e437a93d91c1423e2ce735dc053a131fd590
# Parent  fa6223b9e2a0d9fbfa81329b83c0512417cee713
[RFC] color: support different styling depending on color support (BC)

What good is support for 16 and 256 colors if we're not going to use
it?

This commit implements support for customizing the default styling
based on how many colors are currently being used.

The _defaultstyles dict now accepts dicts as values. If a dict is
present, keys indicate styling for that amount of available
colors. This allows us to tailor styling based on what is
available, allowing us to provide an optimal experience depending
on the level of color support.

This patch is RFC quality because there are no tests and I basically
threw the implementation together without much thought to see what
the reaction would be to the feature.

I'm not sure if a simple color count in the dict is appropriate
because depending on color support on Windows, our color "namespace
may not align. In addition, we should probably consider how color
blindness comes into play. I'd *really* like a config knob to
declare your color blindness. Presumably then we'd have style
variations for each color blind scenario.

diff --git a/mercurial/color.py b/mercurial/color.py
--- a/mercurial/color.py
+++ b/mercurial/color.py
@@ -135,13 +135,31 @@ except ImportError:
 'diff.inserted': 'green',
 'diff.tab': '',
 'diff.trailingwhitespace': 'bold red_background',
-'changeset.public' : '',
-'changeset.draft' : '',
-'changeset.secret' : '',
+'changeset.public': {
+8: '',
+16: 'brightred',
+},
+'changeset.draft': {
+8: '',
+16: 'brightyellow',
+},
+'changeset.secret': {
+8: '',
+16: 'brightmagenta',
+},
 'diffstat.deleted': 'red',
 'diffstat.inserted': 'green',
 'histedit.remaining': 'red bold',
 'ui.prompt': 'yellow',
+'log.bookmark': {
+16: 'brightgreen',
+},
+'log.branch': {
+16: 'brightcyan',
+},
+'log.tag': {
+16: 'brightblue',
+},
 'log.changeset': 'yellow',
 'patchbomb.finalsummary': '',
 'patchbomb.from': 'magenta',
@@ -168,7 +186,23 @@ except ImportError:
 }
 
 def loadcolortable(ui, extname, colortable):
-_defaultstyles.update(colortable)
+for k, v in colortable.items():
+# Incoming value is a dict. Ensure existing key is a dict then
+# overlay.
+if isinstance(v, dict):
+if (k in _defaultstyles
+and not isinstance(_defaultstyles[k], dict)):
+_defaultstyles[k] = {8: _defaultstyles[k]}
+else:
+_defaultstyles[k] = {}
+
+_defaultstyles[k].update(v)
+else:
+# It's a string. Interpret as legacy and assume it is for 8 colors.
+if k in _defaultstyles and isinstance(_defaultstyles[k], dict):
+_defaultstyles[k][8] = v
+else:
+_defaultstyles[k] = v
 
 def _terminfocolors(ui):
 """Obtain defined terminfo colors as a dict.
@@ -202,6 +236,7 @@ def _terminfocolors(ui):
 colorlimit = 8
 
 usecolors = min(termcolors, colorlimit)
+ui._colorcount = usecolors
 
 if usecolors >= 16:
 for color, value in TERMINFO_COLOR_16.items():
@@ -364,7 +399,19 @@ def _modesetup(ui):
 return None
 
 def configstyles(ui):
-ui._styles.update(_defaultstyles)
+colorcount = ui._colorcount
+
+for k, v in _defaultstyles.iteritems():
+if isinstance(v, dict):
+for colors, colorvalue in sorted(v.items(), reverse=True):
+# Use the first value that is within our acceptable color 
count.
+if colors <= colorcount:
+ui._styles[k] = colorvalue
+break
+
+else:
+ui._styles[k] = v
+
 for status, cfgeffects in ui.configitems('color'):
 if '.' not in status or status.startswith(('color.', 'terminfo.')):
 continue
diff --git a/mercurial/ui.py b/mercurial/ui.py
--- a/mercurial/ui.py
+++ b/mercurial/ui.py
@@ -192,6 +192,7 @@ class ui(object):
 self.logblockedtimes = False
 # color mode: see mercurial/color.py for possible value
 self._colormode = None
+self._colorcount = None
 self._terminfoparams = {}
 self._styles = {}
 
@@ -213,6 +214,7 @@ class ui(object):
 self.callhooks = src.callhooks
 self.insecureconnections = src.insecureconnections
 self._colormode = src._colormode
+self._colorcount = src._colorcount
 self._terminfoparams = src._terminfoparams.copy()
 self._styles = src._styles.copy()
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org

[PATCH 2 of 4] color: refactor how terminfo colors are defined

[Gregory Szorc]

# HG changeset patch
# User Gregory Szorc 
# Date 1499639997 25200
#  Sun Jul 09 15:39:57 2017 -0700
# Node ID 98c54db9407a7e0ba94a632aafdbbec3fc76fa8b
# Parent  129c9709980108beecfa1c02a52c679654de8b21
color: refactor how terminfo colors are defined

Terminfo colors are currently lumped together with terminfo control
and styling attributes. As part of implementing 16 and 256 color
support, it will help to differentiate terminfo colors from other
attributes. So this commit does that.

Colors are split to their own dict. A new function for obtaining
a dict of color name to color number has been implemented and code
has been refactored to use it.

The color dict preserves whether the color is user-defined so that
`hg debugcolors` output remains consistent. I'm not sure if this is
worth it. I kept the old behavior to avoid a BC break.

diff --git a/mercurial/color.py b/mercurial/color.py
--- a/mercurial/color.py
+++ b/mercurial/color.py
@@ -32,18 +32,23 @@ try:
 'bold': (True, 'bold', ''),
 'invisible': (True, 'invis', ''),
 'italic': (True, 'sitm', ''),
-'black': (False, curses.COLOR_BLACK, ''),
-'red': (False, curses.COLOR_RED, ''),
-'green': (False, curses.COLOR_GREEN, ''),
-'yellow': (False, curses.COLOR_YELLOW, ''),
-'blue': (False, curses.COLOR_BLUE, ''),
-'magenta': (False, curses.COLOR_MAGENTA, ''),
-'cyan': (False, curses.COLOR_CYAN, ''),
-'white': (False, curses.COLOR_WHITE, ''),
 }
+
+TERMINFO_COLOR_8 = {
+'black': curses.COLOR_BLACK,
+'red': curses.COLOR_RED,
+'green': curses.COLOR_GREEN,
+'yellow': curses.COLOR_YELLOW,
+'blue': curses.COLOR_BLUE,
+'magenta': curses.COLOR_MAGENTA,
+'cyan': curses.COLOR_CYAN,
+'white': curses.COLOR_WHITE,
+}
+
 except ImportError:
 curses = None
 _baseterminfoparams = {}
+TERMINFO_COLOR_8 = {}
 
 _enabledbydefault = True
 
@@ -132,6 +137,21 @@ except ImportError:
 def loadcolortable(ui, extname, colortable):
 _defaultstyles.update(colortable)
 
+def _terminfocolors(ui):
+"""Obtain defined terminfo colors as a dict.
+
+Keys are color names. Values are tuples of (value, user-defined).
+"""
+colors = {}
+for color, value in TERMINFO_COLOR_8.items():
+colors[color] = (value, False)
+
+for key, value in ui.configitems('color'):
+if key.startswith('color.'):
+colors[key[6:]] = (int(value), True)
+
+return colors
+
 def _terminfosetup(ui, mode):
 '''Initialize terminfo data and the terminal if we're in terminfo mode.'''
 
@@ -150,11 +170,11 @@ def _terminfosetup(ui, mode):
 
 ui._terminfoparams.update(_baseterminfoparams)
 
+for color, value in _terminfocolors(ui).items():
+ui._terminfoparams[color] = (False, value[0], '')
+
 for key, val in ui.configitems('color'):
-if key.startswith('color.'):
-newval = (False, int(val), '')
-ui._terminfoparams[key[6:]] = newval
-elif key.startswith('terminfo.'):
+if key.startswith('terminfo.'):
 newval = (True, '', val.replace('\\E', '\x1b'))
 ui._terminfoparams[key[9:]] = newval
 
diff --git a/mercurial/debugcommands.py b/mercurial/debugcommands.py
--- a/mercurial/debugcommands.py
+++ b/mercurial/debugcommands.py
@@ -405,10 +405,11 @@ def _debugdisplaycolor(ui):
 for effect in color._activeeffects(ui).keys():
 ui._styles[effect] = effect
 if ui._terminfoparams:
+for k, v in color._terminfocolors(ui).items():
+ui._styles['color.%s' % k if v[1] else k] = k
+
 for k, v in ui.configitems('color'):
-if k.startswith('color.'):
-ui._styles[k] = k[6:]
-elif k.startswith('terminfo.'):
+if k.startswith('terminfo.'):
 ui._styles[k] = k[9:]
 ui.write(_('available colors:\n'))
 # sort label with a '_' after the other to group '_background' entry.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 5 of 8] archival: flag missing files as a dirty wdir() in the metadata file (BC)

[Matt Harbison]

On Sun, 09 Jul 2017 19:34:53 -0400, Matt Harbison   
wrote:



# HG changeset patch
# User Matt Harbison 
# Date 1499582763 14400
#  Sun Jul 09 02:46:03 2017 -0400
# Node ID 81a3a3e44a191cfea66e25d472d65cce6cf2020d
# Parent  3026ee006b20cf266d90181edbc9e2061cce8f92
archival: flag missing files as a dirty wdir() in the metadata file (BC)

Since the identify command adds a '+' for missing files, it's reasonable  
that
this does too.  Perhaps the node field's hex value should be p1+p2 for  
merges?


Or we could add another field ('identity'?), that is the equivalent of `hg  
id -i`, and avoid the BC.  That might be more useful (and solves the p1+p2  
inconsistency), especially if `hg id -i` is used for versioning.



diff --git a/mercurial/archival.py b/mercurial/archival.py
--- a/mercurial/archival.py
+++ b/mercurial/archival.py
@@ -83,7 +83,7 @@
 hex = ctx.hex()
 if ctx.rev() is None:
 hex = ctx.p1().hex()
-if ctx.dirty():
+if ctx.dirty(missing=True):
 hex += '+'
  base = 'repo: %s\nnode: %s\nbranch: %s\n' % (
diff --git a/tests/test-subrepo-deep-nested-change.t  
b/tests/test-subrepo-deep-nested-change.t

--- a/tests/test-subrepo-deep-nested-change.t
+++ b/tests/test-subrepo-deep-nested-change.t
@@ -238,6 +238,31 @@
   committing subrepository sub1
   committing subrepository sub1/sub2 (glob)
 +  $ rm -r main
+  $ hg archive -S -qr 'wdir()' ../wdir
+  $ cat ../wdir/.hg_archival.txt
+  repo: 7f491f53a367861f47ee64a80eb997d1f341b77a
+  node: 9bb10eebee29dc0f1201dcf5977b811a540255fd+
+  branch: default
+  latesttag: null
+  latesttagdistance: 4
+  changessincelatesttag: 4
+  $ hg update -Cq .
+
+TODO: add the dirty flag for missing subrepo files
+
+  $ rm -r ../wdir sub1/sub2/folder/test.txt
+  $ hg archive -S -qr 'wdir()' ../wdir
+  $ cat ../wdir/.hg_archival.txt
+  repo: 7f491f53a367861f47ee64a80eb997d1f341b77a
+  node: 9bb10eebee29dc0f1201dcf5977b811a540255fd
+  branch: default
+  latesttag: null
+  latesttagdistance: 4
+  changessincelatesttag: 4
+  $ hg update -Cq .
+  $ rm -r ../wdir
+
 .. but first take a detour through some deep removal testing
$ hg remove -S -I 're:.*.txt' .

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 8 of 8] test-subrepo: demonstrate a status problem when merge deletes a file

[Matt Harbison]

# HG changeset patch
# User Matt Harbison 
# Date 1499632684 14400
#  Sun Jul 09 16:38:04 2017 -0400
# Node ID 3ecc0fb587347c91f730eecf261ec707d91e367a
# Parent  52be0dbe23036b919defce0c417175ab615a05b2
test-subrepo: demonstrate a status problem when merge deletes a file

At the interactive update prompt, if (c) is chosen and then followed by `hg rm`,
both `status -R` and `status -S` show the file as 'R', and `files -R` shows no
files (OK, because explicitly removed files aren't supposed to be listed).  If
`rm` follows selecting (c), then both flavors of `status` list the file as '!',
and `files -R` lists the missing file.  So somehow, the (d) option has followed
a third path.

diff --git a/tests/test-merge-subrepos.t b/tests/test-merge-subrepos.t
--- a/tests/test-merge-subrepos.t
+++ b/tests/test-merge-subrepos.t
@@ -107,13 +107,21 @@
   [255]
 
   $ hg up -r '.^' --config ui.interactive=True << EOF
-  > c
+  > d
   > EOF
   other [destination] changed b which local [working copy] deleted
-  use (c)hanged version, leave (d)eleted, or leave (u)nresolved? c
+  use (c)hanged version, leave (d)eleted, or leave (u)nresolved? d
   1 files updated, 0 files merged, 0 files removed, 0 files unresolved
 
+XXX: There's a difference between wdir() and '.', so there should be a status.
+`hg files -S` from the top is also missing 'subrepo/b'.
+
   $ hg st -S
+  $ hg st -R subrepo
+  $ hg files -R subrepo
+  [1]
+  $ hg files -R subrepo -r '.'
+  subrepo/b (glob)
 
   $ hg bookmark -r tip @other
   $ echo xyz > subrepo/c
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 7 of 8] subrepo: make the output references to subrepositories consistent

[Matt Harbison]

# HG changeset patch
# User Matt Harbison 
# Date 1499631210 14400
#  Sun Jul 09 16:13:30 2017 -0400
# Node ID 52be0dbe23036b919defce0c417175ab615a05b2
# Parent  97e2a1b90d6f74fbff1f7c62dc97b16462b9a7d2
subrepo: make the output references to subrepositories consistent

Well, mostly.  The annotation on subrepo functions tacks on a parenthetical to
the abort message, which seems reasonable for a generic mechanism.  But now all
messages consistently spell out 'subrepository', and double quote the name of
the repo.  I noticed the inconsistency in the change for the last commit.

diff --git a/mercurial/subrepo.py b/mercurial/subrepo.py
--- a/mercurial/subrepo.py
+++ b/mercurial/subrepo.py
@@ -73,7 +73,7 @@
 raise ex
 except error.Abort as ex:
 subrepo = subrelpath(self)
-errormsg = str(ex) + ' ' + _('(in subrepo %s)') % subrepo
+errormsg = str(ex) + ' ' + _('(in subrepository "%s")') % subrepo
 # avoid handling this exception by raising a SubrepoAbort exception
 raise SubrepoAbort(errormsg, hint=ex.hint, subrepo=subrepo,
cause=sys.exc_info())
@@ -147,7 +147,7 @@
 kind = 'hg'
 if src.startswith('['):
 if ']' not in src:
-raise error.Abort(_('missing ] in subrepo source'))
+raise error.Abort(_('missing ] in subrepository source'))
 kind, src = src.split(']', 1)
 kind = kind[1:]
 src = src.lstrip() # strip any extra whitespace after ']'
@@ -477,7 +477,7 @@
 This returns None, otherwise.
 """
 if self.dirty(ignoreupdate=ignoreupdate, missing=missing):
-return _("uncommitted changes in subrepository '%s'"
+return _('uncommitted changes in subrepository "%s"'
  ) % subrelpath(self)
 
 def bailifchanged(self, ignoreupdate=False, hint=None):
@@ -896,7 +896,7 @@
 ctx = urepo[revision]
 if ctx.hidden():
 urepo.ui.warn(
-_('revision %s in subrepo %s is hidden\n') \
+_('revision %s in subrepository "%s" is hidden\n') \
 % (revision[0:12], self._path))
 repo = urepo
 hg.updaterepo(repo, revision, overwrite)
@@ -910,12 +910,14 @@
 
 def mergefunc():
 if anc == cur and dst.branch() == cur.branch():
-self.ui.debug("updating subrepo %s\n" % subrelpath(self))
+self.ui.debug('updating subrepository "%s"\n'
+  % subrelpath(self))
 hg.update(self._repo, state[1])
 elif anc == dst:
-self.ui.debug("skipping subrepo %s\n" % subrelpath(self))
+self.ui.debug('skipping subrepository "%s"\n'
+  % subrelpath(self))
 else:
-self.ui.debug("merging subrepo %s\n" % subrelpath(self))
+self.ui.debug('merging subrepository "%s"\n' % 
subrelpath(self))
 hg.merge(self._repo, state[1], remind=False)
 
 wctx = self._repo[None]
@@ -1555,8 +1557,8 @@
 # try only origin: the originally cloned repo
 self._gitcommand(['fetch'])
 if not self._githavelocally(revision):
-raise error.Abort(_("revision %s does not exist in subrepo %s\n") %
-   (revision, self._relpath))
+raise error.Abort(_('revision %s does not exist in subrepository '
+'"%s"\n') % (revision, self._relpath))
 
 @annotatesubrepoerror
 def dirty(self, ignoreupdate=False, missing=False):
@@ -1611,8 +1613,8 @@
 
 def rawcheckout():
 # no branch to checkout, check it out with no branch
-self.ui.warn(_('checking out detached HEAD in subrepo %s\n') %
-  self._relpath)
+self.ui.warn(_('checking out detached HEAD in '
+   'subrepository "%s"\n') % self._relpath)
 self.ui.warn(_('check out a git branch if you intend '
 'to make changes\n'))
 checkout(['-q', revision])
@@ -1731,14 +1733,14 @@
 # determine if the current branch is even useful
 if not self._gitisancestor(self._state[1], current):
 self.ui.warn(_('unrelated git branch checked out '
-'in subrepo %s\n') % self._relpath)
+   'in subrepository "%s"\n') % self._relpath)
 return False
-self.ui.status(_('pushing branch %s of subrepo %s\n') %
+self.ui.status(_('pushing branch %s of subrepository "%s"\n') %
(current.split('/', 2)[2], self._relpath))
 ret = self._gitdir(cmd + ['origin', current])
 return ret[1] == 0
 else:
-

[PATCH 2 of 8] identify: simplify the dirty check

[Matt Harbison]

# HG changeset patch
# User Matt Harbison 
# Date 1499573943 14400
#  Sun Jul 09 00:19:03 2017 -0400
# Node ID 09a3c95bdd28d00bc0071a4ade1ab31cf61262b2
# Parent  19ffd0a7aa46964067e35dd6a092adac2c238502
identify: simplify the dirty check

This is equivalent to the previous code, but it seems better to be explicit
about what aspects of dirty are being ignored.  Perhaps they shouldn't be, since
the help text says 'followed by a "+" if the working directory has uncommitted
changes'.  Both merges and branch changes are committable, even if the files are
unchanged.

Additionally, this will make the `identify` command notice missing subrepo
files, once subrepos are taught to look for missing files.

diff --git a/mercurial/commands.py b/mercurial/commands.py
--- a/mercurial/commands.py
+++ b/mercurial/commands.py
@@ -2775,8 +2775,7 @@
 taglist.extend(p.tags())
 
 dirty = ""
-if (any(repo.status())
-or any(ctx.sub(s).dirty() for s in ctx.substate)):
+if ctx.dirty(missing=True, merge=False, branch=False):
 dirty = '+'
 fm.data(dirty=dirty)
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 6 of 8] subrepo: consider the parent repo dirty when a file is missing

[Matt Harbison]

# HG changeset patch
# User Matt Harbison 
# Date 1499583346 14400
#  Sun Jul 09 02:55:46 2017 -0400
# Node ID 97e2a1b90d6f74fbff1f7c62dc97b16462b9a7d2
# Parent  81a3a3e44a191cfea66e25d472d65cce6cf2020d
subrepo: consider the parent repo dirty when a file is missing

This simply passes the 'missing' argument down from the context of the parent
repo, so the same rules apply.  subrepo.bailifchanged() is hardcoded to care
about missing files, because cmdutil.bailifchanged() is too.

In the end, it looks like this addresses inconsistencies with 'archive',
'identify', blackbox logs, 'merge', and 'update --check'.  I wasn't sure how to
implement this in git, so that's left for someone more familiar with it.

diff --git a/hgext/largefiles/overrides.py b/hgext/largefiles/overrides.py
--- a/hgext/largefiles/overrides.py
+++ b/hgext/largefiles/overrides.py
@@ -289,10 +289,10 @@
 finally:
 repo.lfstatus = False
 
-def overridedirty(orig, repo, ignoreupdate=False):
+def overridedirty(orig, repo, ignoreupdate=False, missing=False):
 try:
 repo._repo.lfstatus = True
-return orig(repo, ignoreupdate)
+return orig(repo, ignoreupdate=ignoreupdate, missing=missing)
 finally:
 repo._repo.lfstatus = False
 
diff --git a/mercurial/context.py b/mercurial/context.py
--- a/mercurial/context.py
+++ b/mercurial/context.py
@@ -1508,7 +1508,7 @@
 "check whether a working directory is modified"
 # check subrepos first
 for s in sorted(self.substate):
-if self.sub(s).dirty():
+if self.sub(s).dirty(missing=missing):
 return True
 # check current working dir
 return ((merge and self.p2()) or
diff --git a/mercurial/subrepo.py b/mercurial/subrepo.py
--- a/mercurial/subrepo.py
+++ b/mercurial/subrepo.py
@@ -460,14 +460,15 @@
 """
 return False
 
-def dirty(self, ignoreupdate=False):
+def dirty(self, ignoreupdate=False, missing=False):
 """returns true if the dirstate of the subrepo is dirty or does not
 match current stored state. If ignoreupdate is true, only check
-whether the subrepo has uncommitted changes in its dirstate.
+whether the subrepo has uncommitted changes in its dirstate.  If 
missing
+is true, check for deleted files.
 """
 raise NotImplementedError
 
-def dirtyreason(self, ignoreupdate=False):
+def dirtyreason(self, ignoreupdate=False, missing=False):
 """return reason string if it is ``dirty()``
 
 Returned string should have enough information for the message
@@ -475,14 +476,15 @@
 
 This returns None, otherwise.
 """
-if self.dirty(ignoreupdate=ignoreupdate):
+if self.dirty(ignoreupdate=ignoreupdate, missing=missing):
 return _("uncommitted changes in subrepository '%s'"
  ) % subrelpath(self)
 
 def bailifchanged(self, ignoreupdate=False, hint=None):
 """raise Abort if subrepository is ``dirty()``
 """
-dirtyreason = self.dirtyreason(ignoreupdate=ignoreupdate)
+dirtyreason = self.dirtyreason(ignoreupdate=ignoreupdate,
+   missing=True)
 if dirtyreason:
 raise error.Abort(dirtyreason, hint=hint)
 
@@ -815,7 +817,7 @@
 return total
 
 @annotatesubrepoerror
-def dirty(self, ignoreupdate=False):
+def dirty(self, ignoreupdate=False, missing=False):
 r = self._state[1]
 if r == '' and not ignoreupdate: # no state recorded
 return True
@@ -823,7 +825,7 @@
 if r != w.p1().hex() and not ignoreupdate:
 # different version checked out
 return True
-return w.dirty() # working directory changed
+return w.dirty(missing=missing) # working directory changed
 
 def basestate(self):
 return self._repo['.'].hex()
@@ -1202,8 +1204,10 @@
 return True, True, bool(missing)
 return bool(changes), False, bool(missing)
 
-def dirty(self, ignoreupdate=False):
-if not self._wcchanged()[0]:
+def dirty(self, ignoreupdate=False, missing=False):
+wcchanged = self._wcchanged()
+changed = wcchanged[0] or (missing and wcchanged[2])
+if not changed:
 if self._state[1] in self._wcrevs() or ignoreupdate:
 return False
 return True
@@ -1555,7 +1559,7 @@
(revision, self._relpath))
 
 @annotatesubrepoerror
-def dirty(self, ignoreupdate=False):
+def dirty(self, ignoreupdate=False, missing=False):
 if self._gitmissing():
 return self._state[1] != ''
 if self._gitisbare():
diff --git a/tests/test-merge-subrepos.t b/tests/test-merge-subrepos.t
--- a/tests/test-merge-subrepos.t
+++ b/tests/test-merge-subrepos.t
@@ -55,13 +55,13 @@
 
   $ rm subrepo/b
 
-TODO: a deleted 

[PATCH 3 of 8] blackbox: simplify the dirty check

[Matt Harbison]

# HG changeset patch
# User Matt Harbison 
# Date 1499574183 14400
#  Sun Jul 09 00:23:03 2017 -0400
# Node ID fca94a77d9cac7b60a4b152fcb42ddb4dd1a
# Parent  09a3c95bdd28d00bc0071a4ade1ab31cf61262b2
blackbox: simplify the dirty check

Same idea (and possibly incorrect behavior) as the previous commit.

diff --git a/hgext/blackbox.py b/hgext/blackbox.py
--- a/hgext/blackbox.py
+++ b/hgext/blackbox.py
@@ -197,10 +197,8 @@
 ctx = ui._bbrepo[None]
 parents = ctx.parents()
 rev = ('+'.join([hex(p.node()) for p in parents]))
-if (ui.configbool('blackbox', 'dirty') and (
-any(ui._bbrepo.status()) or
-any(ctx.sub(s).dirty() for s in ctx.substate)
-)):
+if (ui.configbool('blackbox', 'dirty') and
+ctx.dirty(missing=True, merge=False, branch=False)):
 changed = '+'
 if ui.configbool('blackbox', 'logsource'):
 src = ' [%s]' % event
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 4 of 8] cmdutil: simplify the dirty check in howtocontinue()

[Matt Harbison]

# HG changeset patch
# User Matt Harbison 
# Date 1499575996 14400
#  Sun Jul 09 00:53:16 2017 -0400
# Node ID 3026ee006b20cf266d90181edbc9e2061cce8f92
# Parent  fca94a77d9cac7b60a4b152fcb42ddb4dd1a
cmdutil: simplify the dirty check in howtocontinue()

This is equivalent to the previous code.  But it seems to me that if the user is
going to be prompted that a commit is needed, missing files should be ignored,
but branch and merge changes shouldn't be.

diff --git a/mercurial/cmdutil.py b/mercurial/cmdutil.py
--- a/mercurial/cmdutil.py
+++ b/mercurial/cmdutil.py
@@ -3547,10 +3547,7 @@
 for f, msg in afterresolvedstates:
 if repo.vfs.exists(f):
 return contmsg % msg, True
-workingctx = repo[None]
-dirty = any(repo.status()) or any(workingctx.sub(s).dirty()
- for s in workingctx.substate)
-if dirty:
+if repo[None].dirty(missing=True, merge=False, branch=False):
 return contmsg % _("hg commit"), False
 return None, None
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 5 of 8] archival: flag missing files as a dirty wdir() in the metadata file (BC)

[Matt Harbison]

# HG changeset patch
# User Matt Harbison 
# Date 1499582763 14400
#  Sun Jul 09 02:46:03 2017 -0400
# Node ID 81a3a3e44a191cfea66e25d472d65cce6cf2020d
# Parent  3026ee006b20cf266d90181edbc9e2061cce8f92
archival: flag missing files as a dirty wdir() in the metadata file (BC)

Since the identify command adds a '+' for missing files, it's reasonable that
this does too.  Perhaps the node field's hex value should be p1+p2 for merges?

diff --git a/mercurial/archival.py b/mercurial/archival.py
--- a/mercurial/archival.py
+++ b/mercurial/archival.py
@@ -83,7 +83,7 @@
 hex = ctx.hex()
 if ctx.rev() is None:
 hex = ctx.p1().hex()
-if ctx.dirty():
+if ctx.dirty(missing=True):
 hex += '+'
 
 base = 'repo: %s\nnode: %s\nbranch: %s\n' % (
diff --git a/tests/test-subrepo-deep-nested-change.t 
b/tests/test-subrepo-deep-nested-change.t
--- a/tests/test-subrepo-deep-nested-change.t
+++ b/tests/test-subrepo-deep-nested-change.t
@@ -238,6 +238,31 @@
   committing subrepository sub1
   committing subrepository sub1/sub2 (glob)
 
+  $ rm -r main
+  $ hg archive -S -qr 'wdir()' ../wdir
+  $ cat ../wdir/.hg_archival.txt
+  repo: 7f491f53a367861f47ee64a80eb997d1f341b77a
+  node: 9bb10eebee29dc0f1201dcf5977b811a540255fd+
+  branch: default
+  latesttag: null
+  latesttagdistance: 4
+  changessincelatesttag: 4
+  $ hg update -Cq .
+
+TODO: add the dirty flag for missing subrepo files
+
+  $ rm -r ../wdir sub1/sub2/folder/test.txt
+  $ hg archive -S -qr 'wdir()' ../wdir
+  $ cat ../wdir/.hg_archival.txt
+  repo: 7f491f53a367861f47ee64a80eb997d1f341b77a
+  node: 9bb10eebee29dc0f1201dcf5977b811a540255fd
+  branch: default
+  latesttag: null
+  latesttagdistance: 4
+  changessincelatesttag: 4
+  $ hg update -Cq .
+  $ rm -r ../wdir
+
 .. but first take a detour through some deep removal testing
 
   $ hg remove -S -I 're:.*.txt' .
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 8] tests: tweak the subrepo dirty state tests

[Matt Harbison]

# HG changeset patch
# User Matt Harbison 
# Date 1499573131 14400
#  Sun Jul 09 00:05:31 2017 -0400
# Node ID 19ffd0a7aa46964067e35dd6a092adac2c238502
# Parent  1e872b08a4e93b21e60cba695e4022bf71d4acf4
tests: tweak the subrepo dirty state tests

This is a continuation of 439b4d005b4a.  I overlooked that blackbox logs also
have a dirty marker.  Also, the `hg update --check` test was updating to a
revision where the deleted file wasn't tracked, which is why status seemed to
show the deleted file was restored.

diff --git a/tests/test-merge-subrepos.t b/tests/test-merge-subrepos.t
--- a/tests/test-merge-subrepos.t
+++ b/tests/test-merge-subrepos.t
@@ -28,16 +28,21 @@
 noticed by `update --check` in the top level repo.
 
   $ hg ci -Sqm 'add b'
+  $ echo change > subrepo/b
+
+  $ hg ci -Sm 'change b'
+  committing subrepository subrepo
+
   $ rm a
   $ hg id
-  cb66ec850af7+ tip
+  9bfe45a197d7+ tip
   $ hg sum
-  parent: 3:cb66ec850af7 tip
-   add b
+  parent: 4:9bfe45a197d7 tip
+   change b
   branch: default
   commit: 1 deleted (clean)
   update: 1 new changesets, 2 branch heads (merge)
-  phases: 4 draft
+  phases: 5 draft
 
   $ hg up --check -r '.^'
   abort: uncommitted changes
@@ -52,18 +57,21 @@
 
 TODO: a deleted subrepo file should be flagged as dirty, like the top level 
repo
 
-  $ hg id
-  cb66ec850af7 tip
+  $ hg id --config extensions.blackbox= --config blackbox.dirty=True
+  9bfe45a197d7 tip
+  $ cat .hg/blackbox.log
+  * @9bfe45a197d7b0ab09bf287729dd57e9619c9da5 (*)> id (glob)
+  * @9bfe45a197d7b0ab09bf287729dd57e9619c9da5 (*)> id --config 
"extensions.blackbox=" --config "blackbox.dirty=True" exited 0 * (glob)
 
 TODO: a deleted file should be listed as such, like the top level repo
 
   $ hg sum
-  parent: 3:cb66ec850af7 tip
-   add b
+  parent: 4:9bfe45a197d7 tip
+   change b
   branch: default
   commit: (clean)
   update: 1 new changesets, 2 branch heads (merge)
-  phases: 4 draft
+  phases: 5 draft
 
 Modified subrepo files are noticed by `update --check` and `summary`
 
@@ -76,12 +84,12 @@
   [255]
 
   $ hg sum
-  parent: 3:cb66ec850af7 tip
-   add b
+  parent: 4:9bfe45a197d7 tip
+   change b
   branch: default
   commit: 1 subrepos
   update: 1 new changesets, 2 branch heads (merge)
-  phases: 4 draft
+  phases: 5 draft
 
 TODO: why is -R needed here?  If it's because the subrepo is treated as a
 discrete unit, then this should probably warn or something.
@@ -94,11 +102,13 @@
 TODO: --check should notice a subrepo with a missing file.  It already notices
 a modified file.
 
-  $ hg up -r '.^' --check
+  $ hg up -r '.^' --check --config ui.interactive=True << EOF
+  > c
+  > EOF
+  other [destination] changed b which local [working copy] deleted
+  use (c)hanged version, leave (d)eleted, or leave (u)nresolved? c
   1 files updated, 0 files merged, 0 files removed, 0 files unresolved
 
-TODO: update without --clean shouldn't restore a deleted subrepo file, since it
-doesn't restore a deleted top level repo file.
   $ hg st -S
 
   $ hg bookmark -r tip @other
@@ -112,7 +122,7 @@
 Merge sees deleted subrepo files as an uncommitted change
 
   $ hg merge @other
-   subrepository subrepo diverged (local revision: 2b4750dcc93f, remote 
revision: cde40f86152f)
+   subrepository subrepo diverged (local revision: de222c2e1eac, remote 
revision: 7d3f8eba8116)
   (M)erge, keep (l)ocal [working copy] or keep (r)emote [merge rev]? m
   abort: uncommitted changes (in subrepo subrepo)
   (use 'hg status' to list changes)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] sparse: access status fields by name instead of deconstructing it

[Gregory Szorc]

On Sun, Jul 9, 2017 at 3:09 PM, Martin von Zweigbergk via Mercurial-devel <
mercurial-devel@mercurial-scm.org> wrote:

> # HG changeset patch
> # User Martin von Zweigbergk 
> # Date 1499404838 25200
> #  Thu Jul 06 22:20:38 2017 -0700
> # Node ID 27c6b35d4637ae4bca2b78c6f2f2f06af0d47873
> # Parent  9087f9997f42ce3035e97a03bce63852a20e
> sparse: access status fields by name instead of deconstructing it
>

Queued for default.


>
> The status tuples has had named fields for a few years now.
>
> diff --git a/mercurial/sparse.py b/mercurial/sparse.py
> --- a/mercurial/sparse.py
> +++ b/mercurial/sparse.py
> @@ -200,9 +200,8 @@
>  if not enabled or not repo.vfs.exists('tempsparse'):
>  return
>
> -origstatus = repo.status()
> -modified, added, removed, deleted, a, b, c = origstatus
> -if modified or added or removed or deleted:
> +s = repo.status()
> +if s.modified or s.added or s.removed or s.deleted:
>  # Still have pending changes. Don't bother trying to prune.
>  return
>
> @@ -389,13 +388,11 @@
>  Will abort if a file with pending changes is being excluded or
> included
>  unless ``force`` is True.
>  """
> -modified, added, removed, deleted, unknown, ignored, clean =
> origstatus
> -
>  # Verify there are no pending changes
>  pending = set()
> -pending.update(modified)
> -pending.update(added)
> -pending.update(removed)
> +pending.update(origstatus.modified)
> +pending.update(origstatus.added)
> +pending.update(origstatus.removed)
>  sparsematch = matcher(repo)
>  abort = False
>
> ___
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
>
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH] sparse: access status fields by name instead of deconstructing it

[Martin von Zweigbergk via Mercurial-devel]

# HG changeset patch
# User Martin von Zweigbergk 
# Date 1499404838 25200
#  Thu Jul 06 22:20:38 2017 -0700
# Node ID 27c6b35d4637ae4bca2b78c6f2f2f06af0d47873
# Parent  9087f9997f42ce3035e97a03bce63852a20e
sparse: access status fields by name instead of deconstructing it

The status tuples has had named fields for a few years now.

diff --git a/mercurial/sparse.py b/mercurial/sparse.py
--- a/mercurial/sparse.py
+++ b/mercurial/sparse.py
@@ -200,9 +200,8 @@
 if not enabled or not repo.vfs.exists('tempsparse'):
 return
 
-origstatus = repo.status()
-modified, added, removed, deleted, a, b, c = origstatus
-if modified or added or removed or deleted:
+s = repo.status()
+if s.modified or s.added or s.removed or s.deleted:
 # Still have pending changes. Don't bother trying to prune.
 return
 
@@ -389,13 +388,11 @@
 Will abort if a file with pending changes is being excluded or included
 unless ``force`` is True.
 """
-modified, added, removed, deleted, unknown, ignored, clean = origstatus
-
 # Verify there are no pending changes
 pending = set()
-pending.update(modified)
-pending.update(added)
-pending.update(removed)
+pending.update(origstatus.modified)
+pending.update(origstatus.added)
+pending.update(origstatus.removed)
 sparsematch = matcher(repo)
 abort = False
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 03 of 11 sparse] sparse: move printing of sparse config changes function into core

[Martin von Zweigbergk via Mercurial-devel]

On Sat, Jul 8, 2017 at 4:28 PM, Gregory Szorc  wrote:
> # HG changeset patch
> # User Gregory Szorc 
> # Date 1499546059 25200
> #  Sat Jul 08 13:34:19 2017 -0700
> # Node ID 43da636d83fc5e4eae97a6c3b526552407a4ec77
> # Parent  b54b5a5d05a19ad9d2b4b52f72a87fc4706cba3e
> sparse: move printing of sparse config changes function into core

I've queued up to here for now, thanks!
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 2 v3] releasenotes: add similarity check function to compare incoming notes

[Rishabh Madan]

# HG changeset patch
# User Rishabh Madan 
# Date 1499636627 -7200
#  Sun Jul 09 23:43:47 2017 +0200
# Node ID c83144b6b48a2374a37114336e58974546e915e3
# Parent  43c97ccdfd39cfa447100b54e923d1d3d753476b
releasenotes: add similarity check function to compare incoming notes

It is possible that the incoming note fragments might have some similar content
as the existing release notes. In case of a bug fix, we match for issue in $
existing notes. For other general cases, it makes use of fuzzywuzzy library to
get a similarity score. If the score is above a certain threshold, we ignore
the fragment otherwise add it. But the score might be misleading for small comm$
messages. So, it uses similarity function if only the length of string (in word$
is above a certain number. The patch also adds tests related to its usage.
But it needs improvement in the sense of combining the incoming notes. We can
use interactive mode for adding the notes. Maybe we can do this if similarity
score is under a certain range.

diff -r 43c97ccdfd39 -r c83144b6b48a hgext/releasenotes.py
--- a/hgext/releasenotes.py Sun Jul 09 23:25:02 2017 +0200
+++ b/hgext/releasenotes.py Sun Jul 09 23:43:47 2017 +0200
@@ -14,6 +14,7 @@
 from __future__ import absolute_import
 
 import errno
+import fuzzywuzzy.fuzz as fuzz
 import re
 import sys
 import textwrap
@@ -45,6 +46,7 @@
 ]
 
 RE_DIRECTIVE = re.compile('^\.\. ([a-zA-Z0-9_]+)::\s*([^$]+)?$')
+RE_ISSUE = r'\bissue [0-9]{4,6}(?![0-9])\b|\bissue[0-9]{4,6}(?![0-9])\b'
 
 BULLET_SECTION = _('Other Changes')
 
@@ -90,26 +92,17 @@
 
 This is used to combine multiple sources of release notes together.
 """
+existing = self
+
 for section in other:
+merge = mergereleasenotes(section, existing)
 for title, paragraphs in other.titledforsection(section):
-if self.hastitledinsection(section, title):
-# TODO prompt for resolution if different and running in
-# interactive mode.
-ui.write(_('%s already exists in %s section; ignoring\n') %
- (title, section))
-continue
-
-# TODO perform similarity comparison and try to match against
-# existing.
-self.addtitleditem(section, title, paragraphs)
+if merge.check_merge(ui, section, paragraphs, existing, title):
+self.addtitleditem(section, title, paragraphs)
 
 for paragraphs in other.nontitledforsection(section):
-if paragraphs in self.nontitledforsection(section):
-continue
-
-# TODO perform similarily comparison and try to match against
-# existing.
-self.addnontitleditem(section, paragraphs)
+if merge.check_merge(ui, section, paragraphs, existing):
+self.addnontitleditem(section, paragraphs)
 
 class releasenotessections(object):
 def __init__(self, ui, repo=None):
@@ -145,6 +138,77 @@
 
 return None
 
+class mergereleasenotes(object):
+def __init__(self, section, existing):
+self.incoming_points = []
+self.existing_points = self.gatherexistingnotes(section, existing)
+
+def gatherexistingnotes(self, section, existing):
+existing_points = []
+for title, paragraphs in existing.titledforsection(section):
+str = ""
+str = converttostring(paragraphs)
+existing_points.append(str)
+
+for paragraphs in existing.nontitledforsection(section):
+str = ""
+str = converttostring(paragraphs)
+existing_points.append(str)
+return existing_points
+
+def check_merge(self, ui, section, paragraphs, existing, title=None):
+if title:
+if existing.hastitledinsection(section, title):
+ui.write(_('%s already exists in %s section; ignoring\n') %
+ (title, section))
+return False
+elif paragraphs in existing.nontitledforsection(section):
+return False
+
+incoming_str = converttostring(paragraphs)
+if section == 'fix':
+issues = re.findall(RE_ISSUE, incoming_str, re.IGNORECASE)
+if len(issues) > 0:
+issuenumber = issues[0]
+issuenumber = "".join(issuenumber.split())
+if any(issuenumber in s for s in self.existing_points):
+ui.write(_("\"%s\" already exists in notes; "
+ "ignoring\n") % issuenumber)
+return False
+else:
+return True
+if len(incoming_str.split()) > 10:
+merge = similaritycheck(incoming_str, self.existing_points)
+if not merge:
+ui.write(_("\"%s\" already exists in notes 

[PATCH 1 of 2 v3] releasenotes: add import check for fuzzywuzzy

[Rishabh Madan]

# HG changeset patch
# User Rishabh Madan 
# Date 1499635502 -7200
#  Sun Jul 09 23:25:02 2017 +0200
# Node ID 43c97ccdfd39cfa447100b54e923d1d3d753476b
# Parent  5f22d3d43d36d46cea98c86b2a49eee2d323fd9e
releasenotes: add import check for fuzzywuzzy

This patch adds the has_fuzzywuzzy for import check of external dependency
fuzzywuzzy.

diff -r 5f22d3d43d36 -r 43c97ccdfd39 tests/hghave.py
--- a/tests/hghave.py   Sun Jul 09 19:04:10 2017 +0200
+++ b/tests/hghave.py   Sun Jul 09 23:25:02 2017 +0200
@@ -658,3 +658,12 @@
 @check("fsmonitor", "running tests with fsmonitor")
 def has_fsmonitor():
 return 'HGFSMONITOR_TESTS' in os.environ
+
+@check("fuzzywuzzy", "Fuzzy string matching library")
+def has_fuzzywuzzy():
+try:
+import fuzzywuzzy
+fuzzywuzzy.__version__
+return True
+except ImportError:
+return False
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 6 of 7] histedit: unify strip backup files on success (BC)

[Jun Wu]

Excerpts from Martin von Zweigbergk's message of 2017-07-09 13:44:26 -0700:
> If we wanted to, we could drop support for the case you in the mention
> there (with the AD and BC strips) and make each call to delayedstrip()
> result in one backup (with its own name/topic). I think I prefer that
> anyway, unless we can think of a concrete example where that case
> actually occurs in practice.

In the split case,
 B  o
|
  B o  B o o A2  A2 o
|  ->| | -> |
  A o  A x o A1  A1 o
||/ |
  Z o  Z o   Z  o

Rebase calls delayedstrip([B], 'rebase'), and strip calls delayedstrip([A],
'split'). It seems better to put them in a single backup file since B
depends on A. That could reduce bundle size, and the user seems to always
want to restore B and A together.

It seems the only difference we care about is whether the node to strip is
intermediate or not. I guess we could figure that out automatically from a
transaction, or a snapshot of all changelog heads for multiple transaction
case.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 6 of 7] histedit: unify strip backup files on success (BC)

[Martin von Zweigbergk via Mercurial-devel]

On Sun, Jul 9, 2017 at 12:54 PM, Jun Wu  wrote:
> Excerpts from Augie Fackler's message of 2017-07-09 15:28:15 -0400:
>>
>> > On Jul 9, 2017, at 2:24 PM, Martin von Zweigbergk  
>> > wrote:
>> >
>> > On Sat, Jul 8, 2017 at 4:51 PM, Jun Wu  wrote:
>> >> # HG changeset patch
>> >> # User Jun Wu 
>> >> # Date 1499557831 25200
>> >> #  Sat Jul 08 16:50:31 2017 -0700
>> >> # Node ID d75e65724d2bbcf17fcaaad705e463e84a3de5d2
>> >> # Parent  9a325ae88021e0e02a87ef1ae6baa8a199405140
>> >> # Available At https://bitbucket.org/quark-zju/hg-draft
>> >> #  hg pull https://bitbucket.org/quark-zju/hg-draft  -r 
>> >> d75e65724d2b
>> >> histedit: unify strip backup files on success (BC)
>> >>
>> >> Previously we wrote two different strip backup files on success. This 
>> >> patch
>> >> unifies them. It will make scmutil.cleanupnodes migration more smooth.
>> >
>> > Makes sense to me. I'm queuing the series, thanks.
>> >
>> > Augie, do you remember if there was a reason to keep them separate?
>>
>> The only reason was structural in the code: histedit has “intermediate”
>> nodes which are an implementation detail, and “old” nodes, which are the
>> precursors to the end state. Internally, the lists are separate because
>> you always want to reap “intermediate" nodes, but might want to preserve
>> “old” and in --keep or --abort cases.
>>
>> It might be nice to keep them separate but have one be marked
>> “histedit-internal” and “histedit-before” or something.
>
> Good point.
>
> Going through the "delayedstrip" code path, it could be a bit tricky to
> separate backup files [1]. I wish we can count on in-memory-merge work so
> those intermediate nodes disappear automatically. But that won't happen
> soon. Maybe "delayedstrip" API could be changed, I've added this to my
> backlog to resolve.
>
> [1]: 
> https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-June/100404.html

If we wanted to, we could drop support for the case you in the mention
there (with the AD and BC strips) and make each call to delayedstrip()
result in one backup (with its own name/topic). I think I prefer that
anyway, unless we can think of a concrete example where that case
actually occurs in practice.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 01 of 14] cache: introduce an abstract class for cache we can upgrade incrementally

[Gregory Szorc]

On Sun, Jul 9, 2017 at 10:52 AM, Boris Feld  wrote:

> # HG changeset patch
> # User Boris Feld 
> # Date 1499458441 -7200
> #  Fri Jul 07 22:14:01 2017 +0200
> # Node ID 6edb62505c697329de034c2fdc47befd5896f31f
> # Parent  89796a25d4bb91fb418ad3e70faad2c586902ffb
> # EXP-Topic obs-cache
> cache: introduce an abstract class for cache we can upgrade incrementally
>
> Right now each class implements their own mechanism for validation, and
> update. We start introducing abstract class to ultimately allow more
> unification of the cache code.
>
> The end goal of this series is to introduce a cache for some obsolescence
> property, not to actually implement the cache. However, taking advantage of
> adding a new cache to introduce the abstract class seems like a win.
>

I don't like saying this, but given the amount of discussion these patches
generated the last time they were proposed and given the proximity to the
code freeze, I doubt these will land in 4.3.

Would you be willing to voluntarily withdrawal them from consideration for
4.3? Or if there are less contentious patches/functionality that you would
like to land in 4.3, would you be willing to pair down the series to the
simpler/less-contentious patches so we can focus on 4.3?


>
> diff -r 89796a25d4bb -r 6edb62505c69 mercurial/cache.py
> --- /dev/null   Thu Jan 01 00:00:00 1970 +
> +++ b/mercurial/cache.pyFri Jul 07 22:14:01 2017 +0200
> @@ -0,0 +1,127 @@
> +# cache.py - utilities for caching
> +#
> +# Copyright 2017 Octobus SAS 
> +#
> +# This software may be used and distributed according to the terms of the
> +# GNU General Public License version 2 or any later version.
> +from __future__ import absolute_import
> +
> +import abc
> +import struct
> +
> +from . import (
> +util,
> +)
> +
> +class incrementalcachebase(object):
> +"""base class for incremental cache from append only source
> +
> +There are multiple append only data source we might want to cache
> +computation from. One of the common pattern is to track the state of
> the
> +file and update the cache with the extra data (eg: branchmap-cache
> tracking
> +changelog). This pattern also needs to detect when a the source is
> striped
> +
> +The overall pattern is similar whatever the actual source is. This
> class
> +introduces the basic patterns.
> +"""
> +
> +__metaclass__ = abc.ABCMeta
> +
> +# default key used for an empty cache
> +emptykey = ()
> +
> +_cachekeyspec = '' # used for serialization
> +_cachename = None # used for debug message
> +
> +@abc.abstractmethod
> +def __init__(self):
> +super(incrementalcachebase, self).__init__()
> +self._cachekey = None
> +
> +@util.propertycache
> +def _cachekeystruct(self):
> +# dynamic property to help subclass to change it
> + return struct.Struct('>' + self._cachekeyspec)
> +
> +@util.propertycache
> +def _cachekeysize(self):
> +# dynamic property to help subclass to change it
> +return self._cachekeystruct.size
> +
> +@abc.abstractmethod
> +def _updatefrom(self, repo, data):
> +"""override this method to update you date from incrementally
> read data.
> +
> +Content of  will depends of the sources.
> +"""
> +raise NotImplementedError
> +
> +@abc.abstractmethod
> +def clear(self, reset=False):
> +"""invalidate the cache content
> +
> +if 'reset' is passed, we detected a strip and the cache will have
> to be
> +recomputed.
> +
> +Subclasses MUST overide this method to actually affect the cache
> data.
> +"""
> +if reset:
> +self._cachekey = self.emptykey if reset else None
> +else:
> +self._cachekey = None
> +
> +@abc.abstractmethod
> +def load(self, repo):
> +"""Load data from disk
> +
> +Subclasses MUST restore the "cachekey" attribute while doing so.
> +"""
> +raise NotImplementedError
> +
> +@abc.abstractmethod
> +def _fetchupdatedata(self, repo):
> +"""Check the source for possible changes and return necessary data
> +
> +The return is a tree elements tuple: reset, data, cachekey
> +
> +* reset: `True` when a strip is detected and cache need to be
> reset
> +* data: new data to take in account, actual type depends of the
> source
> +* cachekey: the cache key covering  and precious covered
> data
> +"""
> +raise NotImplementedError
> +
> +# Useful "public" function (no need to override them)
> +
> +def update(self, repo):
> +"""update the cache with new repository data
> +
> +The update will be incremental when possible"""
> +repo = repo.unfiltered()
> +
> +# If we do not have any data, try loading from disk
> +if self._cachekey is None:
> +

Re: [PATCH 10 of 10] effectflag: detect when diff changed

[Jun Wu]

It's not very clear to me that what do we plan to use the effectflags.

I guess the most useful case is divergent resolution, like Alice changes
message, Bob changes content, so we know what to do. But a chain of changes
could cancel effects with each other, like:

  A0 - (amend by Alice) -> A1 - (amend by Alice) -> A2
   \
-- (a chain of rewrites by Bob) -> A9

"A0 -> A1" changes diff content, "A1 -> A2" changes diff content. But
"A0 -> A2" does not necessarily change diff content.

This is a divergent case where A0 has two visible successors: A2 and A9. A2
does not have diff content change but with effectflags, A2 might be treated
as having diff content change.

I think we should store the hash of actual diff text as metadata to detect
diff changes across a chain of rewrites. That hash could be stored in a
cache.

For other information like branch, desc, meta, parents. Since reading them
is very fast using changelog, I'm not sure about the benefit of storing them
in obsmarkers. It might be helpful to provide extra information for nodes
unknown to local changelog (I'd also argue those markers are meaningless and
should be hidden from the user, but that's another topic), but I don't think
this feature is useful for an end-user.


Excerpts from Boris Feld's message of 2017-07-07 14:38:39 +0200:
> # HG changeset patch
> # User Boris Feld 
> # Date 1499346007 -7200
> #  Thu Jul 06 15:00:07 2017 +0200
> # Node ID 449fc1c748c6e058e892a4c940e20137e52e7808
> # Parent  6a40d87dfedcce4064eb4bcdb131ed4d427fd4de
> # EXP-Topic effectflag
> effectflag: detect when diff changed
> 
> Store in effect flag when the diff changed between the predecessor and
> its successors.
> 
> Comparing the diff is not easy because we do not want to incorrectly detect a
> diff modification when the changeset has only been rebased.
> 
> diff -r 6a40d87dfedc -r 449fc1c748c6 mercurial/obsutil.py
> --- a/mercurial/obsutil.pyThu Jul 06 14:58:44 2017 +0200
> +++ b/mercurial/obsutil.pyThu Jul 06 15:00:07 2017 +0200
> @@ -542,6 +542,7 @@
>  DESCCHANGED = 1 << 0 # action changed the description
>  METACHANGED = 1 << 1 # action change the meta
>  PARENTCHANGED = 1 << 2 # action change the parent
> +DIFFCHANGED = 1 << 3 # action change diff introduced by the changeset
>  USERCHANGED = 1 << 4 # the user changed
>  DATECHANGED = 1 << 5 # the date changed
>  BRANCHCHANGED = 1 << 6 # the branch changed
> @@ -565,6 +566,46 @@
>  
>  return True
>  
> +def _prepare_hunk(hunk):
> +"""Drop all information but the username and patch"""
> +cleanunk = []
> +for line in hunk.splitlines():
> +if line.startswith(b'# User') or not line.startswith(b'#'):
> +if line.startswith(b'@@'):
> +line = b'@@\n'
> +cleanunk.append(line)
> +return cleanunk
> +
> +def _getdifflines(iterdiff):
> +"""return a cleaned up lines"""
> +try:
> +lines = next(iterdiff)
> +except StopIteration:
> +return None
> +return _prepare_hunk(lines)
> +
> +def _cmpdiff(leftctx, rightctx):
> +"""return True if both ctx introduce the "same diff"
> +
> +This is a first and basic implementation, with many shortcoming.
> +"""
> +
> +# Leftctx or right ctx might be filtered, so we need to use the contexts
> +# with an unfiltered repository to safely compute the diff
> +leftunfi = leftctx._repo.unfiltered()[leftctx.rev()]
> +leftdiff = leftunfi.diff(git=1)
> +rightunfi = rightctx._repo.unfiltered()[rightctx.rev()]
> +rightdiff = rightunfi.diff(git=1)
> +
> +left, right = (0, 0)
> +while None not in (left, right):
> +left = _getdifflines(leftdiff)
> +right = _getdifflines(rightdiff)
> +
> +if left != right:
> +return False
> +return True
> +
>  def geteffectflag(relation):
>  """ From an obs-marker relation, compute what changed between the
>  predecessor and the successor.
> @@ -604,4 +645,12 @@
>  if ctxmeta != srcmeta:
>  effects |= METACHANGED
>  
> +# Check if at least one of the parent has changed
> +if changectx.parents() != source.parents():
> +effects |= PARENTCHANGED
> +
> +# Check if the diff has changed
> +if not _cmpdiff(source, changectx):
> +effects |= DIFFCHANGED
> +
>  return effects
> diff -r 6a40d87dfedc -r 449fc1c748c6 tests/test-obsmarkers-effectflag.t
> --- a/tests/test-obsmarkers-effectflag.tThu Jul 06 14:58:44 2017 +0200
> +++ b/tests/test-obsmarkers-effectflag.tThu Jul 06 15:00:07 2017 +0200
> @@ -95,7 +95,7 @@
>  
>$ hg debugobsolete --rev .
>d6f4d8b8d3c8cde990f13915bced7f92ce1cc54f 0 
> {ebfe0333e0d96f68a917afd97c0a0af87f1c3b5f} (Thu Jan 01 00:00:00 1970 +) 
> {'ef1': '0', 'user': 'test'}
> -  ebfe0333e0d96f68a917afd97c0a0af87f1c3b5f 
> 75781fdbdbf58a987516b00c980bccda1e9ae588 0 (Thu Jan 01 00:00:00 1970 +) 
> {'ef1': '0', 'user': 'test'}
> +  

Re: [PATCH 7 of 7] histedit: use scmutil.cleanupnodes (BC)

[Jun Wu]

Excerpts from Martin von Zweigbergk's message of 2017-07-09 12:54:50 -0700:
> I agree that amend and metaedit should not need it. I could be made an
> option, since at least rebase and histedit seem to need it. Anyway, we
> can always deal with that when someone files a bug report.

An option sounds good to me. I'll send a follow-up once the number of my
pending patches decreases a bit.

> >
> >> > +scmutil.cleanupnodes(repo, mapping, 'histedit')
> >> >
> >> >  state.clear()
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] cleanupnode: do not use generator for node mapping

[Martin von Zweigbergk via Mercurial-devel]

On Sun, Jul 9, 2017 at 10:06 AM, Jun Wu  wrote:
> Looks good to me. Thanks for the cleanup!

Yep. Queued, thanks!

>
> Excerpts from Boris Feld's message of 2017-07-09 18:49:04 +0200:
>> # HG changeset patch
>> # User Octobus 
>> # Date 1499605879 -7200
>> #  Sun Jul 09 15:11:19 2017 +0200
>> # Node ID a6ed7f0670010e5a6551d736b00dc0bf7367bba8
>> # Parent  4672db164c986da4442bd864cd044512d975c3f2
>> # EXP-Topic fix-cleanup
>> cleanupnode: do not use generator for node mapping
>>
>> The 'successors' part of the mappings used of be a tuple. This avoid issue 
>> from
>> code consuming the generator "by mistake". For example, an extension 
>> inspecting the
>> mapping content used to be able to iterate over the successors mapping 
>> without
>> consequence.
>>
>> Since the mapping are small we do not expect any performance impact we use 
>> tuple
>> again for this.
>>
>> diff -r 4672db164c98 -r a6ed7f067001 mercurial/scmutil.py
>> --- a/mercurial/scmutil.pySat Jun 24 15:29:42 2017 -0700
>> +++ b/mercurial/scmutil.pySun Jul 09 15:11:19 2017 +0200
>> @@ -638,7 +638,7 @@
>>  isobs = unfi.obsstore.successors.__contains__
>>  torev = unfi.changelog.rev
>>  sortfunc = lambda ns: torev(ns[0])
>> -rels = [(unfi[n], (unfi[m] for m in s))
>> +rels = [(unfi[n], tuple(unfi[m] for m in s))
>>  for n, s in sorted(mapping.items(), key=sortfunc)
>>  if s or not isobs(n)]
>>  obsolete.createmarkers(repo, rels, operation=operation)
> ___
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 7 of 7] histedit: use scmutil.cleanupnodes (BC)

[Martin von Zweigbergk via Mercurial-devel]

On Sun, Jul 9, 2017 at 12:21 PM, Jun Wu  wrote:
> Excerpts from Martin von Zweigbergk's message of 2017-07-09 12:04:11 -0700:
>> On Sat, Jul 8, 2017 at 4:51 PM, Jun Wu  wrote:
>> > diff --git a/hgext/histedit.py b/hgext/histedit.py
>> > --- a/hgext/histedit.py
>> > +++ b/hgext/histedit.py
>> > @@ -1182,5 +1182,9 @@ def _finishhistedit(ui, repo, state):
>> >  mapping[n] = ()
>> >
>> > -safecleanupnode(ui, repo, mapping)
>> > +# remove entries about unknown nodes
>> > +nodemap = repo.unfiltered().changelog.nodemap
>> > +mapping = {k: v for k, v in mapping.items()
>> > +   if k in nodemap and all(n in nodemap for n in v)}
>>
>> I suppose these few lines could potentially move into cleanupnodes?
>> IIUC, it's to prevent crashing if you had stripped nodes during
>> histedit. The same can be done during rebase (while stopped to resolve
>> conflicts) and
>
> The current rebase and amend do not have the "remove unknown node" logic
> (might be a bug for rebase, but amend does not seem to need this). Since
> it's only useful in a subset of all cases, and commands like metaedit,
> split, amend, etc do not need it, I think it's cleaner to require explicit
> node removal from the callsite, so when the callsite made a mistake, we can
> catch it instead of ignoring it silently.

I agree that amend and metaedit should not need it. I could be made an
option, since at least rebase and histedit seem to need it. Anyway, we
can always deal with that when someone files a bug report.

>
>> > +scmutil.cleanupnodes(repo, mapping, 'histedit')
>> >
>> >  state.clear()
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 6 of 7] histedit: unify strip backup files on success (BC)

[Jun Wu]

Excerpts from Augie Fackler's message of 2017-07-09 15:28:15 -0400:
> 
> > On Jul 9, 2017, at 2:24 PM, Martin von Zweigbergk  
> > wrote:
> > 
> > On Sat, Jul 8, 2017 at 4:51 PM, Jun Wu  wrote:
> >> # HG changeset patch
> >> # User Jun Wu 
> >> # Date 1499557831 25200
> >> #  Sat Jul 08 16:50:31 2017 -0700
> >> # Node ID d75e65724d2bbcf17fcaaad705e463e84a3de5d2
> >> # Parent  9a325ae88021e0e02a87ef1ae6baa8a199405140
> >> # Available At https://bitbucket.org/quark-zju/hg-draft 
> >> #  hg pull https://bitbucket.org/quark-zju/hg-draft  -r 
> >> d75e65724d2b
> >> histedit: unify strip backup files on success (BC)
> >> 
> >> Previously we wrote two different strip backup files on success. This patch
> >> unifies them. It will make scmutil.cleanupnodes migration more smooth.
> > 
> > Makes sense to me. I'm queuing the series, thanks.
> > 
> > Augie, do you remember if there was a reason to keep them separate?
> 
> The only reason was structural in the code: histedit has “intermediate”
> nodes which are an implementation detail, and “old” nodes, which are the
> precursors to the end state. Internally, the lists are separate because
> you always want to reap “intermediate" nodes, but might want to preserve
> “old” and in --keep or --abort cases.
> 
> It might be nice to keep them separate but have one be marked
> “histedit-internal” and “histedit-before” or something.

Good point.

Going through the "delayedstrip" code path, it could be a bit tricky to
separate backup files [1]. I wish we can count on in-memory-merge work so
those intermediate nodes disappear automatically. But that won't happen
soon. Maybe "delayedstrip" API could be changed, I've added this to my
backlog to resolve.

[1]: 
https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-June/100404.html
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 6 of 7] histedit: unify strip backup files on success (BC)

[Augie Fackler]

> On Jul 9, 2017, at 2:24 PM, Martin von Zweigbergk  
> wrote:
> 
> On Sat, Jul 8, 2017 at 4:51 PM, Jun Wu  wrote:
>> # HG changeset patch
>> # User Jun Wu 
>> # Date 1499557831 25200
>> #  Sat Jul 08 16:50:31 2017 -0700
>> # Node ID d75e65724d2bbcf17fcaaad705e463e84a3de5d2
>> # Parent  9a325ae88021e0e02a87ef1ae6baa8a199405140
>> # Available At https://bitbucket.org/quark-zju/hg-draft
>> #  hg pull https://bitbucket.org/quark-zju/hg-draft -r 
>> d75e65724d2b
>> histedit: unify strip backup files on success (BC)
>> 
>> Previously we wrote two different strip backup files on success. This patch
>> unifies them. It will make scmutil.cleanupnodes migration more smooth.
> 
> Makes sense to me. I'm queuing the series, thanks.
> 
> Augie, do you remember if there was a reason to keep them separate?

The only reason was structural in the code: histedit has “intermediate” nodes 
which are an implementation detail, and “old” nodes, which are the precursors 
to the end state. Internally, the lists are separate because you always want to 
reap “intermediate" nodes, but might want to preserve “old” and in --keep or 
--abort cases.

It might be nice to keep them separate but have one be marked 
“histedit-internal” and “histedit-before” or something.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 7 of 7] histedit: use scmutil.cleanupnodes (BC)

[Jun Wu]

Excerpts from Martin von Zweigbergk's message of 2017-07-09 12:04:11 -0700:
> On Sat, Jul 8, 2017 at 4:51 PM, Jun Wu  wrote:
> > diff --git a/hgext/histedit.py b/hgext/histedit.py
> > --- a/hgext/histedit.py
> > +++ b/hgext/histedit.py
> > @@ -1182,5 +1182,9 @@ def _finishhistedit(ui, repo, state):
> >  mapping[n] = ()
> >
> > -safecleanupnode(ui, repo, mapping)
> > +# remove entries about unknown nodes
> > +nodemap = repo.unfiltered().changelog.nodemap
> > +mapping = {k: v for k, v in mapping.items()
> > +   if k in nodemap and all(n in nodemap for n in v)}
> 
> I suppose these few lines could potentially move into cleanupnodes?
> IIUC, it's to prevent crashing if you had stripped nodes during
> histedit. The same can be done during rebase (while stopped to resolve
> conflicts) and

The current rebase and amend do not have the "remove unknown node" logic
(might be a bug for rebase, but amend does not seem to need this). Since
it's only useful in a subset of all cases, and commands like metaedit,
split, amend, etc do not need it, I think it's cleaner to require explicit
node removal from the callsite, so when the callsite made a mistake, we can
catch it instead of ignoring it silently.

> > +scmutil.cleanupnodes(repo, mapping, 'histedit')
> >
> >  state.clear()
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 4 of 6] rebase: remove revprecursor and revpruned states (BC)

[Jun Wu]

# HG changeset patch
# User Jun Wu 
# Date 1499571514 25200
#  Sat Jul 08 20:38:34 2017 -0700
# Node ID e633aabf34125ef5d5453c211cee7e63654346d2
# Parent  8ba4291b60f1b98903bdac9f18057ae8b7f64d58
# Available At https://bitbucket.org/quark-zju/hg-draft
#  hg pull https://bitbucket.org/quark-zju/hg-draft -r e633aabf3412
rebase: remove revprecursor and revpruned states (BC)

Those states are no longer necessary for rebase to function properly. Remove
them to make the code cleaner.

Marked as BC because in a corner case where working parent is obsoleted, and
is skipped for rebase, we no longer move working parent after rebase
completes. That is better since if working parent is obsoleted, it should be
the user moving working parent back there (after a rebase) explicitly, in
that case, we shouldn't move working parent again.

diff --git a/hgext/rebase.py b/hgext/rebase.py
--- a/hgext/rebase.py
+++ b/hgext/rebase.py
@@ -63,8 +63,8 @@ revtodo = -1
 nullmerge = -2
 revignored = -3
-# successor in rebase destination
-revprecursor = -4
-# plain prune (no successor)
-revpruned = -5
+
+# legacy revstates no longer needed in current code
+# -4: revprecursor, -5: revpruned
+legacystates = {'-4', '-5'}
 
 cmdtable = {}
@@ -232,6 +232,7 @@ class rebaseruntime(object):
 else:
 oldrev, newrev = l.split(':')
-if newrev in (str(nullmerge), str(revignored),
-  str(revprecursor), str(revpruned)):
+if newrev in legacystates:
+continue
+if newrev in (str(nullmerge), str(revignored)):
 state[repo[oldrev].rev()] = int(newrev)
 elif newrev == nullid:
@@ -309,7 +310,4 @@ class rebaseruntime(object):
  self.state, activebookmark=self.activebookmark)
 
-obsrevs = (r for r, st in self.state.items() if st == revprecursor)
-self._handleskippingobsolete(self.state.keys(), obsrevs, self.dest)
-
 def _preparenewrebase(self, dest, rebaseset):
 if dest is None:
@@ -446,8 +444,4 @@ class rebaseruntime(object):
 elif self.state[rev] == revignored:
 ui.status(_('not rebasing ignored %s\n') % desc)
-elif self.state[rev] == revprecursor:
-pass
-elif self.state[rev] == revpruned:
-pass
 else:
 ui.status(_('already rebased %s as %s\n') %
@@ -495,7 +489,5 @@ class rebaseruntime(object):
 # (we do this before stripping)
 newwd = self.state.get(self.originalwd, self.originalwd)
-if newwd == revprecursor:
-newwd = self.obsoletenotrebased[self.originalwd]
-elif newwd < 0:
+if newwd < 0:
 # original directory is a parent of rebase set root or ignored
 newwd = self.originalwd
@@ -1327,5 +1319,5 @@ def buildstate(repo, dest, rebaseset, co
 if succ is None:
 msg = _('note: not rebasing %s, it has no successor\n') % desc
-state[r] = revpruned
+del state[r]
 else:
 destctx = unfi[succ]
@@ -1334,5 +1326,5 @@ def buildstate(repo, dest, rebaseset, co
 msg = (_('note: not rebasing %s, already in destination as %s\n')
% (desc, destdesc))
-state[r] = revprecursor
+del state[r]
 repo.ui.status(msg)
 return originalwd, dest.rev(), state
diff --git a/tests/test-rebase-obsolete.t b/tests/test-rebase-obsolete.t
--- a/tests/test-rebase-obsolete.t
+++ b/tests/test-rebase-obsolete.t
@@ -970,8 +970,8 @@ equivalents in destination
   $ hg rebase -r 2 -d 3 --config experimental.evolution.track-operation=1
   note: not rebasing 2:1e9a3c00cbe9 "b" (mybook), already in destination as 
3:be1832deae9a "b"
-Check that working directory was updated to rev 3 although rev 2 was skipped
+Check that working directory was not updated to rev 3 because rev 2 was skipped
 during the rebase operation
   $ hg log -r .
-  3:be1832deae9a b (no-eol)
+  2:1e9a3c00cbe9 b (no-eol)
 
 Check that bookmark was not moved to rev 3 if rev 2 was skipped during the
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 6 of 6] rebase: remove revprecursor and revpruned states

[Jun Wu]

# HG changeset patch
# User Jun Wu 
# Date 1499571514 25200
#  Sat Jul 08 20:38:34 2017 -0700
# Node ID 22d54f1e4c09d8fb68ea897afcdd0b705db94b5a
# Parent  49d1672e7567f2a16c5ce9afa64babe08891718f
# Available At https://bitbucket.org/quark-zju/hg-draft
#  hg pull https://bitbucket.org/quark-zju/hg-draft -r 22d54f1e4c09
rebase: remove revprecursor and revpruned states

They are no longer necessary to make rebase behavior correct. Therefore
remove them to make the code cleaner and easier to reason about.

diff --git a/hgext/rebase.py b/hgext/rebase.py
--- a/hgext/rebase.py
+++ b/hgext/rebase.py
@@ -61,10 +61,8 @@ templateopts = cmdutil.templateopts
 # Indicates that a revision needs to be rebased
 revtodo = -1
-nullmerge = -2
-revignored = -3
 
 # legacy revstates no longer needed in current code
-# -4: revprecursor, -5: revpruned
-legacystates = {'-4', '-5'}
+# -2: nullmerge, -3: revignored, -4: revprecursor, -5: revpruned
+legacystates = {'-2', '-3', '-4', '-5'}
 
 cmdtable = {}
@@ -234,6 +232,4 @@ class rebaseruntime(object):
 if newrev in legacystates:
 continue
-if newrev in (str(nullmerge), str(revignored)):
-state[repo[oldrev].rev()] = int(newrev)
 elif newrev == nullid:
 state[repo[oldrev].rev()] = revtodo
@@ -440,8 +436,4 @@ class rebaseruntime(object):
 self.state[rev] = p1
 ui.debug('next revision set to %s\n' % p1)
-elif self.state[rev] == nullmerge:
-pass
-elif self.state[rev] == revignored:
-pass
 else:
 ui.status(_('already rebased %s as %s\n') %
@@ -464,5 +456,5 @@ class rebaseruntime(object):
 for rebased in self.state:
 if rebased not in self.skipped and\
-   self.state[rebased] > nullmerge:
+   self.state[rebased] >= revtodo:
 commitmsg += '\n* %s' % repo[rebased].description()
 editopt = True
@@ -480,5 +472,5 @@ class rebaseruntime(object):
 newrev = repo[newnode].rev()
 for oldrev in self.state.iterkeys():
-if self.state[oldrev] > nullmerge:
+if self.state[oldrev] >= revtodo:
 self.state[oldrev] = newrev
 
@@ -1231,5 +1223,4 @@ def buildstate(repo, dest, rebaseset, co
 roots.sort()
 state = dict.fromkeys(rebaseset, revtodo)
-detachset = set()
 emptyrebase = True
 for root in roots:
@@ -1253,45 +1244,4 @@ def buildstate(repo, dest, rebaseset, co
 emptyrebase = False
 repo.ui.debug('rebase onto %s starting from %s\n' % (dest, root))
-# Rebase tries to turn  into a parent of  while
-# preserving the number of parents of rebased changesets:
-#
-# - A changeset with a single parent will always be rebased as a
-#   changeset with a single parent.
-#
-# - A merge will be rebased as merge unless its parents are both
-#   ancestors of  or are themselves in the rebased set and
-#   pruned while rebased.
-#
-# If one parent of  is an ancestor of , the rebased
-# version of this parent will be . This is always true with
-# --base option.
-#
-# Otherwise, we need to *replace* the original parents with
-# . This "detaches" the rebased set from its former location
-# and rebases it onto . Changes introduced by ancestors of
-#  not common with  (the detachset, marked as
-# nullmerge) are "removed" from the rebased changesets.
-#
-# - If  has a single parent, set it to .
-#
-# - If  is a merge, we cannot decide which parent to
-#   replace, the rebase operation is not clearly defined.
-#
-# The table below sums up this behavior:
-#
-# +--+--+-+
-# |  | one parent   |  merge  |
-# +--+--+-+
-# | parent in| new parent is  | parents in :: are |
-# | :: |  | remapped to   |
-# +--+--+-+
-# | unrelated source | new parent is  | ambiguous, abort|
-# +--+--+-+
-#
-# The actual abort is handled by `defineparents`
-if len(root.parents()) <= 1:
-# ancestors of  not ancestors of 
-detachset.update(repo.changelog.findmissingrevs([commonbase.rev()],
-[root.rev()]))
 if emptyrebase:
 return None

[PATCH 1 of 6] rebase: rewrite defineparents

[Jun Wu]

# HG changeset patch
# User Jun Wu 
# Date 1499568916 25200
#  Sat Jul 08 19:55:16 2017 -0700
# Node ID 5a13cf09bacd7912367c4f15b73920fadaf0a457
# Parent  1e872b08a4e93b21e60cba695e4022bf71d4acf4
# Available At https://bitbucket.org/quark-zju/hg-draft
#  hg pull https://bitbucket.org/quark-zju/hg-draft -r 5a13cf09bacd
rebase: rewrite defineparents

The old code has too many tech debts (like outdated comments, confusing
assertions, etc) and is very error-prone to be improved. This patch rewrites
"defineparents" to make future changes easier to reason about.

diff --git a/hgext/rebase.py b/hgext/rebase.py
--- a/hgext/rebase.py
+++ b/hgext/rebase.py
@@ -67,5 +67,4 @@ revprecursor = -4
 # plain prune (no successor)
 revpruned = -5
-revskipped = (revignored, revprecursor, revpruned)
 
 cmdtable = {}
@@ -391,8 +390,5 @@ class rebaseruntime(object):
 ui.progress(_("rebasing"), pos, ("%d:%s" % (rev, ctx)),
 _('changesets'), total)
-p1, p2, base = defineparents(repo, rev, self.dest,
- self.state,
- self.destancestors,
- self.obsoletenotrebased)
+p1, p2, base = defineparents(repo, rev, self.dest, self.state)
 self.storestatus()
 storecollapsemsg(repo, self.collapsemsg)
@@ -464,7 +460,5 @@ class rebaseruntime(object):
 if self.collapsef and not self.keepopen:
 p1, p2, _base = defineparents(repo, min(self.state),
-  self.dest, self.state,
-  self.destancestors,
-  self.obsoletenotrebased)
+  self.dest, self.state)
 editopt = opts.get('edit')
 editform = 'rebase.collapse'
@@ -884,13 +878,4 @@ def rebasenode(repo, rev, p1, base, stat
 return stats
 
-def nearestrebased(repo, rev, state):
-"""return the nearest ancestors of rev in the rebase result"""
-rebased = [r for r in state if state[r] > nullmerge]
-candidates = repo.revs('max(%ld  and (::%d))', rebased, rev)
-if candidates:
-return state[candidates.first()]
-else:
-return None
-
 def _checkobsrebase(repo, ui, rebaseobsrevs, rebasesetrevs, rebaseobsskipped):
 """
@@ -916,105 +901,159 @@ def _checkobsrebase(repo, ui, rebaseobsr
 raise error.Abort(msg % (",".join(divhashes),), hint=h)
 
-def defineparents(repo, rev, dest, state, destancestors,
-  obsoletenotrebased):
-'Return the new parent relationship of the revision that will be rebased'
-parents = repo[rev].parents()
-p1 = p2 = nullrev
-rp1 = None
+def adjusteddest(repo, rev, prev, dest, state):
+"""adjust rebase destination given the current rebase state
+
+rev is being rebased, prev is one of its parent.
+
+For example, when rebasing E to F:
 
-p1n = parents[0].rev()
-if p1n in destancestors:
-p1 = dest
-elif p1n in state:
-if state[p1n] == nullmerge:
-p1 = dest
-elif state[p1n] in revskipped:
-p1 = nearestrebased(repo, p1n, state)
-if p1 is None:
-p1 = dest
-else:
-p1 = state[p1n]
-else: # p1n external
-p1 = dest
-p2 = p1n
+B' <- written during the rebase
+|
+F <- original destination of B, E
+|
+| E <- rev, which is being rebased
+| |
+| D <- prev, one parent of rev being checked
+| |
+| x <- skipped
+| |
+| C <- rebased as C'   C' <- written during the rebase
+| ||
+| B <- rebased as B'   G <- destination of C, separate subgraph
+|/
+A
+
+The destination will be adjusted from F to B'.
+
+Besides, adjust dest according to existing rebase information. For example,
+
+  B C DB wants to be rebased on top of C, C wants to be rebased on top
+   \|/ of D.
+A
 
-if len(parents) == 2 and parents[1].rev() not in destancestors:
-p2n = parents[1].rev()
-# interesting second parent
-if p2n in state:
-if p1 == dest: # p1n in destancestors or external
-p1 = state[p2n]
-if p1 == revprecursor:
-rp1 = obsoletenotrebased[p2n]
-elif state[p2n] in revskipped:
-p2 = nearestrebased(repo, p2n, state)
-if p2 is None:
-# no ancestors rebased yet, detach
-p2 = dest
-else:
-p2 = state[p2n]
-else: # p2n external
-if p2 != nullrev: # p1n external too => rev is a merged revision
-raise error.Abort(_('cannot use revision %d as base, 

[PATCH 2 of 6] rebase: extract ctx description logic to a function

[Jun Wu]

# HG changeset patch
# User Jun Wu 
# Date 1499569552 25200
#  Sat Jul 08 20:05:52 2017 -0700
# Node ID 85aa943fb16e8fe9f8fed13871a28bdd20ff31b0
# Parent  5a13cf09bacd7912367c4f15b73920fadaf0a457
# Available At https://bitbucket.org/quark-zju/hg-draft
#  hg pull https://bitbucket.org/quark-zju/hg-draft -r 85aa943fb16e
rebase: extract ctx description logic to a function

The function will be used in a later patch.

diff --git a/hgext/rebase.py b/hgext/rebase.py
--- a/hgext/rebase.py
+++ b/hgext/rebase.py
@@ -123,4 +123,14 @@ def _revsetdestrebase(repo, subset, x):
 return subset & smartset.baseset([_destrebase(repo, sourceset)])
 
+def _ctxdesc(ctx):
+"""short description for a context"""
+desc = '%d:%s "%s"' % (ctx.rev(), ctx,
+   ctx.description().split('\n', 1)[0])
+repo = ctx.repo()
+names = repo.nodetags(ctx.node()) + repo.nodebookmarks(ctx.node())
+if names:
+desc += ' (%s)' % ' '.join(names)
+return desc
+
 class rebaseruntime(object):
 """This class is a container for rebase runtime state"""
@@ -378,9 +388,5 @@ class rebaseruntime(object):
 for rev in sortedrevs:
 ctx = repo[rev]
-desc = '%d:%s "%s"' % (ctx.rev(), ctx,
-   ctx.description().split('\n', 1)[0])
-names = repo.nodetags(ctx.node()) + repo.nodebookmarks(ctx.node())
-if names:
-desc += ' (%s)' % ' '.join(names)
+desc = _ctxdesc(ctx)
 if self.state[rev] == rev:
 ui.status(_('already rebased %s\n') % desc)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 3 of 6] rebase: move obsoleted not rebased messages earlier (BC)

[Jun Wu]

# HG changeset patch
# User Jun Wu 
# Date 1499570073 25200
#  Sat Jul 08 20:14:33 2017 -0700
# Node ID 8ba4291b60f1b98903bdac9f18057ae8b7f64d58
# Parent  85aa943fb16e8fe9f8fed13871a28bdd20ff31b0
# Available At https://bitbucket.org/quark-zju/hg-draft
#  hg pull https://bitbucket.org/quark-zju/hg-draft -r 8ba4291b60f1
rebase: move obsoleted not rebased messages earlier (BC)

A later patch will clean up those states. This patch moves the messages
earlier.

Marked as BC since the order of message has changed.

diff --git a/hgext/rebase.py b/hgext/rebase.py
--- a/hgext/rebase.py
+++ b/hgext/rebase.py
@@ -447,12 +447,7 @@ class rebaseruntime(object):
 ui.status(_('not rebasing ignored %s\n') % desc)
 elif self.state[rev] == revprecursor:
-destctx = repo[self.obsoletenotrebased[rev]]
-descdest = '%d:%s "%s"' % (destctx.rev(), destctx,
-   destctx.description().split('\n', 1)[0])
-msg = _('note: not rebasing %s, already in destination as 
%s\n')
-ui.status(msg % (desc, descdest))
+pass
 elif self.state[rev] == revpruned:
-msg = _('note: not rebasing %s, it has no successor\n')
-ui.status(msg % desc)
+pass
 else:
 ui.status(_('already rebased %s as %s\n') %
@@ -1326,9 +1321,19 @@ def buildstate(repo, dest, rebaseset, co
 for ignored in set(rebasedomain) - set(rebaseset):
 state[ignored] = revignored
+unfi = repo.unfiltered()
 for r in obsoletenotrebased:
-if obsoletenotrebased[r] is None:
+desc = _ctxdesc(unfi[r])
+succ = obsoletenotrebased[r]
+if succ is None:
+msg = _('note: not rebasing %s, it has no successor\n') % desc
 state[r] = revpruned
 else:
+destctx = unfi[succ]
+destdesc = '%d:%s "%s"' % (destctx.rev(), destctx,
+   destctx.description().split('\n', 1)[0])
+msg = (_('note: not rebasing %s, already in destination as %s\n')
+   % (desc, destdesc))
 state[r] = revprecursor
+repo.ui.status(msg)
 return originalwd, dest.rev(), state
 
diff --git a/tests/test-rebase-obsolete.t b/tests/test-rebase-obsolete.t
--- a/tests/test-rebase-obsolete.t
+++ b/tests/test-rebase-obsolete.t
@@ -205,6 +205,6 @@ More complex case where part of the reba
   
   $ hg rebase --source 'desc(B)' --dest 'tip' --config 
experimental.rebaseskipobsolete=True
+  note: not rebasing 9:08483444fef9 "D", already in destination as 
11:4596109a6a43 "D"
   rebasing 8:8877864f1edb "B"
-  note: not rebasing 9:08483444fef9 "D", already in destination as 
11:4596109a6a43 "D"
   rebasing 10:5ae4c968c6ac "C"
   $ hg debugobsolete
@@ -712,6 +712,6 @@ Even when the chain include missing node
   $ hg debugobsolete `hg log -r 7 -T '{node}\n'` --config 
experimental.evolution=all
   $ hg rebase -d 6 -r "4::"
+  note: not rebasing 7:360bbaa7d3ce "O", it has no successor
   rebasing 4:ff2c4d47b71d "C"
-  note: not rebasing 7:360bbaa7d3ce "O", it has no successor
   rebasing 8:8d47583e023f "P" (tip)
 
@@ -878,4 +878,5 @@ Create the changes that we will rebase
   ~
   $ hg rebase -r ".^^ + .^ + ." -d 19
+  note: not rebasing 21:8b31da3c4919 "dummy change", already in destination as 
19:601db7a18f51 "dummy change successor"
   rebasing 20:b82fb57ea638 "willconflict second version"
   merging willconflict
@@ -889,5 +890,4 @@ Create the changes that we will rebase
   $ hg rebase --continue
   rebasing 20:b82fb57ea638 "willconflict second version"
-  note: not rebasing 21:8b31da3c4919 "dummy change", already in destination as 
19:601db7a18f51 "dummy change successor"
   rebasing 22:7bdc8a87673d "dummy change" (tip)
   $ cd ..
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 5 of 6] rebase: remove messages for nullmerge and revignored (BC)

[Jun Wu]

# HG changeset patch
# User Jun Wu 
# Date 1499571514 25200
#  Sat Jul 08 20:38:34 2017 -0700
# Node ID 49d1672e7567f2a16c5ce9afa64babe08891718f
# Parent  e633aabf34125ef5d5453c211cee7e63654346d2
# Available At https://bitbucket.org/quark-zju/hg-draft
#  hg pull https://bitbucket.org/quark-zju/hg-draft -r 49d1672e7567
rebase: remove messages for nullmerge and revignored (BC)

These states will be removed to make the code cleaner and more robust.
Remove their messages first to make review easier.

diff --git a/hgext/rebase.py b/hgext/rebase.py
--- a/hgext/rebase.py
+++ b/hgext/rebase.py
@@ -441,7 +441,7 @@ class rebaseruntime(object):
 ui.debug('next revision set to %s\n' % p1)
 elif self.state[rev] == nullmerge:
-ui.debug('ignoring null merge rebase of %s\n' % rev)
+pass
 elif self.state[rev] == revignored:
-ui.status(_('not rebasing ignored %s\n') % desc)
+pass
 else:
 ui.status(_('already rebased %s as %s\n') %
diff --git a/tests/test-rebase-conflicts.t b/tests/test-rebase-conflicts.t
--- a/tests/test-rebase-conflicts.t
+++ b/tests/test-rebase-conflicts.t
@@ -221,8 +221,4 @@ Check that the right ancestors is used w
   rebase onto 4bc80088dc6b starting from e31216eec445
   rebase status stored
-  ignoring null merge rebase of 3
-  ignoring null merge rebase of 4
-  ignoring null merge rebase of 6
-  ignoring null merge rebase of 8
   rebasing 9:e31216eec445 "more changes to f1"
future parents are 2 and -1
@@ -399,5 +395,4 @@ Test rebase with obsstore turned on and 
   $ hg rebase --continue --config experimental.evolution=none
   rebasing 1:112478962961 "B" (B)
-  not rebasing ignored 2:26805aba1e60 "C" (C)
   rebasing 3:f585351a92f8 "D" (D)
   warning: orphaned descendants detected, not stripping 112478962961
diff --git a/tests/test-rebase-obsolete.t b/tests/test-rebase-obsolete.t
--- a/tests/test-rebase-obsolete.t
+++ b/tests/test-rebase-obsolete.t
@@ -450,5 +450,4 @@ Test multiple root handling
   rebasing 9:cf44d2f5a9f4 "D"
   rebasing 7:02de42196ebe "H"
-  not rebasing ignored 10:7c6027df6a99 "B"
   rebasing 11:0d8f238b634c "C" (tip)
   $ hg log -G
@@ -520,5 +519,4 @@ test on rebase dropping a merge
   rebasing 3:32af7686d403 "D"
   rebasing 7:02de42196ebe "H"
-  not rebasing ignored 8:53a6a128b2b7 "M"
   rebasing 9:4bde274eefcf "I" (tip)
   $ hg log -G
diff --git a/tests/test-rebase-partial.t b/tests/test-rebase-partial.t
--- a/tests/test-rebase-partial.t
+++ b/tests/test-rebase-partial.t
@@ -82,5 +82,4 @@ the hole (B below), not on top of the de
   > EOF
   already rebased 1:112478962961 "B" (B)
-  not rebasing ignored 2:26805aba1e60 "C" (C)
   rebasing 3:f585351a92f8 "D" (D tip)
   o  4: D
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Next plans on radixlink and hash-preserving obsstore

[Jun Wu]

I'd like to comment on next plans of radixlink and hash-preserving obsstore,
since they look promising and I didn't send followups promptly.

The previous radixlink code works, but comparing to the radixtree in
revlog.c, the radixlink series I have sent is 4x larger in space usage, and
70% slower on insertion. I want to improve these areas.

A key factor for revlog.c to be space efficient is it uses revision numbers
so it does not store nodes directly, and it does not need to store values
(it's a "set" instead of a "map"). I'd like to reuse the "int as key"
strategy. So a uint32 could represent a hash. That reduces space usage
significantly.

Using offsets of strings in obsstore is possible but requires some
non-trivial changes on top of the current obsstore. Since I do want to
change obsstore for hash-preserving, I'm not very interested in improving
the current obsstore for now.

Therefore the rough plan is to have a new obsstore format with indexing
(radixlink) in a native implementation. For "native", I'm also experimenting
with Rust. I have some Rust code for the revised radixlink and its
performance is on par with revlog.c.

The current plans are: 

  1. To reduce the review burden from the community about experimental
 stuff, and to iterate quickly, I'm going to prototype in fb-hgext
 first, utilizing Facebook's Rust review resources. Review discussions
 will be visible on phab.mercurial-scm.org so you're welcomed to join if
 you have interest.
  2. Once we have some confidence about file format (like in 3 weeks or so),
 we will start a discussion about final file format and APIs in the
 mailing list.  From there we can figure out the upstream approach, like
 having a Python fallback, choosing between Rust and C, having an
 upgrade path, etc.

Meanwhile, I'll continue contribute other changes.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 7 of 7] histedit: use scmutil.cleanupnodes (BC)

[Martin von Zweigbergk via Mercurial-devel]

On Sat, Jul 8, 2017 at 4:51 PM, Jun Wu  wrote:
> diff --git a/hgext/histedit.py b/hgext/histedit.py
> --- a/hgext/histedit.py
> +++ b/hgext/histedit.py
> @@ -1182,5 +1182,9 @@ def _finishhistedit(ui, repo, state):
>  mapping[n] = ()
>
> -safecleanupnode(ui, repo, mapping)
> +# remove entries about unknown nodes
> +nodemap = repo.unfiltered().changelog.nodemap
> +mapping = {k: v for k, v in mapping.items()
> +   if k in nodemap and all(n in nodemap for n in v)}

I suppose these few lines could potentially move into cleanupnodes?
IIUC, it's to prevent crashing if you had stripped nodes during
histedit. The same can be done during rebase (while stopped to resolve
conflicts) and

> +scmutil.cleanupnodes(repo, mapping, 'histedit')
>
>  state.clear()
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 6 of 7] histedit: unify strip backup files on success (BC)

[Martin von Zweigbergk via Mercurial-devel]

On Sat, Jul 8, 2017 at 4:51 PM, Jun Wu  wrote:
> # HG changeset patch
> # User Jun Wu 
> # Date 1499557831 25200
> #  Sat Jul 08 16:50:31 2017 -0700
> # Node ID d75e65724d2bbcf17fcaaad705e463e84a3de5d2
> # Parent  9a325ae88021e0e02a87ef1ae6baa8a199405140
> # Available At https://bitbucket.org/quark-zju/hg-draft
> #  hg pull https://bitbucket.org/quark-zju/hg-draft -r 
> d75e65724d2b
> histedit: unify strip backup files on success (BC)
>
> Previously we wrote two different strip backup files on success. This patch
> unifies them. It will make scmutil.cleanupnodes migration more smooth.

Makes sense to me. I'm queuing the series, thanks.

Augie, do you remember if there was a reason to keep them separate?
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 02 of 14] cache: introduce a changelogsourcebase class

[Boris Feld]

On Sun, 2017-07-09 at 19:52 +0200, Boris Feld wrote:
> # HG changeset patch
> # User Boris Feld 
> # Date 1499458552 -7200
> #  Fri Jul 07 22:15:52 2017 +0200
> # Node ID 8b71290526ddb77f157e075191dd748793d85601
> # Parent  6edb62505c697329de034c2fdc47befd5896f31f
> # EXP-Topic obs-cache
> cache: introduce a changelogsourcebase class
> 
Sorry, my battery has discharged while sending the series.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 11 of 14] obscache: instantiate the cache and keep it warm

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1495198006 -7200
#  Fri May 19 14:46:46 2017 +0200
# Node ID f8953ed43f8d1b146dcff688030133f0d6123a49
# Parent  985d753d4f5799f2a332140adedb06efd465d62b
# EXP-Topic obs-cache
obscache: instantiate the cache and keep it warm

We are not using it yet, but we make sure we have a cache and that we keep it up
to date after each transaction.

The two "warning: ignoring unknown working parent" output in the tests are
caused by 'blackbox' accessing the dirstate during the strip. I've not found an
easy way to work around this so I kept them as they are harmless.

diff -r 985d753d4f57 -r f8953ed43f8d mercurial/localrepo.py
--- a/mercurial/localrepo.pyFri May 19 14:44:22 2017 +0200
+++ b/mercurial/localrepo.pyFri May 19 14:46:46 2017 +0200
@@ -1274,6 +1274,10 @@
 self.ui.debug('updating the branch cache\n')
 branchmap.updatecache(self.filtered('served'))
 
+if self.obsstore:
+self.obsstore.obscache.update(self)
+self.obsstore.obscache.save(self)
+
 def invalidatecaches(self):
 
 if '_tagscache' in vars(self):
diff -r 985d753d4f57 -r f8953ed43f8d mercurial/obsolete.py
--- a/mercurial/obsolete.py Fri May 19 14:44:22 2017 +0200
+++ b/mercurial/obsolete.py Fri May 19 14:46:46 2017 +0200
@@ -525,6 +525,7 @@
 self.svfs = repo.svfs
 self._defaultformat = defaultformat
 self._readonly = readonly
+self.obscache = obscache(repo)
 
 def __iter__(self):
 return iter(self._all)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 06 of 14] cache: add a obsstoresourcebase class

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1499458924 -7200
#  Fri Jul 07 22:22:04 2017 +0200
# Node ID 5b49f653a4a50607127e37e7511c8a8e343cc8d9
# Parent  7c41d1484f53c53d516153d19a90c331c3f87c16
# EXP-Topic obs-cache
cache: add a obsstoresourcebase class

This class provides an abstract base for cache based on the obsstore. Having
this as a separared class is useful as it could be used for the obsindexes
cache that have been discussed in June this year.

diff -r 7c41d1484f53 -r 5b49f653a4a5 mercurial/cache.py
--- a/mercurial/cache.pySat Jun 17 06:39:43 2017 +0200
+++ b/mercurial/cache.pyFri Jul 07 22:22:04 2017 +0200
@@ -164,3 +164,20 @@
 
 def _fetchupdatedata(self, repo):
 return self._fetchchangelogdata(self._cachekey, repo.changelog)
+
+class obsstoresourcebase(incrementalcachebase):
+"""an abstract class for cache that source data from the obsstore
+
+For this purpose it use a cache key covering obsstore
+content provided by the obsstore itself
+"""
+
+__metaclass__ = abc.ABCMeta
+
+# default key used for an empty cache
+emptykey = (0, node.nullid)
+_cachekeyspec = 'I20s'
+_cachename = None # used for debug message
+
+def _fetchupdatedata(self, repo):
+return repo.obsstore.getmarkerssince(self._cachekey)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 09 of 14] obscache: add a cache for 1/2 of the "obsolete" property

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1495197986 -7200
#  Fri May 19 14:46:26 2017 +0200
# Node ID 63214f4d9a766761259b650539eede424413e6a2
# Parent  774ff18cc36b72822f598b4fa5a51628513926e3
# EXP-Topic obs-cache
obscache: add a cache for 1/2 of the "obsolete" property

Knowing if a changeset is obsolete requires two data:

1) is the change non-public,
2) is the changeset affected by any obsolescence markers.

The phase related data has some fast implementation already. However, the
obsmarkers based property currently requires to parse the whole obsstore, a
slow operation.

That information is monotonic (new changeset are not affected, and once they
are affected, they will be for ever), making it is easy to cache. We introduce
a new class dedicated to caching of this information. That first
implementation still needs to parse the full obsstore when updating for the
sake of simplicity. It will be improved later to allow lighter upgrade.

The next changesets will put this new cache to use.

That code is coming from the evolve extension, were it matured. To keep this
changeset simple, there are a couple of improvement in the extension that will
be ported later.

diff -r 774ff18cc36b -r 63214f4d9a76 mercurial/obsolete.py
--- a/mercurial/obsolete.py Sat Jul 08 16:26:16 2017 +0200
+++ b/mercurial/obsolete.py Fri May 19 14:46:26 2017 +0200
@@ -75,6 +75,7 @@
 
 from .i18n import _
 from . import (
+cache,
 error,
 node,
 obsutil,
@@ -907,6 +908,145 @@
 repo.ui.deprecwarn(movemsg, '4.3')
 return obsutil.successorssets(repo, initialnode, cache=cache)
 
+class obscache(cache.dualsourcecache):
+"""cache "does a rev is the precursors of some obsmarkers" property
+
+This is not directly holding the "is this revision obsolete" information,
+because phases data gets into play here. However, it allow to compute the
+"obsolescence" set without reading the obsstore content.
+
+The cache use a bytearray to store that data and simply write it on disk
+for storage.
+
+Implementation note #1:
+
+  The obsstore is implementing only half of the transaction logic it
+  should. It properly record the starting point of the obsstore to allow
+  clean rollback. However it still write to the obsstore file directly
+  during the transaction. Instead it should be keeping data in memory and
+  write to a '.pending' file to make the data vailable for hooks.
+
+  This cache is not going further than what the obsstore is doing, so it
+  does not has any '.pending' logic. When the obsstore gains proper
+  '.pending' support, adding it to this cache should not be too hard. As
+  the flag always move from 0 to 1, we could have a second '.pending' cache
+  file to be read. If flag is set in any of them, the value is 1. For the
+  same reason, updating the file in place should be possible.
+
+Implementation note #2:
+
+Storage-wise, we could have a "start rev" to avoid storing useless
+zero. That would be especially useful for the '.pending' overlay.
+"""
+
+_filepath = 'cache/obscache-v01'
+_cachename = 'obscache' # used for error message
+
+def __init__(self, repo):
+super(obscache, self).__init__()
+self._ondiskkey = None
+self._vfs = repo.vfs
+self._data = bytearray()
+
+def get(self, rev):
+"""return True if "rev" is used as "precursors for any obsmarkers
+
+IMPORTANT: make sure the cache has been updated to match the repository
+content before using it"""
+return self._data[rev]
+
+def clear(self, reset=False):
+"""invalidate the in memory cache content"""
+super(obscache, self).clear(reset=reset)
+self._data = bytearray()
+
+def _updatefrom(self, repo, data):
+if data[0]:
+self._updaterevs(repo,  data[0])
+if data[1]:
+self._updatemarkers(repo,  data[1])
+
+def _updaterevs(self, repo, revs):
+"""update the cache with new revisions
+
+Newly added changesets might be affected by obsolescence markers we
+already have locally. So we needs to have some global knowledge about
+the markers to handle that question.
+
+XXX performance note:
+
+Right now this requires parsing all markers in the obsstore. We could
+imagine using various optimisation (eg: another cache, network
+exchange, etc).
+
+A possible approach to this is to build a set of all nodes used as
+precursors in `obsstore._obscandidate`. If markers are not loaded yet,
+we could initialize it by doing a quick scan through the obsstore data
+and filling a (pre-sized) set. Doing so would be much faster than
+parsing all the obsmarkers since we would access less data, not create
+any object beside the nodes and not have to decode any complex data.
+
+For 

[PATCH 05 of 14] obsstore: add a method to incrementally retrieve obsmarkers

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1497674383 -7200
#  Sat Jun 17 06:39:43 2017 +0200
# Node ID 7c41d1484f53c53d516153d19a90c331c3f87c16
# Parent  e73d330f6fdb0cf1635920f4302b61b321ec58e2
# EXP-Topic obs-cache
obsstore: add a method to incrementally retrieve obsmarkers

Parsing the full obsstore is slow, so cache that depends on obsstore content
wants a way to know if the obsstore changed, and it this change was append
only.

For this purpose we introduce an official cache-key for the obsstore. This
cache-key work in a way similar to the '(tiprev, tipnode)' pair used for the
changelog. We use the size of the obsstore file and the hash of its tail. That
way, we can check if the obsstore grew and if the content we knew is still
present in the obsstore.

The cachekey comes with a method that only returns the obsmarkers that changed
between the last seen "cache-key" value and the current state of the
repository. That method is race free and should be used by caches. See
documentation for details.

note: we are reading the full obsstore file from disk as part of the key
validation process. This could be avoided but we keep the implementation
simple for now. Once it is in place and running we can iterate to make it
better.

diff -r e73d330f6fdb -r 7c41d1484f53 mercurial/obsolete.py
--- a/mercurial/obsolete.py Sun Jun 04 10:02:09 2017 -0700
+++ b/mercurial/obsolete.py Sat Jun 17 06:39:43 2017 +0200
@@ -70,6 +70,7 @@
 from __future__ import absolute_import
 
 import errno
+import hashlib
 import struct
 
 from .i18n import _
@@ -513,6 +514,10 @@
 # parents: (tuple of nodeid) or None, parents of precursors
 #  None is used when no data has been recorded
 
+# how much data to read at the end of file,
+# 1024 byte should be about 10 markers in average
+_obskeyspan = 1024
+
 def __init__(self, svfs, defaultformat=_fm1version, readonly=False):
 # caches for various obsolescence related cache
 self.caches = {}
@@ -540,6 +545,72 @@
 
 __bool__ = __nonzero__
 
+def getmarkerssince(self, previouscontentkey):
+"""retrieve all new markers (since "contentkey") + updated content key
+
+This function is to be used by cache depending on obsstore content. It
+make sure cache can incrementally update themselves without fear
+obsstore stripping or race condition.  If the content key is invalid
+(some obsstore content got stripped), all markers in the obsstore will
+be returned.
+
+return: (reset, obsmarkers, contentkey)
+
+:reset: boolean, True if previouscontentkey was invalided. Previously
+stored data are invalid and should be discarded. Full markers
+content is return in this case.
+
+:obsmarkers: the list of new obsmarkers.
+
+:contentkey: a key matching the content of 'previouscontentkey' +
+ 'obsmarkers'.
+
+The content key is a pair of:
+
+(obsstore size, hash of last N bytes of the obsstore)
+
+It must be kept around by cache and provided again for the next
+incremental read of new obsmarkers.
+"""
+# XXX This function could avoid loading the whole data from disk (and
+# only read new markers). It currently use 'self._data' to keep the
+# code simpler.
+keysize, keyhash = previouscontentkey
+fulldata = self._data
+
+# check the existing cache key
+reset = False
+if len(fulldata) < keysize: # less data than expected, this is a strip
+reset = True
+else:
+if keysize == 0: # no obsstore
+actualhash = node.nullid
+else:
+first = max(0, keysize - self._obskeyspan)
+keydata = fulldata[first:keysize]
+actualhash = hashlib.sha1(keydata).digest()
+reset = actualhash != keyhash
+newsize = len(fulldata)
+
+# read the new data
+if reset:
+keysize = None # read all data
+elif keysize == newsize:
+# no update since last change, exist early
+return False, [], previouscontentkey
+if newsize:
+start = max(0, newsize - self._obskeyspan)
+newhash = hashlib.sha1(fulldata[start:newsize]).digest()
+__, obsmarkers = _readmarkers(fulldata, keysize, newsize)
+else:
+# obsstore is empty, use a generic hash and skip reading markers.
+newhash == node.nullid
+obsmarkers = []
+
+# for now and for the sake of simplicity make sure obsmarkers is a list
+obsmarkers = list(obsmarkers)
+return reset, obsmarkers, (newsize, newhash)
+
 @property
 def readonly(self):
 """True if marker creation is disabled
___
Mercurial-devel mailing list

[PATCH 12 of 14] obscache: use the obscache to compute the obsolete set

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1495198021 -7200
#  Fri May 19 14:47:01 2017 +0200
# Node ID 3a93e99b0e718befd57a32615a14fd0d3c194456
# Parent  f8953ed43f8d1b146dcff688030133f0d6123a49
# EXP-Topic obs-cache
obscache: use the obscache to compute the obsolete set

Now that we have a cache and that the cache is kept up to date, we can use it to
speeds up the obsolete set computation. This way, we no longer need to load the
obsstore for most operation.

On the mercurial-core repository, this provides a significant speed up for
operation that do not need further obsolescence information:

| command  | before | after |
| id -r @  |  0.670 | 0.093 |
| log -r @ |  0.951 | 0.916 |
| diff |  0.070 | 0.069 |
| diff -r .~1  |  0.705 | 0.120 |
| export @ |  0.672 | 0.103 |
| log -G -r draft()|  1.117 | 1.084 |
| log -G -r draft()|  1.063 | 1.064 |
|   -T "{node} {troubles}" ||   |
| log -G -r tip|  0.658 | 0.086 |
|   -T "{node} {desc}" ||   |

The obsstore loading operation usually disappear from execution profile.

The speeds up rely on a couple of different mechanism:

* First, not having to parse the obsstore provides a massive speedup:

  Time spent computing 'obsolete', no obsstore pre-loaded.

before: wall 0.449502 comb 0.46 user 0.42 sys 0.04 (best of 17)
after:  wall 0.005752 comb 0.01 user 0.01 sys 0.00 (best of 340)

* Second keeping the computation fully in the revision space (no usage of node),
  raise and extra 4x speedup.

  Time spent computing 'obsolete', obsstore preloaded:

before: wall 0.007684 comb 0.00 user 0.00 sys 0.00 (best of 305)
after:  wall 0.001928 comb 0.00 user 0.00 sys 0.00 (best of 
1250)

To keep the changeset simple, we assume the cache is up to date (from the last
transaction). This won't be true when both old and new clients access the
repository. (with the special case of no new transactions since last upgrade).
We'll address this issue in the next couple of changesets.

This changesets is a first step to install the basic. There are a couple of easy
improvement that can further improve this cache:

 * Improving handling of outdated cache on read only operation (see above),

 * Avoid reaading the full obsstore data from disk to check the cache key
   (about -4ms, 3x speedup)

 * Optimise the python code to reduce attribute lookup
   (about 25% of the remaining of the time spent there).

diff -r f8953ed43f8d -r 3a93e99b0e71 mercurial/obsolete.py
--- a/mercurial/obsolete.py Fri May 19 14:46:46 2017 +0200
+++ b/mercurial/obsolete.py Fri May 19 14:47:01 2017 +0200
@@ -1096,11 +1096,24 @@
 @cachefor('obsolete')
 def _computeobsoleteset(repo):
 """the set of obsolete revisions"""
-getnode = repo.changelog.node
 notpublic = _mutablerevs(repo)
-isobs = repo.obsstore.successors.__contains__
-obs = set(r for r in notpublic if isobs(getnode(r)))
-return obs
+if not notpublic or not repo.obsstore:
+# all changeset are public, none are obsolete
+return set()
+
+# XXX There are a couple of case where the cache could not be up to date:
+#
+# 1) no transaction happened in the repository since the upgrade,
+# 2) both old and new client touches that repository
+#
+# recomputing the whole cache in these case is a bit slower that using the
+# good old version (parsing markers and checking them). We could add some
+# logic to fall back to the old way in these cases.
+obscache = repo.obsstore.obscache
+obscache.update(repo) # ensure it is up to date:
+isobs = obscache.get
+
+return set(r for r in notpublic if isobs(r))
 
 @cachefor('unstable')
 def _computeunstableset(repo):
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 01 of 14] cache: introduce an abstract class for cache we can upgrade incrementally

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1499458441 -7200
#  Fri Jul 07 22:14:01 2017 +0200
# Node ID 6edb62505c697329de034c2fdc47befd5896f31f
# Parent  89796a25d4bb91fb418ad3e70faad2c586902ffb
# EXP-Topic obs-cache
cache: introduce an abstract class for cache we can upgrade incrementally

Right now each class implements their own mechanism for validation, and
update. We start introducing abstract class to ultimately allow more
unification of the cache code.

The end goal of this series is to introduce a cache for some obsolescence
property, not to actually implement the cache. However, taking advantage of
adding a new cache to introduce the abstract class seems like a win.

diff -r 89796a25d4bb -r 6edb62505c69 mercurial/cache.py
--- /dev/null   Thu Jan 01 00:00:00 1970 +
+++ b/mercurial/cache.pyFri Jul 07 22:14:01 2017 +0200
@@ -0,0 +1,127 @@
+# cache.py - utilities for caching
+#
+# Copyright 2017 Octobus SAS 
+#
+# This software may be used and distributed according to the terms of the
+# GNU General Public License version 2 or any later version.
+from __future__ import absolute_import
+
+import abc
+import struct
+
+from . import (
+util,
+)
+
+class incrementalcachebase(object):
+"""base class for incremental cache from append only source
+
+There are multiple append only data source we might want to cache
+computation from. One of the common pattern is to track the state of the
+file and update the cache with the extra data (eg: branchmap-cache tracking
+changelog). This pattern also needs to detect when a the source is striped
+
+The overall pattern is similar whatever the actual source is. This class
+introduces the basic patterns.
+"""
+
+__metaclass__ = abc.ABCMeta
+
+# default key used for an empty cache
+emptykey = ()
+
+_cachekeyspec = '' # used for serialization
+_cachename = None # used for debug message
+
+@abc.abstractmethod
+def __init__(self):
+super(incrementalcachebase, self).__init__()
+self._cachekey = None
+
+@util.propertycache
+def _cachekeystruct(self):
+# dynamic property to help subclass to change it
+ return struct.Struct('>' + self._cachekeyspec)
+
+@util.propertycache
+def _cachekeysize(self):
+# dynamic property to help subclass to change it
+return self._cachekeystruct.size
+
+@abc.abstractmethod
+def _updatefrom(self, repo, data):
+"""override this method to update you date from incrementally read 
data.
+
+Content of  will depends of the sources.
+"""
+raise NotImplementedError
+
+@abc.abstractmethod
+def clear(self, reset=False):
+"""invalidate the cache content
+
+if 'reset' is passed, we detected a strip and the cache will have to be
+recomputed.
+
+Subclasses MUST overide this method to actually affect the cache data.
+"""
+if reset:
+self._cachekey = self.emptykey if reset else None
+else:
+self._cachekey = None
+
+@abc.abstractmethod
+def load(self, repo):
+"""Load data from disk
+
+Subclasses MUST restore the "cachekey" attribute while doing so.
+"""
+raise NotImplementedError
+
+@abc.abstractmethod
+def _fetchupdatedata(self, repo):
+"""Check the source for possible changes and return necessary data
+
+The return is a tree elements tuple: reset, data, cachekey
+
+* reset: `True` when a strip is detected and cache need to be reset
+* data: new data to take in account, actual type depends of the source
+* cachekey: the cache key covering  and precious covered data
+"""
+raise NotImplementedError
+
+# Useful "public" function (no need to override them)
+
+def update(self, repo):
+"""update the cache with new repository data
+
+The update will be incremental when possible"""
+repo = repo.unfiltered()
+
+# If we do not have any data, try loading from disk
+if self._cachekey is None:
+self.load(repo)
+
+reset, data, newkey = self._fetchupdatedata(repo)
+if newkey == self._cachekey:
+return
+if reset or self._cachekey is None:
+repo.ui.log('cache', 'strip detected, %s cache reset\n'
+% self._cachename)
+self.clear(reset=True)
+
+starttime = util.timer()
+self._updatefrom(repo, data)
+duration = util.timer() - starttime
+repo.ui.log('cache', 'updated %s in %.4f seconds\n',
+self._cachename, duration)
+
+self._cachekey = newkey
+
+def _serializecachekey(self):
+"""provide a bytes version of the cachekey"""
+return self._cachekeystruct.pack(*self._cachekey)
+
+def _deserializecachekey(self, data):
+"""read the cachekey 

[PATCH 10 of 14] obsstore: pass a repository object for initialisation

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1495197862 -7200
#  Fri May 19 14:44:22 2017 +0200
# Node ID 985d753d4f5799f2a332140adedb06efd465d62b
# Parent  63214f4d9a766761259b650539eede424413e6a2
# EXP-Topic obs-cache
obsstore: pass a repository object for initialisation

The cache will needs a repository object (to grab a 'vfs'), so we pass a repo 
object instead of
just the 'svfs' and we grab the 'svfs' from there.

diff -r 63214f4d9a76 -r 985d753d4f57 contrib/perf.py
--- a/contrib/perf.py   Fri May 19 14:46:26 2017 +0200
+++ b/contrib/perf.py   Fri May 19 14:44:22 2017 +0200
@@ -1391,8 +1391,7 @@
 
 Result is the number of markers in the repo."""
 timer, fm = gettimer(ui)
-svfs = getsvfs(repo)
-timer(lambda: len(obsolete.obsstore(svfs)))
+timer(lambda: len(obsolete.obsstore(repo)))
 fm.end()
 
 @command('perflrucachedict', formatteropts +
diff -r 63214f4d9a76 -r 985d753d4f57 mercurial/obsolete.py
--- a/mercurial/obsolete.py Fri May 19 14:46:26 2017 +0200
+++ b/mercurial/obsolete.py Fri May 19 14:44:22 2017 +0200
@@ -519,10 +519,10 @@
 # 1024 byte should be about 10 markers in average
 _obskeyspan = 1024
 
-def __init__(self, svfs, defaultformat=_fm1version, readonly=False):
+def __init__(self, repo, defaultformat=_fm1version, readonly=False):
 # caches for various obsolescence related cache
 self.caches = {}
-self.svfs = svfs
+self.svfs = repo.svfs
 self._defaultformat = defaultformat
 self._readonly = readonly
 
@@ -799,7 +799,7 @@
 if defaultformat is not None:
 kwargs['defaultformat'] = defaultformat
 readonly = not isenabled(repo, createmarkersopt)
-store = obsstore(repo.svfs, readonly=readonly, **kwargs)
+store = obsstore(repo, readonly=readonly, **kwargs)
 if store and readonly:
 ui.warn(_('obsolete feature not enabled but %i markers found!\n')
 % len(list(store)))
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 14 of 14] obscache: skip updating outdated obscache obscache for readonly operation

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1496358960 -7200
#  Fri Jun 02 01:16:00 2017 +0200
# Node ID 59528ec2969e10de1c3eec16149b483083b89218
# Parent  7774ff78ef2e7433368184e7de743342b771b91c
# EXP-Topic obs-cache
obscache: skip updating outdated obscache obscache for readonly operation

During read only operation, we fallback to the old way of computing obsolescence
markers when we detect that the obscache is behind/invalid. This might happen
when both old and new clients touch the same repository.

This is a simple way to avoiding paying extra cache warming overhead without
making the cache write pattern more complex.

diff -r 7774ff78ef2e -r 59528ec2969e mercurial/obsolete.py
--- a/mercurial/obsolete.py Sun Jul 09 03:56:12 2017 +0200
+++ b/mercurial/obsolete.py Fri Jun 02 01:16:00 2017 +0200
@@ -1125,9 +1125,14 @@
 # good old version (parsing markers and checking them). We could add some
 # logic to fall back to the old way in these cases.
 obscache = repo.obsstore.obscache
-obscache.update(repo) # ensure it is up to date:
-isobs = obscache.get
-
+if obscache.uptodate(repo) and repo.currenttransaction() is None:
+hasnode = repo.obsstore.successors.__contains__
+node = repo.changelog.node
+def isobs(rev):
+return hasnode(node(rev))
+else:
+obscache.update(repo) # ensure it is up to date:
+isobs = obscache.get
 return set(r for r in notpublic if isobs(r))
 
 @cachefor('unstable')
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 13 of 14] cache: add a way to check if a cache is up to date

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1499565372 -7200
#  Sun Jul 09 03:56:12 2017 +0200
# Node ID 7774ff78ef2e7433368184e7de743342b771b91c
# Parent  3a93e99b0e718befd57a32615a14fd0d3c194456
# EXP-Topic obs-cache
cache: add a way to check if a cache is up to date

This will be useful to decide if a cache can be used as-is or not.

diff -r 3a93e99b0e71 -r 7774ff78ef2e mercurial/cache.py
--- a/mercurial/cache.pyFri May 19 14:47:01 2017 +0200
+++ b/mercurial/cache.pySun Jul 09 03:56:12 2017 +0200
@@ -80,6 +80,15 @@
 raise NotImplementedError
 
 @abc.abstractmethod
+def _uptodate(self, repo):
+"""True if the cache is up to date with the repository
+
+This function is for overriding only. Use the public 'uptodate(repo)'
+for actual usage.
+"""
+raise NotImplementedError
+
+@abc.abstractmethod
 def _fetchupdatedata(self, repo):
 """Check the source for possible changes and return necessary data
 
@@ -98,6 +107,12 @@
 
 # Useful "public" function (no need to override them)
 
+def uptodate(self, repo):
+"""True if the cache is up to date with the repository"""
+if self._cachekey is None:
+self.load(repo)
+return self._uptodate(repo)
+
 def update(self, repo):
 """update the cache with new repository data
 
@@ -168,6 +183,17 @@
 else:
 return False, list(cl.revs(start=keyrev + 1, stop=tiprev)), newkey
 
+def _checkchangelogkey(self, cachekey, cl):
+# Exists as its own method to help subclass to reuse it.
+tiprev = len(cl) - 1
+tipnode = cl.node(tiprev)
+newkey = (tiprev, tipnode)
+tiprev = len(cl) - 1
+return newkey == cachekey
+
+def _uptodate(self, repo):
+return self._checkchangelogkey(self._cachekey, repo.changelog)
+
 def _fetchupdatedata(self, repo):
 return self._fetchchangelogdata(self._cachekey, repo.changelog)
 
@@ -188,6 +214,9 @@
 _cachekeyspec = 'I20s'
 _cachename = None # used for debug message
 
+def _uptodate(self, repo):
+return repo.obsstore.matchcontentkey(self._cachekey)
+
 def _fetchupdatedata(self, repo):
 return repo.obsstore.getmarkerssince(self._cachekey)
 
@@ -210,6 +239,10 @@
  + obsstoresourcebase._cachekeyspec)
 _cachename = None # used for debug message
 
+def _uptodate(self, repo):
+return (self._checkchangelogkey(self._cachekey, repo.changelog)
+and repo.obsstore.matchcontentkey(self._cachekey))
+
 def _fetchupdatedata(self, repo):
 clkey = self._cachekey[0:2]
 obskey = self._cachekey[2:4]
diff -r 3a93e99b0e71 -r 7774ff78ef2e mercurial/obsolete.py
--- a/mercurial/obsolete.py Fri May 19 14:47:01 2017 +0200
+++ b/mercurial/obsolete.py Sun Jul 09 03:56:12 2017 +0200
@@ -547,6 +547,21 @@
 
 __bool__ = __nonzero__
 
+def matchcontentkey(self, key):
+"""provide the cachekey (as usable by """
+keysize, keyhash = key
+# XXX This function could avoid loading the whole data from disk (and
+# only read new markers). It currently use 'self._data' to keep the
+# code simpler.
+fulldata = self._data
+currentsize = len(fulldata)
+if currentsize != keysize:
+return False
+first = max(0, keysize - self._obskeyspan)
+keydata = fulldata[first:keysize]
+actualhash = hashlib.sha1(keydata).digest()
+return actualhash == keyhash
+
 def getmarkerssince(self, previouscontentkey):
 """retrieve all new markers (since "contentkey") + updated content key
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 08 of 14] cache: adds debug details about what the content of the update

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1499523976 -7200
#  Sat Jul 08 16:26:16 2017 +0200
# Node ID 774ff18cc36b72822f598b4fa5a51628513926e3
# Parent  5d3e659c979aa428ba44b138cfac30b7cca28fb3
# EXP-Topic obs-cache
cache: adds debug details about what the content of the update

Having more data in debug is good, we add a small method to help providing them.

diff -r 5d3e659c979a -r 774ff18cc36b mercurial/cache.py
--- a/mercurial/cache.pyFri Jul 07 22:22:39 2017 +0200
+++ b/mercurial/cache.pySat Jul 08 16:26:16 2017 +0200
@@ -91,6 +91,11 @@
 """
 raise NotImplementedError
 
+@abc.abstractmethod
+def _updatesummary(self, data):
+"""return a small string to be included in debug output"""
+raise NotImplementedError
+
 # Useful "public" function (no need to override them)
 
 def update(self, repo):
@@ -114,8 +119,9 @@
 starttime = util.timer()
 self._updatefrom(repo, data)
 duration = util.timer() - starttime
-repo.ui.log('cache', 'updated %s in %.4f seconds\n',
-self._cachename, duration)
+summary = self._updatesummary(data)
+repo.ui.log('cache', 'updated %s in %.4f seconds (%s)\n',
+self._cachename, duration, summary)
 
 self._cachekey = newkey
 
@@ -165,6 +171,9 @@
 def _fetchupdatedata(self, repo):
 return self._fetchchangelogdata(self._cachekey, repo.changelog)
 
+def _updatesummary(self, data):
+return '%ir' % len(data)
+
 class obsstoresourcebase(incrementalcachebase):
 """an abstract class for cache that source data from the obsstore
 
@@ -182,6 +191,9 @@
 def _fetchupdatedata(self, repo):
 return repo.obsstore.getmarkerssince(self._cachekey)
 
+def _updatesummary(self, data):
+return '%io' % len(data)
+
 class dualsourcecache(obsstoresourcebase, changelogsourcebase):
 """An abstract class for cache that needs both changelog and obsstore
 
@@ -216,3 +228,6 @@
 newkey = newclkey + newobskey
 data = (revs, obsmarkers)
 return reset, data, newkey
+
+def _updatesummary(self, data):
+return '%ir, %io' % (len(data[0]), len(data[1]))
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 07 of 14] cache: introduce a dualsourcebase class

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1499458959 -7200
#  Fri Jul 07 22:22:39 2017 +0200
# Node ID 5d3e659c979aa428ba44b138cfac30b7cca28fb3
# Parent  5b49f653a4a50607127e37e7511c8a8e343cc8d9
# EXP-Topic obs-cache
cache: introduce a dualsourcebase class

This abstract class cover the case where cache are computed from data
contained in the changelog -and- the obsstore.

diff -r 5b49f653a4a5 -r 5d3e659c979a mercurial/cache.py
--- a/mercurial/cache.pyFri Jul 07 22:22:04 2017 +0200
+++ b/mercurial/cache.pyFri Jul 07 22:22:39 2017 +0200
@@ -181,3 +181,38 @@
 
 def _fetchupdatedata(self, repo):
 return repo.obsstore.getmarkerssince(self._cachekey)
+
+class dualsourcecache(obsstoresourcebase, changelogsourcebase):
+"""An abstract class for cache that needs both changelog and obsstore
+
+The cache key used is a combinaison of the one used for the changelog and
+the one used for the obsstore. See inherited class for details.
+"""
+
+__metaclass__ = abc.ABCMeta
+
+# default key used for an empty cache
+emptykey = (changelogsourcebase.emptykey
++ obsstoresourcebase.emptykey)
+_cachekeyspec = (changelogsourcebase._cachekeyspec
+ + obsstoresourcebase._cachekeyspec)
+_cachename = None # used for debug message
+
+def _fetchupdatedata(self, repo):
+clkey = self._cachekey[0:2]
+obskey = self._cachekey[2:4]
+
+reset, revs, newclkey = self._fetchchangelogdata(clkey, repo.changelog)
+if reset:
+obskey = obsstoresourcebase.emptykey
+obsreturn = repo.obsstore.getmarkerssince(obskey)
+obsreset, obsmarkers, newobskey = obsreturn
+if obsreset:
+reset = True
+clkey = changelogsourcebase.emptykey
+clreturn = self._fetchchangelogdata(clkey, repo.changelog)
+__, revs, newclkey = clreturn
+
+newkey = newclkey + newobskey
+data = (revs, obsmarkers)
+return reset, data, newkey
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 03 of 14] obsstore: keep self._data updated with _addmarkers

[Boris Feld]

# HG changeset patch
# User Jun Wu 
# Date 1496552183 25200
#  Sat Jun 03 21:56:23 2017 -0700
# Node ID ff2ebc11f12a26a4e0bda3ccf5fde63f5f690813
# Parent  8b71290526ddb77f157e075191dd748793d85601
# EXP-Topic obs-cache
obsstore: keep self._data updated with _addmarkers

This makes sure obsstore._data is still correct with added markers.

The '_data' propertycache was added in 17ce57b7873f.

diff -r 8b71290526dd -r ff2ebc11f12a mercurial/obsolete.py
--- a/mercurial/obsolete.py Fri Jul 07 22:15:52 2017 +0200
+++ b/mercurial/obsolete.py Sat Jun 03 21:56:23 2017 -0700
@@ -607,8 +607,8 @@
 offset = f.tell()
 transaction.add('obsstore', offset)
 # offset == 0: new file - add the version header
-for bytes in encodemarkers(new, offset == 0, self._version):
-f.write(bytes)
+data = b''.join(encodemarkers(new, offset == 0, self._version))
+f.write(data)
 finally:
 # XXX: f.close() == filecache invalidation == obsstore rebuilt.
 # call 'filecacheentry.refresh()'  here
@@ -616,7 +616,7 @@
 addedmarkers = transaction.changes.get('obsmarkers')
 if addedmarkers is not None:
 addedmarkers.update(new)
-self._addmarkers(new)
+self._addmarkers(new, data)
 # new marker *may* have changed several set. invalidate the cache.
 self.caches.clear()
 # records the number of new markers for the transaction hooks
@@ -673,8 +673,9 @@
 def _cached(self, attr):
 return attr in self.__dict__
 
-def _addmarkers(self, markers):
+def _addmarkers(self, markers, rawdata):
 markers = list(markers) # to allow repeated iteration
+self._data = self._data + rawdata
 self._all.extend(markers)
 if self._cached('successors'):
 _addsuccessors(self.successors, markers)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 04 of 14] obsstore: let read marker API take a range of offsets

[Boris Feld]

# HG changeset patch
# User Jun Wu 
# Date 1496595729 25200
#  Sun Jun 04 10:02:09 2017 -0700
# Node ID e73d330f6fdb0cf1635920f4302b61b321ec58e2
# Parent  ff2ebc11f12a26a4e0bda3ccf5fde63f5f690813
# EXP-Topic obs-cache
obsstore: let read marker API take a range of offsets

This allows us to read a customized range of markers, instead of loading all
of them.

The condition of stop is made consistent across C and Python implementation
so we will still read marker when offset=a, stop=a+1.

diff -r ff2ebc11f12a -r e73d330f6fdb mercurial/obsolete.py
--- a/mercurial/obsolete.py Sat Jun 03 21:56:23 2017 -0700
+++ b/mercurial/obsolete.py Sun Jun 04 10:02:09 2017 -0700
@@ -178,10 +178,9 @@
 _fm0fsize = _calcsize(_fm0fixed)
 _fm0fnodesize = _calcsize(_fm0node)
 
-def _fm0readmarkers(data, off):
+def _fm0readmarkers(data, off, stop):
 # Loop on markers
-l = len(data)
-while off + _fm0fsize <= l:
+while off < stop:
 # read fixed part
 cur = data[off:off + _fm0fsize]
 off += _fm0fsize
@@ -317,7 +316,7 @@
 _fm1metapair = 'BB'
 _fm1metapairsize = _calcsize('BB')
 
-def _fm1purereadmarkers(data, off):
+def _fm1purereadmarkers(data, off, stop):
 # make some global constants local for performance
 noneflag = _fm1parentnone
 sha2flag = usingsha256
@@ -331,10 +330,9 @@
 unpack = _unpack
 
 # Loop on markers
-stop = len(data) - _fm1fsize
 ufixed = struct.Struct(_fm1fixed).unpack
 
-while off <= stop:
+while off < stop:
 # read fixed part
 o1 = off + fsize
 t, secs, tz, flags, numsuc, numpar, nummeta, prec = 
ufixed(data[off:o1])
@@ -428,11 +426,10 @@
 data.append(value)
 return ''.join(data)
 
-def _fm1readmarkers(data, off):
+def _fm1readmarkers(data, off, stop):
 native = getattr(parsers, 'fm1readmarkers', None)
 if not native:
-return _fm1purereadmarkers(data, off)
-stop = len(data) - _fm1fsize
+return _fm1purereadmarkers(data, off, stop)
 return native(data, off, stop)
 
 # mapping to read/write various marker formats
@@ -444,14 +441,17 @@
 return _unpack('>B', data[0:1])[0]
 
 @util.nogc
-def _readmarkers(data):
+def _readmarkers(data, off=None, stop=None):
 """Read and enumerate markers from raw data"""
 diskversion = _readmarkerversion(data)
-off = 1
+if not off:
+off = 1  # skip 1 byte version number
+if stop is None:
+stop = len(data)
 if diskversion not in formats:
 msg = _('parsing obsolete marker: unknown version %r') % diskversion
 raise error.UnknownVersion(msg, version=diskversion)
-return diskversion, formats[diskversion][0](data, off)
+return diskversion, formats[diskversion][0](data, off, stop)
 
 def encodeheader(version=_fm0version):
 return _pack('>B', version)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 02 of 14] cache: introduce a changelogsourcebase class

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1499458552 -7200
#  Fri Jul 07 22:15:52 2017 +0200
# Node ID 8b71290526ddb77f157e075191dd748793d85601
# Parent  6edb62505c697329de034c2fdc47befd5896f31f
# EXP-Topic obs-cache
cache: introduce a changelogsourcebase class

This abstract class will help code that need a cache tracking the changelog
content (eg: the branchmap cache). The cache key used is the same as what the
branchmap uses.

diff -r 6edb62505c69 -r 8b71290526dd mercurial/cache.py
--- a/mercurial/cache.pyFri Jul 07 22:14:01 2017 +0200
+++ b/mercurial/cache.pyFri Jul 07 22:15:52 2017 +0200
@@ -10,6 +10,7 @@
 import struct
 
 from . import (
+node,
 util,
 )
 
@@ -125,3 +126,41 @@
 def _deserializecachekey(self, data):
 """read the cachekey from bytes"""
 return self._cachekeystruct.unpack(data)
+
+class changelogsourcebase(incrementalcachebase):
+"""an abstract class for cache sourcing data from the changelog
+
+For this purpose it use a cache key covering changelog content.
+The cache key parts are: (tiprev, tipnode)
+"""
+
+__metaclass__ = abc.ABCMeta
+
+# default key used for an empty cache
+emptykey = (0, node.nullid)
+_cachekeyspec = 'i20s'
+_cachename = None # used for debug message
+
+# Useful "public" function (no need to override them)
+
+def _fetchchangelogdata(self, cachekey, cl):
+"""use a cachekey to fetch incremental data
+
+Exists as its own method to help subclass to reuse it."""
+tiprev = len(cl) - 1
+tipnode = cl.node(tiprev)
+newkey = (tiprev, tipnode)
+tiprev = len(cl) - 1
+if newkey == cachekey:
+return False, [], newkey
+keyrev, keynode = cachekey
+if tiprev < keyrev or cl.node(keyrev) != keynode:
+revs = ()
+if len(cl):
+revs = list(cl.revs(stop=tiprev))
+return True, revs, newkey
+else:
+return False, list(cl.revs(start=keyrev + 1, stop=tiprev)), newkey
+
+def _fetchupdatedata(self, repo):
+return self._fetchchangelogdata(self._cachekey, repo.changelog)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 01 of 14] cache: introduce an abstract class for cache we can upgrade incrementally

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1499458441 -7200
#  Fri Jul 07 22:14:01 2017 +0200
# Node ID 6edb62505c697329de034c2fdc47befd5896f31f
# Parent  89796a25d4bb91fb418ad3e70faad2c586902ffb
# EXP-Topic obs-cache
cache: introduce an abstract class for cache we can upgrade incrementally

Right now each class implements their own mechanism for validation, and
update. We start introducing abstract class to ultimately allow more
unification of the cache code.

The end goal of this series is to introduce a cache for some obsolescence
property, not to actually implement the cache. However, taking advantage of
adding a new cache to introduce the abstract class seems like a win.

diff -r 89796a25d4bb -r 6edb62505c69 mercurial/cache.py
--- /dev/null   Thu Jan 01 00:00:00 1970 +
+++ b/mercurial/cache.pyFri Jul 07 22:14:01 2017 +0200
@@ -0,0 +1,127 @@
+# cache.py - utilities for caching
+#
+# Copyright 2017 Octobus SAS 
+#
+# This software may be used and distributed according to the terms of the
+# GNU General Public License version 2 or any later version.
+from __future__ import absolute_import
+
+import abc
+import struct
+
+from . import (
+util,
+)
+
+class incrementalcachebase(object):
+"""base class for incremental cache from append only source
+
+There are multiple append only data source we might want to cache
+computation from. One of the common pattern is to track the state of the
+file and update the cache with the extra data (eg: branchmap-cache tracking
+changelog). This pattern also needs to detect when a the source is striped
+
+The overall pattern is similar whatever the actual source is. This class
+introduces the basic patterns.
+"""
+
+__metaclass__ = abc.ABCMeta
+
+# default key used for an empty cache
+emptykey = ()
+
+_cachekeyspec = '' # used for serialization
+_cachename = None # used for debug message
+
+@abc.abstractmethod
+def __init__(self):
+super(incrementalcachebase, self).__init__()
+self._cachekey = None
+
+@util.propertycache
+def _cachekeystruct(self):
+# dynamic property to help subclass to change it
+ return struct.Struct('>' + self._cachekeyspec)
+
+@util.propertycache
+def _cachekeysize(self):
+# dynamic property to help subclass to change it
+return self._cachekeystruct.size
+
+@abc.abstractmethod
+def _updatefrom(self, repo, data):
+"""override this method to update you date from incrementally read 
data.
+
+Content of  will depends of the sources.
+"""
+raise NotImplementedError
+
+@abc.abstractmethod
+def clear(self, reset=False):
+"""invalidate the cache content
+
+if 'reset' is passed, we detected a strip and the cache will have to be
+recomputed.
+
+Subclasses MUST overide this method to actually affect the cache data.
+"""
+if reset:
+self._cachekey = self.emptykey if reset else None
+else:
+self._cachekey = None
+
+@abc.abstractmethod
+def load(self, repo):
+"""Load data from disk
+
+Subclasses MUST restore the "cachekey" attribute while doing so.
+"""
+raise NotImplementedError
+
+@abc.abstractmethod
+def _fetchupdatedata(self, repo):
+"""Check the source for possible changes and return necessary data
+
+The return is a tree elements tuple: reset, data, cachekey
+
+* reset: `True` when a strip is detected and cache need to be reset
+* data: new data to take in account, actual type depends of the source
+* cachekey: the cache key covering  and precious covered data
+"""
+raise NotImplementedError
+
+# Useful "public" function (no need to override them)
+
+def update(self, repo):
+"""update the cache with new repository data
+
+The update will be incremental when possible"""
+repo = repo.unfiltered()
+
+# If we do not have any data, try loading from disk
+if self._cachekey is None:
+self.load(repo)
+
+reset, data, newkey = self._fetchupdatedata(repo)
+if newkey == self._cachekey:
+return
+if reset or self._cachekey is None:
+repo.ui.log('cache', 'strip detected, %s cache reset\n'
+% self._cachename)
+self.clear(reset=True)
+
+starttime = util.timer()
+self._updatefrom(repo, data)
+duration = util.timer() - starttime
+repo.ui.log('cache', 'updated %s in %.4f seconds\n',
+self._cachename, duration)
+
+self._cachekey = newkey
+
+def _serializecachekey(self):
+"""provide a bytes version of the cachekey"""
+return self._cachekeystruct.pack(*self._cachekey)
+
+def _deserializecachekey(self, data):
+"""read the cachekey 

[PATCH 02 of 14] cache: introduce a changelogsourcebase class

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1499458552 -7200
#  Fri Jul 07 22:15:52 2017 +0200
# Node ID 8b71290526ddb77f157e075191dd748793d85601
# Parent  6edb62505c697329de034c2fdc47befd5896f31f
# EXP-Topic obs-cache
cache: introduce a changelogsourcebase class

This abstract class will help code that need a cache tracking the changelog
content (eg: the branchmap cache). The cache key used is the same as what the
branchmap uses.

diff -r 6edb62505c69 -r 8b71290526dd mercurial/cache.py
--- a/mercurial/cache.pyFri Jul 07 22:14:01 2017 +0200
+++ b/mercurial/cache.pyFri Jul 07 22:15:52 2017 +0200
@@ -10,6 +10,7 @@
 import struct
 
 from . import (
+node,
 util,
 )
 
@@ -125,3 +126,41 @@
 def _deserializecachekey(self, data):
 """read the cachekey from bytes"""
 return self._cachekeystruct.unpack(data)
+
+class changelogsourcebase(incrementalcachebase):
+"""an abstract class for cache sourcing data from the changelog
+
+For this purpose it use a cache key covering changelog content.
+The cache key parts are: (tiprev, tipnode)
+"""
+
+__metaclass__ = abc.ABCMeta
+
+# default key used for an empty cache
+emptykey = (0, node.nullid)
+_cachekeyspec = 'i20s'
+_cachename = None # used for debug message
+
+# Useful "public" function (no need to override them)
+
+def _fetchchangelogdata(self, cachekey, cl):
+"""use a cachekey to fetch incremental data
+
+Exists as its own method to help subclass to reuse it."""
+tiprev = len(cl) - 1
+tipnode = cl.node(tiprev)
+newkey = (tiprev, tipnode)
+tiprev = len(cl) - 1
+if newkey == cachekey:
+return False, [], newkey
+keyrev, keynode = cachekey
+if tiprev < keyrev or cl.node(keyrev) != keynode:
+revs = ()
+if len(cl):
+revs = list(cl.revs(stop=tiprev))
+return True, revs, newkey
+else:
+return False, list(cl.revs(start=keyrev + 1, stop=tiprev)), newkey
+
+def _fetchupdatedata(self, repo):
+return self._fetchchangelogdata(self._cachekey, repo.changelog)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] cleanupnode: do not use generator for node mapping

[Jun Wu]

Looks good to me. Thanks for the cleanup!

Excerpts from Boris Feld's message of 2017-07-09 18:49:04 +0200:
> # HG changeset patch
> # User Octobus 
> # Date 1499605879 -7200
> #  Sun Jul 09 15:11:19 2017 +0200
> # Node ID a6ed7f0670010e5a6551d736b00dc0bf7367bba8
> # Parent  4672db164c986da4442bd864cd044512d975c3f2
> # EXP-Topic fix-cleanup
> cleanupnode: do not use generator for node mapping
> 
> The 'successors' part of the mappings used of be a tuple. This avoid issue 
> from
> code consuming the generator "by mistake". For example, an extension 
> inspecting the
> mapping content used to be able to iterate over the successors mapping without
> consequence.
> 
> Since the mapping are small we do not expect any performance impact we use 
> tuple
> again for this.
> 
> diff -r 4672db164c98 -r a6ed7f067001 mercurial/scmutil.py
> --- a/mercurial/scmutil.pySat Jun 24 15:29:42 2017 -0700
> +++ b/mercurial/scmutil.pySun Jul 09 15:11:19 2017 +0200
> @@ -638,7 +638,7 @@
>  isobs = unfi.obsstore.successors.__contains__
>  torev = unfi.changelog.rev
>  sortfunc = lambda ns: torev(ns[0])
> -rels = [(unfi[n], (unfi[m] for m in s))
> +rels = [(unfi[n], tuple(unfi[m] for m in s))
>  for n, s in sorted(mapping.items(), key=sortfunc)
>  if s or not isobs(n)]
>  obsolete.createmarkers(repo, rels, operation=operation)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH v3] releasenotes: add custom admonitions support for release notes

[Rishabh Madan]

# HG changeset patch
# User Rishabh Madan 
# Date 1499619850 -7200
#  Sun Jul 09 19:04:10 2017 +0200
# Node ID 5f22d3d43d36d46cea98c86b2a49eee2d323fd9e
# Parent  4672db164c986da4442bd864cd044512d975c3f2
releasenotes: add custom admonitions support for release notes

By default, the extension has default sections like fix, feature, perf etc.. 
This
patch allow user to add support for custom admonition. In order to add a custom
admonition, one needs to have a .hgreleasenotes file inside the repository. All 
the
custom directive with name specified under the tag [sections] will be
usable by the extension. One important thing to keep in mind is if there exists 
any
custom admonitions with same key as default then they will override the default 
ones.

diff -r 4672db164c98 -r 5f22d3d43d36 hgext/releasenotes.py
--- a/hgext/releasenotes.py Sat Jun 24 15:29:42 2017 -0700
+++ b/hgext/releasenotes.py Sun Jul 09 19:04:10 2017 +0200
@@ -20,6 +20,7 @@
 
 from mercurial.i18n import _
 from mercurial import (
+config,
 error,
 minirst,
 registrar,
@@ -111,9 +112,25 @@
 self.addnontitleditem(section, paragraphs)
 
 class releasenotessections(object):
-def __init__(self, ui):
-# TODO support defining custom sections from config.
-self._sections = list(DEFAULT_SECTIONS)
+def __init__(self, ui, repo=None):
+if repo:
+custom_sections = getcustomadmonitions(repo)
+if custom_sections:
+self._sections = custom_sections
+for default_name, default_value in DEFAULT_SECTIONS:
+duplicate_key = False
+for name, value in custom_sections:
+if name == default_name:
+duplicate_key = True
+break
+if duplicate_key:
+continue
+else:
+self._sections.append((default_name, default_value))
+else:
+self._sections = list(DEFAULT_SECTIONS)
+else:
+self._sections = list(DEFAULT_SECTIONS)
 
 def __iter__(self):
 return iter(self._sections)
@@ -128,6 +145,29 @@
 
 return None
 
+def getcustomadmonitions(repo):
+custom_sections = list()
+ctx = repo['.']
+p = config.config()
+repo = ctx.repo()
+
+def read(f, sections=None, remap=None):
+if f in ctx:
+data = ctx[f].data()
+p.parse(f, data, sections, remap, read)
+sectiondict = p['sections']
+sectionlist = list()
+for key, value in sectiondict.iteritems():
+temp = (key, value)
+sectionlist.append(temp)
+return sectionlist
+else:
+return
+
+if '.hgreleasenotes' in ctx:
+custom_sections = read('.hgreleasenotes')
+return custom_sections
+
 def parsenotesfromrevisions(repo, directives, revs):
 notes = parsedreleasenotes()
 
@@ -311,7 +351,6 @@
 lines.extend(wrapper.wrap(' '.join(para)))
 
 lines.append('')
-
 if lines[-1]:
 lines.append('')
 
@@ -396,7 +435,7 @@
 that file. A particular use case for this is to tweak the wording of a
 release note after it has been added to the release notes file.
 """
-sections = releasenotessections(ui)
+sections = releasenotessections(ui, repo)
 
 revs = scmutil.revrange(repo, [rev or 'not public()'])
 incoming = parsenotesfromrevisions(repo, sections.names(), revs)
@@ -411,12 +450,11 @@
 notes = parsedreleasenotes()
 
 notes.merge(ui, incoming)
-
 with open(file_, 'wb') as fh:
 fh.write(serializenotes(sections, notes))
 
 @command('debugparsereleasenotes', norepo=True)
-def debugparsereleasenotes(ui, path):
+def debugparsereleasenotes(ui, path, repo=None):
 """parse release notes and print resulting data structure"""
 if path == '-':
 text = sys.stdin.read()
@@ -424,7 +462,7 @@
 with open(path, 'rb') as fh:
 text = fh.read()
 
-sections = releasenotessections(ui)
+sections = releasenotessections(ui, repo)
 
 notes = parsereleasenotesfile(sections, text)
 
diff -r 4672db164c98 -r 5f22d3d43d36 tests/test-releasenotes-formatting.t
--- a/tests/test-releasenotes-formatting.t  Sat Jun 24 15:29:42 2017 -0700
+++ b/tests/test-releasenotes-formatting.t  Sun Jul 09 19:04:10 2017 +0200
@@ -255,6 +255,8 @@
   
   * Short summary of fix 3
 
+  $ cd ..
+
 Multiple 'Other Changes' sub-sections for every section
 
   $ hg init multiple-otherchanges
@@ -324,3 +326,52 @@
   
   * Short summary of fix 2
 
+  $ cd ..
+
+Using custom sections in notes
+
+  $ hg init custom-section
+  $ cd custom-section
+  $ cat >> .hgreleasenotes << EOF
+  > [sections]
+  > testsection=Name of Section
+  > EOF
+
+  $ touch a
+  $ hg -q commit -A -l - << EOF
+  > commit 

Re: [PATCH 5 of 7] histedit: pass multiple nodes to strip (BC)

[Martin von Zweigbergk via Mercurial-devel]

On Sat, Jul 8, 2017 at 4:51 PM, Jun Wu  wrote:
> # HG changeset patch
> # User Jun Wu 
> # Date 1499557831 25200
> #  Sat Jul 08 16:50:31 2017 -0700
> # Node ID 9a325ae88021e0e02a87ef1ae6baa8a199405140
> # Parent  d1b9eb1ad8ac5f9e645337034594862e1ba37092
> # Available At https://bitbucket.org/quark-zju/hg-draft
> #  hg pull https://bitbucket.org/quark-zju/hg-draft -r 
> 9a325ae88021
> histedit: pass multiple nodes to strip (BC)
>
> Previously, histedit.cleanupnode pass root nodes one by one. Since
> repair.strip takes multiple nodes and can handle them just fine, pass all
> strip roots at once.
>
> This is BC because the number of strip backup files may change from N to 1.
>
> diff --git a/hgext/histedit.py b/hgext/histedit.py
> --- a/hgext/histedit.py
> +++ b/hgext/histedit.py
> @@ -1604,9 +1604,6 @@ def cleanupnode(ui, repo, nodes):
>  nodes = sorted(n for n in nodes if n in nm)
>  roots = [c.node() for c in repo.set("roots(%ln)", nodes)]
> -for c in roots:
> -# We should process node in reverse order to strip tip most 
> first.
> -# but this trigger a bug in changegroup hook.
> -# This would reduce bundle overhead

Oddly enough, all the histedit tests pass with your patch applied even
in the revision that introduced this comment -- 4eb13b619785
(histedit: factorise node stripping logic, 2012-09-26). The commit
message didn't mention the problem, so I don't know how to reproduce
the "bug in changegroup hook". We'll just have to assume it's been
fixed (if it was ever there).

> -repair.strip(ui, repo, c)
> +if roots:
> +repair.strip(ui, repo, roots)
>
>  def safecleanupnode(ui, repo, nodes):
> ___
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH] cleanupnode: do not use generator for node mapping

[Boris Feld]

# HG changeset patch
# User Octobus 
# Date 1499605879 -7200
#  Sun Jul 09 15:11:19 2017 +0200
# Node ID a6ed7f0670010e5a6551d736b00dc0bf7367bba8
# Parent  4672db164c986da4442bd864cd044512d975c3f2
# EXP-Topic fix-cleanup
cleanupnode: do not use generator for node mapping

The 'successors' part of the mappings used of be a tuple. This avoid issue from
code consuming the generator "by mistake". For example, an extension inspecting 
the
mapping content used to be able to iterate over the successors mapping without
consequence.

Since the mapping are small we do not expect any performance impact we use tuple
again for this.

diff -r 4672db164c98 -r a6ed7f067001 mercurial/scmutil.py
--- a/mercurial/scmutil.py  Sat Jun 24 15:29:42 2017 -0700
+++ b/mercurial/scmutil.py  Sun Jul 09 15:11:19 2017 +0200
@@ -638,7 +638,7 @@
 isobs = unfi.obsstore.successors.__contains__
 torev = unfi.changelog.rev
 sortfunc = lambda ns: torev(ns[0])
-rels = [(unfi[n], (unfi[m] for m in s))
+rels = [(unfi[n], tuple(unfi[m] for m in s))
 for n, s in sorted(mapping.items(), key=sortfunc)
 if s or not isobs(n)]
 obsolete.createmarkers(repo, rels, operation=operation)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] releasenotes: add custom admonitions support for release notes

[Rishabh Madan]

>
> >  class releasenotessections(object):
> > -def __init__(self, ui):
> > -# TODO support defining custom sections from config.
> > -self._sections = list(DEFAULT_SECTIONS)
> > +def __init__(self, repo, revs, ui):
> > +custom_sections = self.getcustomadmonitions(repo, revs, ui)
> > +if custom_sections:
> > +self._sections = custom_sections
> > +else:
> > +self._sections = list(DEFAULT_SECTIONS) + custom_sections
>  ^^^
>
> Here custom_sections is empty, right?


Yes. I made a mistake here. For the if condition I'll take the summation of
both of them and else would just have default_sections.
Later, I'll just override the key, value pairs in case there is something
common between the two.


> > +custom_sections = list()
> > +for rev in revs:
> > +ctx = repo[rev]
>
> What do you want to do here? Processing all revs or just taking the last
> rev?
>

I was discussing this with Pulkit. And just repo['.'] would be enough here.

>
> > +p = config.config()
> > +repo = ctx.repo()
> > +
> > +def read(f, sections=None, remap=None):
> > +if f in ctx:
> > +try:
> > +data = ctx[f].data()
> > +
> > +except IOError as err:
> > +if err.errno != errno.ENOENT:
> > +raise
> > +ui.warn(_("warning: .hgadmonitions file \'%s\' not
> found\n") %
> > +repo.pathto(f))
> > +return
> > +p.parse(f, data, sections, remap, read)
> > +sectiondict = p.__getitem__(sections)
>  ^^^
> p[sections]
>
> BTW, this sections should not be the same as the sections passed to
> parse().
> Perhaps, you would just need to parse the all sections in config file, and
> pick the one you want.
>

Yes. This is a better approach. That way I can use it in case we plan to
add other type of configs also.


>   p.parse(src, data)
>   return p.items('releasenotes.sections')
>
> > +sectionlist = list()
> > +for key, value in sectiondict.iteritems():
> > +temp = (key, value)
> > +sectionlist.append(temp)
> > +else:
> > +raise error.Abort(_(".hgadmonitions file \'%s\' not
> found") %
> > +  repo.pathto(f))
> > +return sectionlist
> > +if '.hgadmonitions' in ctx:
> > +custom_sections = read('.hgadmonitions',
> 'releasenotes.sections')
>
> .hgadmonitions sounds too obscure. .hgreleasenotes is probably better, and
> the section name could be just [sections] or [admonitions].
>
> > +return custom_sections
> > +
> >  def parsenotesfromrevisions(repo, directives, revs):
> >  notes = parsedreleasenotes()
> >
> > @@ -396,9 +432,9 @@
> >  that file. A particular use case for this is to tweak the wording
> of a
> >  release note after it has been added to the release notes file.
> >  """
> > -sections = releasenotessections(ui)
> > +revs = scmutil.revrange(repo, [rev or 'not public()'])
> >
> > -revs = scmutil.revrange(repo, [rev or 'not public()'])
> > +sections = releasenotessections(repo, revs, ui)
>
> I doubt it's correct to read all .hgadmonitions files. Maybe we would
> simply
> want to use the latest config for the revision you're generating a release
> note?
>

I guess now that I'll use repo['.'] this problem is answered.

>
> > -
> > +print(incoming)
>
> :)
>
> Oops! Totally forgot about it.

And, can you write some tests?


Sure.
ᐧ
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH] releasenotes: add custom admonitions support for release notes

[Yuya Nishihara]

On Fri, 07 Jul 2017 19:13:58 +0200, Rishabh Madan wrote:
> # HG changeset patch
> # User Rishabh Madan 
> # Date 1499447586 -7200
> #  Fri Jul 07 19:13:06 2017 +0200
> # Node ID 766932e4c2768577173f1bfaa10e919b9df739ec
> # Parent  e714159860fd0872ae0555bb07546aa7e9f700e0
> releasenotes: add custom admonitions support for release notes

>  class releasenotessections(object):
> -def __init__(self, ui):
> -# TODO support defining custom sections from config.
> -self._sections = list(DEFAULT_SECTIONS)
> +def __init__(self, repo, revs, ui):
> +custom_sections = self.getcustomadmonitions(repo, revs, ui)
> +if custom_sections:
> +self._sections = custom_sections
> +else:
> +self._sections = list(DEFAULT_SECTIONS) + custom_sections
 ^^^

Here custom_sections is empty, right?

> +def getcustomadmonitions(self, repo, revs, ui):

Nit: this can be a free function since it doesn't depend on self.

> +custom_sections = list()
> +for rev in revs:
> +ctx = repo[rev]

What do you want to do here? Processing all revs or just taking the last rev?

> +p = config.config()
> +repo = ctx.repo()
> +
> +def read(f, sections=None, remap=None):
> +if f in ctx:
> +try:
> +data = ctx[f].data()
> +
> +except IOError as err:
> +if err.errno != errno.ENOENT:
> +raise
> +ui.warn(_("warning: .hgadmonitions file \'%s\' not 
> found\n") %
> +repo.pathto(f))
> +return
> +p.parse(f, data, sections, remap, read)
> +sectiondict = p.__getitem__(sections)
 ^^^
p[sections]

BTW, this sections should not be the same as the sections passed to parse().
Perhaps, you would just need to parse the all sections in config file, and
pick the one you want.

  p.parse(src, data)
  return p.items('releasenotes.sections')

> +sectionlist = list()
> +for key, value in sectiondict.iteritems():
> +temp = (key, value)
> +sectionlist.append(temp)
> +else:
> +raise error.Abort(_(".hgadmonitions file \'%s\' not found") %
> +  repo.pathto(f))
> +return sectionlist
> +if '.hgadmonitions' in ctx:
> +custom_sections = read('.hgadmonitions', 'releasenotes.sections')

.hgadmonitions sounds too obscure. .hgreleasenotes is probably better, and
the section name could be just [sections] or [admonitions].

> +return custom_sections
> +
>  def parsenotesfromrevisions(repo, directives, revs):
>  notes = parsedreleasenotes()
>  
> @@ -396,9 +432,9 @@
>  that file. A particular use case for this is to tweak the wording of a
>  release note after it has been added to the release notes file.
>  """
> -sections = releasenotessections(ui)
> +revs = scmutil.revrange(repo, [rev or 'not public()'])
>  
> -revs = scmutil.revrange(repo, [rev or 'not public()'])
> +sections = releasenotessections(repo, revs, ui)

I doubt it's correct to read all .hgadmonitions files. Maybe we would simply
want to use the latest config for the revision you're generating a release
note?

> -
> +print(incoming)

:)

And, can you write some tests?
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 3 topic-experiment] tutorial: word wrap long lines

[Josef 'Jeff' Sipek]

# HG changeset patch
# User Josef 'Jeff' Sipek 
# Date 1499599407 -10800
#  Sun Jul 09 14:23:27 2017 +0300
# Branch stable
# Node ID c497d62fdcea9861369a15896a1f59d6f3787765
# Parent  db3830646e34220cfcac0837a33f9a8503dea5d2
tutorial: word wrap long lines

diff --git a/tests/test-topic-tutorial.t b/tests/test-topic-tutorial.t
--- a/tests/test-topic-tutorial.t
+++ b/tests/test-topic-tutorial.t
@@ -57,7 +57,8 @@ a topic. Creating a new topic is done us
 
   $ hg topic food
 
-As for named branch, our topic is active but it does not contains any 
changesets yet::
+As for named branch, our topic is active but it does not contains any
+changesets yet::
 
   $ hg topic
* food
@@ -117,7 +118,8 @@ And future commit will be part of that t
  summary: adding condiments
   
 
-We can get a compact view of the content of our topic using the ``stack`` 
command::
+We can get a compact view of the content of our topic using the ``stack``
+command::
 
   $ hg stack
   ### topic: food
@@ -133,7 +135,8 @@ The topic desactivate when we update awa
   $ hg topic
  food
 
-Note that ``default`` (name of the branch) now refers to the tipmost changeset 
of default without a topic::
+Note that ``default`` (name of the branch) now refers to the tipmost
+changeset of default without a topic::
 
   $ hg log --graph
   o  changeset:   2:287de11b401f
@@ -163,7 +166,8 @@ And updating back to the topic reactivat
   $ hg topic
* food
 
-The name used for updating does not affect the activation of the topic, 
updating to a revision part of a topic will activate it in all cases::
+The name used for updating does not affect the activation of the topic,
+updating to a revision part of a topic will activate it in all cases::
 
   $ hg up default
   1 files updated, 0 files merged, 0 files removed, 0 files unresolved
@@ -190,7 +194,8 @@ The name used for updating does not affe
   $ hg commit -A -m "Adding clothes"
   $ cd ../client
 
-Topic will also affect rebase and merge destination. Let's pull the latest 
update from the main server::
+Topic will also affect rebase and merge destination. Let's pull the latest
+update from the main server::
 
   $ hg pull
   pulling from $TESTTMP/server (glob)
@@ -226,7 +231,8 @@ Topic will also affect rebase and merge 
  summary: Shopping list
   
 
-The topic head will not be considered when merge from the new head of the 
branch::
+The topic head will not be considered when merge from the new head of the
+branch::
 
   $ hg up default
   1 files updated, 0 files merged, 0 files removed, 0 files unresolved
@@ -367,7 +373,8 @@ The information ``hg stack`` command ada
   t1: Adding hammer
 ^ adding fruits
 
-They are seen as independant branch by Mercurial. No rebase or merge betwen 
them will be attempted by default::
+They are seen as independant branch by Mercurial. No rebase or merge betwen
+them will be attempted by default::
 
   $ hg rebase
   nothing to rebase
@@ -400,7 +407,8 @@ Lets see what other people did in the me
   added 2 changesets with 2 changes to 1 files (+1 heads)
   (run 'hg heads' to see heads)
 
-There is new changes! We can simply use ``hg rebase`` to update our changeset 
on top of the latest::
+There is new changes! We can simply use ``hg rebase`` to update our
+changeset on top of the latest::
 
   $ hg rebase
   rebasing 6:183984ef46d1 "Adding hammer"
@@ -411,13 +419,15 @@ There is new changes! We can simply use 
   rebasing 8:34255b455dac "Adding drill"
   merging shopping
 
-But what about the other topic? You can use 'hg topic --verbose' to see 
information about them::
+But what about the other topic? You can use 'hg topic --verbose' to see
+information about them::
 
   $ hg topic --verbose
  drinks (on branch: default, 2 changesets, 2 behind)
* tools  (on branch: default, 3 changesets)
 
-The "2 behind" is telling you that there is 2 new changesets on the named 
branch of the topic. You need to merge or rebase to incorporate them.
+The "2 behind" is telling you that there is 2 new changesets on the named
+branch of the topic. You need to merge or rebase to incorporate them.
 
 Pushing that topic would create a new heads will be prevented::
 
@@ -429,7 +439,8 @@ Pushing that topic would create a new he
   [255]
 
 
-Even after a rebase Pushing all active topics at the same time will complains 
about the multiple heads it would create on that branch::
+Even after a rebase Pushing all active topics at the same time will
+complains about the multiple heads it would create on that branch::
 
   $ hg rebase -b drinks
   rebasing 9:8dfa45bd5e0c "Adding apple juice"
@@ -445,7 +456,8 @@ Even after a rebase Pushing all active t
   (merge or see 'hg help push' for details about pushing new heads)
   [255]
 
-Publishing only one of them is allowed (as long as it does not create a new 
branch head has we just saw in the previous case)::
+Publishing only one of them is allowed (as long as it does not create a new
+branch 

[PATCH 3 of 3 topic-experiment] tutorial: fix grammar and spelling

[Josef 'Jeff' Sipek]

# HG changeset patch
# User Josef 'Jeff' Sipek 
# Date 1499601692 -10800
#  Sun Jul 09 15:01:32 2017 +0300
# Branch stable
# Node ID d3297cb2c810432906d8cf4b45e412f2f87241b0
# Parent  c497d62fdcea9861369a15896a1f59d6f3787765
tutorial: fix grammar and spelling

diff --git a/tests/test-topic-tutorial.t b/tests/test-topic-tutorial.t
--- a/tests/test-topic-tutorial.t
+++ b/tests/test-topic-tutorial.t
@@ -42,7 +42,7 @@ their unfinished work.
 Topic Basics
 
 
-Let's says we use Mercurial to manage our shopping list::
+Let's say we use Mercurial to manage our shopping list::
 
   $ hg log --graph
   @  changeset:   0:38da43f0a2ea
@@ -52,12 +52,12 @@ Let's says we use Mercurial to manage ou
  summary: Shopping list
   
 
-We are about to do some edition to this list and would like to do them within
-a topic. Creating a new topic is done using the ``topic`` command::
+We are about to make some additions to this list and would like to do them
+within a topic. Creating a new topic is done using the ``topic`` command::
 
   $ hg topic food
 
-As for named branch, our topic is active but it does not contains any
+Much like a named branch, our topic is active but it does not contain any
 changesets yet::
 
   $ hg topic
@@ -95,7 +95,7 @@ Our next commit will be part of the acti
  summary: adding condiments
   
 
-And future commit will be part of that topic too::
+And future commits will be part of that topic too::
 
   $ cat >> shopping << EOF
   > Bananas
@@ -128,7 +128,7 @@ command::
   t1: adding condiments
 ^ Shopping list
 
-The topic desactivate when we update away from it::
+The topic deactivates when we update away from it::
 
   $ hg up default
   1 files updated, 0 files merged, 0 files removed, 0 files unresolved
@@ -158,7 +158,7 @@ changeset of default without a topic::
  summary: Shopping list
   
 
-And updating back to the topic reactivate it::
+And updating back to the topic reactivates it::
 
   $ hg up food
   switching to topic food
@@ -166,8 +166,8 @@ And updating back to the topic reactivat
   $ hg topic
* food
 
-The name used for updating does not affect the activation of the topic,
-updating to a revision part of a topic will activate it in all cases::
+Updating to any changeset that is part of a topic activates the topic
+regardless of how the revision was specified::
 
   $ hg up default
   1 files updated, 0 files merged, 0 files removed, 0 files unresolved
@@ -194,8 +194,8 @@ updating to a revision part of a topic w
   $ hg commit -A -m "Adding clothes"
   $ cd ../client
 
-Topic will also affect rebase and merge destination. Let's pull the latest
-update from the main server::
+The topic will also affect the rebase and the merge destinations. Let's pull
+the latest update from the main server::
 
   $ hg pull
   pulling from $TESTTMP/server (glob)
@@ -231,7 +231,7 @@ update from the main server::
  summary: Shopping list
   
 
-The topic head will not be considered when merge from the new head of the
+The topic head will not be considered when merging from the new head of the
 branch::
 
   $ hg up default
@@ -278,7 +278,7 @@ But the topic will see that branch head 
  summary: Shopping list
   
 
-The topic information will fade out when we publish the changesets::
+The topic information will disappear when we publish the changesets::
 
   $ hg topic
* food
@@ -321,12 +321,12 @@ The topic information will fade out when
 Working with Multiple Topics
 
 
-In the above example, topic are not bring much benefit since you only have one
-line of developement. Topic start to be more useful when you have to work on
-multiple features are the same time.
+In the above example, topics do not bring much benefit since you only have one
+line of development. Topics start to be more useful when you have to work on
+multiple features at the same time.
 
 We might go shopping in a hardware store in the same go, so let's add some
-tools to the shopping list withing a new topic::
+tools to the shopping list within a new topic::
 
   $ hg topic tools
   $ echo hammer >> shopping
@@ -336,9 +336,9 @@ tools to the shopping list withing a new
   $ echo drill >> shopping
   $ hg ci -m 'Adding drill'
 
-But are not sure to actually go in the hardward store, so in the meantime, we
-want to extend the list with drinks. We go back to the official default branch
-and start a new topic::
+But we are not sure we will actually go to the hardware store, so in the
+meantime, we want to extend the list with drinks. We go back to the official
+default branch and start a new topic::
 
   $ hg up default
   1 files updated, 0 files merged, 0 files removed, 0 files unresolved
@@ -354,7 +354,7 @@ We now have two topics::
* drinks
  tools
 
-The information ``hg stack`` command adapt to the active topic::
+The information displayed by ``hg stack`` adapts to the active topic::
 
   $ hg stack
   ### topic: 

[PATCH 1 of 3 topic-experiment] tutorial: use rm instead of 'hg rm' for an untracked temporary file

[Josef 'Jeff' Sipek]

# HG changeset patch
# User Josef 'Jeff' Sipek 
# Date 1499599224 -10800
#  Sun Jul 09 14:20:24 2017 +0300
# Branch stable
# Node ID db3830646e34220cfcac0837a33f9a8503dea5d2
# Parent  61e73c8fe169717105e832b23086683848a9ef53
tutorial: use rm instead of 'hg rm' for an untracked temporary file

diff --git a/tests/test-topic-tutorial.t b/tests/test-topic-tutorial.t
--- a/tests/test-topic-tutorial.t
+++ b/tests/test-topic-tutorial.t
@@ -385,9 +385,7 @@ They are seen as independant branch by M
   $ echo 'Coat' > shopping
   $ echo 'Shoes' >> shopping
   $ cat foo >> shopping
-  $ hg rm foo
-  not removing foo: file is untracked
-  [1]
+  $ rm foo
   $ hg ci -m 'add a pair of shoes'
   $ cd ../client
 
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel