D1393: bundle2: inline struct operations

[indygreg (Gregory Szorc)]

indygreg added a comment.


  FWIW, when I last looked at adding stream clones to bundle2, I held off 
because of the performance overhead. With the performance work in this series, 
I suspect bundle2's I/O is fast enough to support stream clones with minimal 
performance degradation. I may have a go at that once this series is queued 
because it is always something I've wanted to do.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1393

To: indygreg, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1394: bundle2: avoid unbound read when seeking

[indygreg (Gregory Szorc)]

indygreg created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  Currently, seekableunbundlepart.seek() will perform a read() during
  seek operations. This will allocate a buffer to hold the raw data
  over the seek distance. This can lead to very large allocations
  and cause performance to suffer.
  
  We change the code to perform read(32768) in a loop to avoid
  potentially large allocations.
  
  `hg perfbundleread` on an uncompressed Firefox bundle reveals
  a performance impact:
  
  ! bundle2 iterparts()
  ! wall 2.992605 comb 2.99 user 2.26 sys 0.73 (best of 4)
  ! bundle2 iterparts() seekable
  ! wall 3.863810 comb 3.86 user 3.00 sys 0.86 (best of 3)
  ! bundle2 part seek()
  ! wall 6.213387 comb 6.20 user 3.35 sys 2.85 (best of 3)
  ! wall 3.820347 comb 3.81 user 2.98 sys 0.83 (best of 3)
  
  Since seekable bundle parts are (only) used by bundlerepo, this /may/
  speed up initial loading of bundle-based repos. But any improvement
  will likely only be noticed on very large bundles.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1394

AFFECTED FILES
  mercurial/bundle2.py

CHANGE DETAILS

diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
--- a/mercurial/bundle2.py
+++ b/mercurial/bundle2.py
@@ -1415,13 +1415,20 @@
 newpos = self._pos + offset
 elif whence == os.SEEK_END:
 if not self.consumed:
-self.read()
+# Can't use self.consume() here because it advances self._pos.
+chunk = self.read(32768)
+while chunk:
+chunk = self.read(32768)
 newpos = self._chunkindex[-1][0] - offset
 else:
 raise ValueError('Unknown whence value: %r' % (whence,))
 
 if newpos > self._chunkindex[-1][0] and not self.consumed:
-self.read()
+# Can't use self.consume() here because it advances self._pos.
+chunk = self.read(32768)
+while chunk:
+chunk = self.read(32668)
+
 if not 0 <= newpos <= self._chunkindex[-1][0]:
 raise ValueError('Offset out of range')
 



To: indygreg, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1393: bundle2: inline struct operations

[indygreg (Gregory Szorc)]

indygreg created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  Before, we were calling struct.unpack() (via an alias) on every
  loop iteration. I'm not sure what Python does under the hood, but
  it would have to look at the struct format and determine what to
  do.
  
  This commit establishes a struct.Struct instance and reuses it for
  struct reading.
  
  We can see the impact from running `hg perfbundleread` on a Firefox
  bundle:
  
  ! read(8k)
  ! wall 0.679730 comb 0.68 user 0.14 sys 0.54 (best of 15)
  ! read(16k)
  ! wall 0.577228 comb 0.57 user 0.08 sys 0.49 (best of 17)
  ! read(32k)
  ! wall 0.516060 comb 0.52 user 0.04 sys 0.48 (best of 20)
  ! read(128k)
  ! wall 0.496378 comb 0.49 user 0.01 sys 0.48 (best of 20)
  ! bundle2 iterparts()
  ! wall 3.056811 comb 3.05 user 2.34 sys 0.71 (best of 4)
  ! wall 2.992605 comb 2.99 user 2.26 sys 0.73 (best of 4)
  ! bundle2 iterparts() seekable
  ! wall 4.007676 comb 4.00 user 3.17 sys 0.83 (best of 3)
  ! wall 3.863810 comb 3.86 user 3.00 sys 0.86 (best of 3)
  ! bundle2 part seek()
  ! wall 6.267110 comb 6.25 user 3.48 sys 2.77 (best of 3)
  ! wall 6.213387 comb 6.20 user 3.35 sys 2.85 (best of 3)
  ! bundle2 part read(8k)
  ! wall 3.404164 comb 3.40 user 2.65 sys 0.75 (best of 3)
  ! wall 3.241099 comb 3.25 user 2.56 sys 0.69 (best of 3)
  ! bundle2 part read(16k)
  ! wall 3.197972 comb 3.20 user 2.49 sys 0.71 (best of 4)
  ! wall 3.003930 comb 3.00 user 2.27 sys 0.73 (best of 4)
  ! bundle2 part read(32k)
  ! wall 3.060557 comb 3.06 user 2.34 sys 0.72 (best of 4)
  ! wall 2.904695 comb 2.90 user 2.16 sys 0.74 (best of 4)
  ! bundle2 part read(128k)
  ! wall 2.952209 comb 2.95 user 2.23 sys 0.72 (best of 4)
  ! wall 2.776140 comb 2.78 user 2.07 sys 0.71 (best of 4)
  
  Profiling now says most remaining time is spent in util.chunkbuffer.
  I already heavily optimized that data structure several releases ago.
  So we'll likely get little more performance out of bundle2 reading
  while still retaining util.chunkbuffer().

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1393

AFFECTED FILES
  mercurial/bundle2.py

CHANGE DETAILS

diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
--- a/mercurial/bundle2.py
+++ b/mercurial/bundle2.py
@@ -1196,11 +1196,14 @@
 dolog = ui.configbool('devel', 'bundle2.debug')
 debug = ui.debug
 
-headersize = struct.calcsize(_fpayloadsize)
+headerstruct = struct.Struct(_fpayloadsize)
+headersize = headerstruct.size
+unpack = headerstruct.unpack
+
 readexactly = changegroup.readexactly
 read = fh.read
 
-chunksize = _unpack(_fpayloadsize, readexactly(fh, headersize))[0]
+chunksize = unpack(readexactly(fh, headersize))[0]
 indebug(ui, 'payload chunk size: %i' % chunksize)
 
 # changegroup.readexactly() is inlined below for performance.
@@ -1227,7 +1230,7 @@
 ' (got %d bytes, expected %d)') %
   (len(s), chunksize))
 
-chunksize = _unpack(_fpayloadsize, s)[0]
+chunksize = unpack(s)[0]
 
 # indebug() inlined for performance.
 if dolog:



To: indygreg, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1391: bundle2: inline debug logging

[indygreg (Gregory Szorc)]

indygreg created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  Profiling revealed that repeated calls to indebug() were
  consuming a fair amount of CPU during bundle2 reading, with
  most of the time spent in ui.configbool().
  
  Inlining indebug() and avoiding extra attribute lookups speeds
  things up substantially. Using `hg perfbundleread` with a Firefox
  bundle:
  
  ! read(8k)
  ! wall 0.679730 comb 0.68 user 0.14 sys 0.54 (best of 15)
  ! read(16k)
  ! wall 0.577228 comb 0.57 user 0.08 sys 0.49 (best of 17)
  ! read(32k)
  ! wall 0.516060 comb 0.52 user 0.04 sys 0.48 (best of 20)
  ! read(128k)
  ! wall 0.496378 comb 0.49 user 0.01 sys 0.48 (best of 20)
  ! bundle2 iterparts()
  ! wall 6.983756 comb 6.98 user 6.22 sys 0.76 (best of 3)
  ! wall 3.460903 comb 3.46 user 2.76 sys 0.70 (best of 3)
  ! bundle2 iterparts() seekable
  ! wall 8.132131 comb 8.11 user 7.16 sys 0.95 (best of 3)
  ! wall 4.312722 comb 4.31 user 3.48 sys 0.83 (best of 3)
  ! bundle2 part seek()
  ! wall 10.860942 comb 10.84 user 7.79 sys 3.05 (best of 3)
  ! wall 6.754764 comb 6.74 user 3.97 sys 2.77 (best of 3)
  ! bundle2 part read(8k)
  ! wall 7.258035 comb 7.26 user 6.47 sys 0.79 (best of 3)
  ! wall 3.668004 comb 3.66 user 2.96 sys 0.70 (best of 3)
  ! bundle2 part read(16k)
  ! wall 7.099891 comb 7.08 user 6.31 sys 0.77 (best of 3)
  ! wall 3.489196 comb 3.48 user 2.75 sys 0.73 (best of 3)
  ! bundle2 part read(32k)
  ! wall 6.964685 comb 6.95 user 6.13 sys 0.82 (best of 3)
  ! wall 3.388569 comb 3.38 user 2.64 sys 0.74 (best of 3)
  ! bundle2 part read(128k)
  ! wall 6.852867 comb 6.85 user 6.06 sys 0.79 (best of 3)
  ! wall 3.276415 comb 3.27 user 2.56 sys 0.71 (best of 4)

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1391

AFFECTED FILES
  mercurial/bundle2.py

CHANGE DETAILS

diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
--- a/mercurial/bundle2.py
+++ b/mercurial/bundle2.py
@@ -1193,6 +1193,9 @@
 Part payload data consists of framed chunks. This function takes
 a file handle and emits those chunks.
 """
+dolog = ui.configbool('devel', 'bundle2.debug')
+debug = ui.debug
+
 headersize = struct.calcsize(_fpayloadsize)
 readexactly = changegroup.readexactly
 
@@ -1211,7 +1214,10 @@
 'negative payload chunk size: %s' % chunksize)
 
 chunksize = _unpack(_fpayloadsize, readexactly(fh, headersize))[0]
-indebug(ui, 'payload chunk size: %i' % chunksize)
+
+# indebug() inlined for performance.
+if dolog:
+debug('bundle2-input: payload chunk size: %i\n' % chunksize)
 
 class unbundlepart(unpackermixin):
 """a bundle part read from a bundle"""



To: indygreg, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1392: bundle2: inline changegroup.readexactly()

[indygreg (Gregory Szorc)]

indygreg created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  Profiling reveals this loop is pretty tight. Literally any
  function call elimination can make a big difference.
  
  This commit inlines the relatively trivial changegroup.readexactly()
  method inside the loop.
  
  The results with `hg perfbundleread` on a bundle of the Firefox repo
  speak for themselves:
  
  ! read(8k)
  ! wall 0.679730 comb 0.68 user 0.14 sys 0.54 (best of 15)
  ! read(16k)
  ! wall 0.577228 comb 0.57 user 0.08 sys 0.49 (best of 17)
  ! read(32k)
  ! wall 0.516060 comb 0.52 user 0.04 sys 0.48 (best of 20)
  ! read(128k)
  ! wall 0.496378 comb 0.49 user 0.01 sys 0.48 (best of 20)
  ! bundle2 iterparts()
  ! wall 3.460903 comb 3.46 user 2.76 sys 0.70 (best of 3)
  ! wall 3.056811 comb 3.05 user 2.34 sys 0.71 (best of 4)
  ! bundle2 iterparts() seekable
  ! wall 4.312722 comb 4.31 user 3.48 sys 0.83 (best of 3)
  ! wall 4.007676 comb 4.00 user 3.17 sys 0.83 (best of 3)
  ! bundle2 part seek()
  ! wall 6.754764 comb 6.74 user 3.97 sys 2.77 (best of 3)
  ! wall 6.267110 comb 6.25 user 3.48 sys 2.77 (best of 3)
  ! bundle2 part read(8k)
  ! wall 3.668004 comb 3.66 user 2.96 sys 0.70 (best of 3)
  ! wall 3.404164 comb 3.40 user 2.65 sys 0.75 (best of 3)
  ! bundle2 part read(16k)
  ! wall 3.489196 comb 3.48 user 2.75 sys 0.73 (best of 3)
  ! wall 3.197972 comb 3.20 user 2.49 sys 0.71 (best of 4)
  ! bundle2 part read(32k)
  ! wall 3.388569 comb 3.38 user 2.64 sys 0.74 (best of 3)
  ! wall 3.060557 comb 3.06 user 2.34 sys 0.72 (best of 4)
  ! bundle2 part read(128k)
  ! wall 3.276415 comb 3.27 user 2.56 sys 0.71 (best of 4)
  ! wall 2.952209 comb 2.95 user 2.23 sys 0.72 (best of 4)

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1392

AFFECTED FILES
  mercurial/bundle2.py

CHANGE DETAILS

diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
--- a/mercurial/bundle2.py
+++ b/mercurial/bundle2.py
@@ -1198,22 +1198,36 @@
 
 headersize = struct.calcsize(_fpayloadsize)
 readexactly = changegroup.readexactly
+read = fh.read
 
 chunksize = _unpack(_fpayloadsize, readexactly(fh, headersize))[0]
 indebug(ui, 'payload chunk size: %i' % chunksize)
 
+# changegroup.readexactly() is inlined below for performance.
 while chunksize:
 if chunksize >= 0:
-yield readexactly(fh, chunksize)
+s = read(chunksize)
+if len(s) < chunksize:
+raise error.Abort(_('stream ended unexpectedly '
+' (got %d bytes, expected %d)') %
+  (len(s), chunksize))
+
+yield s
 elif chunksize == flaginterrupt:
 # Interrupt "signal" detected. The regular stream is interrupted
 # and a bundle2 part follows. Consume it.
 interrupthandler(ui, fh)()
 else:
 raise error.BundleValueError(
 'negative payload chunk size: %s' % chunksize)
 
-chunksize = _unpack(_fpayloadsize, readexactly(fh, headersize))[0]
+s = read(headersize)
+if len(s) < headersize:
+raise error.Abort(_('stream ended unexpectedly '
+' (got %d bytes, expected %d)') %
+  (len(s), chunksize))
+
+chunksize = _unpack(_fpayloadsize, s)[0]
 
 # indebug() inlined for performance.
 if dolog:



To: indygreg, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1390: bundle2: don't use seekable bundle2 parts by default (issue5691)

[indygreg (Gregory Szorc)]

indygreg created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  The last commit removed the last use of the bundle2 part seek() API
  in the generic bundle2 part iteration code. This means we can now
  switch to using unseekable bundle2 parts by default and have the
  special consumers that actually need the behavior request it.
  
  This commit changes unbundle20.iterparts() to expose non-seekable
  unbundlepart instances by default. If seekable parts are needed,
  callers can pass "seekable=True." The bundlerepo class needs
  seekable parts, so it does this.
  
  The interrupt handler is also changed to use a regular unbundlepart.
  So, by default, all consumers except bundlerepo will see unseekable
  parts.
  
  Because the behavior of the iterparts() benchmark changed, we add
  a variation to test seekable parts vs unseekable parts. And because
  parts no longer have seek() unless "seekable=True," we update the
  "part seek" benchmark.
  
  Speaking of benchmarks, this change has the following impact to
  `hg perfbundleread` on an uncompressed bundle of the Firefox repo
  (6,070,036,163 bytes):
  
  ! read(8k)
  ! wall 0.722709 comb 0.72 user 0.15 sys 0.57 (best of 14)
  ! read(16k)
  ! wall 0.602208 comb 0.59 user 0.08 sys 0.51 (best of 17)
  ! read(32k)
  ! wall 0.554018 comb 0.56 user 0.05 sys 0.51 (best of 18)
  ! read(128k)
  ! wall 0.520086 comb 0.53 user 0.02 sys 0.51 (best of 20)
  ! bundle2 forwardchunks()
  ! wall 2.996329 comb 3.00 user 2.30 sys 0.70 (best of 4)
  ! bundle2 iterparts()
  ! wall 8.070791 comb 8.06 user 7.18 sys 0.88 (best of 3)
  ! wall 6.983756 comb 6.98 user 6.22 sys 0.76 (best of 3)
  ! bundle2 iterparts() seekable
  ! wall 8.132131 comb 8.11 user 7.16 sys 0.95 (best of 3)
  ! bundle2 part seek()
  ! wall 10.370142 comb 10.35 user 7.43 sys 2.92 (best of 3)
  ! wall 10.860942 comb 10.84 user 7.79 sys 3.05 (best of 3)
  ! bundle2 part read(8k)
  ! wall 8.599892 comb 8.58 user 7.72 sys 0.86 (best of 3)
  ! wall 7.258035 comb 7.26 user 6.47 sys 0.79 (best of 3)
  ! bundle2 part read(16k)
  ! wall 8.265361 comb 8.25 user 7.36 sys 0.89 (best of 3)
  ! wall 7.099891 comb 7.08 user 6.31 sys 0.77 (best of 3)
  ! bundle2 part read(32k)
  ! wall 8.290308 comb 8.28 user 7.33 sys 0.95 (best of 3)
  ! wall 6.964685 comb 6.95 user 6.13 sys 0.82 (best of 3)
  ! bundle2 part read(128k)
  ! wall 8.204900 comb 8.15 user 7.21 sys 0.94 (best of 3)
  ! wall 6.852867 comb 6.85 user 6.06 sys 0.79 (best of 3)
  
  The significant speedup is due to not incurring the overhead to track
  payload offset data. Of course, this overhead is proportional to
  bundle2 part size. So a multiple gigabyte changegroup part is on the
  extreme side of the spectrum for real-world impact.
  
  In addition to the CPU efficiency wins, not tracking offset data
  also means not using memory to hold that data. Using a bundle based on
  the example BSD repository in issue 5691, this change has a drastic
  impact to memory usage during `hg unbundle` (`hg clone` would behave
  similarly). Before, memory usage incrementally increased for the
  duration of bundle processing. In other words, as we advanced through
  the changegroup and bundle2 part, we kept allocating more memory to
  hold offset data. After this change, we still increase memory during
  changegroup application. But the rate of increase is significantly
  slower. (I suspect there is a memory leak somewhere, but that's for
  another commit.)
  
  The RSS at the end of filelog application is as follows:
  
  Before: ~752 MB
  After:  ~567 MB
  
  So, we were storing ~185 MB of offset data that we never even used.
  Talk about wasteful!
  
  .. api::
  
bundle2 parts are no longer seekable by default.
  
  .. perf::
  
bundle2 read I/O throughput significantly increased.
  
  .. perf::
  
Significant memory use reductions when reading from bundle2 bundles.

On the BSD repository, peak RSS during changegroup application
decreased by ~185 MB from ~752 MB to ~567 MB.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1390

AFFECTED FILES
  contrib/perf.py
  mercurial/bundle2.py
  mercurial/bundlerepo.py

CHANGE DETAILS

diff --git a/mercurial/bundlerepo.py b/mercurial/bundlerepo.py
--- a/mercurial/bundlerepo.py
+++ b/mercurial/bundlerepo.py
@@ -289,7 +289,7 @@
 self._cgunpacker = None
 
 cgpart = None
-for part in bundle.iterparts():
+for part in bundle.iterparts(seekable=True):
 if part.type == 'changegroup':
 if cgpart:
 raise NotImplementedError("can't process "
diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
--- a

D1389: bundle2: only seek to beginning of part in bundlerepo

[indygreg (Gregory Szorc)]

indygreg created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  For reasons still not yet fully understood by me, bundlerepo
  requires its changegroup bundle2 part to be seeked to beginning
  after part iteration. As far as I can tell, it is the only
  bundle2 part consumer that relies on this behavior.
  
  This seeking was performed in the generic iterparts() API. Again,
  I don't fully understand why it was here and not in bundlerepo.
  Probably historical reasons.
  
  What I do know is that all other bundle2 part consumers don't
  need this special behavior (assuming the tests are comprehensive).
  So, we move the code from bundle2's iterparts() to bundlerepo's
  consumption of iterparts().

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1389

AFFECTED FILES
  mercurial/bundle2.py
  mercurial/bundlerepo.py

CHANGE DETAILS

diff --git a/mercurial/bundlerepo.py b/mercurial/bundlerepo.py
--- a/mercurial/bundlerepo.py
+++ b/mercurial/bundlerepo.py
@@ -288,18 +288,26 @@
 self._bundlefile = bundle
 self._cgunpacker = None
 
-hadchangegroup = False
+cgpart = None
 for part in bundle.iterparts():
 if part.type == 'changegroup':
-if hadchangegroup:
+if cgpart:
 raise NotImplementedError("can't process "
   "multiple changegroups")
-hadchangegroup = True
+cgpart = part
 
 self._handlebundle2part(bundle, part)
 
-if not hadchangegroup:
+if not cgpart:
 raise error.Abort(_("No changegroups found"))
+
+# This is required to placate a later consumer, which expects
+# the payload offset to be at the beginning of the changegroup.
+# We need to do this after the iterparts() generator advances
+# because iterparts() will seek to end of payload after the
+# generator returns control to iterparts().
+cgpart.seek(0, os.SEEK_SET)
+
 elif isinstance(bundle, changegroup.cg1unpacker):
 if bundle.compressed():
 f = self._writetempbundle(bundle.read, '.hg10un',
diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
--- a/mercurial/bundle2.py
+++ b/mercurial/bundle2.py
@@ -859,9 +859,7 @@
 # Ensure part is fully consumed so we can start reading the next
 # part.
 part.consume()
-# But then seek back to the beginning so the code consuming this
-# generator has a part that starts at 0.
-part.seek(0, os.SEEK_SET)
+
 headerblock = self._readpartheader()
 indebug(self.ui, 'end of bundle2 stream')
 



To: indygreg, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1386: bundle2: extract logic for seeking bundle2 part into own class

[indygreg (Gregory Szorc)]

indygreg created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  Currently, unbundlepart classes support bi-directional seeking.
  Most consumers of unbundlepart only ever seek forward - typically
  as part of moving to the end of the bundle part so they can move
  on to the next one. But regardless of the actual usage of the
  part, instances maintain an index mapping offsets within the
  underlying raw payload to offsets within the decoded payload.
  
  Maintaining the mapping of offset data can be expensive in terms of
  memory use. Furthermore, many bundle2 consumers don't have access
  to an underlying seekable stream. This includes all compressed
  bundles. So maintaining offset data when the underlying stream
  can't be seeked anyway is wasteful. And since many bundle2 streams
  can't be seeked, it seems like a bad idea to expose a seek API
  in bundle2 parts by default. If you provide them, people will
  attempt to use them.
  
  Seekable bundle2 parts should be the exception, not the rule. This
  commit starts the process dividing unbundlepart into 2 classes: a
  base class that supports linear, one-time reads and a child class
  that supports bi-directional seeking. In this first commit, we
  split various methods and attributes out into a new
  "seekableunbundlepart" class. Previous instantiators of "unbundlepart"
  now instantiate "seekableunbundlepart." This preserves backwards
  compatibility. The coupling between the classes is still tight:
  "unbundlepart" cannot be used on its own. This will be addressed
  in subsequent commits.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1386

AFFECTED FILES
  mercurial/bundle2.py

CHANGE DETAILS

diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
--- a/mercurial/bundle2.py
+++ b/mercurial/bundle2.py
@@ -854,7 +854,7 @@
 indebug(self.ui, 'start extraction of bundle2 parts')
 headerblock = self._readpartheader()
 while headerblock is not None:
-part = unbundlepart(self.ui, headerblock, self._fp)
+part = seekableunbundlepart(self.ui, headerblock, self._fp)
 yield part
 # Seek to the end of the part to force it's consumption so the next
 # part can be read. But then seek back to the beginning so the
@@ -1155,7 +1155,7 @@
 if headerblock is None:
 indebug(self.ui, 'no part found during interruption.')
 return
-part = unbundlepart(self.ui, headerblock, self._fp)
+part = seekableunbundlepart(self.ui, headerblock, self._fp)
 op = interruptoperation(self.ui)
 hardabort = False
 try:
@@ -1207,10 +1207,8 @@
 self.advisoryparams = None
 self.params = None
 self.mandatorykeys = ()
-self._payloadstream = None
 self._readheader()
 self._mandatory = None
-self._chunkindex = [] #(payload, file) position tuples for chunk starts
 self._pos = 0
 
 def _fromheader(self, size):
@@ -1237,46 +1235,6 @@
 self.params.update(self.advisoryparams)
 self.mandatorykeys = frozenset(p[0] for p in mandatoryparams)
 
-def _payloadchunks(self, chunknum=0):
-'''seek to specified chunk and start yielding data'''
-if len(self._chunkindex) == 0:
-assert chunknum == 0, 'Must start with chunk 0'
-self._chunkindex.append((0, self._tellfp()))
-else:
-assert chunknum < len(self._chunkindex), \
-   'Unknown chunk %d' % chunknum
-self._seekfp(self._chunkindex[chunknum][1])
-
-pos = self._chunkindex[chunknum][0]
-payloadsize = self._unpack(_fpayloadsize)[0]
-indebug(self.ui, 'payload chunk size: %i' % payloadsize)
-while payloadsize:
-if payloadsize == flaginterrupt:
-# interruption detection, the handler will now read a
-# single part and process it.
-interrupthandler(self.ui, self._fp)()
-elif payloadsize < 0:
-msg = 'negative payload chunk size: %i' %  payloadsize
-raise error.BundleValueError(msg)
-else:
-result = self._readexact(payloadsize)
-chunknum += 1
-pos += payloadsize
-if chunknum == len(self._chunkindex):
-self._chunkindex.append((pos, self._tellfp()))
-yield result
-payloadsize = self._unpack(_fpayloadsize)[0]
-indebug(self.ui, 'payload chunk size: %i' % payloadsize)
-
-def _findchunk(self, pos):
-'''for a given payload position, return a chunk number and offset'''
-for chunk, (ppos, fpos) in enumerate(self._chunkindex):
-if ppos == pos:
-return chunk, 0
-elif ppos > pos:
-return chunk - 1, pos - self._chunkind

D1385: perf: add command to benchmark bundle reading

[indygreg (Gregory Szorc)]

indygreg created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  Upcoming commits will be refactoring bundle2 I/O code.
  
  This commit establishes a `hg perfbundleread` command that measures
  how long it takes to read a bundle using various mechanisms.
  
  As a baseline, here's output from an uncompressed bundle1
  bundle of my Firefox repo (7,098,622,890 bytes):
  
  ! read(8k)
  ! wall 0.763481 comb 0.76 user 0.16 sys 0.60 (best of 6)
  ! read(16k)
  ! wall 0.644512 comb 0.64 user 0.11 sys 0.53 (best of 16)
  ! read(32k)
  ! wall 0.581172 comb 0.59 user 0.06 sys 0.53 (best of 18)
  ! read(128k)
  ! wall 0.535183 comb 0.53 user 0.01 sys 0.52 (best of 19)
  ! cg1 deltaiter()
  ! wall 0.873500 comb 0.88 user 0.84 sys 0.04 (best of 12)
  ! cg1 getchunks()
  ! wall 6.283797 comb 6.27 user 5.57 sys 0.70 (best of 3)
  ! cg1 read(8k)
  ! wall 1.097173 comb 1.10 user 0.40 sys 0.70 (best of 10)
  ! cg1 read(16k)
  ! wall 0.810750 comb 0.80 user 0.20 sys 0.60 (best of 13)
  ! cg1 read(32k)
  ! wall 0.671215 comb 0.67 user 0.11 sys 0.56 (best of 15)
  ! cg1 read(128k)
  ! wall 0.597857 comb 0.60 user 0.02 sys 0.58 (best of 15)
  
  And from an uncompressed bundle2 bundle (6,070,036,163 bytes):
  
  ! read(8k)
  ! wall 0.676997 comb 0.68 user 0.16 sys 0.52 (best of 15)
  ! read(16k)
  ! wall 0.592706 comb 0.59 user 0.08 sys 0.51 (best of 17)
  ! read(32k)
  ! wall 0.529395 comb 0.53 user 0.05 sys 0.48 (best of 16)
  ! read(128k)
  ! wall 0.491270 comb 0.49 user 0.01 sys 0.48 (best of 19)
  ! bundle2 forwardchunks()
  ! wall 2.997131 comb 2.99 user 2.27 sys 0.72 (best of 4)
  ! bundle2 iterparts()
  ! wall 12.247197 comb 10.67 user 8.17 sys 2.50 (best of 3)
  ! bundle2 part seek()
  ! wall 11.761675 comb 10.50 user 8.24 sys 2.26 (best of 3)
  ! bundle2 part read(8k)
  ! wall 9.116163 comb 9.11 user 8.24 sys 0.87 (best of 3)
  ! bundle2 part read(16k)
  ! wall 8.984362 comb 8.97 user 8.11 sys 0.86 (best of 3)
  ! bundle2 part read(32k)
  ! wall 8.758364 comb 8.74 user 7.86 sys 0.88 (best of 3)
  ! bundle2 part read(128k)
  ! wall 8.749040 comb 8.73 user 7.83 sys 0.90 (best of 3)
  
  We already see some interesting data. Notably that bundle2 has
  significant overhead compared to bundle1. This matters for e.g. stream
  clone bundles, which can be applied at >1Gbps.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1385

AFFECTED FILES
  contrib/perf.py
  tests/test-contrib-perf.t

CHANGE DETAILS

diff --git a/tests/test-contrib-perf.t b/tests/test-contrib-perf.t
--- a/tests/test-contrib-perf.t
+++ b/tests/test-contrib-perf.t
@@ -55,6 +55,8 @@
  benchmark parsing bookmarks from disk to memory
perfbranchmap
  benchmark the update of a branchmap
+   perfbundleread
+ Benchmark reading of bundle files.
perfcca   (no help text available)
perfchangegroupchangelog
  Benchmark producing a changelog group for a changegroup.
@@ -173,3 +175,7 @@
   $ (testrepohg files -r 1.2 glob:mercurial/*.c glob:mercurial/*.py;
   >  testrepohg files -r tip glob:mercurial/*.c glob:mercurial/*.py) |
   > "$TESTDIR"/check-perf-code.py contrib/perf.py
+  contrib/perf.py:498:
+   > from mercurial import (
+   import newer module separately in try clause for early Mercurial
+  [1]
diff --git a/contrib/perf.py b/contrib/perf.py
--- a/contrib/perf.py
+++ b/contrib/perf.py
@@ -488,6 +488,117 @@
 timer(d)
 fm.end()
 
+@command('perfbundleread', formatteropts, 'BUNDLE')
+def perfbundleread(ui, repo, bundlepath, **opts):
+"""Benchmark reading of bundle files.
+
+This command is meant to isolate the I/O part of bundle reading as
+much as possible.
+"""
+from mercurial import (
+bundle2,
+exchange,
+streamclone,
+)
+
+def makebench(fn):
+def run():
+with open(bundlepath, 'rb') as fh:
+bundle = exchange.readbundle(ui, fh, bundlepath)
+fn(bundle)
+
+return run
+
+def makereadnbytes(size):
+def run():
+with open(bundlepath, 'rb') as fh:
+bundle = exchange.readbundle(ui, fh, bundlepath)
+while bundle.read(size):
+pass
+
+return run
+
+def makestdioread(size):
+def run():
+with open(bundlepath, 'rb') as fh:
+while fh.read(size):
+pass
+
+return run
+
+# bundle1
+
+def deltaiter(bundle):
+for delta in bundle.deltaiter():
+pass
+
+def iterchunks(bundle):
+for chunk in bundle.getchunks():
+pass
+
+# bundle2
+
+  

D1388: bundle2: implement consume() API on unbundlepart

[indygreg (Gregory Szorc)]

indygreg created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  We want bundle parts to not be seekable by default. That means
  eliminating the generic seek() method.
  
  A common pattern in bundle2.py is to seek to the end of the part
  data. This is mainly used by the part iteration code to ensure
  the underlying stream is advanced to the next bundle part.
  
  In this commit, we establish a dedicated API for consuming a
  bundle2 part data. We switch users of seek() to it.
  
  The old implementation of seek(0, os.SEEK_END) would effectively
  call self.read(). The new implementation calls self.read(32768)
  in a loop. The old implementation would therefore assemble a
  buffer to hold all remaining data being seeked over. For seeking
  over large bundle parts, this would involve a large allocation and
  a lot of overhead to collect intermediate data! This overhead can
  be seen in the results for `hg perfbundleread`:
  
  ! bundle2 iterparts()
  ! wall 10.891305 comb 10.82 user 7.99 sys 2.83 (best of 3)
  ! wall 8.070791 comb 8.06 user 7.18 sys 0.88 (best of 3)
  ! bundle2 part seek()
  ! wall 12.991478 comb 10.39 user 7.72 sys 2.67 (best of 3)
  ! wall 10.370142 comb 10.35 user 7.43 sys 2.92 (best of 3)
  
  Of course, skipping over large payload data isn't likely very common.
  So I doubt the performance wins will be observed in the wild.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1388

AFFECTED FILES
  mercurial/bundle2.py

CHANGE DETAILS

diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
--- a/mercurial/bundle2.py
+++ b/mercurial/bundle2.py
@@ -363,7 +363,7 @@
 self.count = count
 self.current = p
 yield p
-p.seek(0, os.SEEK_END)
+p.consume()
 self.current = None
 self.iterator = func()
 return self.iterator
@@ -385,11 +385,11 @@
 try:
 if self.current:
 # consume the part content to not corrupt the stream.
-self.current.seek(0, os.SEEK_END)
+self.current.consume()
 
 for part in self.iterator:
 # consume the bundle content
-part.seek(0, os.SEEK_END)
+part.consume()
 except Exception:
 seekerror = True
 
@@ -856,10 +856,11 @@
 while headerblock is not None:
 part = seekableunbundlepart(self.ui, headerblock, self._fp)
 yield part
-# Seek to the end of the part to force it's consumption so the next
-# part can be read. But then seek back to the beginning so the
-# code consuming this generator has a part that starts at 0.
-part.seek(0, os.SEEK_END)
+# Ensure part is fully consumed so we can start reading the next
+# part.
+part.consume()
+# But then seek back to the beginning so the code consuming this
+# generator has a part that starts at 0.
 part.seek(0, os.SEEK_SET)
 headerblock = self._readpartheader()
 indebug(self.ui, 'end of bundle2 stream')
@@ -1165,7 +1166,7 @@
 raise
 finally:
 if not hardabort:
-part.seek(0, os.SEEK_END)
+part.consume()
 self.ui.debug('bundle2-input-stream-interrupt:'
   ' closing out of band context\n')
 
@@ -1300,6 +1301,20 @@
 """Generator of decoded chunks in the payload."""
 return decodepayloadchunks(self.ui, self._fp)
 
+def consume(self):
+"""Read the part payload until completion.
+
+By consuming the part data, the underlying stream read offset will
+be advanced to the next part (or end of stream).
+"""
+if self.consumed:
+return
+
+chunk = self.read(32768)
+while chunk:
+self._pos += len(chunk)
+chunk = self.read(32768)
+
 def read(self, size=None):
 """read payload data"""
 if not self._initialized:



To: indygreg, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1387: bundle2: implement generic part payload decoder

[indygreg (Gregory Szorc)]

indygreg created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  The previous commit extracted _payloadchunks() to a new derived class.
  There was still a reference to this method in unbundlepart, making
  unbundlepart unusable on its own.
  
  This commit implements a generic version of a bundle2 part payload
  decoder, without offset tracking. seekableunbundlepart._payloadchunks()
  has been refactored to consume it, adding offset tracking like before.
  We also implement unbundlepart._payloadchunks(), which is a thin
  wrapper for it. Since we never instantiate unbundlepart directly,
  this new method is not used. This will be changed in subsequent
  commits.
  
  The new implementation also inlines some simple code from unpackermixin
  and adds some local variable to prevent extra function calls and
  attribute lookups. `hg perfbundleread` on an uncompressed Firefox
  bundle seems to show a minor win:
  
  ! bundle2 iterparts()
  ! wall 12.593258 comb 12.25 user 8.87 sys 3.38 (best of 3)
  ! wall 10.891305 comb 10.82 user 7.99 sys 2.83 (best of 3)
  ! bundle2 part seek()
  ! wall 13.173163 comb 11.10 user 8.39 sys 2.71 (best of 3)
  ! wall 12.991478 comb 10.39 user 7.72 sys 2.67 (best of 3)
  ! bundle2 part read(8k)
  ! wall 9.483612 comb 9.48 user 8.42 sys 1.06 (best of 3)
  ! wall 8.599892 comb 8.58 user 7.72 sys 0.86 (best of 3)
  ! bundle2 part read(16k)
  ! wall 9.159815 comb 9.15 user 8.22 sys 0.93 (best of 3)
  ! wall 8.265361 comb 8.25 user 7.36 sys 0.89 (best of 3)
  ! bundle2 part read(32k)
  ! wall 9.141308 comb 9.13 user 8.22 sys 0.91 (best of 3)
  ! wall 8.290308 comb 8.28 user 7.33 sys 0.95 (best of 3)
  ! bundle2 part read(128k)
  ! wall 8.880587 comb 8.85 user 7.96 sys 0.89 (best of 3)
  ! wall 8.204900 comb 8.15 user 7.21 sys 0.94 (best of 3)
  
  Function call overhead in Python strikes again!
  
  Of course, bundle2 decoding CPU overhead is likely small compared to
  decompression and changegroup application. But every little bit helps.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1387

AFFECTED FILES
  mercurial/bundle2.py

CHANGE DETAILS

diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
--- a/mercurial/bundle2.py
+++ b/mercurial/bundle2.py
@@ -1187,6 +1187,32 @@
 def gettransaction(self):
 raise TransactionUnavailable('no repo access from stream interruption')
 
+def decodepayloadchunks(ui, fh):
+"""Reads bundle2 part payload data into chunks.
+
+Part payload data consists of framed chunks. This function takes
+a file handle and emits those chunks.
+"""
+headersize = struct.calcsize(_fpayloadsize)
+readexactly = changegroup.readexactly
+
+chunksize = _unpack(_fpayloadsize, readexactly(fh, headersize))[0]
+indebug(ui, 'payload chunk size: %i' % chunksize)
+
+while chunksize:
+if chunksize >= 0:
+yield readexactly(fh, chunksize)
+elif chunksize == flaginterrupt:
+# Interrupt "signal" detected. The regular stream is interrupted
+# and a bundle2 part follows. Consume it.
+interrupthandler(ui, fh)()
+else:
+raise error.BundleValueError(
+'negative payload chunk size: %s' % chunksize)
+
+chunksize = _unpack(_fpayloadsize, readexactly(fh, headersize))[0]
+indebug(ui, 'payload chunk size: %i' % chunksize)
+
 class unbundlepart(unpackermixin):
 """a bundle part read from a bundle"""
 
@@ -1270,6 +1296,10 @@
 # we read the data, tell it
 self._initialized = True
 
+def _payloadchunks(self):
+"""Generator of decoded chunks in the payload."""
+return decodepayloadchunks(self.ui, self._fp)
+
 def read(self, size=None):
 """read payload data"""
 if not self._initialized:
@@ -1320,25 +1350,14 @@
 self._seekfp(self._chunkindex[chunknum][1])
 
 pos = self._chunkindex[chunknum][0]
-payloadsize = self._unpack(_fpayloadsize)[0]
-indebug(self.ui, 'payload chunk size: %i' % payloadsize)
-while payloadsize:
-if payloadsize == flaginterrupt:
-# interruption detection, the handler will now read a
-# single part and process it.
-interrupthandler(self.ui, self._fp)()
-elif payloadsize < 0:
-msg = 'negative payload chunk size: %i' %  payloadsize
-raise error.BundleValueError(msg)
-else:
-result = self._readexact(payloadsize)
-chunknum += 1
-pos += payloadsize
-if chunknum == len(self._chunkindex):
-self._chunkindex.append((pos, self._tellfp()))
-yield result
-payloadsize

D1383: bundlerepo: rename "bundle" arguments to "cgunpacker"

[indygreg (Gregory Szorc)]

indygreg created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  "bundle" was appropriate for the bundle1 days where a bundle
  was a changegroup. In a bundle2 world, changegroup readers
  are referred to as "changegroup unpackers."

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1383

AFFECTED FILES
  mercurial/bundlerepo.py

CHANGE DETAILS

diff --git a/mercurial/bundlerepo.py b/mercurial/bundlerepo.py
--- a/mercurial/bundlerepo.py
+++ b/mercurial/bundlerepo.py
@@ -42,7 +42,7 @@
 )
 
 class bundlerevlog(revlog.revlog):
-def __init__(self, opener, indexfile, bundle, linkmapper):
+def __init__(self, opener, indexfile, cgunpacker, linkmapper):
 # How it works:
 # To retrieve a revision, we need to know the offset of the revision in
 # the bundle (an unbundle object). We store this offset in the index
@@ -52,15 +52,15 @@
 # check revision against repotiprev.
 opener = vfsmod.readonlyvfs(opener)
 revlog.revlog.__init__(self, opener, indexfile)
-self.bundle = bundle
+self.bundle = cgunpacker
 n = len(self)
 self.repotiprev = n - 1
 self.bundlerevs = set() # used by 'bundle()' revset expression
-for deltadata in bundle.deltaiter():
+for deltadata in cgunpacker.deltaiter():
 node, p1, p2, cs, deltabase, delta, flags = deltadata
 
 size = len(delta)
-start = bundle.tell() - size
+start = cgunpacker.tell() - size
 
 link = linkmapper(cs)
 if node in self.nodemap:
@@ -165,10 +165,10 @@
 raise NotImplementedError
 
 class bundlechangelog(bundlerevlog, changelog.changelog):
-def __init__(self, opener, bundle):
+def __init__(self, opener, cgunpacker):
 changelog.changelog.__init__(self, opener)
 linkmapper = lambda x: x
-bundlerevlog.__init__(self, opener, self.indexfile, bundle,
+bundlerevlog.__init__(self, opener, self.indexfile, cgunpacker,
   linkmapper)
 
 def baserevision(self, nodeorrev):
@@ -186,9 +186,10 @@
 self.filteredrevs = oldfilter
 
 class bundlemanifest(bundlerevlog, manifest.manifestrevlog):
-def __init__(self, opener, bundle, linkmapper, dirlogstarts=None, dir=''):
+def __init__(self, opener, cgunpacker, linkmapper, dirlogstarts=None,
+ dir=''):
 manifest.manifestrevlog.__init__(self, opener, dir=dir)
-bundlerevlog.__init__(self, opener, self.indexfile, bundle,
+bundlerevlog.__init__(self, opener, self.indexfile, cgunpacker,
   linkmapper)
 if dirlogstarts is None:
 dirlogstarts = {}
@@ -217,9 +218,9 @@
 return super(bundlemanifest, self).dirlog(d)
 
 class bundlefilelog(bundlerevlog, filelog.filelog):
-def __init__(self, opener, path, bundle, linkmapper):
+def __init__(self, opener, path, cgunpacker, linkmapper):
 filelog.filelog.__init__(self, opener, path)
-bundlerevlog.__init__(self, opener, self.indexfile, bundle,
+bundlerevlog.__init__(self, opener, self.indexfile, cgunpacker,
   linkmapper)
 
 def baserevision(self, nodeorrev):
@@ -246,12 +247,12 @@
 self.invalidate()
 self.dirty = True
 
-def _getfilestarts(bundle):
 bundlefilespos = {}
-for chunkdata in iter(bundle.filelogheader, {}):
+def _getfilestarts(cgunpacker):
+for chunkdata in iter(cgunpacker.filelogheader, {}):
 fname = chunkdata['filename']
 bundlefilespos[fname] = bundle.tell()
-for chunk in iter(lambda: bundle.deltachunk(None), {}):
+for chunk in iter(lambda: cgunpacker.deltachunk(None), {}):
 pass
 return bundlefilespos
 



To: indygreg, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1384: bundlerepo: rename "bundlefilespos" variable and attribute

[indygreg (Gregory Szorc)]

indygreg created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  Strictly speaking, this variable tracks offsets within the
  changegroup, not the bundle.
  
  While we're here, mark a class attribute as private because
  it is.
  
  .. api::
  
Rename bundlerepo.bundlerepository.bundlefilespos to
_cgfilespos.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1384

AFFECTED FILES
  mercurial/bundlerepo.py

CHANGE DETAILS

diff --git a/mercurial/bundlerepo.py b/mercurial/bundlerepo.py
--- a/mercurial/bundlerepo.py
+++ b/mercurial/bundlerepo.py
@@ -247,14 +247,14 @@
 self.invalidate()
 self.dirty = True
 
-bundlefilespos = {}
 def _getfilestarts(cgunpacker):
+filespos = {}
 for chunkdata in iter(cgunpacker.filelogheader, {}):
 fname = chunkdata['filename']
-bundlefilespos[fname] = bundle.tell()
+filespos[fname] = cgunpacker.tell()
 for chunk in iter(lambda: cgunpacker.deltachunk(None), {}):
 pass
-return bundlefilespos
+return filespos
 
 class bundlerepository(localrepo.localrepository):
 """A repository instance that is a union of a local repo and a bundle.
@@ -312,8 +312,8 @@
 raise error.Abort(_('bundle type %s cannot be read') %
   type(bundle))
 
-# dict with the mapping 'filename' -> position in the bundle
-self.bundlefilespos = {}
+# dict with the mapping 'filename' -> position in the changegroup.
+self._cgfilespos = {}
 
 self.firstnewrev = self.changelog.repotiprev + 1
 phases.retractboundary(self, None, phases.draft,
@@ -403,12 +403,12 @@
 return self._url
 
 def file(self, f):
-if not self.bundlefilespos:
+if not self._cgfilespos:
 self._cgunpacker.seek(self.filestart)
-self.bundlefilespos = _getfilestarts(self._cgunpacker)
+self._cgfilespos = _getfilestarts(self._cgunpacker)
 
-if f in self.bundlefilespos:
-self._cgunpacker.seek(self.bundlefilespos[f])
+if f in self._cgfilespos:
+self._cgunpacker.seek(self._cgfilespos[f])
 linkmapper = self.unfiltered().changelog.rev
 return bundlefilelog(self.svfs, f, self._cgunpacker, linkmapper)
 else:



To: indygreg, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1382: bundlerepo: use early return

[indygreg (Gregory Szorc)]

indygreg created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  I like avoiding patterns that lead to the pyramid of doom.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1382

AFFECTED FILES
  mercurial/bundlerepo.py

CHANGE DETAILS

diff --git a/mercurial/bundlerepo.py b/mercurial/bundlerepo.py
--- a/mercurial/bundlerepo.py
+++ b/mercurial/bundlerepo.py
@@ -319,19 +319,19 @@
[ctx.node() for ctx in self[self.firstnewrev:]])
 
 def _handlebundle2part(self, bundle, part):
-if part.type == 'changegroup':
-cgstream = part
-version = part.params.get('version', '01')
-legalcgvers = changegroup.supportedincomingversions(self)
-if version not in legalcgvers:
-msg = _('Unsupported changegroup version: %s')
-raise error.Abort(msg % version)
-if bundle.compressed():
-cgstream = self._writetempbundle(part.read,
- ".cg%sun" % version)
+if part.type != 'changegroup':
+return
 
-self._cgunpacker = changegroup.getunbundler(version, cgstream,
- 'UN')
+cgstream = part
+version = part.params.get('version', '01')
+legalcgvers = changegroup.supportedincomingversions(self)
+if version not in legalcgvers:
+msg = _('Unsupported changegroup version: %s')
+raise error.Abort(msg % version)
+if bundle.compressed():
+cgstream = self._writetempbundle(part.read, '.cg%sun' % version)
+
+self._cgunpacker = changegroup.getunbundler(version, cgstream, 'UN')
 
 def _writetempbundle(self, readfn, suffix, header=''):
 """Write a temporary file to disk



To: indygreg, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: Upstreaming Facebook extensions

[Matt Harbison]

On Mon, 13 Nov 2017 22:12:31 -0500, Jun Wu  wrote:


Excerpts from Matt Harbison's message of 2017-11-13 21:50:29 -0500:

For LFS, it has been used in a repo synced from p4 for about half a year.
It's mostly good except for lack of features (ex. support the Git-LFS SSH
authentication, support gc, etc.). It was actually written with some  
extra
care of upstream-friendliness. For example, I put remotefilelog  
integration

in remotefilelog instead of LFS intentionally.

Help is definitely welcomed!


OK, good to know.

So is upstreaming lfs in a single patch and marking it experimental a  
reasonable next step, or does this need to incubate in hg-experimental a  
bit more?  I didn't get it working outside of the tests (it said something  
about not having a common changegroup version), but it looks like there's  
some low hanging fruit like registering the config options, and tracking  
down some warning in url.py about an "unquoted realm" or similar.


I know BC doesn't apply to experimental things, but realistically, I  
assume things won't need to change other that maybe config stuff to add  
features?  I wouldn't mind using this sooner rather than later if the  
files can always be retrieved.

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: Upstreaming Facebook extensions

[Jun Wu]

Excerpts from Matt Harbison's message of 2017-11-13 21:50:29 -0500:
> I see some things are being upstreamed out of the hg-experimental repo on  
> Bitbucket, but I haven't seen mention of how these are being selected.   
> I'm interested in the absorb and lfs extensions.  The former works great  
> on Linux in my limited use, but the linelog module dependency doesn't  
> build on Windows.  I only found the latter today, so I haven't used it  
> much other than to run the tests.
> 
> Are there any (other) showstoppers, and/or design TBDs to getting these  
> bundled?  I should be able to throw some time at this if it helps  
> (especially the LFS extension, as there's a need for it at work).

I think difficulty of upstreaming cleanly is the main deciding factor. So
"commitextras" got upstreamed first, then "uncommit/unamend/directaccess".

For absorb, it needs to migrate away from using Cython, and has a pure
implementation. Unfortunately I cannot prioritize this work for now.

For LFS, it has been used in a repo synced from p4 for about half a year.
It's mostly good except for lack of features (ex. support the Git-LFS SSH
authentication, support gc, etc.). It was actually written with some extra
care of upstream-friendliness. For example, I put remotefilelog integration
in remotefilelog instead of LFS intentionally.

Help is definitely welcomed!
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Upstreaming Facebook extensions

[Matt Harbison]

I see some things are being upstreamed out of the hg-experimental repo on  
Bitbucket, but I haven't seen mention of how these are being selected.   
I'm interested in the absorb and lfs extensions.  The former works great  
on Linux in my limited use, but the linelog module dependency doesn't  
build on Windows.  I only found the latter today, so I haven't used it  
much other than to run the tests.


Are there any (other) showstoppers, and/or design TBDs to getting these  
bundled?  I should be able to throw some time at this if it helps  
(especially the LFS extension, as there's a need for it at work).

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1381: crecord: fix revert -r '.^' crash caused by 3649c3f2cd

[quark (Jun Wu)]

quark created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  https://phab.mercurial-scm.org/rHG3649c3f2cd90c8aec395ca8af5adae33defff12c 
(revert: do not reverse hunks in interactive when REV is not
  parent (issue5096)) changed the record "operation" for the text version but
  missed the curses version. Without this patch, running
  `hg revert -r '.^' --config ui.interface=curses` would crash with:
  
ProgrammingError: unexpected operation: apply

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1381

AFFECTED FILES
  mercurial/crecord.py

CHANGE DETAILS

diff --git a/mercurial/crecord.py b/mercurial/crecord.py
--- a/mercurial/crecord.py
+++ b/mercurial/crecord.py
@@ -555,6 +555,7 @@
 return chunkselector.opts
 
 _headermessages = { # {operation: text}
+'apply': _('Select hunks to apply'),
 'revert': _('Select hunks to revert'),
 'discard': _('Select hunks to discard'),
 None: _('Select hunks to record'),



To: quark, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: Desired use case for obsmarkers / visibility

[Jun Wu]

Excerpts from Augie Fackler's message of 2017-11-10 17:58:34 -0500:
> Here's a proposal I ran by Martin probably a year ago, and that he
> didn't immediately hate:
> 
> Prunes are different from regular obsolete markers. By default, you
> don't push prunes, and pulled prunes are not applied, but merely
> warned about. Push and pull could both grow a new flag
> (`--exchange-death` is my intentionally-bad name because I haven't
> come up with a good one).

I think this is reasonable.

Practically, instead of having a separate layer of visibility (which could
take some time to build right). We could implement it as an option of the
exchange protocol so the problematic marker does not get sent to the client
at all. Implementation-wise, this is to stop following the "parent" field
when calculating the obsmarker chain. It sounds easier to implement.
 
> Anecdotally, nearly all of the prune markers in my repo of hg are
> mine, and probably haven't ever been exchanged. In the entire history
> of hg, we've exchanged 21062 prune markers, but only 361 prune markers
> (minus any that are mine, I'm too lazy to make ane extra clone to get
> that info), out of that 21k are actually prunes of something I've got
> locally. I don't know how many of those locally-pruned revisions then
> later got resurrected and finally landed in the repo. That suggests to
> me that we could actually do pretty well by only exchanging prunes
> when it's actually relevant to the destination.

Not directly related to this thread. I'd like note that the above situation
is hg-committed workflow specific. There are users at fb who have tons of
markers which are all their own that even `unfi.revs('draft')` is not
considered cheap.  Shall we start planning for a new visibility layer, I'd
like to see it reduces time complexity calculating phases by filtering out
invisible roots (ex. it is at a lower layer than phases).

> Thoughts?
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1336: remove: print message for each file in verbose mode only while using `-A`

[mharbison72 (Matt Harbison)]

mharbison72 added a comment.


  If this can't be taken as-is, what about something in tweakdefaults to hide 
this by default?  I just assumed that --quiet already worked, and the problem 
was that this throws away the other more useful warnings in the non --after 
case.  (Or people don't remember to use --quiet before running it.)
  
  FWIW, I watched someone run `hg rm -A` on a repo with 67K files on Friday, 
and that made me much more sympathetic.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1336

To: pavanpc, #hg-reviewers, lothiraldan
Cc: durin42, lothiraldan, mitrandir, mharbison72, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: linkrev corruption (was Re: Corrupted repo? Full of unexpected changesets)

[Jun Wu]

Excerpts from Martin von Zweigbergk's message of 2017-10-29 22:46:25 +:
> I think Jun (on CC) once wrote some extension that would cache linkrevs. I
> think it could be configured not to care about the linkrevs in the revlog
> at all. If that's correct, it seems like one could run a server with that
> config and it would produce a correct clone.
> 
> Jun, do I remember the right?

Sorry for the slow response. linkrevcache [1] recalculates linkrevs from
scratch and in theory works in this linkrev corruption case regardless of
what config options are used.

The extension was initially built for server-side usecase. So its interface 
is a bit low-level and it is not "strip-friendly":

  hg debugbuildlinkrevcache --end REV

If you know the max number of corruption rev, pass that to the command and
it would probably "fix" your repo. Note that if you strip rev < REV in the
repo by running strip, amend, rebase, histedit without obsstore enabled, or
rebase --abort, histedit --abort with obsstore enabled, the cache needs to
be rebuilt manually.

[1]: 
https://bitbucket.org/facebook/hg-experimental/src/39a1cf89e6a24441ac7d68615b206d4f9ee63257/hgext3rd/linkrevcache.py
> 
> On Sun, Oct 29, 2017, 15:37 Augie Fackler  wrote:
> 
> >
> > > On Oct 28, 2017, at 9:23 PM, Kim Alvefur  wrote:
> > >
> > > Hi,
> > >
> > > I've got a repo that, when running `hg verify`, produces a number of
> > > messages like:
> > >
> > >> rev 42 points to unexpected changeset 6074
> > >
> > > A tarball of the repo can be downloaded from
> > >  > >
> > > It's a clone of  > >
> > > I'm not entirely sure what I have done to it. `hg debugupgraderepo` may
> > > have been involved, as well as hardlinking to other repos.
> > >
> > > Oddly enough, the problem appears to follow along when it is cloned.
> >
> > I’ve known about problems like this for a while - in fact, for a long time
> > the way to fix them was to push them to code.google.com and then clone
> > back out, as the nature of Google’s custom hg server meant it scrubbed
> > these problems away.
> >
> > I suppose for an “easy” fix we could add some sort of debug config setting
> > that would re-guess linkrevs on the way in - it’d be *really* slow on large
> > repos, but would work.
> >
> > > I don't consider recovery of this particulary important, I'm more
> > > interested in knowing how this could have happened and, if this can be
> > > traced down no a bug in hg, seeing that fixed.
> >
> > It’d be interesting to know if you can reproduce this with `hg
> > debugupgraderepo`: if you can, please file a bug.
> >
> > This corruption is partially harmless - it can result in pushes omitting
> > file contents in such a way that the server has a broken history (which is
> > how I discovered the problem.) We should probably figure out a way to fix
> > it, especially now that GCode is gone.
> >
> > AF
> >
> > >
> > > --
> > > Regards,
> > > Kim "Zash" Alvefur
> > > ___
> > > Mercurial-devel mailing list
> > > Mercurial-devel@mercurial-scm.org
> > > https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
> >
> > ___
> > Mercurial-devel mailing list
> > Mercurial-devel@mercurial-scm.org
> > https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
> >
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1354: dirstate: change all writes to dirstatemap._map to go through one method

[durin42 (Augie Fackler)]

durin42 added a comment.


  In https://phab.mercurial-scm.org/D1354#22991, @mbolin wrote:
  
  > @durin42 The issue is that `dirstatemap` is doing so much "stuff" beyond 
just storage that Eden prefers to subclass it rather than copy/paste all of the 
business logic.
  
  
  That's an argument for further splitting dirstatemap. That said, does 
https://phab.mercurial-scm.org/D1347 resolve the specific concern that 
motivated this change?

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1354

To: mbolin, mbthomas, durham, #hg-reviewers
Cc: durin42, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1354: dirstate: change all writes to dirstatemap._map to go through one method

[mbolin (Michael Bolin)]

mbolin added a comment.


  @durin42 The issue is that `dirstatemap` is doing so much "stuff" beyond just 
storage that Eden prefers to subclass it rather than copy/paste all of the 
business logic.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1354

To: mbolin, mbthomas, durham, #hg-reviewers
Cc: durin42, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 02 of 15 V2] bookmark: add methods to binary encode and decode bookmark values

[Augie Fackler]

> On Nov 13, 2017, at 05:16, Boris Feld  wrote:
> 
> On Fri, 2017-11-10 at 17:35 -0500, Augie Fackler wrote:
>> (+indygreg, who also is a formats enthusiast)
>> 
>> On Thu, Nov 02, 2017 at 02:17:59PM +0100, Boris Feld wrote:
>>> # HG changeset patch
>>> # User Boris Feld 
>>> # Date 1508072395 -7200
>>> #  Sun Oct 15 14:59:55 2017 +0200
>>> # Node ID 4d0c6772a81aa1e2b25f32f944563db1f33fd327
>>> # Parent  8c9a9eecdcd61401a1604a08a5272f7dabd4b912
>>> # EXP-Topic b2.bookmarks
>>> # Available At https://bitbucket.org/octobus/mercurial-devel/
>>> #  hg pull https://bitbucket.org/octobus/mercurial-deve
>>> l/ -r 4d0c6772a81a
>>> bookmark: add methods to binary encode and decode bookmark values
>>> 
>>> Coming new bundle2 parts related to bookmark will use a binary
>>> encoding. It
>>> encodes a series of '(bookmark, node)' pairs. Bookmark name has a
>>> high enough
>>> size limit to not be affected by issue5165. (64K length, we are
>>> well covered)
>> 
>> I'm not thrilled here. Could we do some sort of varint encoding,
>> which
>> would be generally useful going forward for a variety of things,
>> rather than just shrugging and setting a limit which we (today) deem
>> absurdly large?
> 
> We are not sure what the value of that would be. Bundle2 makes it very
> easy to move to a newer encoding version if needed.

I mean, not really. We have to add a b2cap, and then we've got to maintain 
multiple versions forever. It's kind of a pain. A varint encoding wouldn't be 
hard, and will probably take O(an hour) to code up. I'd entertain that much 
more eagerly than just dropping a uint64 in here and asserting it should be big 
enough forever.

> We are already
> doing so for changegroup and obsmarkers.
> 
> The biggest bookmark we have seen was about 800 characters, that should
> leave us some room for the future without blocking us.

That you know about. I seem to remember violent arguments that 255 bytes should 
be more than enough. I agree a 64k bookmark sounds totally insane, but what if 
I wanted to put base64'd test result metadata in a bookmark?

> 
>> 
>> I agree that practically speaking, nobody should have a 64k bookmark,
>> but we can do better. We also know that \0 shouldn't appear in a
>> bookmark name, so we could just make the format be
>> null-terminated. I'd much rather we get in the practice of
>> deliberately crafting formats that are maximally flexible and
>> reusable.
> 
> That would be significantly harder to parse since we would need to
> search for the '\0' in the stream. We do not think it brings much value
> either since any new data we sneak into the last field would have to be
> understood by the receiver. If we need to negotiate for such
> capabilities, this is equivalent to rolling out a new encoding.

Then let's explore the varint option. Here's a rough sketch of a reader:

import io
src = io.BytesIO(b'\xff\x01')
cur = ord(src.read(1))
i = cur & 0x7f
while cur & 0x80:
i = i << 7
cur = ord(src.read(1))
i += cur & 0x7f
assert 0b111001 == i

Gives you 7 bits of value per byte transmitted, which means that we'll get most 
realistic bookmark lengths in two bytes of "here's how much string" (probably 
one byte for the vast majority of ascii cases!)

I'd much rather we just build up the infrastructure for stuff like this _now_ 
and start using it, rather than continue to punt and shove massive uint64 
fields in and then pray it'll be enough for all eternity. It does mean we can't 
use struct anymore, but that seems like a comparatively small price to pay for 
being able to trivially make this future proof.

(As before, if other reviewers think I'm being too hard on this, please speak 
up.)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1377: bundlerepo: make bundle and bundlefile attributes private

[indygreg (Gregory Szorc)]

This revision was automatically updated to reflect the committed changes.
Closed by commit rHGdf2a676a2e9e: bundlerepo: make bundle and bundlefile 
attributes private (authored by indygreg, committed by ).

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1377?vs=3432&id=3454

REVISION DETAIL
  https://phab.mercurial-scm.org/D1377

AFFECTED FILES
  mercurial/bundlerepo.py

CHANGE DETAILS

diff --git a/mercurial/bundlerepo.py b/mercurial/bundlerepo.py
--- a/mercurial/bundlerepo.py
+++ b/mercurial/bundlerepo.py
@@ -281,11 +281,11 @@
 
 self.tempfile = None
 f = util.posixfile(bundlepath, "rb")
-self.bundlefile = self.bundle = exchange.readbundle(ui, f, bundlepath)
+self._bundlefile = self._bundle = exchange.readbundle(ui, f, 
bundlepath)
 
-if isinstance(self.bundle, bundle2.unbundle20):
+if isinstance(self._bundle, bundle2.unbundle20):
 hadchangegroup = False
-for part in self.bundle.iterparts():
+for part in self._bundle.iterparts():
 if part.type == 'changegroup':
 if hadchangegroup:
 raise NotImplementedError("can't process "
@@ -296,16 +296,15 @@
 
 if not hadchangegroup:
 raise error.Abort(_("No changegroups found"))
-elif isinstance(self.bundle, changegroup.cg1unpacker):
-if self.bundle.compressed():
-f = self._writetempbundle(self.bundle.read, '.hg10un',
+elif isinstance(self._bundle, changegroup.cg1unpacker):
+if self._bundle.compressed():
+f = self._writetempbundle(self._bundle.read, '.hg10un',
   header='HG10UN')
-self.bundlefile = self.bundle = exchange.readbundle(ui, f,
-bundlepath,
-self.vfs)
+self._bundlefile = self._bundle = exchange.readbundle(
+ui, f, bundlepath, self.vfs)
 else:
 raise error.Abort(_('bundle type %s cannot be read') %
-  type(self.bundle))
+  type(self._bundle))
 
 # dict with the mapping 'filename' -> position in the bundle
 self.bundlefilespos = {}
@@ -322,11 +321,11 @@
 if version not in legalcgvers:
 msg = _('Unsupported changegroup version: %s')
 raise error.Abort(msg % version)
-if self.bundle.compressed():
+if self._bundle.compressed():
 cgstream = self._writetempbundle(part.read,
  ".cg%sun" % version)
 
-self.bundle = changegroup.getunbundler(version, cgstream, 'UN')
+self._bundle = changegroup.getunbundler(version, cgstream, 'UN')
 
 def _writetempbundle(self, readfn, suffix, header=''):
 """Write a temporary file to disk
@@ -352,28 +351,28 @@
 @localrepo.unfilteredpropertycache
 def changelog(self):
 # consume the header if it exists
-self.bundle.changelogheader()
-c = bundlechangelog(self.svfs, self.bundle)
-self.manstart = self.bundle.tell()
+self._bundle.changelogheader()
+c = bundlechangelog(self.svfs, self._bundle)
+self.manstart = self._bundle.tell()
 return c
 
 def _constructmanifest(self):
-self.bundle.seek(self.manstart)
+self._bundle.seek(self.manstart)
 # consume the header if it exists
-self.bundle.manifestheader()
+self._bundle.manifestheader()
 linkmapper = self.unfiltered().changelog.rev
-m = bundlemanifest(self.svfs, self.bundle, linkmapper)
-self.filestart = self.bundle.tell()
+m = bundlemanifest(self.svfs, self._bundle, linkmapper)
+self.filestart = self._bundle.tell()
 return m
 
 def _consumemanifest(self):
 """Consumes the manifest portion of the bundle, setting filestart so 
the
 file portion can be read."""
-self.bundle.seek(self.manstart)
-self.bundle.manifestheader()
-for delta in self.bundle.deltaiter():
+self._bundle.seek(self.manstart)
+self._bundle.manifestheader()
+for delta in self._bundle.deltaiter():
 pass
-self.filestart = self.bundle.tell()
+self.filestart = self._bundle.tell()
 
 @localrepo.unfilteredpropertycache
 def manstart(self):
@@ -398,19 +397,19 @@
 
 def file(self, f):
 if not self.bundlefilespos:
-self.bundle.seek(self.filestart)
-self.bundlefilespos = _getfilestarts(self.bundle)
+self._bundle.seek(self.filestart)
+self.bundlefilespos = _getfilestarts(self._bundle)
 
 if f in self.bundlefilespos:
-self.bundle.seek(self.

D1379: bundlerepo: rename _bundle to _cgunpacker

[indygreg (Gregory Szorc)]

This revision was automatically updated to reflect the committed changes.
Closed by commit rHG3f393e4593f2: bundlerepo: rename _bundle to _cgunpacker 
(authored by indygreg, committed by ).

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1379?vs=3434&id=3455

REVISION DETAIL
  https://phab.mercurial-scm.org/D1379

AFFECTED FILES
  mercurial/bundlerepo.py

CHANGE DETAILS

diff --git a/mercurial/bundlerepo.py b/mercurial/bundlerepo.py
--- a/mercurial/bundlerepo.py
+++ b/mercurial/bundlerepo.py
@@ -285,7 +285,7 @@
 
 if isinstance(bundle, bundle2.unbundle20):
 self._bundlefile = bundle
-self._bundle = None
+self._cgunpacker = None
 
 hadchangegroup = False
 for part in bundle.iterparts():
@@ -306,7 +306,7 @@
 bundle = exchange.readbundle(ui, f, bundlepath, self.vfs)
 
 self._bundlefile = bundle
-self._bundle = bundle
+self._cgunpacker = bundle
 else:
 raise error.Abort(_('bundle type %s cannot be read') %
   type(bundle))
@@ -330,7 +330,8 @@
 cgstream = self._writetempbundle(part.read,
  ".cg%sun" % version)
 
-self._bundle = changegroup.getunbundler(version, cgstream, 'UN')
+self._cgunpacker = changegroup.getunbundler(version, cgstream,
+ 'UN')
 
 def _writetempbundle(self, readfn, suffix, header=''):
 """Write a temporary file to disk
@@ -356,28 +357,28 @@
 @localrepo.unfilteredpropertycache
 def changelog(self):
 # consume the header if it exists
-self._bundle.changelogheader()
-c = bundlechangelog(self.svfs, self._bundle)
-self.manstart = self._bundle.tell()
+self._cgunpacker.changelogheader()
+c = bundlechangelog(self.svfs, self._cgunpacker)
+self.manstart = self._cgunpacker.tell()
 return c
 
 def _constructmanifest(self):
-self._bundle.seek(self.manstart)
+self._cgunpacker.seek(self.manstart)
 # consume the header if it exists
-self._bundle.manifestheader()
+self._cgunpacker.manifestheader()
 linkmapper = self.unfiltered().changelog.rev
-m = bundlemanifest(self.svfs, self._bundle, linkmapper)
-self.filestart = self._bundle.tell()
+m = bundlemanifest(self.svfs, self._cgunpacker, linkmapper)
+self.filestart = self._cgunpacker.tell()
 return m
 
 def _consumemanifest(self):
 """Consumes the manifest portion of the bundle, setting filestart so 
the
 file portion can be read."""
-self._bundle.seek(self.manstart)
-self._bundle.manifestheader()
-for delta in self._bundle.deltaiter():
+self._cgunpacker.seek(self.manstart)
+self._cgunpacker.manifestheader()
+for delta in self._cgunpacker.deltaiter():
 pass
-self.filestart = self._bundle.tell()
+self.filestart = self._cgunpacker.tell()
 
 @localrepo.unfilteredpropertycache
 def manstart(self):
@@ -402,13 +403,13 @@
 
 def file(self, f):
 if not self.bundlefilespos:
-self._bundle.seek(self.filestart)
-self.bundlefilespos = _getfilestarts(self._bundle)
+self._cgunpacker.seek(self.filestart)
+self.bundlefilespos = _getfilestarts(self._cgunpacker)
 
 if f in self.bundlefilespos:
-self._bundle.seek(self.bundlefilespos[f])
+self._cgunpacker.seek(self.bundlefilespos[f])
 linkmapper = self.unfiltered().changelog.rev
-return bundlefilelog(self.svfs, f, self._bundle, linkmapper)
+return bundlefilelog(self.svfs, f, self._cgunpacker, linkmapper)
 else:
 return filelog.filelog(self.svfs, f)
 



To: indygreg, #hg-reviewers, dlax, durin42
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1376: bundlerepo: don't assume there are only two bundle classes

[indygreg (Gregory Szorc)]

This revision was automatically updated to reflect the committed changes.
Closed by commit rHG4f04c9207a76: bundlerepo: don't assume there are only 
two bundle classes (authored by indygreg, committed by ).

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1376?vs=3431&id=3453

REVISION DETAIL
  https://phab.mercurial-scm.org/D1376

AFFECTED FILES
  mercurial/bundlerepo.py

CHANGE DETAILS

diff --git a/mercurial/bundlerepo.py b/mercurial/bundlerepo.py
--- a/mercurial/bundlerepo.py
+++ b/mercurial/bundlerepo.py
@@ -296,13 +296,16 @@
 
 if not hadchangegroup:
 raise error.Abort(_("No changegroups found"))
-
-elif self.bundle.compressed():
-f = self._writetempbundle(self.bundle.read, '.hg10un',
-  header='HG10UN')
-self.bundlefile = self.bundle = exchange.readbundle(ui, f,
-bundlepath,
-self.vfs)
+elif isinstance(self.bundle, changegroup.cg1unpacker):
+if self.bundle.compressed():
+f = self._writetempbundle(self.bundle.read, '.hg10un',
+  header='HG10UN')
+self.bundlefile = self.bundle = exchange.readbundle(ui, f,
+bundlepath,
+self.vfs)
+else:
+raise error.Abort(_('bundle type %s cannot be read') %
+  type(self.bundle))
 
 # dict with the mapping 'filename' -> position in the bundle
 self.bundlefilespos = {}



To: indygreg, #hg-reviewers, dlax, durin42
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1378: bundlerepo: assign bundle attributes in bundle type blocks

[indygreg (Gregory Szorc)]

This revision was automatically updated to reflect the committed changes.
Closed by commit rHG495fcff10124: bundlerepo: assign bundle attributes in 
bundle type blocks (authored by indygreg, committed by ).

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1378?vs=3433&id=3456

REVISION DETAIL
  https://phab.mercurial-scm.org/D1378

AFFECTED FILES
  mercurial/bundlerepo.py

CHANGE DETAILS

diff --git a/mercurial/bundlerepo.py b/mercurial/bundlerepo.py
--- a/mercurial/bundlerepo.py
+++ b/mercurial/bundlerepo.py
@@ -281,47 +281,52 @@
 
 self.tempfile = None
 f = util.posixfile(bundlepath, "rb")
-self._bundlefile = self._bundle = exchange.readbundle(ui, f, 
bundlepath)
+bundle = exchange.readbundle(ui, f, bundlepath)
 
-if isinstance(self._bundle, bundle2.unbundle20):
+if isinstance(bundle, bundle2.unbundle20):
+self._bundlefile = bundle
+self._bundle = None
+
 hadchangegroup = False
-for part in self._bundle.iterparts():
+for part in bundle.iterparts():
 if part.type == 'changegroup':
 if hadchangegroup:
 raise NotImplementedError("can't process "
   "multiple changegroups")
 hadchangegroup = True
 
-self._handlebundle2part(part)
+self._handlebundle2part(bundle, part)
 
 if not hadchangegroup:
 raise error.Abort(_("No changegroups found"))
-elif isinstance(self._bundle, changegroup.cg1unpacker):
-if self._bundle.compressed():
-f = self._writetempbundle(self._bundle.read, '.hg10un',
+elif isinstance(bundle, changegroup.cg1unpacker):
+if bundle.compressed():
+f = self._writetempbundle(bundle.read, '.hg10un',
   header='HG10UN')
-self._bundlefile = self._bundle = exchange.readbundle(
-ui, f, bundlepath, self.vfs)
+bundle = exchange.readbundle(ui, f, bundlepath, self.vfs)
+
+self._bundlefile = bundle
+self._bundle = bundle
 else:
 raise error.Abort(_('bundle type %s cannot be read') %
-  type(self._bundle))
+  type(bundle))
 
 # dict with the mapping 'filename' -> position in the bundle
 self.bundlefilespos = {}
 
 self.firstnewrev = self.changelog.repotiprev + 1
 phases.retractboundary(self, None, phases.draft,
[ctx.node() for ctx in self[self.firstnewrev:]])
 
-def _handlebundle2part(self, part):
+def _handlebundle2part(self, bundle, part):
 if part.type == 'changegroup':
 cgstream = part
 version = part.params.get('version', '01')
 legalcgvers = changegroup.supportedincomingversions(self)
 if version not in legalcgvers:
 msg = _('Unsupported changegroup version: %s')
 raise error.Abort(msg % version)
-if self._bundle.compressed():
+if bundle.compressed():
 cgstream = self._writetempbundle(part.read,
  ".cg%sun" % version)
 



To: indygreg, #hg-reviewers, dlax, durin42
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1374: bundlerepo: rename arguments to bundlerepository.__init__

[indygreg (Gregory Szorc)]

This revision was automatically updated to reflect the committed changes.
Closed by commit rHG4696938d40f9: bundlerepo: rename arguments to 
bundlerepository.__init__ (authored by indygreg, committed by ).

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1374?vs=3429&id=3451

REVISION DETAIL
  https://phab.mercurial-scm.org/D1374

AFFECTED FILES
  mercurial/bundlerepo.py

CHANGE DETAILS

diff --git a/mercurial/bundlerepo.py b/mercurial/bundlerepo.py
--- a/mercurial/bundlerepo.py
+++ b/mercurial/bundlerepo.py
@@ -256,24 +256,24 @@
 return bundlefilespos
 
 class bundlerepository(localrepo.localrepository):
-def __init__(self, ui, path, bundlename):
+def __init__(self, ui, repopath, bundlepath):
 self._tempparent = None
 try:
-localrepo.localrepository.__init__(self, ui, path)
+localrepo.localrepository.__init__(self, ui, repopath)
 except error.RepoError:
 self._tempparent = tempfile.mkdtemp()
 localrepo.instance(ui, self._tempparent, 1)
 localrepo.localrepository.__init__(self, ui, self._tempparent)
 self.ui.setconfig('phases', 'publish', False, 'bundlerepo')
 
-if path:
-self._url = 'bundle:' + util.expandpath(path) + '+' + bundlename
+if repopath:
+self._url = 'bundle:' + util.expandpath(repopath) + '+' + 
bundlepath
 else:
-self._url = 'bundle:' + bundlename
+self._url = 'bundle:' + bundlepath
 
 self.tempfile = None
-f = util.posixfile(bundlename, "rb")
-self.bundlefile = self.bundle = exchange.readbundle(ui, f, bundlename)
+f = util.posixfile(bundlepath, "rb")
+self.bundlefile = self.bundle = exchange.readbundle(ui, f, bundlepath)
 
 if isinstance(self.bundle, bundle2.unbundle20):
 hadchangegroup = False
@@ -293,7 +293,7 @@
 f = self._writetempbundle(self.bundle.read, '.hg10un',
   header='HG10UN')
 self.bundlefile = self.bundle = exchange.readbundle(ui, f,
-bundlename,
+bundlepath,
 self.vfs)
 
 # dict with the mapping 'filename' -> position in the bundle



To: indygreg, #hg-reviewers, dlax, durin42
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1375: bundlerepo: add docstring for bundlerepository class

[indygreg (Gregory Szorc)]

This revision was automatically updated to reflect the committed changes.
Closed by commit rHGd2458ba810c5: bundlerepo: add docstring for 
bundlerepository class (authored by indygreg, committed by ).

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1375?vs=3430&id=3452

REVISION DETAIL
  https://phab.mercurial-scm.org/D1375

AFFECTED FILES
  mercurial/bundlerepo.py

CHANGE DETAILS

diff --git a/mercurial/bundlerepo.py b/mercurial/bundlerepo.py
--- a/mercurial/bundlerepo.py
+++ b/mercurial/bundlerepo.py
@@ -256,6 +256,14 @@
 return bundlefilespos
 
 class bundlerepository(localrepo.localrepository):
+"""A repository instance that is a union of a local repo and a bundle.
+
+Instances represent a read-only repository composed of a local repository
+with the contents of a bundle file applied. The repository instance is
+conceptually similar to the state of a repository after an
+``hg unbundle`` operation. However, the contents of the bundle are never
+applied to the actual base repository.
+"""
 def __init__(self, ui, repopath, bundlepath):
 self._tempparent = None
 try:



To: indygreg, #hg-reviewers, dlax, durin42
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1373: bundlerepo: use suffix variable

[indygreg (Gregory Szorc)]

This revision was automatically updated to reflect the committed changes.
Closed by commit rHG80e9b85d96e5: bundlerepo: use suffix variable (authored by 
indygreg, committed by ).

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1373?vs=3428&id=3450

REVISION DETAIL
  https://phab.mercurial-scm.org/D1373

AFFECTED FILES
  mercurial/bundlerepo.py

CHANGE DETAILS

diff --git a/mercurial/bundlerepo.py b/mercurial/bundlerepo.py
--- a/mercurial/bundlerepo.py
+++ b/mercurial/bundlerepo.py
@@ -321,7 +321,7 @@
 """Write a temporary file to disk
 """
 fdtemp, temp = self.vfs.mkstemp(prefix="hg-bundle-",
-suffix=".hg10un")
+suffix=suffix)
 self.tempfile = temp
 
 with os.fdopen(fdtemp, pycompat.sysstr('wb')) as fptemp:



To: indygreg, #hg-reviewers, dlax, durin42
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 12 of 15 V2] bundle2: support a 'records' mode for the 'bookmarks' part

[Augie Fackler]

> On Nov 13, 2017, at 05:24, Boris Feld  wrote:
> 
> On Fri, 2017-11-10 at 17:43 -0500, Augie Fackler wrote:
>> On Thu, Nov 02, 2017 at 02:18:09PM +0100, Boris Feld wrote:
>>> # HG changeset patch
>>> # User Boris Feld 
>>> # Date 1508246776 -7200
>>> #  Tue Oct 17 15:26:16 2017 +0200
>>> # Node ID 68bbec307c142b6b41893512b1c76320c87c2fa1
>>> # Parent  bd3927325fe48e104b1627e5681ccd09a9a49e44
>>> # EXP-Topic b2.bookmarks
>>> # Available At https://bitbucket.org/octobus/mercurial-devel/
>>> #  hg pull https://bitbucket.org/octobus/mercurial-deve
>>> l/ -r 68bbec307c14
>>> bundle2: support a 'records' mode for the 'bookmarks' part
>>> 
>>> In this mode, the bookmarks changes are record in the
>>> 'bundleoperation' records
>>> instead of inflicted to the repository. This is necessary to use
>>> the part when
>> 
>> s/inflicted/applied/?
>> 
>>> pulling.
>>> 
>> 
>> I'm confused. Why do we not want to apply the part when we're
>> pulling?
>> This log message could use some expansion.
> 
> When pulling, the remote bookmark value is not just applied -as-is-
> into the local repository. There is an extra step to detect bookmark
> divergence. The remote bookmarks data are stored until this processing
> happens.

Ah. That wasn't obvious. Let's make sure that comment makes it into v3 of this 
series in the commit message.___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1372: bundlerepo: make methods agree with base class

[indygreg (Gregory Szorc)]

This revision was automatically updated to reflect the committed changes.
Closed by commit rHG32d079f37207: bundlerepo: make methods agree with base 
class (authored by indygreg, committed by ).

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1372?vs=3427&id=3449

REVISION DETAIL
  https://phab.mercurial-scm.org/D1372

AFFECTED FILES
  mercurial/bundlerepo.py

CHANGE DETAILS

diff --git a/mercurial/bundlerepo.py b/mercurial/bundlerepo.py
--- a/mercurial/bundlerepo.py
+++ b/mercurial/bundlerepo.py
@@ -86,7 +86,7 @@
 self.bundlerevs.add(n)
 n += 1
 
-def _chunk(self, rev):
+def _chunk(self, rev, df=None):
 # Warning: in case of bundle, the diff is against what we stored as
 # delta base, not against rev - 1
 # XXX: could use some caching
@@ -108,7 +108,7 @@
 return mdiff.textdiff(self.revision(rev1, raw=True),
   self.revision(rev2, raw=True))
 
-def revision(self, nodeorrev, raw=False):
+def revision(self, nodeorrev, _df=None, raw=False):
 """return an uncompressed revision of a given node or revision
 number.
 """
@@ -152,12 +152,15 @@
 # needs to override 'baserevision' and make more specific call here.
 return revlog.revlog.revision(self, nodeorrev, raw=True)
 
-def addrevision(self, text, transaction, link, p1=None, p2=None, d=None):
+def addrevision(self, *args, **kwargs):
+raise NotImplementedError
+
+def addgroup(self, *args, **kwargs):
 raise NotImplementedError
-def addgroup(self, deltas, transaction, addrevisioncb=None):
+
+def strip(self, *args, **kwargs):
 raise NotImplementedError
-def strip(self, rev, minlink):
-raise NotImplementedError
+
 def checksize(self):
 raise NotImplementedError
 



To: indygreg, #hg-reviewers, durin42
Cc: durin42, lothiraldan, dlax, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 4 of 4 STABLE] dispatch: abort if early boolean options can't be parsed

[Augie Fackler]

On Sun, Nov 12, 2017 at 12:02:49AM +0900, Yuya Nishihara wrote:
> # HG changeset patch
> # User Yuya Nishihara 
> # Date 1510376561 -32400
> #  Sat Nov 11 14:02:41 2017 +0900
> # Branch stable
> # Node ID a8d8c3229692d77881a6242329160803cb5ab8c5
> # Parent  ed13a7f6d26a28f73593a091a6a57c756678245b
> dispatch: abort if early boolean options can't be parsed

queued for stable, thanks
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 5 of 4 STABLE] dispatch: fix early parsing of short option with value like -R=foo

[Augie Fackler]

On Tue, Nov 14, 2017 at 12:43:46AM +0900, Yuya Nishihara wrote:
> # HG changeset patch
> # User Yuya Nishihara 
> # Date 1510586759 -32400
> #  Tue Nov 14 00:25:59 2017 +0900
> # Branch stable
> # Node ID 6bccada86df7f70c24f2e62bedf9bfb5d796cbd1
> # Parent  a8d8c3229692d77881a6242329160803cb5ab8c5
> dispatch: fix early parsing of short option with value like -R=foo

queued for stable, thanks
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 6 of 6 V2] test-pattern: actually update tests using the patterns

[Augie Fackler]

On Mon, Nov 13, 2017 at 12:11:20PM +0100, Boris Feld wrote:
> # HG changeset patch
> # User Boris Feld 
> # Date 1509866592 -3600
> #  Sun Nov 05 08:23:12 2017 +0100
> # Node ID 715948f2256fa181c4f3730825492f9da0f74b2e
> # Parent  c06ed80fc1063905dc6a711fc32c88d5913b0db6
> # EXP-Topic better-substitute
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> #  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
> 715948f2256f
> test-pattern: actually update tests using the patterns


I love this feature. But, could we get away with a non-.py file
for configuring the substitutions? Maybe just use a configparser file
for now?

I'm hesitant to use Python files for config languages, it's been a bad
experience in the past.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 4 of 4] rebase: exclude obsoletes without a successor in destination (issue5300)

[Augie Fackler]

On Sun, Nov 12, 2017 at 06:06:37PM +0100, Denis Laxalde wrote:
> # HG changeset patch
> # User Denis Laxalde 
> # Date 1510489041 -3600
> #  Sun Nov 12 13:17:21 2017 +0100
> # Node ID 25d1891bd300bc527cb7e878532c8512ff5489de
> # Parent  1d16e28225e01d2fac9f31ed058ae1f4c00bb9da
> # EXP-Topic rebase-obsolete
> rebase: exclude obsoletes without a successor in destination (issue5300)

I took patches 1-3. I think this is right. I'm 90% sure this is
right. But I'm a little tired, and so I want Martin's more careful eye
on it if he has time.

>
> In the following example, when trying to rebase 3:: onto 2, the rebase
> will abort with "this rebase will cause divergence from: 4":
>
> o  7 f
> |
> | o  6 e
> | |
> | o  5 d'
> | |
> x |  4 d (rewritten as 5)
> |/
> o  3 c
> |
> | o  2 x
> | |
> o |  1 b
> |/
> o  0 a
>
> By excluding obsolete changesets without a successor in destination (4
> in the example above) and their descendants, we make rebase work in this
> case, thus giving:
>
> o  11 e
> |
> o  10 d'
> |
> o  9 c
> |
> o  8 b
> |
> | o  7 f
> | |
> | | x  6 e (rewritten using rebase as 11)
> | | |
> | | x  5 d' (rewritten using rebase as 10)
> | | |
> | x |  4 d
> | |/
> | x  3 c (rewritten using rebase as 9)
> | |
> o |  2 x
> | |
> | x  1 b (rewritten using rebase as 8)
> |/
> o  0 a
>
> where branch 4:: is left behind while branch 5:: is rebased as expected.
>
> The rationale is that users may not be interested in rebasing orphan
> changesets when specifying a rebase set that include them but would
> still want "stable" ones to be rebased. Currently, the user is suggested
> to allow divergence (but probably does not want it) or they must specify
> a rebase set excluding problematic changesets (which might be a bit
> cumbersome). The approach proposed here corresponds to "Option 2" in
> https://www.mercurial-scm.org/wiki/CEDRebase.
>
> .. feature::
>
>Let 'hg rebase' avoid content-divergence when obsolete changesets are
>present in the rebase set by skipping possible sources of divergence.
>
>
> We extend _computeobsoletenotrebased() so that it also return a set of
> obsolete changesets in rebaseset without a successor in destination.
> This 'obsoletewithoutsuccessorindestination' is then stored as an
> attribute of rebaseruntime and used in _performrebasesubset() to:
>
> * filter out descendants of these changesets from the revisions to
>   rebase;
> * issue a message about these revisions being skipped.
>
> This only occurs if 'evolution.allowdivergence' option is off and
> 'rebaseskipobsolete' is on.
>
> diff --git a/hgext/rebase.py b/hgext/rebase.py
> --- a/hgext/rebase.py
> +++ b/hgext/rebase.py
> @@ -179,6 +179,7 @@ class rebaseruntime(object):
>  # other extensions
>  self.keepopen = opts.get('keepopen', False)
>  self.obsoletenotrebased = {}
> +self.obsoletewithoutsuccessorindestination = set()
>
>  @property
>  def repo(self):
> @@ -311,9 +312,10 @@ class rebaseruntime(object):
>  if not self.ui.configbool('experimental', 'rebaseskipobsolete'):
>  return
>  obsoleteset = set(obsoleterevs)
> -self.obsoletenotrebased = _computeobsoletenotrebased(self.repo,
> -obsoleteset, destmap)
> +self.obsoletenotrebased, self.obsoletewithoutsuccessorindestination 
> = \
> +_computeobsoletenotrebased(self.repo, obsoleteset, destmap)
>  skippedset = set(self.obsoletenotrebased)
> +skippedset.update(self.obsoletewithoutsuccessorindestination)
>  _checkobsrebase(self.repo, self.ui, obsoleteset, skippedset)
>
>  def _prepareabortorcontinue(self, isabort):
> @@ -419,12 +421,26 @@ class rebaseruntime(object):
>  def _performrebasesubset(self, tr, subset, pos, total):
>  repo, ui, opts = self.repo, self.ui, self.opts
>  sortedrevs = repo.revs('sort(%ld, -topo)', subset)
> +allowdivergence = self.ui.configbool(
> +'experimental', 'evolution.allowdivergence')
> +if not allowdivergence:
> +sortedrevs -= repo.revs(
> +'descendants(%ld) and not %ld',
> +self.obsoletewithoutsuccessorindestination,
> +self.obsoletewithoutsuccessorindestination,
> +)
>  for rev in sortedrevs:
>  dest = self.destmap[rev]
>  ctx = repo[rev]
>  desc = _ctxdesc(ctx)
>  if self.state[rev] == rev:
>  ui.status(_('already rebased %s\n') % desc)
> +elif (not allowdivergence
> +  and rev in self.obsoletewithoutsuccessorindestination):
> +msg = _('note: not rebasing %s and its descendants as '
> +'this would cause divergences\n') % desc
> +repo.ui.status(msg)
> + 

D1371: global: remove redundant parenthesis

[indygreg (Gregory Szorc)]

indygreg created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  My code editor was spewing several warnings for redundant
  parenthesis. It supports easily fixing them. So I just did it
  globally.
  
  What redundant parenthesis are left are in 3rd party code or
  are needed around string literals so our style checker doesn't
  complain about the lack of _().

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1371

AFFECTED FILES
  contrib/check-config.py
  contrib/import-checker.py
  contrib/perf.py
  contrib/revsetbenchmarks.py
  doc/check-seclevel.py
  doc/docchecker
  doc/gendoc.py
  doc/hgmanpage.py
  hgext/bugzilla.py
  hgext/churn.py
  hgext/convert/convcmd.py
  hgext/convert/cvs.py
  hgext/convert/git.py
  hgext/convert/monotone.py
  hgext/convert/p4.py
  hgext/convert/subversion.py
  hgext/factotum.py
  hgext/hgk.py
  hgext/keyword.py
  hgext/largefiles/basestore.py
  hgext/largefiles/lfcommands.py
  hgext/largefiles/lfutil.py
  hgext/largefiles/overrides.py
  hgext/mq.py
  hgext/patchbomb.py
  hgext/show.py
  hgext/transplant.py
  hgext/uncommit.py
  hgext/win32mbcs.py
  mercurial/bookmarks.py
  mercurial/bundlerepo.py
  mercurial/byterange.py
  mercurial/changegroup.py
  mercurial/chgserver.py
  mercurial/cmdutil.py
  mercurial/commands.py
  mercurial/config.py
  mercurial/context.py
  mercurial/crecord.py
  mercurial/debugcommands.py
  mercurial/destutil.py
  mercurial/dirstate.py
  mercurial/discovery.py
  mercurial/dispatch.py
  mercurial/exchange.py
  mercurial/extensions.py
  mercurial/filelog.py
  mercurial/filemerge.py
  mercurial/fileset.py
  mercurial/hbisect.py
  mercurial/hg.py
  mercurial/hgweb/webutil.py
  mercurial/localrepo.py
  mercurial/mail.py
  mercurial/manifest.py
  mercurial/match.py
  mercurial/merge.py
  mercurial/minirst.py
  mercurial/obsolete.py
  mercurial/parser.py
  mercurial/patch.py
  mercurial/posix.py
  mercurial/progress.py
  mercurial/pure/base85.py
  mercurial/pure/mpatch.py
  mercurial/pure/parsers.py
  mercurial/revlog.py
  mercurial/revsetlang.py
  mercurial/setdiscovery.py
  mercurial/sshpeer.py
  mercurial/statprof.py
  mercurial/subrepo.py
  mercurial/tags.py
  mercurial/templater.py
  mercurial/transaction.py
  mercurial/txnutil.py
  mercurial/ui.py
  mercurial/url.py
  mercurial/util.py
  tests/check-perf-code.py
  tests/flagprocessorext.py
  tests/hghave.py
  tests/run-tests.py

CHANGE DETAILS

diff --git a/tests/run-tests.py b/tests/run-tests.py
--- a/tests/run-tests.py
+++ b/tests/run-tests.py
@@ -550,7 +550,7 @@
 if options.showchannels:
 options.nodiff = True
 
-return (options, args)
+return options, args
 
 def rename(src, dst):
 """Like os.rename(), trade atomicity and opened files friendliness
@@ -947,7 +947,7 @@
 
 def _portmap(self, i):
 offset = b'' if i == 0 else b'%d' % i
-return (br':%d\b' % (self._startport + i), b':$HGPORT%s' % offset)
+return br':%d\b' % (self._startport + i), b':$HGPORT%s' % offset
 
 def _getreplacements(self):
 """Obtain a mapping of text replacements to apply to test output.
@@ -1108,7 +1108,7 @@
 proc = subprocess.Popen(cmd, shell=True, cwd=self._testtmp,
 env=env)
 ret = proc.wait()
-return (ret, None)
+return ret, None
 
 proc = Popen4(cmd, self._testtmp, self._timeout, env)
 def cleanup():
diff --git a/tests/hghave.py b/tests/hghave.py
--- a/tests/hghave.py
+++ b/tests/hghave.py
@@ -243,8 +243,8 @@
 def gethgversion():
 m = matchoutput('hg --version --quiet 2>&1', br'(\d+)\.(\d+)')
 if not m:
-return (0, 0)
-return (int(m.group(1)), int(m.group(2)))
+return 0, 0
+return int(m.group(1)), int(m.group(2))
 
 @checkvers("hg", "Mercurial >= %s",
 list([(1.0 * x) / 10 for x in range(9, 99)]))
@@ -281,8 +281,8 @@
 def getgitversion():
 m = matchoutput('git --version 2>&1', br'git version (\d+)\.(\d+)')
 if not m:
-return (0, 0)
-return (int(m.group(1)), int(m.group(2)))
+return 0, 0
+return int(m.group(1)), int(m.group(2))
 
 @checkvers("git", "git client (with ext::sh support) version >= %s", (1.9,))
 def has_git_range(v):
@@ -301,8 +301,8 @@
 def getsvnversion():
 m = matchoutput('svn --version --quiet 2>&1', br'^(\d+)\.(\d+)')
 if not m:
-return (0, 0)
-return (int(m.group(1)), int(m.group(2)))
+return 0, 0
+return int(m.group(1)), int(m.group(2))
 
 @checkvers("svn", "subversion client and admin tools >= %s", (1.3, 1.5))
 def has_svn_range(v):
@@ -607,7 +607,7 @@
 @check("demandimport", "demandimport enabled")
 def has_demandimport():
 # chg disables demandimport intentionally for performance wins.
-return ((not has_chg()) and os.environ.get('HGDEMANDIMPORT') != 'disable')
+return (not has_chg()) and os.environ.get('HGDEMANDIMPORT') !=

D1224: merge: cache unknown dir checks (issue5716)

[durin42 (Augie Fackler)]

durin42 requested changes to this revision.
durin42 added a comment.
This revision now requires changes to proceed.


  needs rebased, otherwise looks fine

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1224

To: mbthomas, #hg-reviewers, durin42
Cc: durin42, sid0, krbullock, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1346: fsmonitor: only access inner dirstate map if it is available

[durin42 (Augie Fackler)]

durin42 requested changes to this revision.
durin42 added inline comments.
This revision now requires changes to proceed.

INLINE COMMENTS

> __init__.py:279
> +# directly access the inner dirstate map if the standard dirstate
> +# implementation is in use.
> +dmap = dmap._map

Do we do this for perf reasons?  If so, add a comment.

(Maybe check and see if this shows up on perfdirstate or similar - if it does, 
document that in the commit message with numbers, and then you don't even need 
a comment because we'll find that with blame.)

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1346

To: mbthomas, #hg-reviewers, durin42
Cc: durin42, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1345: dirstate: add explicit methods for querying directories

[durin42 (Augie Fackler)]

durin42 accepted this revision.
durin42 added a comment.
This revision is now accepted and ready to land.


  Looks good. For bonus points do one more upload with the right `.. api::` 
annotation for the releasenotes extension.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1345

To: mbthomas, #hg-reviewers, durin42
Cc: durin42, mbolin, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1343: dirstate: move dropping of folded filenames into the dirstate map

[durin42 (Augie Fackler)]

durin42 added a comment.


  btw I'm dubious of the 'filefoldmap' in self.__dict__ checks, as that very 
much looks like it should be an invariant now

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1343

To: mbthomas, #hg-reviewers, durin42
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1380: dirstate: document dirstatemap interface

[durin42 (Augie Fackler)]

durin42 requested changes to this revision.
durin42 added inline comments.
This revision now requires changes to proceed.

INLINE COMMENTS

> dirstate.py:1198
>  class dirstatemap(object):
> +'''
> +Dirstate contents

Nit: let's try and prefer """ to '''

Try and get a pithy one-line summary of the class on the first line of the 
docstring with the """

> dirstate.py:1214
> +
> +- `copymap` is a dict mapping filenames to the filename they were copied
> +  from.

`copymap` maps destination filenames to their source filename

perhaps? either way, optional.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1380

To: mbthomas, #hg-reviewers, durin42
Cc: durin42, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1347: dirstate: make map implementation overridable

[durin42 (Augie Fackler)]

durin42 added subscribers: mbolin, durin42.
durin42 accepted this revision as: durin42.
durin42 added a comment.


  I'd prefer this as a class attr, but would like to hear from @mbolin as to 
whether or not that solves the problem(s) that inspired 
https://phab.mercurial-scm.org/D1354.

INLINE COMMENTS

> dirstate.py:83
>  self._updatedfiles = set()
> +self._mapcls = dirstatemap
>  

Could you set this at class-level, rather than in __init__? I think then 
@mbolin would be able to make his subclass replace the class attribute and 
that'd be sufficient?

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1347

To: mbthomas, #hg-reviewers, durin42
Cc: durin42, mbolin, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1357: remotenames: consider existing data while storing newer data

[durin42 (Augie Fackler)]

durin42 accepted this revision as: durin42.
durin42 added a comment.


  Seems fine, though this use case does make me wonder (a little) if we should 
give up and use sqlite for this so we don't end up constantly rewriting a 
massive file.
  
  Don't rewrite in that direction though unless someone else wants to see that 
though.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1357

To: pulkit, #hg-reviewers, durin42
Cc: durin42, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D939: remotenames: add functionality to store remotenames under .hg/hgremotenames/

[durin42 (Augie Fackler)]

durin42 added a comment.


  I'm confused by the state of this diff - it appears to be modifying a file 
which doesn't yet exist in hg?

INLINE COMMENTS

> pulkit wrote in test-remotenames.t:69
> @durin42 related to your comment on list[0], is space separated format okay 
> or null separated is preferred?
> 
> [0]: 
> https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-November/107528.html

This is sort of okay, but won't scale well. I'd rather we used something we 
know we'll never see in a path like NUL rather than space separated, but this 
is probably fine.

I might bias towards making the first line an int followed by a newline so we 
can easily iterate on the file format and migrate things in the future.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D939

To: pulkit, #hg-reviewers, dlax, ryanmce
Cc: durin42, ryanmce, dlax, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D971: uncommit: unify functions _uncommitdirstate and _unamenddirstate to one

[durin42 (Augie Fackler)]

durin42 added subscribers: hooper, martinvonz, durin42.
durin42 accepted this revision as: durin42.
durin42 added a comment.


  (Marking accepted, but consider just moving this to `scmutil` or 
`rewriteutil` (I made that second one up, but it might be a sensible thing to 
have?) in parallel with the inevitable bikeshedding over on 
https://phab.mercurial-scm.org/D821.

INLINE COMMENTS

> uncommit.py:101
>  
> -def _uncommitdirstate(repo, oldctx, match):
> -"""Fix the dirstate after switching the working directory from
> -oldctx to a copy of oldctx not containing changed files matched by
> -match.
> +def _fixdirstate(repo, oldctx, newctx, status):
> +""" fix the dirstate after switching the working directory from oldctx to

This looks like it should be a utility method in core. I think @martinvonz or 
@hooper ran into a case where this would have been helpful.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D971

To: pulkit, #hg-reviewers, mbthomas, durin42
Cc: durin42, martinvonz, hooper, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D821: unamend: move fb extension unamend to core

[durin42 (Augie Fackler)]

durin42 added inline comments.

INLINE COMMENTS

> uncommit.py:247
> +def unamend(ui, repo, **opts):
> +"""undo the amend operation on a current changeset
> +

undo the _most recent_ amend? or can I run this iteratively and undo many 
amends in sequence?

> uncommit.py:265
> +markers = list(predecessormarkers(curctx))
> +if len(markers) != 1:
> +e = _("changeset must have one predecessor, found %i 
> predecessors")

Should we also look for unamend_source in the extra, and potentially refuse to 
unamend an unamend? Or not?

> uncommit.py:273
> +if curctx.children():
> +raise error.Abort(_("cannot unamend in the middle of a stack"))
> +

cannot unamend a changeset with children

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D821

To: pulkit, #hg-reviewers, durham
Cc: durin42, ryanmce, singhsrb, durham, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1336: remove: print message for each file in verbose mode only while using `-A`

[durin42 (Augie Fackler)]

durin42 added a comment.


  I think this probably violates our BC policy. We could probably stand to add 
a `--quiet` for suppressing this output, I suppose.
  
  I'll leave this open for other reviewers to mull over, and if it's still open 
in a week without comment I'll probably pass final judgement.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1336

To: pavanpc, #hg-reviewers, lothiraldan
Cc: durin42, lothiraldan, mitrandir, mharbison72, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1372: bundlerepo: make methods agree with base class

[durin42 (Augie Fackler)]

durin42 added a comment.


  In https://phab.mercurial-scm.org/D1372#22818, @lothiraldan wrote:
  
  > In https://phab.mercurial-scm.org/D1372#22814, @dlax wrote:
  >
  > > > For methods that are implemented, we change arguments to match the base.
  > >
  > > Alternatively, we could use `**kwargs` for keywords arguments unused in a 
method. I think that's a common pattern and it avoids confusing the reader with 
unused arguments.
  >
  >
  > 👍 for using `**kwargs` and even maybe `*args` if necessary.
  
  
  The standard we've recently moved to at Google is to allow `del foo  # 
unused` for unused arguments, so they're forcibly unused, but also still 
demonstrably passed. It makes things easier overall for various inference 
tools, and I'd be happy to adopt that in hg too.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1372

To: indygreg, #hg-reviewers
Cc: durin42, lothiraldan, dlax, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1358: remotenames: store journal entry for bookmarks if journal is loaded

[durin42 (Augie Fackler)]

durin42 added a comment.


  In https://phab.mercurial-scm.org/D1358#22899, @smf wrote:
  
  > pulkit (Pulkit Goyal)  writes:
  >
  > > pulkit created this revision.
  > >  Herald added a subscriber: mercurial-devel.
  > >  Herald added a reviewer: hg-reviewers.
  > > 
  > > REPOSITORY
  > > 
  > >   rHG Mercurial
  > > 
  > > REVISION DETAIL
  > > 
  > >   https://phab.mercurial-scm.org/D1358
  > > 
  > > AFFECTED FILES
  > > 
  > >   mercurial/remotenames.py
  > > 
  > > CHANGE DETAILS
  > > 
  > > diff --git a/mercurial/remotenames.py b/mercurial/remotenames.py
  > > 
  > > - a/mercurial/remotenames.py +++ b/mercurial/remotenames.py
  >
  > A bit jumping the gun, no? I don't think we should be skipping putting
  >  this initial stuff into hgext/remotenames.py.
  
  
  I don't think so. IMO the storage-only parts can and should go straight in 
core and be on by default. Exposing the data (by default, at least) we're 
collecting should probably go through an extension rodeo first.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1358

To: pulkit, #hg-reviewers
Cc: durin42, smf, dlax, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[Bug 5740] New: No obvious way to disable graph output added from alias

[mercurial-bugs]

https://bz.mercurial-scm.org/show_bug.cgi?id=5740

Bug ID: 5740
   Summary: No obvious way to disable graph output added from
alias
   Product: Mercurial
   Version: default branch
  Hardware: PC
OS: Windows
Status: UNCONFIRMED
  Severity: feature
  Priority: wish
 Component: Mercurial
  Assignee: bugzi...@mercurial-scm.org
  Reporter: gregory.sz...@gmail.com
CC: mercurial-devel@mercurial-scm.org

https://bugzilla.mozilla.org/show_bug.cgi?id=1415922 is an interesting bug
report.

At the crux of it, some automated code attempts to run `hg log -T '{node}'` to
resolve a node value. However, the user's hgrc has an [alias] for "log = log
-G". So the `hg log` output contains a leading graph node before the node.

The behavior is the same if HGPLAIN is used.

A workaround is to use `--config alias.log=log`. But that feels hacky.

It feels like there should be a more scripting-friendly way to ensure aliases
don't come into play when running commands.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: D1358: remotenames: store journal entry for bookmarks if journal is loaded

[Sean Farley]

pulkit (Pulkit Goyal)  writes:

> pulkit created this revision.
> Herald added a subscriber: mercurial-devel.
> Herald added a reviewer: hg-reviewers.
>
> REPOSITORY
>   rHG Mercurial
>
> REVISION DETAIL
>   https://phab.mercurial-scm.org/D1358
>
> AFFECTED FILES
>   mercurial/remotenames.py
>
> CHANGE DETAILS
>
> diff --git a/mercurial/remotenames.py b/mercurial/remotenames.py
> --- a/mercurial/remotenames.py
> +++ b/mercurial/remotenames.py

A bit jumping the gun, no? I don't think we should be skipping putting
this initial stuff into hgext/remotenames.py.


signature.asc
Description: PGP signature
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D939: remotenames: add functionality to store remotenames under .hg/hgremotenames/

[pulkit (Pulkit Goyal)]

pulkit added a subscriber: durin42.
pulkit added inline comments.

INLINE COMMENTS

> test-remotenames.t:69
> +  ec2426147f0e39dbc9cef599b066be6035ce691d file:$TESTTMP/server default
> +  3e1487808078543b0af6d10dadf5d46943578db0 file:$TESTTMP/server wat

@durin42 related to your comment on list[0], is space separated format okay or 
null separated is preferred?

[0]: 
https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-November/107528.html

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D939

To: pulkit, #hg-reviewers, dlax, ryanmce
Cc: durin42, ryanmce, dlax, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1378: bundlerepo: assign bundle attributes in bundle type blocks

[indygreg (Gregory Szorc)]

indygreg added inline comments.

INLINE COMMENTS

> dlax wrote in bundlerepo.py:298
> Within `_handlebundle2part`, there's still a `self._bundle = 
> changegroup.getunbundler(...)` statement. Perhaps, it'd be even clearer if 
> this method would `return changegroup.getunbundler(...)` and let the caller 
> eventually assign this as an attribute.

Yes, this API would be cleaner.

The reason I hesitated is `_handlebundle2part()` is generic. It feels weird to 
return a changegroup unpacker from such a generic method.

Since I'm incurring an API break in this series anyway and since treemanifests 
is the only thing extending this code AFAICT, I may just refactor things 
further.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1378

To: indygreg, #hg-reviewers, dlax
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: Desired use case for obsmarkers / visibility

[Gregory Szorc]

On Fri, Nov 3, 2017 at 12:34 PM, Boris Feld  wrote:

> On Thu, 2017-11-02 at 10:06 -0700, Gregory Szorc wrote:
> > I have a potential use case for obsmarkers / visibility that I want
> > to run by people to see if it can be supported.
> >
> > Changesets are pushed to the Firefox repo via a landing service. This
> > service essentially imports changesets [submitted by the author] and
> > rebases them onto the repo head.
> >
> > Today, when your changesets are landed, you need to perform "garbage
> > collection" on your local repo to remove the old versions of
> > changesets. We want landed changesets to disappear after `hg pull`
> > picks up the rebased versions.
> >
> > This is a pretty straightforward scenario and is supported by
> > obsmarkers today. If we enabled the writing of obsolescence markers
> > in the landing service, things would essentially "just work."
> >
> > Here's where things get a little more complicated.
> >
> > When changesets are landed to the Firefox repo today, they are first
> > pushed to an "integration" repository. Logically, this can be modeled
> > as a single repo divided into public and draft parts. e.g.
> >
> > o D (draft) (head)
> > o C (draft)
> > o B (public)
> > o A (public) (root)
> >
> > When our CI says a changeset is "good," it is promoted to public.
> > e.g.
> >
> > o D (draft)
> > o C (public) (formerly draft)
> > o B (public)
> > o A (public) (root)
> >
> > Today, when we encounter a "bad" changeset, we perform a backout.
> > e.g.
> >
> > o D' (draft) (backout of D)
> > o D (draft)
> > o C (public)
> > o B (public)
> > o A (public) (root)
> >
> > Given our push velocity, it is common to have intermediary changesets
> > land before a changeset is identified as "bad." This means there are
> > changesets between the initial landings and its backout. e.g.
> >
> > o D' (draft) (backout of D)
> > o E (draft)
> > o D (draft)
> > o C (public)
> > o B (public)
> > o A (public) (root)
> >
> > The repo with the backouts is eventually published and the final
> > history of the repo is littered with "bad" changesets and backouts.
> > This causes all kinds of problems for bisection, annotate, file
> > history, etc.
> >
> > Instead of performing backouts and leaving the final repo history in
> > a sub-optimal state, we want to instead "drop" "bad" changesets
> > before they are published. e.g.
> >
> > o E' (draft) (rebased from discarded D to C)
> > | x D (draft) (discarded)
> > o C (public)
> > o B (public)
> > o A (public) (root)
> >
> > Since we can identify "bad" changesets relatively quickly, this would
> > enable us to remove the vast majority of backouts and "bad"
> > changesets from the final, published repo history.
> >
> > Again, obsolescence as it exists today facilitates this. We can
> > perform these drops via `hg histedit` (or similar) and the
> > appropriate "prune" obsmarkers are written so the canonical repo has
> > the appropriate final history.
> >
> > However, the way it works today isn't friendly to end-user workflows.
> >
> > If we were to deploy this, the following would happen:
> >
> > 1) User creates changeset X and submits for landing.
> > 2) Landing service rebases to X' and writes X->X' marker.
> > 3) X' turns out to be bad and is dropped. X'->null marker is written
> > to convey the prune.
> > 4) User pulls and sees X->X'->null and hides X because its most
> > recent successor is pruned.
> > 5) User is left wondering what happened to X. They possibly forget
> > they need to fix and reland X.
> >
> > This is bad UX. What we want to happen instead is:
> >
> > a) User pulls after X' drop and X is still visible.
> > b) Something else happens and some form of X remains
> > visible/accessible to user
> >
> > The server can't expose X' because everyone would see it. We have 1
> > head per repo and don't want to be exposing random "bad" changesets
> > to everyone. This seems to rule out the traditional evolve solution
> > of "touch" a changeset to revive a prune because I'm not sure how
> > we'd send X' to only the user that cares about it. There's also no
> > way in obsolescence today to unhide X once it has been obsoleted.
> >
> > In the obsmarker world of today, the best solution I can think of is
> > "delete obsmarkers on the server." If we discarded the X->X' marker
> > (or didn't write it until X' became public), the end-user's original
> > changeset X wouldn't be hidden on pull because there is no marker on
> > the server referencing X. But this approach feels hacky and is extra
> > server-side complexity, which I'd prefer to avoid.
>
> First, `hg strip` get ride of X' and the obsmarkers for you, but that
> is a more hacky and traumatic for the repository that you will want
> (especially the caches).
>

We have a strong desire to avoid strip. We have a writable master server
and separate read-only mirrors. Performing a strip means having to
replicate that strip. And when the repo is being stripped, recent
changesets may not be available in the revl

Re: Desired use case for obsmarkers / visibility

[Gregory Szorc]

On Mon, Nov 13, 2017 at 8:33 AM, Boris Feld  wrote:

> On Fri, 2017-11-03 at 20:34 +0100, Boris Feld wrote:
> > On Thu, 2017-11-02 at 10:06 -0700, Gregory Szorc wrote:
> > > I have a potential use case for obsmarkers / visibility that I want
> > > to run by people to see if it can be supported.
> > >
> > > Changesets are pushed to the Firefox repo via a landing service.
> > > This
> > > service essentially imports changesets [submitted by the author]
> > > and
> > > rebases them onto the repo head.
> > >
> > > Today, when your changesets are landed, you need to perform
> > > "garbage
> > > collection" on your local repo to remove the old versions of
> > > changesets. We want landed changesets to disappear after `hg pull`
> > > picks up the rebased versions.
> > >
> > > This is a pretty straightforward scenario and is supported by
> > > obsmarkers today. If we enabled the writing of obsolescence markers
> > > in the landing service, things would essentially "just work."
> > >
> > > Here's where things get a little more complicated.
> > >
> > > When changesets are landed to the Firefox repo today, they are
> > > first
> > > pushed to an "integration" repository. Logically, this can be
> > > modeled
> > > as a single repo divided into public and draft parts. e.g.
> > >
> > > o D (draft) (head)
> > > o C (draft)
> > > o B (public)
> > > o A (public) (root)
> > >
> > > When our CI says a changeset is "good," it is promoted to public.
> > > e.g.
> > >
> > > o D (draft)
> > > o C (public) (formerly draft)
> > > o B (public)
> > > o A (public) (root)
> > >
> > > Today, when we encounter a "bad" changeset, we perform a backout.
> > > e.g.
> > >
> > > o D' (draft) (backout of D)
> > > o D (draft)
> > > o C (public)
> > > o B (public)
> > > o A (public) (root)
> > >
> > > Given our push velocity, it is common to have intermediary
> > > changesets
> > > land before a changeset is identified as "bad." This means there
> > > are
> > > changesets between the initial landings and its backout. e.g.
> > >
> > > o D' (draft) (backout of D)
> > > o E (draft)
> > > o D (draft)
> > > o C (public)
> > > o B (public)
> > > o A (public) (root)
> > >
> > > The repo with the backouts is eventually published and the final
> > > history of the repo is littered with "bad" changesets and backouts.
> > > This causes all kinds of problems for bisection, annotate, file
> > > history, etc.
> > >
> > > Instead of performing backouts and leaving the final repo history
> > > in
> > > a sub-optimal state, we want to instead "drop" "bad" changesets
> > > before they are published. e.g.
> > >
> > > o E' (draft) (rebased from discarded D to C)
> > > >  x D (draft) (discarded)
> > >
> > > o C (public)
> > > o B (public)
> > > o A (public) (root)
> > >
> > > Since we can identify "bad" changesets relatively quickly, this
> > > would
> > > enable us to remove the vast majority of backouts and "bad"
> > > changesets from the final, published repo history.
> > >
> > > Again, obsolescence as it exists today facilitates this. We can
> > > perform these drops via `hg histedit` (or similar) and the
> > > appropriate "prune" obsmarkers are written so the canonical repo
> > > has
> > > the appropriate final history.
> > >
> > > However, the way it works today isn't friendly to end-user
> > > workflows.
> > >
> > > If we were to deploy this, the following would happen:
> > >
> > > 1) User creates changeset X and submits for landing.
> > > 2) Landing service rebases to X' and writes X->X' marker.
> > > 3) X' turns out to be bad and is dropped. X'->null marker is
> > > written
> > > to convey the prune.
> > > 4) User pulls and sees X->X'->null and hides X because its most
> > > recent successor is pruned.
> > > 5) User is left wondering what happened to X. They possibly forget
> > > they need to fix and reland X.
> > >
> > > This is bad UX. What we want to happen instead is:
> > >
> > > a) User pulls after X' drop and X is still visible.
> > > b) Something else happens and some form of X remains
> > > visible/accessible to user
> > >
> > > The server can't expose X' because everyone would see it. We have 1
> > > head per repo and don't want to be exposing random "bad" changesets
> > > to everyone. This seems to rule out the traditional evolve solution
> > > of "touch" a changeset to revive a prune because I'm not sure how
> > > we'd send X' to only the user that cares about it. There's also no
> > > way in obsolescence today to unhide X once it has been obsoleted.
> > >
> > > In the obsmarker world of today, the best solution I can think of
> > > is
> > > "delete obsmarkers on the server." If we discarded the X->X' marker
> > > (or didn't write it until X' became public), the end-user's
> > > original
> > > changeset X wouldn't be hidden on pull because there is no marker
> > > on
> > > the server referencing X. But this approach feels hacky and is
> > > extra
> > > server-side complexity, which I'd prefer to avoid.
> >
> > First, `hg strip` get ride of X' 

D1371: global: remove redundant parenthesis

[indygreg (Gregory Szorc)]

indygreg added a comment.


  I used my editor's feature to fix violations. That's probably what introduced 
the whitespace changes. If only Mercurial conformed to PEP-8 style...

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1371

To: indygreg, #hg-reviewers, yuja
Cc: yuja, dlax, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: Desired use case for obsmarkers / visibility

[Boris Feld]

On Fri, 2017-11-03 at 20:34 +0100, Boris Feld wrote:
> On Thu, 2017-11-02 at 10:06 -0700, Gregory Szorc wrote:
> > I have a potential use case for obsmarkers / visibility that I want
> > to run by people to see if it can be supported.
> > 
> > Changesets are pushed to the Firefox repo via a landing service.
> > This
> > service essentially imports changesets [submitted by the author]
> > and
> > rebases them onto the repo head.
> > 
> > Today, when your changesets are landed, you need to perform
> > "garbage
> > collection" on your local repo to remove the old versions of
> > changesets. We want landed changesets to disappear after `hg pull`
> > picks up the rebased versions.
> > 
> > This is a pretty straightforward scenario and is supported by
> > obsmarkers today. If we enabled the writing of obsolescence markers
> > in the landing service, things would essentially "just work."
> > 
> > Here's where things get a little more complicated.
> > 
> > When changesets are landed to the Firefox repo today, they are
> > first
> > pushed to an "integration" repository. Logically, this can be
> > modeled
> > as a single repo divided into public and draft parts. e.g.
> > 
> > o D (draft) (head)
> > o C (draft)
> > o B (public)
> > o A (public) (root)
> > 
> > When our CI says a changeset is "good," it is promoted to public.
> > e.g.
> > 
> > o D (draft)
> > o C (public) (formerly draft)
> > o B (public)
> > o A (public) (root)
> > 
> > Today, when we encounter a "bad" changeset, we perform a backout.
> > e.g.
> > 
> > o D' (draft) (backout of D)
> > o D (draft)
> > o C (public)
> > o B (public)
> > o A (public) (root)
> > 
> > Given our push velocity, it is common to have intermediary
> > changesets
> > land before a changeset is identified as "bad." This means there
> > are
> > changesets between the initial landings and its backout. e.g.
> > 
> > o D' (draft) (backout of D)
> > o E (draft)
> > o D (draft)
> > o C (public)
> > o B (public)
> > o A (public) (root)
> > 
> > The repo with the backouts is eventually published and the final
> > history of the repo is littered with "bad" changesets and backouts.
> > This causes all kinds of problems for bisection, annotate, file
> > history, etc.
> > 
> > Instead of performing backouts and leaving the final repo history
> > in
> > a sub-optimal state, we want to instead "drop" "bad" changesets
> > before they are published. e.g.
> > 
> > o E' (draft) (rebased from discarded D to C)
> > >  x D (draft) (discarded)
> > 
> > o C (public)
> > o B (public)
> > o A (public) (root)
> > 
> > Since we can identify "bad" changesets relatively quickly, this
> > would
> > enable us to remove the vast majority of backouts and "bad"
> > changesets from the final, published repo history.
> > 
> > Again, obsolescence as it exists today facilitates this. We can
> > perform these drops via `hg histedit` (or similar) and the
> > appropriate "prune" obsmarkers are written so the canonical repo
> > has
> > the appropriate final history.
> > 
> > However, the way it works today isn't friendly to end-user
> > workflows.
> > 
> > If we were to deploy this, the following would happen:
> > 
> > 1) User creates changeset X and submits for landing.
> > 2) Landing service rebases to X' and writes X->X' marker.
> > 3) X' turns out to be bad and is dropped. X'->null marker is
> > written
> > to convey the prune.
> > 4) User pulls and sees X->X'->null and hides X because its most
> > recent successor is pruned.
> > 5) User is left wondering what happened to X. They possibly forget
> > they need to fix and reland X.
> > 
> > This is bad UX. What we want to happen instead is:
> > 
> > a) User pulls after X' drop and X is still visible.
> > b) Something else happens and some form of X remains
> > visible/accessible to user
> > 
> > The server can't expose X' because everyone would see it. We have 1
> > head per repo and don't want to be exposing random "bad" changesets
> > to everyone. This seems to rule out the traditional evolve solution
> > of "touch" a changeset to revive a prune because I'm not sure how
> > we'd send X' to only the user that cares about it. There's also no
> > way in obsolescence today to unhide X once it has been obsoleted.
> > 
> > In the obsmarker world of today, the best solution I can think of
> > is
> > "delete obsmarkers on the server." If we discarded the X->X' marker
> > (or didn't write it until X' became public), the end-user's
> > original
> > changeset X wouldn't be hidden on pull because there is no marker
> > on
> > the server referencing X. But this approach feels hacky and is
> > extra
> > server-side complexity, which I'd prefer to avoid.
> 
> First, `hg strip` get ride of X' and the obsmarkers for you, but that
> is a more hacky and traumatic for the repository that you will want 
> (especially the caches).
> 
> Fortunately there are a couple of other low tech solutions available 
> with today implementation:
> 
> 
> For your usecase. If people are barely pulli

[PATCH 5 of 4 STABLE] dispatch: fix early parsing of short option with value like -R=foo

[Yuya Nishihara]

# HG changeset patch
# User Yuya Nishihara 
# Date 1510586759 -32400
#  Tue Nov 14 00:25:59 2017 +0900
# Branch stable
# Node ID 6bccada86df7f70c24f2e62bedf9bfb5d796cbd1
# Parent  a8d8c3229692d77881a6242329160803cb5ab8c5
dispatch: fix early parsing of short option with value like -R=foo

Before, -R=foo was parsed as '-R' 'foo', which disagrees with the standard
getopt behavior.

diff --git a/mercurial/dispatch.py b/mercurial/dispatch.py
--- a/mercurial/dispatch.py
+++ b/mercurial/dispatch.py
@@ -665,6 +665,10 @@ def _earlygetopt(aliases, args):
 >>> _earlygetopt([b'-R'], args), args
 (['bar'], ['x', 'y'])
 
+>>> args = [b'x', b'-R=bar', b'y']
+>>> _earlygetopt([b'-R'], args), args
+(['=bar'], ['x', 'y'])
+
 >>> args = [b'x', b'-R', b'--', b'y']
 >>> _earlygetopt([b'-R'], args), args
 ([], ['x', '-R', '--', 'y'])
@@ -678,7 +682,9 @@ def _earlygetopt(aliases, args):
 pos = 0
 while pos < argcount:
 fullarg = arg = args[pos]
-equals = arg.find('=')
+equals = -1
+if arg.startswith('--'):
+equals = arg.find('=')
 if equals > -1:
 arg = arg[:equals]
 if arg in aliases:
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1336: remove: print message for each file in verbose mode only while using `-A`

[lothiraldan (Boris Feld)]

lothiraldan accepted this revision.
lothiraldan added a comment.


  In https://phab.mercurial-scm.org/D1336#22841, @pavanpc wrote:
  
  > @lothiraldan  I modified a test case for hg rm -A  case where we 
get the message 'not removing : file still exists' . For other cases, 
test cases already exist with 'hg rm -A' and 'hg rm --after' without verbose 
mode.
  
  
  Thx, LGTM

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1336

To: pavanpc, #hg-reviewers, lothiraldan
Cc: lothiraldan, mitrandir, mharbison72, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1370: bundle2: use os.SEEK_* constants

[indygreg (Gregory Szorc)]

This revision was automatically updated to reflect the committed changes.
Closed by commit rHG241d9caca11e: bundle2: use os.SEEK_* constants (authored by 
indygreg, committed by ).

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1370?vs=3425&id=3446

REVISION DETAIL
  https://phab.mercurial-scm.org/D1370

AFFECTED FILES
  mercurial/bundle2.py

CHANGE DETAILS

diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
--- a/mercurial/bundle2.py
+++ b/mercurial/bundle2.py
@@ -148,6 +148,7 @@
 from __future__ import absolute_import, division
 
 import errno
+import os
 import re
 import string
 import struct
@@ -362,7 +363,7 @@
 self.count = count
 self.current = p
 yield p
-p.seek(0, 2)
+p.seek(0, os.SEEK_END)
 self.current = None
 self.iterator = func()
 return self.iterator
@@ -384,11 +385,11 @@
 try:
 if self.current:
 # consume the part content to not corrupt the stream.
-self.current.seek(0, 2)
+self.current.seek(0, os.SEEK_END)
 
 for part in self.iterator:
 # consume the bundle content
-part.seek(0, 2)
+part.seek(0, os.SEEK_END)
 except Exception:
 seekerror = True
 
@@ -858,8 +859,8 @@
 # Seek to the end of the part to force it's consumption so the next
 # part can be read. But then seek back to the beginning so the
 # code consuming this generator has a part that starts at 0.
-part.seek(0, 2)
-part.seek(0)
+part.seek(0, os.SEEK_END)
+part.seek(0, os.SEEK_SET)
 headerblock = self._readpartheader()
 indebug(self.ui, 'end of bundle2 stream')
 
@@ -1164,7 +1165,7 @@
 raise
 finally:
 if not hardabort:
-part.seek(0, 2)
+part.seek(0, os.SEEK_END)
 self.ui.debug('bundle2-input-stream-interrupt:'
   ' closing out of band context\n')
 
@@ -1330,12 +1331,12 @@
 def tell(self):
 return self._pos
 
-def seek(self, offset, whence=0):
-if whence == 0:
+def seek(self, offset, whence=os.SEEK_SET):
+if whence == os.SEEK_SET:
 newpos = offset
-elif whence == 1:
+elif whence == os.SEEK_CUR:
 newpos = self._pos + offset
-elif whence == 2:
+elif whence == os.SEEK_END:
 if not self.consumed:
 self.read()
 newpos = self._chunkindex[-1][0] - offset



To: indygreg, #hg-reviewers, dlax, yuja
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D974: py3: handle keyword arguments correctly in hgext/patchbomb.py

[pulkit (Pulkit Goyal)]

This revision was automatically updated to reflect the committed changes.
Closed by commit rHG71e63fe6b1ab: py3: handle keyword arguments correctly in 
hgext/patchbomb.py (authored by pulkit, committed by ).

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D974?vs=3376&id=3444

REVISION DETAIL
  https://phab.mercurial-scm.org/D974

AFFECTED FILES
  hgext/patchbomb.py

CHANGE DETAILS

diff --git a/hgext/patchbomb.py b/hgext/patchbomb.py
--- a/hgext/patchbomb.py
+++ b/hgext/patchbomb.py
@@ -89,6 +89,7 @@
 mail,
 node as nodemod,
 patch,
+pycompat,
 registrar,
 repair,
 scmutil,
@@ -318,7 +319,7 @@
 tmpfn = os.path.join(tmpdir, 'bundle')
 btype = ui.config('patchbomb', 'bundletype')
 if btype:
-opts['type'] = btype
+opts[r'type'] = btype
 try:
 commands.bundle(ui, repo, tmpfn, dest, **opts)
 return util.readfile(tmpfn)
@@ -338,8 +339,8 @@
 the user through the editor.
 """
 ui = repo.ui
-if opts.get('desc'):
-body = open(opts.get('desc')).read()
+if opts.get(r'desc'):
+body = open(opts.get(r'desc')).read()
 else:
 ui.write(_('\nWrite the introductory message for the '
'patch series.\n\n'))
@@ -359,21 +360,21 @@
 """
 ui = repo.ui
 _charsets = mail._charsets(ui)
-subj = (opts.get('subject')
+subj = (opts.get(r'subject')
 or prompt(ui, 'Subject:', 'A bundle for your repository'))
 
 body = _getdescription(repo, '', sender, **opts)
 msg = emailmod.MIMEMultipart.MIMEMultipart()
 if body:
-msg.attach(mail.mimeencode(ui, body, _charsets, opts.get('test')))
+msg.attach(mail.mimeencode(ui, body, _charsets, opts.get(r'test')))
 datapart = emailmod.MIMEBase.MIMEBase('application', 'x-mercurial-bundle')
 datapart.set_payload(bundle)
-bundlename = '%s.hg' % opts.get('bundlename', 'bundle')
+bundlename = '%s.hg' % opts.get(r'bundlename', 'bundle')
 datapart.add_header('Content-Disposition', 'attachment',
 filename=bundlename)
 emailmod.Encoders.encode_base64(datapart)
 msg.attach(datapart)
-msg['Subject'] = mail.headencode(ui, subj, _charsets, opts.get('test'))
+msg['Subject'] = mail.headencode(ui, subj, _charsets, opts.get(r'test'))
 return [(msg, subj, None)]
 
 def _makeintro(repo, sender, revs, patches, **opts):
@@ -384,27 +385,27 @@
 _charsets = mail._charsets(ui)
 
 # use the last revision which is likely to be a bookmarked head
-prefix = _formatprefix(ui, repo, revs.last(), opts.get('flag'),
+prefix = _formatprefix(ui, repo, revs.last(), opts.get(r'flag'),
0, len(patches), numbered=True)
-subj = (opts.get('subject') or
+subj = (opts.get(r'subject') or
 prompt(ui, '(optional) Subject: ', rest=prefix, default=''))
 if not subj:
 return None # skip intro if the user doesn't bother
 
 subj = prefix + ' ' + subj
 
 body = ''
-if opts.get('diffstat'):
+if opts.get(r'diffstat'):
 # generate a cumulative diffstat of the whole patch series
 diffstat = patch.diffstat(sum(patches, []))
 body = '\n' + diffstat
 else:
 diffstat = None
 
 body = _getdescription(repo, body, sender, **opts)
-msg = mail.mimeencode(ui, body, _charsets, opts.get('test'))
+msg = mail.mimeencode(ui, body, _charsets, opts.get(r'test'))
 msg['Subject'] = mail.headencode(ui, subj, _charsets,
- opts.get('test'))
+ opts.get(r'test'))
 return (msg, subj, diffstat)
 
 def _getpatchmsgs(repo, sender, revs, patchnames=None, **opts):
@@ -414,6 +415,7 @@
 
 This function returns a list of "email" tuples (subject, content, None).
 """
+bytesopts = pycompat.byteskwargs(opts)
 ui = repo.ui
 _charsets = mail._charsets(ui)
 patches = list(_getpatches(repo, revs, **opts))
@@ -423,7 +425,7 @@
  % len(patches))
 
 # build the intro message, or skip it if the user declines
-if introwanted(ui, opts, len(patches)):
+if introwanted(ui, bytesopts, len(patches)):
 msg = _makeintro(repo, sender, revs, patches, **opts)
 if msg:
 msgs.append(msg)
@@ -437,8 +439,8 @@
 for i, (r, p) in enumerate(zip(revs, patches)):
 if patchnames:
 name = patchnames[i]
-msg = makepatch(ui, repo, r, p, opts, _charsets, i + 1,
-len(patches), numbered, name)
+msg = makepatch(ui, repo, r, p, bytesopts, _charsets,
+i + 1, len(patches), numbered, name)
 msgs.append(msg)
 
 return msgs
@@ -579,6 +581,7 @@
 Before using this command, you will need to enable email in your
 hgrc. See the [email] section in hgrc(5) for details.
 '''
+opts = pycompat.byteskwargs(opts)
 
 _charsets = mail._cha

D1270: help: adding a topic on flags

[rdamazio (Rodrigo Damazio Bovendorp)]

This revision was automatically updated to reflect the committed changes.
Closed by commit rHGb0262b25ab48: help: adding a topic on flags (authored by 
rdamazio, committed by ).

CHANGED PRIOR TO COMMIT
  https://phab.mercurial-scm.org/D1270?vs=3359&id=3445#toc

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1270?vs=3359&id=3445

REVISION DETAIL
  https://phab.mercurial-scm.org/D1270

AFFECTED FILES
  contrib/wix/help.wxs
  mercurial/help.py
  mercurial/help/flags.txt
  tests/test-globalopts.t
  tests/test-help.t
  tests/test-hgweb-json.t

CHANGE DETAILS

diff --git a/tests/test-hgweb-json.t b/tests/test-hgweb-json.t
--- a/tests/test-hgweb-json.t
+++ b/tests/test-hgweb-json.t
@@ -1581,6 +1581,10 @@
 "topic": "filesets"
   },
   {
+"summary": "Command-line flags",
+"topic": "flags"
+  },
+  {
 "summary": "Glossary",
 "topic": "glossary"
   },
diff --git a/tests/test-help.t b/tests/test-help.t
--- a/tests/test-help.t
+++ b/tests/test-help.t
@@ -110,6 +110,7 @@
environment   Environment Variables
extensionsUsing Additional Features
filesets  Specifying File Sets
+   flags Command-line flags
glossary  Glossary
hgignore  Syntax for Mercurial Ignore Files
hgweb Configuring hgweb
@@ -188,6 +189,7 @@
environment   Environment Variables
extensionsUsing Additional Features
filesets  Specifying File Sets
+   flags Command-line flags
glossary  Glossary
hgignore  Syntax for Mercurial Ignore Files
hgweb Configuring hgweb
@@ -865,6 +867,7 @@
environment   Environment Variables
extensionsUsing Additional Features
filesets  Specifying File Sets
+   flags Command-line flags
glossary  Glossary
hgignore  Syntax for Mercurial Ignore Files
hgweb Configuring hgweb
@@ -2013,6 +2016,13 @@
   Specifying File Sets
   
   
+  
+  flags
+  
+  
+  Command-line flags
+  
+  
   
   glossary
   
diff --git a/tests/test-globalopts.t b/tests/test-globalopts.t
--- a/tests/test-globalopts.t
+++ b/tests/test-globalopts.t
@@ -355,6 +355,7 @@
environment   Environment Variables
extensionsUsing Additional Features
filesets  Specifying File Sets
+   flags Command-line flags
glossary  Glossary
hgignore  Syntax for Mercurial Ignore Files
hgweb Configuring hgweb
@@ -439,6 +440,7 @@
environment   Environment Variables
extensionsUsing Additional Features
filesets  Specifying File Sets
+   flags Command-line flags
glossary  Glossary
hgignore  Syntax for Mercurial Ignore Files
hgweb Configuring hgweb
diff --git a/mercurial/help/flags.txt b/mercurial/help/flags.txt
new file mode 100644
--- /dev/null
+++ b/mercurial/help/flags.txt
@@ -0,0 +1,104 @@
+Most Mercurial commands accept various flags.
+
+Flag names
+==
+
+Flags for each command are listed in :hg:`help` for that command.
+Additionally, some flags, such as --repository, are global and can be used with
+any command - those are seen in :hg:`help -v`, and can be specified before or
+after the command.
+
+Every flag has at least a long name, such as --repository. Some flags may also
+have a short one-letter name, such as the equivalent -R. Using the short or 
long
+name is equivalent and has the same effect.
+
+Flags that have a short name can also be bundled together - for instance, to
+specify both --edit (short -e) and --interactive (short -i), one could use::
+
+hg commit -ei
+
+If any of the bundled flags takes a value (i.e. is not a boolean), it must be
+last, followed by the value::
+
+hg commit -im 'Message'
+
+Flag types
+==
+
+Mercurial command-line flags can be strings, numbers, booleans, or lists of
+strings.
+
+Specifying flag values
+==
+
+The following syntaxes are allowed, assuming a flag 'flagname' with short name
+'f'::
+
+--flagname=foo
+--flagname foo
+-f foo
+-ffoo
+
+This syntax applies to all non-boolean flags (strings, numbers or lists).
+
+Specifying boolean flags
+
+
+Boolean flags do not take a value parameter. To specify a boolean, use the flag
+name to set it to true, or the same name prefixed with 'no-' to set it to
+false::
+
+hg commit --interactive
+hg commit --no-interactive
+
+Specifying list flags
+=
+
+List flags take multiple values. To specify them, pass the flag multiple 
times::
+
+hg files --include mercurial --include tests
+
+Setting flag defaults
+=
+
+In order to set a default value for a flag in an hgrc file, it is recommended 
to
+use aliases::
+
+[alias]
+commit = commit --interactive
+
+For more information on hgrc files, see :hg:`help config`.
+
+Overriding flags on the command line
+
+
+If the same 

D1348: histedit: add support to output nodechanges using formatter

[yuja (Yuya Nishihara)]

yuja requested changes to this revision.
yuja added inline comments.
This revision now requires changes to proceed.

INLINE COMMENTS

> histedit.py:1208
> +fd = fm.formatdict
> +nodechanges = fd({hf(oldn): fl([hf(n) for n in newn], 'succ')
> +  for oldn, newn in mapping.iteritems()})

Can you fix the formatter issue in rebase.py first so we won't
have to apply the same change again on histedit.py?

Probably we wouldn't want to call `hf(oldn)` and `fl([hf(n)...])` as `{key}` 
and `{value}` in template respectively. I think something
like `{oldnode}` and `{newnodes}` would be better, but I don't
know what are the correct terms here.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1348

To: pulkit, durin42, #hg-reviewers, dlax, yuja
Cc: yuja, dlax, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1270: help: adding a topic on flags

[yuja (Yuya Nishihara)]

yuja accepted this revision.
yuja added a comment.
This revision is now accepted and ready to land.


  Removed -f=foo and queued, thanks.
  
  > it makes permanent my --no- boolean prefix for flags,
  
  Perhaps we can add some note saying --no- is still experimental?

INLINE COMMENTS

> flags.txt:39
> +--flagname foo
> +-f=foo
> +-f foo

This is wrong. -f=foo is identical to -f =foo.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1270

To: rdamazio, #hg-reviewers, durin42, yuja
Cc: yuja, durin42, av6, dlax, martinvonz, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1371: global: remove redundant parenthesis

[yuja (Yuya Nishihara)]

yuja requested changes to this revision.
yuja added inline comments.
This revision now requires changes to proceed.

INLINE COMMENTS

> revsetlang.py:298
>  # x + y + z -> (or x y z) -> (or (list x y z))
> -return (op, _fixops(('list',) + x[1:]))
>  elif op == 'subscript' and x[1][0] == 'relation':

I don't like this sort of changes because here a tuple is a
data structure used thoroughly, not an arbitrary set of values
to be returned.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1371

To: indygreg, #hg-reviewers, yuja
Cc: yuja, dlax, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D974: py3: handle keyword arguments correctly in hgext/patchbomb.py

[yuja (Yuya Nishihara)]

yuja accepted this revision.
yuja added a comment.
This revision is now accepted and ready to land.


  Queued this, thanks.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D974

To: pulkit, #hg-reviewers, yuja
Cc: yuja, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [POLL] Mass renaming options for consistency + guidelines

[Yuya Nishihara]

On Sun, 12 Nov 2017 12:26:18 -0500, Augie Fackler wrote:
> > On Nov 12, 2017, at 4:42 AM, Yuya Nishihara  wrote:
> > On Sat, 14 Oct 2017 14:02:03 +0900, Yuya Nishihara wrote:
> >> On Sat, 14 Oct 2017 00:41:04 -0400, Augie Fackler wrote:
> >>> 
> >>> durin42: +1
> >>> martinvonz: +1
> >>> David Demelier: +1
> >>> Yuya: ? (I think +0 on hyphens at least for new config knobs?)
> >> 
> >> Yep, +0. More precisely:
> >> 
> >> * okay: allow_pull
> >> * preferred: allow-pull
> >> * also preferred: allowpull
> >> 
> >> FWIW, dashes can't be used in some config sections.
> >> 
> >> * [hooks] - pre-/post- is reserved for command hooks
> >> * [revsetaliases], [templatealiases] - name must be a valid symbol
> > 
> > So we're agreed to do the mass renaming, right?
> > I thought it was only for new options, but maybe I was wrong.
> 
> I’m happy to have aliases for things with the new config registrar, since it 
> centralizes the list of aliases and greatly reduces the maintenance burden.
> 
> I don’t feel strongly about renaming the canonical version in the docs 
> though. If we think it’s better to leave the authoritative name the 
> longer-standing name, that’s completely fine with me.

FWIW, introducing bunch of permanent aliases might not be as simple as it
sounds. It will basically add another axis to the current config layer,
[global, user, repo] * [oldname, new-name].

Should user's "ui.tweakdefaults" precede global "ui.tweak-defaults" for example?
Probably it should. Should they both be listed in "hg config"? Maybe, but how
can we know which is in effect? No idea...
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1345: dirstate: add explicit methods for querying directories

[mbthomas (Mark Thomas)]

mbthomas updated this revision to Diff 3442.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1345?vs=3345&id=3442

REVISION DETAIL
  https://phab.mercurial-scm.org/D1345

AFFECTED FILES
  contrib/perf.py
  hgext/largefiles/reposetup.py
  mercurial/dirstate.py

CHANGE DETAILS

diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -157,8 +157,8 @@
 def _pl(self):
 return self._map.parents()
 
-def dirs(self):
-return self._map.dirs
+def hasdir(self, d):
+return self._map.hastrackeddir(d)
 
 @rootcache('.hgignore')
 def _ignore(self):
@@ -390,11 +390,11 @@
 oldstate = self[f]
 if state == 'a' or oldstate == 'r':
 scmutil.checkfilename(f)
-if f in self._map.dirs:
+if self._map.hastrackeddir(f):
 raise error.Abort(_('directory %r already in dirstate') % f)
 # shadows
 for d in util.finddirs(f):
-if d in self._map.dirs:
+if self._map.hastrackeddir(d):
 break
 entry = self._map.get(d)
 if entry is not None and entry[0] != 'r':
@@ -770,7 +770,6 @@
 results = dict.fromkeys(subrepos)
 results['.hg'] = None
 
-alldirs = None
 for ff in files:
 # constructing the foldmap is expensive, so don't do it for the
 # common case where files is ['.']
@@ -801,9 +800,7 @@
 if nf in dmap: # does it exactly match a missing file?
 results[nf] = None
 else: # does it match a missing directory?
-if alldirs is None:
-alldirs = util.dirs(dmap._map)
-if nf in alldirs:
+if self._map.hasdir(nf):
 if matchedir:
 matchedir(nf)
 notfoundadd(nf)
@@ -1199,9 +1196,6 @@
 - `otherparentset` is a set of the filenames that are marked as coming
   from the second parent when the dirstate is currently being merged.
 
-- `dirs` is a set-like object containing all the directories that contain
-  files in the dirstate, excluding any files that are marked as removed.
-
 - `filefoldmap` is a dict mapping normalized filenames to the denormalized
   form that they appear as in the dirstate.
 
@@ -1237,7 +1231,8 @@
 self._map.clear()
 self.copymap.clear()
 self.setparents(nullid, nullid)
-util.clearcachedproperty(self, "dirs")
+util.clearcachedproperty(self, "_dirs")
+util.clearcachedproperty(self, "_alldirs")
 util.clearcachedproperty(self, "filefoldmap")
 util.clearcachedproperty(self, "dirfoldmap")
 util.clearcachedproperty(self, "nonnormalset")
@@ -1270,8 +1265,10 @@
 
 def addfile(self, f, oldstate, state, mode, size, mtime):
 """Add a tracked file to the dirstate."""
-if oldstate in "?r" and "dirs" in self.__dict__:
-self.dirs.addpath(f)
+if oldstate in "?r" and "_dirs" in self.__dict__:
+self._dirs.addpath(f)
+if oldstate == "?" and "_alldirs" in self.__dict__:
+self._alldirs.addpath(f)
 self._map[f] = dirstatetuple(state, mode, size, mtime)
 if state != 'n' or mtime == -1:
 self.nonnormalset.add(f)
@@ -1286,8 +1283,10 @@
 the file's previous state.  In the future, we should refactor this
 to be more explicit about what that state is.
 """
-if oldstate not in "?r" and "dirs" in self.__dict__:
-self.dirs.delpath(f)
+if oldstate not in "?r" and "_dirs" in self.__dict__:
+self._dirs.delpath(f)
+if oldstate == "?" and "_alldirs" in self.__dict__:
+self._alldirs.addpath(f)
 if "filefoldmap" in self.__dict__:
 normed = util.normcase(f)
 self.filefoldmap.pop(normed, None)
@@ -1301,8 +1300,10 @@
 """
 exists = f in self._map
 if exists:
-if oldstate != "r" and "dirs" in self.__dict__:
-self.dirs.delpath(f)
+if oldstate != "r" and "_dirs" in self.__dict__:
+self._dirs.delpath(f)
+if "_alldirs" in self.__dict__:
+self._alldirs.delpath(f)
 del self._map[f]
 if "filefoldmap" in self.__dict__:
 normed = util.normcase(f)
@@ -1352,13 +1353,28 @@
 f['.'] = '.' # prevents useless util.fspath() invocation
 return f
 
+def hastrackeddir(self, d):
+"""
+Returns True if the dirstate contains a tracked (not removed) file
+in this directory.
+"""
+return d in self._dirs
+
+def hasdir(self, d):
+"""
+Returns True if the dirstate contains a file (tracked or removed)
+

D1347: dirstate: make map implementation overridable

[mbthomas (Mark Thomas)]

mbthomas updated this revision to Diff 3443.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1347?vs=3347&id=3443

REVISION DETAIL
  https://phab.mercurial-scm.org/D1347

AFFECTED FILES
  mercurial/dirstate.py

CHANGE DETAILS

diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -80,6 +80,7 @@
 self._plchangecallbacks = {}
 self._origpl = None
 self._updatedfiles = set()
+self._mapcls = dirstatemap
 
 @contextlib.contextmanager
 def parentchange(self):
@@ -128,7 +129,7 @@
 @propertycache
 def _map(self):
 '''Return the dirstate contents (see documentation for dirstatemap).'''
-self._map = dirstatemap(self._ui, self._opener, self._root)
+self._map = self._mapcls(self._ui, self._opener, self._root)
 return self._map
 
 @property



To: mbthomas, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1343: dirstate: move dropping of folded filenames into the dirstate map

[mbthomas (Mark Thomas)]

mbthomas updated this revision to Diff 3440.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1343?vs=3343&id=3440

REVISION DETAIL
  https://phab.mercurial-scm.org/D1343

AFFECTED FILES
  mercurial/dirstate.py

CHANGE DETAILS

diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -387,11 +387,6 @@
 return self._map.copymap
 
 def _droppath(self, f):
-if "filefoldmap" in self._map.__dict__:
-normed = util.normcase(f)
-if normed in self._map.filefoldmap:
-del self._map.filefoldmap[normed]
-
 self._updatedfiles.add(f)
 
 def _addpath(self, f, state, mode, size, mtime):
@@ -1215,9 +1210,6 @@
 
 - `dirfoldmap` is a dict mapping normalized directory names to the
   denormalized form that they appear as in the dirstate.
-
-Once instantiated, the filefoldmap and dirfoldmap views must be maintained
-by the caller.
 '''
 
 def __init__(self, ui, opener, root):
@@ -1299,6 +1291,9 @@
 """
 if oldstate not in "?r" and "dirs" in self.__dict__:
 self.dirs.delpath(f)
+if "filefoldmap" in self.__dict__:
+normed = util.normcase(f)
+self.filefoldmap.pop(normed, None)
 self._map[f] = dirstatetuple('r', 0, size, 0)
 self.nonnormalset.add(f)
 
@@ -1312,6 +1307,9 @@
 if oldstate != "r" and "dirs" in self.__dict__:
 self.dirs.delpath(f)
 del self._map[f]
+if "filefoldmap" in self.__dict__:
+normed = util.normcase(f)
+self.filefoldmap.pop(normed, None)
 self.nonnormalset.discard(f)
 return exists
 



To: mbthomas, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1341: dirstate: move management of nonnormal sets into dirstate map

[mbthomas (Mark Thomas)]

mbthomas updated this revision to Diff 3438.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1341?vs=3341&id=3438

REVISION DETAIL
  https://phab.mercurial-scm.org/D1341

AFFECTED FILES
  mercurial/dirstate.py

CHANGE DETAILS

diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -415,10 +415,6 @@
 self._map.dirs.addpath(f)
 self._dirty = True
 self._updatedfiles.add(f)
-if state != 'n' or mtime == -1:
-self._map.nonnormalset.add(f)
-if size == -2:
-self._map.otherparentset.add(f)
 self._map.addfile(f, state, mode, size, mtime)
 
 def normal(self, f):
@@ -490,7 +486,6 @@
 elif entry[0] == 'n' and entry[2] == -2: # other parent
 size = -2
 self._map.otherparentset.add(f)
-self._map.nonnormalset.add(f)
 self._map.removefile(f, size)
 if size == 0:
 self._map.copymap.pop(f, None)
@@ -506,8 +501,6 @@
 if self._map.dropfile(f):
 self._dirty = True
 self._droppath(f)
-if f in self._map.nonnormalset:
-self._map.nonnormalset.remove(f)
 self._map.copymap.pop(f, None)
 
 def _discoverpath(self, path, normed, ignoremissing, exists, storemap):
@@ -631,12 +624,7 @@
 
 # emulate dropping timestamp in 'parsers.pack_dirstate'
 now = _getfsnow(self._opener)
-dmap = self._map
-for f in self._updatedfiles:
-e = dmap.get(f)
-if e is not None and e[0] == 'n' and e[3] == now:
-dmap.addfile(f, e[0], e[1], e[2], -1)
-self._map.nonnormalset.add(f)
+self._map.clearambiguoustimes(self._updatedfiles, now)
 
 # emulate that all 'dirstate.normal' results are written out
 self._lastnormaltime = 0
@@ -1231,8 +1219,8 @@
 - `dirfoldmap` is a dict mapping normalized directory names to the
   denormalized form that they appear as in the dirstate.
 
-Once instantiated, the nonnormalset, otherparentset, dirs, filefoldmap and
-dirfoldmap views must be maintained by the caller.
+Once instantiated, the dirs, filefoldmap and dirfoldmap views must be
+maintained by the caller.
 '''
 
 def __init__(self, ui, opener, root):
@@ -1297,6 +1285,10 @@
 def addfile(self, f, state, mode, size, mtime):
 """Add a tracked file to the dirstate."""
 self._map[f] = dirstatetuple(state, mode, size, mtime)
+if state != 'n' or mtime == -1:
+self.nonnormalset.add(f)
+if size == -2:
+self.otherparentset.add(f)
 
 def removefile(self, f, size):
 """
@@ -1307,6 +1299,7 @@
 to be more explicit about what that state is.
 """
 self._map[f] = dirstatetuple('r', 0, size, 0)
+self.nonnormalset.add(f)
 
 def dropfile(self, f):
 """
@@ -1316,8 +1309,16 @@
 exists = f in self._map
 if exists:
 del self._map[f]
+self.nonnormalset.discard(f)
 return exists
 
+def clearambiguoustimes(self, files, now):
+for f in files:
+e = self.get(f)
+if e is not None and e[0] == 'n' and e[3] == now:
+self._map[f] = dirstatetuple(e[0], e[1], e[2], -1)
+self.nonnormalset.add(f)
+
 def nonnormalentries(self):
 '''Compute the nonnormal dirstate entries from the dmap'''
 try:



To: mbthomas, #hg-reviewers
Cc: durin42, mbolin, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1342: dirstate: move management of the dirstate dirs into the dirstatemap

[mbthomas (Mark Thomas)]

mbthomas updated this revision to Diff 3439.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1342?vs=3342&id=3439

REVISION DETAIL
  https://phab.mercurial-scm.org/D1342

AFFECTED FILES
  mercurial/dirstate.py

CHANGE DETAILS

diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -387,9 +387,6 @@
 return self._map.copymap
 
 def _droppath(self, f):
-if self[f] not in "?r" and "dirs" in self._map.__dict__:
-self._map.dirs.delpath(f)
-
 if "filefoldmap" in self._map.__dict__:
 normed = util.normcase(f)
 if normed in self._map.filefoldmap:
@@ -411,11 +408,9 @@
 if entry is not None and entry[0] != 'r':
 raise error.Abort(
 _('file %r in dirstate clashes with %r') % (d, f))
-if oldstate in "?r" and "dirs" in self._map.__dict__:
-self._map.dirs.addpath(f)
 self._dirty = True
 self._updatedfiles.add(f)
-self._map.addfile(f, state, mode, size, mtime)
+self._map.addfile(f, oldstate, state, mode, size, mtime)
 
 def normal(self, f):
 '''Mark a file normal and clean.'''
@@ -476,6 +471,7 @@
 '''Mark a file removed.'''
 self._dirty = True
 self._droppath(f)
+oldstate = self[f]
 size = 0
 if self._pl[1] != nullid:
 entry = self._map.get(f)
@@ -486,7 +482,7 @@
 elif entry[0] == 'n' and entry[2] == -2: # other parent
 size = -2
 self._map.otherparentset.add(f)
-self._map.removefile(f, size)
+self._map.removefile(f, oldstate, size)
 if size == 0:
 self._map.copymap.pop(f, None)
 
@@ -498,7 +494,8 @@
 
 def drop(self, f):
 '''Drop a file from the dirstate'''
-if self._map.dropfile(f):
+oldstate = self[f]
+if self._map.dropfile(f, oldstate):
 self._dirty = True
 self._droppath(f)
 self._map.copymap.pop(f, None)
@@ -1219,8 +1216,8 @@
 - `dirfoldmap` is a dict mapping normalized directory names to the
   denormalized form that they appear as in the dirstate.
 
-Once instantiated, the dirs, filefoldmap and dirfoldmap views must be
-maintained by the caller.
+Once instantiated, the filefoldmap and dirfoldmap views must be maintained
+by the caller.
 '''
 
 def __init__(self, ui, opener, root):
@@ -1282,32 +1279,38 @@
 """Loads the underlying data, if it's not already loaded"""
 self._map
 
-def addfile(self, f, state, mode, size, mtime):
+def addfile(self, f, oldstate, state, mode, size, mtime):
 """Add a tracked file to the dirstate."""
+if oldstate in "?r" and "dirs" in self.__dict__:
+self.dirs.addpath(f)
 self._map[f] = dirstatetuple(state, mode, size, mtime)
 if state != 'n' or mtime == -1:
 self.nonnormalset.add(f)
 if size == -2:
 self.otherparentset.add(f)
 
-def removefile(self, f, size):
+def removefile(self, f, oldstate, size):
 """
 Mark a file as removed in the dirstate.
 
 The `size` parameter is used to store sentinel values that indicate
 the file's previous state.  In the future, we should refactor this
 to be more explicit about what that state is.
 """
+if oldstate not in "?r" and "dirs" in self.__dict__:
+self.dirs.delpath(f)
 self._map[f] = dirstatetuple('r', 0, size, 0)
 self.nonnormalset.add(f)
 
-def dropfile(self, f):
+def dropfile(self, f, oldstate):
 """
 Remove a file from the dirstate.  Returns True if the file was
 previously recorded.
 """
 exists = f in self._map
 if exists:
+if oldstate != "r" and "dirs" in self.__dict__:
+self.dirs.delpath(f)
 del self._map[f]
 self.nonnormalset.discard(f)
 return exists



To: mbthomas, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1344: dirstate: remove _droppath method

[mbthomas (Mark Thomas)]

mbthomas updated this revision to Diff 3441.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1344?vs=3344&id=3441

REVISION DETAIL
  https://phab.mercurial-scm.org/D1344

AFFECTED FILES
  mercurial/dirstate.py

CHANGE DETAILS

diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -386,9 +386,6 @@
 def copies(self):
 return self._map.copymap
 
-def _droppath(self, f):
-self._updatedfiles.add(f)
-
 def _addpath(self, f, state, mode, size, mtime):
 oldstate = self[f]
 if state == 'a' or oldstate == 'r':
@@ -465,7 +462,6 @@
 def remove(self, f):
 '''Mark a file removed.'''
 self._dirty = True
-self._droppath(f)
 oldstate = self[f]
 size = 0
 if self._pl[1] != nullid:
@@ -477,6 +473,7 @@
 elif entry[0] == 'n' and entry[2] == -2: # other parent
 size = -2
 self._map.otherparentset.add(f)
+self._updatedfiles.add(f)
 self._map.removefile(f, oldstate, size)
 if size == 0:
 self._map.copymap.pop(f, None)
@@ -492,7 +489,7 @@
 oldstate = self[f]
 if self._map.dropfile(f, oldstate):
 self._dirty = True
-self._droppath(f)
+self._updatedfiles.add(f)
 self._map.copymap.pop(f, None)
 
 def _discoverpath(self, path, normed, ignoremissing, exists, storemap):



To: mbthomas, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1380: dirstate: document dirstatemap interface

[mbthomas (Mark Thomas)]

mbthomas created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1380

AFFECTED FILES
  mercurial/dirstate.py

CHANGE DETAILS

diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -127,8 +127,7 @@
 
 @propertycache
 def _map(self):
-'''Return the dirstate contents as a map from filename to
-(state, mode, size, time).'''
+'''Return the dirstate contents (see documentation for dirstatemap).'''
 self._map = dirstatemap(self._ui, self._opener, self._root)
 return self._map
 
@@ -1196,6 +1195,46 @@
 self._opener.unlink(backupname)
 
 class dirstatemap(object):
+'''
+Dirstate contents
+
+The dirstate contains the following state:
+
+- `identity` is the identity of the dirstate file, which can be used to
+  detect when changes have occurred to the dirstate file.
+
+- `parents` is a pair containing the parents of the working copy. The
+  parents are updated by calling `setparents`.
+
+- the state map maps filenames to tuples of (state, mode, size, mtime),
+  where state is a single character representing 'normal', 'added',
+  'removed', or 'merged'. It is accessed by treating the dirstate as a
+  dict.
+
+- `copymap` is a dict mapping filenames to the filename they were copied
+  from.
+
+The dirstate also provides the following views onto the state:
+
+- `nonnormalset` is a set of the filenames that have state other
+  than 'normal', or are normal but have an mtime of -1 ('normallookup').
+
+- `otherparentset` is a set of the filenames that are marked as coming
+  from the second parent when the dirstate is currently being merged.
+
+- `dirs` is a set-like object containing all the directories that contain
+  files in the dirstate, excluding any files that are marked as removed.
+
+- `filefoldmap` is a dict mapping normalized filenames to the denormalized
+  form that they appear as in the dirstate.
+
+- `dirfoldmap` is a dict mapping normalized directory names to the
+  denormalized form that they appear as in the dirstate.
+
+Once instantiated, the nonnormalset, otherparentset, dirs, filefoldmap and
+dirfoldmap views must be maintained by the caller.
+'''
+
 def __init__(self, ui, opener, root):
 self._ui = ui
 self._opener = opener



To: mbthomas, #hg-reviewers
Cc: mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1340: dirstate: add explicit methods for modifying dirstate

[mbthomas (Mark Thomas)]

mbthomas updated this revision to Diff 3437.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1340?vs=3340&id=3437

REVISION DETAIL
  https://phab.mercurial-scm.org/D1340

AFFECTED FILES
  mercurial/dirstate.py

CHANGE DETAILS

diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -415,11 +415,11 @@
 self._map.dirs.addpath(f)
 self._dirty = True
 self._updatedfiles.add(f)
-self._map[f] = dirstatetuple(state, mode, size, mtime)
 if state != 'n' or mtime == -1:
 self._map.nonnormalset.add(f)
 if size == -2:
 self._map.otherparentset.add(f)
+self._map.addfile(f, state, mode, size, mtime)
 
 def normal(self, f):
 '''Mark a file normal and clean.'''
@@ -490,8 +490,8 @@
 elif entry[0] == 'n' and entry[2] == -2: # other parent
 size = -2
 self._map.otherparentset.add(f)
-self._map[f] = dirstatetuple('r', 0, size, 0)
 self._map.nonnormalset.add(f)
+self._map.removefile(f, size)
 if size == 0:
 self._map.copymap.pop(f, None)
 
@@ -503,10 +503,9 @@
 
 def drop(self, f):
 '''Drop a file from the dirstate'''
-if f in self._map:
+if self._map.dropfile(f):
 self._dirty = True
 self._droppath(f)
-del self._map[f]
 if f in self._map.nonnormalset:
 self._map.nonnormalset.remove(f)
 self._map.copymap.pop(f, None)
@@ -636,7 +635,7 @@
 for f in self._updatedfiles:
 e = dmap.get(f)
 if e is not None and e[0] == 'n' and e[3] == now:
-dmap[f] = dirstatetuple(e[0], e[1], e[2], -1)
+dmap.addfile(f, e[0], e[1], e[2], -1)
 self._map.nonnormalset.add(f)
 
 # emulate that all 'dirstate.normal' results are written out
@@ -1208,8 +1207,9 @@
 
 - the state map maps filenames to tuples of (state, mode, size, mtime),
   where state is a single character representing 'normal', 'added',
-  'removed', or 'merged'. It is accessed by treating the dirstate as a
-  dict.
+  'removed', or 'merged'. It is read by treating the dirstate as a
+  dict.  File state is updated by calling the `addfile`, `removefile` and
+  `dropfile` methods.
 
 - `copymap` is a dict mapping filenames to the filename they were copied
   from.
@@ -1284,22 +1284,40 @@
 def __contains__(self, key):
 return key in self._map
 
-def __setitem__(self, key, value):
-self._map[key] = value
-
 def __getitem__(self, key):
 return self._map[key]
 
-def __delitem__(self, key):
-del self._map[key]
-
 def keys(self):
 return self._map.keys()
 
 def preload(self):
 """Loads the underlying data, if it's not already loaded"""
 self._map
 
+def addfile(self, f, state, mode, size, mtime):
+"""Add a tracked file to the dirstate."""
+self._map[f] = dirstatetuple(state, mode, size, mtime)
+
+def removefile(self, f, size):
+"""
+Mark a file as removed in the dirstate.
+
+The `size` parameter is used to store sentinel values that indicate
+the file's previous state.  In the future, we should refactor this
+to be more explicit about what that state is.
+"""
+self._map[f] = dirstatetuple('r', 0, size, 0)
+
+def dropfile(self, f):
+"""
+Remove a file from the dirstate.  Returns True if the file was
+previously recorded.
+"""
+exists = f in self._map
+if exists:
+del self._map[f]
+return exists
+
 def nonnormalentries(self):
 '''Compute the nonnormal dirstate entries from the dmap'''
 try:
@@ -1429,8 +1447,6 @@
 # Avoid excess attribute lookups by fast pathing certain checks
 self.__contains__ = self._map.__contains__
 self.__getitem__ = self._map.__getitem__
-self.__setitem__ = self._map.__setitem__
-self.__delitem__ = self._map.__delitem__
 self.get = self._map.get
 
 def write(self, st, now):



To: mbthomas, #hg-reviewers, durin42
Cc: durin42, mbolin, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1358: remotenames: store journal entry for bookmarks if journal is loaded

[pulkit (Pulkit Goyal)]

pulkit added a comment.


  In https://phab.mercurial-scm.org/D1358#22836, @dlax wrote:
  
  > The state of this stack is not quite clear: there are abandoned revisions 
and the first changeset (introducing "mercurial/remotenames.py" file) seems to 
be missing.
  
  
  Ah, I will resend the series maybe with new differentials. Thanks!

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1358

To: pulkit, #hg-reviewers
Cc: dlax, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 3 of 6 V2] test-pattern: register current the bundlecaps string

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1509860620 -3600
#  Sun Nov 05 06:43:40 2017 +0100
# Node ID 16289f5358a17c0ee629dfa6910b00bb8dec7fe9
# Parent  15aa75b64e1dff38fcce4a8f283daf0ce37a1261
# EXP-Topic better-substitute
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
16289f5358a1
test-pattern: register current the bundlecaps string

The bundle capabilites sent with every getbundle commands. Every time the
protocol is updated, that string is altered. We get that string replace by
$USUAL_BUNDLE_CAPS$ so that we only have to change the substitution whenever
this happens.

diff --git a/tests/common-pattern.py b/tests/common-pattern.py
--- a/tests/common-pattern.py
+++ b/tests/common-pattern.py
@@ -6,4 +6,17 @@ substitutions = [
 (br'zstd,zlib,none,bzip2',
  br'$USUAL_COMPRESSIONS$'
 ),
+# capabilities sent through http
+(br'bundlecaps=HG20%2Cbundle2%3DHG20%250A'
+ br'changegroup%253D01%252C02%250A'
+ br'digests%253Dmd5%252Csha1%252Csha512%250A'
+ br'error%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250A'
+ br'hgtagsfnodes%250A'
+ br'listkeys%250A'
+ br'phases%253Dheads%250A'
+ br'pushkey%250A'
+ br'remote-changegroup%253Dhttp%252Chttps',
+ # (the replacement patterns)
+ br'$USUAL_BUNDLE_CAPS$'
+),
 ]
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 4 of 6 V2] test-pattern: register the current the bundle2 capabilities string

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1510545585 -3600
#  Mon Nov 13 04:59:45 2017 +0100
# Node ID 8f0e6aaf04b6bdd6535b6bc03008538c95085105
# Parent  16289f5358a17c0ee629dfa6910b00bb8dec7fe9
# EXP-Topic better-substitute
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
8f0e6aaf04b6
test-pattern: register the current the bundle2 capabilities string

The bundle capabilites are sent with every getbundle ssh connection. Every time
the protocol is updated, that string is altered. We get the part about bundle2
string replaced by $USUAL_BUNDLE2_CAPS$ so that we only have to change the
substitution whenever this happens.

diff --git a/tests/common-pattern.py b/tests/common-pattern.py
--- a/tests/common-pattern.py
+++ b/tests/common-pattern.py
@@ -19,4 +19,17 @@ substitutions = [
  # (the replacement patterns)
  br'$USUAL_BUNDLE_CAPS$'
 ),
+# bundle2 capabilities sent through ssh
+(br'bundle2=HG20%0A'
+ br'changegroup%3D01%2C02%0A'
+ br'digests%3Dmd5%2Csha1%2Csha512%0A'
+ br'error%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0A'
+ br'hgtagsfnodes%0A'
+ br'listkeys%0A'
+ br'phases%3Dheads%0A'
+ br'pushkey%0A'
+ br'remote-changegroup%3Dhttp%2Chttps',
+ # (replacement patterns)
+ br'$USUAL_BUNDLE2_CAPS$'
+),
 ]
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 2 of 6 V2] test-pattern: substitute common compression list

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1509860498 -3600
#  Sun Nov 05 06:41:38 2017 +0100
# Node ID 15aa75b64e1dff38fcce4a8f283daf0ce37a1261
# Parent  a7d1303d69b73deac107d78ed37506f662ec661b
# EXP-Topic better-substitute
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
15aa75b64e1d
test-pattern: substitute common compression list

The compression list as to be matched with a glob because zstd might not be part
of the option. By using a substitution for these, we won't have to re-glob them
over and over.

diff --git a/tests/common-pattern.py b/tests/common-pattern.py
new file mode 100644
--- /dev/null
+++ b/tests/common-pattern.py
@@ -0,0 +1,9 @@
+# common patterns in test at can safely be replaced
+from __future__ import absolute_import
+
+substitutions = [
+# list of possible compressions
+(br'zstd,zlib,none,bzip2',
+ br'$USUAL_COMPRESSIONS$'
+),
+]
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 1 of 6 V2] run-tests: allow to register any arbitrary pattern for replacement

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1509860067 -3600
#  Sun Nov 05 06:34:27 2017 +0100
# Node ID a7d1303d69b73deac107d78ed37506f662ec661b
# Parent  5d4369079c861d8fb01ab4505bf7d5911cf7a6e1
# EXP-Topic better-substitute
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
a7d1303d69b7
run-tests: allow to register any arbitrary pattern for replacement

We add a 'common-pattern.py' file that allow to define extra pattern. This seems
a cleaner approach than editing the 'run-test.py' file over and over. In
addition allowing arbitrary pattern registration will also help extension.

The format used is a python file is picked out of convenience defining a list of
tuple in 'substitutions' variable. This is picked out of convenience since it is
dead simple to implement.

The end goal is to register more pattern for Mercurial test. There are multiple
common patterns that change over time. That impact is annoying. Using pattern
emplacement for them would be handy.

The next patches will define all the needed patterns and the last patch will
mass-update the tests outputs as it was easier to do in a single pass.

diff --git a/tests/run-tests.py b/tests/run-tests.py
--- a/tests/run-tests.py
+++ b/tests/run-tests.py
@@ -968,6 +968,13 @@ class Test(unittest.TestCase):
 ]
 r.append((self._escapepath(self._testtmp), b'$TESTTMP'))
 
+testdir = os.path.dirname(self.path)
+replacementfile = os.path.join(testdir, 'common-pattern.py')
+
+if os.path.exists(replacementfile):
+data = {}
+execfile(replacementfile, data)
+r.extend(data.get('substitutions', ()))
 return r
 
 def _escapepath(self, p):
diff --git a/tests/test-run-tests.t b/tests/test-run-tests.t
--- a/tests/test-run-tests.t
+++ b/tests/test-run-tests.t
@@ -1503,3 +1503,43 @@ Test cases in .t files
   # Ran 2 tests, 0 skipped, 1 failed.
   python hash seed: * (glob)
   [1]
+
+Test automatic pattern replacement
+
+  $ cat << EOF >> common-pattern.py
+  > substitutions = [
+  > (br'foo-(.*)\\b',
+  >  br'\$XXX=\\1\$'),
+  > (br'bar\\n',
+  >  br'\$YYY$\\n'),
+  > ]
+  > EOF
+
+  $ cat << EOF >> test-substitution.t
+  >   $ echo foo-12
+  >   \$XXX=12$
+  >   $ echo foo-42
+  >   \$XXX=42$
+  >   $ echo bar prior
+  >   bar prior
+  >   $ echo lastbar
+  >   last\$YYY$
+  >   $ echo foo-bar foo-baz
+  > EOF
+
+  $ rt test-substitution.t
+  
+  --- $TESTTMP/anothertests/cases/test-substitution.t
+  +++ $TESTTMP/anothertests/cases/test-substitution.t.err
+  @@ -7,3 +7,4 @@
+ $ echo lastbar
+ last$YYY$
+ $ echo foo-bar foo-baz
+  +  $XXX=bar foo-baz$
+  
+  ERROR: test-substitution.t output changed
+  !
+  Failed test-substitution.t: output changed
+  # Ran 1 tests, 0 skipped, 1 failed.
+  python hash seed: * (glob)
+  [1]
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 5 of 6 V2] test-pattern: substitute the HTTP log timestamp too

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1509866633 -3600
#  Sun Nov 05 08:23:53 2017 +0100
# Node ID c06ed80fc1063905dc6a711fc32c88d5913b0db6
# Parent  8f0e6aaf04b6bdd6535b6bc03008538c95085105
# EXP-Topic better-substitute
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
c06ed80fc106
test-pattern: substitute the HTTP log timestamp too

We add a pattern matching the infamous timestamp in http log. Now, we should be
able to have change appears in https log without having to re-glob the whole
thing over and over.

diff --git a/tests/common-pattern.py b/tests/common-pattern.py
--- a/tests/common-pattern.py
+++ b/tests/common-pattern.py
@@ -32,4 +32,8 @@ substitutions = [
  # (replacement patterns)
  br'$USUAL_BUNDLE2_CAPS$'
 ),
+# HTTP log dates
+(br' - - \[\d\d/.../2\d\d\d \d\d:\d\d:\d\d] "GET',
+ br' - - [$LOGDATE$] "GET'
+),
 ]
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 6 of 6 V2] test-pattern: actually update tests using the patterns

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1509866592 -3600
#  Sun Nov 05 08:23:12 2017 +0100
# Node ID 715948f2256fa181c4f3730825492f9da0f74b2e
# Parent  c06ed80fc1063905dc6a711fc32c88d5913b0db6
# EXP-Topic better-substitute
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
715948f2256f
test-pattern: actually update tests using the patterns

We mass update the tests now. This will help the next soul touching the http
protocol.

diff --git a/tests/test-clonebundles.t b/tests/test-clonebundles.t
--- a/tests/test-clonebundles.t
+++ b/tests/test-clonebundles.t
@@ -32,8 +32,8 @@ Missing manifest should not result in se
 
   $ cat server/access.log
   * - - [*] "GET /?cmd=capabilities HTTP/1.1" 200 - (glob)
-  * - - [*] "GET /?cmd=batch HTTP/1.1" 200 - 
x-hgarg-1:cmds=heads+%3Bknown+nodes%3D x-hgproto-1:0.1 0.2 
comp=*zlib,none,bzip2 (glob)
-  * - - [*] "GET /?cmd=getbundle HTTP/1.1" 200 - 
x-hgarg-1:bundlecaps=HG20%2Cbundle2%3DHG20%250Achangegroup%253D01%252C02%250Adigests%253Dmd5%252Csha1%252Csha512%250Aerror%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250Ahgtagsfnodes%250Alistkeys%250Aphases%253Dheads%250Apushkey%250Aremote-changegroup%253Dhttp%252Chttps&cg=1&common=&heads=aaff8d2ffbbf07a46dd1f05d8ae7877e3f56e2a2&listkeys=bookmarks&phases=1
 x-hgproto-1:0.1 0.2 comp=*zlib,none,bzip2 (glob)
+  $LOCALIP - - [$LOGDATE$] "GET /?cmd=batch HTTP/1.1" 200 - 
x-hgarg-1:cmds=heads+%3Bknown+nodes%3D x-hgproto-1:0.1 0.2 
comp=$USUAL_COMPRESSIONS$
+  $LOCALIP - - [$LOGDATE$] "GET /?cmd=getbundle HTTP/1.1" 200 - 
x-hgarg-1:$USUAL_BUNDLE_CAPS$&cg=1&common=&heads=aaff8d2ffbbf07a46dd1f05d8ae7877e3f56e2a2&listkeys=bookmarks&phases=1
 x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
 
 Empty manifest file results in retrieval
 (the extension only checks if the manifest file exists)
diff --git a/tests/test-debugcommands.t b/tests/test-debugcommands.t
--- a/tests/test-debugcommands.t
+++ b/tests/test-debugcommands.t
@@ -162,7 +162,7 @@ Test debugcapabilities command:
   $ hg debugcapabilities ./debugrevlog/
   Main capabilities:
 branchmap
-
bundle2=HG20%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Aphases%3Dheads%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps
+$USUAL_BUNDLE2_CAPS$
 getbundle
 known
 lookup
diff --git a/tests/test-getbundle.t b/tests/test-getbundle.t
--- a/tests/test-getbundle.t
+++ b/tests/test-getbundle.t
@@ -264,9 +264,9 @@ Verify we hit the HTTP server:
 
   $ cat access.log
   * - - [*] "GET /?cmd=capabilities HTTP/1.1" 200 - (glob)
-  * - - [*] "GET /?cmd=getbundle HTTP/1.1" 200 - x-hgproto-1:0.1 0.2 
comp=*zlib,none,bzip2 (glob)
+  $LOCALIP - - [$LOGDATE$] "GET /?cmd=getbundle HTTP/1.1" 200 - 
x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   * - - [*] "GET /?cmd=capabilities HTTP/1.1" 200 - (glob)
-  * - - [*] "GET /?cmd=getbundle HTTP/1.1" 200 - 
x-hgarg-1:common=700b7e19db54103633c4bf4a6a6b6d55f4d50c03+d5f6e1ea452285324836a49d7d3c2a63cfed1d31&heads=13c0170174366b441dc68e8e33757232fa744458+bac16991d12ff45f9dc43c52da1946dfadb83e80
 x-hgproto-1:0.1 0.2 comp=*zlib,none,bzip2 (glob)
+  $LOCALIP - - [$LOGDATE$] "GET /?cmd=getbundle HTTP/1.1" 200 - 
x-hgarg-1:common=700b7e19db54103633c4bf4a6a6b6d55f4d50c03+d5f6e1ea452285324836a49d7d3c2a63cfed1d31&heads=13c0170174366b441dc68e8e33757232fa744458+bac16991d12ff45f9dc43c52da1946dfadb83e80
 x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
 
   $ cat error.log
 
diff --git a/tests/test-hgweb-commands.t b/tests/test-hgweb-commands.t
--- a/tests/test-hgweb-commands.t
+++ b/tests/test-hgweb-commands.t
@@ -1926,7 +1926,7 @@ capabilities
   $ get-with-headers.py $LOCALIP:$HGPORT '?cmd=capabilities'; echo
   200 Script output follows
   
-  lookup changegroupsubset branchmap pushkey known getbundle unbundlehash 
batch 
bundle2=HG20%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Aphases%3Dheads%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps
 unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx 
compression=*zlib (glob)
+  lookup changegroupsubset branchmap pushkey known getbundle unbundlehash 
batch $USUAL_BUNDLE2_CAPS$ unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 
httpmediatype=0.1rx,0.1tx,0.2tx compression=zstd,zlib
 
 heads
 
@@ -2174,7 +2174,7 @@ capabilities
   batch
   stream-preferred
   streamreqs=generaldelta,revlogv1
-  
bundle2=HG20%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Aphases%3Dheads%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps
+  $USUAL_BUNDLE2_CAPS$
   unbundle=HG10GZ,HG10BZ,HG10UN
   httpheader=1024
   httpmediatype=0.1rx,0.1tx,0.2tx
diff --

Re: [PATCH 11 of 11] test-pattern: actually update tests using the patterns

[Boris Feld]

Sorry I included some obsolete changesets by mistake, I will resend a
new series.

On Mon, 2017-11-13 at 12:05 +0100, Boris Feld wrote:
> # HG changeset patch
> # User Boris Feld 
> # Date 1509866592 -3600
> #  Sun Nov 05 08:23:12 2017 +0100
> # Node ID 715948f2256fa181c4f3730825492f9da0f74b2e
> # Parent  c06ed80fc1063905dc6a711fc32c88d5913b0db6
> # EXP-Topic better-substitute
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> #  hg pull https://bitbucket.org/octobus/mercurial-devel/
>  -r 715948f2256f
> test-pattern: actually update tests using the patterns
> 
> We mass update the tests now. This will help the next soul touching
> the http
> protocol.
> 
> diff --git a/tests/test-clonebundles.t b/tests/test-clonebundles.t
> --- a/tests/test-clonebundles.t
> +++ b/tests/test-clonebundles.t
> @@ -32,8 +32,8 @@ Missing manifest should not result in se
>  
>    $ cat server/access.log
>    * - - [*] "GET /?cmd=capabilities HTTP/1.1" 200 - (glob)
> -  * - - [*] "GET /?cmd=batch HTTP/1.1" 200 - x-hgarg-
> 1:cmds=heads+%3Bknown+nodes%3D x-hgproto-1:0.1 0.2
> comp=*zlib,none,bzip2 (glob)
> -  * - - [*] "GET /?cmd=getbundle HTTP/1.1" 200 - x-hgarg-
> 1:bundlecaps=HG20%2Cbundle2%3DHG20%250Achangegroup%253D01%252C02%250A
> digests%253Dmd5%252Csha1%252Csha512%250Aerror%253Dabort%252Cunsupport
> edcontent%252Cpushraced%252Cpushkey%250Ahgtagsfnodes%250Alistkeys%250
> Aphases%253Dheads%250Apushkey%250Aremote-
> changegroup%253Dhttp%252Chttps&cg=1&common=00
> 00&heads=aaff8d2ffbbf07a46dd1f05d8ae7877e3f56e2a2&listkey
> s=bookmarks&phases=1 x-hgproto-1:0.1 0.2 comp=*zlib,none,bzip2 (glob)
> +  $LOCALIP - - [$LOGDATE$] "GET /?cmd=batch HTTP/1.1" 200 - x-hgarg-
> 1:cmds=heads+%3Bknown+nodes%3D x-hgproto-1:0.1 0.2
> comp=$USUAL_COMPRESSIONS$
> +  $LOCALIP - - [$LOGDATE$] "GET /?cmd=getbundle HTTP/1.1" 200 - x-
> hgarg-
> 1:$USUAL_BUNDLE_CAPS$&cg=1&common=000
> 0&heads=aaff8d2ffbbf07a46dd1f05d8ae7877e3f56e2a2&listkeys=bookmar
> ks&phases=1 x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
>  
>  Empty manifest file results in retrieval
>  (the extension only checks if the manifest file exists)
> diff --git a/tests/test-debugcommands.t b/tests/test-debugcommands.t
> --- a/tests/test-debugcommands.t
> +++ b/tests/test-debugcommands.t
> @@ -162,7 +162,7 @@ Test debugcapabilities command:
>    $ hg debugcapabilities ./debugrevlog/
>    Main capabilities:
>  branchmap
> -bundle2=HG20%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csh
> a512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgt
> agsfnodes%0Alistkeys%0Aphases%3Dheads%0Apushkey%0Aremote-
> changegroup%3Dhttp%2Chttps
> +$USUAL_BUNDLE2_CAPS$
>  getbundle
>  known
>  lookup
> diff --git a/tests/test-getbundle.t b/tests/test-getbundle.t
> --- a/tests/test-getbundle.t
> +++ b/tests/test-getbundle.t
> @@ -264,9 +264,9 @@ Verify we hit the HTTP server:
>  
>    $ cat access.log
>    * - - [*] "GET /?cmd=capabilities HTTP/1.1" 200 - (glob)
> -  * - - [*] "GET /?cmd=getbundle HTTP/1.1" 200 - x-hgproto-1:0.1 0.2
> comp=*zlib,none,bzip2 (glob)
> +  $LOCALIP - - [$LOGDATE$] "GET /?cmd=getbundle HTTP/1.1" 200 - x-
> hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
>    * - - [*] "GET /?cmd=capabilities HTTP/1.1" 200 - (glob)
> -  * - - [*] "GET /?cmd=getbundle HTTP/1.1" 200 - x-hgarg-
> 1:common=700b7e19db54103633c4bf4a6a6b6d55f4d50c03+d5f6e1ea45228532483
> 6a49d7d3c2a63cfed1d31&heads=13c0170174366b441dc68e8e33757232fa744458+
> bac16991d12ff45f9dc43c52da1946dfadb83e80 x-hgproto-1:0.1 0.2
> comp=*zlib,none,bzip2 (glob)
> +  $LOCALIP - - [$LOGDATE$] "GET /?cmd=getbundle HTTP/1.1" 200 - x-
> hgarg-
> 1:common=700b7e19db54103633c4bf4a6a6b6d55f4d50c03+d5f6e1ea45228532483
> 6a49d7d3c2a63cfed1d31&heads=13c0170174366b441dc68e8e33757232fa744458+
> bac16991d12ff45f9dc43c52da1946dfadb83e80 x-hgproto-1:0.1 0.2
> comp=$USUAL_COMPRESSIONS$
>  
>    $ cat error.log
>  
> diff --git a/tests/test-hgweb-commands.t b/tests/test-hgweb-
> commands.t
> --- a/tests/test-hgweb-commands.t
> +++ b/tests/test-hgweb-commands.t
> @@ -1926,7 +1926,7 @@ capabilities
>    $ get-with-headers.py $LOCALIP:$HGPORT '?cmd=capabilities'; echo
>    200 Script output follows
>    
> -  lookup changegroupsubset branchmap pushkey known getbundle
> unbundlehash batch
> bundle2=HG20%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%
> 0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfn
> odes%0Alistkeys%0Aphases%3Dheads%0Apushkey%0Aremote-
> changegroup%3Dhttp%2Chttps unbundle=HG10GZ,HG10BZ,HG10UN
> httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx compression=*zlib
> (glob)
> +  lookup changegroupsubset branchmap pushkey known getbundle
> unbundlehash batch $USUAL_BUNDLE2_CAPS$ unbundle=HG10GZ,HG10BZ,HG10UN
> httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx compression=zstd,zlib
>  
>  heads
>  
> @@ -2174,7 +2174,7 @@ capabilities
>    batch
>    stream-preferred
>    strea

[PATCH 09 of 11] test-pattern: register the current the bundle2 capabilities string

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1510545585 -3600
#  Mon Nov 13 04:59:45 2017 +0100
# Node ID 8f0e6aaf04b6bdd6535b6bc03008538c95085105
# Parent  16289f5358a17c0ee629dfa6910b00bb8dec7fe9
# EXP-Topic better-substitute
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
8f0e6aaf04b6
test-pattern: register the current the bundle2 capabilities string

The bundle capabilites are sent with every getbundle ssh connection. Every time
the protocol is updated, that string is altered. We get the part about bundle2
string replaced by $USUAL_BUNDLE2_CAPS$ so that we only have to change the
substitution whenever this happens.

diff --git a/tests/common-pattern.py b/tests/common-pattern.py
--- a/tests/common-pattern.py
+++ b/tests/common-pattern.py
@@ -19,4 +19,17 @@ substitutions = [
  # (the replacement patterns)
  br'$USUAL_BUNDLE_CAPS$'
 ),
+# bundle2 capabilities sent through ssh
+(br'bundle2=HG20%0A'
+ br'changegroup%3D01%2C02%0A'
+ br'digests%3Dmd5%2Csha1%2Csha512%0A'
+ br'error%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0A'
+ br'hgtagsfnodes%0A'
+ br'listkeys%0A'
+ br'phases%3Dheads%0A'
+ br'pushkey%0A'
+ br'remote-changegroup%3Dhttp%2Chttps',
+ # (replacement patterns)
+ br'$USUAL_BUNDLE2_CAPS$'
+),
 ]
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 04 of 11] test-pattern: register the current the bundle2 capabilities string

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1510545585 -3600
#  Mon Nov 13 04:59:45 2017 +0100
# Node ID 5714516a59c40bd83187bdb2a1615710a6cf
# Parent  6e8aa2adeceaa523f214b79970df6471d728faf9
# EXP-Topic better-substitute
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
5714516a
test-pattern: register the current the bundle2 capabilities string

The bundle capabilites are sent with every getbundle ssh connection. Every time
the protocol is updated, that string is altered. We get the part about bundle2
string replaced by $USUAL_BUNDLE2_CAPS$ so that we only have to change the
substitution whenever this happens.

diff --git a/tests/common-pattern.py b/tests/common-pattern.py
--- a/tests/common-pattern.py
+++ b/tests/common-pattern.py
@@ -19,4 +19,17 @@ substitutions = [
  # (the replacement patterns)
  br'$USUAL_BUNDLE_CAPS$'
 ),
+# bundle2 capabilities sent through ssh
+(br'bundle2=HG20%0A'
+ br'changegroup%3D01%2C02%0A'
+ br'digests%3Dmd5%2Csha1%2Csha512%0A'
+ br'error%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0A'
+ br'hgtagsfnodes%0A'
+ br'listkeys%0A'
+ br'phases%3Dheads%0A'
+ br'pushkey%0A'
+ br'remote-changegroup%3Dhttp%2Chttps',
+ # (replacement patterns)
+ br'$USUAL_BUNDLE2_CAPS$'
+),
 ]
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 05 of 11] test-pattern: substitute the HTTP log timestamp too

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1509866633 -3600
#  Sun Nov 05 08:23:53 2017 +0100
# Node ID bacfe13eb9b21a13c8144ae4d2337415ea93f2e3
# Parent  5714516a59c40bd83187bdb2a1615710a6cf
# EXP-Topic better-substitute
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
bacfe13eb9b2
test-pattern: substitute the HTTP log timestamp too

We add a pattern matching the infamous timestamp in http log. Now, we should be
able to have change appears in https log without having to re-glob the whole
thing over and over.

diff --git a/tests/common-pattern.py b/tests/common-pattern.py
--- a/tests/common-pattern.py
+++ b/tests/common-pattern.py
@@ -32,4 +32,8 @@ substitutions = [
  # (replacement patterns)
  br'$USUAL_BUNDLE2_CAPS$'
 ),
+# HTTP log dates
+(br' - - \[\d\d/.../2\d\d\d \d\d:\d\d:\d\d] "GET',
+ br' - - [$LOGDATE$] "GET'
+),
 ]
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 06 of 11] run-tests: allow to register any arbitrary pattern for replacement

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1509860067 -3600
#  Sun Nov 05 06:34:27 2017 +0100
# Node ID a7d1303d69b73deac107d78ed37506f662ec661b
# Parent  5d4369079c861d8fb01ab4505bf7d5911cf7a6e1
# EXP-Topic better-substitute
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
a7d1303d69b7
run-tests: allow to register any arbitrary pattern for replacement

We add a 'common-pattern.py' file that allow to define extra pattern. This seems
a cleaner approach than editing the 'run-test.py' file over and over. In
addition allowing arbitrary pattern registration will also help extension.

The format used is a python file is picked out of convenience defining a list of
tuple in 'substitutions' variable. This is picked out of convenience since it is
dead simple to implement.

The end goal is to register more pattern for Mercurial test. There are multiple
common patterns that change over time. That impact is annoying. Using pattern
emplacement for them would be handy.

The next patches will define all the needed patterns and the last patch will
mass-update the tests outputs as it was easier to do in a single pass.

diff --git a/tests/run-tests.py b/tests/run-tests.py
--- a/tests/run-tests.py
+++ b/tests/run-tests.py
@@ -968,6 +968,13 @@ class Test(unittest.TestCase):
 ]
 r.append((self._escapepath(self._testtmp), b'$TESTTMP'))
 
+testdir = os.path.dirname(self.path)
+replacementfile = os.path.join(testdir, 'common-pattern.py')
+
+if os.path.exists(replacementfile):
+data = {}
+execfile(replacementfile, data)
+r.extend(data.get('substitutions', ()))
 return r
 
 def _escapepath(self, p):
diff --git a/tests/test-run-tests.t b/tests/test-run-tests.t
--- a/tests/test-run-tests.t
+++ b/tests/test-run-tests.t
@@ -1503,3 +1503,43 @@ Test cases in .t files
   # Ran 2 tests, 0 skipped, 1 failed.
   python hash seed: * (glob)
   [1]
+
+Test automatic pattern replacement
+
+  $ cat << EOF >> common-pattern.py
+  > substitutions = [
+  > (br'foo-(.*)\\b',
+  >  br'\$XXX=\\1\$'),
+  > (br'bar\\n',
+  >  br'\$YYY$\\n'),
+  > ]
+  > EOF
+
+  $ cat << EOF >> test-substitution.t
+  >   $ echo foo-12
+  >   \$XXX=12$
+  >   $ echo foo-42
+  >   \$XXX=42$
+  >   $ echo bar prior
+  >   bar prior
+  >   $ echo lastbar
+  >   last\$YYY$
+  >   $ echo foo-bar foo-baz
+  > EOF
+
+  $ rt test-substitution.t
+  
+  --- $TESTTMP/anothertests/cases/test-substitution.t
+  +++ $TESTTMP/anothertests/cases/test-substitution.t.err
+  @@ -7,3 +7,4 @@
+ $ echo lastbar
+ last$YYY$
+ $ echo foo-bar foo-baz
+  +  $XXX=bar foo-baz$
+  
+  ERROR: test-substitution.t output changed
+  !
+  Failed test-substitution.t: output changed
+  # Ran 1 tests, 0 skipped, 1 failed.
+  python hash seed: * (glob)
+  [1]
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 01 of 11] run-tests: allow to register any arbitrary pattern for replacement

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1509860067 -3600
#  Sun Nov 05 06:34:27 2017 +0100
# Node ID fda6ff97ddead2610694668042f58f892130d982
# Parent  5d4369079c861d8fb01ab4505bf7d5911cf7a6e1
# EXP-Topic better-substitute
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
fda6ff97ddea
run-tests: allow to register any arbitrary pattern for replacement

We add a 'common-pattern.py' file that allow to define extra pattern. This seems
a cleaner approach than editing the 'run-test.py' file over and over. In
addition allowing arbitrary pattern registration will also help extension.

The format used is a python file is picked out of convenience defining a list of
tuple in 'substitutions' variable. This is picked out of convenience since it is
dead simple to implement.

The end goal is to register more pattern for Mercurial test. There are multiple
common patterns that change over time. That impact is annoying. Using pattern
emplacement for them would be handy.

diff --git a/tests/run-tests.py b/tests/run-tests.py
--- a/tests/run-tests.py
+++ b/tests/run-tests.py
@@ -968,6 +968,13 @@ class Test(unittest.TestCase):
 ]
 r.append((self._escapepath(self._testtmp), b'$TESTTMP'))
 
+testdir = os.path.dirname(self.path)
+replacementfile = os.path.join(testdir, 'common-pattern.py')
+
+if os.path.exists(replacementfile):
+data = {}
+execfile(replacementfile, data)
+r.extend(data.get('substitutions', ()))
 return r
 
 def _escapepath(self, p):
diff --git a/tests/test-run-tests.t b/tests/test-run-tests.t
--- a/tests/test-run-tests.t
+++ b/tests/test-run-tests.t
@@ -1503,3 +1503,43 @@ Test cases in .t files
   # Ran 2 tests, 0 skipped, 1 failed.
   python hash seed: * (glob)
   [1]
+
+Test automatic pattern replacement
+
+  $ cat << EOF >> common-pattern.py
+  > substitutions = [
+  > (br'foo-(.*)\\b',
+  >  br'\$XXX=\\1\$'),
+  > (br'bar\\n',
+  >  br'\$YYY$\\n'),
+  > ]
+  > EOF
+
+  $ cat << EOF >> test-substitution.t
+  >   $ echo foo-12
+  >   \$XXX=12$
+  >   $ echo foo-42
+  >   \$XXX=42$
+  >   $ echo bar prior
+  >   bar prior
+  >   $ echo lastbar
+  >   last\$YYY$
+  >   $ echo foo-bar foo-baz
+  > EOF
+
+  $ rt test-substitution.t
+  
+  --- $TESTTMP/anothertests/cases/test-substitution.t
+  +++ $TESTTMP/anothertests/cases/test-substitution.t.err
+  @@ -7,3 +7,4 @@
+ $ echo lastbar
+ last$YYY$
+ $ echo foo-bar foo-baz
+  +  $XXX=bar foo-baz$
+  
+  ERROR: test-substitution.t output changed
+  !
+  Failed test-substitution.t: output changed
+  # Ran 1 tests, 0 skipped, 1 failed.
+  python hash seed: * (glob)
+  [1]
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 08 of 11] test-pattern: register current the bundlecaps string

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1509860620 -3600
#  Sun Nov 05 06:43:40 2017 +0100
# Node ID 16289f5358a17c0ee629dfa6910b00bb8dec7fe9
# Parent  15aa75b64e1dff38fcce4a8f283daf0ce37a1261
# EXP-Topic better-substitute
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
16289f5358a1
test-pattern: register current the bundlecaps string

The bundle capabilites sent with every getbundle commands. Every time the
protocol is updated, that string is altered. We get that string replace by
$USUAL_BUNDLE_CAPS$ so that we only have to change the substitution whenever
this happens.

diff --git a/tests/common-pattern.py b/tests/common-pattern.py
--- a/tests/common-pattern.py
+++ b/tests/common-pattern.py
@@ -6,4 +6,17 @@ substitutions = [
 (br'zstd,zlib,none,bzip2',
  br'$USUAL_COMPRESSIONS$'
 ),
+# capabilities sent through http
+(br'bundlecaps=HG20%2Cbundle2%3DHG20%250A'
+ br'changegroup%253D01%252C02%250A'
+ br'digests%253Dmd5%252Csha1%252Csha512%250A'
+ br'error%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250A'
+ br'hgtagsfnodes%250A'
+ br'listkeys%250A'
+ br'phases%253Dheads%250A'
+ br'pushkey%250A'
+ br'remote-changegroup%253Dhttp%252Chttps',
+ # (the replacement patterns)
+ br'$USUAL_BUNDLE_CAPS$'
+),
 ]
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 07 of 11] test-pattern: substitute common compression list

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1509860498 -3600
#  Sun Nov 05 06:41:38 2017 +0100
# Node ID 15aa75b64e1dff38fcce4a8f283daf0ce37a1261
# Parent  a7d1303d69b73deac107d78ed37506f662ec661b
# EXP-Topic better-substitute
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
15aa75b64e1d
test-pattern: substitute common compression list

The compression list as to be matched with a glob because zstd might not be part
of the option. By using a substitution for these, we won't have to re-glob them
over and over.

diff --git a/tests/common-pattern.py b/tests/common-pattern.py
new file mode 100644
--- /dev/null
+++ b/tests/common-pattern.py
@@ -0,0 +1,9 @@
+# common patterns in test at can safely be replaced
+from __future__ import absolute_import
+
+substitutions = [
+# list of possible compressions
+(br'zstd,zlib,none,bzip2',
+ br'$USUAL_COMPRESSIONS$'
+),
+]
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 11 of 11] test-pattern: actually update tests using the patterns

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1509866592 -3600
#  Sun Nov 05 08:23:12 2017 +0100
# Node ID 715948f2256fa181c4f3730825492f9da0f74b2e
# Parent  c06ed80fc1063905dc6a711fc32c88d5913b0db6
# EXP-Topic better-substitute
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
715948f2256f
test-pattern: actually update tests using the patterns

We mass update the tests now. This will help the next soul touching the http
protocol.

diff --git a/tests/test-clonebundles.t b/tests/test-clonebundles.t
--- a/tests/test-clonebundles.t
+++ b/tests/test-clonebundles.t
@@ -32,8 +32,8 @@ Missing manifest should not result in se
 
   $ cat server/access.log
   * - - [*] "GET /?cmd=capabilities HTTP/1.1" 200 - (glob)
-  * - - [*] "GET /?cmd=batch HTTP/1.1" 200 - 
x-hgarg-1:cmds=heads+%3Bknown+nodes%3D x-hgproto-1:0.1 0.2 
comp=*zlib,none,bzip2 (glob)
-  * - - [*] "GET /?cmd=getbundle HTTP/1.1" 200 - 
x-hgarg-1:bundlecaps=HG20%2Cbundle2%3DHG20%250Achangegroup%253D01%252C02%250Adigests%253Dmd5%252Csha1%252Csha512%250Aerror%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250Ahgtagsfnodes%250Alistkeys%250Aphases%253Dheads%250Apushkey%250Aremote-changegroup%253Dhttp%252Chttps&cg=1&common=&heads=aaff8d2ffbbf07a46dd1f05d8ae7877e3f56e2a2&listkeys=bookmarks&phases=1
 x-hgproto-1:0.1 0.2 comp=*zlib,none,bzip2 (glob)
+  $LOCALIP - - [$LOGDATE$] "GET /?cmd=batch HTTP/1.1" 200 - 
x-hgarg-1:cmds=heads+%3Bknown+nodes%3D x-hgproto-1:0.1 0.2 
comp=$USUAL_COMPRESSIONS$
+  $LOCALIP - - [$LOGDATE$] "GET /?cmd=getbundle HTTP/1.1" 200 - 
x-hgarg-1:$USUAL_BUNDLE_CAPS$&cg=1&common=&heads=aaff8d2ffbbf07a46dd1f05d8ae7877e3f56e2a2&listkeys=bookmarks&phases=1
 x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
 
 Empty manifest file results in retrieval
 (the extension only checks if the manifest file exists)
diff --git a/tests/test-debugcommands.t b/tests/test-debugcommands.t
--- a/tests/test-debugcommands.t
+++ b/tests/test-debugcommands.t
@@ -162,7 +162,7 @@ Test debugcapabilities command:
   $ hg debugcapabilities ./debugrevlog/
   Main capabilities:
 branchmap
-
bundle2=HG20%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Aphases%3Dheads%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps
+$USUAL_BUNDLE2_CAPS$
 getbundle
 known
 lookup
diff --git a/tests/test-getbundle.t b/tests/test-getbundle.t
--- a/tests/test-getbundle.t
+++ b/tests/test-getbundle.t
@@ -264,9 +264,9 @@ Verify we hit the HTTP server:
 
   $ cat access.log
   * - - [*] "GET /?cmd=capabilities HTTP/1.1" 200 - (glob)
-  * - - [*] "GET /?cmd=getbundle HTTP/1.1" 200 - x-hgproto-1:0.1 0.2 
comp=*zlib,none,bzip2 (glob)
+  $LOCALIP - - [$LOGDATE$] "GET /?cmd=getbundle HTTP/1.1" 200 - 
x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
   * - - [*] "GET /?cmd=capabilities HTTP/1.1" 200 - (glob)
-  * - - [*] "GET /?cmd=getbundle HTTP/1.1" 200 - 
x-hgarg-1:common=700b7e19db54103633c4bf4a6a6b6d55f4d50c03+d5f6e1ea452285324836a49d7d3c2a63cfed1d31&heads=13c0170174366b441dc68e8e33757232fa744458+bac16991d12ff45f9dc43c52da1946dfadb83e80
 x-hgproto-1:0.1 0.2 comp=*zlib,none,bzip2 (glob)
+  $LOCALIP - - [$LOGDATE$] "GET /?cmd=getbundle HTTP/1.1" 200 - 
x-hgarg-1:common=700b7e19db54103633c4bf4a6a6b6d55f4d50c03+d5f6e1ea452285324836a49d7d3c2a63cfed1d31&heads=13c0170174366b441dc68e8e33757232fa744458+bac16991d12ff45f9dc43c52da1946dfadb83e80
 x-hgproto-1:0.1 0.2 comp=$USUAL_COMPRESSIONS$
 
   $ cat error.log
 
diff --git a/tests/test-hgweb-commands.t b/tests/test-hgweb-commands.t
--- a/tests/test-hgweb-commands.t
+++ b/tests/test-hgweb-commands.t
@@ -1926,7 +1926,7 @@ capabilities
   $ get-with-headers.py $LOCALIP:$HGPORT '?cmd=capabilities'; echo
   200 Script output follows
   
-  lookup changegroupsubset branchmap pushkey known getbundle unbundlehash 
batch 
bundle2=HG20%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Aphases%3Dheads%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps
 unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx 
compression=*zlib (glob)
+  lookup changegroupsubset branchmap pushkey known getbundle unbundlehash 
batch $USUAL_BUNDLE2_CAPS$ unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 
httpmediatype=0.1rx,0.1tx,0.2tx compression=zstd,zlib
 
 heads
 
@@ -2174,7 +2174,7 @@ capabilities
   batch
   stream-preferred
   streamreqs=generaldelta,revlogv1
-  
bundle2=HG20%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Aphases%3Dheads%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps
+  $USUAL_BUNDLE2_CAPS$
   unbundle=HG10GZ,HG10BZ,HG10UN
   httpheader=1024
   httpmediatype=0.1rx,0.1tx,0.2tx
diff --

[PATCH 02 of 11] test-pattern: substitute common compression list

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1509860498 -3600
#  Sun Nov 05 06:41:38 2017 +0100
# Node ID aaaceee35dc84c5a5a7f3287d2605dd9d5f27688
# Parent  fda6ff97ddead2610694668042f58f892130d982
# EXP-Topic better-substitute
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
aaaceee35dc8
test-pattern: substitute common compression list

The compression list as to be matched with a glob because zstd might not be part
of the option. By using a substitution for these, we won't have to re-glob them
over and over.

diff --git a/tests/common-pattern.py b/tests/common-pattern.py
new file mode 100644
--- /dev/null
+++ b/tests/common-pattern.py
@@ -0,0 +1,9 @@
+# common patterns in test at can safely be replaced
+from __future__ import absolute_import
+
+substitutions = [
+# list of possible compressions
+(br'zstd,zlib,none,bzip2',
+ br'$USUAL_COMPRESSIONS$'
+),
+]
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 03 of 11] test-pattern: register current the bundlecaps string

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1509860620 -3600
#  Sun Nov 05 06:43:40 2017 +0100
# Node ID 6e8aa2adeceaa523f214b79970df6471d728faf9
# Parent  aaaceee35dc84c5a5a7f3287d2605dd9d5f27688
# EXP-Topic better-substitute
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
6e8aa2adecea
test-pattern: register current the bundlecaps string

The bundle capabilites sent with every getbundle commands. Every time the
protocol is updated, that string is altered. We get that string replace by
$USUAL_BUNDLE_CAPS$ so that we only have to change the substitution whenever
this happens.

diff --git a/tests/common-pattern.py b/tests/common-pattern.py
--- a/tests/common-pattern.py
+++ b/tests/common-pattern.py
@@ -6,4 +6,17 @@ substitutions = [
 (br'zstd,zlib,none,bzip2',
  br'$USUAL_COMPRESSIONS$'
 ),
+# capabilities sent through http
+(br'bundlecaps=HG20%2Cbundle2%3DHG20%250A'
+ br'changegroup%253D01%252C02%250A'
+ br'digests%253Dmd5%252Csha1%252Csha512%250A'
+ br'error%253Dabort%252Cunsupportedcontent%252Cpushraced%252Cpushkey%250A'
+ br'hgtagsfnodes%250A'
+ br'listkeys%250A'
+ br'phases%253Dheads%250A'
+ br'pushkey%250A'
+ br'remote-changegroup%253Dhttp%252Chttps',
+ # (the replacement patterns)
+ br'$USUAL_BUNDLE_CAPS$'
+),
 ]
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH 10 of 11] test-pattern: substitute the HTTP log timestamp too

[Boris Feld]

# HG changeset patch
# User Boris Feld 
# Date 1509866633 -3600
#  Sun Nov 05 08:23:53 2017 +0100
# Node ID c06ed80fc1063905dc6a711fc32c88d5913b0db6
# Parent  8f0e6aaf04b6bdd6535b6bc03008538c95085105
# EXP-Topic better-substitute
# Available At https://bitbucket.org/octobus/mercurial-devel/
#  hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 
c06ed80fc106
test-pattern: substitute the HTTP log timestamp too

We add a pattern matching the infamous timestamp in http log. Now, we should be
able to have change appears in https log without having to re-glob the whole
thing over and over.

diff --git a/tests/common-pattern.py b/tests/common-pattern.py
--- a/tests/common-pattern.py
+++ b/tests/common-pattern.py
@@ -32,4 +32,8 @@ substitutions = [
  # (replacement patterns)
  br'$USUAL_BUNDLE2_CAPS$'
 ),
+# HTTP log dates
+(br' - - \[\d\d/.../2\d\d\d \d\d:\d\d:\d\d] "GET',
+ br' - - [$LOGDATE$] "GET'
+),
 ]
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1336: remove: print message for each file in verbose mode only while using `-A`

[pavanpc (Pavan Kumar PC)]

pavanpc added a comment.


  @lothiraldan  I modified a test case for hg rm -A  case where we 
get the message 'not removing : file still exists' . For other cases, 
test cases already exist with 'hg rm -A' and 'hg rm --after' without verbose 
mode.

INLINE COMMENTS

> mharbison72 wrote in cmdutil.py:2982
> This will need to be protected by seeing if there are any files in modified + 
> added + clean, like it was before.  Otherwise, using -A will always return 
> non zero, even if it succeeded without warning cases.  Maybe hoist the 
> 'remaining = ...' line out of the conditional?

Moved the 'remaining ' outside the conditional. Making the ui.verbose check 
only while adding it to the warnings list

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D1336

To: pavanpc, #hg-reviewers
Cc: lothiraldan, mitrandir, mharbison72, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D1336: remove: print message for each file in verbose mode only while using `-A`

[pavanpc (Pavan Kumar PC)]

pavanpc updated this revision to Diff 3435.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D1336?vs=3424&id=3435

REVISION DETAIL
  https://phab.mercurial-scm.org/D1336

AFFECTED FILES
  mercurial/cmdutil.py
  tests/test-remove.t

CHANGE DETAILS

diff --git a/tests/test-remove.t b/tests/test-remove.t
--- a/tests/test-remove.t
+++ b/tests/test-remove.t
@@ -179,7 +179,6 @@
   \r (no-eol) (esc)
   skipping [===>] 1/1\r (no-eol) (esc)
   \r (no-eol) (esc)
-  not removing bar: file still exists
   exit code: 1
   A bar
   ./bar
@@ -189,9 +188,9 @@
   \r (no-eol) (esc)
   0 files updated, 0 files merged, 0 files removed, 0 files unresolved
 
-21 state clean, options -A
+21 state clean, options -Av
 
-  $ remove -A foo
+  $ remove -Av foo
   \r (no-eol) (esc)
   deleting [===>] 1/1\r (no-eol) (esc)
   \r (no-eol) (esc)
@@ -205,10 +204,10 @@
   ./foo
   0 files updated, 0 files merged, 0 files removed, 0 files unresolved
 
-22 state modified, options -A
+22 state modified, options -Av
 
   $ echo b >> foo
-  $ remove -A foo
+  $ remove -Av foo
   \r (no-eol) (esc)
   deleting [===>] 1/1\r (no-eol) (esc)
   \r (no-eol) (esc)
@@ -357,10 +356,10 @@
   \r (no-eol) (esc)
   2 files updated, 0 files merged, 0 files removed, 0 files unresolved
 
-dir, options -A
+dir, options -Av
 
   $ rm test/bar
-  $ remove -A test
+  $ remove -Av test
   \r (no-eol) (esc)
   deleting [===>] 1/1\r (no-eol) (esc)
   \r (no-eol) (esc)
diff --git a/mercurial/cmdutil.py b/mercurial/cmdutil.py
--- a/mercurial/cmdutil.py
+++ b/mercurial/cmdutil.py
@@ -2975,8 +2975,9 @@
 for f in remaining:
 count += 1
 ui.progress(_('skipping'), count, total=total, unit=_('files'))
-warnings.append(_('not removing %s: file still exists\n')
-% m.rel(f))
+if ui.verbose:
+warnings.append(_('not removing %s: file still exists\n')
+% m.rel(f))
 ret = 1
 ui.progress(_('skipping'), None)
 else:



To: pavanpc, #hg-reviewers
Cc: lothiraldan, mitrandir, mharbison72, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: [PATCH 12 of 15 V2] bundle2: support a 'records' mode for the 'bookmarks' part

[Boris Feld]

On Fri, 2017-11-10 at 17:43 -0500, Augie Fackler wrote:
> On Thu, Nov 02, 2017 at 02:18:09PM +0100, Boris Feld wrote:
> > # HG changeset patch
> > # User Boris Feld 
> > # Date 1508246776 -7200
> > #  Tue Oct 17 15:26:16 2017 +0200
> > # Node ID 68bbec307c142b6b41893512b1c76320c87c2fa1
> > # Parent  bd3927325fe48e104b1627e5681ccd09a9a49e44
> > # EXP-Topic b2.bookmarks
> > # Available At https://bitbucket.org/octobus/mercurial-devel/
> > #  hg pull https://bitbucket.org/octobus/mercurial-deve
> > l/ -r 68bbec307c14
> > bundle2: support a 'records' mode for the 'bookmarks' part
> > 
> > In this mode, the bookmarks changes are record in the
> > 'bundleoperation' records
> > instead of inflicted to the repository. This is necessary to use
> > the part when
> 
> s/inflicted/applied/?
> 
> > pulling.
> > 
> 
> I'm confused. Why do we not want to apply the part when we're
> pulling?
> This log message could use some expansion.

When pulling, the remote bookmark value is not just applied -as-is-
into the local repository. There is an extra step to detect bookmark
divergence. The remote bookmarks data are stored until this processing
happens.

> 
> > 
> > diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
> > --- a/mercurial/bundle2.py
> > +++ b/mercurial/bundle2.py
> > @@ -1897,40 +1897,55 @@ def handlepushkey(op, inpart):
> >  def handlebookmark(op, inpart):
> >  """transmit bookmark information
> > 
> > -The part contains binary encoded bookmark information. The
> > bookmark
> > -information is applied as is to the unbundling repository.
> > Make sure a
> > -'check:bookmarks' part is issued earlier to check for race
> > condition in
> > -such update.
> > +The part contains binary encoded bookmark information.
> > +
> > +The exact behavior of this part can be controlled by the
> > 'bookmarks' mode
> > +on the bundle operation.
> > 
> > -This behavior is suitable for pushing. Semantic adjustment
> > will be needed
> > -for pull.
> > +When mode is 'apply' (the default) the bookmark information is
> > applied as
> > +is to the unbundling repository. Make sure a 'check:bookmarks'
> > part is
> > +issued earlier to check for push races in such update. This
> > behavior is
> > +suitable for pushing.
> > +
> > +When mode is 'records', the information is recorded into the
> > 'bookmarks'
> > +records of the bundle operation. This behavior is suitable for
> > pulling.
> >  """
> >  changes = bookmarks.binarydecode(inpart)
> > 
> > -tr = op.gettransaction()
> > -bookstore = op.repo._bookmarks
> > +pushkeycompat = op.repo.ui.configbool('server', 'bookmarks-
> > pushkey-compat')
> > +bookmarksmode = op.modes.get('bookmarks', 'apply')
> > 
> > -pushkeycompat = op.repo.ui.configbool('server', 'bookmarks-
> > pushkey-compat')
> > -if pushkeycompat:
> > -allhooks = []
> > +if bookmarksmode == 'apply':
> > +tr = op.gettransaction()
> > +bookstore = op.repo._bookmarks
> > +if pushkeycompat:
> > +allhooks = []
> > +for book, node in changes:
> > +hookargs = tr.hookargs.copy()
> > +hookargs['pushkeycompat'] = '1'
> > +hookargs['namespace'] = 'bookmark'
> > +hookargs['key'] = book
> > +hookargs['old'] = nodemod.hex(bookstore.get(book,
> > ''))
> > +hookargs['new'] = nodemod.hex(node if node is not
> > None else '')
> > +allhooks.append(hookargs)
> > +
> > +for hookargs in allhooks:
> > +op.repo.hook('prepushkey', throw=True, **hookargs)
> > +
> > +bookstore.applychanges(op.repo, op.gettransaction(),
> > changes)
> > +
> > +if pushkeycompat:
> > +def runhook():
> > +for hookargs in allhooks:
> > +op.repo.hook('prepushkey', **hookargs)
> > +op.repo._afterlock(runhook)
> > +
> > +elif bookmarksmode == 'records':
> >  for book, node in changes:
> > -hookargs = tr.hookargs.copy()
> > -hookargs['pushkeycompat'] = '1'
> > -hookargs['namespace'] = 'bookmark'
> > -hookargs['key'] = book
> > -hookargs['old'] = nodemod.hex(bookstore.get(book, ''))
> > -hookargs['new'] = nodemod.hex(node if node is not None
> > else '')
> > -allhooks.append(hookargs)
> > -for hookargs in allhooks:
> > -op.repo.hook('prepushkey', throw=True, **hookargs)
> > -
> > -bookstore.applychanges(op.repo, tr, changes)
> > -
> > -if pushkeycompat:
> > -def runhook():
> > -for hookargs in allhooks:
> > -op.repo.hook('prepushkey', **hookargs)
> > -op.repo._afterlock(runhook)
> > +record = {'bookmark': book, 'node': node}
> > +op.records.add('bookmarks', record)
> > +else:
> > +raise error.

Re: [PATCH 02 of 15 V2] bookmark: add methods to binary encode and decode bookmark values

[Boris Feld]

On Fri, 2017-11-10 at 17:35 -0500, Augie Fackler wrote:
> (+indygreg, who also is a formats enthusiast)
> 
> On Thu, Nov 02, 2017 at 02:17:59PM +0100, Boris Feld wrote:
> > # HG changeset patch
> > # User Boris Feld 
> > # Date 1508072395 -7200
> > #  Sun Oct 15 14:59:55 2017 +0200
> > # Node ID 4d0c6772a81aa1e2b25f32f944563db1f33fd327
> > # Parent  8c9a9eecdcd61401a1604a08a5272f7dabd4b912
> > # EXP-Topic b2.bookmarks
> > # Available At https://bitbucket.org/octobus/mercurial-devel/
> > #  hg pull https://bitbucket.org/octobus/mercurial-deve
> > l/ -r 4d0c6772a81a
> > bookmark: add methods to binary encode and decode bookmark values
> > 
> > Coming new bundle2 parts related to bookmark will use a binary
> > encoding. It
> > encodes a series of '(bookmark, node)' pairs. Bookmark name has a
> > high enough
> > size limit to not be affected by issue5165. (64K length, we are
> > well covered)
> 
> I'm not thrilled here. Could we do some sort of varint encoding,
> which
> would be generally useful going forward for a variety of things,
> rather than just shrugging and setting a limit which we (today) deem
> absurdly large?

We are not sure what the value of that would be. Bundle2 makes it very
easy to move to a newer encoding version if needed. We are already
doing so for changegroup and obsmarkers.

The biggest bookmark we have seen was about 800 characters, that should
leave us some room for the future without blocking us.

> 
> I agree that practically speaking, nobody should have a 64k bookmark,
> but we can do better. We also know that \0 shouldn't appear in a
> bookmark name, so we could just make the format be
> null-terminated. I'd much rather we get in the practice of
> deliberately crafting formats that are maximally flexible and
> reusable.

That would be significantly harder to parse since we would need to
search for the '\0' in the stream. We do not think it brings much value
either since any new data we sneak into the last field would have to be
understood by the receiver. If we need to negotiate for such
capabilities, this is equivalent to rolling out a new encoding.

> 
> IOW, encode something like this:
> 
> struct {
>   node [20]byte
>   name [] byte // continues until you see the first NUL
> }
> 
> Greg, am I talking crazy?
> 
> 
> > 
> > diff --git a/mercurial/bookmarks.py b/mercurial/bookmarks.py
> > --- a/mercurial/bookmarks.py
> > +++ b/mercurial/bookmarks.py
> > @@ -8,12 +8,14 @@
> >  from __future__ import absolute_import
> > 
> >  import errno
> > +import struct
> > 
> >  from .i18n import _
> >  from .node import (
> >  bin,
> >  hex,
> >  short,
> > +wdirid,
> >  )
> >  from . import (
> >  encoding,
> > @@ -550,6 +552,60 @@ def unhexlifybookmarks(marks):
> >  binremotemarks[name] = bin(node)
> >  return binremotemarks
> > 
> > +_binaryentry = struct.Struct('>20sH')
> > +
> > +def binaryencode(bookmarks):
> > +"""encode a '(bookmark, node)' iterable into a binary stream
> > +
> > +the binary format is:
> > +
> > +
> > +
> > +:node: is a 20 bytes binary node,
> > +:bookmark-length: an unsigned short,
> > +:bookmark-name: the name of the bookmark (of length  > length>)
> > +
> > +wdirid (all bits set) will be used as a special value for
> > "missing"
> > +"""
> > +binarydata = []
> > +for book, node in bookmarks:
> > +if not node: # None or ''
> > +node = wdirid
> > +binarydata.append(_binaryentry.pack(node, len(book)))
> > +binarydata.append(book)
> > +return ''.join(binarydata)
> > +
> > +def binarydecode(stream):
> > +"""decode a binary stream into an '(bookmark, node)' iterable
> > +
> > +the binary format is:
> > +
> > +
> > +
> > +:node: is a 20 bytes binary node,
> > +:bookmark-length: an unsigned short,
> > +:bookmark-name: the name of the bookmark (of length  > length>))
> > +
> > +wdirid (all bits set) will be used as a special value for
> > "missing"
> > +"""
> > +entrysize = _binaryentry.size
> > +books = []
> > +while True:
> > +entry = stream.read(entrysize)
> > +if len(entry) < entrysize:
> > +if entry:
> > +raise error.Abort(_('bad bookmark stream'))
> > +break
> > +node, length = _binaryentry.unpack(entry)
> > +bookmark = stream.read(length)
> > +if len(bookmark) < length:
> > +if entry:
> > +raise error.Abort(_('bad bookmark stream'))
> > +if node == wdirid:
> > +node = None
> > +books.append((bookmark, node))
> > +return books
> > +
> >  def updatefromremote(ui, repo, remotemarks, path, trfunc,
> > explicit=()):
> >  ui.debug("checking for updated bookmarks\n")
> >  localmarks = repo._bookmarks
> > ___
> > Mercurial-devel mailing list
> > Mercurial-devel@mercurial-scm.org
> > https://www.mercurial-scm.