D10962: rust: bump rust-cpython version for 0.6.0

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  That function as a nice `is_none` method on `PyObject` that we will put to 
use.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10962

AFFECTED FILES
  rust/Cargo.lock
  rust/hg-cpython/Cargo.toml

CHANGE DETAILS

diff --git a/rust/hg-cpython/Cargo.toml b/rust/hg-cpython/Cargo.toml
--- a/rust/hg-cpython/Cargo.toml
+++ b/rust/hg-cpython/Cargo.toml
@@ -28,5 +28,5 @@
 env_logger = "0.7.1"
 
 [dependencies.cpython]
-version = "0.5.2"
+version = "0.6.0"
 default-features = false
diff --git a/rust/Cargo.lock b/rust/Cargo.lock
--- a/rust/Cargo.lock
+++ b/rust/Cargo.lock
@@ -157,9 +157,9 @@
 
 [[package]]
 name = "cpython"
-version = "0.5.2"
+version = "0.6.0"
 source = "registry+https://github.com/rust-lang/crates.io-index;
-checksum = "0f11357af68648b6a227e7e2384d439cec8595de65970f45e3f7f4b2600be472"
+checksum = "8094679a4e9bfc8035572162624bc800eda35b5f9eff2537b9cd9aacc3d9782e"
 dependencies = [
  "libc",
  "num-traits",
@@ -604,22 +604,9 @@
 
 [[package]]
 name = "paste"
-version = "0.1.18"
+version = "1.0.5"
 source = "registry+https://github.com/rust-lang/crates.io-index;
-checksum = "45ca20c77d80be666aef2b45486da86238fabe33e38306bd3118fe4af33fa880"
-dependencies = [
- "paste-impl",
- "proc-macro-hack",
-]
-
-[[package]]
-name = "paste-impl"
-version = "0.1.18"
-source = "registry+https://github.com/rust-lang/crates.io-index;
-checksum = "d95a7db200b97ef370c8e6de0088252f7e0dfff7d047a28528e47456c0fc98b6"
-dependencies = [
- "proc-macro-hack",
-]
+checksum = "acbf547ad0c65e31259204bd90935776d1c693cec2f4ff7abb7a1bbbd40dfe58"
 
 [[package]]
 name = "pkg-config"
@@ -662,9 +649,9 @@
 
 [[package]]
 name = "python27-sys"
-version = "0.5.2"
+version = "0.6.0"
 source = "registry+https://github.com/rust-lang/crates.io-index;
-checksum = "f485897ed7048f5032317c4e427800ef9f2053355516524d73952b8b07032054"
+checksum = "5826ddbc5366eb0b0492040fdc25bf50bb49092c192bd45e80fb7a24dc6832ab"
 dependencies = [
  "libc",
  "regex",
@@ -672,9 +659,9 @@
 
 [[package]]
 name = "python3-sys"
-version = "0.5.2"
+version = "0.6.0"
 source = "registry+https://github.com/rust-lang/crates.io-index;
-checksum = "5b29b99c6868eb02beb3bf6ed025c8bcdf02efc149b8e80347d3e5d059a806db"
+checksum = "b78af21b29594951a47fc3dac9b9eff0a3f077dec2f780ee943ae16a668f3b6a"
 dependencies = [
  "libc",
  "regex",



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10961: rust-dirstate: make the MTIME_UNSET public

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  We will use it in the cpython layer soon.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10961

AFFECTED FILES
  rust/hg-core/src/dirstate.rs

CHANGE DETAILS

diff --git a/rust/hg-core/src/dirstate.rs b/rust/hg-core/src/dirstate.rs
--- a/rust/hg-core/src/dirstate.rs
+++ b/rust/hg-core/src/dirstate.rs
@@ -77,7 +77,7 @@
 length: unaligned::I32Be,
 }
 
-const MTIME_UNSET: i32 = -1;
+pub const MTIME_UNSET: i32 = -1;
 
 /// A `DirstateEntry` with a size of `-2` means that it was merged from the
 /// other parent. This allows revert to pick the right status back during a



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10959: dirstate-entry: restrict `from_p2` property to tracked file

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  The other case is now handled in `from_p2_removed`, and should ideally be 
moved
  to a lower level. So we can drop the `state == 'n'` narrowing and move it in 
the
  property itself.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10959

AFFECTED FILES
  mercurial/cext/parsers.c
  mercurial/dirstate.py
  mercurial/pure/parsers.py

CHANGE DETAILS

diff --git a/mercurial/pure/parsers.py b/mercurial/pure/parsers.py
--- a/mercurial/pure/parsers.py
+++ b/mercurial/pure/parsers.py
@@ -98,9 +98,11 @@
 def from_p2(self):
 """True if the file have been fetched from p2 during the current merge
 
+This is only True is the file is currently tracked.
+
 Should only be set if a merge is in progress in the dirstate
 """
-return self._size == FROM_P2
+return self._state == b'n' and self._size == FROM_P2
 
 @property
 def from_p2_removed(self):
diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -393,7 +393,7 @@
 copies[f] = source
 self.normallookup(f)
 # Also fix up otherparent markers
-elif s.state == b'n' and s.from_p2:
+elif s.from_p2:
 source = self._map.copymap.get(f)
 if source:
 copies[f] = source
@@ -542,7 +542,7 @@
 if source is not None:
 self.copy(source, f)
 return
-elif entry.merged or entry.state == b'n' and entry.from_p2:
+elif entry.merged or entry.from_p2:
 return
 self._addpath(f, b'n', 0, possibly_dirty=True)
 self._map.copymap.pop(f, None)
diff --git a/mercurial/cext/parsers.c b/mercurial/cext/parsers.c
--- a/mercurial/cext/parsers.c
+++ b/mercurial/cext/parsers.c
@@ -178,7 +178,7 @@
 
 static PyObject *dirstatetuple_get_from_p2(dirstateTupleObject *self)
 {
-   if (self->size == dirstate_v1_from_p2) {
+   if (self->state == 'n' && self->size == dirstate_v1_from_p2) {
Py_RETURN_TRUE;
} else {
Py_RETURN_FALSE;



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10960: rust-dirstatemap: expand the wrapping code a bit

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is easier to read.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10960

AFFECTED FILES
  rust/hg-cpython/src/dirstate/dirstate_map.rs

CHANGE DETAILS

diff --git a/rust/hg-cpython/src/dirstate/dirstate_map.rs 
b/rust/hg-cpython/src/dirstate/dirstate_map.rs
--- a/rust/hg-cpython/src/dirstate/dirstate_map.rs
+++ b/rust/hg-cpython/src/dirstate/dirstate_map.rs
@@ -112,23 +112,31 @@
 size: PyObject,
 mtime: PyObject
 ) -> PyResult {
+let f = f.extract::(py)?;
+let filename = HgPath::new(f.data(py));
+let oldstate = oldstate.extract::(py)?.data(py)[0]
+.try_into()
+.map_err(|e: HgError| {
+PyErr::new::(py, e.to_string())
+})?;
+let state = state.extract::(py)?.data(py)[0]
+.try_into()
+.map_err(|e: HgError| {
+PyErr::new::(py, e.to_string())
+})?;
+let mode = mode.extract(py)?;
+let size = size.extract(py)?;
+let mtime = mtime.extract(py)?;
+let entry = DirstateEntry {
+state: state,
+mode: mode,
+size: size,
+mtime: mtime,
+};
 self.inner(py).borrow_mut().add_file(
-HgPath::new(f.extract::(py)?.data(py)),
-oldstate.extract::(py)?.data(py)[0]
-.try_into()
-.map_err(|e: HgError| {
-PyErr::new::(py, e.to_string())
-})?,
-DirstateEntry {
-state: state.extract::(py)?.data(py)[0]
-.try_into()
-.map_err(|e: HgError| {
-PyErr::new::(py, e.to_string())
-})?,
-mode: mode.extract(py)?,
-size: size.extract(py)?,
-mtime: mtime.extract(py)?,
-},
+filename,
+oldstate,
+entry,
 ).and(Ok(py.None())).or_else(|e: DirstateError| {
 Err(PyErr::new::(py, e.to_string()))
 })



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10953: dirstate: move most of the `remove` logic with dirstatemap `removefile`

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This code deal with special logic to preserving "merged" and "from_p2" 
information when removing a file. These are implementation details that are 
more suitable for the dirstatemap layer. Since the dirstatemap layer alreaday 
have most of the information necessary to do so, the move is easy.
  
  This move helps us to encapsulate more implementation details within the 
dirstatemap and its entry. Easing the use of a different storage for dirstate 
v2.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10953

AFFECTED FILES
  mercurial/dirstate.py
  mercurial/dirstatemap.py
  rust/hg-core/src/dirstate.rs
  rust/hg-core/src/dirstate/dirstate_map.rs
  rust/hg-core/src/dirstate_tree/dirstate_map.rs
  rust/hg-core/src/dirstate_tree/dispatch.rs
  rust/hg-cpython/src/dirstate/dirstate_map.rs
  rust/hg-cpython/src/dirstate/dispatch.rs

CHANGE DETAILS

diff --git a/rust/hg-cpython/src/dirstate/dispatch.rs 
b/rust/hg-cpython/src/dirstate/dispatch.rs
--- a/rust/hg-cpython/src/dirstate/dispatch.rs
+++ b/rust/hg-cpython/src/dirstate/dispatch.rs
@@ -33,10 +33,9 @@
 fn remove_file(
  self,
 filename: ,
-old_state: EntryState,
-size: i32,
+in_merge: bool,
 ) -> Result<(), DirstateError> {
-self.get_mut().remove_file(filename, old_state, size)
+self.get_mut().remove_file(filename, in_merge)
 }
 
 fn drop_file(
diff --git a/rust/hg-cpython/src/dirstate/dirstate_map.rs 
b/rust/hg-cpython/src/dirstate/dirstate_map.rs
--- a/rust/hg-cpython/src/dirstate/dirstate_map.rs
+++ b/rust/hg-cpython/src/dirstate/dirstate_map.rs
@@ -137,18 +137,12 @@
 def removefile(
 ,
 f: PyObject,
-oldstate: PyObject,
-size: PyObject
+in_merge: PyObject
 ) -> PyResult {
 self.inner(py).borrow_mut()
 .remove_file(
 HgPath::new(f.extract::(py)?.data(py)),
-oldstate.extract::(py)?.data(py)[0]
-.try_into()
-.map_err(|e: HgError| {
-PyErr::new::(py, e.to_string())
-})?,
-size.extract(py)?,
+in_merge.extract::(py)?.is_true(),
 )
 .or_else(|_| {
 Err(PyErr::new::(
diff --git a/rust/hg-core/src/dirstate_tree/dispatch.rs 
b/rust/hg-core/src/dirstate_tree/dispatch.rs
--- a/rust/hg-core/src/dirstate_tree/dispatch.rs
+++ b/rust/hg-core/src/dirstate_tree/dispatch.rs
@@ -61,8 +61,7 @@
 fn remove_file(
  self,
 filename: ,
-old_state: EntryState,
-size: i32,
+in_merge: bool,
 ) -> Result<(), DirstateError>;
 
 /// Drop information about this file from the map if any, and return
@@ -295,10 +294,9 @@
 fn remove_file(
  self,
 filename: ,
-old_state: EntryState,
-size: i32,
+in_merge: bool,
 ) -> Result<(), DirstateError> {
-self.remove_file(filename, old_state, size)
+self.remove_file(filename, in_merge)
 }
 
 fn drop_file(
diff --git a/rust/hg-core/src/dirstate_tree/dirstate_map.rs 
b/rust/hg-core/src/dirstate_tree/dirstate_map.rs
--- a/rust/hg-core/src/dirstate_tree/dirstate_map.rs
+++ b/rust/hg-core/src/dirstate_tree/dirstate_map.rs
@@ -11,6 +11,8 @@
 use crate::dirstate::parsers::packed_entry_size;
 use crate::dirstate::parsers::parse_dirstate_entries;
 use crate::dirstate::parsers::Timestamp;
+use crate::dirstate::SIZE_FROM_OTHER_PARENT;
+use crate::dirstate::SIZE_NON_NORMAL;
 use crate::matchers::Matcher;
 use crate::utils::hg_path::{HgPath, HgPathBuf};
 use crate::CopyMapIter;
@@ -726,9 +728,34 @@
 fn remove_file(
  self,
 filename: ,
-old_state: EntryState,
-size: i32,
+in_merge: bool,
 ) -> Result<(), DirstateError> {
+let old_entry_opt = self.get(filename)?;
+let old_state = match old_entry_opt {
+Some(e) => e.state,
+None => EntryState::Unknown,
+};
+let mut size = 0;
+if in_merge {
+// XXX we should not be able to have 'm' state and 'FROM_P2' if not
+// during a merge. So I (marmoute) am not sure we need the
+// conditionnal at all. Adding double checking this with assert
+// would be nice.
+if let Some(old_entry) = old_entry_opt {
+// backup the previous state
+if old_entry.state == EntryState::Merged {
+size = SIZE_NON_NORMAL;
+} else if old_entry.state == EntryState::Normal
+&& old_entry.size == SIZE_FROM_OTHER_PARENT
+{
+// other parent
+size = SIZE_FROM_OTHER_PARENT;
+}
+}
+}
+ 

D10958: dirstate-entry: `merged_removed` and `from_p2_removed` properties

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Lets start to define and use more semantic property. These two might be a bit
  too low level and could be shaved off later, however this seems an improvement
  for now.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10958

AFFECTED FILES
  mercurial/cext/parsers.c
  mercurial/dirstate.py
  mercurial/pure/parsers.py

CHANGE DETAILS

diff --git a/mercurial/pure/parsers.py b/mercurial/pure/parsers.py
--- a/mercurial/pure/parsers.py
+++ b/mercurial/pure/parsers.py
@@ -36,6 +36,9 @@
 # a special value used internally for `size` if the file come from the other 
parent
 FROM_P2 = -2
 
+# a special value used internally for `size` if the file is 
modified/merged/added
+NONNORMAL = -1
+
 
 class dirstatetuple(object):
 """represente a dirstate entry
@@ -100,10 +103,28 @@
 return self._size == FROM_P2
 
 @property
+def from_p2_removed(self):
+"""True if the file has been removed, but was merged initially
+
+This property seems like an attraction leakage and should probably be
+dealt in this class (or maybe the dirstatemap one directly.
+"""
+return self._state == b'r' and self._size == FROM_P2
+
+@property
 def removed(self):
 """True if the file has been removed"""
 return self._state == b'r'
 
+@property
+def merged_removed(self):
+"""True if the file has been removed, but was merged initially
+
+This property seems like an attraction leakage and should probably be
+dealt in this class (or maybe the dirstatemap one directly.
+"""
+return self._state == b'r' and self._size == NONNORMAL
+
 def v1_state(self):
 """return a "state" suitable for v1 serialization"""
 return self._state
diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -531,16 +531,18 @@
 # being removed, restore that state.
 entry = self._map.get(f)
 if entry is not None:
-if entry.removed and (entry[2] == NONNORMAL or entry.from_p2):
+# XXX this should probably be dealt with a a lower level
+# (see `merged_removed` and `from_p2_removed`)
+if entry.merged_removed or entry.from_p2_removed:
 source = self._map.copymap.get(f)
-if entry[2] == NONNORMAL:
+if entry.merged_removed:
 self.merge(f)
-elif entry.from_p2:
+elif entry.from_p2_removed:
 self.otherparent(f)
-if source:
+if source is not None:
 self.copy(source, f)
 return
-if entry.merged or entry.state == b'n' and entry.from_p2:
+elif entry.merged or entry.state == b'n' and entry.from_p2:
 return
 self._addpath(f, b'n', 0, possibly_dirty=True)
 self._map.copymap.pop(f, None)
diff --git a/mercurial/cext/parsers.c b/mercurial/cext/parsers.c
--- a/mercurial/cext/parsers.c
+++ b/mercurial/cext/parsers.c
@@ -30,6 +30,7 @@
 static const char *const versionerrortext = "Python minor version mismatch";
 
 static const int dirstate_v1_from_p2 = -2;
+static const int dirstate_v1_nonnormal = -1;
 
 static PyObject *dict_new_presized(PyObject *self, PyObject *args)
 {
@@ -166,6 +167,15 @@
}
 };
 
+static PyObject *dirstatetuple_get_merged_removed(dirstateTupleObject *self)
+{
+   if (self->state == 'r' && self->size == dirstate_v1_nonnormal) {
+   Py_RETURN_TRUE;
+   } else {
+   Py_RETURN_FALSE;
+   }
+};
+
 static PyObject *dirstatetuple_get_from_p2(dirstateTupleObject *self)
 {
if (self->size == dirstate_v1_from_p2) {
@@ -175,6 +185,15 @@
}
 };
 
+static PyObject *dirstatetuple_get_from_p2_removed(dirstateTupleObject *self)
+{
+   if (self->state == 'r' && self->size == dirstate_v1_from_p2) {
+   Py_RETURN_TRUE;
+   } else {
+   Py_RETURN_FALSE;
+   }
+};
+
 static PyObject *dirstatetuple_get_removed(dirstateTupleObject *self)
 {
if (self->state == 'r') {
@@ -186,7 +205,11 @@
 
 static PyGetSetDef dirstatetuple_getset[] = {
 {"state", (getter)dirstatetuple_get_state, NULL, "state", NULL},
+{"merged_removed", (getter)dirstatetuple_get_merged_removed, NULL,
+ "merged_removed", NULL},
 {"merged", (getter)dirstatetuple_get_merged, NULL, "merged", NULL},
+{"from_p2_removed", (getter)dirstatetuple_get_from_p2_removed, NULL,
+ "from_p2_removed", NULL},
 {"from_p2", (getter)dirstatetuple_get_from_p2, NULL, "from_p2", NULL},
 {"removed", (getter)dirstatetuple_get_removed, NULL, "removed", 

D10957: dirstate-entry: add a `removed` property

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Lets start to define and use more semantic property.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10957

AFFECTED FILES
  mercurial/cext/parsers.c
  mercurial/dirstate.py
  mercurial/pure/parsers.py

CHANGE DETAILS

diff --git a/mercurial/pure/parsers.py b/mercurial/pure/parsers.py
--- a/mercurial/pure/parsers.py
+++ b/mercurial/pure/parsers.py
@@ -99,6 +99,11 @@
 """
 return self._size == FROM_P2
 
+@property
+def removed(self):
+"""True if the file has been removed"""
+return self._state == b'r'
+
 def v1_state(self):
 """return a "state" suitable for v1 serialization"""
 return self._state
diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -472,7 +472,7 @@
 if self._map.hastrackeddir(d):
 break
 entry = self._map.get(d)
-if entry is not None and entry.state != b'r':
+if entry is not None and not entry.removed:
 msg = _(b'file %r in dirstate clashes with %r')
 msg %= (pycompat.bytestr(d), pycompat.bytestr(f))
 raise error.Abort(msg)
@@ -531,9 +531,7 @@
 # being removed, restore that state.
 entry = self._map.get(f)
 if entry is not None:
-if entry.state == b'r' and (
-entry[2] == NONNORMAL or entry.from_p2
-):
+if entry.removed and (entry[2] == NONNORMAL or entry.from_p2):
 source = self._map.copymap.get(f)
 if entry[2] == NONNORMAL:
 self.merge(f)
@@ -1364,7 +1362,7 @@
 madd(fn)
 elif state == b'a':
 aadd(fn)
-elif state == b'r':
+elif t.removed:
 radd(fn)
 status = scmutil.status(
 modified, added, removed, deleted, unknown, ignored, clean
diff --git a/mercurial/cext/parsers.c b/mercurial/cext/parsers.c
--- a/mercurial/cext/parsers.c
+++ b/mercurial/cext/parsers.c
@@ -175,10 +175,20 @@
}
 };
 
+static PyObject *dirstatetuple_get_removed(dirstateTupleObject *self)
+{
+   if (self->state == 'r') {
+   Py_RETURN_TRUE;
+   } else {
+   Py_RETURN_FALSE;
+   }
+};
+
 static PyGetSetDef dirstatetuple_getset[] = {
 {"state", (getter)dirstatetuple_get_state, NULL, "state", NULL},
 {"merged", (getter)dirstatetuple_get_merged, NULL, "merged", NULL},
 {"from_p2", (getter)dirstatetuple_get_from_p2, NULL, "from_p2", NULL},
+{"removed", (getter)dirstatetuple_get_removed, NULL, "removed", NULL},
 {NULL} /* Sentinel */
 };
 



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10956: dirstate-entry: add a `from_p2` property

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Lets start to define and use more semantic property.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10956

AFFECTED FILES
  mercurial/cext/parsers.c
  mercurial/dirstate.py
  mercurial/dirstatemap.py
  mercurial/pure/parsers.py

CHANGE DETAILS

diff --git a/mercurial/pure/parsers.py b/mercurial/pure/parsers.py
--- a/mercurial/pure/parsers.py
+++ b/mercurial/pure/parsers.py
@@ -33,6 +33,10 @@
 _decompress = zlib.decompress
 
 
+# a special value used internally for `size` if the file come from the other 
parent
+FROM_P2 = -2
+
+
 class dirstatetuple(object):
 """represente a dirstate entry
 
@@ -87,6 +91,14 @@
 """
 return self._state == b'm'
 
+@property
+def from_p2(self):
+"""True if the file have been fetched from p2 during the current merge
+
+Should only be set if a merge is in progress in the dirstate
+"""
+return self._size == FROM_P2
+
 def v1_state(self):
 """return a "state" suitable for v1 serialization"""
 return self._state
diff --git a/mercurial/dirstatemap.py b/mercurial/dirstatemap.py
--- a/mercurial/dirstatemap.py
+++ b/mercurial/dirstatemap.py
@@ -173,7 +173,7 @@
 # backup the previous state
 if entry.merged:  # merge
 size = NONNORMAL
-elif entry[0] == b'n' and entry[2] == FROM_P2:  # other parent
+elif entry[0] == b'n' and entry.from_p2:
 size = FROM_P2
 self.otherparentset.add(f)
 if size == 0:
diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -393,7 +393,7 @@
 copies[f] = source
 self.normallookup(f)
 # Also fix up otherparent markers
-elif s.state == b'n' and s[2] == FROM_P2:
+elif s.state == b'n' and s.from_p2:
 source = self._map.copymap.get(f)
 if source:
 copies[f] = source
@@ -531,16 +531,18 @@
 # being removed, restore that state.
 entry = self._map.get(f)
 if entry is not None:
-if entry.state == b'r' and entry[2] in (NONNORMAL, FROM_P2):
+if entry.state == b'r' and (
+entry[2] == NONNORMAL or entry.from_p2
+):
 source = self._map.copymap.get(f)
 if entry[2] == NONNORMAL:
 self.merge(f)
-elif entry[2] == FROM_P2:
+elif entry.from_p2:
 self.otherparent(f)
 if source:
 self.copy(source, f)
 return
-if entry.merged or entry.state == b'n' and entry[2] == FROM_P2:
+if entry.merged or entry.state == b'n' and entry.from_p2:
 return
 self._addpath(f, b'n', 0, possibly_dirty=True)
 self._map.copymap.pop(f, None)
@@ -1336,7 +1338,7 @@
 (size != st.st_size and size != st.st_size & 
_rangemask)
 or ((mode ^ st.st_mode) & 0o100 and checkexec)
 )
-or size == FROM_P2  # other parent
+or t.from_p2
 or fn in copymap
 ):
 if stat.S_ISLNK(st.st_mode) and size != st.st_size:
diff --git a/mercurial/cext/parsers.c b/mercurial/cext/parsers.c
--- a/mercurial/cext/parsers.c
+++ b/mercurial/cext/parsers.c
@@ -29,6 +29,8 @@
 
 static const char *const versionerrortext = "Python minor version mismatch";
 
+static const int dirstate_v1_from_p2 = -2;
+
 static PyObject *dict_new_presized(PyObject *self, PyObject *args)
 {
Py_ssize_t expected_size;
@@ -164,9 +166,19 @@
}
 };
 
+static PyObject *dirstatetuple_get_from_p2(dirstateTupleObject *self)
+{
+   if (self->size == dirstate_v1_from_p2) {
+   Py_RETURN_TRUE;
+   } else {
+   Py_RETURN_FALSE;
+   }
+};
+
 static PyGetSetDef dirstatetuple_getset[] = {
 {"state", (getter)dirstatetuple_get_state, NULL, "state", NULL},
 {"merged", (getter)dirstatetuple_get_merged, NULL, "merged", NULL},
+{"from_p2", (getter)dirstatetuple_get_from_p2, NULL, "from_p2", NULL},
 {NULL} /* Sentinel */
 };
 



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10955: dirstate-entry: add a `merged` property

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Lets start to define and use more semantic property.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10955

AFFECTED FILES
  mercurial/cext/parsers.c
  mercurial/dirstate.py
  mercurial/dirstatemap.py
  mercurial/pure/parsers.py

CHANGE DETAILS

diff --git a/mercurial/pure/parsers.py b/mercurial/pure/parsers.py
--- a/mercurial/pure/parsers.py
+++ b/mercurial/pure/parsers.py
@@ -79,6 +79,14 @@
 """
 return self._state
 
+@property
+def merged(self):
+"""True if the file has been merged
+
+Should only be set if a merge is in progress in the dirstate
+"""
+return self._state == b'm'
+
 def v1_state(self):
 """return a "state" suitable for v1 serialization"""
 return self._state
diff --git a/mercurial/dirstatemap.py b/mercurial/dirstatemap.py
--- a/mercurial/dirstatemap.py
+++ b/mercurial/dirstatemap.py
@@ -171,7 +171,7 @@
 # would be nice.
 if entry is not None:
 # backup the previous state
-if entry[0] == b'm':  # merge
+if entry.merged:  # merge
 size = NONNORMAL
 elif entry[0] == b'n' and entry[2] == FROM_P2:  # other parent
 size = FROM_P2
diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -355,7 +355,7 @@
 def setparents(self, p1, p2=None):
 """Set dirstate parents to p1 and p2.
 
-When moving from two parents to one, 'm' merged entries a
+When moving from two parents to one, "merged" entries a
 adjusted to normal and previous copy records discarded and
 returned by the call.
 
@@ -386,8 +386,8 @@
 if s is None:
 continue
 
-# Discard 'm' markers when moving away from a merge state
-if s.state == b'm':
+# Discard "merged" markers when moving away from a merge state
+if s.merged:
 source = self._map.copymap.get(f)
 if source:
 copies[f] = source
@@ -527,7 +527,7 @@
 '''Mark a file normal, but possibly dirty.'''
 if self.in_merge:
 # if there is a merge going on and the file was either
-# in state 'm' (-1) or coming from other parent (-2) before
+# "merged" or coming from other parent (-2) before
 # being removed, restore that state.
 entry = self._map.get(f)
 if entry is not None:
@@ -540,11 +540,7 @@
 if source:
 self.copy(source, f)
 return
-if (
-entry.state == b'm'
-or entry.state == b'n'
-and entry[2] == FROM_P2
-):
+if entry.merged or entry.state == b'n' and entry[2] == FROM_P2:
 return
 self._addpath(f, b'n', 0, possibly_dirty=True)
 self._map.copymap.pop(f, None)
@@ -1362,7 +1358,7 @@
 ladd(fn)
 elif listclean:
 cadd(fn)
-elif state == b'm':
+elif t.merged:
 madd(fn)
 elif state == b'a':
 aadd(fn)
diff --git a/mercurial/cext/parsers.c b/mercurial/cext/parsers.c
--- a/mercurial/cext/parsers.c
+++ b/mercurial/cext/parsers.c
@@ -155,8 +155,18 @@
return PyBytes_FromStringAndSize(>state, 1);
 };
 
+static PyObject *dirstatetuple_get_merged(dirstateTupleObject *self)
+{
+   if (self->state == 'm') {
+   Py_RETURN_TRUE;
+   } else {
+   Py_RETURN_FALSE;
+   }
+};
+
 static PyGetSetDef dirstatetuple_getset[] = {
 {"state", (getter)dirstatetuple_get_state, NULL, "state", NULL},
+{"merged", (getter)dirstatetuple_get_merged, NULL, "merged", NULL},
 {NULL} /* Sentinel */
 };
 



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10954: dirstate-entry: add a `state` property (and use it)

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This replace the [0] access. Ultimately is we should probably get ride of this
  in its current form. However this is a good transitional solution to move away
  for tuple indexing for now.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10954

AFFECTED FILES
  mercurial/cext/parsers.c
  mercurial/dirstate.py
  mercurial/pure/parsers.py

CHANGE DETAILS

diff --git a/mercurial/pure/parsers.py b/mercurial/pure/parsers.py
--- a/mercurial/pure/parsers.py
+++ b/mercurial/pure/parsers.py
@@ -64,6 +64,21 @@
 else:
 raise IndexError(idx)
 
+@property
+def state(self):
+"""
+States are:
+  n  normal
+  m  needs merging
+  r  marked for removal
+  a  marked for addition
+
+XXX This "state" is a bit obscure and mostly a direct expression of the
+dirstatev1 format. It would make sense to ultimately deprecate it in
+favor of the more "semantic" attributes.
+"""
+return self._state
+
 def v1_state(self):
 """return a "state" suitable for v1 serialization"""
 return self._state
diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -311,8 +311,15 @@
   r  marked for removal
   a  marked for addition
   ?  not tracked
+
+XXX The "state" is a bit obscure to be in the "public" API. we should
+consider migrating all user of this to going through the dirstate entry
+instead.
 """
-return self._map.get(key, (b"?",))[0]
+entry = self._map.get(key)
+if entry is not None:
+return entry.state
+return b'?'
 
 def __contains__(self, key):
 return key in self._map
@@ -380,13 +387,13 @@
 continue
 
 # Discard 'm' markers when moving away from a merge state
-if s[0] == b'm':
+if s.state == b'm':
 source = self._map.copymap.get(f)
 if source:
 copies[f] = source
 self.normallookup(f)
 # Also fix up otherparent markers
-elif s[0] == b'n' and s[2] == FROM_P2:
+elif s.state == b'n' and s[2] == FROM_P2:
 source = self._map.copymap.get(f)
 if source:
 copies[f] = source
@@ -465,7 +472,7 @@
 if self._map.hastrackeddir(d):
 break
 entry = self._map.get(d)
-if entry is not None and entry[0] != b'r':
+if entry is not None and entry.state != b'r':
 msg = _(b'file %r in dirstate clashes with %r')
 msg %= (pycompat.bytestr(d), pycompat.bytestr(f))
 raise error.Abort(msg)
@@ -524,7 +531,7 @@
 # being removed, restore that state.
 entry = self._map.get(f)
 if entry is not None:
-if entry[0] == b'r' and entry[2] in (NONNORMAL, FROM_P2):
+if entry.state == b'r' and entry[2] in (NONNORMAL, FROM_P2):
 source = self._map.copymap.get(f)
 if entry[2] == NONNORMAL:
 self.merge(f)
@@ -533,7 +540,11 @@
 if source:
 self.copy(source, f)
 return
-if entry[0] == b'm' or entry[0] == b'n' and entry[2] == 
FROM_P2:
+if (
+entry.state == b'm'
+or entry.state == b'n'
+and entry[2] == FROM_P2
+):
 return
 self._addpath(f, b'n', 0, possibly_dirty=True)
 self._map.copymap.pop(f, None)
@@ -761,7 +772,7 @@
 if delaywrite > 0:
 # do we have any files to delay for?
 for f, e in pycompat.iteritems(self._map):
-if e[0] == b'n' and e[3] == now:
+if e.state == b'n' and e[3] == now:
 import time  # to avoid useless import
 
 # rather than sleep n seconds, sleep until the next
@@ -1315,7 +1326,7 @@
 # general. That is much slower than simply accessing and storing 
the
 # tuple members one by one.
 t = dget(fn)
-state = t[0]
+state = t.state
 mode = t[1]
 size = t[2]
 time = t[3]
diff --git a/mercurial/cext/parsers.c b/mercurial/cext/parsers.c
--- a/mercurial/cext/parsers.c
+++ b/mercurial/cext/parsers.c
@@ -150,6 +150,16 @@
 {NULL} /* Sentinel */
 };
 
+static PyObject *dirstatetuple_get_state(dirstateTupleObject *self)
+{
+   return 

D10952: dirstate: add a `in_merge` property

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This factor the "p2 is not null" check and is fairly simpler to read.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10952

AFFECTED FILES
  mercurial/dirstate.py

CHANGE DETAILS

diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -337,6 +337,11 @@
 def p2(self):
 return self._validate(self._pl[1])
 
+@property
+def in_merge(self):
+"""True if a merge is in progress"""
+return self._pl[1] != self._nodeconstants.nullid
+
 def branch(self):
 return encoding.tolocal(self._branch)
 
@@ -513,7 +518,7 @@
 
 def normallookup(self, f):
 '''Mark a file normal, but possibly dirty.'''
-if self._pl[1] != self._nodeconstants.nullid:
+if self.in_merge:
 # if there is a merge going on and the file was either
 # in state 'm' (-1) or coming from other parent (-2) before
 # being removed, restore that state.
@@ -535,7 +540,7 @@
 
 def otherparent(self, f):
 '''Mark as coming from the other parent, always dirty.'''
-if self._pl[1] == self._nodeconstants.nullid:
+if not self.in_merge:
 msg = _(b"setting %r to other parent only allowed in merges") % f
 raise error.Abort(msg)
 if f in self and self[f] == b'n':
@@ -556,7 +561,7 @@
 self._dirty = True
 oldstate = self[f]
 size = 0
-if self._pl[1] != self._nodeconstants.nullid:
+if self.in_merge:
 entry = self._map.get(f)
 if entry is not None:
 # backup the previous state
@@ -572,7 +577,7 @@
 
 def merge(self, f):
 '''Mark a file merged.'''
-if self._pl[1] == self._nodeconstants.nullid:
+if not self.in_merge:
 return self.normallookup(f)
 return self.otherparent(f)
 



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10951: dirstate-entry: introduce dedicated accessors for v1 serialization

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  In the spirit of changing the content and storage of the dirstate entry, we 
add
  new method that the code doing v1 serialisation can use.
  
  Adding such method to the C object is quite trivial.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10951

AFFECTED FILES
  mercurial/cext/parsers.c
  mercurial/pure/parsers.py

CHANGE DETAILS

diff --git a/mercurial/pure/parsers.py b/mercurial/pure/parsers.py
--- a/mercurial/pure/parsers.py
+++ b/mercurial/pure/parsers.py
@@ -64,6 +64,22 @@
 else:
 raise IndexError(idx)
 
+def v1_state(self):
+"""return a "state" suitable for v1 serialization"""
+return self._state
+
+def v1_mode(self):
+"""return a "mode" suitable for v1 serialization"""
+return self._mode
+
+def v1_size(self):
+"""return a "size" suitable for v1 serialization"""
+return self._size
+
+def v1_mtime(self):
+"""return a "mtime" suitable for v1 serialization"""
+return self._mtime
+
 
 def gettype(q):
 return int(q & 0x)
@@ -450,7 +466,14 @@
 
 if f in copymap:
 f = b"%s\0%s" % (f, copymap[f])
-e = _pack(b">c", e[0], e[1], e[2], e[3], len(f))
+e = _pack(
+b">c",
+e.v1_state(),
+e.v1_mode(),
+e.v1_size(),
+e.v1_mtime(),
+len(f),
+)
 write(e)
 write(f)
 return cs.getvalue()
diff --git a/mercurial/cext/parsers.c b/mercurial/cext/parsers.c
--- a/mercurial/cext/parsers.c
+++ b/mercurial/cext/parsers.c
@@ -118,6 +118,38 @@
 0  /* sq_inplace_repeat */
 };
 
+static PyObject *dirstatetuple_v1_state(dirstateTupleObject *self)
+{
+   return PyBytes_FromStringAndSize(>state, 1);
+};
+
+static PyObject *dirstatetuple_v1_mode(dirstateTupleObject *self)
+{
+   return PyInt_FromLong(self->mode);
+};
+
+static PyObject *dirstatetuple_v1_size(dirstateTupleObject *self)
+{
+   return PyInt_FromLong(self->size);
+};
+
+static PyObject *dirstatetuple_v1_mtime(dirstateTupleObject *self)
+{
+   return PyInt_FromLong(self->mtime);
+};
+
+static PyMethodDef dirstatetuple_methods[] = {
+{"v1_state", (PyCFunction)dirstatetuple_v1_state, METH_NOARGS,
+ "return a \"state\" suitable for v1 serialization"},
+{"v1_mode", (PyCFunction)dirstatetuple_v1_mode, METH_NOARGS,
+ "return a \"mode\" suitable for v1 serialization"},
+{"v1_size", (PyCFunction)dirstatetuple_v1_size, METH_NOARGS,
+ "return a \"size\" suitable for v1 serialization"},
+{"v1_mtime", (PyCFunction)dirstatetuple_v1_mtime, METH_NOARGS,
+ "return a \"mtime\" suitable for v1 serialization"},
+{NULL} /* Sentinel */
+};
+
 PyTypeObject dirstateTupleType = {
 PyVarObject_HEAD_INIT(NULL, 0)  /* header */
 "dirstate_tuple",   /* tp_name */
@@ -146,7 +178,7 @@
 0,  /* tp_weaklistoffset */
 0,  /* tp_iter */
 0,  /* tp_iternext */
-0,  /* tp_methods */
+dirstatetuple_methods,  /* tp_methods */
 0,  /* tp_members */
 0,  /* tp_getset */
 0,  /* tp_base */



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10950: dirstate-entry: goes through the `dirstatetuple` constructor in all cases

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  We need to make sure we build an object.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10950

AFFECTED FILES
  mercurial/pure/parsers.py

CHANGE DETAILS

diff --git a/mercurial/pure/parsers.py b/mercurial/pure/parsers.py
--- a/mercurial/pure/parsers.py
+++ b/mercurial/pure/parsers.py
@@ -425,7 +425,7 @@
 if b'\0' in f:
 f, c = f.split(b'\0')
 copymap[f] = c
-dmap[f] = e[:4]
+dmap[f] = dirstatetuple(*e[:4])
 return parents
 
 



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10949: dirstate-entry: turn dirstate tuple into a real object (like in C)

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  With dirstate V2, the stored information and actual format will change. This 
mean we need to start an a better abstraction for a dirstate entry that a tuple 
directly accessed.
  
  By chance, the C code is already doing this and pretend to be a tuple. So it
  should be fairly easy. We start with turning the tuple into an object, we will
  slowly migrate the dirstate code to no longer use the tuple directly in later
  changesets.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10949

AFFECTED FILES
  mercurial/pure/parsers.py

CHANGE DETAILS

diff --git a/mercurial/pure/parsers.py b/mercurial/pure/parsers.py
--- a/mercurial/pure/parsers.py
+++ b/mercurial/pure/parsers.py
@@ -32,18 +32,37 @@
 _compress = zlib.compress
 _decompress = zlib.decompress
 
-# Some code below makes tuples directly because it's more convenient. However,
-# code outside this module should always use dirstatetuple.
-def dirstatetuple(*x):
-"""the four items are:
+
+class dirstatetuple(object):
+"""represente a dirstate entry
+
+It constains:
+
 - state (one of 'n', 'a', 'r', 'm')
 - mode,
 - size,
 - mtime,
 """
 
-# x is a tuple
-return x
+__slot__ = ('_state', '_mode', '_size', '_mtime')
+
+def __init__(self, state, mode, size, mtime):
+self._state = state
+self._mode = mode
+self._size = size
+self._mtime = mtime
+
+def __getitem__(self, idx):
+if idx == 0 or idx == -4:
+return self._state
+elif idx == 1 or idx == -3:
+return self._mode
+elif idx == 2 or idx == -2:
+return self._size
+elif idx == 3 or idx == -1:
+return self._mtime
+else:
+raise IndexError(idx)
 
 
 def gettype(q):



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10947: test: use a python script in `test-transaction-rollback-on-sigpipe.t`

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This still does not work on Windows, but at least this is a python script now.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10947

AFFECTED FILES
  tests/test-transaction-rollback-on-sigpipe.t
  tests/testlib/sigpipe-remote.py

CHANGE DETAILS

diff --git a/tests/testlib/sigpipe-remote.py b/tests/testlib/sigpipe-remote.py
new file mode 100755
--- /dev/null
+++ b/tests/testlib/sigpipe-remote.py
@@ -0,0 +1,176 @@
+#!/usr/bin/env -S python3 -u
+from __future__ import print_function
+
+import os
+import subprocess
+import sys
+import threading
+import time
+
+# we cannot use mercurial.testing as long as python2 is not dropped as the 
test will only install the mercurial module for python2 in python2 run
+
+
+def _timeout_factor():
+"""return the current modification to timeout"""
+default = int(os.environ.get('HGTEST_TIMEOUT_DEFAULT', 360))
+current = int(os.environ.get('HGTEST_TIMEOUT', default))
+if current == 0:
+return 1
+return current / float(default)
+
+
+def wait_file(path, timeout=10):
+timeout *= _timeout_factor()
+start = time.time()
+while not os.path.exists(path):
+if (time.time() - start) > timeout:
+raise RuntimeError(b"timed out waiting for file: %s" % path)
+time.sleep(0.01)
+
+
+def write_file(path, content=b''):
+with open(path, 'wb') as f:
+f.write(content)
+
+
+# end of mercurial.testing content
+
+if sys.version_info[0] < 3:
+print('SIGPIPE-HELPER: script should run with Python 3', file=sys.stderr)
+sys.exit(255)
+
+
+def sysbytes(s):
+return s.encode('utf-8')
+
+
+def sysstr(s):
+return s.decode('latin-1')
+
+
+piped_stdout = os.pipe2(os.O_NONBLOCK | os.O_CLOEXEC)
+piped_stderr = os.pipe2(os.O_NONBLOCK | os.O_CLOEXEC)
+
+stdout_writer = os.fdopen(piped_stdout[1], "rb")
+stdout_reader = os.fdopen(piped_stdout[0], "rb")
+stderr_writer = os.fdopen(piped_stderr[1], "rb")
+stderr_reader = os.fdopen(piped_stderr[0], "rb")
+
+DEBUG_FILE = os.environ.get('SIGPIPE_REMOTE_DEBUG_FILE')
+if DEBUG_FILE is None:
+debug_stream = sys.stderr.buffer
+else:
+debug_stream = open(DEBUG_FILE, 'bw', buffering=0)
+
+SYNCFILE1 = os.environ.get('SYNCFILE1')
+SYNCFILE2 = os.environ.get('SYNCFILE2')
+if SYNCFILE1 is None:
+print('SIGPIPE-HELPER: missing variable $SYNCFILE1', file=sys.stderr)
+sys.exit(255)
+if SYNCFILE2 is None:
+print('SIGPIPE-HELPER: missing variable $SYNCFILE2', file=sys.stderr)
+sys.exit(255)
+
+debug_stream.write(b'SIGPIPE-HELPER: Starting\n')
+
+TESTLIB_DIR = os.path.dirname(sys.argv[0])
+WAIT_SCRIPT = os.path.join(TESTLIB_DIR, 'wait-on-file')
+
+hooks_cmd = '%s 10 %s %s'
+hooks_cmd %= (
+WAIT_SCRIPT,
+SYNCFILE2,
+SYNCFILE1,
+)
+
+cmd = ['hg']
+cmd += sys.argv[1:]
+sub = subprocess.Popen(
+cmd,
+bufsize=0,
+close_fds=True,
+stdin=sys.stdin,
+stdout=stdout_writer,
+stderr=stderr_writer,
+)
+
+debug_stream.write(b'SIGPIPE-HELPER: Mercurial started\n')
+
+
+shut_down = threading.Event()
+
+close_lock = threading.Lock()
+
+
+def _read(stream):
+try:
+return stream.read()
+except ValueError:
+# read on closed file
+return None
+
+
+def forward_stdout():
+while not shut_down.is_set():
+c = _read(stdout_reader)
+while c is not None:
+sys.stdout.buffer.write(c)
+c = _read(stdout_reader)
+time.sleep(0.001)
+with close_lock:
+if not stdout_reader.closed:
+stdout_reader.close()
+debug_stream.write(b'SIGPIPE-HELPER: stdout closed\n')
+
+
+def forward_stderr():
+while not shut_down.is_set():
+c = _read(stderr_reader)
+if c is not None:
+sys.stderr.buffer.write(c)
+c = _read(stderr_reader)
+time.sleep(0.001)
+with close_lock:
+if not stderr_reader.closed:
+stderr_reader.close()
+debug_stream.write(b'SIGPIPE-HELPER: stderr closed\n')
+
+
+stdout_thread = threading.Thread(target=forward_stdout, daemon=True)
+stderr_thread = threading.Thread(target=forward_stderr, daemon=True)
+
+try:
+stdout_thread.start()
+stderr_thread.start()
+
+debug_stream.write(b'SIGPIPE-HELPER: Redirection in place\n')
+
+try:
+wait_file(sysbytes(SYNCFILE1))
+except RuntimeError as exc:
+msg = sysbytes(str(exc))
+debug_stream.write(b'SIGPIPE-HELPER: wait failed: %s\n' % msg)
+else:
+debug_stream.write(b'SIGPIPE-HELPER: SYNCFILE1 detected\n')
+with close_lock:
+if not stdout_reader.closed:
+stdout_reader.close()
+if not stderr_reader.closed:
+stderr_reader.close()
+sys.stdin.close()
+debug_stream.write(b'SIGPIPE-HELPER: pipes closed\n')
+debug_stream.write(b'SIGPIPE-HELPER: 

D10946: test: make sure we hit the SIGPIPE in test-transaction-rollback-on-sigpipe

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  With a coming change, we might not hit the sig pipe without issue and extra 
line
  of output. We do this early to make the next change clearer.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10946

AFFECTED FILES
  tests/test-transaction-rollback-on-sigpipe.t

CHANGE DETAILS

diff --git a/tests/test-transaction-rollback-on-sigpipe.t 
b/tests/test-transaction-rollback-on-sigpipe.t
--- a/tests/test-transaction-rollback-on-sigpipe.t
+++ b/tests/test-transaction-rollback-on-sigpipe.t
@@ -47,7 +47,8 @@
 
   $ cat >remote/.hg/hgrc < [hooks]
-  > pretxnchangegroup.break-things=$hook_script
+  > pretxnchangegroup.00-break-things=$hook_script
+  > pretxnchangegroup.01-output-things=echo "some remote output to be forward 
to the closed pipe"
   > EOF
 
   $ hg --cwd ./remote tip -T '{node|short}\n'



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10945: test: clarify some output in `test-transaction-rollback-on-sigpipe`

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  We don't want a dirty transaction to remain, but need to check the transaction
  was aborted. Otherwise it is easy to have no abandoned transaction if the
  transaction succeed.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10945

AFFECTED FILES
  tests/test-transaction-rollback-on-sigpipe.t

CHANGE DETAILS

diff --git a/tests/test-transaction-rollback-on-sigpipe.t 
b/tests/test-transaction-rollback-on-sigpipe.t
--- a/tests/test-transaction-rollback-on-sigpipe.t
+++ b/tests/test-transaction-rollback-on-sigpipe.t
@@ -50,12 +50,16 @@
   > pretxnchangegroup.break-things=$hook_script
   > EOF
 
+  $ hg --cwd ./remote tip -T '{node|short}\n'
+  
   $ cd local
   $ echo foo > foo ; hg commit -qAm "commit"
   $ hg push -q -e "\"$PYTHON\" \"$TESTDIR/dummyssh\"" --remotecmd $remotecmd 
2>&1 | grep -v $killable_pipe
   abort: stream ended unexpectedly (got 0 bytes, expected 4)
 
 The remote should be left in a good state
+  $ hg --cwd ../remote tip -T '{node|short}\n'
+  
   $ hg --cwd ../remote recover
   no interrupted transaction available
   [1]



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10943: run-tests: stop writing a `python3` symlink pointing to python2

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Having `python3` actually pointing to `python2` is bad. So we stop doing so.
  
  In addition we need to re-introduce a `python` executable since some of the
  script really need to be able to say "current python" in their shbang. For
  example, `hghave` is one of such script.
  
  The faulty changes where introduced by c102b704edb5 
.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10943

AFFECTED FILES
  tests/dumbhttp.py
  tests/dummysmtpd.py
  tests/get-with-headers.py
  tests/hghave
  tests/run-tests.py
  tests/test-filelog.py
  tests/test-remotefilelog-datapack.py
  tests/test-remotefilelog-histpack.py
  tests/test-status-inprocess.py
  tests/test-stdio.py
  tests/tinyproxy.py

CHANGE DETAILS

diff --git a/tests/tinyproxy.py b/tests/tinyproxy.py
--- a/tests/tinyproxy.py
+++ b/tests/tinyproxy.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python3
+#!/usr/bin/env python
 
 from __future__ import absolute_import, print_function
 
diff --git a/tests/test-stdio.py b/tests/test-stdio.py
--- a/tests/test-stdio.py
+++ b/tests/test-stdio.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python3
+#!/usr/bin/env python
 """
 Tests the buffering behavior of stdio streams in `mercurial.utils.procutil`.
 """
diff --git a/tests/test-status-inprocess.py b/tests/test-status-inprocess.py
--- a/tests/test-status-inprocess.py
+++ b/tests/test-status-inprocess.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python3
+#!/usr/bin/env python
 from __future__ import absolute_import, print_function
 
 import sys
diff --git a/tests/test-remotefilelog-histpack.py 
b/tests/test-remotefilelog-histpack.py
--- a/tests/test-remotefilelog-histpack.py
+++ b/tests/test-remotefilelog-histpack.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python3
+#!/usr/bin/env python
 from __future__ import absolute_import
 
 import hashlib
diff --git a/tests/test-remotefilelog-datapack.py 
b/tests/test-remotefilelog-datapack.py
--- a/tests/test-remotefilelog-datapack.py
+++ b/tests/test-remotefilelog-datapack.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python3
+#!/usr/bin/env python
 from __future__ import absolute_import, print_function
 
 import hashlib
diff --git a/tests/test-filelog.py b/tests/test-filelog.py
--- a/tests/test-filelog.py
+++ b/tests/test-filelog.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python3
+#!/usr/bin/env python
 """
 Tests the behavior of filelog w.r.t. data starting with '\1\n'
 """
diff --git a/tests/run-tests.py b/tests/run-tests.py
--- a/tests/run-tests.py
+++ b/tests/run-tests.py
@@ -3530,9 +3530,11 @@
 """Configure the environment to use the appropriate Python in tests."""
 # Tests must use the same interpreter as us or bad things will happen.
 if sys.platform == 'win32':
-pyexename = b'python.exe'
+pyexe_names = [b'python', b'python.exe']
+elif sys.version_info[0] < 3:
+pyexe_names = [b'python', b'python2']
 else:
-pyexename = b'python3'  # XXX this is wrong with python2...
+pyexe_names = [b'python', b'python3']
 
 # os.symlink() is a thing with py3 on Windows, but it requires
 # Administrator rights.
@@ -3540,7 +3542,7 @@
 msg = "# Making python executable in test path a symlink to '%s'"
 msg %= sysexecutable
 vlog(msg)
-for pyexename in [pyexename]:
+for pyexename in pyexe_names:
 mypython = os.path.join(self._tmpbindir, pyexename)
 try:
 if os.readlink(mypython) == sysexecutable:
@@ -3566,11 +3568,16 @@
 with open(osenvironb[b'RUNTESTDIR'] + b'/python3', 'wb') as f:
 f.write(b'#!/bin/sh\n')
 f.write(b'py -3.%d "$@"\n' % sys.version_info[1])
+if os.getenv('MSYSTEM'):
+with open(osenvironb[b'RUNTESTDIR'] + b'/python2', 'wb') as f:
+f.write(b'#!/bin/sh\n')
+f.write(b'py -2.%d "$@"\n' % sys.version_info[1])
 
 exedir, exename = os.path.split(sysexecutable)
-msg = "# Modifying search path to find %s as %s in '%s'"
-msg %= (exename, pyexename, exedir)
-vlog(msg)
+for pyexename in pyexe_names:
+msg = "# Modifying search path to find %s as %s in '%s'"
+msg %= (exename, pyexename, exedir)
+vlog(msg)
 path = os.environ['PATH'].split(os.pathsep)
 while exedir in path:
 path.remove(exedir)
@@ -3598,8 +3605,9 @@
 extra_paths.append(scripts_dir)
 
 os.environ['PATH'] = os.pathsep.join(extra_paths + path)
-if not self._findprogram(pyexename):
-print("WARNING: Cannot find %s in search path" % 

D10942: run-tests: avoid an early return

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  To fix the "python3 pointing to python2" we will also need to create a 
"python"
  pointer. So we will need to create multiple pointer. So we need to stop using
  early return.
  
  We replace the early return with a loop and a continue, since the next
  changeset will introduce that loop anyway.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10942

AFFECTED FILES
  tests/run-tests.py

CHANGE DETAILS

diff --git a/tests/run-tests.py b/tests/run-tests.py
--- a/tests/run-tests.py
+++ b/tests/run-tests.py
@@ -3540,22 +3540,23 @@
 msg = "# Making python executable in test path a symlink to '%s'"
 msg %= sysexecutable
 vlog(msg)
-mypython = os.path.join(self._tmpbindir, pyexename)
-try:
-if os.readlink(mypython) == sysexecutable:
-return
-os.unlink(mypython)
-except OSError as err:
-if err.errno != errno.ENOENT:
-raise
-if self._findprogram(pyexename) != sysexecutable:
+for pyexename in [pyexename]:
+mypython = os.path.join(self._tmpbindir, pyexename)
 try:
-os.symlink(sysexecutable, mypython)
-self._createdfiles.append(mypython)
+if os.readlink(mypython) == sysexecutable:
+continue
+os.unlink(mypython)
 except OSError as err:
-# child processes may race, which is harmless
-if err.errno != errno.EEXIST:
+if err.errno != errno.ENOENT:
 raise
+if self._findprogram(pyexename) != sysexecutable:
+try:
+os.symlink(sysexecutable, mypython)
+self._createdfiles.append(mypython)
+except OSError as err:
+# child processes may race, which is harmless
+if err.errno != errno.EEXIST:
+raise
 else:
 # Windows doesn't have `python3.exe`, and MSYS cannot understand 
the
 # reparse point with that name provided by Microsoft.  Create a



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10944: check-shbang: accept -S with env

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is useful to use `python3 -u` in a coming patch.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10944

AFFECTED FILES
  tests/test-check-shbang.t

CHANGE DETAILS

diff --git a/tests/test-check-shbang.t b/tests/test-check-shbang.t
--- a/tests/test-check-shbang.t
+++ b/tests/test-check-shbang.t
@@ -5,7 +5,7 @@
 
 look for python scripts that do not use /usr/bin/env
 
-  $ testrepohg files 'set:grep(r"^#!.*?python") and not 
grep(r"^#!/usr/bi{1}n/env python") - **/*.t'
+  $ testrepohg files 'set:grep(r"^#!.*?python") and not 
grep(r"^#!/usr/bi{1}n/env (-S )?python") - **/*.t'
   [1]
 
 In tests, enforce $PYTHON and *not* /usr/bin/env python or similar:



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10941: run-tests: clarify pyexename assignement

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  That assignement is wrong, we clarify that it is wrong before starting to fix
  it. This will make the fix clearer.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10941

AFFECTED FILES
  tests/run-tests.py

CHANGE DETAILS

diff --git a/tests/run-tests.py b/tests/run-tests.py
--- a/tests/run-tests.py
+++ b/tests/run-tests.py
@@ -3529,7 +3529,10 @@
 def _usecorrectpython(self):
 """Configure the environment to use the appropriate Python in tests."""
 # Tests must use the same interpreter as us or bad things will happen.
-pyexename = sys.platform == 'win32' and b'python.exe' or b'python3'
+if sys.platform == 'win32':
+pyexename = b'python.exe'
+else:
+pyexename = b'python3'  # XXX this is wrong with python2...
 
 # os.symlink() is a thing with py3 on Windows, but it requires
 # Administrator rights.



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10940: run-tests: factor appdata out

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is a gratuitous readability change.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10940

AFFECTED FILES
  tests/run-tests.py

CHANGE DETAILS

diff --git a/tests/run-tests.py b/tests/run-tests.py
--- a/tests/run-tests.py
+++ b/tests/run-tests.py
@@ -3575,9 +3575,10 @@
 # not be in PATH by default.
 extra_paths = [exedir]
 vi = sys.version_info
-if 'APPDATA' in os.environ:
+appdata = os.environ.get('APPDATA')
+if appdata is not None:
 scripts_dir = os.path.join(
-os.environ['APPDATA'],
+appdata,
 'Python',
 'Python%d%d' % (vi[0], vi[1]),
 'Scripts',
@@ -3585,7 +3586,7 @@
 
 if vi.major == 2:
 scripts_dir = os.path.join(
-os.environ['APPDATA'],
+appdata,
 'Python',
 'Scripts',
 )



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10939: run-tests: unroll a non-so-one-liner

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is shorter and clearer.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10939

AFFECTED FILES
  tests/run-tests.py

CHANGE DETAILS

diff --git a/tests/run-tests.py b/tests/run-tests.py
--- a/tests/run-tests.py
+++ b/tests/run-tests.py
@@ -3564,10 +3564,9 @@
 f.write(b'py -3.%d "$@"\n' % sys.version_info[1])
 
 exedir, exename = os.path.split(sysexecutable)
-vlog(
-"# Modifying search path to find %s as %s in '%s'"
-% (exename, pyexename, exedir)
-)
+msg = "# Modifying search path to find %s as %s in '%s'"
+msg %= (exename, pyexename, exedir)
+vlog(msg)
 path = os.environ['PATH'].split(os.pathsep)
 while exedir in path:
 path.remove(exedir)



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10938: run-tests: unroll a non-so-one-liner

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is shorter and clearer.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10938

AFFECTED FILES
  tests/run-tests.py

CHANGE DETAILS

diff --git a/tests/run-tests.py b/tests/run-tests.py
--- a/tests/run-tests.py
+++ b/tests/run-tests.py
@@ -3534,10 +3534,9 @@
 # os.symlink() is a thing with py3 on Windows, but it requires
 # Administrator rights.
 if getattr(os, 'symlink', None) and os.name != 'nt':
-vlog(
-"# Making python executable in test path a symlink to '%s'"
-% sysexecutable
-)
+msg = "# Making python executable in test path a symlink to '%s'"
+msg %= sysexecutable
+vlog(msg)
 mypython = os.path.join(self._tmpbindir, pyexename)
 try:
 if os.readlink(mypython) == sysexecutable:



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10937: testing: fix _timeout_factor

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  With `--debug`, `run-tests.py` set the timeout to 0... that breaks the logic 
in
  `mercurial.testing`.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10937

AFFECTED FILES
  mercurial/testing/__init__.py

CHANGE DETAILS

diff --git a/mercurial/testing/__init__.py b/mercurial/testing/__init__.py
--- a/mercurial/testing/__init__.py
+++ b/mercurial/testing/__init__.py
@@ -16,8 +16,10 @@
 
 def _timeout_factor():
 """return the current modification to timeout"""
-default = int(environ.get('HGTEST_TIMEOUT_DEFAULT', 1))
+default = int(environ.get('HGTEST_TIMEOUT_DEFAULT', 360))
 current = int(environ.get('HGTEST_TIMEOUT', default))
+if current == 0:
+return 1
 return current / float(default)
 
 



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10934: dirstate: split dirstatemap in its own file

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  The dirstate file is large enough and the dirstatemap is quite insulated logic
  already.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10934

AFFECTED FILES
  mercurial/dirstate.py
  mercurial/dirstatemap.py
  tests/fakedirstatewritetime.py

CHANGE DETAILS

diff --git a/tests/fakedirstatewritetime.py b/tests/fakedirstatewritetime.py
--- a/tests/fakedirstatewritetime.py
+++ b/tests/fakedirstatewritetime.py
@@ -10,6 +10,7 @@
 from mercurial import (
 context,
 dirstate,
+dirstatemap as dirstatemapmod,
 extensions,
 policy,
 registrar,
@@ -66,11 +67,11 @@
 if rustmod is not None:
 # The Rust implementation does not use public parse/pack dirstate
 # to prevent conversion round-trips
-orig_dirstatemap_write = dirstate.dirstatemap.write
+orig_dirstatemap_write = dirstatemapmod.dirstatemap.write
 wrapper = lambda self, st, now: orig_dirstatemap_write(
 self, st, fakenow
 )
-dirstate.dirstatemap.write = wrapper
+dirstatemapmod.dirstatemap.write = wrapper
 
 orig_dirstate_getfsnow = dirstate._getfsnow
 wrapper = lambda *args: pack_dirstate(fakenow, orig_pack_dirstate, *args)
@@ -86,7 +87,7 @@
 orig_module.pack_dirstate = orig_pack_dirstate
 dirstate._getfsnow = orig_dirstate_getfsnow
 if rustmod is not None:
-dirstate.dirstatemap.write = orig_dirstatemap_write
+dirstatemapmod.dirstatemap.write = orig_dirstatemap_write
 
 
 def _poststatusfixup(orig, workingctx, status, fixup):
diff --git a/mercurial/dirstatemap.py b/mercurial/dirstatemap.py
new file mode 100644
--- /dev/null
+++ b/mercurial/dirstatemap.py
@@ -0,0 +1,610 @@
+# dirstatemap.py
+#
+# This software may be used and distributed according to the terms of the
+# GNU General Public License version 2 or any later version.
+
+from __future__ import absolute_import
+
+import errno
+
+from .i18n import _
+
+from . import (
+error,
+pathutil,
+policy,
+pycompat,
+txnutil,
+util,
+)
+
+parsers = policy.importmod('parsers')
+rustmod = policy.importrust('dirstate')
+
+propertycache = util.propertycache
+
+dirstatetuple = parsers.dirstatetuple
+
+
+# a special value used internally for `size` if the file come from the other 
parent
+FROM_P2 = -2
+
+# a special value used internally for `size` if the file is 
modified/merged/added
+NONNORMAL = -1
+
+# a special value used internally for `time` if the time is ambigeous
+AMBIGUOUS_TIME = -1
+
+
+class dirstatemap(object):
+"""Map encapsulating the dirstate's contents.
+
+The dirstate contains the following state:
+
+- `identity` is the identity of the dirstate file, which can be used to
+  detect when changes have occurred to the dirstate file.
+
+- `parents` is a pair containing the parents of the working copy. The
+  parents are updated by calling `setparents`.
+
+- the state map maps filenames to tuples of (state, mode, size, mtime),
+  where state is a single character representing 'normal', 'added',
+  'removed', or 'merged'. It is read by treating the dirstate as a
+  dict.  File state is updated by calling the `addfile`, `removefile` and
+  `dropfile` methods.
+
+- `copymap` maps destination filenames to their source filename.
+
+The dirstate also provides the following views onto the state:
+
+- `nonnormalset` is a set of the filenames that have state other
+  than 'normal', or are normal but have an mtime of -1 ('normallookup').
+
+- `otherparentset` is a set of the filenames that are marked as coming
+  from the second parent when the dirstate is currently being merged.
+
+- `filefoldmap` is a dict mapping normalized filenames to the denormalized
+  form that they appear as in the dirstate.
+
+- `dirfoldmap` is a dict mapping normalized directory names to the
+  denormalized form that they appear as in the dirstate.
+"""
+
+def __init__(self, ui, opener, root, nodeconstants, use_dirstate_v2):
+self._ui = ui
+self._opener = opener
+self._root = root
+self._filename = b'dirstate'
+self._nodelen = 20
+self._nodeconstants = nodeconstants
+assert (
+not use_dirstate_v2
+), "should have detected unsupported requirement"
+
+self._parents = None
+self._dirtyparents = False
+
+# for consistent view between _pl() and _read() invocations
+self._pendingmode = None
+
+@propertycache
+def _map(self):
+self._map = {}
+self.read()
+return self._map
+
+@propertycache
+def copymap(self):
+self.copymap = {}
+self._map
+return self.copymap
+
+def directories(self):
+# Rust / dirstate-v2 only

D10933: dirstate: explicitely deal with the `added` case in `_addpath`

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This special case is now directly handled, this clarify the possible value for
  the other options.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10933

AFFECTED FILES
  mercurial/dirstate.py

CHANGE DETAILS

diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -464,17 +464,22 @@
 msg = _(b'file %r in dirstate clashes with %r')
 msg %= (pycompat.bytestr(d), pycompat.bytestr(f))
 raise error.Abort(msg)
-if from_p2:
+if state == b'a':
+assert not possibly_dirty
+assert not from_p2
+size = NONNORMAL
+mtime = AMBIGUOUS_TIME
+elif from_p2:
+assert not possibly_dirty
 size = FROM_P2
 mtime = AMBIGUOUS_TIME
 elif possibly_dirty:
 mtime = AMBIGUOUS_TIME
 else:
 assert size != FROM_P2
-if size != NONNORMAL:
-size = size & _rangemask
-if mtime != AMBIGUOUS_TIME:
-mtime = mtime & _rangemask
+assert size != NONNORMAL
+size = size & _rangemask
+mtime = mtime & _rangemask
 self._dirty = True
 self._updatedfiles.add(f)
 self._map.addfile(f, oldstate, state, mode, size, mtime)



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10932: dirstate: add an explicit `possibly_dirty` parameter to `_addpath`

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This lets says what we mean instead of using magic value. The lower level can
  then decide how to express that.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10932

AFFECTED FILES
  mercurial/dirstate.py

CHANGE DETAILS

diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -446,6 +446,7 @@
 size=NONNORMAL,
 mtime=AMBIGUOUS_TIME,
 from_p2=False,
+possibly_dirty=False,
 ):
 oldstate = self[f]
 if state == b'a' or oldstate == b'r':
@@ -466,6 +467,8 @@
 if from_p2:
 size = FROM_P2
 mtime = AMBIGUOUS_TIME
+elif possibly_dirty:
+mtime = AMBIGUOUS_TIME
 else:
 assert size != FROM_P2
 if size != NONNORMAL:
@@ -522,7 +525,7 @@
 return
 if entry[0] == b'm' or entry[0] == b'n' and entry[2] == 
FROM_P2:
 return
-self._addpath(f, b'n', 0)
+self._addpath(f, b'n', 0, possibly_dirty=True)
 self._map.copymap.pop(f, None)
 
 def otherparent(self, f):



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10935: dirstate: document the dirstatetuple content

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  We are about to modify this, so let us document the existing code.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10935

AFFECTED FILES
  mercurial/pure/parsers.py

CHANGE DETAILS

diff --git a/mercurial/pure/parsers.py b/mercurial/pure/parsers.py
--- a/mercurial/pure/parsers.py
+++ b/mercurial/pure/parsers.py
@@ -35,6 +35,13 @@
 # Some code below makes tuples directly because it's more convenient. However,
 # code outside this module should always use dirstatetuple.
 def dirstatetuple(*x):
+"""the four items are:
+- state (one of 'n', 'a', 'r', 'm')
+- mode,
+- size,
+- mtime,
+"""
+
 # x is a tuple
 return x
 



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10930: dirstate: add default value to _addpath

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  We want to remove the magic value usage from the higher level, so lets stop
  passing them explicitely when possible.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10930

AFFECTED FILES
  mercurial/dirstate.py

CHANGE DETAILS

diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -438,7 +438,7 @@
 def copies(self):
 return self._map.copymap
 
-def _addpath(self, f, state, mode, size, mtime):
+def _addpath(self, f, state, mode, size=NONNORMAL, mtime=AMBIGUOUS_TIME):
 oldstate = self[f]
 if state == b'a' or oldstate == b'r':
 scmutil.checkfilename(f)
@@ -509,7 +509,7 @@
 return
 if entry[0] == b'm' or entry[0] == b'n' and entry[2] == 
FROM_P2:
 return
-self._addpath(f, b'n', 0, NONNORMAL, AMBIGUOUS_TIME)
+self._addpath(f, b'n', 0)
 self._map.copymap.pop(f, None)
 
 def otherparent(self, f):
@@ -519,15 +519,15 @@
 raise error.Abort(msg)
 if f in self and self[f] == b'n':
 # merge-like
-self._addpath(f, b'm', 0, FROM_P2, AMBIGUOUS_TIME)
+self._addpath(f, b'm', 0, FROM_P2)
 else:
 # add-like
-self._addpath(f, b'n', 0, FROM_P2, AMBIGUOUS_TIME)
+self._addpath(f, b'n', 0, FROM_P2)
 self._map.copymap.pop(f, None)
 
 def add(self, f):
 '''Mark a file added.'''
-self._addpath(f, b'a', 0, NONNORMAL, AMBIGUOUS_TIME)
+self._addpath(f, b'a', 0)
 self._map.copymap.pop(f, None)
 
 def remove(self, f):



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10931: dirstate: add an explicit `from_p2` parameter to `_addpath`

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This lets says what we mean instead of using magic value. The lower level can
  then decide how to express that.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10931

AFFECTED FILES
  mercurial/dirstate.py

CHANGE DETAILS

diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -438,7 +438,15 @@
 def copies(self):
 return self._map.copymap
 
-def _addpath(self, f, state, mode, size=NONNORMAL, mtime=AMBIGUOUS_TIME):
+def _addpath(
+self,
+f,
+state,
+mode,
+size=NONNORMAL,
+mtime=AMBIGUOUS_TIME,
+from_p2=False,
+):
 oldstate = self[f]
 if state == b'a' or oldstate == b'r':
 scmutil.checkfilename(f)
@@ -455,10 +463,15 @@
 msg = _(b'file %r in dirstate clashes with %r')
 msg %= (pycompat.bytestr(d), pycompat.bytestr(f))
 raise error.Abort(msg)
-if size != NONNORMAL and size != FROM_P2:
-size = size & _rangemask
-if mtime != AMBIGUOUS_TIME:
-mtime = mtime & _rangemask
+if from_p2:
+size = FROM_P2
+mtime = AMBIGUOUS_TIME
+else:
+assert size != FROM_P2
+if size != NONNORMAL:
+size = size & _rangemask
+if mtime != AMBIGUOUS_TIME:
+mtime = mtime & _rangemask
 self._dirty = True
 self._updatedfiles.add(f)
 self._map.addfile(f, oldstate, state, mode, size, mtime)
@@ -519,10 +532,10 @@
 raise error.Abort(msg)
 if f in self and self[f] == b'n':
 # merge-like
-self._addpath(f, b'm', 0, FROM_P2)
+self._addpath(f, b'm', 0, from_p2=True)
 else:
 # add-like
-self._addpath(f, b'n', 0, FROM_P2)
+self._addpath(f, b'n', 0, from_p2=True)
 self._map.copymap.pop(f, None)
 
 def add(self, f):



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10929: dirstate: move the _rangemask filtering closer to its storage

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  The goal is to get ready to move these kind of processing at a lower level. 
We start with move
  move _rangemask filtering close to where it is sent to the lower level to 
make the future move trivial.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10929

AFFECTED FILES
  mercurial/dirstate.py

CHANGE DETAILS

diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -455,6 +455,10 @@
 msg = _(b'file %r in dirstate clashes with %r')
 msg %= (pycompat.bytestr(d), pycompat.bytestr(f))
 raise error.Abort(msg)
+if size != NONNORMAL and size != FROM_P2:
+size = size & _rangemask
+if mtime != AMBIGUOUS_TIME:
+mtime = mtime & _rangemask
 self._dirty = True
 self._updatedfiles.add(f)
 self._map.addfile(f, oldstate, state, mode, size, mtime)
@@ -476,7 +480,7 @@
 mode = s.st_mode
 size = s.st_size
 mtime = s[stat.ST_MTIME]
-self._addpath(f, b'n', mode, size & _rangemask, mtime & _rangemask)
+self._addpath(f, b'n', mode, size, mtime)
 self._map.copymap.pop(f, None)
 if f in self._map.nonnormalset:
 self._map.nonnormalset.remove(f)



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10928: dirstate: introduce a symbolic constant for the AMBIGUOUS_TIME marker

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is going to be clearer and easier to track than -1. Ultimately I would
  like to get ride of this special value everywhere but in the lower level,
  however we need to clarify the API first. This changeset is part of such
  clarification.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10928

AFFECTED FILES
  mercurial/dirstate.py

CHANGE DETAILS

diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -54,6 +54,9 @@
 # a special value used internally for `size` if the file is 
modified/merged/added
 NONNORMAL = -1
 
+# a special value used internally for `time` if the time is ambigeous
+AMBIGUOUS_TIME = -1
+
 
 class repocache(filecache):
 """filecache for files in .hg/"""
@@ -502,7 +505,7 @@
 return
 if entry[0] == b'm' or entry[0] == b'n' and entry[2] == 
FROM_P2:
 return
-self._addpath(f, b'n', 0, NONNORMAL, -1)
+self._addpath(f, b'n', 0, NONNORMAL, AMBIGUOUS_TIME)
 self._map.copymap.pop(f, None)
 
 def otherparent(self, f):
@@ -512,15 +515,15 @@
 raise error.Abort(msg)
 if f in self and self[f] == b'n':
 # merge-like
-self._addpath(f, b'm', 0, FROM_P2, -1)
+self._addpath(f, b'm', 0, FROM_P2, AMBIGUOUS_TIME)
 else:
 # add-like
-self._addpath(f, b'n', 0, FROM_P2, -1)
+self._addpath(f, b'n', 0, FROM_P2, AMBIGUOUS_TIME)
 self._map.copymap.pop(f, None)
 
 def add(self, f):
 '''Mark a file added.'''
-self._addpath(f, b'a', 0, NONNORMAL, -1)
+self._addpath(f, b'a', 0, NONNORMAL, AMBIGUOUS_TIME)
 self._map.copymap.pop(f, None)
 
 def remove(self, f):
@@ -1537,7 +1540,7 @@
 if oldstate == b"?" and "_alldirs" in self.__dict__:
 self._alldirs.addpath(f)
 self._map[f] = dirstatetuple(state, mode, size, mtime)
-if state != b'n' or mtime == -1:
+if state != b'n' or mtime == AMBIGUOUS_TIME:
 self.nonnormalset.add(f)
 if size == FROM_P2:
 self.otherparentset.add(f)
@@ -1581,7 +1584,7 @@
 for f in files:
 e = self.get(f)
 if e is not None and e[0] == b'n' and e[3] == now:
-self._map[f] = dirstatetuple(e[0], e[1], e[2], -1)
+self._map[f] = dirstatetuple(e[0], e[1], e[2], AMBIGUOUS_TIME)
 self.nonnormalset.add(f)
 
 def nonnormalentries(self):
@@ -1592,7 +1595,7 @@
 nonnorm = set()
 otherparent = set()
 for fname, e in pycompat.iteritems(self._map):
-if e[0] != b'n' or e[3] == -1:
+if e[0] != b'n' or e[3] == AMBIGUOUS_TIME:
 nonnorm.add(fname)
 if e[0] == b'n' and e[2] == FROM_P2:
 otherparent.add(fname)



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10927: dirstate: introduce a symbolic constant for the NONNORMAL marker

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is going to be clearer and easier to track than -1. Ultimately I would
  like to get ride of this special value everywhere but in the lower level,
  however we need to clarify the API first. This changeset is part of such
  clarification.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10927

AFFECTED FILES
  mercurial/dirstate.py

CHANGE DETAILS

diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -51,6 +51,9 @@
 # a special value used internally for `size` if the file come from the other 
parent
 FROM_P2 = -2
 
+# a special value used internally for `size` if the file is 
modified/merged/added
+NONNORMAL = -1
+
 
 class repocache(filecache):
 """filecache for files in .hg/"""
@@ -488,9 +491,9 @@
 # being removed, restore that state.
 entry = self._map.get(f)
 if entry is not None:
-if entry[0] == b'r' and entry[2] in (-1, FROM_P2):
+if entry[0] == b'r' and entry[2] in (NONNORMAL, FROM_P2):
 source = self._map.copymap.get(f)
-if entry[2] == -1:
+if entry[2] == NONNORMAL:
 self.merge(f)
 elif entry[2] == FROM_P2:
 self.otherparent(f)
@@ -499,7 +502,7 @@
 return
 if entry[0] == b'm' or entry[0] == b'n' and entry[2] == 
FROM_P2:
 return
-self._addpath(f, b'n', 0, -1, -1)
+self._addpath(f, b'n', 0, NONNORMAL, -1)
 self._map.copymap.pop(f, None)
 
 def otherparent(self, f):
@@ -517,7 +520,7 @@
 
 def add(self, f):
 '''Mark a file added.'''
-self._addpath(f, b'a', 0, -1, -1)
+self._addpath(f, b'a', 0, NONNORMAL, -1)
 self._map.copymap.pop(f, None)
 
 def remove(self, f):
@@ -530,7 +533,7 @@
 if entry is not None:
 # backup the previous state
 if entry[0] == b'm':  # merge
-size = -1
+size = NONNORMAL
 elif entry[0] == b'n' and entry[2] == FROM_P2:  # other parent
 size = FROM_P2
 self._map.otherparentset.add(f)



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10926: dirstate: introduce a symbolic constant for the FROM_P2 marker

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is going to be clearer and easier to track than -2. Ultimately I would
  like to get ride of this special value everywhere but in the lower level,
  however we need to clarify the API first. This changeset is part of such
  clarification.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10926

AFFECTED FILES
  mercurial/dirstate.py

CHANGE DETAILS

diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -48,6 +48,10 @@
 dirstatetuple = parsers.dirstatetuple
 
 
+# a special value used internally for `size` if the file come from the other 
parent
+FROM_P2 = -2
+
+
 class repocache(filecache):
 """filecache for files in .hg/"""
 
@@ -371,7 +375,7 @@
 copies[f] = source
 self.normallookup(f)
 # Also fix up otherparent markers
-elif s[0] == b'n' and s[2] == -2:
+elif s[0] == b'n' and s[2] == FROM_P2:
 source = self._map.copymap.get(f)
 if source:
 copies[f] = source
@@ -484,16 +488,16 @@
 # being removed, restore that state.
 entry = self._map.get(f)
 if entry is not None:
-if entry[0] == b'r' and entry[2] in (-1, -2):
+if entry[0] == b'r' and entry[2] in (-1, FROM_P2):
 source = self._map.copymap.get(f)
 if entry[2] == -1:
 self.merge(f)
-elif entry[2] == -2:
+elif entry[2] == FROM_P2:
 self.otherparent(f)
 if source:
 self.copy(source, f)
 return
-if entry[0] == b'm' or entry[0] == b'n' and entry[2] == -2:
+if entry[0] == b'm' or entry[0] == b'n' and entry[2] == 
FROM_P2:
 return
 self._addpath(f, b'n', 0, -1, -1)
 self._map.copymap.pop(f, None)
@@ -505,10 +509,10 @@
 raise error.Abort(msg)
 if f in self and self[f] == b'n':
 # merge-like
-self._addpath(f, b'm', 0, -2, -1)
+self._addpath(f, b'm', 0, FROM_P2, -1)
 else:
 # add-like
-self._addpath(f, b'n', 0, -2, -1)
+self._addpath(f, b'n', 0, FROM_P2, -1)
 self._map.copymap.pop(f, None)
 
 def add(self, f):
@@ -527,8 +531,8 @@
 # backup the previous state
 if entry[0] == b'm':  # merge
 size = -1
-elif entry[0] == b'n' and entry[2] == -2:  # other parent
-size = -2
+elif entry[0] == b'n' and entry[2] == FROM_P2:  # other parent
+size = FROM_P2
 self._map.otherparentset.add(f)
 self._updatedfiles.add(f)
 self._map.removefile(f, oldstate, size)
@@ -1302,7 +1306,7 @@
 (size != st.st_size and size != st.st_size & 
_rangemask)
 or ((mode ^ st.st_mode) & 0o100 and checkexec)
 )
-or size == -2  # other parent
+or size == FROM_P2  # other parent
 or fn in copymap
 ):
 if stat.S_ISLNK(st.st_mode) and size != st.st_size:
@@ -1532,7 +1536,7 @@
 self._map[f] = dirstatetuple(state, mode, size, mtime)
 if state != b'n' or mtime == -1:
 self.nonnormalset.add(f)
-if size == -2:
+if size == FROM_P2:
 self.otherparentset.add(f)
 
 def removefile(self, f, oldstate, size):
@@ -1587,7 +1591,7 @@
 for fname, e in pycompat.iteritems(self._map):
 if e[0] != b'n' or e[3] == -1:
 nonnorm.add(fname)
-if e[0] == b'n' and e[2] == -2:
+if e[0] == b'n' and e[2] == FROM_P2:
 otherparent.add(fname)
 return nonnorm, otherparent
 



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10925: dirstate: split a not-so-one-liner

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is shorter and simpler to read.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10925

AFFECTED FILES
  mercurial/dirstate.py

CHANGE DETAILS

diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -442,10 +442,9 @@
 break
 entry = self._map.get(d)
 if entry is not None and entry[0] != b'r':
-raise error.Abort(
-_(b'file %r in dirstate clashes with %r')
-% (pycompat.bytestr(d), pycompat.bytestr(f))
-)
+msg = _(b'file %r in dirstate clashes with %r')
+msg %= (pycompat.bytestr(d), pycompat.bytestr(f))
+raise error.Abort(msg)
 self._dirty = True
 self._updatedfiles.add(f)
 self._map.addfile(f, oldstate, state, mode, size, mtime)



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10924: dirstate: split a not-so-one-liner

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is simpler to read.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10924

AFFECTED FILES
  mercurial/dirstate.py

CHANGE DETAILS

diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -433,9 +433,9 @@
 if state == b'a' or oldstate == b'r':
 scmutil.checkfilename(f)
 if self._map.hastrackeddir(f):
-raise error.Abort(
-_(b'directory %r already in dirstate') % 
pycompat.bytestr(f)
-)
+msg = _(b'directory %r already in dirstate')
+msg %= pycompat.bytestr(f)
+raise error.Abort(msg)
 # shadows
 for d in pathutil.finddirs(f):
 if self._map.hastrackeddir(d):



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10923: dirstate: split a not-so-one-liner

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is shorter and simpler to read.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10923

AFFECTED FILES
  mercurial/dirstate.py

CHANGE DETAILS

diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -502,9 +502,8 @@
 def otherparent(self, f):
 '''Mark as coming from the other parent, always dirty.'''
 if self._pl[1] == self._nodeconstants.nullid:
-raise error.Abort(
-_(b"setting %r to other parent only allowed in merges") % f
-)
+msg = _(b"setting %r to other parent only allowed in merges") % f
+raise error.Abort(msg)
 if f in self and self[f] == b'n':
 # merge-like
 self._addpath(f, b'm', 0, -2, -1)



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10905: stream: double check that self.vfs is *not* in the vfsmap

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  The stream clone logic allows for writing any content to any file under 
various
  vfs. This is *not* suitable for *vfs*, since writing in `.hg/` directly allow 
to
  modify the configuration and is a great and simple gateway for remote code
  execution.

REPOSITORY
  rHG Mercurial

BRANCH
  stable

REVISION DETAIL
  https://phab.mercurial-scm.org/D10905

AFFECTED FILES
  mercurial/streamclone.py

CHANGE DETAILS

diff --git a/mercurial/streamclone.py b/mercurial/streamclone.py
--- a/mercurial/streamclone.py
+++ b/mercurial/streamclone.py
@@ -560,6 +560,12 @@
 def _emit2(repo, entries, totalfilesize):
 """actually emit the stream bundle"""
 vfsmap = _makemap(repo)
+# we keep repo.vfs out of the on purpose, ther are too many danger there
+# (eg: .hg/hgrc),
+#
+# this assert is duplicated (from _makemap) as author might think this is
+# fine, while this is really not fine.
+assert repo.vfs not in vfsmap.values()
 progress = repo.ui.makeprogress(
 _(b'bundle'), total=totalfilesize, unit=_(b'bytes')
 )
@@ -685,6 +691,12 @@
 progress.update(0)
 
 vfsmap = _makemap(repo)
+# we keep repo.vfs out of the on purpose, ther are too many danger
+# there (eg: .hg/hgrc),
+#
+# this assert is duplicated (from _makemap) as author might think this
+# is fine, while this is really not fine.
+assert repo.vfs not in vfsmap.values()
 
 with repo.transaction(b'clone'):
 ctxs = (vfs.backgroundclosing(repo.ui) for vfs in vfsmap.values())



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10904: censor: split the core of the logic into its own function

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  We now have a "generic" rewrite function (only able to do censoring for now)
  and a thin wrapper that implement the `censor` API with it.
  
  We are now ready to start incorporating strip specific changes.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10904

AFFECTED FILES
  mercurial/revlogutils/rewrite.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/rewrite.py b/mercurial/revlogutils/rewrite.py
--- a/mercurial/revlogutils/rewrite.py
+++ b/mercurial/revlogutils/rewrite.py
@@ -127,26 +127,35 @@
 
 def v2_censor(revlog, tr, censornode, tombstone=b''):
 """censors a revision in a "version 2" revlog"""
-# General principle
-#
-# We create new revlog files (index/data/sidedata) to copy the content of
-# the existing data without the censored data.
-#
-# We need to recompute new delta for any revision that used the censored
-# revision as delta base. As the cumulative size of the new delta may be
-# large, we store them in a temporary file until they are stored in their
-# final destination.
-#
-# All data before the censored data can be blindly copied. The rest needs
-# to be copied as we go and the associated index entry needs adjustement.
+assert revlog._format_version != REVLOGV0, revlog._format_version
+assert revlog._format_version != REVLOGV1, revlog._format_version
+
+censor_revs = {revlog.rev(censornode)}
+_rewrite_v2(revlog, tr, censor_revs, tombstone)
+
+
+def _rewrite_v2(revlog, tr, censor_revs, tombstone=b''):
+"""rewrite a revlog to censor some of its content
+
+General principle
 
+We create new revlog files (index/data/sidedata) to copy the content of
+the existing data without the censored data.
+
+We need to recompute new delta for any revision that used the censored
+revision as delta base. As the cumulative size of the new delta may be
+large, we store them in a temporary file until they are stored in their
+final destination.
+
+All data before the censored data can be blindly copied. The rest needs
+to be copied as we go and the associated index entry needs adjustement.
+"""
 assert revlog._format_version != REVLOGV0, revlog._format_version
 assert revlog._format_version != REVLOGV1, revlog._format_version
 
 old_index = revlog.index
 docket = revlog._docket
 
-censor_revs = {revlog.rev(censornode)}
 tombstone = storageutil.packmeta({b'censored': tombstone}, b'')
 
 first_excl_rev = min(censor_revs)



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10903: censor: migrate the logic to a set of `censor_revs`

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Instead of considering a special unique censored revision within the code, we
  consider a set of revision (currently always of size 1). This make the main 
code
  less censor-centric and prepare for the usage of a similar approach for
  stripping changesets.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10903

AFFECTED FILES
  mercurial/revlogutils/rewrite.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/rewrite.py b/mercurial/revlogutils/rewrite.py
--- a/mercurial/revlogutils/rewrite.py
+++ b/mercurial/revlogutils/rewrite.py
@@ -146,20 +146,22 @@
 old_index = revlog.index
 docket = revlog._docket
 
-censor_rev = revlog.rev(censornode)
+censor_revs = {revlog.rev(censornode)}
 tombstone = storageutil.packmeta({b'censored': tombstone}, b'')
 
-censored_entry = revlog.index[censor_rev]
-index_cutoff = revlog.index.entry_size * censor_rev
-data_cutoff = censored_entry[ENTRY_DATA_OFFSET] >> 16
-sidedata_cutoff = revlog.sidedata_cut_off(censor_rev)
+first_excl_rev = min(censor_revs)
+
+first_excl_entry = revlog.index[first_excl_rev]
+index_cutoff = revlog.index.entry_size * first_excl_rev
+data_cutoff = first_excl_entry[ENTRY_DATA_OFFSET] >> 16
+sidedata_cutoff = revlog.sidedata_cut_off(first_excl_rev)
 
 with pycompat.unnamedtempfile(mode=b"w+b") as tmp_storage:
 # rev → (new_base, data_start, data_end, compression_mode)
 rewritten_entries = _precompute_rewritten_delta(
 revlog,
 old_index,
-{censor_rev},
+censor_revs,
 tmp_storage,
 )
 
@@ -182,24 +184,26 @@
 ) = open_files
 
 # writing the censored revision
-_rewrite_censor(
-revlog,
-old_index,
-open_files,
-censor_rev,
-tombstone,
-)
 
 # Writing all subsequent revisions
-for rev in range(censor_rev + 1, len(old_index)):
-_rewrite_simple(
-revlog,
-old_index,
-open_files,
-rev,
-rewritten_entries,
-tmp_storage,
-)
+for rev in range(first_excl_rev, len(old_index)):
+if rev in censor_revs:
+_rewrite_censor(
+revlog,
+old_index,
+open_files,
+rev,
+tombstone,
+)
+else:
+_rewrite_simple(
+revlog,
+old_index,
+open_files,
+rev,
+rewritten_entries,
+tmp_storage,
+)
 docket.write(transaction=None, stripping=True)
 
 



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10902: censor: rename `rl` to `revlog` in the main function

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Now that the bulk of the code moved to smaller function we are less restricted
  on line length and we can use more explicite naming.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10902

AFFECTED FILES
  mercurial/revlogutils/rewrite.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/rewrite.py b/mercurial/revlogutils/rewrite.py
--- a/mercurial/revlogutils/rewrite.py
+++ b/mercurial/revlogutils/rewrite.py
@@ -125,7 +125,7 @@
 rl._loadindex()
 
 
-def v2_censor(rl, tr, censornode, tombstone=b''):
+def v2_censor(revlog, tr, censornode, tombstone=b''):
 """censors a revision in a "version 2" revlog"""
 # General principle
 #
@@ -140,31 +140,31 @@
 # All data before the censored data can be blindly copied. The rest needs
 # to be copied as we go and the associated index entry needs adjustement.
 
-assert rl._format_version != REVLOGV0, rl._format_version
-assert rl._format_version != REVLOGV1, rl._format_version
+assert revlog._format_version != REVLOGV0, revlog._format_version
+assert revlog._format_version != REVLOGV1, revlog._format_version
 
-old_index = rl.index
-docket = rl._docket
+old_index = revlog.index
+docket = revlog._docket
 
-censor_rev = rl.rev(censornode)
+censor_rev = revlog.rev(censornode)
 tombstone = storageutil.packmeta({b'censored': tombstone}, b'')
 
-censored_entry = rl.index[censor_rev]
-index_cutoff = rl.index.entry_size * censor_rev
+censored_entry = revlog.index[censor_rev]
+index_cutoff = revlog.index.entry_size * censor_rev
 data_cutoff = censored_entry[ENTRY_DATA_OFFSET] >> 16
-sidedata_cutoff = rl.sidedata_cut_off(censor_rev)
+sidedata_cutoff = revlog.sidedata_cut_off(censor_rev)
 
 with pycompat.unnamedtempfile(mode=b"w+b") as tmp_storage:
 # rev → (new_base, data_start, data_end, compression_mode)
 rewritten_entries = _precompute_rewritten_delta(
-rl,
+revlog,
 old_index,
 {censor_rev},
 tmp_storage,
 )
 
 all_files = _setup_new_files(
-rl,
+revlog,
 index_cutoff,
 data_cutoff,
 sidedata_cutoff,
@@ -183,7 +183,7 @@
 
 # writing the censored revision
 _rewrite_censor(
-rl,
+revlog,
 old_index,
 open_files,
 censor_rev,
@@ -193,7 +193,7 @@
 # Writing all subsequent revisions
 for rev in range(censor_rev + 1, len(old_index)):
 _rewrite_simple(
-rl,
+revlog,
 old_index,
 open_files,
 rev,



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10901: censor: extract the part about creating and opening new files in a function

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  The v2_censor function is huge, now that its content has settled a bit it is a
  good time to split individual part inside dedicated function.
  
  The last part is the file copying and opening logic. It now have its own
  function.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10901

AFFECTED FILES
  mercurial/revlogutils/rewrite.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/rewrite.py b/mercurial/revlogutils/rewrite.py
--- a/mercurial/revlogutils/rewrite.py
+++ b/mercurial/revlogutils/rewrite.py
@@ -163,55 +163,12 @@
 tmp_storage,
 )
 
-old_index_filepath = rl.opener.join(docket.index_filepath())
-old_data_filepath = rl.opener.join(docket.data_filepath())
-old_sidedata_filepath = rl.opener.join(docket.sidedata_filepath())
-
-new_index_filepath = rl.opener.join(docket.new_index_file())
-new_data_filepath = rl.opener.join(docket.new_data_file())
-new_sidedata_filepath = rl.opener.join(docket.new_sidedata_file())
-
-util.copyfile(
-old_index_filepath, new_index_filepath, nb_bytes=index_cutoff
-)
-util.copyfile(
-old_data_filepath, new_data_filepath, nb_bytes=data_cutoff
-)
-util.copyfile(
-old_sidedata_filepath,
-new_sidedata_filepath,
-nb_bytes=sidedata_cutoff,
+all_files = _setup_new_files(
+rl,
+index_cutoff,
+data_cutoff,
+sidedata_cutoff,
 )
-rl.opener.register_file(docket.index_filepath())
-rl.opener.register_file(docket.data_filepath())
-rl.opener.register_file(docket.sidedata_filepath())
-
-docket.index_end = index_cutoff
-docket.data_end = data_cutoff
-docket.sidedata_end = sidedata_cutoff
-
-# reload the revlog internal information
-rl.clearcaches()
-rl._loadindex(docket=docket)
-
-@contextlib.contextmanager
-def all_files():
-# hide opening in an helper function to please check-code, black
-# and various python ersion at the same time
-with open(old_data_filepath, 'rb') as old_data_file:
-with open(old_sidedata_filepath, 'rb') as old_sidedata_file:
-with open(new_index_filepath, 'r+b') as new_index_file:
-with open(new_data_filepath, 'r+b') as new_data_file:
-with open(
-new_sidedata_filepath, 'r+b'
-) as new_sidedata_file:
-yield (
-old_data_file,
-old_sidedata_file,
-new_index_file,
-new_data_file,
-new_sidedata_file,
-)
 
 # we dont need to open the old index file since its content already
 # exist in a usable form in `old_index`.
@@ -223,12 +180,6 @@
 new_data_file,
 new_sidedata_file,
 ) = open_files
-new_index_file.seek(0, os.SEEK_END)
-assert new_index_file.tell() == index_cutoff
-new_data_file.seek(0, os.SEEK_END)
-assert new_data_file.tell() == data_cutoff
-new_sidedata_file.seek(0, os.SEEK_END)
-assert new_sidedata_file.tell() == sidedata_cutoff
 
 # writing the censored revision
 _rewrite_censor(
@@ -305,6 +256,80 @@
 return rewritten_entries
 
 
+def _setup_new_files(
+revlog,
+index_cutoff,
+data_cutoff,
+sidedata_cutoff,
+):
+"""
+
+return a context manager to open all the relevant files:
+- old_data_file,
+- old_sidedata_file,
+- new_index_file,
+- new_data_file,
+- new_sidedata_file,
+
+The old_index_file is not here because it is accessed through the
+`old_index` object if the caller function.
+"""
+docket = revlog._docket
+old_index_filepath = revlog.opener.join(docket.index_filepath())
+old_data_filepath = revlog.opener.join(docket.data_filepath())
+old_sidedata_filepath = revlog.opener.join(docket.sidedata_filepath())
+
+new_index_filepath = revlog.opener.join(docket.new_index_file())
+new_data_filepath = revlog.opener.join(docket.new_data_file())
+new_sidedata_filepath = revlog.opener.join(docket.new_sidedata_file())
+
+util.copyfile(old_index_filepath, new_index_filepath, 
nb_bytes=index_cutoff)
+util.copyfile(old_data_filepath, new_data_filepath, nb_bytes=data_cutoff)
+util.copyfile(
+old_sidedata_filepath,
+new_sidedata_filepath,
+nb_bytes=sidedata_cutoff,
+)
+

D10896: revlog: rewrite `censors.py` to `rewrite.py`

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: indygreg.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  The logic we use for censoring revision will be mostly common with the one we
  needs for stripping. So we rename the module to `rewrite` to better match its
  future content.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10896

AFFECTED FILES
  mercurial/revlog.py
  mercurial/revlogutils/censor.py
  mercurial/revlogutils/rewrite.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/censor.py b/mercurial/revlogutils/rewrite.py
rename from mercurial/revlogutils/censor.py
rename to mercurial/revlogutils/rewrite.py
diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -81,13 +81,13 @@
 util as interfaceutil,
 )
 from .revlogutils import (
-censor,
 deltas as deltautil,
 docket as docketutil,
 flagutil,
 nodemap as nodemaputil,
 randomaccessfile,
 revlogv0,
+rewrite,
 sidedata as sidedatautil,
 )
 from .utils import (
@@ -3068,9 +3068,9 @@
 % self._format_version
 )
 elif self._format_version == REVLOGV1:
-censor.v1_censor(self, tr, censornode, tombstone)
+rewrite.v1_censor(self, tr, censornode, tombstone)
 else:
-censor.v2_censor(self, tr, censornode, tombstone)
+rewrite.v2_censor(self, tr, censornode, tombstone)
 
 def verifyintegrity(self, state):
 """Verifies the integrity of the revlog.



To: marmoute, indygreg, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10900: censor: extract the part about recomputing delta in a function

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  The v2_censor function is huge, now that its content has settled a bit it is a
  good time to split individual part inside dedicated function.
  
  It is now the turn of the logic that recompute new delta to replace the one
  based on the revision that are going away.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10900

AFFECTED FILES
  mercurial/revlogutils/rewrite.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/rewrite.py b/mercurial/revlogutils/rewrite.py
--- a/mercurial/revlogutils/rewrite.py
+++ b/mercurial/revlogutils/rewrite.py
@@ -154,45 +154,14 @@
 data_cutoff = censored_entry[ENTRY_DATA_OFFSET] >> 16
 sidedata_cutoff = rl.sidedata_cut_off(censor_rev)
 
-# rev → (new_base, data_start, data_end)
-rewritten_entries = {}
-
-dc = deltas.deltacomputer(rl)
-excl = [censor_rev]
-
 with pycompat.unnamedtempfile(mode=b"w+b") as tmp_storage:
-with rl._segmentfile._open_read() as dfh:
-for rev in range(censor_rev + 1, len(old_index)):
-entry = old_index[rev]
-if censor_rev != entry[ENTRY_DELTA_BASE]:
-continue
-# This is a revision that use the censored revision as the base
-# for its delta. We need a need new deltas
-if entry[ENTRY_DATA_UNCOMPRESSED_LENGTH] == 0:
-# this revision is empty, we can delta against nullrev
-rewritten_entries[rev] = (nullrev, 0, 0)
-else:
-
-text = rl.rawdata(rev, _df=dfh)
-info = revlogutils.revisioninfo(
-node=entry[ENTRY_NODE_ID],
-p1=rl.node(entry[ENTRY_PARENT_1]),
-p2=rl.node(entry[ENTRY_PARENT_2]),
-btext=[text],
-textlen=len(text),
-cachedelta=None,
-flags=entry[ENTRY_DATA_OFFSET] & 0x,
-)
-d = dc.finddeltainfo(
-info, dfh, excluded_bases=excl, target_rev=rev
-)
-default_comp = rl._docket.default_compression_header
-comp_mode, d = deltas.delta_compression(default_comp, d)
-# using `tell` is a bit lazy, but we are not here for speed
-start = tmp_storage.tell()
-tmp_storage.write(d.data[1])
-end = tmp_storage.tell()
-rewritten_entries[rev] = (d.base, start, end, comp_mode)
+# rev → (new_base, data_start, data_end, compression_mode)
+rewritten_entries = _precompute_rewritten_delta(
+rl,
+old_index,
+{censor_rev},
+tmp_storage,
+)
 
 old_index_filepath = rl.opener.join(docket.index_filepath())
 old_data_filepath = rl.opener.join(docket.data_filepath())
@@ -283,6 +252,59 @@
 docket.write(transaction=None, stripping=True)
 
 
+def _precompute_rewritten_delta(
+revlog,
+old_index,
+excluded_revs,
+tmp_storage,
+):
+"""Compute new delta for revisions whose delta is based on revision that
+will not survives as is.
+
+return a mapping: {rev → (new_base, data_start, data_end, 
compression_mode)}
+"""
+dc = deltas.deltacomputer(revlog)
+rewritten_entries = {}
+first_excl_rev = min(excluded_revs)
+with revlog._segmentfile._open_read() as dfh:
+for rev in range(first_excl_rev, len(old_index)):
+if rev in excluded_revs:
+# this revision will be preserved as is, so we don't need to
+# consider recomputing a delta.
+continue
+entry = old_index[rev]
+if entry[ENTRY_DELTA_BASE] not in excluded_revs:
+continue
+# This is a revision that use the censored revision as the base
+# for its delta. We need a need new deltas
+if entry[ENTRY_DATA_UNCOMPRESSED_LENGTH] == 0:
+# this revision is empty, we can delta against nullrev
+rewritten_entries[rev] = (nullrev, 0, 0, COMP_MODE_PLAIN)
+else:
+
+text = revlog.rawdata(rev, _df=dfh)
+info = revlogutils.revisioninfo(
+node=entry[ENTRY_NODE_ID],
+p1=revlog.node(entry[ENTRY_PARENT_1]),
+p2=revlog.node(entry[ENTRY_PARENT_2]),
+btext=[text],
+textlen=len(text),
+cachedelta=None,
+flags=entry[ENTRY_DATA_OFFSET] & 0x,
+)
+d = dc.finddeltainfo(
+info, dfh, excluded_bases=excluded_revs, 

D10899: censor: extract the part about writing the other revision in a function

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  The v2_censor function is huge, now that its content has settled a bit it is a
  good time to split individual part inside dedicated function.
  
  We continue with a small function that add a non-censored revision back to the
  revlog.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10899

AFFECTED FILES
  mercurial/revlogutils/rewrite.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/rewrite.py b/mercurial/revlogutils/rewrite.py
--- a/mercurial/revlogutils/rewrite.py
+++ b/mercurial/revlogutils/rewrite.py
@@ -270,72 +270,97 @@
 tombstone,
 )
 
- Writing all subsequent revisions
+# Writing all subsequent revisions
 for rev in range(censor_rev + 1, len(old_index)):
-entry = old_index[rev]
-flags = entry[ENTRY_DATA_OFFSET] & 0x
-old_data_offset = entry[ENTRY_DATA_OFFSET] >> 16
+_rewrite_simple(
+rl,
+old_index,
+open_files,
+rev,
+rewritten_entries,
+tmp_storage,
+)
+docket.write(transaction=None, stripping=True)
+
 
-if rev not in rewritten_entries:
-old_data_file.seek(old_data_offset)
-new_data_size = entry[ENTRY_DATA_COMPRESSED_LENGTH]
-new_data = old_data_file.read(new_data_size)
-data_delta_base = entry[ENTRY_DELTA_BASE]
-d_comp_mode = entry[ENTRY_DATA_COMPRESSION_MODE]
-else:
-(
-data_delta_base,
-start,
-end,
-d_comp_mode,
-) = rewritten_entries[rev]
-new_data_size = end - start
-tmp_storage.seek(start)
-new_data = tmp_storage.read(new_data_size)
-
-# It might be faster to group continuous read/write operation,
-# however, this is censor, an operation that is not focussed
-# around stellar performance. So I have not written this
-# optimisation yet.
-new_data_offset = new_data_file.tell()
-new_data_file.write(new_data)
+def _rewrite_simple(
+revlog,
+old_index,
+all_files,
+rev,
+rewritten_entries,
+tmp_storage,
+):
+"""append a "normal" revision to the index"""
+(
+old_data_file,
+old_sidedata_file,
+new_index_file,
+new_data_file,
+new_sidedata_file,
+) = all_files
+entry = old_index[rev]
+flags = entry[ENTRY_DATA_OFFSET] & 0x
+old_data_offset = entry[ENTRY_DATA_OFFSET] >> 16
 
-sidedata_size = entry[ENTRY_SIDEDATA_COMPRESSED_LENGTH]
-new_sidedata_offset = new_sidedata_file.tell()
-if 0 < sidedata_size:
-old_sidedata_offset = entry[ENTRY_SIDEDATA_OFFSET]
-old_sidedata_file.seek(old_sidedata_offset)
-new_sidedata = old_sidedata_file.read(sidedata_size)
-new_sidedata_file.write(new_sidedata)
+if rev not in rewritten_entries:
+old_data_file.seek(old_data_offset)
+new_data_size = entry[ENTRY_DATA_COMPRESSED_LENGTH]
+new_data = old_data_file.read(new_data_size)
+data_delta_base = entry[ENTRY_DELTA_BASE]
+d_comp_mode = entry[ENTRY_DATA_COMPRESSION_MODE]
+else:
+(
+data_delta_base,
+start,
+end,
+d_comp_mode,
+) = rewritten_entries[rev]
+new_data_size = end - start
+tmp_storage.seek(start)
+new_data = tmp_storage.read(new_data_size)
 
-data_uncompressed_length = 
entry[ENTRY_DATA_UNCOMPRESSED_LENGTH]
-sd_com_mode = entry[ENTRY_SIDEDATA_COMPRESSION_MODE]
-assert data_delta_base <= rev, (data_delta_base, rev)
+# It might be faster to group continuous read/write operation,
+# however, this is censor, an operation that is not focussed
+# around stellar performance. So I have not written this
+# optimisation yet.
+new_data_offset = new_data_file.tell()
+new_data_file.write(new_data)
 
-new_entry = revlogutils.entry(
-flags=flags,
-data_offset=new_data_offset,
-data_compressed_length=new_data_size,
-data_uncompressed_length=data_uncompressed_length,
-data_delta_base=data_delta_base,
-link_rev=entry[ENTRY_LINK_REV],
-parent_rev_1=entry[ENTRY_PARENT_1],
-

D10898: censor: extract the part about writing the censored revision in a function

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  The v2_censor function is huge, now that its content has settled a bit it is a
  good time to split individual part inside dedicated function. We start with a
  small function that process the censored revision.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10898

AFFECTED FILES
  mercurial/revlogutils/rewrite.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/rewrite.py b/mercurial/revlogutils/rewrite.py
--- a/mercurial/revlogutils/rewrite.py
+++ b/mercurial/revlogutils/rewrite.py
@@ -261,36 +261,14 @@
 new_sidedata_file.seek(0, os.SEEK_END)
 assert new_sidedata_file.tell() == sidedata_cutoff
 
-### writing the censored revision
-entry = old_index[censor_rev]
-
-# XXX consider trying the default compression too
-new_data_size = len(tombstone)
-new_data_offset = new_data_file.tell()
-new_data_file.write(tombstone)
-
-# we are not adding any sidedata as they might leak info about the 
censored version
-
-new_entry = revlogutils.entry(
-flags=constants.REVIDX_ISCENSORED,
-data_offset=new_data_offset,
-data_compressed_length=new_data_size,
-data_uncompressed_length=new_data_size,
-data_delta_base=censor_rev,
-link_rev=entry[ENTRY_LINK_REV],
-parent_rev_1=entry[ENTRY_PARENT_1],
-parent_rev_2=entry[ENTRY_PARENT_2],
-node_id=entry[ENTRY_NODE_ID],
-sidedata_offset=0,
-sidedata_compressed_length=0,
-data_compression_mode=COMP_MODE_PLAIN,
-sidedata_compression_mode=COMP_MODE_PLAIN,
+# writing the censored revision
+_rewrite_censor(
+rl,
+old_index,
+open_files,
+censor_rev,
+tombstone,
 )
-rl.index.append(new_entry)
-entry_bin = rl.index.entry_binary(censor_rev)
-new_index_file.write(entry_bin)
-docket.index_end = new_index_file.tell()
-docket.data_end = new_data_file.tell()
 
  Writing all subsequent revisions
 for rev in range(censor_rev + 1, len(old_index)):
@@ -358,3 +336,54 @@
 docket.sidedata_end = new_sidedata_file.tell()
 
 docket.write(transaction=None, stripping=True)
+
+
+def _rewrite_censor(
+revlog,
+old_index,
+all_files,
+rev,
+tombstone,
+):
+"""rewrite and append a censored revision"""
+(
+old_data_file,
+old_sidedata_file,
+new_index_file,
+new_data_file,
+new_sidedata_file,
+) = all_files
+entry = old_index[rev]
+
+# XXX consider trying the default compression too
+new_data_size = len(tombstone)
+new_data_offset = new_data_file.tell()
+new_data_file.write(tombstone)
+
+# we are not adding any sidedata as they might leak info about the 
censored version
+
+link_rev = entry[ENTRY_LINK_REV]
+
+p1 = entry[ENTRY_PARENT_1]
+p2 = entry[ENTRY_PARENT_2]
+
+new_entry = revlogutils.entry(
+flags=constants.REVIDX_ISCENSORED,
+data_offset=new_data_offset,
+data_compressed_length=new_data_size,
+data_uncompressed_length=new_data_size,
+data_delta_base=rev,
+link_rev=link_rev,
+parent_rev_1=p1,
+parent_rev_2=p2,
+node_id=entry[ENTRY_NODE_ID],
+sidedata_offset=0,
+sidedata_compressed_length=0,
+data_compression_mode=COMP_MODE_PLAIN,
+sidedata_compression_mode=COMP_MODE_PLAIN,
+)
+revlog.index.append(new_entry)
+entry_bin = revlog.index.entry_binary(rev)
+new_index_file.write(entry_bin)
+revlog._docket.index_end = new_index_file.tell()
+revlog._docket.data_end = new_data_file.tell()



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10897: censor: put the tuple of open files in an explicit variable

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This will be simpler to pass these file around in future changesets.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10897

AFFECTED FILES
  mercurial/revlogutils/rewrite.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/rewrite.py b/mercurial/revlogutils/rewrite.py
--- a/mercurial/revlogutils/rewrite.py
+++ b/mercurial/revlogutils/rewrite.py
@@ -246,13 +246,14 @@
 
 # we dont need to open the old index file since its content already
 # exist in a usable form in `old_index`.
-with all_files() as (
-old_data_file,
-old_sidedata_file,
-new_index_file,
-new_data_file,
-new_sidedata_file,
-):
+with all_files() as open_files:
+(
+old_data_file,
+old_sidedata_file,
+new_index_file,
+new_data_file,
+new_sidedata_file,
+) = open_files
 new_index_file.seek(0, os.SEEK_END)
 assert new_index_file.tell() == index_cutoff
 new_data_file.seek(0, os.SEEK_END)



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10889: revlog: factor the logic to determine the delta compression out

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: indygreg.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This make the logic clearly isolated and documented and it will help use to
  reuse it during censors/strip.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10889

AFFECTED FILES
  mercurial/revlog.py
  mercurial/revlogutils/deltas.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/deltas.py b/mercurial/revlogutils/deltas.py
--- a/mercurial/revlogutils/deltas.py
+++ b/mercurial/revlogutils/deltas.py
@@ -18,6 +18,9 @@
 from ..pycompat import getattr
 
 from .constants import (
+COMP_MODE_DEFAULT,
+COMP_MODE_INLINE,
+COMP_MODE_PLAIN,
 REVIDX_ISCENSORED,
 REVIDX_RAWTEXT_CHANGING_FLAGS,
 )
@@ -1113,3 +1116,28 @@
 if deltainfo is None:
 deltainfo = self._fullsnapshotinfo(fh, revinfo)
 return deltainfo
+
+
+def delta_compression(default_compression_header, deltainfo):
+"""return (COMPRESSION_MODE, deltainfo)
+
+used by revlog v2+ format to dispatch between PLAIN and DEFAULT
+compression.
+"""
+h, d = deltainfo.data
+compression_mode = COMP_MODE_INLINE
+if not h and not d:
+# not data to store at all... declare them uncompressed
+compression_mode = COMP_MODE_PLAIN
+elif not h:
+t = d[0:1]
+if t == b'\0':
+compression_mode = COMP_MODE_PLAIN
+elif t == default_compression_header:
+compression_mode = COMP_MODE_DEFAULT
+elif h == b'u':
+# we have a more efficient way to declare uncompressed
+h = b''
+compression_mode = COMP_MODE_PLAIN
+deltainfo = drop_u_compression(deltainfo)
+return compression_mode, deltainfo
diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -2437,21 +2437,9 @@
 
 compression_mode = COMP_MODE_INLINE
 if self._docket is not None:
-h, d = deltainfo.data
-if not h and not d:
-# not data to store at all... declare them uncompressed
-compression_mode = COMP_MODE_PLAIN
-elif not h:
-t = d[0:1]
-if t == b'\0':
-compression_mode = COMP_MODE_PLAIN
-elif t == self._docket.default_compression_header:
-compression_mode = COMP_MODE_DEFAULT
-elif h == b'u':
-# we have a more efficient way to declare uncompressed
-h = b''
-compression_mode = COMP_MODE_PLAIN
-deltainfo = deltautil.drop_u_compression(deltainfo)
+default_comp = self._docket.default_compression_header
+r = deltautil.delta_compression(default_comp, deltainfo)
+compression_mode, deltainfo = r
 
 sidedata_compression_mode = COMP_MODE_INLINE
 if sidedata and self.hassidedata:



To: marmoute, indygreg, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10887: deltas: at a `target_rev` parameter to finddeltainfo

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Otherwise, recomputing a delta for a revision might result in a delta against 
a
  later revision or a full snapshot thinking we are appending a new revision.
  
  We will make use of this during censoring (and later, stripping).

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10887

AFFECTED FILES
  mercurial/revlogutils/deltas.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/deltas.py b/mercurial/revlogutils/deltas.py
--- a/mercurial/revlogutils/deltas.py
+++ b/mercurial/revlogutils/deltas.py
@@ -1030,8 +1030,7 @@
 snapshotdepth,
 )
 
-def _fullsnapshotinfo(self, fh, revinfo):
-curr = len(self.revlog)
+def _fullsnapshotinfo(self, fh, revinfo, curr):
 rawtext = self.buildtext(revinfo, fh)
 data = self.revlog.compress(rawtext)
 compresseddeltalen = deltalen = dist = len(data[1]) + len(data[0])
@@ -1050,7 +1049,7 @@
 snapshotdepth,
 )
 
-def finddeltainfo(self, revinfo, fh, excluded_bases=None):
+def finddeltainfo(self, revinfo, fh, excluded_bases=None, target_rev=None):
 """Find an acceptable delta against a candidate revision
 
 revinfo: information about the revision (instance of _revisioninfo)
@@ -1067,8 +1066,11 @@
 a delta base. Use this to recompute delta suitable in censor or strip
 context.
 """
+if target_rev is None:
+curr = len(self.revlog)
+
 if not revinfo.textlen:
-return self._fullsnapshotinfo(fh, revinfo)
+return self._fullsnapshotinfo(fh, revinfo, target_rev)
 
 if excluded_bases is None:
 excluded_bases = set()
@@ -1077,7 +1079,7 @@
 # not calling candelta since only one revision needs test, also to
 # avoid overhead fetching flags again.
 if revinfo.flags & REVIDX_RAWTEXT_CHANGING_FLAGS:
-return self._fullsnapshotinfo(fh, revinfo)
+return self._fullsnapshotinfo(fh, revinfo, target_rev)
 
 cachedelta = revinfo.cachedelta
 p1 = revinfo.p1
@@ -1099,6 +1101,8 @@
 for candidaterev in candidaterevs:
 if candidaterev in excluded_bases:
 continue
+if candidaterev >= target_rev:
+continue
 candidatedelta = self._builddeltainfo(revinfo, candidaterev, 
fh)
 if candidatedelta is not None:
 if isgooddeltainfo(self.revlog, candidatedelta, revinfo):
@@ -,5 +1115,5 @@
 candidaterevs = next(groups)
 
 if deltainfo is None:
-deltainfo = self._fullsnapshotinfo(fh, revinfo)
+deltainfo = self._fullsnapshotinfo(fh, revinfo, target_rev)
 return deltainfo



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: How to fix integrity errors ?

[Pierre-Yves David]

Hi M. Pertin

On 6/10/21 4:26 PM, PERTIN Francois wrote:

We are using mercurial version 3.7.3 on a Linux server.



This version of Mercurial was released 5 years ago and is considered 
quite ancient. There have been 21 other major releases of Mercurial in 
the meantime, including multiple security releases.


I strongly encourage you to upgrade your client and server.



At one time a user got an error while trying to push.

He did it again, the error disappear and push succeed but all users 
had to re-clone the repository from the server to be able to push again.




This is quite suspicious. Do you have any record of the error that user 
got while pushing? Or of the error the other people got while pushing?


When did this happened?


Since that time, running hg verify on user repository and on server 
repository returns the following error.


myfile@23805  
<mailto:cpu/val3_nonRegression/val3_nonRegScara/usr/app/rdd3.dll@23805>: 
ddb482e379a1 in manifests not found
21293 files, 24338 changesets, 98658 total revisions
1 integrity errors encountered!
(first damaged changeset appears to be 23805)


So it looks like a corruption was pushed to you repository, the revision 
of a given file is referenced in the history but is missing from the 
storage for that file. Have you kept backups of the users' repositories 
at the time of the error?


Does revision `23805` (number to use on the same repository you used to 
run verify) have any children?




This is not an issue for all the users except for fisheye.

It is not able to index the repository, it always restart after the 
following error.


Atlasian ask us to fix the server repository by using the method 
described here: RepositoryCorruption - Mercurial 
<https://urldefense.com/v3/__https:/www.mercurial-scm.org/wiki/RepositoryCorruption__;!!PxtyCg5I!G2l-3QNOufA29psGGqSMt4sgqbUUA6rP_CwMmyGjzx3Q-2mhJ0nxW66if53H5_0A$>


I spend a lot of time trying to fix this without success.\


Does any one know how to fix this issue?



It looks like Fisheye is trying to load the entire history into its own 
database and is chocking on the specific revision that contains the 
corruption. That file revision is corrupted so you will never be able to 
restore it as is. However they are multiple options to get out of this 
situation:


1) remove the changeset with the corruption from the repository (if it 
is an unused head), or rewrite the dependant history,

2) find an un-corrupted version of that file in some backup,
3) use "censors" to mark the content of that revision non-accessible 
(might need some development to the mercurial code-base),

4) convince Fisheye to be less picky.


I don't know which of the above options are actually possible in your 
case. And I don't know which one will be the best. However we can 
probably get to a solution with some email roundtrips. Can you please 
have a look at the question I wrote earlier in this email?


Cheers,

--

Pierre-Yves David

___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10869: censor: implement censoring for revlogv2

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: indygreg.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  It is a bit verbose and rough, but it works. Most of that logic can be common
  for `stripping`, so we can expect more refactoring of that code to accommodate
  both needs. However I wanted to keep this changesets "simple enough" and 
before
  moving forward.
  
  We also need to properly delete the older index/data/sidedata file, but this 
has
  implication for streaming clone and transaction, so this will come later.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10869

AFFECTED FILES
  mercurial/revlog.py
  mercurial/revlogutils/censor.py
  tests/test-censor.t

CHANGE DETAILS

diff --git a/tests/test-censor.t b/tests/test-censor.t
--- a/tests/test-censor.t
+++ b/tests/test-censor.t
@@ -1,4 +1,14 @@
 #require no-reposimplestore
+#testcases revlogv1 revlogv2
+
+#if revlogv2
+
+  $ cat >> $HGRCPATH < [experimental]
+  > revlogv2=enable-unstable-format-and-corrupt-my-data
+  > EOF
+
+#endif
 
   $ cat >> $HGRCPATH < [extensions]
diff --git a/mercurial/revlogutils/censor.py b/mercurial/revlogutils/censor.py
--- a/mercurial/revlogutils/censor.py
+++ b/mercurial/revlogutils/censor.py
@@ -1,4 +1,5 @@
 # censor code related to censoring revision
+# coding: utf8
 #
 # Copyright 2021 Pierre-Yves David 
 # Copyright 2015 Google, Inc 
@@ -6,17 +7,44 @@
 # This software may be used and distributed according to the terms of the
 # GNU General Public License version 2 or any later version.
 
+import contextlib
+import os
+
 from ..node import (
 nullrev,
 )
+from .constants import (
+COMP_MODE_PLAIN,
+ENTRY_DATA_COMPRESSED_LENGTH,
+ENTRY_DATA_COMPRESSION_MODE,
+ENTRY_DATA_OFFSET,
+ENTRY_DATA_UNCOMPRESSED_LENGTH,
+ENTRY_DELTA_BASE,
+ENTRY_LINK_REV,
+ENTRY_NODE_ID,
+ENTRY_PARENT_1,
+ENTRY_PARENT_2,
+ENTRY_SIDEDATA_COMPRESSED_LENGTH,
+ENTRY_SIDEDATA_COMPRESSION_MODE,
+ENTRY_SIDEDATA_OFFSET,
+REVLOGV0,
+REVLOGV1,
+)
 from ..i18n import _
+
 from .. import (
 error,
+pycompat,
+revlogutils,
+util,
 )
 from ..utils import (
 storageutil,
 )
-from . import constants
+from . import (
+constants,
+deltas,
+)
 
 
 def v1_censor(rl, tr, censornode, tombstone=b''):
@@ -95,3 +123,227 @@
 
 rl.clearcaches()
 rl._loadindex()
+
+
+def v2_censor(rl, tr, censornode, tombstone=b''):
+"""censors a revision in a "version 2" revlog"""
+# General principle
+#
+# We create new revlog files (index/data/sidedata) to copy the content of
+# the existing data without the censored data.
+#
+# We need to recompute new delta for any revision that used the censored
+# revision as delta base. As the cumulative size of the new delta may be
+# large, we store them in a temporary file until they are stored in their
+# final destination.
+#
+# All data before the censored data can be blindly copied. The rest needs
+# to be copied as we go and the associated index entry needs adjustement.
+
+assert rl._format_version != REVLOGV0, rl._format_version
+assert rl._format_version != REVLOGV1, rl._format_version
+
+old_index = rl.index
+docket = rl._docket
+
+censor_rev = rl.rev(censornode)
+tombstone = storageutil.packmeta({b'censored': tombstone}, b'')
+
+censored_entry = rl.index[censor_rev]
+index_cut_off = rl.index.entry_size * censor_rev
+data_cut_off = censored_entry[ENTRY_DATA_OFFSET] >> 16
+sidedata_cut_off = rl.sidedata_cut_off(censor_rev)
+
+# rev → (new_base, data_start, data_end)
+rewritten_entries = {}
+
+dc = deltas.deltacomputer(rl)
+excl = [censor_rev]
+
+with pycompat.unnamedtempfile(mode="w+b") as tmp_storage:
+with rl._datareadfp() as dfh:
+for rev in range(censor_rev + 1, len(old_index)):
+entry = old_index[rev]
+if censor_rev != entry[ENTRY_DELTA_BASE]:
+continue
+# This is a revision that use the censored revision as the base
+# for its delta. We need a need new deltas
+if entry[ENTRY_DATA_UNCOMPRESSED_LENGTH] == 0:
+# this revision is empty, we can delta against nullrev
+rewritten_entries[rev] = (nullrev, 0, 0)
+else:
+
+info = revlogutils.revisioninfo(
+entry[ENTRY_NODE_ID],
+entry[ENTRY_PARENT_1],
+entry[ENTRY_PARENT_2],
+[rl.revision(rev, _df=dfh, raw=True)],
+None,
+entry[ENTRY_DATA_OFFSET] & 0x,
+)
+
+d = dc.finddeltainfo(info, dfh, exclud

D10868: revlog: list older-but-still-around file in `files`

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: indygreg.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  These older files will eventually be removed, but we have to take them in 
account until they are. So we update `files` to keep fncache happy.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10868

AFFECTED FILES
  mercurial/revlog.py
  mercurial/revlogutils/docket.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/docket.py b/mercurial/revlogutils/docket.py
--- a/mercurial/revlogutils/docket.py
+++ b/mercurial/revlogutils/docket.py
@@ -180,6 +180,13 @@
 self._index_uuid = make_uid()
 return self.index_filepath()
 
+def old_index_filepaths(self, include_empty=True):
+"""yield file path to older index files associated to this docket"""
+# very simplistic version at first
+for uuid, size in self._older_index_uuids:
+if include_empty or size > 0:
+yield b"%s-%s.idx" % (self._radix, uuid)
+
 def data_filepath(self):
 """file path to the current data file associated to this docket"""
 # very simplistic version at first
@@ -196,6 +203,13 @@
 self._data_uuid = make_uid()
 return self.data_filepath()
 
+def old_data_filepaths(self, include_empty=True):
+"""yield file path to older data files associated to this docket"""
+# very simplistic version at first
+for uuid, size in self._older_data_uuids:
+if include_empty or size > 0:
+yield b"%s-%s.dat" % (self._radix, uuid)
+
 def sidedata_filepath(self):
 """file path to the current sidedata file associated to this docket"""
 # very simplistic version at first
@@ -212,6 +226,13 @@
 self._sidedata_uuid = make_uid()
 return self.sidedata_filepath()
 
+def old_sidedata_filepaths(self, include_empty=True):
+"""yield file path to older sidedata files associated to this docket"""
+# very simplistic version at first
+for uuid, size in self._older_sidedata_uuids:
+if include_empty or size > 0:
+yield b"%s-%s.sda" % (self._radix, uuid)
+
 @property
 def index_end(self):
 return self._index_end
diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -2928,10 +2928,13 @@
 res.append(self._datafile)
 else:
 res.append(self._docket_file)
+res.extend(self._docket.old_index_filepaths(include_empty=False))
 if self._docket.data_end:
 res.append(self._datafile)
+res.extend(self._docket.old_data_filepaths(include_empty=False))
 if self._docket.sidedata_end:
 res.append(self._sidedatafile)
+
res.extend(self._docket.old_sidedata_filepaths(include_empty=False))
 return res
 
 def emitrevisions(



To: marmoute, indygreg, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10867: revlog: add docket method to request new content files

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  If we want to write content into new files, we need to be able to ask for 
them.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10867

AFFECTED FILES
  mercurial/revlogutils/docket.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/docket.py b/mercurial/revlogutils/docket.py
--- a/mercurial/revlogutils/docket.py
+++ b/mercurial/revlogutils/docket.py
@@ -171,6 +171,15 @@
 self._index_uuid = make_uid()
 return b"%s-%s.idx" % (self._radix, self._index_uuid)
 
+def new_index_file(self):
+"""switch index file to a new UID
+
+The previous index UID is moved to the "older" list."""
+old = (self._index_uuid, self._index_end)
+self._older_index_uuids.insert(0, old)
+self._index_uuid = make_uid()
+return self.index_filepath()
+
 def data_filepath(self):
 """file path to the current data file associated to this docket"""
 # very simplistic version at first
@@ -178,6 +187,15 @@
 self._data_uuid = make_uid()
 return b"%s-%s.dat" % (self._radix, self._data_uuid)
 
+def new_data_file(self):
+"""switch data file to a new UID
+
+The previous data UID is moved to the "older" list."""
+old = (self._data_uuid, self._data_end)
+self._older_data_uuids.insert(0, old)
+self._data_uuid = make_uid()
+return self.data_filepath()
+
 def sidedata_filepath(self):
 """file path to the current sidedata file associated to this docket"""
 # very simplistic version at first
@@ -185,6 +203,15 @@
 self._sidedata_uuid = make_uid()
 return b"%s-%s.sda" % (self._radix, self._sidedata_uuid)
 
+def new_sidedata_file(self):
+"""switch sidedata file to a new UID
+
+The previous sidedata UID is moved to the "older" list."""
+old = (self._sidedata_uuid, self._sidedata_end)
+self._older_sidedata_uuids.insert(0, old)
+self._sidedata_uuid = make_uid()
+return self.sidedata_filepath()
+
 @property
 def index_end(self):
 return self._index_end



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10865: nodemap: fix some comment formatting

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Black seems unhappy about it.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10865

AFFECTED FILES
  mercurial/revlogutils/nodemap.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/nodemap.py b/mercurial/revlogutils/nodemap.py
--- a/mercurial/revlogutils/nodemap.py
+++ b/mercurial/revlogutils/nodemap.py
@@ -133,7 +133,7 @@
 
 
 def delete_nodemap(tr, repo, revlog):
-""" Delete nodemap data on disk for a given revlog"""
+"""Delete nodemap data on disk for a given revlog"""
 if revlog._nodemap_file is None:
 msg = "calling persist nodemap on a revlog without the feature enabled"
 raise error.ProgrammingError(msg)



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10866: revlog: add a way to keep track of older uids in the docket

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  When the revlog content is rewritten, we will use new files, to avoid 
truncating
  the previous ones. We need some way to keep track of the older files before we
  clean them up.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10866

AFFECTED FILES
  mercurial/revlogutils/docket.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/docket.py b/mercurial/revlogutils/docket.py
--- a/mercurial/revlogutils/docket.py
+++ b/mercurial/revlogutils/docket.py
@@ -89,8 +89,11 @@
 #  |   This is mandatory as docket must be compatible with the previous
 #  |   revlog index header.
 # * 1 bytes: size of index uuid
+# * 1 bytes: number of outdated index uuid
 # * 1 bytes: size of data uuid
+# * 1 bytes: number of outdated data uuid
 # * 1 bytes: size of sizedata uuid
+# * 1 bytes: number of outdated data uuid
 # * 8 bytes: size of index-data
 # * 8 bytes: pending size of index-data
 # * 8 bytes: size of data
@@ -98,7 +101,10 @@
 # * 8 bytes: pending size of data
 # * 8 bytes: pending size of sidedata
 # * 1 bytes: default compression header
-S_HEADER = struct.Struct(constants.INDEX_HEADER_FMT + b'BBBLLc')
+S_HEADER = struct.Struct(constants.INDEX_HEADER_FMT + b'BBLLc')
+# * 1 bytes: size of index uuid
+# * 8 bytes: size of file
+S_OLD_UID = struct.Struct('>BL')
 
 
 class RevlogDocket(object):
@@ -110,8 +116,11 @@
 use_pending=False,
 version_header=None,
 index_uuid=None,
+older_index_uuids=(),
 data_uuid=None,
+older_data_uuids=(),
 sidedata_uuid=None,
+older_sidedata_uuids=(),
 index_end=0,
 pending_index_end=0,
 data_end=0,
@@ -127,8 +136,14 @@
 self._path = revlog._docket_file
 self._opener = revlog.opener
 self._index_uuid = index_uuid
+self._older_index_uuids = older_index_uuids
 self._data_uuid = data_uuid
+self._older_data_uuids = older_data_uuids
 self._sidedata_uuid = sidedata_uuid
+self._older_sidedata_uuids = older_sidedata_uuids
+assert not set(older_index_uuids) & set(older_data_uuids)
+assert not set(older_data_uuids) & set(older_sidedata_uuids)
+assert not set(older_index_uuids) & set(older_sidedata_uuids)
 # thes asserts should be True as long as we have a single index 
filename
 assert index_end <= pending_index_end
 assert data_end <= pending_data_end
@@ -237,8 +252,11 @@
 data = (
 self._version_header,
 len(self._index_uuid),
+len(self._older_index_uuids),
 len(self._data_uuid),
+len(self._older_data_uuids),
 len(self._sidedata_uuid),
+len(self._older_sidedata_uuids),
 official_index_end,
 self._index_end,
 official_data_end,
@@ -249,9 +267,24 @@
 )
 s = []
 s.append(S_HEADER.pack(*data))
+
 s.append(self._index_uuid)
+for u, size in self._older_index_uuids:
+s.append(S_OLD_UID.pack(len(u), size))
+for u, size in self._older_index_uuids:
+s.append(u)
+
 s.append(self._data_uuid)
+for u, size in self._older_data_uuids:
+s.append(S_OLD_UID.pack(len(u), size))
+for u, size in self._older_data_uuids:
+s.append(u)
+
 s.append(self._sidedata_uuid)
+for u, size in self._older_sidedata_uuids:
+s.append(S_OLD_UID.pack(len(u), size))
+for u, size in self._older_sidedata_uuids:
+s.append(u)
 return b''.join(s)
 
 
@@ -270,6 +303,19 @@
 return docket
 
 
+def _parse_old_uids(get_data, count):
+all_sizes = []
+all_uids = []
+for i in range(0, count):
+raw = get_data(S_OLD_UID.size)
+all_sizes.append(S_OLD_UID.unpack(raw))
+
+for uid_size, file_size in all_sizes:
+uid = get_data(uid_size)
+all_uids.append((uid, file_size))
+return all_uids
+
+
 def parse_docket(revlog, data, use_pending=False):
 """given some docket data return a docket object for the given revlog"""
 header = S_HEADER.unpack(data[: S_HEADER.size])
@@ -295,12 +341,21 @@
 index_uuid_size = next(iheader)
 index_uuid = get_data(index_uuid_size)
 
+older_index_uuid_count = next(iheader)
+older_index_uuids = _parse_old_uids(get_data, older_index_uuid_count)
+
 data_uuid_size = next(iheader)
 data_uuid = get_data(data_uuid_size)
 
+older_data_uuid_count = next(iheader)
+older_data_uuids = _parse_old_uids(get_data, older_data_uuid_count)
+
 sidedata_uuid_size = next(iheader)
 sidedata_uuid = get_data(sidedata_uuid_size)
 
+older_sidedata_uuid_count = next(iheader)
+older_sidedata_uuids = _parse_old_uids(get_data, 

D10855: clone: reuse the stream clone logic for local clone

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Streaming clone and local (non `--pull`) clone do mostly the same thing,
  however they were using different logic to do so. This means the logic
  frequently went out of sync and that new case had to be dealt with twice.
  
  This is fragile and anoying. So we replace this with a re-use of the logic we
  use for streaming clone.
  
  I can see various test changes:
  
  - a more precise progress output,
  - armless fncache loading during clone,
  - fncache is no longer hardlinked (since we write it by hand).
  
  I am not reinstalling the `reposimplestore` specific output, as far as I
  understand this variant have been broken for years.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10855

AFFECTED FILES
  mercurial/hg.py
  mercurial/streamclone.py
  tests/test-clone.t
  tests/test-empty.t
  tests/test-fncache.t
  tests/test-hardlinks.t
  tests/test-subrepo-deep-nested-change.t
  tests/test-subrepo-recursion.t

CHANGE DETAILS

diff --git a/tests/test-subrepo-recursion.t b/tests/test-subrepo-recursion.t
--- a/tests/test-subrepo-recursion.t
+++ b/tests/test-subrepo-recursion.t
@@ -454,19 +454,15 @@
 #if hardlink
   $ hg clone -U . ../empty
   \r (no-eol) (esc)
-  linking [ <=>   ] 1\r (no-eol) (esc)
-  linking [  <=>  ] 2\r (no-eol) (esc)
-  linking [   <=> ] 3\r (no-eol) (esc)
-  linking [<=>] 4\r (no-eol) (esc)
-  linking [ <=>   ] 5\r (no-eol) (esc)
-  linking [  <=>  ] 6\r (no-eol) (esc)
-  linking [   <=> ] 7\r (no-eol) (esc)
-  linking [<=>] 8\r (no-eol) (esc)
-  linking [ <=>   ] 9\r (no-eol) (esc) 
(reposimplestore !)
-  linking [  <=> ] 10\r (no-eol) (esc) 
(reposimplestore !)
-  linking [   <=>] 11\r (no-eol) (esc) 
(reposimplestore !)
-  linking [<=>   ] 12\r (no-eol) (esc) 
(reposimplestore !)
-  linking [ <=>  ] 13\r (no-eol) (esc) 
(reposimplestore !)
+  linking [>] 1/9\r (no-eol) (esc)
+  linking [=>   ] 2/9\r (no-eol) (esc)
+  linking [==>  ] 3/9\r (no-eol) (esc)
+  linking [===> ] 4/9\r (no-eol) (esc)
+  linking [>] 5/9\r (no-eol) (esc)
+  linking [=>   ] 6/9\r (no-eol) (esc)
+  linking [==>  ] 7/9\r (no-eol) (esc)
+  linking [===> ] 8/9\r (no-eol) (esc)
+  linking [>] 9/9\r (no-eol) (esc)
   \r (no-eol) (esc)
 #else
   $ hg clone -U . ../empty
@@ -484,22 +480,14 @@
   archiving [==>] 3/3\r (no-eol) (esc)
   \r (no-eol) (esc)
   \r (no-eol) (esc)
-  linking [ <=>   ] 1\r (no-eol) (esc)
-  linking [  <=>  ] 2\r (no-eol) (esc)
-  linking [   <=> ] 3\r (no-eol) (esc)
-  linking [<=>] 4\r (no-eol) (esc)
-  linking [ <=>   ] 5\r (no-eol) (esc)
-  linking [  <=>  ] 6\r (no-eol) (esc)
-  linking [   <=> ] 7\r (no-eol) (esc)
-  linking [<=>] 8\r (no-eol) (esc)
-  linking [ <=>   ] 9\r (no-eol) (esc) 
(reposimplestore !)
-  linking [  <=> ] 10\r (no-eol) (esc) 
(reposimplestore !)
-  linking [   <=>] 11\r (no-eol) (esc) 
(reposimplestore !)
-  linking [<=>   ] 12\r (no-eol) (esc) 
(reposimplestore !)
-  linking [ <=>  ] 13\r (no-eol) (esc) 
(reposimplestore !)
-  linking [  <=> ] 14\r (no-eol) (esc) 
(reposimplestore !)
-  linking [   <=>] 15\r (no-eol) (esc) 
(reposimplestore !)
-  linking [<=>   ] 16\r (no-eol) (esc) 
(reposimplestore !)

D10858: revlogv2: fix `hg verify` with revlog v2

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: indygreg.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  We added a bunch of new files but we never teached some internal about it. 
This
  is now fixed.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10858

AFFECTED FILES
  mercurial/revlog.py
  tests/test-revlog-v2.t

CHANGE DETAILS

diff --git a/tests/test-revlog-v2.t b/tests/test-revlog-v2.t
--- a/tests/test-revlog-v2.t
+++ b/tests/test-revlog-v2.t
@@ -119,3 +119,21 @@
 The two repository should be identical, this diff MUST be empty
 
   $ cmp ../tip-new ../tip-cloned || diff -U8 ../tip-new ../tip-cloned
+
+
+hg verify should be happy
+-
+
+  $ hg verify
+  checking changesets
+  checking manifests
+  crosschecking files in changesets and manifests
+  checking files
+  checked 1 changesets with 1 changes to 1 files
+
+  $ hg verify -R ../cloned-repo
+  checking changesets
+  checking manifests
+  crosschecking files in changesets and manifests
+  checking files
+  checked 1 changesets with 1 changes to 1 files
diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -2923,8 +2923,15 @@
 
 def files(self):
 res = [self._indexfile]
-if not self._inline:
-res.append(self._datafile)
+if self._docket_file is None:
+if not self._inline:
+res.append(self._datafile)
+else:
+res.append(self._docket_file)
+if self._docket.data_end:
+res.append(self._datafile)
+if self._docket.sidedata_end:
+res.append(self._sidedatafile)
 return res
 
 def emitrevisions(



To: marmoute, indygreg, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10857: revlogv2: also test that local clone works

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is now the case so lets add a test.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10857

AFFECTED FILES
  tests/test-revlog-v2.t

CHANGE DETAILS

diff --git a/tests/test-revlog-v2.t b/tests/test-revlog-v2.t
--- a/tests/test-revlog-v2.t
+++ b/tests/test-revlog-v2.t
@@ -18,8 +18,8 @@
   > revlogv2 = enable-unstable-format-and-corrupt-my-data
   > EOF
 
-  $ hg init empty-repo
-  $ cd empty-repo
+  $ hg init new-repo
+  $ cd new-repo
   $ cat .hg/requires
   dotencode
   exp-dirstate-v2 (dirstate-v2 !)
@@ -94,3 +94,28 @@
   .hg/store/00manifest-43c37dde.dat
   .hg/store/00manifest-e2c9362a.sda
   .hg/store/00manifest.i
+
+Local clone works
+-
+
+  $ hg clone . ../cloned-repo
+  updating to branch default
+  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  $ hg tip | tee ../tip-new
+  changeset:   0:96ee1d7354c4
+  tag: tip
+  user:test
+  date:Thu Jan 01 00:00:00 1970 +
+  summary: initial
+  
+  $ hg tip -R ../cloned-repo | tee ../tip-cloned
+  changeset:   0:96ee1d7354c4
+  tag: tip
+  user:test
+  date:Thu Jan 01 00:00:00 1970 +
+  summary: initial
+  
+
+The two repository should be identical, this diff MUST be empty
+
+  $ cmp ../tip-new ../tip-cloned || diff -U8 ../tip-new ../tip-cloned



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10853: copyfile: add a option callback for failed hardlinking

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Local clone, adjust its UI depending on the success of using hardlinking, so 
we
  add a small callback making it possible for `copyfile` to signal if the
  requested hardlinking failed.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10853

AFFECTED FILES
  mercurial/util.py

CHANGE DETAILS

diff --git a/mercurial/util.py b/mercurial/util.py
--- a/mercurial/util.py
+++ b/mercurial/util.py
@@ -1911,7 +1911,13 @@
 
 
 def copyfile(
-src, dest, hardlink=False, copystat=False, checkambig=False, nb_bytes=None
+src,
+dest,
+hardlink=False,
+copystat=False,
+checkambig=False,
+nb_bytes=None,
+no_hardlink_cb=None,
 ):
 """copy a file, preserving mode and optionally other stat info like
 atime/mtime
@@ -1938,6 +1944,8 @@
 except OSError:
 fstype = None
 if fstype not in _hardlinkfswhitelist:
+if no_hardlink_cb is not None:
+no_hardlink_cb()
 hardlink = False
 if hardlink:
 try:
@@ -1946,8 +1954,10 @@
 m = "the `nb_bytes` argument is incompatible with `hardlink`"
 raise error.ProgrammingError(m)
 return
-except (IOError, OSError):
-pass  # fall back to normal copy
+except (IOError, OSError) as exc:
+if exc.errno != errno.EEXIST and no_hardlink_cb is not None:
+no_hardlink_cb()
+# fall back to normal copy
 if os.path.islink(src):
 os.symlink(os.readlink(src), dest)
 # copytime is ignored for symlinks, but in general copytime isn't 
needed



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10854: copyfiles: add a way to relax the file system checking for hardlink

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is critical for transaction file, less for hardlink/copy clone as we are
  about to do. Since `pure` build does not have a `getfstype` implementation 
this
  would disable hardlink clone for all pure build. So we add a parameter to
  control that extra check.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10854

AFFECTED FILES
  mercurial/util.py

CHANGE DETAILS

diff --git a/mercurial/util.py b/mercurial/util.py
--- a/mercurial/util.py
+++ b/mercurial/util.py
@@ -1918,6 +1918,7 @@
 checkambig=False,
 nb_bytes=None,
 no_hardlink_cb=None,
+check_fs_hardlink=True,
 ):
 """copy a file, preserving mode and optionally other stat info like
 atime/mtime
@@ -1936,7 +1937,7 @@
 if checkambig:
 oldstat = checkambig and filestat.frompath(dest)
 unlink(dest)
-if hardlink:
+if hardlink and check_fs_hardlink:
 # Hardlinks are problematic on CIFS (issue4546), do not allow hardlinks
 # unless we are confident that dest is on a whitelisted filesystem.
 try:



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10856: clone: also report the bookmark file as copied

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is a small UI adjustement, but this is easy enough to do.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10856

AFFECTED FILES
  mercurial/streamclone.py

CHANGE DETAILS

diff --git a/mercurial/streamclone.py b/mercurial/streamclone.py
--- a/mercurial/streamclone.py
+++ b/mercurial/streamclone.py
@@ -824,6 +824,16 @@
 
 with dest_repo.lock():
 with src_repo.lock():
+
+# bookmark is not integrated to the streaming as it might the
+# `repo.vfs` and they are too many sentitive data accessible
+# through `repo.vfs` to expose it to streaming clone.
+src_book_vfs = bookmarks.bookmarksvfs(src_repo)
+srcbookmarks = src_book_vfs.join(b'bookmarks')
+bm_count = 0
+if os.path.exists(srcbookmarks):
+bm_count = 1
+
 entries, totalfilesize = _v2_walk(
 src_repo,
 includes=None,
@@ -834,7 +844,7 @@
 dest_vfs_map = _makemap(dest_repo)
 progress = src_repo.ui.makeprogress(
 topic=_(b'linking'),
-total=len(entries),
+total=len(entries) + bm_count,
 unit=_(b'files'),
 )
 # copy  files
@@ -848,18 +858,16 @@
 hardlink = _copy_files(src_vfs_map, dest_vfs_map, files, progress)
 
 # copy bookmarks over
-src_book_vfs = bookmarks.bookmarksvfs(src_repo)
-srcbookmarks = src_book_vfs.join(b'bookmarks')
-dst_book_vfs = bookmarks.bookmarksvfs(dest_repo)
-dstbookmarks = dst_book_vfs.join(b'bookmarks')
-if os.path.exists(srcbookmarks):
+if bm_count:
+dst_book_vfs = bookmarks.bookmarksvfs(dest_repo)
+dstbookmarks = dst_book_vfs.join(b'bookmarks')
 util.copyfile(srcbookmarks, dstbookmarks)
 progress.complete()
 if hardlink:
 msg = b'linked %d files\n'
 else:
 msg = b'copied %d files\n'
-src_repo.ui.debug(msg % len(entries))
+src_repo.ui.debug(msg % (len(entries) + bm_count))
 
 with dest_repo.transaction(b"localclone") as tr:
 dest_repo.store.write(tr)



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10850: clone: use "official" API to create local clone destination

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This make sure we have a properly created, fully functional repository early.
  This will be useful to simply the hardlink/copy phases of the local clone to
  make it share more of its logic with the similar "stream" cloning.
  
  This has a minor impact of the test and the resulting repository has is 
better initialized (eg: the `wcache` directory is pre-created.)

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10850

AFFECTED FILES
  mercurial/hg.py
  tests/test-empty.t
  tests/test-extension.t

CHANGE DETAILS

diff --git a/tests/test-extension.t b/tests/test-extension.t
--- a/tests/test-extension.t
+++ b/tests/test-extension.t
@@ -111,6 +111,7 @@
   reposetup called for a
   ui == repo.ui
   uipopulate called (1 times)
+  uipopulate called (1 times)
   reposetup called for b
   ui == repo.ui
   updating to branch default
diff --git a/tests/test-empty.t b/tests/test-empty.t
--- a/tests/test-empty.t
+++ b/tests/test-empty.t
@@ -48,6 +48,7 @@
   hgrc
   requires
   store
+  wcache
 
 Should be empty:
 
diff --git a/mercurial/hg.py b/mercurial/hg.py
--- a/mercurial/hg.py
+++ b/mercurial/hg.py
@@ -857,20 +857,17 @@
 
 if copy:
 srcrepo.hook(b'preoutgoing', throw=True, source=b'clone')
-hgdir = os.path.realpath(os.path.join(dest, b".hg"))
-if not os.path.exists(dest):
-util.makedirs(dest)
-try:
-destpath = hgdir
-util.makedir(destpath, notindexed=True)
-except OSError as inst:
-if inst.errno == errno.EEXIST:
-cleandir = None
-raise error.Abort(
-_(b"destination '%s' already exists") % dest
-)
-raise
 
+destrootpath = urlutil.urllocalpath(dest)
+dest_reqs = localrepo.clone_requirements(ui, createopts, srcrepo)
+localrepo.createrepository(
+ui,
+destrootpath,
+requirements=dest_reqs,
+)
+destrepo = localrepo.makelocalrepository(ui, destrootpath)
+
+destpath = destrepo.vfs.base
 destlock = copystore(ui, srcrepo, destpath)
 # copy bookmarks over
 srcbookmarks = srcrepo.vfs.join(b'bookmarks')



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10852: streamingclone: extract the scanning part from the generation part

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  We will reuse the scanning part for local clone, so we need it in a dedicated
  function.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10852

AFFECTED FILES
  mercurial/streamclone.py

CHANGE DETAILS

diff --git a/mercurial/streamclone.py b/mercurial/streamclone.py
--- a/mercurial/streamclone.py
+++ b/mercurial/streamclone.py
@@ -603,6 +603,47 @@
 """a function for synchronisation during tests"""
 
 
+def _v2_walk(repo, includes, excludes, includeobsmarkers):
+"""emit a seris of files information useful to clone a repo
+
+return (entries, totalfilesize)
+
+entries is a list of tuple (vfs-key, file-path, file-type, size)
+
+- `vfs-key`: is a key to the right vfs to write the file (see _makemap)
+- `name`: file path of the file to copy (to be feed to the vfss)
+- `file-type`: do this file need to be copied with the source lock ?
+- `size`: the size of the file (or None)
+"""
+assert repo._currentlock(repo._lockref) is not None
+entries = []
+totalfilesize = 0
+
+matcher = None
+if includes or excludes:
+matcher = narrowspec.match(repo.root, includes, excludes)
+
+for rl_type, name, ename, size in _walkstreamfiles(repo, matcher):
+if size:
+ft = _fileappend
+if rl_type & store.FILEFLAGS_VOLATILE:
+ft = _filefull
+entries.append((_srcstore, name, ft, size))
+totalfilesize += size
+for name in _walkstreamfullstorefiles(repo):
+if repo.svfs.exists(name):
+totalfilesize += repo.svfs.lstat(name).st_size
+entries.append((_srcstore, name, _filefull, None))
+if includeobsmarkers and repo.svfs.exists(b'obsstore'):
+totalfilesize += repo.svfs.lstat(b'obsstore').st_size
+entries.append((_srcstore, b'obsstore', _filefull, None))
+for name in cacheutil.cachetocopy(repo):
+if repo.cachevfs.exists(name):
+totalfilesize += repo.cachevfs.lstat(name).st_size
+entries.append((_srccache, name, _filefull, None))
+return entries, totalfilesize
+
+
 def generatev2(repo, includes, excludes, includeobsmarkers):
 """Emit content for version 2 of a streaming clone.
 
@@ -618,32 +659,14 @@
 
 with repo.lock():
 
-entries = []
-totalfilesize = 0
-
-matcher = None
-if includes or excludes:
-matcher = narrowspec.match(repo.root, includes, excludes)
+repo.ui.debug(b'scanning\n')
 
-repo.ui.debug(b'scanning\n')
-for rl_type, name, ename, size in _walkstreamfiles(repo, matcher):
-if size:
-ft = _fileappend
-if rl_type & store.FILEFLAGS_VOLATILE:
-ft = _filefull
-entries.append((_srcstore, name, ft, size))
-totalfilesize += size
-for name in _walkstreamfullstorefiles(repo):
-if repo.svfs.exists(name):
-totalfilesize += repo.svfs.lstat(name).st_size
-entries.append((_srcstore, name, _filefull, None))
-if includeobsmarkers and repo.svfs.exists(b'obsstore'):
-totalfilesize += repo.svfs.lstat(b'obsstore').st_size
-entries.append((_srcstore, b'obsstore', _filefull, None))
-for name in cacheutil.cachetocopy(repo):
-if repo.cachevfs.exists(name):
-totalfilesize += repo.cachevfs.lstat(name).st_size
-entries.append((_srccache, name, _filefull, None))
+entries, totalfilesize = _v2_walk(
+repo,
+includes=includes,
+excludes=excludes,
+includeobsmarkers=includeobsmarkers,
+)
 
 chunks = _emit2(repo, entries, totalfilesize)
 first = next(chunks)



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10849: localrepo: introduce a clone_requirements function

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This function take a source repository and return a relevant set of
  requirements that should be used by a copy clone.
  
  This will help make the creation of the destination repository during copy
  clone simpler.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10849

AFFECTED FILES
  mercurial/localrepo.py

CHANGE DETAILS

diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -3485,6 +3485,24 @@
 return createopts
 
 
+def clone_requirements(ui, createopts, srcrepo):
+"""clone the requirements of a local repo for a local clone
+
+The store requirements are unchanged while the working copy requirements
+depends on the configuration
+"""
+target_requirements = set()
+createopts = defaultcreateopts(ui, createopts=createopts)
+for r in newreporequirements(ui, createopts):
+if r in requirementsmod.WORKING_DIR_REQUIREMENTS:
+target_requirements.add(r)
+
+for r in srcrepo.requirements:
+if r not in requirementsmod.WORKING_DIR_REQUIREMENTS:
+target_requirements.add(r)
+return target_requirements
+
+
 def newreporequirements(ui, createopts):
 """Determine the set of requirements for a new local repository.
 



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10851: vfs: add a `register_file` method on the vfs class

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is used by the fncache vfs to register new file. Until now, `fncache` 
have
  been doing this "automatically" by monitoring write pattern. However this is
  fragile and when we copy files in place by other means, we need something more
  robuts. So we add an explicit method to do so.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10851

AFFECTED FILES
  mercurial/store.py
  mercurial/vfs.py

CHANGE DETAILS

diff --git a/mercurial/vfs.py b/mercurial/vfs.py
--- a/mercurial/vfs.py
+++ b/mercurial/vfs.py
@@ -329,6 +329,9 @@
 None  # pytype: disable=attribute-error
 )
 
+def register_file(self, path):
+"""generic hook point to lets fncache steer its stew"""
+
 
 class vfs(abstractvfs):
 """Operate files relative to a base directory
diff --git a/mercurial/store.py b/mercurial/store.py
--- a/mercurial/store.py
+++ b/mercurial/store.py
@@ -730,6 +730,11 @@
 else:
 return self.vfs.join(path)
 
+def register_file(self, path):
+"""generic hook point to lets fncache steer its stew"""
+if path.startswith(b'data/') or path.startswith(b'meta/'):
+self.fncache.add(path)
+
 
 class fncachestore(basicstore):
 def __init__(self, path, vfstype, dotencode):



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10848: createrepository: allow to directly pass the target requirements

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is useful when doing a local clone that copies store contents, it will
  requires the destination to use the very same store requirements so directly
  providing them will be simpler and safer

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10848

AFFECTED FILES
  mercurial/localrepo.py

CHANGE DETAILS

diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -3677,11 +3677,13 @@
 return {k: v for k, v in createopts.items() if k not in known}
 
 
-def createrepository(ui, path, createopts=None):
+def createrepository(ui, path, createopts=None, requirements=None):
 """Create a new repository in a vfs.
 
 ``path`` path to the new repo's working directory.
 ``createopts`` options for the new repository.
+``requirement`` predefined set of requirements.
+(incompatible with ``createopts``)
 
 The following keys for ``createopts`` are recognized:
 
@@ -3704,27 +3706,34 @@
Indicates that storage for files should be shallow (not all ancestor
revisions are known).
 """
-createopts = defaultcreateopts(ui, createopts=createopts)
-
-unknownopts = filterknowncreateopts(ui, createopts)
-
-if not isinstance(unknownopts, dict):
-raise error.ProgrammingError(
-b'filterknowncreateopts() did not return a dict'
-)
-
-if unknownopts:
-raise error.Abort(
-_(
-b'unable to create repository because of unknown '
-b'creation option: %s'
+
+if requirements is not None:
+if createopts is not None:
+msg = b'cannot specify both createopts and requirements'
+raise error.ProgrammingError(msg)
+createopts = {}
+else:
+createopts = defaultcreateopts(ui, createopts=createopts)
+
+unknownopts = filterknowncreateopts(ui, createopts)
+
+if not isinstance(unknownopts, dict):
+raise error.ProgrammingError(
+b'filterknowncreateopts() did not return a dict'
 )
-% b', '.join(sorted(unknownopts)),
-hint=_(b'is a required extension not loaded?'),
-)
-
-requirements = newreporequirements(ui, createopts=createopts)
-requirements -= checkrequirementscompat(ui, requirements)
+
+if unknownopts:
+raise error.Abort(
+_(
+b'unable to create repository because of unknown '
+b'creation option: %s'
+)
+% b', '.join(sorted(unknownopts)),
+hint=_(b'is a required extension not loaded?'),
+)
+
+requirements = newreporequirements(ui, createopts=createopts)
+requirements -= checkrequirementscompat(ui, requirements)
 
 wdirvfs = vfsmod.vfs(path, expandpath=True, realpath=True)
 



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10847: copyfile: add overlay in the whitelist for hardlink

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is the "default" file system for docker. We don't know if the hardlink
  will actually succeed, but the odds are good it will. They are a second layer
  of fallback in case the hardlink do not actually works.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10847

AFFECTED FILES
  mercurial/util.py

CHANGE DETAILS

diff --git a/mercurial/util.py b/mercurial/util.py
--- a/mercurial/util.py
+++ b/mercurial/util.py
@@ -1901,6 +1901,7 @@
 b'hfs',
 b'jfs',
 b'NTFS',
+b'overlay',
 b'reiserfs',
 b'tmpfs',
 b'ufs',



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10845: transaction: narrow the error filtering when failing to rename undo file

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Having inconsistent "undo" files can be quite serious so we narrow the error
  filtering to the intent that the comment explain.
  
  (This is an opportunity improvement while looking at something next to that.)

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10845

AFFECTED FILES
  mercurial/localrepo.py

CHANGE DETAILS

diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -3445,8 +3445,9 @@
 vfs.tryunlink(dest)
 try:
 vfs.rename(src, dest)
-except OSError:  # journal file does not yet exist
-pass
+except OSError as exc:  # journal file does not yet exist
+if exc.errno != errno.ENOENT:
+raise
 
 return a
 



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10846: transaction: simplify `undo.backupfiles` file creation with a variable

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This make the code shorter and clearer.
  
  (This is an opportunity improvement while looking at something next to that.)

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10846

AFFECTED FILES
  mercurial/transaction.py

CHANGE DETAILS

diff --git a/mercurial/transaction.py b/mercurial/transaction.py
--- a/mercurial/transaction.py
+++ b/mercurial/transaction.py
@@ -633,9 +633,9 @@
 """write transaction data for possible future undo call"""
 if self._undoname is None:
 return
-undobackupfile = self._opener.open(
-b"%s.backupfiles" % self._undoname, b'w'
-)
+
+undo_backup_path = b"%s.backupfiles" % self._undoname
+undobackupfile = self._opener.open(undo_backup_path, b'w')
 undobackupfile.write(b'%d\n' % version)
 for l, f, b, c in self._backupentries:
 if not f:  # temporary file



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10841: copyfiles: deal with existing file when hardlinking

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  If the hardlinking fails, we fallback to `shutil.copy`, but do not consider
  future hardlinking doomed.
  
  This is a slight improvement from the current situation, we still avoid
  hardliking in a case we might be able to do it. However this does not have an
  impact of the rest of the operation.
  
  (This is an opportunity improvement while looking at something next to that.)

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10841

AFFECTED FILES
  mercurial/util.py

CHANGE DETAILS

diff --git a/mercurial/util.py b/mercurial/util.py
--- a/mercurial/util.py
+++ b/mercurial/util.py
@@ -2009,8 +2009,10 @@
 if hardlink:
 try:
 oslink(src, dst)
-except (IOError, OSError):
-hardlink = False
+except (IOError, OSError) as exc:
+if exc.errno != errno.EEXIST:
+hardlink = False
+# XXX maybe try to relink if the file exist ?
 shutil.copy(src, dst)
 else:
 shutil.copy(src, dst)



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10844: transaction: clarify a conditionnal about version check

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Let us move the short branch early.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10844

AFFECTED FILES
  mercurial/transaction.py

CHANGE DETAILS

diff --git a/mercurial/transaction.py b/mercurial/transaction.py
--- a/mercurial/transaction.py
+++ b/mercurial/transaction.py
@@ -743,15 +743,15 @@
 lines = fp.readlines()
 if lines:
 ver = lines[0][:-1]
-if ver == (b'%d' % version):
+if ver != (b'%d' % version):
+report(BAD_VERSION_MSG)
+else:
 for line in lines[1:]:
 if line:
 # Shave off the trailing newline
 line = line[:-1]
 l, f, b, c = line.split(b'\0')
 backupentries.append((l, f, b, bool(c)))
-else:
-report(BAD_VERSION_MSG)
 
 _playback(
 file,



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10843: transaction: extract message about different version in a constants

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This make the code a bit simpler and clearer.
  
  (This is an opportunity improvement while looking at something next to that.)

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10843

AFFECTED FILES
  mercurial/transaction.py

CHANGE DETAILS

diff --git a/mercurial/transaction.py b/mercurial/transaction.py
--- a/mercurial/transaction.py
+++ b/mercurial/transaction.py
@@ -702,6 +702,11 @@
 self._releasefn = None  # Help prevent cycles.
 
 
+BAD_VERSION_MSG = _(
+b"journal was created by a different version of Mercurial\n"
+)
+
+
 def rollback(opener, vfsmap, file, report, checkambigfiles=None):
 """Rolls back the transaction contained in the given file
 
@@ -746,12 +751,7 @@
 l, f, b, c = line.split(b'\0')
 backupentries.append((l, f, b, bool(c)))
 else:
-report(
-_(
-b"journal was created by a different version of "
-b"Mercurial\n"
-)
-)
+report(BAD_VERSION_MSG)
 
 _playback(
 file,



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10842: transaction: explain why some recovery failed

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Right now we issue a message about failing to recover some file, but not why.
  It seems useful to add some information about that.
  
  (This is an opportunity improvement while looking at something next to that.)

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10842

AFFECTED FILES
  mercurial/transaction.py

CHANGE DETAILS

diff --git a/mercurial/transaction.py b/mercurial/transaction.py
--- a/mercurial/transaction.py
+++ b/mercurial/transaction.py
@@ -94,8 +94,9 @@
 try:
 util.copyfile(backuppath, filepath, checkambig=checkambig)
 backupfiles.append(b)
-except IOError:
-report(_(b"failed to recover %s\n") % f)
+except IOError as exc:
+e_msg = stringutil.forcebytestr(exc)
+report(_(b"failed to recover %s (%s)\n") % (f, e_msg))
 else:
 target = f or b
 try:



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10840: clone: cleanup the "cleanup dir" logic used during local clone

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  With the previous code, there was a small windows were we could delete too 
much
  data. This should no longer be the case.
  
  (This is an opportunity improvement while looking at something next to that.)

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10840

AFFECTED FILES
  mercurial/hg.py

CHANGE DETAILS

diff --git a/mercurial/hg.py b/mercurial/hg.py
--- a/mercurial/hg.py
+++ b/mercurial/hg.py
@@ -825,7 +825,12 @@
 abspath = os.path.abspath(urlutil.urllocalpath(origsource))
 
 if islocal(dest):
-cleandir = dest
+if os.path.exists(dest):
+# only clean up directories we create ourselves
+hgdir = os.path.realpath(os.path.join(dest, b".hg"))
+cleandir = hgdir
+else:
+cleandir = dest
 
 copy = False
 if (
@@ -855,9 +860,6 @@
 hgdir = os.path.realpath(os.path.join(dest, b".hg"))
 if not os.path.exists(dest):
 util.makedirs(dest)
-else:
-# only clean up directories we create ourselves
-cleandir = hgdir
 try:
 destpath = hgdir
 util.makedir(destpath, notindexed=True)



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10831: index: use an explicit constant for INDEX_HEADER format and use it for docket

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This avoid leaking python-3.6 compatibility details too much.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10831

AFFECTED FILES
  mercurial/revlogutils/constants.py
  mercurial/revlogutils/docket.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/docket.py b/mercurial/revlogutils/docket.py
--- a/mercurial/revlogutils/docket.py
+++ b/mercurial/revlogutils/docket.py
@@ -95,9 +95,7 @@
 # * 8 bytes: size of data
 # * 8 bytes: pending size of data
 # * 1 bytes: default compression header
-S_HEADER = struct.Struct(
-pycompat.sysstr(constants.INDEX_HEADER.format) + 'BBc'
-)
+S_HEADER = struct.Struct(constants.INDEX_HEADER_FMT + b'BBc')
 
 
 class RevlogDocket(object):
diff --git a/mercurial/revlogutils/constants.py 
b/mercurial/revlogutils/constants.py
--- a/mercurial/revlogutils/constants.py
+++ b/mercurial/revlogutils/constants.py
@@ -29,7 +29,9 @@
 
 ### main revlog header
 
-INDEX_HEADER = struct.Struct(b">I")
+# We cannot rely on  Struct.format is inconsistent for python <=3.6 versus 
above
+INDEX_HEADER_FMT = b">I"
+INDEX_HEADER = struct.Struct(INDEX_HEADER_FMT)
 
 ## revlog version
 REVLOGV0 = 0



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10822: verify: use some intermediate variables instead of a multi-liner

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is shorter and easier to read as the indentation remains the same.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10822

AFFECTED FILES
  mercurial/verify.py

CHANGE DETAILS

diff --git a/mercurial/verify.py b/mercurial/verify.py
--- a/mercurial/verify.py
+++ b/mercurial/verify.py
@@ -575,12 +575,8 @@
 if f in filenodes:
 fns = [(v, k) for k, v in pycompat.iteritems(filenodes[f])]
 for lr, node in sorted(fns):
-self._err(
-lr,
-_(b"manifest refers to unknown revision %s")
-% short(node),
-f,
-)
+msg = _(b"manifest refers to unknown revision %s")
+self._err(lr, msg % short(node), f)
 progress.complete()
 
 if self.warnorphanstorefiles:



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10821: verify: use some intermediate variables instead of a multi-liner

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is shorter and easier to read as the indentation remains the same.
  
  We extract the long message in a module level constant for clarity.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10821

AFFECTED FILES
  mercurial/verify.py

CHANGE DETAILS

diff --git a/mercurial/verify.py b/mercurial/verify.py
--- a/mercurial/verify.py
+++ b/mercurial/verify.py
@@ -50,6 +50,10 @@
 b"warning: copy source of '%s' not in parents of %s"
 )
 
+WARN_NULLID_COPY_SOURCE = _(
+b"warning: %s@%s: copy source revision is nullid %s:%s\n"
+)
+
 
 class verifier(object):
 def __init__(self, repo, level=None):
@@ -557,13 +561,9 @@
 m = _(b"empty or missing copy source revlog %s:%s")
 self._err(lr, m % (rp[0], short(rp[1])), f)
 elif rp[1] == self.repo.nullid:
-ui.note(
-_(
-b"warning: %s@%s: copy source"
-b" revision is nullid %s:%s\n"
-)
-% (f, lr, rp[0], short(rp[1]))
-)
+msg = WARN_NULLID_COPY_SOURCE
+msg %= (f, lr, rp[0], short(rp[1]))
+ui.note(msg)
 else:
 fl2.rev(rp[1])
 except Exception as inst:



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10820: verify: use some intermediate variables instead of a multi-liner

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is shorter and easier to read as the indentation remains the same.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10820

AFFECTED FILES
  mercurial/verify.py

CHANGE DETAILS

diff --git a/mercurial/verify.py b/mercurial/verify.py
--- a/mercurial/verify.py
+++ b/mercurial/verify.py
@@ -554,15 +554,8 @@
 self._warn(WARN_UNKNOWN_COPY_SOURCE % (f, ctx))
 fl2 = repo.file(rp[0])
 if not len(fl2):
-self._err(
-lr,
-_(
-b"empty or missing copy source revlog "
-b"%s:%s"
-)
-% (rp[0], short(rp[1])),
-f,
-)
+m = _(b"empty or missing copy source revlog %s:%s")
+self._err(lr, m % (rp[0], short(rp[1])), f)
 elif rp[1] == self.repo.nullid:
 ui.note(
 _(



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10819: verify: use some intermediate variables instead of a multi-liner

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is shorter and easier to read as the indentation remains the same.
  
  We extract the long message in a module level constant for clarity.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10819

AFFECTED FILES
  mercurial/verify.py

CHANGE DETAILS

diff --git a/mercurial/verify.py b/mercurial/verify.py
--- a/mercurial/verify.py
+++ b/mercurial/verify.py
@@ -46,6 +46,10 @@
 b"parent-directory manifest refers to unknown revision %s"
 )
 
+WARN_UNKNOWN_COPY_SOURCE = _(
+b"warning: copy source of '%s' not in parents of %s"
+)
+
 
 class verifier(object):
 def __init__(self, repo, level=None):
@@ -547,13 +551,7 @@
 if lr is not None and ui.verbose:
 ctx = lrugetctx(lr)
 if not any(rp[0] in pctx for pctx in 
ctx.parents()):
-self._warn(
-_(
-b"warning: copy source of '%s' not"
-b" in parents of %s"
-)
-% (f, ctx)
-)
+self._warn(WARN_UNKNOWN_COPY_SOURCE % (f, ctx))
 fl2 = repo.file(rp[0])
 if not len(fl2):
 self._err(



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10818: verify: use some intermediate variables instead of a multi-liner

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is shorter and easier to read as the indentation remains the same.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10818

AFFECTED FILES
  mercurial/verify.py

CHANGE DETAILS

diff --git a/mercurial/verify.py b/mercurial/verify.py
--- a/mercurial/verify.py
+++ b/mercurial/verify.py
@@ -514,11 +514,8 @@
 if problem.warning:
 self._warn(problem.warning)
 elif problem.error:
-self._err(
-linkrev if linkrev is not None else lr,
-problem.error,
-f,
-)
+linkrev_msg = linkrev if linkrev is not None else lr
+self._err(linkrev_msg, problem.error, f)
 else:
 raise error.ProgrammingError(
 b'problem instance does not set warning or error '



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10817: verify: use some intermediate variables instead of a multi-liner

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is shorter and easier to read as the indentation remains the same.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10817

AFFECTED FILES
  mercurial/verify.py

CHANGE DETAILS

diff --git a/mercurial/verify.py b/mercurial/verify.py
--- a/mercurial/verify.py
+++ b/mercurial/verify.py
@@ -494,9 +494,8 @@
 storefiles.remove(ff)
 except KeyError:
 if self.warnorphanstorefiles:
-self._warn(
-_(b" warning: revlog '%s' not in fncache!") % ff
-)
+msg = _(b" warning: revlog '%s' not in fncache!")
+self._warn(msg % ff)
 self.fncachewarned = True
 
 if not len(fl) and (self.havecl or self.havemf):



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10816: verify: use some intermediate variables instead of a multi-liner

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is shorter and easier to read as the indentation remains the same.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10816

AFFECTED FILES
  mercurial/verify.py

CHANGE DETAILS

diff --git a/mercurial/verify.py b/mercurial/verify.py
--- a/mercurial/verify.py
+++ b/mercurial/verify.py
@@ -378,12 +378,9 @@
 if dir:
 self._err(c, WARN_PARENT_DIR_UNKNOWN_REV % short(m), label)
 else:
-self._err(
-c,
-_(b"changeset refers to unknown revision %s")
-% short(m),
-label,
-)
+msg = _(b"changeset refers to unknown revision %s")
+msg %= short(m)
+self._err(c, msg, label)
 
 if not dir and subdirnodes:
 self.ui.status(_(b"checking directory manifests\n"))



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10815: verify: use some intermediate variables instead of a multi-liner

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is shorter and easier to read as the indentation remains the same.
  
  We extract the long message in a module level constant for clarity.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10815

AFFECTED FILES
  mercurial/verify.py

CHANGE DETAILS

diff --git a/mercurial/verify.py b/mercurial/verify.py
--- a/mercurial/verify.py
+++ b/mercurial/verify.py
@@ -42,6 +42,10 @@
 b'hint: run "hg debugrebuildfncache" to recover from corrupt fncache\n'
 )
 
+WARN_PARENT_DIR_UNKNOWN_REV = _(
+b"parent-directory manifest refers to unknown revision %s"
+)
+
 
 class verifier(object):
 def __init__(self, repo, level=None):
@@ -372,15 +376,7 @@
 changesetpairs = [(c, m) for m in mflinkrevs for c in 
mflinkrevs[m]]
 for c, m in sorted(changesetpairs):
 if dir:
-self._err(
-c,
-_(
-b"parent-directory manifest refers to unknown"
-b" revision %s"
-)
-% short(m),
-label,
-)
+self._err(c, WARN_PARENT_DIR_UNKNOWN_REV % short(m), label)
 else:
 self._err(
 c,



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10814: verify: use some intermediate variables instead of a multi-liner

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is shorter and easier to read as the indentation remains the same.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10814

AFFECTED FILES
  mercurial/verify.py

CHANGE DETAILS

diff --git a/mercurial/verify.py b/mercurial/verify.py
--- a/mercurial/verify.py
+++ b/mercurial/verify.py
@@ -345,9 +345,8 @@
 if fl == b't':
 if not match.visitdir(fullpath):
 continue
-subdirnodes.setdefault(fullpath + b'/', {}).setdefault(
-fn, []
-).append(lr)
+sdn = subdirnodes.setdefault(fullpath + b'/', {})
+sdn.setdefault(fn, []).append(lr)
 else:
 if not match(fullpath):
 continue



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10813: verify: use some intermediate variables instead of a multi-liner

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is shorter and easier to read as the indentation remains the same.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10813

AFFECTED FILES
  mercurial/verify.py

CHANGE DETAILS

diff --git a/mercurial/verify.py b/mercurial/verify.py
--- a/mercurial/verify.py
+++ b/mercurial/verify.py
@@ -361,12 +361,8 @@
 # code (eg: hash verification, filename are ordered, etc.)
 mfdelta = mfl.get(dir, n).read()
 except Exception as inst:
-self._exc(
-lr,
-_(b"reading full manifest %s") % short(n),
-inst,
-label,
-)
+msg = _(b"reading full manifest %s") % short(n)
+self._exc(lr, msg, inst, label)
 
 if not dir:
 progress.complete()



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10812: verify: use some intermediate variables instead of a multi-liner

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is shorter and easier to read as the indentation remains the same.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10812

AFFECTED FILES
  mercurial/verify.py

CHANGE DETAILS

diff --git a/mercurial/verify.py b/mercurial/verify.py
--- a/mercurial/verify.py
+++ b/mercurial/verify.py
@@ -329,11 +329,8 @@
 if n in mflinkrevs:
 del mflinkrevs[n]
 elif dir:
-self._err(
-lr,
-_(b"%s not in parent-directory manifest") % short(n),
-label,
-)
+msg = _(b"%s not in parent-directory manifest") % short(n)
+self._err(lr, msg, label)
 else:
 self._err(lr, _(b"%s not in changesets") % short(n), label)
 



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10811: verify: use some intermediate variables instead of a multi-liner

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is shorter and easier to read as the indentation remains the same.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10811

AFFECTED FILES
  mercurial/verify.py

CHANGE DETAILS

diff --git a/mercurial/verify.py b/mercurial/verify.py
--- a/mercurial/verify.py
+++ b/mercurial/verify.py
@@ -211,10 +211,9 @@
 if self.errors:
 ui.warn(_(b"%d integrity errors encountered!\n") % self.errors)
 if self.badrevs:
-ui.warn(
-_(b"(first damaged changeset appears to be %d)\n")
-% min(self.badrevs)
-)
+msg = _(b"(first damaged changeset appears to be %d)\n")
+msg %= min(self.badrevs)
+ui.warn(msg)
 return 1
 return 0
 



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10810: verify: use some intermediate variables instead of a multi-liner

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is shorter and easier to read as the indentation remains the same.
  
  We extract the long message in a module level constant for clarity.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10810

AFFECTED FILES
  mercurial/verify.py

CHANGE DETAILS

diff --git a/mercurial/verify.py b/mercurial/verify.py
--- a/mercurial/verify.py
+++ b/mercurial/verify.py
@@ -38,6 +38,11 @@
 return f
 
 
+HINT_FNCACHE = _(
+b'hint: run "hg debugrebuildfncache" to recover from corrupt fncache\n'
+)
+
+
 class verifier(object):
 def __init__(self, repo, level=None):
 self.repo = repo.unfiltered()
@@ -202,12 +207,7 @@
 if self.warnings:
 ui.warn(_(b"%d warnings encountered!\n") % self.warnings)
 if self.fncachewarned:
-ui.warn(
-_(
-b'hint: run "hg debugrebuildfncache" to recover from '
-b'corrupt fncache\n'
-)
-)
+ui.warn(HINT_FNCACHE)
 if self.errors:
 ui.warn(_(b"%d integrity errors encountered!\n") % self.errors)
 if self.badrevs:



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10809: verify: use some intermediate variables instead of a multi-liner

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is shorter and easier to read as the indentation remains the same.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10809

AFFECTED FILES
  mercurial/verify.py

CHANGE DETAILS

diff --git a/mercurial/verify.py b/mercurial/verify.py
--- a/mercurial/verify.py
+++ b/mercurial/verify.py
@@ -156,11 +156,8 @@
 msg = _(b"unknown parent 1 %s of %s") % (short(p1), 
short(node))
 self._err(lr, msg, f)
 if p2 not in seen and p2 != self.repo.nullid:
-self._err(
-lr,
-_(b"unknown parent 2 %s of %s") % (short(p2), short(node)),
-f,
-)
+msg = _(b"unknown parent 2 %s of %s") % (short(p2), 
short(node))
+self._err(lr, msg, f)
 except Exception as inst:
 self._exc(lr, _(b"checking parents of %s") % short(node), inst, f)
 



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10808: verify: use some intermediate variables instead of a multi-liner

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is shorter and easier to read as the indentation remains the same.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10808

AFFECTED FILES
  mercurial/verify.py

CHANGE DETAILS

diff --git a/mercurial/verify.py b/mercurial/verify.py
--- a/mercurial/verify.py
+++ b/mercurial/verify.py
@@ -153,11 +153,8 @@
 try:
 p1, p2 = obj.parents(node)
 if p1 not in seen and p1 != self.repo.nullid:
-self._err(
-lr,
-_(b"unknown parent 1 %s of %s") % (short(p1), short(node)),
-f,
-)
+msg = _(b"unknown parent 1 %s of %s") % (short(p1), 
short(node))
+self._err(lr, msg, f)
 if p2 not in seen and p2 != self.repo.nullid:
 self._err(
 lr,



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10807: verify: use some intermediate variables instead of a multi-liner

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This is shorter and easier to read as the indentation remains the same.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10807

AFFECTED FILES
  mercurial/verify.py

CHANGE DETAILS

diff --git a/mercurial/verify.py b/mercurial/verify.py
--- a/mercurial/verify.py
+++ b/mercurial/verify.py
@@ -145,10 +145,9 @@
 linkrevs.append(lr)
 except Exception:
 pass
-self._warn(
-_(b" (expected %s)")
-% b" ".join(map(pycompat.bytestr, linkrevs))
-)
+msg = _(b" (expected %s)")
+msg %= b" ".join(map(pycompat.bytestr, linkrevs))
+self._warn(msg)
 lr = None  # can't be trusted
 
 try:



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10806: verify: expand a one liner into explicit commands

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  The result is not longer, but it is more edible.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10806

AFFECTED FILES
  mercurial/verify.py

CHANGE DETAILS

diff --git a/mercurial/verify.py b/mercurial/verify.py
--- a/mercurial/verify.py
+++ b/mercurial/verify.py
@@ -139,11 +139,10 @@
 if f and len(linkrevs) > 1:
 try:
 # attempt to filter down to real linkrevs
-linkrevs = [
-l
-for l in linkrevs
-if self.lrugetctx(l)[f].filenode() == node
-]
+linkrevs = []
+for lr in linkrevs:
+if self.lrugetctx(lr)[f].filenode() == node:
+linkrevs.append(lr)
 except Exception:
 pass
 self._warn(



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10805: verify: align a comment line

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This argument description is back with it comrade.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10805

AFFECTED FILES
  mercurial/verify.py

CHANGE DETAILS

diff --git a/mercurial/verify.py b/mercurial/verify.py
--- a/mercurial/verify.py
+++ b/mercurial/verify.py
@@ -114,7 +114,7 @@
 arguments are:
 - obj:  the source revlog
 - i:the revision number
-- node:the revision node id
+- node: the revision node id
 - seen: nodes previously seen for this revlog
 - linkrevs: [changelog-revisions] introducing "node"
 - f:string label ("changelog", "manifest", or filename)



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10802: revlogv2: make sure bundling pick a compatible bundle format

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  Before this change, revlog-v2 repository where bundled using the incompatible
  "v1" format.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10802

AFFECTED FILES
  mercurial/bundlecaches.py
  tests/test-revlog-v2.t

CHANGE DETAILS

diff --git a/tests/test-revlog-v2.t b/tests/test-revlog-v2.t
--- a/tests/test-revlog-v2.t
+++ b/tests/test-revlog-v2.t
@@ -69,6 +69,14 @@
   .hg/store/data/foo.i:
   : 00 00 de ad ||
 
+Bundle use a compatible changegroup format
+--
+
+  $ hg bundle --all ../basic.hg
+  1 changesets found
+  $ hg debugbundle --spec ../basic.hg
+  bzip2-v2
+
 The expected files are generated
 
 
diff --git a/mercurial/bundlecaches.py b/mercurial/bundlecaches.py
--- a/mercurial/bundlecaches.py
+++ b/mercurial/bundlecaches.py
@@ -167,6 +167,8 @@
 # Generaldelta repos require v2.
 if requirementsmod.GENERALDELTA_REQUIREMENT in repo.requirements:
 version = b'v2'
+elif requirementsmod.REVLOGV2_REQUIREMENT in repo.requirements:
+version = b'v2'
 # Modern compression engines require v2.
 if compression not in _bundlespecv1compengines:
 version = b'v2'



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10801: censor: do not process sidedata of censored revision while bundling

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  The revision is censored, we should ignore it.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10801

AFFECTED FILES
  mercurial/utils/storageutil.py

CHANGE DETAILS

diff --git a/mercurial/utils/storageutil.py b/mercurial/utils/storageutil.py
--- a/mercurial/utils/storageutil.py
+++ b/mercurial/utils/storageutil.py
@@ -481,13 +481,18 @@
 serialized_sidedata = None
 sidedata_flags = (0, 0)
 if sidedata_helpers:
-old_sidedata = store.sidedata(rev)
-sidedata, sidedata_flags = sidedatamod.run_sidedata_helpers(
-store=store,
-sidedata_helpers=sidedata_helpers,
-sidedata=old_sidedata,
-rev=rev,
-)
+try:
+old_sidedata = store.sidedata(rev)
+except error.CensoredNodeError:
+# skip any potential sidedata of the censored revision
+sidedata = {}
+else:
+sidedata, sidedata_flags = sidedatamod.run_sidedata_helpers(
+store=store,
+sidedata_helpers=sidedata_helpers,
+sidedata=old_sidedata,
+rev=rev,
+)
 if sidedata:
 serialized_sidedata = sidedatamod.serialize_sidedata(sidedata)
 



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10797: revlog: move `revisioninfo` in `revlogutils`

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: indygreg.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  We will need it in other utility module. So lets extract it from `revlog.py`,
  the module is too large already anyway.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10797

AFFECTED FILES
  mercurial/revlog.py
  mercurial/revlogutils/__init__.py
  mercurial/revlogutils/deltas.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/deltas.py b/mercurial/revlogutils/deltas.py
--- a/mercurial/revlogutils/deltas.py
+++ b/mercurial/revlogutils/deltas.py
@@ -932,7 +932,7 @@
 def buildtext(self, revinfo, fh):
 """Builds a fulltext version of a revision
 
-revinfo: _revisioninfo instance that contains all needed info
+revinfo: revisioninfo instance that contains all needed info
 fh:  file handle to either the .i or the .d revlog file,
  depending on whether it is inlined or not
 """
diff --git a/mercurial/revlogutils/__init__.py 
b/mercurial/revlogutils/__init__.py
--- a/mercurial/revlogutils/__init__.py
+++ b/mercurial/revlogutils/__init__.py
@@ -7,6 +7,7 @@
 
 from __future__ import absolute_import
 
+from ..thirdparty import attr
 from ..interfaces import repository
 
 # See mercurial.revlogutils.constants for doc
@@ -56,3 +57,24 @@
 data_compression_mode,
 sidedata_compression_mode,
 )
+
+
+@attr.s(slots=True, frozen=True)
+class revisioninfo(object):
+"""Information about a revision that allows building its fulltext
+node:   expected hash of the revision
+p1, p2: parent revs of the revision
+btext:  built text cache consisting of a one-element list
+cachedelta: (baserev, uncompressed_delta) or None
+flags:  flags associated to the revision storage
+
+One of btext[0] or cachedelta must be set.
+"""
+
+node = attr.ib()
+p1 = attr.ib()
+p2 = attr.ib()
+btext = attr.ib()
+textlen = attr.ib()
+cachedelta = attr.ib()
+flags = attr.ib()
diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -167,27 +167,6 @@
 )
 
 
-@attr.s(slots=True, frozen=True)
-class _revisioninfo(object):
-"""Information about a revision that allows building its fulltext
-node:   expected hash of the revision
-p1, p2: parent revs of the revision
-btext:  built text cache consisting of a one-element list
-cachedelta: (baserev, uncompressed_delta) or None
-flags:  flags associated to the revision storage
-
-One of btext[0] or cachedelta must be set.
-"""
-
-node = attr.ib()
-p1 = attr.ib()
-p2 = attr.ib()
-btext = attr.ib()
-textlen = attr.ib()
-cachedelta = attr.ib()
-flags = attr.ib()
-
-
 @interfaceutil.implementer(repository.irevisiondelta)
 @attr.s(slots=True)
 class revlogrevisiondelta(object):
@@ -2530,7 +2509,15 @@
 if deltacomputer is None:
 deltacomputer = deltautil.deltacomputer(self)
 
-revinfo = _revisioninfo(node, p1, p2, btext, textlen, cachedelta, 
flags)
+revinfo = revlogutils.revisioninfo(
+node,
+p1,
+p2,
+btext,
+textlen,
+cachedelta,
+flags,
+)
 
 deltainfo = deltacomputer.finddeltainfo(revinfo, fh)
 



To: marmoute, indygreg, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10800: revlog: allow to pass an existing docket to `_loadindex()`

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: indygreg.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This will be when switching undelying files, for examples during censors and 
strip operation with rvlog-v2.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10800

AFFECTED FILES
  mercurial/revlog.py

CHANGE DETAILS

diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -487,7 +487,7 @@
 raise
 return b''
 
-def _loadindex(self):
+def _loadindex(self, docket=None):
 
 new_header, mmapindexthreshold, force_nodemap = self._init_opts()
 
@@ -498,45 +498,51 @@
 else:
 entry_point = b'%s.i' % self.radix
 
-entry_data = b''
-self._initempty = True
-entry_data = self._get_data(entry_point, mmapindexthreshold)
-if len(entry_data) > 0:
-header = INDEX_HEADER.unpack(entry_data[:4])[0]
-self._initempty = False
+if docket is not None:
+self._docket = docket
+self._docket_file = entry_point
 else:
-header = new_header
-
-self._format_flags = header & ~0x
-self._format_version = header & 0x
-
-supported_flags = SUPPORTED_FLAGS.get(self._format_version)
-if supported_flags is None:
-msg = _(b'unknown version (%d) in revlog %s')
-msg %= (self._format_version, self.display_id)
-raise error.RevlogError(msg)
-elif self._format_flags & ~supported_flags:
-msg = _(b'unknown flags (%#04x) in version %d revlog %s')
-display_flag = self._format_flags >> 16
-msg %= (display_flag, self._format_version, self.display_id)
-raise error.RevlogError(msg)
-
-features = FEATURES_BY_VERSION[self._format_version]
-self._inline = features[b'inline'](self._format_flags)
-self._generaldelta = features[b'generaldelta'](self._format_flags)
-self.hassidedata = features[b'sidedata']
-
-if not features[b'docket']:
-self._indexfile = entry_point
-index_data = entry_data
-else:
-self._docket_file = entry_point
-if self._initempty:
-self._docket = docketutil.default_docket(self, header)
+entry_data = b''
+self._initempty = True
+entry_data = self._get_data(entry_point, mmapindexthreshold)
+if len(entry_data) > 0:
+header = INDEX_HEADER.unpack(entry_data[:4])[0]
+self._initempty = False
 else:
-self._docket = docketutil.parse_docket(
-self, entry_data, use_pending=self._trypending
-)
+header = new_header
+
+self._format_flags = header & ~0x
+self._format_version = header & 0x
+
+supported_flags = SUPPORTED_FLAGS.get(self._format_version)
+if supported_flags is None:
+msg = _(b'unknown version (%d) in revlog %s')
+msg %= (self._format_version, self.display_id)
+raise error.RevlogError(msg)
+elif self._format_flags & ~supported_flags:
+msg = _(b'unknown flags (%#04x) in version %d revlog %s')
+display_flag = self._format_flags >> 16
+msg %= (display_flag, self._format_version, self.display_id)
+raise error.RevlogError(msg)
+
+features = FEATURES_BY_VERSION[self._format_version]
+self._inline = features[b'inline'](self._format_flags)
+self._generaldelta = features[b'generaldelta'](self._format_flags)
+self.hassidedata = features[b'sidedata']
+
+if not features[b'docket']:
+self._indexfile = entry_point
+index_data = entry_data
+else:
+self._docket_file = entry_point
+if self._initempty:
+self._docket = docketutil.default_docket(self, header)
+else:
+self._docket = docketutil.parse_docket(
+self, entry_data, use_pending=self._trypending
+)
+
+if self._docket is not None:
 self._indexfile = self._docket.index_filepath()
 index_data = b''
 index_size = self._docket.index_end



To: marmoute, indygreg, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10798: util: add `nb_bytes` argument to `copyfile` to partially copy a file

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  When set, this allow to copy only the first `nb_bytes` of a file. This will be
  useful for censor/strip operation with revlogv2.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10798

AFFECTED FILES
  mercurial/util.py

CHANGE DETAILS

diff --git a/mercurial/util.py b/mercurial/util.py
--- a/mercurial/util.py
+++ b/mercurial/util.py
@@ -1909,7 +1909,9 @@
 }
 
 
-def copyfile(src, dest, hardlink=False, copystat=False, checkambig=False):
+def copyfile(
+src, dest, hardlink=False, copystat=False, checkambig=False, nb_bytes=None
+):
 """copy a file, preserving mode and optionally other stat info like
 atime/mtime
 
@@ -1918,6 +1920,8 @@
 repo.wlock).
 
 copystat and checkambig should be exclusive.
+
+nb_bytes: if set only copy the first `nb_bytes` of the source file.
 """
 assert not (copystat and checkambig)
 oldstat = None
@@ -1937,6 +1941,9 @@
 if hardlink:
 try:
 oslink(src, dest)
+if nb_bytes is not None:
+m = "the `nb_bytes` argument is incompatible with nb_bytes"
+raise error.ProgrammingError(m)
 return
 except (IOError, OSError):
 pass  # fall back to normal copy
@@ -1944,6 +1951,9 @@
 os.symlink(os.readlink(src), dest)
 # copytime is ignored for symlinks, but in general copytime isn't 
needed
 # for them anyway
+if nb_bytes is not None:
+m = "cannot use `nb_bytes` on a symlink"
+raise error.ProgrammingError(m)
 else:
 try:
 shutil.copyfile(src, dest)
@@ -1960,6 +1970,10 @@
 oldstat.stat[stat.ST_MTIME] + 1
 ) & 0x7FFF
 os.utime(dest, (advanced, advanced))
+# We could do something smarter using `copy_file_range` call or 
similar
+if nb_bytes is not None:
+with open(dest, mode='r+') as f:
+f.truncate(nb_bytes)
 except shutil.Error as inst:
 raise error.Abort(stringutil.forcebytestr(inst))
 



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D10799: revlog: add a ways to blacklist some revision when searching for a delta

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  This will be useful to recompute appropriate deltas one the fly during
  censor/strip operation with revlog-v2.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10799

AFFECTED FILES
  mercurial/revlogutils/deltas.py

CHANGE DETAILS

diff --git a/mercurial/revlogutils/deltas.py b/mercurial/revlogutils/deltas.py
--- a/mercurial/revlogutils/deltas.py
+++ b/mercurial/revlogutils/deltas.py
@@ -1050,7 +1050,7 @@
 snapshotdepth,
 )
 
-def finddeltainfo(self, revinfo, fh):
+def finddeltainfo(self, revinfo, fh, excluded_base=None):
 """Find an acceptable delta against a candidate revision
 
 revinfo: information about the revision (instance of _revisioninfo)
@@ -1066,6 +1066,9 @@
 if not revinfo.textlen:
 return self._fullsnapshotinfo(fh, revinfo)
 
+if excluded_base is None:
+excluded_base = set()
+
 # no delta for flag processor revision (see "candelta" for why)
 # not calling candelta since only one revision needs test, also to
 # avoid overhead fetching flags again.
@@ -1090,6 +1093,8 @@
 # challenge it against refined candidates
 nominateddeltas.append(deltainfo)
 for candidaterev in candidaterevs:
+if candidaterev in excluded_base:
+continue
 candidatedelta = self._builddeltainfo(revinfo, candidaterev, 
fh)
 if candidatedelta is not None:
 if isgooddeltainfo(self.revlog, candidatedelta, revinfo):



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel