[PATCH] hgweb: don't try to wrap mod_wsgi loggers

[Ludovic Chabant]

# HG changeset patch
# User Ludovic Chabant 
# Date 1601413962 25200
#  Tue Sep 29 14:12:42 2020 -0700
# Node ID f8726d166fcaa0d690f6783146172758b2e14deb
# Parent  80bf7b1ada15622ea45b8ecc5647404f5acb2905
hgweb: don't try to wrap mod_wsgi loggers

diff --git a/mercurial/utils/procutil.py b/mercurial/utils/procutil.py
--- a/mercurial/utils/procutil.py
+++ b/mercurial/utils/procutil.py
@@ -108,6 +108,10 @@
 # The io.BufferedIOBase.write() contract guarantees that all data is
 # written.
 return stream
+if str(type(stream).__module__) == 'mod_wsgi':
+   # WSGI loggers write all data. We couldn't even wrap them anyway since
+   # they typically don't return the number of bytes written.
+   return stream
 # In general, the write() method of streams is free to write only part of
 # the data.
 return WriteAllWrapper(stream)
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

[PATCH] hgdemandimport: exclude more sqlalchemy modules

[Ludovic Chabant]

# HG changeset patch
# User Ludovic Chabant 
# Date 1601415526 25200
#  Tue Sep 29 14:38:46 2020 -0700
# Node ID 0b78eb21c79f02980e59ae30c6049075534b
# Parent  f8726d166fcaa0d690f6783146172758b2e14deb
hgdemandimport: exclude more sqlalchemy modules

We could potentially exclude the entire sqlalchemy library.

diff --git a/hgdemandimport/__init__.py b/hgdemandimport/__init__.py
--- a/hgdemandimport/__init__.py
+++ b/hgdemandimport/__init__.py
@@ -52,6 +52,7 @@
 'rfc822',
 'mimetools',
 'sqlalchemy.events',  # has import-time side effects (issue5085)
+'sqlalchemy.dialects', # similar problems as above
 # setuptools 8 expects this module to explode early when not on windows
 'distutils.msvc9compiler',
 '__builtin__',
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9121: hg-core: fix path encoding usage

[acezar (Antoine Cezar)]

acezar created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  1. Hash encoded path are in `.hg/store/dh` instead of `.hg/store/data`.
  
  2. Path encoded index and data files may not have the same parent path. It is 
not just about replacing `.i` by `.d`

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9121

AFFECTED FILES
  rust/hg-core/src/operations/cat.rs
  rust/hg-core/src/operations/debugdata.rs
  rust/hg-core/src/revlog/changelog.rs
  rust/hg-core/src/revlog/manifest.rs
  rust/hg-core/src/revlog/revlog.rs

CHANGE DETAILS

diff --git a/rust/hg-core/src/revlog/revlog.rs 
b/rust/hg-core/src/revlog/revlog.rs
--- a/rust/hg-core/src/revlog/revlog.rs
+++ b/rust/hg-core/src/revlog/revlog.rs
@@ -47,7 +47,10 @@
 /// It will also open the associated data file if index and data are not
 /// interleaved.
 #[timed]
-pub fn open(index_path: ) -> Result {
+pub fn open(
+index_path: ,
+data_path: Option<>,
+) -> Result {
 let index_mmap =
 mmap_open(_path).map_err(RevlogError::IoError)?;
 
@@ -58,6 +61,8 @@
 
 let index = Index::new(Box::new(index_mmap))?;
 
+let default_data_path = index_path.with_extension("d");
+
 // TODO load data only when needed //
 // type annotation required
 // won't recognize Mmap as Deref
@@ -65,9 +70,9 @@
 if index.is_inline() {
 None
 } else {
-let data_path = index_path.with_extension("d");
+let data_path = data_path.unwrap_or(_data_path);
 let data_mmap =
-mmap_open(_path).map_err(RevlogError::IoError)?;
+mmap_open(data_path).map_err(RevlogError::IoError)?;
 Some(Box::new(data_mmap))
 };
 
diff --git a/rust/hg-core/src/revlog/manifest.rs 
b/rust/hg-core/src/revlog/manifest.rs
--- a/rust/hg-core/src/revlog/manifest.rs
+++ b/rust/hg-core/src/revlog/manifest.rs
@@ -13,7 +13,7 @@
 /// Open the `manifest` of a repository given by its root.
 pub fn open(root: ) -> Result {
 let index_file = root.join(".hg/store/00manifest.i");
-let revlog = Revlog::open(_file)?;
+let revlog = Revlog::open(_file, None)?;
 Ok(Self { revlog })
 }
 
diff --git a/rust/hg-core/src/revlog/changelog.rs 
b/rust/hg-core/src/revlog/changelog.rs
--- a/rust/hg-core/src/revlog/changelog.rs
+++ b/rust/hg-core/src/revlog/changelog.rs
@@ -12,7 +12,7 @@
 /// Open the `changelog` of a repository given by its root.
 pub fn open(root: ) -> Result {
 let index_file = root.join(".hg/store/00changelog.i");
-let revlog = Revlog::open(_file)?;
+let revlog = Revlog::open(_file, None)?;
 Ok(Self { revlog })
 }
 
diff --git a/rust/hg-core/src/operations/debugdata.rs 
b/rust/hg-core/src/operations/debugdata.rs
--- a/rust/hg-core/src/operations/debugdata.rs
+++ b/rust/hg-core/src/operations/debugdata.rs
@@ -102,7 +102,7 @@
 DebugDataKind::Changelog => root.join(".hg/store/00changelog.i"),
 DebugDataKind::Manifest => root.join(".hg/store/00manifest.i"),
 };
-let revlog = Revlog::open(_file)?;
+let revlog = Revlog::open(_file, None)?;
 let data = revlog.get_rev_data(rev)?;
 
 Ok(data)
diff --git a/rust/hg-core/src/operations/cat.rs 
b/rust/hg-core/src/operations/cat.rs
--- a/rust/hg-core/src/operations/cat.rs
+++ b/rust/hg-core/src/operations/cat.rs
@@ -6,7 +6,7 @@
 // GNU General Public License version 2 or any later version.
 
 use std::convert::From;
-use std::path::PathBuf;
+use std::path::{Path, PathBuf};
 
 use crate::revlog::changelog::Changelog;
 use crate::revlog::manifest::{Manifest, ManifestEntry};
@@ -14,7 +14,7 @@
 use crate::revlog::revlog::Revlog;
 use crate::revlog::revlog::RevlogError;
 use crate::revlog::Revision;
-use crate::utils::hg_path::HgPathBuf;
+use crate::utils::hg_path::{HgPath, HgPathBuf};
 
 /// Kind of error encountered by `CatRev`
 #[derive(Debug)]
@@ -119,15 +119,13 @@
 {
 for cat_file in self.files.iter() {
 if cat_file.as_bytes() == manifest_file.as_bytes() {
-let encoded_bytes =
-path_encode(manifest_file.as_bytes());
-let revlog_index_string = format!(
-".hg/store/data/{}.i",
-String::from_utf8_lossy(_bytes),
-);
-let revlog_index_path =
-self.root.join(_index_string);
-let file_log = Revlog::open(_index_path)?;
+let index_path =
+store_path(self.root, manifest_file, b".i");
+let data_path =
+

Re: Improving the CI experience

[Georges Racinet]

On 9/29/20 10:54 AM, Raphaël Gomès wrote:
>
>
> On 9/28/20 8:12 PM, Augie Fackler wrote:
>>
>>
>>> On Sep 28, 2020, at 10:43, Raphaël Gomès >> > wrote:
>>>
>>> Hi all,
>>>
>>> As you may know, we (Octobus) and other Mercurial contributors have
>>> been using
>>> the Heptapod [1] CI to vet our patches pre-submission against 7
>>> variants of
>>> Mercurial:
>>>   - Python 2 pure
>>>   - Python 3 pure
>>>   - Python 2 python+c
>>>   - Python 3 python+c
>>>   - Python 2 rust+c
>>>   - Python 3 rust+c
>>>   - Python 2 chg
>>>   - (also code checks and Rust unit tests, of course)
>>>
>>> All variants above are run on x86_64 Debian machines. Our CI does
>>> not (yet) have
>>> the MacOS, Windows, or FreeBSDoptionsthat the Mercurial Buildbot
>>> has, but
>>> does test more variants on Linux.
>>>
>>> Nevertheless, it catches a lot of the mistakes that we all make
>>> sometimes, 
>>> before we even send the patches upstream: regressions in Python 3,
>>> API breakage
>>> with Rust, bad formatting, and other bugs of various nature. It does
>>> so way
>>> before it touches the `committed` repo, which means that if I screw
>>> up my
>>> patch, it only has to bother me.
>>>
>>> The value of such a system is highly dependent on it being
>>> trustworthy: flaky
>>> tests do exist, but they should be addressed and fixed in a timely
>>> manner and
>>> my CI being red should mean that *I* missed something. Yet, I have
>>> heard the
>>> phrase "I don't even look at the pipelines anymore, they're always
>>> broken for
>>> something I haven't done".
>>>
>>> Most of the time it's something minor: I've even been guilty of
>>> breaking the CI
>>> myself in my last patch series (oops) due to bad formatting!
>>> However, these
>>> issues should be few and far between: the developer experience
>>> suffers greatly
>>> and we grow to accept that the tests are always broken, don't bother
>>> to run
>>> them at all, etc.
>>>
>>> # Proposal
>>>
>>> I thought a start to fixing this situation would be for me to setup 
>>> an automation to send a small email to this mailing-list every time
>>> either the
>>> `stable` or `default` branches (that track `hg-committed`) are
>>> broken, with a link 
>>> to the pipeline results.
>>> This will not catch everything (only a subset of the possible
>>> targets is tested), but
>>> I think this would be an improvement for all contributors, even
>>> those that don't
>>> use Heptapod.
>>>
>>> What do you all think?
>>
>> I'm open to it. Do you think the CI that you're running on heptapod
>> is faster than the buildbot? Should we keep the buildbot or shut it down?
>>
> Faster in terms of latency, almost certainly, at least for now because
> of the number of workers, and probably in terms of the actual speed of
> the runners, though I'm just making a guess.
>
Looking at a random build on buildbot [1], it took about 2 hours to run
the full test suite without option and with --pure.

In terms of usefulness to the community as a whole, I think we probably
want to define the full latency as the full time it takes between a push
to hg-committed to available results and potential alerts on the mailing
lists.

We can split that in

1. the time it takes for the changesets to arrive in Heptapod.

    That part is not automatic yet, we'd need to improve on that and
that's fairly easy.

    The most efficient would be for hg-committed to send a trigger in a
hook. The service listening to that trigger would then pull from
hg-committed and push to Heptapod.

    Raphaël or I could submit this to mercurial-infra, what do you think ?

2. the time it takes for workers to take the jobs.

    That part depends on how busy the CI system is. In practice, the
jobs start right away more often than not, but we can get 15-20 minutes
of wait sometimes. This is good enough for us now, we have lots of
options if that becomes a problem.

3. the time it takes to run the jobs.

    The equivalent in Heptapod CI of the buildbot build mentioned above
would be the test-py2 and test-py2-pure jobs. They run in parallel in
less than half an hour each. Picking again randomly a pipeline [2], I
see the full set of 11 jobs complete in 25 minutes.

    Some jobs are much faster (Rust cargo tests about 2 minutes, check
tests about 4 minutes). I don't know yet if it would be easy to get fast
feedback about them, just saying there's room for follow-up improvements.

[1] https://buildbot.mercurial-scm.org/builders/hg%20tests/builds/3078

[2] https://foss.heptapod.net/octobus/mercurial-devel/-/pipelines/10794

> However, the Buildbot still has value since it covers cases that our
> CI still does not cover and not everyone is using Heptapod, so let's
> keep it around for a while.
>
Yes these are complementary. However, if we get to the point where the
Heptapod pipelines run after each push to hg-committed and it works
smoothly, we could consider lightening the weight on the buildbot by
removing redundant builds.

> To 

D9119: salvaged: track removal-candidates in more cases

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  If we want to use this information for copy tracing, then we need to be able 
to
  record it.
  
  First we unlock the recording of deletion candidate, and we will actually 
record
  the data in the next changeset.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9119

AFFECTED FILES
  mercurial/merge.py

CHANGE DETAILS

diff --git a/mercurial/merge.py b/mercurial/merge.py
--- a/mercurial/merge.py
+++ b/mercurial/merge.py
@@ -781,7 +781,10 @@
 if (
 pa not in ([wctx, p2] + wctx.parents())
 and not forcefulldiff
-and not repo.ui.configbool(b'experimental', b'merge-track-salvaged')
+and not (
+repo.ui.configbool(b'experimental', b'merge-track-salvaged')
+or repo.filecopiesmode == b'changeset-sidedata'
+)
 ):
 # Identify which files are relevant to the merge, so we can limit the
 # total m1-vs-m2 diff to just those files. This has significant



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9120: salvaged: record salvaged in ChangingFiles at commit time

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  The new code is a simple but effective way to detect this information. We 
might
  be able to move it inside the various conditionnal above, but I want to focus
  on simplicity until we have a full working stack.
  
  It is worth noting that if we record the information in the ChangingFiles
  object, it is not persisted yet. This will comes with later changesets.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9120

AFFECTED FILES
  mercurial/commit.py

CHANGE DETAILS

diff --git a/mercurial/commit.py b/mercurial/commit.py
--- a/mercurial/commit.py
+++ b/mercurial/commit.py
@@ -140,6 +140,21 @@
 files.update_copies_from_p1(ctx.p1copies())
 files.update_copies_from_p2(ctx.p2copies())
 
+copy_sd = ctx.repo().filecopiesmode == b'changeset-sidedata'
+if copy_sd and len(ctx.parents()) > 1:
+# XXX this `mergestate.read` could be duplicated with a the merge state
+# reading in _process_files So we could refactor further to reuse it in
+# some cases.
+ms = mergestate.mergestate.read(repo)
+if ms.active():
+for fname in sorted(ms._stateextras.keys()):
+if (
+ms.extras(fname).get(b'FILE_MIGHT_DELETE_BY_MERGE')
+== b'yes'
+):
+if fname in ctx:
+files.mark_undeleted(fname)
+
 return mn, files
 
 



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D9118: changing-files: add a "salvaged" set to track file that were not removed

[marmoute (Pierre-Yves David)]

marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  We need this set for the copy tracing algorithm. See documentation for details
  about this set.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9118

AFFECTED FILES
  mercurial/metadata.py

CHANGE DETAILS

diff --git a/mercurial/metadata.py b/mercurial/metadata.py
--- a/mercurial/metadata.py
+++ b/mercurial/metadata.py
@@ -31,6 +31,7 @@
 - added:   files actively added in the changeset.
 - merged:  files whose history got merged
 - removed: files removed in the revision
+- salvaged: files that might have been deleted by a merge but were not
 - touched: files affected by the merge
 
 and copies information is held by 2 mappings
@@ -47,6 +48,7 @@
 added=None,
 removed=None,
 merged=None,
+salvaged=None,
 p1_copies=None,
 p2_copies=None,
 ):
@@ -54,6 +56,7 @@
 self._merged = set(() if merged is None else merged)
 self._removed = set(() if removed is None else removed)
 self._touched = set(() if touched is None else touched)
+self._salvaged = set(() if touched is None else touched)
 self._touched.update(self._added)
 self._touched.update(self._merged)
 self._touched.update(self._removed)
@@ -65,6 +68,7 @@
 self.added == other.added
 and self.merged == other.merged
 and self.removed == other.removed
+and self.salvaged == other.salvaged
 and self.touched == other.touched
 and self.copied_from_p1 == other.copied_from_p1
 and self.copied_from_p2 == other.copied_from_p2
@@ -160,6 +164,27 @@
 self.mark_removed(f)
 
 @util.propertycache
+def salvaged(self):
+"""files that might had been deleted by a merge, but were not
+
+During a merge, the manifest merging might select some file for
+removal, or for a removed/changed conflict. If at commit time the file
+still exist in the final result. It the removal is "reverted" and the
+file is "salvaged"
+"""
+return frozenset(self._salvaged)
+
+def mark_salvaged(self, filename):
+if "salvaged" in vars(self):
+del self.salvaged
+self.mark_salvaged(filename)
+self._salvaged.add(filename)
+
+def update_salvaged(self, filenames):
+for f in filenames:
+self.mark_salvaged(f)
+
+@property
 def touched(self):
 """files either actively modified, added or removed"""
 return frozenset(self._touched)



To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: Improving the CI experience

[Raphaël Gomès]

On 9/28/20 8:12 PM, Augie Fackler wrote:



On Sep 28, 2020, at 10:43, Raphaël Gomès > wrote:


Hi all,

As you may know, we (Octobus) and other Mercurial contributors have 
been using
the Heptapod [1] CI to vet our patches pre-submission against 7 
variants of

Mercurial:
- Python 2 pure
- Python 3 pure
- Python 2 python+c
- Python 3 python+c
- Python 2 rust+c
- Python 3 rust+c
- Python 2 chg
- (also code checks and Rust unit tests, of course)

All variants above are run on x86_64 Debian machines. Our CI does not 
(yet) have

the MacOS, Windows, or FreeBSDoptionsthat the Mercurial Buildbot has, but
does test more variants on Linux.

Nevertheless, it catches a lot of the mistakes that we all make 
sometimes,
before we even send the patches upstream: regressions in Python 3, 
API breakage
with Rust, bad formatting, and other bugs of various nature. It does 
so way

before it touches the `committed` repo, which means that if I screw up my
patch, it only has to bother me.

The value of such a system is highly dependent on it being 
trustworthy: flaky
tests do exist, but they should be addressed and fixed in a timely 
manner and
my CI being red should mean that *I* missed something. Yet, I have 
heard the
phrase "I don't even look at the pipelines anymore, they're always 
broken for

something I haven't done".

Most of the time it's something minor: I've even been guilty of 
breaking the CI
myself in my last patch series (oops) due to bad formatting! However, 
these
issues should be few and far between: the developer experience 
suffers greatly
and we grow to accept that the tests are always broken, don't bother 
to run

them at all, etc.

# Proposal

I thought a start to fixing this situation would be for me to setup
an automation to send a small email to this mailing-list every time 
either the
`stable` or `default` branches (that track `hg-committed`) are 
broken, with a link

to the pipeline results.
This will not catch everything (only a subset of the possible targets 
is tested), but
I think this would be an improvement for all contributors, even those 
that don't

use Heptapod.

What do you all think?


I'm open to it. Do you think the CI that you're running on heptapod is 
faster than the buildbot? Should we keep the buildbot or shut it down?


Faster in terms of latency, almost certainly, at least for now because 
of the number of workers, and probably in terms of the actual speed of 
the runners, though I'm just making a guess.


However, the Buildbot still has value since it covers cases that our CI 
still does not cover and not everyone is using Heptapod, so let's keep 
it around for a while. To further drive the point home, I'm not 
comfortable recommending Heptapod for the project or anything of that 
sort yet, even ruling out personal preferences for workflow. Maybe we 
can have that discussion in a few months, who knows.


The fact that the Buildbot runs *after* the patches have landed makes me 
sad, but that's an issue for another day probably!



Raphaël

[1] https://foss.heptapod.net/octobus/mercurial-devel/-/pipelines
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org 


https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel



___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel