Re: [gentoo-portage-dev] [PATCH] _EbuildFetcherProcess: Handle SIGTERM
On 7/22/24 23:40, Joey Pabalinas wrote: On Mon, Jul 22, 2024 at 09:40:48PM GMT, Zac Medico wrote: Fix _EbuildFetcherProcess to handle SIGTERM, so that FETCHCOMMAND processes will not be left running in the background: * Convert the fetch function to an async_fetch coroutine function so that it can use asyncio.CancelledError handlers to terminate running processes. * Use multiprocessing.active_children() to detect and terminate any processes that asyncio.CancelledError handlers did not have an opportunity to terminate because the exception arrived too soon after fork/spawn. * Add unit test to verify that a child process is correctly killed when EbuildFetcher is cancelled, with short timeout in case it takes some time for the process to disappear. Bug: https://bugs.gentoo.org/936273 Closes: https://github.com/gentoo/portage/pull/1361 Signed-off-by: Zac Medico Looks good. Reviewed-by: Joey Pabalinas Thanks! There's now an update in https://github.com/gentoo/portage/pull/1361 to use multiple multiprocessing.active_children() loops in order to allow the children a little more time to terminate. -- Thanks, Zac OpenPGP_signature.asc Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH] _EbuildFetcherProcess: Handle SIGTERM
On Mon, Jul 22, 2024 at 09:40:48PM GMT, Zac Medico wrote: > Fix _EbuildFetcherProcess to handle SIGTERM, so that FETCHCOMMAND > processes will not be left running in the background: > > * Convert the fetch function to an async_fetch coroutine function > so that it can use asyncio.CancelledError handlers to terminate > running processes. > > * Use multiprocessing.active_children() to detect and terminate > any processes that asyncio.CancelledError handlers did not have > an opportunity to terminate because the exception arrived too > soon after fork/spawn. > > * Add unit test to verify that a child process is correctly > killed when EbuildFetcher is cancelled, with short timeout in > case it takes some time for the process to disappear. > > Bug: https://bugs.gentoo.org/936273 > Closes: https://github.com/gentoo/portage/pull/1361 > Signed-off-by: Zac Medico Looks good. Reviewed-by: Joey Pabalinas -- Cheers, Joey Pabalinas signature.asc Description: PGP signature
[gentoo-portage-dev] [PATCH] _EbuildFetcherProcess: Handle SIGTERM
Fix _EbuildFetcherProcess to handle SIGTERM, so that FETCHCOMMAND
processes will not be left running in the background:
* Convert the fetch function to an async_fetch coroutine function
so that it can use asyncio.CancelledError handlers to terminate
running processes.
* Use multiprocessing.active_children() to detect and terminate
any processes that asyncio.CancelledError handlers did not have
an opportunity to terminate because the exception arrived too
soon after fork/spawn.
* Add unit test to verify that a child process is correctly
killed when EbuildFetcher is cancelled, with short timeout in
case it takes some time for the process to disappear.
Bug: https://bugs.gentoo.org/936273
Closes: https://github.com/gentoo/portage/pull/1361
Signed-off-by: Zac Medico
---
lib/_emerge/EbuildFetcher.py | 58 +++---
lib/portage/package/ebuild/fetch.py| 97 ++--
lib/portage/tests/ebuild/test_fetch.py | 100 -
lib/portage/tests/util/test_socks5.py | 16 +++-
4 files changed, 245 insertions(+), 26 deletions(-)
diff --git a/lib/_emerge/EbuildFetcher.py b/lib/_emerge/EbuildFetcher.py
index 81d4b1054b..ece33dd630 100644
--- a/lib/_emerge/EbuildFetcher.py
+++ b/lib/_emerge/EbuildFetcher.py
@@ -4,6 +4,8 @@
import copy
import functools
import io
+import multiprocessing
+import signal
import sys
import portage
@@ -17,11 +19,12 @@ from portage.package.ebuild.fetch import (
_check_distfile,
_drop_privs_userfetch,
_want_userfetch,
-fetch,
+async_fetch,
)
from portage.util._async.AsyncTaskFuture import AsyncTaskFuture
from portage.util._async.ForkProcess import ForkProcess
from portage.util._pty import _create_pty_or_pipe
+from portage.util.futures import asyncio
from _emerge.CompositeTask import CompositeTask
@@ -34,6 +37,7 @@ class EbuildFetcher(CompositeTask):
"logfile",
"pkg",
"prefetch",
+"pre_exec",
"_fetcher_proc",
)
@@ -253,6 +257,7 @@ class _EbuildFetcherProcess(ForkProcess):
self._get_manifest(),
self._uri_map,
self.fetchonly,
+self.pre_exec,
)
ForkProcess._start(self)
@@ -263,7 +268,10 @@ class _EbuildFetcherProcess(ForkProcess):
self._settings = None
@staticmethod
-def _target(settings, manifest, uri_map, fetchonly):
+def _target(settings, manifest, uri_map, fetchonly, pre_exec):
+if pre_exec is not None:
+pre_exec()
+
# Force consistent color output, in case we are capturing fetch
# output through a normal pipe due to unavailability of ptys.
portage.output.havecolor = settings.get("NOCOLOR") not in ("yes",
"true")
@@ -273,17 +281,43 @@ class _EbuildFetcherProcess(ForkProcess):
if _want_userfetch(settings):
_drop_privs_userfetch(settings)
-rval = 1
allow_missing = manifest.allow_missing or "digest" in settings.features
-if fetch(
-uri_map,
-settings,
-fetchonly=fetchonly,
-digests=copy.deepcopy(manifest.getTypeDigests("DIST")),
-allow_missing_digests=allow_missing,
-):
-rval = os.EX_OK
-return rval
+
+async def main():
+loop = asyncio.get_event_loop()
+task = asyncio.ensure_future(
+async_fetch(
+uri_map,
+settings,
+fetchonly=fetchonly,
+digests=copy.deepcopy(manifest.getTypeDigests("DIST")),
+allow_missing_digests=allow_missing,
+)
+)
+
+def sigterm_handler(signum, _frame):
+loop.call_soon_threadsafe(task.cancel)
+signal.signal(signal.SIGTERM, signal.SIG_IGN)
+
+signal.signal(signal.SIGTERM, sigterm_handler)
+try:
+await task
+except asyncio.CancelledError:
+# If asyncio.CancelledError arrives too soon after fork/spawn
+# then handers will not have an opportunity to termintate
+# the corresponding process, so clean up after this race.
+for proc in multiprocessing.active_children():
+proc.terminate()
+proc.join(0)
+if proc.exitcode is None:
+proc.kill()
+proc.join(0)
+signal.signal(signal.SIGTERM, signal.SIG_DFL)
+os.kill(os.getpid(), signal.SIGTERM)
+
+return os.EX_OK if task.result() else 1
+
+return asyncio.run(main())
def _get_ebuild_path(self):
if self.ebuild_path is not None:
diff --git a/lib/portage/package/ebuild/fetch.py
b/lib/portage/package/ebuild/fetch.py
index ed40cf6ede..124b2b7056 100644
--- a/lib/portage/package/ebuild/fetch.py
+++
Re: [gentoo-portage-dev] [PATCH 0/2] Minor documentation fixups
Bryan Gardiner writes: > Just fixes for a couple of small documentation issues I found. Applied - thanks! > > Bryan Gardiner (2): > depgraph.py: Fix the URL for the changed-deps wiki page > ebuild.5: Escape carats which are special to groff > > lib/_emerge/depgraph.py | 2 +- > man/ebuild.5| 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-)
[gentoo-portage-dev] [PATCH 1/2] depgraph.py: Fix the URL for the changed-deps wiki page
Signed-off-by: Bryan Gardiner --- lib/_emerge/depgraph.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py index 6853ec491..9673d85f8 100644 --- a/lib/_emerge/depgraph.py +++ b/lib/_emerge/depgraph.py @@ -1181,7 +1181,7 @@ class depgraph: "NOTE: Refer to the following page for more information about dependency", " change(s) without revision bump:", "", -" https://wiki.gentoo.org/wiki/Project:Portage/Changed_Deps;, +" https://wiki.gentoo.org/wiki/Project:Portage/Changed_dependencies;, "", " In order to suppress reports about dependency changes, add", " --changed-deps-report=n to the EMERGE_DEFAULT_OPTS variable in", -- 2.43.2
[gentoo-portage-dev] [PATCH 2/2] ebuild.5: Escape carats which are special to groff
Raw carats are converted to circumflex characters which are not searchable, so they need to be escaped. See groff_char(7). Signed-off-by: Bryan Gardiner --- man/ebuild.5 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/man/ebuild.5 b/man/ebuild.5 index a32ba4828..36d65eb18 100644 --- a/man/ebuild.5 +++ b/man/ebuild.5 @@ -734,7 +734,7 @@ If flag1 enabled then flag2 enabled flag1? ( flag2 ) If flag1 disabled then flag2 enabled !flag1? ( flag2 ) If flag1 disabled then flag2 disabled !flag1? ( !flag2 ) Must enable any one or more (inclusive or) || ( flag1 flag2 flag3 ) -Must enable exactly one but not more (exclusive or)^^ ( flag1 flag2 flag3 ) +Must enable exactly one but not more (exclusive or)\(ha\(ha ( flag1 flag2 flag3 ) May enable at most one (EAPI 5 or later) ?? ( flag1 flag2 flag3 ) .TE .TP -- 2.43.2
[gentoo-portage-dev] [PATCH 0/2] Minor documentation fixups
Just fixes for a couple of small documentation issues I found. Bryan Gardiner (2): depgraph.py: Fix the URL for the changed-deps wiki page ebuild.5: Escape carats which are special to groff lib/_emerge/depgraph.py | 2 +- man/ebuild.5| 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) -- 2.43.2
[gentoo-portage-dev] [PATCH 1/2] Add get_repo_revision_history function and repo_revisions file
The history of synced revisions is provided by a new
get_repo_revision_history function and corresponding
/var/lib/portage/repo_revisions file, with history
limit currently capped at 25 revisions. If a change
is detected and the current process has permission
to update the repo_revisions file, then the file will
be updated with any newly detected revisions.
For volatile repos the revisions may be unordered,
which makes them unusable for the purposes of the
revision history, so the revisions of volatile repos
are not tracked. This functions detects revisions
which are not yet visible to the current process due
to the sync-rcu option.
The emaint revisions --purgerepos and --purgeallrepos
options allow revisions for some or all repos to be
easily purged from the history. For example, the
emerge-webrsync script uses this emaint commmand to
purge the revision history of the gentoo repo when
the emerge-webrsync --revert option is used to roll
back to a previous snapshot:
emaint revisions --purgerepos="${repo_name}"
Bug: https://bugs.gentoo.org/924772
Signed-off-by: Zac Medico
---
bin/emerge-webrsync | 3 +-
lib/portage/const.py | 1 +
lib/portage/emaint/modules/meson.build| 1 +
.../emaint/modules/revisions/__init__.py | 36 +
.../emaint/modules/revisions/meson.build | 8 ++
.../emaint/modules/revisions/revisions.py | 95 +
lib/portage/sync/controller.py| 8 +-
lib/portage/sync/meson.build | 1 +
lib/portage/sync/revision_history.py | 133 ++
lib/portage/tests/sync/test_sync_local.py | 75 +-
man/emaint.1 | 18 ++-
man/portage.5 | 17 ++-
12 files changed, 388 insertions(+), 8 deletions(-)
create mode 100644 lib/portage/emaint/modules/revisions/__init__.py
create mode 100644 lib/portage/emaint/modules/revisions/meson.build
create mode 100644 lib/portage/emaint/modules/revisions/revisions.py
create mode 100644 lib/portage/sync/revision_history.py
diff --git a/bin/emerge-webrsync b/bin/emerge-webrsync
index 99da05543a..caa4986da2 100755
--- a/bin/emerge-webrsync
+++ b/bin/emerge-webrsync
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
-# Copyright 1999-2023 Gentoo Authors
+# Copyright 1999-2024 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
# Author: Karl Trygve Kalleberg
# Rewritten from the old, Perl-based emerge-webrsync script
@@ -732,6 +732,7 @@ main() {
[[ ${do_debug} -eq 1 ]] && set -x
if [[ -n ${revert_date} ]] ; then
+ emaint revisions --purgerepos="${repo_name}"
do_snapshot 1 "${revert_date}"
else
do_latest_snapshot
diff --git a/lib/portage/const.py b/lib/portage/const.py
index 2154213b7b..c9a71009a7 100644
--- a/lib/portage/const.py
+++ b/lib/portage/const.py
@@ -51,6 +51,7 @@ PRIVATE_PATH = "var/lib/portage"
WORLD_FILE = f"{PRIVATE_PATH}/world"
WORLD_SETS_FILE = f"{PRIVATE_PATH}/world_sets"
CONFIG_MEMORY_FILE = f"{PRIVATE_PATH}/config"
+REPO_REVISIONS = f"{PRIVATE_PATH}/repo_revisions"
NEWS_LIB_PATH = "var/lib/gentoo"
# these variables get EPREFIX prepended automagically when they are
diff --git a/lib/portage/emaint/modules/meson.build
b/lib/portage/emaint/modules/meson.build
index 48f4f77d83..33b396be94 100644
--- a/lib/portage/emaint/modules/meson.build
+++ b/lib/portage/emaint/modules/meson.build
@@ -12,5 +12,6 @@ subdir('logs')
subdir('merges')
subdir('move')
subdir('resume')
+subdir('revisions')
subdir('sync')
subdir('world')
diff --git a/lib/portage/emaint/modules/revisions/__init__.py
b/lib/portage/emaint/modules/revisions/__init__.py
new file mode 100644
index 00..c51cbb4bf3
--- /dev/null
+++ b/lib/portage/emaint/modules/revisions/__init__.py
@@ -0,0 +1,36 @@
+# Copyright 2024 Gentoo Authors
+# Distributed under the terms of the GNU General Public License v2
+
+doc = """Purge repo_revisions history file."""
+__doc__ = doc
+
+
+module_spec = {
+"name": "revisions",
+"description": doc,
+"provides": {
+"purgerevisions": {
+"name": "revisions",
+"sourcefile": "revisions",
+"class": "PurgeRevisions",
+"description": "Purge repo_revisions history",
+"functions": ["purgeallrepos", "purgerepos"],
+"func_desc": {
+"repo": {
+"long": "--purgerepos",
+"help": "(revisions module only): --purgerepos Purge
revisions for the specified repo(s)",
+"status": "Purging %s",
+"action": "store",
+"func": "purgerepos",
+},
+"allrepos": {
+"long": "--purgeallrepos",
+"help": "(revisions module only): --purgeallrepos Purge
revisions for all repos",
+
[gentoo-portage-dev] [PATCH 2/2] bintree: Add REPO_REVISIONS to package index header
As a means for binhost clients to select source repo
revisions which are consistent with binhosts, inject
REPO_REVISIONS from a package into the index header,
using a history of synced revisions to guarantee
forward progress. This queries the relevant repos to
check if any new revisions have appeared in the
absence of a proper sync operation.
Bug: https://bugs.gentoo.org/924772
Signed-off-by: Zac Medico
---
lib/portage/dbapi/bintree.py | 66 -
lib/portage/tests/sync/test_sync_local.py | 71 +++
2 files changed, 123 insertions(+), 14 deletions(-)
diff --git a/lib/portage/dbapi/bintree.py b/lib/portage/dbapi/bintree.py
index 7bc1f60f6d..fbf60e74eb 100644
--- a/lib/portage/dbapi/bintree.py
+++ b/lib/portage/dbapi/bintree.py
@@ -48,6 +48,7 @@ from portage.exception import (
from portage.localization import _
from portage.output import colorize
from portage.package.ebuild.profile_iuse import iter_iuse_vars
+from portage.sync.revision_history import get_repo_revision_history
from portage.util import ensure_dirs
from portage.util.file_copy import copyfile
from portage.util.futures import asyncio
@@ -62,6 +63,7 @@ from portage import _unicode_encode
import codecs
import errno
import io
+import json
import re
import stat
import subprocess
@@ -134,13 +136,19 @@ class bindbapi(fakedbapi):
"USE",
"_mtime_",
}
+# Keys required only when initially adding a package.
+self._init_aux_keys = {
+"REPO_REVISIONS",
+}
self._aux_cache = {}
self._aux_cache_slot_dict_cache = None
@property
def _aux_cache_slot_dict(self):
if self._aux_cache_slot_dict_cache is None:
-self._aux_cache_slot_dict_cache =
slot_dict_class(self._aux_cache_keys)
+self._aux_cache_slot_dict_cache = slot_dict_class(
+chain(self._aux_cache_keys, self._init_aux_keys)
+)
return self._aux_cache_slot_dict_cache
def __getstate__(self):
@@ -1791,6 +1799,10 @@ class binarytree:
pkgindex = self._new_pkgindex()
d = self._inject_file(pkgindex, cpv, full_path)
+repo_revisions = d.get("REPO_REVISIONS")
+if repo_revisions:
+repo_revisions = json.loads(repo_revisions)
+self._inject_repo_revisions(pkgindex.header, repo_revisions)
self._update_pkgindex_header(pkgindex.header)
self._pkgindex_write(pkgindex)
@@ -1872,7 +1884,7 @@ class binarytree:
@return: package metadata
"""
if keys is None:
-keys = self.dbapi._aux_cache_keys
+keys = chain(self.dbapi._aux_cache_keys, self.dbapi._init_aux_keys)
metadata = self.dbapi._aux_cache_slot_dict()
else:
metadata = {}
@@ -1916,6 +1928,56 @@ class binarytree:
return metadata
+def _inject_repo_revisions(self, header, repo_revisions):
+"""
+Inject REPO_REVISIONS from a package into the index header,
+using a history of synced revisions to guarantee forward
+progress. This queries the relevant repos to check if any
+new revisions have appeared in the absence of a proper sync
+operation.
+
+This does not expose REPO_REVISIONS that do not appear in
+the sync history, since such revisions suggest that the
+package was not built locally, and in this case its
+REPO_REVISIONS are not intended to be exposed.
+"""
+synced_repo_revisions = get_repo_revision_history(
+self.settings["EROOT"],
+[self.settings.repositories[repo_name] for repo_name in
repo_revisions],
+)
+header_repo_revisions = (
+json.loads(header["REPO_REVISIONS"]) if
header.get("REPO_REVISIONS") else {}
+)
+for repo_name, repo_revision in repo_revisions.items():
+rev_list = synced_repo_revisions.get(repo_name, [])
+header_rev = header_repo_revisions.get(repo_name)
+if not rev_list or header_rev in (repo_revision, rev_list[0]):
+continue
+try:
+header_rev_index = (
+None if header_rev is None else rev_list.index(header_rev)
+)
+except ValueError:
+header_rev_index = None
+try:
+repo_revision_index = rev_list.index(repo_revision)
+except ValueError:
+repo_revision_index = None
+if repo_revision_index is not None and (
+header_rev_index is None or repo_revision_index <
header_rev_index
+):
+# There is forward progress when repo_revision is more recent
+# than header_rev or header_rev was not found in the history.
+# Do not expose repo_revision here if it does not appear in
+
[gentoo-portage-dev] [PATCH 0/2] Add REPO_REVISIONS to package index header
As a means for binhost clients to select source repo
revisions which are consistent with binhosts, inject
REPO_REVISIONS from a package into the index header,
using a history of synced revisions to guarantee
forward progress. This queries the relevant repos to
check if any new revisions have appeared in the
absence of a proper sync operation.
The history of synced revisions is provided by a new
get_repo_revision_history function and corresponding
/var/lib/portage/repo_revisions file, with history
limit currently capped at 25 revisions. If a change
is detected and the current process has permission
to update the repo_revisions file, then the file will
be updated with any newly detected revisions.
For volatile repos the revisions may be unordered,
which makes them unusable for the purposes of the
revision history, so the revisions of volatile repos
are not tracked. This functions detects revisions
which are not yet visible to the current process due
to the sync-rcu option.
The emaint revisions --purgerepos and --purgeallrepos
options allow revisions for some or all repos to be
easily purged from the history. For example, the
emerge-webrsync script uses this emaint commmand to
purge the revision history of the gentoo repo when
the emerge-webrsync --revert option is used to roll
back to a previous snapshot:
emaint revisions --purgerepos="${repo_name}"
This series can also be reviewed at https://github.com/gentoo/portage/pull/1306.
Bug: https://bugs.gentoo.org/924772
Signed-off-by: Zac Medico
Zac Medico (2):
Add get_repo_revision_history function and repo_revisions file
bintree: Add REPO_REVISIONS to package index header
bin/emerge-webrsync | 3 +-
lib/portage/const.py | 1 +
lib/portage/dbapi/bintree.py | 66 +++-
lib/portage/emaint/modules/meson.build| 1 +
.../emaint/modules/revisions/__init__.py | 36 +
.../emaint/modules/revisions/meson.build | 8 +
.../emaint/modules/revisions/revisions.py | 95
lib/portage/sync/controller.py| 8 +-
lib/portage/sync/meson.build | 1 +
lib/portage/sync/revision_history.py | 133
lib/portage/tests/sync/test_sync_local.py | 146 --
man/emaint.1 | 18 ++-
man/portage.5 | 17 +-
13 files changed, 511 insertions(+), 22 deletions(-)
create mode 100644 lib/portage/emaint/modules/revisions/__init__.py
create mode 100644 lib/portage/emaint/modules/revisions/meson.build
create mode 100644 lib/portage/emaint/modules/revisions/revisions.py
create mode 100644 lib/portage/sync/revision_history.py
--
2.41.0
[gentoo-portage-dev] [PATCH 9/9] EbuildPhase: async_check_locale
Change config.environ() check_locale calls to async_check_locale
calls in the EbuildPhase _async_start method in order to eliminate
synchronous waiting for child processes in the main event loop
thread.
Bug: https://bugs.gentoo.org/923841
Signed-off-by: Zac Medico
---
lib/_emerge/EbuildMetadataPhase.py| 4 +++
lib/_emerge/EbuildPhase.py| 28 ++-
lib/portage/package/ebuild/config.py | 26 -
lib/portage/util/futures/_asyncio/__init__.py | 9 ++
lib/portage/util/locale.py| 28 +--
5 files changed, 70 insertions(+), 25 deletions(-)
diff --git a/lib/_emerge/EbuildMetadataPhase.py
b/lib/_emerge/EbuildMetadataPhase.py
index 9fcdabe840..90a3ea05aa 100644
--- a/lib/_emerge/EbuildMetadataPhase.py
+++ b/lib/_emerge/EbuildMetadataPhase.py
@@ -8,6 +8,7 @@ import portage
portage.proxy.lazyimport.lazyimport(
globals(),
+"_emerge.EbuildPhase:_setup_locale",
"portage.package.ebuild._metadata_invalid:eapi_invalid",
)
from portage import os
@@ -83,6 +84,9 @@ class EbuildMetadataPhase(SubProcess):
settings.setcpv(self.cpv)
settings.configdict["pkg"]["EAPI"] = parsed_eapi
+# This requires above setcpv and EAPI setup.
+await _setup_locale(self.settings)
+
debug = settings.get("PORTAGE_DEBUG") == "1"
master_fd = None
slave_fd = None
diff --git a/lib/_emerge/EbuildPhase.py b/lib/_emerge/EbuildPhase.py
index c81bf54a81..c8caf73722 100644
--- a/lib/_emerge/EbuildPhase.py
+++ b/lib/_emerge/EbuildPhase.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2021 Gentoo Authors
+# Copyright 1999-2024 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
import functools
@@ -24,6 +24,7 @@ from portage.package.ebuild.prepare_build_dirs import (
_prepare_fake_distdir,
_prepare_fake_filesdir,
)
+from portage.eapi import _get_eapi_attrs
from portage.util import writemsg, ensure_dirs
from portage.util._async.AsyncTaskFuture import AsyncTaskFuture
from portage.util._async.BuildLogger import BuildLogger
@@ -54,12 +55,34 @@ portage.proxy.lazyimport.lazyimport(
+ "_post_src_install_write_metadata,"
+ "_preinst_bsdflags",
"portage.util.futures.unix_events:_set_nonblocking",
+"portage.util.locale:async_check_locale,split_LC_ALL",
)
from portage import os
from portage import _encodings
from portage import _unicode_encode
+async def _setup_locale(settings):
+eapi_attrs = _get_eapi_attrs(settings["EAPI"])
+if eapi_attrs.posixish_locale:
+split_LC_ALL(settings)
+settings["LC_COLLATE"] = "C"
+# check_locale() returns None when check can not be executed.
+if await async_check_locale(silent=True, env=settings.environ()) is
False:
+# try another locale
+for l in ("C.UTF-8", "en_US.UTF-8", "en_GB.UTF-8", "C"):
+settings["LC_CTYPE"] = l
+if await async_check_locale(silent=True,
env=settings.environ()):
+# TODO: output the following only once
+# writemsg(
+# _("!!! LC_CTYPE unsupported, using %s instead\n")
+# % self.settings["LC_CTYPE"]
+# )
+break
+else:
+raise AssertionError("C locale did not pass the test!")
+
+
class EbuildPhase(CompositeTask):
__slots__ = ("actionmap", "fd_pipes", "phase", "settings") +
("_ebuild_lock",)
@@ -94,6 +117,9 @@ class EbuildPhase(CompositeTask):
self._start_task(AsyncTaskFuture(future=future),
self._async_start_exit)
async def _async_start(self):
+
+await _setup_locale(self.settings)
+
need_builddir = self.phase not in
EbuildProcess._phases_without_builddir
if need_builddir:
diff --git a/lib/portage/package/ebuild/config.py
b/lib/portage/package/ebuild/config.py
index d7b0ca5676..35c77486ec 100644
--- a/lib/portage/package/ebuild/config.py
+++ b/lib/portage/package/ebuild/config.py
@@ -29,7 +29,6 @@ portage.proxy.lazyimport.lazyimport(
"portage.dbapi.vartree:vartree",
"portage.package.ebuild.doebuild:_phase_func_map",
"portage.util.compression_probe:_compressors",
-"portage.util.locale:check_locale,split_LC_ALL",
)
from portage import bsd_chflags, load_mod, os, selinux, _unicode_decode
from portage.const import (
@@ -3368,20 +3367,17 @@ class config:
mydict["EBUILD_PHASE_FUNC"] = phase_func
if eapi_attrs.posixish_locale:
-split_LC_ALL(mydict)
-mydict["LC_COLLATE"] = "C"
-# check_locale() returns None when check can not be executed.
-if check_locale(silent=True, env=mydict) is False:
-# try another locale
-for l in ("C.UTF-8", "en_US.UTF-8", "en_GB.UTF-8", "C"):
-mydict["LC_CTYPE"] = l
-if
[gentoo-portage-dev] [PATCH 8/9] EbuildMetadataPhase: Migrate to _async_start
Bug: https://bugs.gentoo.org/923841 Signed-off-by: Zac Medico --- lib/_emerge/EbuildMetadataPhase.py | 17 ++--- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/lib/_emerge/EbuildMetadataPhase.py b/lib/_emerge/EbuildMetadataPhase.py index 784712e8cb..9fcdabe840 100644 --- a/lib/_emerge/EbuildMetadataPhase.py +++ b/lib/_emerge/EbuildMetadataPhase.py @@ -46,6 +46,12 @@ class EbuildMetadataPhase(SubProcess): _files_dict = slot_dict_class(_file_names, prefix="") def _start(self): +asyncio.ensure_future( +self._async_start(), loop=self.scheduler +).add_done_callback(self._async_start_done) +self._registered = True + +async def _async_start(self): ebuild_path = self.ebuild_hash.location with open( @@ -116,7 +122,6 @@ class EbuildMetadataPhase(SubProcess): self._raw_metadata = [] files.ebuild = master_fd self.scheduler.add_reader(files.ebuild, self._output_handler) -self._registered = True retval = portage.doebuild( ebuild_path, @@ -150,16 +155,6 @@ class EbuildMetadataPhase(SubProcess): self._proc = retval -asyncio.ensure_future( -self._async_start(), loop=self.scheduler -).add_done_callback(self._async_start_done) - -async def _async_start(self): -# Call async check_locale here for bug 923841, but code -# also needs to migrate from _start to here, including -# the self.deallocate_config set_result call. -pass - def _async_start_done(self, future): future.cancelled() or future.result() if self._was_cancelled(): -- 2.41.0
[gentoo-portage-dev] [PATCH 7/9] actions: disable pytest-xdist for spawn start-method (workers crash)
Bug: https://bugs.gentoo.org/924416
Signed-off-by: Zac Medico
---
.github/workflows/ci.yml | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 5bffd97206..762999b7cc 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -90,5 +90,8 @@ jobs:
- name: Run tests for ${{ matrix.python-version }}
run: |
[[ "${{ matrix.start-method }}" == "spawn" ]] && export
PORTAGE_MULTIPROCESSING_START_METHOD=spawn
- export PYTEST_ADDOPTS="-vv -ra -l -o console_output_style=count -n
$(nproc) --dist=worksteal"
+ # spawn start-method crashes pytest-xdist workers (bug 924416)
+ [[ "${{ matrix.start-method }}" == "spawn" ]] && \
+export PYTEST_ADDOPTS="-vv -ra -l -o console_output_style=count"
|| \
+export PYTEST_ADDOPTS="-vv -ra -l -o console_output_style=count -n
$(nproc) --dist=worksteal"
meson test -C /tmp/build --verbose
--
2.41.0
[gentoo-portage-dev] [PATCH 6/9] async_aux_get: Use EbuildMetadataPhase deallocate_config future
For the portdbapi async_aux_get method, there is not a very
good place to store a config pool, so instead use asyncio.Lock
to manage access to the portdbapi doebuild_settings attribute
when using the main event loop in the main thread. For other
threads, clone a config instance since we do not have a
thread-safe config pool. This cloning is expensive, but since
portage internals do not trigger this case, it suffices for now
(an AssertionError ensures that internals do not trigger it).
For the main event loop running in the main thread, performance
with the asyncio.Lock should not be significantly different to
performance prior to commit c95fc64abf96, since check_locale
results are typically cached and before there was only a single
shared doebuild_settings instance with access serialized via
the EbuildMetadataPhase _start method.
Update async_aux_get callers to use asyncio.ensure_future on
the returned coroutine when needed, since it used to return
a future instead of a coroutine, and sometimes a future is
needed for add_done_callback usage.
In the portdbapi async_fetch_map method, fix a broken reference
to "future" which should have been "aux_get_future", an error
discovered while testing this patch.
Bug: https://bugs.gentoo.org/924319
Fixes: c95fc64abf96 ("EbuildPhase: async_check_locale")
Signed-off-by: Zac Medico
---
lib/portage/_emirrordist/FetchIterator.py | 10 +-
lib/portage/dbapi/porttree.py | 129 +++---
lib/portage/tests/update/test_move_ent.py | 3 +
3 files changed, 97 insertions(+), 45 deletions(-)
diff --git a/lib/portage/_emirrordist/FetchIterator.py
b/lib/portage/_emirrordist/FetchIterator.py
index eaf3e53596..e4fdd092af 100644
--- a/lib/portage/_emirrordist/FetchIterator.py
+++ b/lib/portage/_emirrordist/FetchIterator.py
@@ -1,4 +1,4 @@
-# Copyright 2013-2018 Gentoo Foundation
+# Copyright 2013-2024 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
import threading
@@ -14,6 +14,7 @@ from portage.exception import PortageException,
PortageKeyError
from portage.package.ebuild.fetch import DistfileName
from portage.util._async.AsyncTaskFuture import AsyncTaskFuture
from portage.util._async.TaskScheduler import TaskScheduler
+from portage.util.futures import asyncio
from portage.util.futures.iter_completed import iter_gather
from .FetchTask import FetchTask
from _emerge.CompositeTask import CompositeTask
@@ -276,8 +277,11 @@ def _async_fetch_tasks(config, hash_filter, repo_config,
digests_future, cpv, lo
result.set_result(fetch_tasks)
def future_generator():
-yield config.portdb.async_aux_get(
-cpv, ("RESTRICT",), myrepo=repo_config.name, loop=loop
+yield asyncio.ensure_future(
+config.portdb.async_aux_get(
+cpv, ("RESTRICT",), myrepo=repo_config.name, loop=loop
+),
+loop,
)
yield config.portdb.async_fetch_map(cpv, mytree=repo_config.location,
loop=loop)
diff --git a/lib/portage/dbapi/porttree.py b/lib/portage/dbapi/porttree.py
index 61d431f917..4eebe1183f 100644
--- a/lib/portage/dbapi/porttree.py
+++ b/lib/portage/dbapi/porttree.py
@@ -1,4 +1,4 @@
-# Copyright 1998-2021 Gentoo Authors
+# Copyright 1998-2024 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
__all__ = ["close_portdbapi_caches", "FetchlistDict", "portagetree",
"portdbapi"]
@@ -41,7 +41,9 @@ from portage.util.futures import asyncio
from portage.util.futures.iter_completed import iter_gather
from _emerge.EbuildMetadataPhase import EbuildMetadataPhase
+import contextlib
import os as _os
+import threading
import traceback
import warnings
import errno
@@ -239,6 +241,7 @@ class portdbapi(dbapi):
# this purpose because doebuild makes many changes to the config
# instance that is passed in.
self.doebuild_settings = config(clone=self.settings)
+self._doebuild_settings_lock = asyncio.Lock()
self.depcachedir = os.path.realpath(self.settings.depcachedir)
if os.environ.get("SANDBOX_ON") == "1":
@@ -356,6 +359,17 @@ class portdbapi(dbapi):
self._better_cache = None
self._broken_ebuilds = set()
+def __getstate__(self):
+state = self.__dict__.copy()
+# These attributes are not picklable, so they are automatically
+# regenerated after unpickling.
+state["_doebuild_settings_lock"] = None
+return state
+
+def __setstate__(self, state):
+self.__dict__.update(state)
+self._doebuild_settings_lock = asyncio.Lock()
+
def _set_porttrees(self, porttrees):
"""
Consumers, such as emirrordist, may modify the porttrees attribute in
@@ -669,7 +683,7 @@ class portdbapi(dbapi):
self.async_aux_get(mycpv, mylist, mytree=mytree, myrepo=myrepo,
loop=loop)
)
-def async_aux_get(self, mycpv, mylist, mytree=None, myrepo=None,
loop=None):
+
[gentoo-portage-dev] [PATCH 5/9] asyncio: Wrap asyncio.Lock for python 3.9 compat
Wrap asyncio.Lock for compatibility with python 3.9 where the
deprecated loop parameter is required in order to avoid "got
Future attached to a different loop" errors.
The pordbapi async_aux_get method can use asyncio.Lock to
serialize access to its doebuild_settings attribute in order
to prevent issues like bug 924319.
Bug: https://bugs.gentoo.org/924319
Signed-off-by: Zac Medico
---
lib/portage/util/futures/_asyncio/__init__.py | 17 +
1 file changed, 17 insertions(+)
diff --git a/lib/portage/util/futures/_asyncio/__init__.py
b/lib/portage/util/futures/_asyncio/__init__.py
index 8f1b8e8275..b6481c281e 100644
--- a/lib/portage/util/futures/_asyncio/__init__.py
+++ b/lib/portage/util/futures/_asyncio/__init__.py
@@ -9,6 +9,7 @@ __all__ = (
"CancelledError",
"Future",
"InvalidStateError",
+"Lock",
"TimeoutError",
"get_child_watcher",
"get_event_loop",
@@ -22,6 +23,7 @@ __all__ = (
"wait_for",
)
+import sys
import types
import weakref
@@ -35,6 +37,7 @@ from asyncio import (
FIRST_EXCEPTION,
Future,
InvalidStateError,
+Lock as _Lock,
shield,
TimeoutError,
wait_for,
@@ -159,6 +162,20 @@ def iscoroutinefunction(func):
return False
+class Lock(_Lock):
+"""
+Inject loop parameter for python3.9 or less in order to avoid
+"got Future attached to a different loop" errors.
+"""
+
+def __init__(self, **kwargs):
+if sys.version_info >= (3, 10):
+kwargs.pop("loop", None)
+elif "loop" not in kwargs:
+kwargs["loop"] = _safe_loop()._loop
+super().__init__(**kwargs)
+
+
class Task(Future):
"""
Schedule the execution of a coroutine: wrap it in a future. A task
--
2.41.0
[gentoo-portage-dev] [PATCH 4/9] ResolverPlayground: Use egencache to create manifests
Make the ResolverPlayground _create_ebuild_manifests method
call egencache --jobs, which reliably triggers the KeyError
from bug 924319 for multiple tests:
lib/portage/tests/bin/test_doins.py::DoIns::testDoInsFallback Exception in
callback EbuildMetadataPhase._async_start_done()
handle: )>
Traceback (most recent call last):
File "/usr/lib/python3.12/asyncio/events.py", line 88, in _run
self._context.run(self._callback, *self._args)
File "lib/_emerge/EbuildMetadataPhase.py", line 154, in _async_start_done
future.cancelled() or future.result()
^^^
File "lib/_emerge/EbuildMetadataPhase.py", line 130, in _async_start
retval = portage.doebuild(
^
File "lib/portage/package/ebuild/doebuild.py", line 1030, in doebuild
doebuild_environment(
File "lib/portage/package/ebuild/doebuild.py", line 519, in
doebuild_environment
eapi = mysettings.configdict["pkg"]["EAPI"]
File "lib/portage/util/__init__.py", line 1684, in __getitem__
return UserDict.__getitem__(self, item_key)
File "lib/portage/cache/mappings.py", line 175, in __getitem__
return self.data[key]
~^
KeyError: 'EAPI'
Bug: https://bugs.gentoo.org/924319
Signed-off-by: Zac Medico
---
lib/portage/tests/dbapi/test_portdb_cache.py | 4 +-
.../tests/resolver/ResolverPlayground.py | 38 ++-
2 files changed, 23 insertions(+), 19 deletions(-)
diff --git a/lib/portage/tests/dbapi/test_portdb_cache.py
b/lib/portage/tests/dbapi/test_portdb_cache.py
index c7c6913b49..c24a4f2098 100644
--- a/lib/portage/tests/dbapi/test_portdb_cache.py
+++ b/lib/portage/tests/dbapi/test_portdb_cache.py
@@ -1,6 +1,7 @@
-# Copyright 2012-2023 Gentoo Authors
+# Copyright 2012-2024 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
+import shutil
import subprocess
import sys
import textwrap
@@ -63,6 +64,7 @@ class PortdbCacheTestCase(TestCase):
python_cmd = (portage_python, "-b", "-Wd", "-c")
test_commands = (
+(lambda: shutil.rmtree(md5_cache_dir) or True,),
(lambda: not os.path.exists(pms_cache_dir),),
(lambda: not os.path.exists(md5_cache_dir),),
python_cmd
diff --git a/lib/portage/tests/resolver/ResolverPlayground.py
b/lib/portage/tests/resolver/ResolverPlayground.py
index 2d26012873..75c86b615c 100644
--- a/lib/portage/tests/resolver/ResolverPlayground.py
+++ b/lib/portage/tests/resolver/ResolverPlayground.py
@@ -3,6 +3,7 @@
import bz2
import fnmatch
+import subprocess
import tempfile
import portage
@@ -18,8 +19,6 @@ from portage.const import (
from portage.process import find_binary
from portage.dep import Atom, _repo_separator
from portage.dbapi.bintree import binarytree
-from portage.package.ebuild.config import config
-from portage.package.ebuild.digestgen import digestgen
from portage._sets import load_default_config
from portage._sets.base import InternalPackageSet
from portage.tests import cnf_path
@@ -323,22 +322,25 @@ class ResolverPlayground:
f.write(misc_content)
def _create_ebuild_manifests(self, ebuilds):
-tmpsettings = config(clone=self.settings)
-tmpsettings["PORTAGE_QUIET"] = "1"
-for cpv in ebuilds:
-a = Atom("=" + cpv, allow_repo=True)
-repo = a.repo
-if repo is None:
-repo = "test_repo"
-
-repo_dir = self._get_repo_dir(repo)
-ebuild_dir = os.path.join(repo_dir, a.cp)
-ebuild_path = os.path.join(ebuild_dir, a.cpv.split("/")[1] +
".ebuild")
-
-portdb = self.trees[self.eroot]["porttree"].dbapi
-tmpsettings["O"] = ebuild_dir
-if not digestgen(mysettings=tmpsettings, myportdb=portdb):
-raise AssertionError(f"digest creation failed for
{ebuild_path}")
+for repo_name in self._repositories:
+if repo_name == "DEFAULT":
+continue
+egencache_cmd = [
+"egencache",
+f"--repo={repo_name}",
+"--update",
+"--update-manifests",
+"--sign-manifests=n",
+"--strict-manifests=n",
+
f"--repositories-configuration={self.settings['PORTAGE_REPOSITORIES']}",
+f"--jobs={portage.util.cpuinfo.get_cpu_count()}",
+]
+result = subprocess.run(
+egencache_cmd,
+env=self.settings.environ(),
+)
+if result.returncode != os.EX_OK:
+raise AssertionError(f"command failed: {egencache_cmd}")
def _create_binpkgs(self, binpkgs):
# When using BUILD_ID, there can be multiple instances for the
--
2.41.0
[gentoo-portage-dev] [PATCH 3/9] MetadataRegen: Use EbuildMetadataPhase deallocate_config
For EbuildMetadataPhase consumers like MetadataRegen and
depgraph, store a pool of config instances in a config_pool
list, and return instaces to the list when the deallocate_config
future is done.
Bug: https://bugs.gentoo.org/924319
Fixes: c95fc64abf96 ("EbuildPhase: async_check_locale")
Signed-off-by: Zac Medico
---
lib/_emerge/MetadataRegen.py | 16 ++--
lib/_emerge/depgraph.py | 11 +++
2 files changed, 25 insertions(+), 2 deletions(-)
diff --git a/lib/_emerge/MetadataRegen.py b/lib/_emerge/MetadataRegen.py
index d29722b94c..538a94b450 100644
--- a/lib/_emerge/MetadataRegen.py
+++ b/lib/_emerge/MetadataRegen.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2020 Gentoo Authors
+# Copyright 1999-2024 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
from _emerge.EbuildMetadataPhase import EbuildMetadataPhase
@@ -44,6 +44,7 @@ class MetadataRegen(AsyncScheduler):
valid_pkgs = self._valid_pkgs
cp_set = self._cp_set
consumer = self._consumer
+config_pool = []
portage.writemsg_stdout("Regenerating cache entries...\n")
for cp in self._cp_iter:
@@ -73,12 +74,23 @@ class MetadataRegen(AsyncScheduler):
consumer(cpv, repo_path, metadata, ebuild_hash,
True)
continue
+if config_pool:
+settings = config_pool.pop()
+else:
+settings = portage.config(clone=portdb.settings)
+
+deallocate_config = self.scheduler.create_future()
+deallocate_config.add_done_callback(
+lambda future: config_pool.append(future.result())
+)
+
yield EbuildMetadataPhase(
cpv=cpv,
ebuild_hash=ebuild_hash,
portdb=portdb,
repo_path=repo_path,
-settings=portdb.doebuild_settings,
+settings=settings,
+deallocate_config=deallocate_config,
write_auxdb=self._write_auxdb,
)
diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index 1674fa289e..70b83ee1f4 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -754,6 +754,7 @@ class depgraph:
def _dynamic_deps_preload(self, fake_vartree):
portdb = fake_vartree._portdb
+config_pool = []
for pkg in fake_vartree.dbapi:
self._spinner_update()
self._dynamic_config._package_tracker.add_installed_pkg(pkg)
@@ -768,12 +769,22 @@ class depgraph:
if metadata is not None:
fake_vartree.dynamic_deps_preload(pkg, metadata)
else:
+if config_pool:
+settings = config_pool.pop()
+else:
+settings = portage.config(clone=portdb.settings)
+
+deallocate_config = portdb._event_loop.create_future()
+deallocate_config.add_done_callback(
+lambda future: config_pool.append(future.result())
+)
proc = EbuildMetadataPhase(
cpv=pkg.cpv,
ebuild_hash=ebuild_hash,
portdb=portdb,
repo_path=repo_path,
settings=portdb.doebuild_settings,
+deallocate_config=deallocate_config,
)
proc.addExitListener(self._dynamic_deps_proc_exit(pkg,
fake_vartree))
yield proc
--
2.41.0
[gentoo-portage-dev] [PATCH 2/9] EbuildMetadataPhase: Add deallocate_config future
Use a deallocate_config future to release self.settings when
it is no longer needed. It's necessary to manage concurrency
since commit c95fc64abf96 because mutation of self.settings
is no longer limited to the EbuildMetadataPhase _start method,
where exclusive access was guaranteed within the main thread.
Add support to the isAlive() method to detect when the
EbuildMetadataPhase has started but the pid is not yet
available (due to async_check_locale usage from commit
c95fc64abf96). This can be used to check if an
EbuildMetadataPhase instance has been successfully started
so that it can be relied upon to set the result of the
deallocate_config future.
Bug: https://bugs.gentoo.org/924319
Signed-off-by: Zac Medico
---
lib/_emerge/EbuildMetadataPhase.py | 36 ++
lib/_emerge/SubProcess.py | 5 -
2 files changed, 40 insertions(+), 1 deletion(-)
diff --git a/lib/_emerge/EbuildMetadataPhase.py
b/lib/_emerge/EbuildMetadataPhase.py
index f4f685e81c..784712e8cb 100644
--- a/lib/_emerge/EbuildMetadataPhase.py
+++ b/lib/_emerge/EbuildMetadataPhase.py
@@ -14,6 +14,7 @@ from portage import os
from portage import _encodings
from portage import _unicode_decode
from portage import _unicode_encode
+from portage.util.futures import asyncio
import fcntl
@@ -33,6 +34,7 @@ class EbuildMetadataPhase(SubProcess):
"portdb",
"repo_path",
"settings",
+"deallocate_config",
"write_auxdb",
) + (
"_eapi",
@@ -127,6 +129,15 @@ class EbuildMetadataPhase(SubProcess):
returnproc=True,
)
settings.pop("PORTAGE_PIPE_FD", None)
+# At this point we can return settings to the caller
+# since we never use it for anything more than an
+# eapi_invalid call after this, and eapi_invalid is
+# insensitive to concurrent modifications.
+if (
+self.deallocate_config is not None
+and not self.deallocate_config.cancelled()
+):
+self.deallocate_config.set_result(settings)
os.close(slave_fd)
null_input.close()
@@ -139,6 +150,31 @@ class EbuildMetadataPhase(SubProcess):
self._proc = retval
+asyncio.ensure_future(
+self._async_start(), loop=self.scheduler
+).add_done_callback(self._async_start_done)
+
+async def _async_start(self):
+# Call async check_locale here for bug 923841, but code
+# also needs to migrate from _start to here, including
+# the self.deallocate_config set_result call.
+pass
+
+def _async_start_done(self, future):
+future.cancelled() or future.result()
+if self._was_cancelled():
+pass
+elif future.cancelled():
+self.cancel()
+self._was_cancelled()
+
+if self.deallocate_config is not None and not
self.deallocate_config.done():
+self.deallocate_config.set_result(self.settings)
+
+if self.returncode is not None:
+self._unregister()
+self.wait()
+
def _output_handler(self):
while True:
buf = self._read_buf(self._files.ebuild)
diff --git a/lib/_emerge/SubProcess.py b/lib/_emerge/SubProcess.py
index 029bbc3f44..057e0adc24 100644
--- a/lib/_emerge/SubProcess.py
+++ b/lib/_emerge/SubProcess.py
@@ -18,9 +18,12 @@ class SubProcess(AbstractPollTask):
# we've sent a kill signal to our subprocess.
_cancel_timeout = 1 # seconds
+def isAlive(self):
+return (self._registered or self.pid is not None) and self.returncode
is None
+
@property
def pid(self):
-return self._proc.pid
+return None if self._proc is None else self._proc.pid
def _poll(self):
# Simply rely on _async_waitpid_cb to set the returncode.
--
2.41.0
[gentoo-portage-dev] [PATCH 1/9] anydbm: Pickle support for multiprocessing spawn
The egencache usage in ResolverPlayground that was used to trigger
bug 924319 triggered a pickling error for AuxdbTestCase.test_anydbm
with the multiprocessing spawn start method, so fix the anydbm
cache module to omit the unpicklable database object from pickled
state, and regenerate it after unpickling.
Bug: https://bugs.gentoo.org/924319
Signed-off-by: Zac Medico
---
lib/portage/cache/anydbm.py | 17 -
lib/portage/tests/dbapi/test_auxdb.py | 4 +---
2 files changed, 17 insertions(+), 4 deletions(-)
diff --git a/lib/portage/cache/anydbm.py b/lib/portage/cache/anydbm.py
index 94a270a483..ad7042ae41 100644
--- a/lib/portage/cache/anydbm.py
+++ b/lib/portage/cache/anydbm.py
@@ -1,4 +1,4 @@
-# Copyright 2005-2020 Gentoo Authors
+# Copyright 2005-2024 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
# Author(s): Brian Harring (ferri...@gentoo.org)
@@ -67,6 +67,21 @@ class database(fs_template.FsBased):
raise cache_errors.InitializationError(self.__class__, e)
self._ensure_access(self._db_path)
+def __getstate__(self):
+state = self.__dict__.copy()
+# These attributes are not picklable, so they are automatically
+# regenerated after unpickling.
+state["_database__db"] = None
+return state
+
+def __setstate__(self, state):
+self.__dict__.update(state)
+mode = "w"
+if dbm.whichdb(self._db_path) in ("dbm.gnu", "gdbm"):
+# Allow multiple concurrent writers (see bug #53607).
+mode += "u"
+self.__db = dbm.open(self._db_path, mode, self._perms)
+
def iteritems(self):
# dbm doesn't implement items()
for k in self.__db.keys():
diff --git a/lib/portage/tests/dbapi/test_auxdb.py
b/lib/portage/tests/dbapi/test_auxdb.py
index 0de0123a5f..aac6ce361c 100644
--- a/lib/portage/tests/dbapi/test_auxdb.py
+++ b/lib/portage/tests/dbapi/test_auxdb.py
@@ -16,9 +16,7 @@ class AuxdbTestCase(TestCase):
from portage.cache.anydbm import database
except ImportError:
self.skipTest("dbm import failed")
-self._test_mod(
-"portage.cache.anydbm.database", multiproc=False, picklable=False
-)
+self._test_mod("portage.cache.anydbm.database", multiproc=False,
picklable=True)
def test_flat_hash_md5(self):
self._test_mod("portage.cache.flat_hash.md5_database")
--
2.41.0
[gentoo-portage-dev] [PATCH 0/9] EbuildPhase/EbuildMetadataPhase: async check_locale
Change config.environ() check_locale calls to async_check_locale calls in the EbuildPhase/EbuildMetadataPhase _async_start method in order to eliminate synchronous waiting for child processes in the main event loop thread. Note that this series of changes causes access to the portdbapi doebuild_settings attribute to no longer be serialized via the EbuildMetadataPhase _start_method. As a result, exclusive access to config instances needs to be guaranteed in some other way to avoid triggering problems (see bug 924319), such as by maintaining a config pool or by serializing access via an asyncio.Lock instance. This series can also be reviewed at https://github.com/gentoo/portage/pull/1267. Bug: https://bugs.gentoo.org/923841 Bug: https://bugs.gentoo.org/924319 Signed-off-by: Zac Medico Zac Medico (9): anydbm: Pickle support for multiprocessing spawn EbuildMetadataPhase: Add deallocate_config future MetadataRegen: Use EbuildMetadataPhase deallocate_config ResolverPlayground: Use egencache to create manifests asyncio: Wrap asyncio.Lock for python 3.9 compat async_aux_get: Use EbuildMetadataPhase deallocate_config future actions: disable pytest-xdist for spawn start-method (workers crash) EbuildMetadataPhase: Migrate to _async_start EbuildPhase: async_check_locale .github/workflows/ci.yml | 5 +- lib/_emerge/EbuildMetadataPhase.py| 37 - lib/_emerge/EbuildPhase.py| 28 +++- lib/_emerge/MetadataRegen.py | 16 ++- lib/_emerge/SubProcess.py | 5 +- lib/_emerge/depgraph.py | 11 ++ lib/portage/_emirrordist/FetchIterator.py | 10 +- lib/portage/cache/anydbm.py | 17 ++- lib/portage/dbapi/porttree.py | 129 -- lib/portage/package/ebuild/config.py | 26 ++-- lib/portage/tests/dbapi/test_auxdb.py | 4 +- lib/portage/tests/dbapi/test_portdb_cache.py | 4 +- .../tests/resolver/ResolverPlayground.py | 38 +++--- lib/portage/tests/update/test_move_ent.py | 3 + lib/portage/util/futures/_asyncio/__init__.py | 26 lib/portage/util/locale.py| 28 ++-- 16 files changed, 289 insertions(+), 98 deletions(-) -- 2.41.0
[gentoo-portage-dev] [PATCH v2] process.spawn: Add returnproc parameter
In order to migrate away from unsafe os.fork() usage in threaded
processes (https://github.com/python/cpython/issues/84559), add a
returnproc parameter that is similar to returnpid, which causes
spawn to return a single Process object instead of a list of pids.
The Process API is a subset of asyncio.subprocess.Process. The
returnproc parameter conflicts with the logfile parameter, since
the caller is expected to use the fd_pipes parameter to implement
logging (this was also true for the returnpid parameter).
In the future, spawn will return objects of a different type but
with a compatible interface to Process, in order to encapsulate
implementation-dependent objects like multiprocessing.Process which
are designed to manage the process lifecycle and need to persist
until it exits.
Trigger a UserWarning when the returnpid parameter is used, in
order to encourage migration to returnproc (do not use
DeprecationWarning since it is hidden by default). This warning
will be temporarily suppressed for portage internals, until they
finish migrating to returnproc. There are probably very few if
any external consumers of spawn with the returnpid parameter,
so it seems safe to move quickly with this deprecation.
Bug: https://bugs.gentoo.org/916566
Signed-off-by: Zac Medico
---
[PATCH v2] Make returnproc return a singlar Process (for
comparison, returnpid never returned a list container more than
a single pid). Also, returnproc explicitly conflicts with the
logfile parameter, since callers are expected to use the fd_pipes
parameter to implement logging (this was also true for returnpid).
lib/portage/process.py| 84 +++
lib/portage/tests/process/meson.build | 1 +
.../tests/process/test_spawn_returnproc.py| 39 +
3 files changed, 124 insertions(+)
create mode 100644 lib/portage/tests/process/test_spawn_returnproc.py
diff --git a/lib/portage/process.py b/lib/portage/process.py
index 71750a715f..6ec52efc4a 100644
--- a/lib/portage/process.py
+++ b/lib/portage/process.py
@@ -15,6 +15,7 @@ import subprocess
import sys
import traceback
import os as _os
+import warnings
from dataclasses import dataclass
from functools import lru_cache
@@ -27,6 +28,7 @@ import portage
portage.proxy.lazyimport.lazyimport(
globals(),
+"portage.util._eventloop.global_event_loop:global_event_loop",
"portage.util:dump_traceback,writemsg,writemsg_level",
)
@@ -277,12 +279,78 @@ def calc_env_stats(env) -> EnvStats:
env_too_large_warnings = 0
+class Process:
+"""
+An object that wraps OS processes created by spawn.
+In the future, spawn will return objects of a different type
+but with a compatible interface to this class, in order
+to encapsulate implementation-dependent objects like
+multiprocessing.Process which are designed to manage
+the process lifecycle and need to persist until it exits.
+"""
+
+def __init__(self, pid):
+self.pid = pid
+self.returncode = None
+self._exit_waiters = []
+
+def __repr__(self):
+return f"<{self.__class__.__name__} {self.pid}>"
+
+async def wait(self):
+"""
+Wait for the child process to terminate.
+
+Set and return the returncode attribute.
+"""
+if self.returncode is not None:
+return self.returncode
+
+loop = global_event_loop()
+if not self._exit_waiters:
+loop._asyncio_child_watcher.add_child_handler(self.pid,
self._child_handler)
+waiter = loop.create_future()
+self._exit_waiters.append(waiter)
+return await waiter
+
+def _child_handler(self, pid, returncode):
+if pid != self.pid:
+raise AssertionError(f"expected pid {self.pid}, got {pid}")
+self.returncode = returncode
+
+for waiter in self._exit_waiters:
+if not waiter.cancelled():
+waiter.set_result(returncode)
+self._exit_waiters = None
+
+def send_signal(self, sig):
+"""Send a signal to the process."""
+if self.returncode is not None:
+# Skip signalling a process that we know has already died.
+return
+
+try:
+os.kill(self.pid, sig)
+except ProcessLookupError:
+# Suppress the race condition error; bpo-40550.
+pass
+
+def terminate(self):
+"""Terminate the process with SIGTERM"""
+self.send_signal(signal.SIGTERM)
+
+def kill(self):
+"""Kill the process with SIGKILL"""
+self.send_signal(signal.SIGKILL)
+
+
def spawn(
mycommand,
env=None,
opt_name=None,
fd_pipes=None,
returnpid=False,
+returnproc=False,
uid=None,
gid=None,
groups=None,
@@ -315,6 +383,9 @@ def spawn(
@param returnpid: Return the Process IDs for a successful spawn.
NOTE: This requires the caller clean up all the PIDs, otherwise spawn will
clean them.
[gentoo-portage-dev] [PATCH] process.spawn: Add returnproc parameter
In order to migrate away from unsafe os.fork() usage in threaded
processes (https://github.com/python/cpython/issues/84559), add a
returnproc parameter that is similar to returnpid, which causes
spawn to return Process objects instead of pids. The Process
API is a subset of asyncio.subprocess.Process.
In the future, spawn will return objects of a different type but
with a compatible interface to Process, in order to encapsulate
implementation-dependent objects like multiprocessing.Process which
are designed to manage the process lifecycle and need to persist
until it exits.
Trigger a UserWarning when the returnpid parameter is used, in
order to encourage migration to returnproc (do not use
DeprecationWarning since it is hidden by default). This warning
will be temporarily suppressed for portage internals, until they
finish migrating to returnproc. There are probably very few if
any external consumers of spawn with the returnpid parameter,
so it seems safe to move quickly with this deprecation.
Bug: https://bugs.gentoo.org/916566
Signed-off-by: Zac Medico
---
lib/portage/process.py| 79 +++
lib/portage/tests/process/meson.build | 1 +
.../tests/process/test_spawn_returnproc.py| 39 +
3 files changed, 119 insertions(+)
create mode 100644 lib/portage/tests/process/test_spawn_returnproc.py
diff --git a/lib/portage/process.py b/lib/portage/process.py
index 71750a715f..74953b3e22 100644
--- a/lib/portage/process.py
+++ b/lib/portage/process.py
@@ -15,6 +15,7 @@ import subprocess
import sys
import traceback
import os as _os
+import warnings
from dataclasses import dataclass
from functools import lru_cache
@@ -27,6 +28,7 @@ import portage
portage.proxy.lazyimport.lazyimport(
globals(),
+"portage.util._eventloop.global_event_loop:global_event_loop",
"portage.util:dump_traceback,writemsg,writemsg_level",
)
@@ -277,12 +279,78 @@ def calc_env_stats(env) -> EnvStats:
env_too_large_warnings = 0
+class Process:
+"""
+An object that wraps OS processes created by spawn.
+In the future, spawn will return objects of a different type
+but with a compatible interface to this class, in order
+to encapsulate implementation-dependent objects like
+multiprocessing.Process which are designed to manage
+the process lifecycle and need to persist until it exits.
+"""
+
+def __init__(self, pid):
+self.pid = pid
+self.returncode = None
+self._exit_waiters = []
+
+def __repr__(self):
+return f"<{self.__class__.__name__} {self.pid}>"
+
+async def wait(self):
+"""
+Wait for the child process to terminate.
+
+Set and return the returncode attribute.
+"""
+if self.returncode is not None:
+return self.returncode
+
+loop = global_event_loop()
+if not self._exit_waiters:
+loop._asyncio_child_watcher.add_child_handler(self.pid,
self._child_handler)
+waiter = loop.create_future()
+self._exit_waiters.append(waiter)
+return await waiter
+
+def _child_handler(self, pid, returncode):
+if pid != self.pid:
+raise AssertionError(f"expected pid {self.pid}, got {pid}")
+self.returncode = returncode
+
+for waiter in self._exit_waiters:
+if not waiter.cancelled():
+waiter.set_result(returncode)
+self._exit_waiters = None
+
+def send_signal(self, sig):
+"""Send a signal to the process."""
+if self.returncode is not None:
+# Skip signalling a process that we know has already died.
+return
+
+try:
+os.kill(self.pid, sig)
+except ProcessLookupError:
+# Suppress the race condition error; bpo-40550.
+pass
+
+def terminate(self):
+"""Terminate the process with SIGTERM"""
+self.send_signal(signal.SIGTERM)
+
+def kill(self):
+"""Kill the process with SIGKILL"""
+self.send_signal(signal.SIGKILL)
+
+
def spawn(
mycommand,
env=None,
opt_name=None,
fd_pipes=None,
returnpid=False,
+returnproc=False,
uid=None,
gid=None,
groups=None,
@@ -315,6 +383,9 @@ def spawn(
@param returnpid: Return the Process IDs for a successful spawn.
NOTE: This requires the caller clean up all the PIDs, otherwise spawn will
clean them.
@type returnpid: Boolean
+@param returnproc: Return the Process objects for a successful spawn.
+NOTE: This requires the caller clean up all the PIDs, otherwise spawn will
clean them.
+@type returnproc: Boolean
@param uid: User ID to spawn as; useful for dropping privilages
@type uid: Integer
@param gid: Group ID to spawn as; useful for dropping privilages
@@ -491,7 +562,15 @@ def spawn(
# If the caller wants to handle cleaning up the processes, we tell
# it about all processes that
[gentoo-portage-dev] [PATCH v2] Stop implicitly manipulating `NO_COLOR`/`NOCOLOR`
Stop implicitly forcing `NO_COLOR` and `NOCOLOR` in ebuild environment.
This is undesired for several reasons:
1. It makes it impossible to control color for emerge output
independently of command output, e.g. when one goes to a pty while
the other goes to logs.
2. It makes it impossible to get color output in logs when running
`emerge --jobs ...`.
3. Forcing `NO_COLOR=1` turns out to cause random test failures,
and while fixing them is commendable, it is a pain for arch testing
and it is currently blocking stabilization requests.
With the new approach, the color output in programs is consistent
between using ``--jobs`` or ``--quiet-build``, and not. Therefore,
both cases generate uniform logs. In order to obtain logs free of color
codes, one can either filter them via `ansifilter(1)` (as the manpages
already recommend) or explicitly set `NO_COLOR`.
Furthermore, one can combine color-free build output (for clean logs)
with colorful emerge output by using:
NO_COLOR=true emerge --color y ...
Bug: https://bugs.gentoo.org/918515
Signed-off-by: Michał Górny
---
NEWS | 5 +
bin/ebuild | 4
lib/_emerge/AbstractEbuildProcess.py | 6 --
lib/_emerge/actions.py | 5 -
lib/portage/tests/resolver/ResolverPlayground.py | 3 ---
5 files changed, 5 insertions(+), 18 deletions(-)
Changes in v2: add a NEWS entry
diff --git a/NEWS b/NEWS
index f1305744f..eb5eb4ab0 100644
--- a/NEWS
+++ b/NEWS
@@ -9,6 +9,11 @@ Release notes take the form of the following optional
categories:
portage-3.0.56 (UNRELEASED)
--
+Breaking changes:
+* ebuild: no longer implicitly set NOCOLOR and NO_COLOR when color output
+ is disabled for emerge itself. In order to obtain color-free logs, please
+ either explicitly set NO_COLOR environment variable or use e.g. ansifilter.
+
Features:
* sync: git: Add sync-git-verify-max-age-days option.
diff --git a/bin/ebuild b/bin/ebuild
index cbefb5816..69db474c0 100755
--- a/bin/ebuild
+++ b/bin/ebuild
@@ -132,10 +132,6 @@ def main():
or not sys.stdout.isatty()
):
portage.output.nocolor()
-portage.settings.unlock()
-portage.settings["NO_COLOR"] = "true"
-portage.settings.backup_changes("NO_COLOR")
-portage.settings.lock()
apply_priorities(portage.settings)
diff --git a/lib/_emerge/AbstractEbuildProcess.py
b/lib/_emerge/AbstractEbuildProcess.py
index 96d91b5da..3a8a18b5f 100644
--- a/lib/_emerge/AbstractEbuildProcess.py
+++ b/lib/_emerge/AbstractEbuildProcess.py
@@ -80,12 +80,6 @@ class AbstractEbuildProcess(SpawnProcess):
self._async_wait()
return
-if self.background:
-# Automatically prevent color codes from showing up in logs,
-# since we're not displaying to a terminal anyway.
-self.settings["NOCOLOR"] = "true"
-self.settings["NO_COLOR"] = "true"
-
start_ipc_daemon = False
if self._enable_ipc_daemon:
self.settings.pop("PORTAGE_EBUILD_EXIT_FILE", None)
diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index 37264114e..dbd9707a8 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -2749,15 +2749,10 @@ def adjust_config(myopts, settings):
if "--color" in myopts:
if "y" == myopts["--color"]:
portage.output.havecolor = 1
-settings["NO_COLOR"] = ""
else:
portage.output.havecolor = 0
-settings["NO_COLOR"] = "true"
-settings.backup_changes("NO_COLOR")
elif settings.get("TERM") == "dumb" or not sys.stdout.isatty():
portage.output.havecolor = 0
-settings["NO_COLOR"] = "true"
-settings.backup_changes("NO_COLOR")
if "--pkg-format" in myopts:
settings["PORTAGE_BINPKG_FORMAT"] = myopts["--pkg-format"]
diff --git a/lib/portage/tests/resolver/ResolverPlayground.py
b/lib/portage/tests/resolver/ResolverPlayground.py
index 475c4aaac..115800a60 100644
--- a/lib/portage/tests/resolver/ResolverPlayground.py
+++ b/lib/portage/tests/resolver/ResolverPlayground.py
@@ -591,9 +591,6 @@ class ResolverPlayground:
"PORTAGE_TMPDIR": os.path.join(self.eroot, "var/tmp"),
}
-if portage.util.no_color(os.environ):
-make_conf["NO_COLOR"] = os.environ["NO_COLOR"]
-
# Pass along PORTAGE_USERNAME and PORTAGE_GRPNAME since they
# need to be inherited by ebuild subprocesses.
if "PORTAGE_USERNAME" in os.environ:
--
2.43.0
Re: [gentoo-portage-dev] [PATCH] Stop implicitly manipulating `NO_COLOR`/`NOCOLOR`
On Sun, 2023-11-26 at 18:33 +0100, Ulrich Mueller wrote: > > > > > > On Sun, 26 Nov 2023, Michał Górny wrote: > > > 3. Forcing `NO_COLOR=1` turns out to cause random test failures, > > and while fixing them is commendable, it is a pain for arch testing > > and it is currently blocking stabilization requests. > > I'd argue that test suites that fail because of NO_COLOR are broken. > IIUC, these failures will show up for users who set NO_COLOR=1 in their > make.conf or in the environment? Sure, they are broken. However, as I said I'd rather fix them in my spare time than have to fix them to unblock a security stablereq because implicit Portage logic blocks all AT work on a minor issue that's non- trivial to fix. > > > With the new approach, the color output in programs is consistent > > between using ``--jobs`` or ``--quiet-build``, and not. Therefore, > > both cases generate uniform logs. In order to obtain logs free of color > > codes, one can either filter them via `ansifilter(1)` (as the manpages > > already recommend) or explicitly set `NO_COLOR`. > > Will this still guarantee that NO_COLOR=1 from the environment is always > passed on to the build system? > > Otherwise, it would break ebuild-mode in some configurations. > Yes, I've tested it and it worked. Both for passing from external env and via make.conf. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] [PATCH] Stop implicitly manipulating `NO_COLOR`/`NOCOLOR`
> On Sun, 26 Nov 2023, Michał Górny wrote: > 3. Forcing `NO_COLOR=1` turns out to cause random test failures, >and while fixing them is commendable, it is a pain for arch testing >and it is currently blocking stabilization requests. I'd argue that test suites that fail because of NO_COLOR are broken. IIUC, these failures will show up for users who set NO_COLOR=1 in their make.conf or in the environment? > With the new approach, the color output in programs is consistent > between using ``--jobs`` or ``--quiet-build``, and not. Therefore, > both cases generate uniform logs. In order to obtain logs free of color > codes, one can either filter them via `ansifilter(1)` (as the manpages > already recommend) or explicitly set `NO_COLOR`. Will this still guarantee that NO_COLOR=1 from the environment is always passed on to the build system? Otherwise, it would break ebuild-mode in some configurations. Ulrich signature.asc Description: PGP signature
[gentoo-portage-dev] [PATCH] Stop implicitly manipulating `NO_COLOR`/`NOCOLOR`
Stop implicitly forcing `NO_COLOR` and `NOCOLOR` in ebuild environment.
This is undesired for several reasons:
1. It makes it impossible to control color for emerge output
independently of command output, e.g. when one goes to a pty while
the other goes to logs.
2. It makes it impossible to get color output in logs when running
`emerge --jobs ...`.
3. Forcing `NO_COLOR=1` turns out to cause random test failures,
and while fixing them is commendable, it is a pain for arch testing
and it is currently blocking stabilization requests.
With the new approach, the color output in programs is consistent
between using ``--jobs`` or ``--quiet-build``, and not. Therefore,
both cases generate uniform logs. In order to obtain logs free of color
codes, one can either filter them via `ansifilter(1)` (as the manpages
already recommend) or explicitly set `NO_COLOR`.
Furthermore, one can combine color-free build output (for clean logs)
with colorful emerge output by using:
NO_COLOR=true emerge --color y ...
Bug: https://bugs.gentoo.org/918515
---
bin/ebuild | 4
lib/_emerge/AbstractEbuildProcess.py | 6 --
lib/_emerge/actions.py | 5 -
lib/portage/tests/resolver/ResolverPlayground.py | 3 ---
4 files changed, 18 deletions(-)
diff --git a/bin/ebuild b/bin/ebuild
index cbefb5816..69db474c0 100755
--- a/bin/ebuild
+++ b/bin/ebuild
@@ -132,10 +132,6 @@ def main():
or not sys.stdout.isatty()
):
portage.output.nocolor()
-portage.settings.unlock()
-portage.settings["NO_COLOR"] = "true"
-portage.settings.backup_changes("NO_COLOR")
-portage.settings.lock()
apply_priorities(portage.settings)
diff --git a/lib/_emerge/AbstractEbuildProcess.py
b/lib/_emerge/AbstractEbuildProcess.py
index 96d91b5da..3a8a18b5f 100644
--- a/lib/_emerge/AbstractEbuildProcess.py
+++ b/lib/_emerge/AbstractEbuildProcess.py
@@ -80,12 +80,6 @@ class AbstractEbuildProcess(SpawnProcess):
self._async_wait()
return
-if self.background:
-# Automatically prevent color codes from showing up in logs,
-# since we're not displaying to a terminal anyway.
-self.settings["NOCOLOR"] = "true"
-self.settings["NO_COLOR"] = "true"
-
start_ipc_daemon = False
if self._enable_ipc_daemon:
self.settings.pop("PORTAGE_EBUILD_EXIT_FILE", None)
diff --git a/lib/_emerge/actions.py b/lib/_emerge/actions.py
index 37264114e..dbd9707a8 100644
--- a/lib/_emerge/actions.py
+++ b/lib/_emerge/actions.py
@@ -2749,15 +2749,10 @@ def adjust_config(myopts, settings):
if "--color" in myopts:
if "y" == myopts["--color"]:
portage.output.havecolor = 1
-settings["NO_COLOR"] = ""
else:
portage.output.havecolor = 0
-settings["NO_COLOR"] = "true"
-settings.backup_changes("NO_COLOR")
elif settings.get("TERM") == "dumb" or not sys.stdout.isatty():
portage.output.havecolor = 0
-settings["NO_COLOR"] = "true"
-settings.backup_changes("NO_COLOR")
if "--pkg-format" in myopts:
settings["PORTAGE_BINPKG_FORMAT"] = myopts["--pkg-format"]
diff --git a/lib/portage/tests/resolver/ResolverPlayground.py
b/lib/portage/tests/resolver/ResolverPlayground.py
index 475c4aaac..115800a60 100644
--- a/lib/portage/tests/resolver/ResolverPlayground.py
+++ b/lib/portage/tests/resolver/ResolverPlayground.py
@@ -591,9 +591,6 @@ class ResolverPlayground:
"PORTAGE_TMPDIR": os.path.join(self.eroot, "var/tmp"),
}
-if portage.util.no_color(os.environ):
-make_conf["NO_COLOR"] = os.environ["NO_COLOR"]
-
# Pass along PORTAGE_USERNAME and PORTAGE_GRPNAME since they
# need to be inherited by ebuild subprocesses.
if "PORTAGE_USERNAME" in os.environ:
--
2.43.0
[gentoo-portage-dev] [PATCH 3/3] prepman: Deprecate it
According to PMS, calling prepman from ebuilds was never allowed.
Last usage in Gentoo repository has been removed on 2019-04-14.
Signed-off-by: Ulrich Müller
---
bin/ebuild-helpers/prepman | 6 +-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/bin/ebuild-helpers/prepman b/bin/ebuild-helpers/prepman
index 871fd589b..0c0a7e812 100755
--- a/bin/ebuild-helpers/prepman
+++ b/bin/ebuild-helpers/prepman
@@ -1,8 +1,12 @@
#!/usr/bin/env bash
-# Copyright 1999-2018 Gentoo Foundation
+# Copyright 1999-2023 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
# Note: this really does nothing these days. It's going to be banned
# when the last consumers are gone.
+source "${PORTAGE_BIN_PATH}"/isolated-functions.sh || exit 1
+
+eqawarn "QA Notice: '${0##*/}' is not allowed in ebuild scope"
+
exit 0
--
2.40.0
[gentoo-portage-dev] [PATCH 2/3] prepinfo: Deprecate it
According to PMS, calling prepinfo from ebuilds was never allowed.
Last usage in Gentoo repository has been removed on 2018-11-22.
Bug: https://bugs.gentoo.org/899898
Signed-off-by: Ulrich Müller
---
bin/ebuild-helpers/prepinfo | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/bin/ebuild-helpers/prepinfo b/bin/ebuild-helpers/prepinfo
index 14fa38226..af5ae9f08 100755
--- a/bin/ebuild-helpers/prepinfo
+++ b/bin/ebuild-helpers/prepinfo
@@ -1,9 +1,11 @@
#!/usr/bin/env bash
-# Copyright 1999-2022 Gentoo Authors
+# Copyright 1999-2023 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
source "${PORTAGE_BIN_PATH}"/isolated-functions.sh || exit 1
+eqawarn "QA Notice: '${0##*/}' is not allowed in ebuild scope"
+
if ! ___eapi_has_prefix_variables ; then
ED=${D}
fi
--
2.40.0
[gentoo-portage-dev] [PATCH 1/3] misc-functions.sh: Inline prepinfo code in install_qa_check()
Add a deprecation warning. Presumably, subdirs of /usr/share/info have
never been used by any package except app-editors/emacs.
Remove duplicate rm statement at the end of the function.
Bug: https://bugs.gentoo.org/899898
Signed-off-by: Ulrich Müller
---
bin/misc-functions.sh | 24 +++-
1 file changed, 19 insertions(+), 5 deletions(-)
diff --git a/bin/misc-functions.sh b/bin/misc-functions.sh
index 55c4afe73..235823afc 100755
--- a/bin/misc-functions.sh
+++ b/bin/misc-functions.sh
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
-# Copyright 1999-2018 Gentoo Authors
+# Copyright 1999-2023 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
#
# Miscellaneous shell functions that make use of the ebuild env but don't need
@@ -143,7 +143,24 @@ install_qa_check() {
chflags -R nosunlnk,nouunlnk "${ED}" 2>/dev/null
fi
- [[ -d ${ED%/}/usr/share/info ]] && prepinfo
+ if [[ -d ${ED%/}/usr/share/info ]]; then
+ # Portage regenerates this on the installed system.
+ rm -f
"${ED%/}"/usr/share/info/dir{,.info}{,.Z,.gz,.bz2,.lzma,.lz,.xz,.zst} \
+ || die "rm failed"
+ # Recurse into subdirs. Remove this code after 2023-12-31.
#899898
+ while read -r -d '' x; do
+ ( shopt -s failglob; : "${x}"/.keepinfodir* )
2>/dev/null \
+ && continue
+ for f in
"${x}"/dir{,.info}{,.Z,.gz,.bz2,.lzma,.lz,.xz,.zst}; do
+ if [[ -e ${f} ]]; then
+ eqawarn "QA Notice: Removing Info
directory file '${f}'."
+ eqawarn "Relying on this behavior is
deprecated and may"
+ eqawarn "cause file collisions in
future."
+ rm -f "${f}" || die "rm failed"
+ fi
+ done
+ done < <(find "${ED%/}"/usr/share/info -mindepth 1 -type d
-print0)
+ fi
# If binpkg-docompress is enabled, apply compression before creating
# the binary package.
@@ -251,9 +268,6 @@ install_qa_check() {
prepallstrip
fi
fi
-
- # Portage regenerates this on the installed system.
- rm -f "${ED%/}"/usr/share/info/dir{,.Z,.gz,.bz2,.lzma,.lz,.xz,.zst} ||
die "rm failed!"
}
__dyn_instprep() {
--
2.40.0
Re: [gentoo-portage-dev] [PATCH] Remove obsolete FEATURES=force-prefix
Sounds fine to me too. On 22-01-2023 09:16:53 +, James Le Cuirot wrote: > On Sun, 2023-01-22 at 09:18 +0100, Ulrich Müller wrote: > > Signed-off-by: Ulrich Müller > > --- > > bin/dohtml.py| 6 ++ > > bin/eapi.sh | 4 ++-- > > lib/portage/const.py | 3 +-- > > lib/portage/package/ebuild/config.py | 11 +++ > > man/make.conf.5 | 8 +--- > > 5 files changed, 9 insertions(+), 23 deletions(-) > > +1 -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] Remove obsolete FEATURES=force-prefix
On Sun, 2023-01-22 at 09:18 +0100, Ulrich Müller wrote: > Signed-off-by: Ulrich Müller > --- > bin/dohtml.py| 6 ++ > bin/eapi.sh | 4 ++-- > lib/portage/const.py | 3 +-- > lib/portage/package/ebuild/config.py | 11 +++ > man/make.conf.5 | 8 +--- > 5 files changed, 9 insertions(+), 23 deletions(-) +1 signature.asc Description: This is a digitally signed message part
[gentoo-portage-dev] [PATCH] Remove obsolete FEATURES=force-prefix
Signed-off-by: Ulrich Müller
---
bin/dohtml.py| 6 ++
bin/eapi.sh | 4 ++--
lib/portage/const.py | 3 +--
lib/portage/package/ebuild/config.py | 11 +++
man/make.conf.5 | 8 +---
5 files changed, 9 insertions(+), 23 deletions(-)
diff --git a/bin/dohtml.py b/bin/dohtml.py
index 4ebee7779..d1f0e3140 100755
--- a/bin/dohtml.py
+++ b/bin/dohtml.py
@@ -1,5 +1,5 @@
#!/usr/bin/env python
-# Copyright 1999-2020 Gentoo Authors
+# Copyright 1999-2023 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
#
@@ -146,9 +146,7 @@ class OptionsClass:
self.PF = os.environ["PF"]
if self.PF:
self.PF = normalize_path(self.PF)
-if "force-prefix" not in os.environ.get(
-"FEATURES", ""
-).split() and os.environ.get("EAPI", "0") in ("0", "1", "2"):
+if os.environ.get("EAPI", "0") in ("0", "1", "2"):
self.ED = os.environ.get("D", "")
else:
self.ED = os.environ.get("ED", "")
diff --git a/bin/eapi.sh b/bin/eapi.sh
index d90c3cd2e..ae815f3a6 100644
--- a/bin/eapi.sh
+++ b/bin/eapi.sh
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
-# Copyright 2012-2022 Gentoo Authors
+# Copyright 2012-2023 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
# PHASES
@@ -27,7 +27,7 @@ ___eapi_has_S_WORKDIR_fallback() {
# VARIABLES
___eapi_has_prefix_variables() {
- [[ ! ${1-${EAPI-0}} =~ ^(0|1|2)$ || " ${FEATURES} " == *" force-prefix
"* ]]
+ [[ ! ${1-${EAPI-0}} =~ ^(0|1|2)$ ]]
}
___eapi_has_BROOT() {
diff --git a/lib/portage/const.py b/lib/portage/const.py
index a6f7a2242..99206fe2c 100644
--- a/lib/portage/const.py
+++ b/lib/portage/const.py
@@ -1,5 +1,5 @@
# portage: Constants
-# Copyright 1998-2021 Gentoo Authors
+# Copyright 1998-2023 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
import os
@@ -156,7 +156,6 @@ SUPPORTED_FEATURES = frozenset(
"fakeroot",
"fixlafiles",
"force-mirror",
-"force-prefix",
"getbinpkg",
"gpg-keepalive",
"icecream",
diff --git a/lib/portage/package/ebuild/config.py
b/lib/portage/package/ebuild/config.py
index 23b524235..f66bf6485 100644
--- a/lib/portage/package/ebuild/config.py
+++ b/lib/portage/package/ebuild/config.py
@@ -1,4 +1,4 @@
-# Copyright 2010-2021 Gentoo Authors
+# Copyright 2010-2023 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
__all__ = [
@@ -3347,14 +3347,9 @@ class config:
if not (src_like_phase and eapi_attrs.broot):
mydict.pop("BROOT", None)
-# Prefix variables are supported beginning with EAPI 3, or when
-# force-prefix is in FEATURES, since older EAPIs would otherwise be
-# useless with prefix configurations. This brings compatibility with
-# the prefix branch of portage, which also supports EPREFIX for all
-# EAPIs (for obvious reasons).
+# Prefix variables are supported beginning with EAPI 3.
if phase == "depend" or (
-"force-prefix" not in self.features
-and eapi is not None
+eapi is not None
and not eapi_supports_prefix(eapi)
):
mydict.pop("ED", None)
diff --git a/man/make.conf.5 b/man/make.conf.5
index 34c3cd5b5..05832ce1c 100644
--- a/man/make.conf.5
+++ b/man/make.conf.5
@@ -1,4 +1,4 @@
-.TH "MAKE.CONF" "5" "Nov 2021" "Portage VERSION" "Portage"
+.TH "MAKE.CONF" "5" "Jan 2023" "Portage VERSION" "Portage"
.SH "NAME"
make.conf \- custom settings for Portage
.SH "SYNOPSIS"
@@ -522,12 +522,6 @@ fixes (order of flags, duplicated entries, ...)
Only fetch files from configured mirrors, ignoring \fBSRC_URI\fR,
except when \fImirror\fR is in the \fBebuild\fR(5) \fBRESTRICT\fR variable.
.TP
-.B force\-prefix
-Enable prefix support for all ebuilds, regardless of EAPI, since older EAPIs
-would otherwise be useless with prefix configurations. This brings
-compatibility with the prefix branch of portage, which also supports EPREFIX
-for all EAPIs (for obvious reasons).
-.TP
.B getbinpkg
Force emerges to always try to fetch files from the \fIPORTAGE_BINHOST\fR. See
\fBmake.conf\fR(5) for more information.
--
2.39.1
Re: [gentoo-portage-dev] [PATCH 2/2] Add caching to _slot_operator_check_reverse_dependencies
On 11/24/22 19:36, Pin-yen Lin wrote: Add lru_cache to speed up the running time of "Calculating dependencies". In a ChromeOS use case, this patch decreases the running time from 311s to 197s with almost no memory usage increase. Signed-off-by: Pin-yen Lin --- lib/_emerge/depgraph.py | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py index ce6cabcc1..9649bb2a8 100644 --- a/lib/_emerge/depgraph.py +++ b/lib/_emerge/depgraph.py @@ -2240,6 +2240,7 @@ class depgraph: return None +@functools.lru_cache(maxsize=100) def _slot_operator_check_reverse_dependencies( self, existing_pkg, candidate_pkg, replacement_parent=None ): Merged. Thank you! https://gitweb.gentoo.org/proj/portage.git/commit/?id=0c42cc962e1926ecbdc83d903a2804f9e037f2a9 https://gitweb.gentoo.org/proj/portage.git/commit/?id=839ab46be1777e5886da28b98b53a462b992c5bf -- Thanks, Zac
Re: [gentoo-portage-dev] [PATCH 2/2] Add caching to _slot_operator_check_reverse_dependencies
On Thu, Nov 24, 2022 at 10:36 PM Pin-yen Lin wrote: > > Add lru_cache to speed up the running time of "Calculating > dependencies". > > In a ChromeOS use case, this patch decreases the running time from > 311s to 197s with almost no memory usage increase. > > Signed-off-by: Pin-yen Lin Thank you! With recent subslot rebuilds (icu, boost, poppler), I measure an improvement of 19%! Benchmark 1: emerge @world -vuNDp Time (mean ± σ): 42.668 s ± 0.555 s[User: 42.095 s, System: 0.315 s] Range (min … max): 41.572 s … 43.342 s10 runs Benchmark 2: emerge @world -vuNDp Time (mean ± σ): 35.991 s ± 0.154 s[User: 35.409 s, System: 0.332 s] Range (min … max): 35.831 s … 36.306 s10 runs Summary 'emerge @world -vuNDp' ran 1.19 ± 0.02 times faster than 'emerge @world -vuNDp'
Re: [gentoo-portage-dev] sys-kernel/vanilla-sources not supported any more?
On Fri, Nov 25, 2022 at 5:50 AM Joakim Tjernlund wrote: > > I have noticed sys-kernel/vanilla-sources lagging behind for a while now and > I wonder if this pkg is obsolete? I think you sent this message to the wrong place. This list is for discussion of Portage development, not packages in the gentoo repository.
[gentoo-portage-dev] sys-kernel/vanilla-sources not supported any more?
I have noticed sys-kernel/vanilla-sources lagging behind for a while now and I wonder if this pkg is obsolete? Jocke
[gentoo-portage-dev] [PATCH 2/2] Add caching to _slot_operator_check_reverse_dependencies
Add lru_cache to speed up the running time of "Calculating dependencies". In a ChromeOS use case, this patch decreases the running time from 311s to 197s with almost no memory usage increase. Signed-off-by: Pin-yen Lin --- lib/_emerge/depgraph.py | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py index ce6cabcc1..9649bb2a8 100644 --- a/lib/_emerge/depgraph.py +++ b/lib/_emerge/depgraph.py @@ -2240,6 +2240,7 @@ class depgraph: return None +@functools.lru_cache(maxsize=100) def _slot_operator_check_reverse_dependencies( self, existing_pkg, candidate_pkg, replacement_parent=None ): -- 2.38.1.584.g0f3c55d4c2-goog
[gentoo-portage-dev] [PATCH 1/2] Move and rename check_reverse_dependencies
Move the function outside the original scope and rename the function to
_slot_operator_check_reverse_dependencies to accommodate the coding
style.
Signed-off-by: Pin-yen Lin
---
lib/_emerge/depgraph.py | 232 +---
1 file changed, 119 insertions(+), 113 deletions(-)
diff --git a/lib/_emerge/depgraph.py b/lib/_emerge/depgraph.py
index c976f8205..ce6cabcc1 100644
--- a/lib/_emerge/depgraph.py
+++ b/lib/_emerge/depgraph.py
@@ -2240,6 +2240,117 @@ class depgraph:
return None
+def _slot_operator_check_reverse_dependencies(
+self, existing_pkg, candidate_pkg, replacement_parent=None
+):
+"""
+Check if candidate_pkg satisfies all of existing_pkg's non-
+slot operator parents.
+"""
+debug = "--debug" in self._frozen_config.myopts
+built_slot_operator_parents = set()
+for parent, atom in self._dynamic_config._parent_atoms.get(
+existing_pkg, []
+):
+if atom.soname or atom.slot_operator_built:
+built_slot_operator_parents.add(parent)
+
+for parent, atom in self._dynamic_config._parent_atoms.get(
+existing_pkg, []
+):
+if isinstance(parent, Package):
+if parent in built_slot_operator_parents:
+if hasattr(atom, "_orig_atom"):
+# If atom is the result of virtual expansion, then
+# derefrence it to _orig_atom so that it will be
correctly
+# handled as a built slot operator dependency when
+# appropriate (see bug 764764).
+atom = atom._orig_atom
+# This parent may need to be rebuilt, therefore
+# discard its soname and built slot operator
+# dependency components which are not necessarily
+# relevant.
+if atom.soname:
+continue
+elif atom.package and atom.slot_operator_built:
+# This discards the slot/subslot component.
+atom = atom.with_slot("=")
+
+if replacement_parent is not None and (
+replacement_parent.slot_atom == parent.slot_atom
+or replacement_parent.cpv == parent.cpv
+):
+# This parent is irrelevant because we intend to
+# replace it with replacement_parent.
+continue
+
+if any(
+pkg is not parent
+and (pkg.slot_atom == parent.slot_atom or pkg.cpv ==
parent.cpv)
+for pkg in self._dynamic_config._package_tracker.match(
+parent.root, Atom(parent.cp)
+)
+):
+# This parent may need to be eliminated due to a
+# slot conflict, so its dependencies aren't
+# necessarily relevant.
+continue
+
+if not self._too_deep(
+parent.depth
+) and not self._frozen_config.excluded_pkgs.findAtomForPackage(
+parent, modified_use=self._pkg_use_enabled(parent)
+):
+# Check for common reasons that the parent's
+# dependency might be irrelevant.
+if self._upgrade_available(parent):
+# This parent could be replaced by
+# an upgrade (bug 584626).
+continue
+if parent.installed and self._in_blocker_conflict(parent):
+# This parent could be uninstalled in order
+# to solve a blocker conflict (bug 612772).
+continue
+if self._dynamic_config.digraph.has_edge(parent,
existing_pkg):
+# There is a direct circular dependency between
+# parent and existing_pkg. This type of
+# relationship tends to prevent updates
+# of packages (bug 612874). Since candidate_pkg
+# is available, we risk a missed update if we
+# don't try to eliminate this parent from the
+# graph. Therefore, we give candidate_pkg a
+# chance, and assume that it will be masked
+# by backtracking if necessary.
+continue
+
+atom_set = InternalPackageSet(initial_atoms=(atom,),
allow_repo=True)
+if not atom_set.findAtomForPackage(
+candidate_pkg,
modified_use=self._pkg_use_enabled(candidate_pkg)
+):
+if debug:
+parent_atoms = []
+
Re: [gentoo-portage-dev] Fixing EAPI 8 --enable-static once and for all
On Sat, Nov 19, 2022 at 3:33 AM Ulrich Mueller wrote: > > > On Sat, 19 Nov 2022, David Seifert wrote: > > > With this extensive analysis, I believe this patch to be safe. > > Still looks like an incompatible change, so it will need an EAPI bump. > > EAPI 9 feature bug is here: https://bugs.gentoo.org/815169 I support this patch to fix Portage, regardless of what PMS says. Coding to the spec doesn't make sense if the spec is broken.
Re: [gentoo-portage-dev] Fixing EAPI 8 --enable-static once and for all
On Sat, 2022-11-19 at 09:33 +0100, Ulrich Mueller wrote: > > > > > > On Sat, 19 Nov 2022, David Seifert wrote: > > > With this extensive analysis, I believe this patch to be safe. > > Still looks like an incompatible change, so it will need an EAPI bump. > > EAPI 9 feature bug is here: https://bugs.gentoo.org/815169 I dare say we've done worse retroactive changes. -- Best regards, Michał Górny
Re: [gentoo-portage-dev] Fixing EAPI 8 --enable-static once and for all
> On Sat, 19 Nov 2022, David Seifert wrote: > With this extensive analysis, I believe this patch to be safe. Still looks like an incompatible change, so it will need an EAPI bump. EAPI 9 feature bug is here: https://bugs.gentoo.org/815169 signature.asc Description: PGP signature
[gentoo-portage-dev] Fixing EAPI 8 --enable-static once and for all
Attached a patch to finally fix https://bugs.gentoo.org/814380.
When I opened the bug, it was clear that I intended to only disable
static libtool archive building. Unfortunately, I hadn't done the
necessary due diligence, and it turns out how we detect the
"--enable-static" flag is overly broad and leads to a sizable number of
false positives in EAPI 8.
I have 937 EAPI 8 packages on my machine, and in 0 cases does my patch
now produce static libraries where none were produced before, while it
successfully reduces the annoying QA warnings on packages that don't use
libtool or use the flag for different purposes:
1. packages that have an "off-target"/false-positive flag triggering
the current logic:
dev-libs/nspr-4.35
--enable-static-rtl Use the MSVC static runtime library
sys-fs/lvm2-2.03.16
--enable-static_linkuse this to link the tools to their libraries
2. packages with a real --{dis,en}able-static flag, but with semantics
that don't actually disable building of static-libs:
dev-util/strace-5.19
--enable-static link strace statically
sys-process/htop-3.2.1
--enable-static build a static htop binary [default=no]
media-video/mkvtoolnix-71.1.0
--enable-static make a static build of the applications (no)
--enable-static-qt link to static versions of the Qt library (no)
dev-util/strace has a $(use_enable static) option, and the other two are
default disabled. Hence in all of these cases, the patch does not change
the build output.
3. packages with a --{dis,en}able-static flag that controls building of
static libraries, but with a slightly different --help output that
doesn't trigger the updated glob anymore:
dev-libs/icu-72.1
--enable-static build static libraries default=no
dev-libs/nettle-3.8.1
--disable-staticDo not build any static library
net-print/cups-2.4.2
--enable-static install static libraries
All of these packages pass a $(use_enable static-libs static) option,
hence none of them would be affected by the missing --disable-static.
With this extensive analysis, I believe this patch to be safe.
[gentoo-portage-dev] [PATCH] Make EAPI 8 `--disable-static` logic libtool-specific
* The intention has always been to only target `configure` scripts that use
libtool, not just any script with a `--disable-static*` option.
* libtool has been using the same `configure` format for at least
the past 15 years (going back to libtool 1.5.22):
1. shared and static libraries enabled (the main use case):
--enable-shared[=PKGS] build shared libraries [default=yes]
--enable-static[=PKGS] build static libraries [default=yes]
2. shared libraries enabled and static libraries disabled:
--enable-static[=PKGS] build static libraries [default=no]
--enable-shared[=PKGS] build shared libraries [default=yes]
3. shared libraries disabled and static libraries enabled:
--enable-shared[=PKGS] build shared libraries [default=no]
--enable-static[=PKGS] build static libraries [default=yes]
Bug: https://bugs.gentoo.org/814380
---
bin/phase-helpers.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/bin/phase-helpers.sh b/bin/phase-helpers.sh
index 2217e5a0b..6f691f6ff 100644
--- a/bin/phase-helpers.sh
+++ b/bin/phase-helpers.sh
@@ -655,8 +655,8 @@ econf() {
fi
if ___eapi_econf_passes_--disable-static; then
- if [[ ${conf_help} == *--disable-static* || \
- ${conf_help} ==
*--enable-static* ]]; then
+ if [[ ${conf_help} ==
*--enable-shared\[=PKGS\]* &&
+ ${conf_help} ==
*--enable-static\[=PKGS\]* ]]; then
conf_args+=( --disable-static )
fi
fi
--
2.38.1
Re: [gentoo-portage-dev] bin/isolated-functions.sh USERLAND setting
On 28-07-2022 20:11:24 +0200, Fabian Groffen wrote:
> On 28-07-2022 19:57:50 +0200, Arfrever Frehtes Taifersar Arahesis wrote:
> > But this code starts with 'if [[ -z ${USERLAND} ]]', so it should not
> > be run when USERLAND="GNU".
> >
> > USERLAND="GNU" is set in make.defaults, which is parsed by Portage
> > early (at Python side) and these values are passed by Portage to
> > ebuild.sh subprocess.
>
> Hmmm, that would be nice, features/prefix/make.defaults includes setting
> that.
>
> Makes me still wonder why we do this, is it a fallback for when the
> profiles aren't available (no tree yet) or something?
>
> It exports XARGS, which only used in the ___parallel_xargs function.
> That function in turn is used in 2 places. So all this stuff is done to
> add -r to xargs if xargs isn't BSD xargs. Iow --no-run-if-empty, which
> is cool, but apparently not strictly necessary (fugly warning I guess at
> worst). We're already checking xargs to support --max-procs=, so we
> could in the same go check for support of --no-run-if-empty, and drop
> the whole XARGS thing and declutter the code quite a lot :)
Sorry, I grepped wrong. XARGS is used throughout the code, so it needs
to be initialised in isolated-functions.sh. It could be initialised by
simply probing instead of relying on uname though.
As for USERLAND, emerge-webrsync and delta-webrsync appear to use it,
but they retrieve it from portageq, so I guess USERLAND is not really
used in Portage itself.
Fabian
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] bin/isolated-functions.sh USERLAND setting
On 28-07-2022 19:57:50 +0200, Arfrever Frehtes Taifersar Arahesis wrote:
> 2022-07-28 17:47 UTCに、Fabian Groffen は書いた:
> > bin/isolated-functions.sh does the following bit:
> >
> > if [[ -z ${USERLAND} ]] ; then
> >case $(uname -s) in
> >*BSD|DragonFly)
> >export USERLAND="BSD"
> >;;
> >*)
> >export USERLAND="GNU"
> >;;
> >esac
> > fi
> >
> > (after which it uses USERLAND for a single purpose, to export XARGS to
> > either GNU '[g]xargs -r', or BSD 'xargs')
> >
> > This bit is problematic for Prefix, because Prefix may run on *BSD, but
> > use USERLAND=GNU.
>
> But this code starts with 'if [[ -z ${USERLAND} ]]', so it should not
> be run when USERLAND="GNU".
>
> USERLAND="GNU" is set in make.defaults, which is parsed by Portage
> early (at Python side) and these values are passed by Portage to
> ebuild.sh subprocess.
Hmmm, that would be nice, features/prefix/make.defaults includes setting
that.
Makes me still wonder why we do this, is it a fallback for when the
profiles aren't available (no tree yet) or something?
It exports XARGS, which only used in the ___parallel_xargs function.
That function in turn is used in 2 places. So all this stuff is done to
add -r to xargs if xargs isn't BSD xargs. Iow --no-run-if-empty, which
is cool, but apparently not strictly necessary (fugly warning I guess at
worst). We're already checking xargs to support --max-procs=, so we
could in the same go check for support of --no-run-if-empty, and drop
the whole XARGS thing and declutter the code quite a lot :)
Thanks,
Fabian
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] bin/isolated-functions.sh USERLAND setting
2022-07-28 17:47 UTCに、Fabian Groffen は書いた:
> bin/isolated-functions.sh does the following bit:
>
> if [[ -z ${USERLAND} ]] ; then
>case $(uname -s) in
>*BSD|DragonFly)
>export USERLAND="BSD"
>;;
>*)
>export USERLAND="GNU"
>;;
>esac
> fi
>
> (after which it uses USERLAND for a single purpose, to export XARGS to
> either GNU '[g]xargs -r', or BSD 'xargs')
>
> This bit is problematic for Prefix, because Prefix may run on *BSD, but
> use USERLAND=GNU.
But this code starts with 'if [[ -z ${USERLAND} ]]', so it should not
be run when USERLAND="GNU".
USERLAND="GNU" is set in make.defaults, which is parsed by Portage
early (at Python side) and these values are passed by Portage to
ebuild.sh subprocess.
[gentoo-portage-dev] bin/isolated-functions.sh USERLAND setting
Hi,
bin/isolated-functions.sh does the following bit:
if [[ -z ${USERLAND} ]] ; then
case $(uname -s) in
*BSD|DragonFly)
export USERLAND="BSD"
;;
*)
export USERLAND="GNU"
;;
esac
fi
(after which it uses USERLAND for a single purpose, to export XARGS to
either GNU '[g]xargs -r', or BSD 'xargs')
This bit is problematic for Prefix, because Prefix may run on *BSD, but
use USERLAND=GNU. The problem is theoretical at this point,
occasionally people come in and want to use Prefix on *BSD again.
Now I think Gentoo/BSD is theoretical too. It officially is no longer
maintained[1].
So, question here is, do we want to retain this bit of historical
compatibility work (also check things like gsed wrapper), or can it go,
basically removing a potential problem for Gentoo Prefix?
[1] https://wiki.gentoo.org/wiki/Gentoo_FreeBSD
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] emake: explicitly set SHELL
2022-07-26 07:03 UTCに、Florian Schmaus は書いた:
> But then I wondered if "make SHELL=$BROOT/bin/sh" wouldn't override
> explicitly set SHELL values in Makefiles. Assume a package has
>
> SHELL = /bin/zsh
>
> in one of its Makefiles. Then emake would reset this to 'sh'. Which
> appears like it could cause build issues.
>
> If this is the case, then I am not sure what we can do about it. It
> appears fragile, if not impossible, to ask 'make' which value for SHELL
> it would assume, so that emake could adjust the path. Another option
> could be that affected packages define a variable in their ebuild, e.g.
> EMAKE_SHELL="zsh", which emake could extend with BROOT before passing
> the resulting value as SHELL to make.
If there was such package, it could just override SHELL on emake invocation:
src_compile() {
emake SHELL="${BROOT}/bin/zsh"
# Or:
emake SHELL="$(type -P zsh)"
}
Re: [gentoo-portage-dev] [PATCH] phase-functions: make ED, EROOT read-only
On 25-07-2022 19:43:21 -0400, Mike Gilbert wrote: > On Mon, Jul 25, 2022 at 1:03 PM Fabian Groffen wrote: > > > > bin/phase-functions.sh: make ED and EROOT read-only too > > > > Like D, make ED and EROOT read-only vars. > > Makes sense. https://github.com/gentoo/portage/pull/870 Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] misc-functions: Prefix fixes
On 25-07-2022 19:42:51 -0400, Mike Gilbert wrote: > On Mon, Jul 25, 2022 at 12:47 PM Fabian Groffen wrote: > > > > bin/misc-functions.sh: some Prefix fixes > > > > - ED needs not to exist, whereas D does, so ensure we check for that, > > and create ED if absent, necessary for further checks to succeed > > - use EPREFIX in INSTALL_MASK > > Seems good to me. https://github.com/gentoo/portage/pull/870 -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] world-writable: tune for Prefix
On 25-07-2022 19:39:38 -0400, Mike Gilbert wrote: > On Mon, Jul 25, 2022 at 12:41 PM Fabian Groffen wrote: > > > > bin/install-qa-check.d/90world-writable: include EPREFIX in reports > > > > It is much less confusing and consistent to report full paths including > > the leading EPREFIX. > > Makes sense to me. https://github.com/gentoo/portage/pull/869 thanks -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] multilib-strict: fix for Prefix
On 25-07-2022 19:38:57 -0400, Mike Gilbert wrote:
> On Mon, Jul 25, 2022 at 12:26 PM Fabian Groffen wrote:
> >
> > bin/install-qa-check.d/80multilib-strict: use file/find from Prefix
> >
> > diff --git a/bin/install-qa-check.d/80multilib-strict
> > b/bin/install-qa-check.d/80multilib-strict
> > index afd223250..3db4ecce3 100644
> > --- a/bin/install-qa-check.d/80multilib-strict
> > +++ b/bin/install-qa-check.d/80multilib-strict
> > @@ -1,7 +1,7 @@
> > # Strict multilib directory checks
> > multilib_strict_check() {
> > if has multilib-strict ${FEATURES} && \
> > - [[ -x /usr/bin/file && -x /usr/bin/find ]] && \
> > + [[ -x ${EPREFIX}/usr/bin/file && -x ${EPREFIX}/usr/bin/find ]]
> > && \
> >[[ -n ${MULTILIB_STRICT_DIRS} && -n ${MULTILIB_STRICT_DENY} ]]
> > then
> > rm -f "${T}/multilib-strict.log"
>
> Seems good to me.
https://github.com/gentoo/portage/pull/868
Thanks
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] 80libraries: add support for Darwin targets
On 26-07-2022 04:01:37 +0100, Sam James wrote:
>
>
> > On 26 Jul 2022, at 00:33, Mike Gilbert wrote:
> >
> > On Mon, Jul 25, 2022 at 11:38 AM Fabian Groffen wrote:
> >>
> >> bin/install-qa-check.d/80libraries: support Darwin/Mach-O objects
> >>
> >> Check for dylib on Darwin, so on everything else.
> >>
> >> Signed-off-by: Fabian Groffen
> >>
> >> diff --git a/bin/install-qa-check.d/80libraries
> >> b/bin/install-qa-check.d/80libraries
> >> index 8dc35bb87..a477ec9cb 100644
> >> --- a/bin/install-qa-check.d/80libraries
> >> +++ b/bin/install-qa-check.d/80libraries
> >> @@ -140,7 +140,9 @@ lib_check() {
> >>local abort="no"
> >>local a s
> >>for a in "${ED%/}"/usr/lib*/*.a ; do
> >> - s=${a%.a}.so
> >> + [[ ${CHOST} == *-darwin* ]] \
> >> + && s=${a%.a}.dylib \
> >> + || s=${a%.a}.so
> >
> > I would find this much easier to read if you converted it to an
> > if/else statement instead of chaining && and ||.
>
> Yes, please.
https://github.com/gentoo/portage/pull/867
Thanks
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] doins: fix D check, add EPREFIX check
On 25-07-2022 19:29:35 -0400, Mike Gilbert wrote: > Could you please create a PR at https://github.com/gentoo/portage so > that the CI system can test the changes for this patch series? https://github.com/gentoo/portage/pull/866 Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] doins: fix D check, add EPREFIX check
On 25-07-2022 19:50:41 +0200, Ulrich Mueller wrote:
> > On Mon, 25 Jul 2022, Fabian Groffen wrote:
>
> > @@ -50,6 +51,16 @@ if [[ ${_E_INSDESTTREE_#${ED}} != "${_E_INSDESTTREE_}"
> > ]]; then
> > __helpers_die "${helper} used with \${D} or \${ED}"
> > exit 1
> > fi
> > +if [[ -n ${EPREFIX} && \
> > + ${_E_INSDESTTREE_#${EPREFIX}} != "${_E_INSDESTTREE_}" ]];
>
> The semicolon is redundant.
removed, thanks
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] emake: explicitly set SHELL
On 26-07-2022 09:03:18 +0200, Florian Schmaus wrote:
> On 26.07.22 05:00, Sam James wrote:
> >> On 25 Jul 2022, at 16:28, Fabian Groffen wrote:
> >>
> >> bin/ebuild-helpers/emake: force SHELL to be set
> >>
[snip]
> >>
> >> diff --git a/bin/ebuild-helpers/emake b/bin/ebuild-helpers/emake
> >> index 60718a2e4..21da85845 100755
> >> --- a/bin/ebuild-helpers/emake
> >> +++ b/bin/ebuild-helpers/emake
> >> @@ -12,7 +12,7 @@
> >> source "${PORTAGE_BIN_PATH}"/isolated-functions.sh || exit 1
> >>
> >> cmd=(
> >> - ${MAKE:-make} ${MAKEOPTS} "$@" ${EXTRA_EMAKE}
> >> + ${MAKE:-make} SHELL="${BASH:-/bin/bash}" ${MAKEOPTS} "$@" ${EXTRA_EMAKE}
> >> )
> >>
> >> if [[ ${PORTAGE_QUIET} != 1 ]] ; then
> >>
> >
> > I don't think I agree with this as it is. Why not just ${EPREFIX}/bin/sh to
> > avoid using
> > an ancient host sh?
>
> I was about to write the same (also using EPREFIX, but EBROOT seems what
> you want, as you figured).
>
> But then I wondered if "make SHELL=$BROOT/bin/sh" wouldn't override
> explicitly set SHELL values in Makefiles. Assume a package has
>
> SHELL = /bin/zsh
>
> in one of its Makefiles. Then emake would reset this to 'sh'. Which
> appears like it could cause build issues.
>
> If this is the case, then I am not sure what we can do about it. It
> appears fragile, if not impossible, to ask 'make' which value for SHELL
> it would assume, so that emake could adjust the path. Another option
> could be that affected packages define a variable in their ebuild, e.g.
> EMAKE_SHELL="zsh", which emake could extend with BROOT before passing
> the resulting value as SHELL to make.
So, I can also envision we drop this patch, and I see if I can patch
make(1) to use $EPREFIX/bin/sh instead of /bin/sh by default. Not sure,
but this would retain the behaviour Portage is doing now for non-Prefix,
and would get the behaviour we want in Prefix.
On an alternative note, there is CONFIG_SHELL (used for setting which shell
to use with configure), which I think in many cases bleeds through to
make, but should there be a MAKE_SHELL perhaps as well? Then the
default would be pretty much ok.
(We never ran into any problems forcing SHELL to bash in Prefix, but
perhaps that's not a representative test for the whole of Gentoo.)
Thanks,
Fabian
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] emake: explicitly set SHELL
On 26.07.22 05:00, Sam James wrote:
On 25 Jul 2022, at 16:28, Fabian Groffen wrote:
bin/ebuild-helpers/emake: force SHELL to be set
On Prefix systems /bin/sh can be anything, including very ancient. So
ensure we're running with bash, since that's what Gentoo Linux is
expecting /bin/sh to be (by default, at least).
Provide a fallback for the (near impossible) case that we use a bash
that doesn't set BASH, or when we don't use bash at all. This is not
expected, though, as we explicitly require bash throughout all Portage,
so we don't really care about using a non-Prefixed one, for this really
shouldn't happen.
Signed-off-by: Fabian Groffen
diff --git a/bin/ebuild-helpers/emake b/bin/ebuild-helpers/emake
index 60718a2e4..21da85845 100755
--- a/bin/ebuild-helpers/emake
+++ b/bin/ebuild-helpers/emake
@@ -12,7 +12,7 @@
source "${PORTAGE_BIN_PATH}"/isolated-functions.sh || exit 1
cmd=(
- ${MAKE:-make} ${MAKEOPTS} "$@" ${EXTRA_EMAKE}
+ ${MAKE:-make} SHELL="${BASH:-/bin/bash}" ${MAKEOPTS} "$@" ${EXTRA_EMAKE}
)
if [[ ${PORTAGE_QUIET} != 1 ]] ; then
I don't think I agree with this as it is. Why not just ${EPREFIX}/bin/sh to
avoid using
an ancient host sh?
I was about to write the same (also using EPREFIX, but EBROOT seems what
you want, as you figured).
But then I wondered if "make SHELL=$BROOT/bin/sh" wouldn't override
explicitly set SHELL values in Makefiles. Assume a package has
SHELL = /bin/zsh
in one of its Makefiles. Then emake would reset this to 'sh'. Which
appears like it could cause build issues.
If this is the case, then I am not sure what we can do about it. It
appears fragile, if not impossible, to ask 'make' which value for SHELL
it would assume, so that emake could adjust the path. Another option
could be that affected packages define a variable in their ebuild, e.g.
EMAKE_SHELL="zsh", which emake could extend with BROOT before passing
the resulting value as SHELL to make.
- Flow
Re: [gentoo-portage-dev] [PATCH] emake: explicitly set SHELL
> On 26 Jul 2022, at 04:00, Sam James wrote:
>
>
>
>> On 25 Jul 2022, at 16:28, Fabian Groffen wrote:
>>
>> bin/ebuild-helpers/emake: force SHELL to be set
>>
>> On Prefix systems /bin/sh can be anything, including very ancient. So
>> ensure we're running with bash, since that's what Gentoo Linux is
>> expecting /bin/sh to be (by default, at least).
>>
>> Provide a fallback for the (near impossible) case that we use a bash
>> that doesn't set BASH, or when we don't use bash at all. This is not
>> expected, though, as we explicitly require bash throughout all Portage,
>> so we don't really care about using a non-Prefixed one, for this really
>> shouldn't happen.
>>
>> Signed-off-by: Fabian Groffen
>>
>> diff --git a/bin/ebuild-helpers/emake b/bin/ebuild-helpers/emake
>> index 60718a2e4..21da85845 100755
>> --- a/bin/ebuild-helpers/emake
>> +++ b/bin/ebuild-helpers/emake
>> @@ -12,7 +12,7 @@
>> source "${PORTAGE_BIN_PATH}"/isolated-functions.sh || exit 1
>>
>> cmd=(
>> -${MAKE:-make} ${MAKEOPTS} "$@" ${EXTRA_EMAKE}
>> +${MAKE:-make} SHELL="${BASH:-/bin/bash}" ${MAKEOPTS} "$@" ${EXTRA_EMAKE}
>> )
>>
>> if [[ ${PORTAGE_QUIET} != 1 ]] ; then
>>
>
> I don't think I agree with this as it is. Why not just ${EPREFIX}/bin/sh to
> avoid using
> an ancient host sh?
>
Sorry, ${BROOT}, I guess.
> I might use Bash for Portage but my /bin/sh is dash usually.
>
>> --
>> Fabian Groffen
>> Gentoo on a different level
signature.asc
Description: Message signed with OpenPGP
Re: [gentoo-portage-dev] [PATCH] 80libraries: add support for Darwin targets
> On 26 Jul 2022, at 00:33, Mike Gilbert wrote:
>
> On Mon, Jul 25, 2022 at 11:38 AM Fabian Groffen wrote:
>>
>> bin/install-qa-check.d/80libraries: support Darwin/Mach-O objects
>>
>> Check for dylib on Darwin, so on everything else.
>>
>> Signed-off-by: Fabian Groffen
>>
>> diff --git a/bin/install-qa-check.d/80libraries
>> b/bin/install-qa-check.d/80libraries
>> index 8dc35bb87..a477ec9cb 100644
>> --- a/bin/install-qa-check.d/80libraries
>> +++ b/bin/install-qa-check.d/80libraries
>> @@ -140,7 +140,9 @@ lib_check() {
>>local abort="no"
>>local a s
>>for a in "${ED%/}"/usr/lib*/*.a ; do
>> - s=${a%.a}.so
>> + [[ ${CHOST} == *-darwin* ]] \
>> + && s=${a%.a}.dylib \
>> + || s=${a%.a}.so
>
> I would find this much easier to read if you converted it to an
> if/else statement instead of chaining && and ||.
Yes, please.
signature.asc
Description: Message signed with OpenPGP
Re: [gentoo-portage-dev] [PATCH] emake: explicitly set SHELL
> On 25 Jul 2022, at 16:28, Fabian Groffen wrote:
>
> bin/ebuild-helpers/emake: force SHELL to be set
>
> On Prefix systems /bin/sh can be anything, including very ancient. So
> ensure we're running with bash, since that's what Gentoo Linux is
> expecting /bin/sh to be (by default, at least).
>
> Provide a fallback for the (near impossible) case that we use a bash
> that doesn't set BASH, or when we don't use bash at all. This is not
> expected, though, as we explicitly require bash throughout all Portage,
> so we don't really care about using a non-Prefixed one, for this really
> shouldn't happen.
>
> Signed-off-by: Fabian Groffen
>
> diff --git a/bin/ebuild-helpers/emake b/bin/ebuild-helpers/emake
> index 60718a2e4..21da85845 100755
> --- a/bin/ebuild-helpers/emake
> +++ b/bin/ebuild-helpers/emake
> @@ -12,7 +12,7 @@
> source "${PORTAGE_BIN_PATH}"/isolated-functions.sh || exit 1
>
> cmd=(
> - ${MAKE:-make} ${MAKEOPTS} "$@" ${EXTRA_EMAKE}
> + ${MAKE:-make} SHELL="${BASH:-/bin/bash}" ${MAKEOPTS} "$@" ${EXTRA_EMAKE}
> )
>
> if [[ ${PORTAGE_QUIET} != 1 ]] ; then
>
I don't think I agree with this as it is. Why not just ${EPREFIX}/bin/sh to
avoid using
an ancient host sh?
I might use Bash for Portage but my /bin/sh is dash usually.
> --
> Fabian Groffen
> Gentoo on a different level
signature.asc
Description: Message signed with OpenPGP
Re: [gentoo-portage-dev] [PATCH] phase-functions: make ED, EROOT read-only
On Mon, Jul 25, 2022 at 1:03 PM Fabian Groffen wrote: > > bin/phase-functions.sh: make ED and EROOT read-only too > > Like D, make ED and EROOT read-only vars. Makes sense.
Re: [gentoo-portage-dev] [PATCH] misc-functions: Prefix fixes
On Mon, Jul 25, 2022 at 12:47 PM Fabian Groffen wrote: > > bin/misc-functions.sh: some Prefix fixes > > - ED needs not to exist, whereas D does, so ensure we check for that, > and create ED if absent, necessary for further checks to succeed > - use EPREFIX in INSTALL_MASK Seems good to me.
Re: [gentoo-portage-dev] [PATCH] world-writable: tune for Prefix
On Mon, Jul 25, 2022 at 12:41 PM Fabian Groffen wrote: > > bin/install-qa-check.d/90world-writable: include EPREFIX in reports > > It is much less confusing and consistent to report full paths including > the leading EPREFIX. Makes sense to me.
Re: [gentoo-portage-dev] [PATCH] multilib-strict: fix for Prefix
On Mon, Jul 25, 2022 at 12:26 PM Fabian Groffen wrote:
>
> bin/install-qa-check.d/80multilib-strict: use file/find from Prefix
>
> diff --git a/bin/install-qa-check.d/80multilib-strict
> b/bin/install-qa-check.d/80multilib-strict
> index afd223250..3db4ecce3 100644
> --- a/bin/install-qa-check.d/80multilib-strict
> +++ b/bin/install-qa-check.d/80multilib-strict
> @@ -1,7 +1,7 @@
> # Strict multilib directory checks
> multilib_strict_check() {
> if has multilib-strict ${FEATURES} && \
> - [[ -x /usr/bin/file && -x /usr/bin/find ]] && \
> + [[ -x ${EPREFIX}/usr/bin/file && -x ${EPREFIX}/usr/bin/find ]] && \
>[[ -n ${MULTILIB_STRICT_DIRS} && -n ${MULTILIB_STRICT_DENY} ]]
> then
> rm -f "${T}/multilib-strict.log"
Seems good to me.
Re: [gentoo-portage-dev] [PATCH] emake: explicitly set SHELL
On Mon, Jul 25, 2022 at 11:28 AM Fabian Groffen wrote: > > bin/ebuild-helpers/emake: force SHELL to be set > > On Prefix systems /bin/sh can be anything, including very ancient. So > ensure we're running with bash, since that's what Gentoo Linux is > expecting /bin/sh to be (by default, at least). > > Provide a fallback for the (near impossible) case that we use a bash > that doesn't set BASH, or when we don't use bash at all. This is not > expected, though, as we explicitly require bash throughout all Portage, > so we don't really care about using a non-Prefixed one, for this really > shouldn't happen. I'm a little on the fence about this: in theory, Makefiles should use POSIX-compatible shell commands unless the author explicitly chooses to use bash. I guess we can get away with this since ebuilds always require bash anyway.
Re: [gentoo-portage-dev] [PATCH] 80libraries: add support for Darwin targets
On Mon, Jul 25, 2022 at 11:38 AM Fabian Groffen wrote:
>
> bin/install-qa-check.d/80libraries: support Darwin/Mach-O objects
>
> Check for dylib on Darwin, so on everything else.
>
> Signed-off-by: Fabian Groffen
>
> diff --git a/bin/install-qa-check.d/80libraries
> b/bin/install-qa-check.d/80libraries
> index 8dc35bb87..a477ec9cb 100644
> --- a/bin/install-qa-check.d/80libraries
> +++ b/bin/install-qa-check.d/80libraries
> @@ -140,7 +140,9 @@ lib_check() {
> local abort="no"
> local a s
> for a in "${ED%/}"/usr/lib*/*.a ; do
> - s=${a%.a}.so
> + [[ ${CHOST} == *-darwin* ]] \
> + && s=${a%.a}.dylib \
> + || s=${a%.a}.so
I would find this much easier to read if you converted it to an
if/else statement instead of chaining && and ||.
Re: [gentoo-portage-dev] [PATCH] doins: fix D check, add EPREFIX check
Could you please create a PR at https://github.com/gentoo/portage so that the CI system can test the changes for this patch series?
Re: [gentoo-portage-dev] [PATCH] doins: fix D check, add EPREFIX check
> On Mon, 25 Jul 2022, Fabian Groffen wrote:
> @@ -50,6 +51,16 @@ if [[ ${_E_INSDESTTREE_#${ED}} != "${_E_INSDESTTREE_}" ]];
> then
> __helpers_die "${helper} used with \${D} or \${ED}"
> exit 1
> fi
> +if [[ -n ${EPREFIX} && \
> + ${_E_INSDESTTREE_#${EPREFIX}} != "${_E_INSDESTTREE_}" ]];
The semicolon is redundant.
> +then
> + __vecho "---" 1>&2
> + __vecho "You should not use \${EPREFIX} with helpers." 1>&2
> + __vecho " --> ${_E_INSDESTTREE_}" 1>&2
> + __vecho "---" 1>&2
> + __helpers_die "${helper} used with \${EPREFIX}"
> + exit 1
> +fi
signature.asc
Description: PGP signature
[gentoo-portage-dev] [PATCH] phase-functions: make ED, EROOT read-only
bin/phase-functions.sh: make ED and EROOT read-only too Like D, make ED and EROOT read-only vars. Signed-off-by: Fabian Groffen diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh index ccf7eeea7..212b19fc1 100644 --- a/bin/phase-functions.sh +++ b/bin/phase-functions.sh @@ -12,7 +12,7 @@ PORTAGE_READONLY_METADATA="BDEPEND DEFINED_PHASES DEPEND DESCRIPTION PDEPEND RDEPEND REPOSITORY RESTRICT SLOT SRC_URI" PORTAGE_READONLY_VARS="D EBUILD EBUILD_PHASE EBUILD_PHASE_FUNC \ - EBUILD_SH_ARGS EMERGE_FROM FILESDIR MERGE_TYPE \ + EBUILD_SH_ARGS ED EMERGE_FROM EROOT FILESDIR MERGE_TYPE \ PM_EBUILD_HOOK_DIR \ PORTAGE_ACTUAL_DISTDIR PORTAGE_ARCHLIST PORTAGE_BASHRC \ PORTAGE_BINPKG_FILE PORTAGE_BINPKG_TAR_OPTS PORTAGE_BINPKG_TMPFILE \ -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
[gentoo-portage-dev] [PATCH] misc-functions: Prefix fixes
bin/misc-functions.sh: some Prefix fixes
- ED needs not to exist, whereas D does, so ensure we check for that,
and create ED if absent, necessary for further checks to succeed
- use EPREFIX in INSTALL_MASK
Signed-off-by: Fabian Groffen
diff --git a/bin/misc-functions.sh b/bin/misc-functions.sh
index c8bac08e7..8fcc23588 100755
--- a/bin/misc-functions.sh
+++ b/bin/misc-functions.sh
@@ -19,6 +19,8 @@ source "${PORTAGE_BIN_PATH}/ebuild.sh" || exit 1
install_symlink_html_docs() {
if ! ___eapi_has_prefix_variables; then
local ED=${D}
+ else
+ [[ ! -d ${ED} && -d ${D} ]] && dodir /
fi
cd "${ED}" || die "cd failed"
# Symlink the html documentation (if DOC_SYMLINKS_DIR is set in
make.conf)
@@ -83,7 +87,7 @@ install_qa_check() {
local EPREFIX= ED=${D}
fi
- cd "${ED}" || die "cd failed"
+ cd "${D}" || die "cd failed"
# Collect the paths for QA checks, highest prio first.
paths=(
@@ -367,7 +718,7 @@ preinst_mask() {
local f x
for f in man info doc; do
if has no${f} ${FEATURES}; then
- INSTALL_MASK+=" /usr/share/${f}"
+ INSTALL_MASK+=" ${EPREFIX}/usr/share/${f}"
fi
done
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
[gentoo-portage-dev] [PATCH] world-writable: tune for Prefix
bin/install-qa-check.d/90world-writable: include EPREFIX in reports
It is much less confusing and consistent to report full paths including
the leading EPREFIX.
Signed-off-by: Fabian Groffen
diff --git a/bin/install-qa-check.d/90world-writable
b/bin/install-qa-check.d/90world-writable
index 820683bd6..90b961a86 100644
--- a/bin/install-qa-check.d/90world-writable
+++ b/bin/install-qa-check.d/90world-writable
@@ -2,7 +2,7 @@
world_writable_check() {
# Now we look for all world writable files.
- local unsafe_files=$(find "${ED}" -type f -perm -2 | sed -e
"s:^${ED}:/:")
+ local unsafe_files=$(find "${ED}" -type f -perm -2 | sed -e
"s:^${D}:/:")
local OLDIFS x prev_shopts=$-
OLDIFS=$IFS
@@ -19,7 +21,7 @@ world_writable_check() {
eqawarn
fi
- local unsafe_files=$(find "${ED}" -type f '(' -perm -2002 -o -perm
-4002 ')' | sed -e "s:^${ED}:/:")
+ local unsafe_files=$(find "${ED}" -type f '(' -perm -2002 -o -perm
-4002 ')' | sed -e "s:^${D}:/:")
if [[ -n ${unsafe_files} ]] ; then
eqawarn "QA Notice: Unsafe files detected (set*id and world
writable)"
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
[gentoo-portage-dev] [PATCH] multilib-strict: fix for Prefix
bin/install-qa-check.d/80multilib-strict: use file/find from Prefix
diff --git a/bin/install-qa-check.d/80multilib-strict
b/bin/install-qa-check.d/80multilib-strict
index afd223250..3db4ecce3 100644
--- a/bin/install-qa-check.d/80multilib-strict
+++ b/bin/install-qa-check.d/80multilib-strict
@@ -1,7 +1,7 @@
# Strict multilib directory checks
multilib_strict_check() {
if has multilib-strict ${FEATURES} && \
- [[ -x /usr/bin/file && -x /usr/bin/find ]] && \
+ [[ -x ${EPREFIX}/usr/bin/file && -x ${EPREFIX}/usr/bin/find ]] && \
[[ -n ${MULTILIB_STRICT_DIRS} && -n ${MULTILIB_STRICT_DENY} ]]
then
rm -f "${T}/multilib-strict.log"
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
[gentoo-portage-dev] [PATCH] 80libraries: add support for Darwin targets
bin/install-qa-check.d/80libraries: support Darwin/Mach-O objects
Check for dylib on Darwin, so on everything else.
Signed-off-by: Fabian Groffen
diff --git a/bin/install-qa-check.d/80libraries
b/bin/install-qa-check.d/80libraries
index 8dc35bb87..a477ec9cb 100644
--- a/bin/install-qa-check.d/80libraries
+++ b/bin/install-qa-check.d/80libraries
@@ -140,7 +140,9 @@ lib_check() {
local abort="no"
local a s
for a in "${ED%/}"/usr/lib*/*.a ; do
- s=${a%.a}.so
+ [[ ${CHOST} == *-darwin* ]] \
+ && s=${a%.a}.dylib \
+ || s=${a%.a}.so
if [[ ! -e ${s} ]] ; then
s=${s%usr/*}${s##*/usr/}
if [[ -e ${s} ]] ; then
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
[gentoo-portage-dev] [PATCH] emake: explicitly set SHELL
bin/ebuild-helpers/emake: force SHELL to be set
On Prefix systems /bin/sh can be anything, including very ancient. So
ensure we're running with bash, since that's what Gentoo Linux is
expecting /bin/sh to be (by default, at least).
Provide a fallback for the (near impossible) case that we use a bash
that doesn't set BASH, or when we don't use bash at all. This is not
expected, though, as we explicitly require bash throughout all Portage,
so we don't really care about using a non-Prefixed one, for this really
shouldn't happen.
Signed-off-by: Fabian Groffen
diff --git a/bin/ebuild-helpers/emake b/bin/ebuild-helpers/emake
index 60718a2e4..21da85845 100755
--- a/bin/ebuild-helpers/emake
+++ b/bin/ebuild-helpers/emake
@@ -12,7 +12,7 @@
source "${PORTAGE_BIN_PATH}"/isolated-functions.sh || exit 1
cmd=(
- ${MAKE:-make} ${MAKEOPTS} "$@" ${EXTRA_EMAKE}
+ ${MAKE:-make} SHELL="${BASH:-/bin/bash}" ${MAKEOPTS} "$@" ${EXTRA_EMAKE}
)
if [[ ${PORTAGE_QUIET} != 1 ]] ; then
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] usage of /bin/bash in shebangs
On 25-07-2022 08:09:02 -0700, Zac Medico wrote: > On 7/24/22 23:17, Fabian Groffen wrote: > > On 24-07-2022 13:58:31 -0700, Zac Medico wrote: > >> On 7/24/22 12:29, Fabian Groffen wrote: > >>> Hi, > >>> > >>> Quick question, I noticed that portage uses /bin/bash hardcoded in > >>> shebang of scripts, while it uses /usr/bin/env python for python > >>> executable files. > >>> > >>> Is there anything against using /usr/bin/env bash for shell scripts? > >>> Changing this would help for Prefix. > >> > >> > >> I can't think of any reason not to do this. > > > > Do you want to see a patch, or is it OK to push this myself? > > It's OK to just push this yourself. Thanks! Pushed, thanks, b716e3591 -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
[gentoo-portage-dev] [PATCH] doins: fix D check, add EPREFIX check
bin/ebuild-helpers/doins: fix D check, add EPREFIX check
ED = D/EPREFIX, so checking for ED includes EPREFIX, which when this is
absent fails to check for D. Simply check for D instead, which catches
both the case for D and ED.
Add check for usage of EPREFIX, like for using D with helpers.
Signed-off-by: Fabian Groffen
diff --git a/bin/ebuild-helpers/doins b/bin/ebuild-helpers/doins
index 24fe48121..4315a038f 100755
--- a/bin/ebuild-helpers/doins
+++ b/bin/ebuild-helpers/doins
@@ -42,7 +42,7 @@ if ! ___eapi_has_prefix_variables; then
export ED="${D}"
fi
-if [[ ${_E_INSDESTTREE_#${ED}} != "${_E_INSDESTTREE_}" ]]; then
+if [[ ${_E_INSDESTTREE_#${D}} != "${_E_INSDESTTREE_}" ]]; then
__vecho "---" 1>&2
__vecho "You should not use \${D} or \${ED} with helpers." 1>&2
__vecho " --> ${_E_INSDESTTREE_}" 1>&2
@@ -50,6 +51,16 @@ if [[ ${_E_INSDESTTREE_#${ED}} != "${_E_INSDESTTREE_}" ]];
then
__helpers_die "${helper} used with \${D} or \${ED}"
exit 1
fi
+if [[ -n ${EPREFIX} && \
+ ${_E_INSDESTTREE_#${EPREFIX}} != "${_E_INSDESTTREE_}" ]];
+then
+ __vecho "---" 1>&2
+ __vecho "You should not use \${EPREFIX} with helpers." 1>&2
+ __vecho " --> ${_E_INSDESTTREE_}" 1>&2
+ __vecho "---" 1>&2
+ __helpers_die "${helper} used with \${EPREFIX}"
+ exit 1
+fi
if ___eapi_doins_and_newins_preserve_symlinks; then
DOINS_ARGS+=( --preserve_symlinks )
--
Fabian Groffen
Gentoo on a different level
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] usage of /bin/bash in shebangs
On 7/24/22 23:17, Fabian Groffen wrote: On 24-07-2022 13:58:31 -0700, Zac Medico wrote: On 7/24/22 12:29, Fabian Groffen wrote: Hi, Quick question, I noticed that portage uses /bin/bash hardcoded in shebang of scripts, while it uses /usr/bin/env python for python executable files. Is there anything against using /usr/bin/env bash for shell scripts? Changing this would help for Prefix. I can't think of any reason not to do this. Do you want to see a patch, or is it OK to push this myself? It's OK to just push this yourself. Thanks! -- Thanks, Zac OpenPGP_signature Description: OpenPGP digital signature
Re: [gentoo-portage-dev] usage of /bin/bash in shebangs
On 24-07-2022 13:58:31 -0700, Zac Medico wrote: > On 7/24/22 12:29, Fabian Groffen wrote: > > Hi, > > > > Quick question, I noticed that portage uses /bin/bash hardcoded in > > shebang of scripts, while it uses /usr/bin/env python for python > > executable files. > > > > Is there anything against using /usr/bin/env bash for shell scripts? > > Changing this would help for Prefix. > > > I can't think of any reason not to do this. Do you want to see a patch, or is it OK to push this myself? Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] usage of /bin/bash in shebangs
On 7/24/22 12:29, Fabian Groffen wrote: Hi, Quick question, I noticed that portage uses /bin/bash hardcoded in shebang of scripts, while it uses /usr/bin/env python for python executable files. Is there anything against using /usr/bin/env bash for shell scripts? Changing this would help for Prefix. I can't think of any reason not to do this. -- Thanks, Zac OpenPGP_signature Description: OpenPGP digital signature
[gentoo-portage-dev] usage of /bin/bash in shebangs
Hi, Quick question, I noticed that portage uses /bin/bash hardcoded in shebang of scripts, while it uses /usr/bin/env python for python executable files. Is there anything against using /usr/bin/env bash for shell scripts? Changing this would help for Prefix. Thanks, Fabian -- Fabian Groffen Gentoo on a different level signature.asc Description: PGP signature
Re: [gentoo-portage-dev] Normaliser function for distfiles
On Tue, 17 May 2022 16:35:11 +0200, Michał Górny wrote: > This would require a new EAPI. We don't really want more Portage- > specific hacks that are going to break for everyone not using Portage or > the very specific Portage version. > > I'm not saying that it's not doable but I'm not convinced the added > complexity is really worth the effort, especially given that this looks > like a very special corner case. In the end, fixing Melpa is the right > thing to do. > > For a start, you'd have to ensure that the "normalizer script" (or its > dependencies, if you put it in the repo) is available at the time of > fetching. This pretty much goes back to the problem of "fetch > dependencies", and requires a major design change in Portage that > reduces separation between fetching and installing that we have now. > I mean, right now Portage pretty much assumes that you can do > a `--fetchonly` with no extra packages necessary. > > The "normalizer" wouldn't be trivial either. In the end, we're talking > about getting 100% consistent results on all platforms, over > a reasonably long timeframe. Oh, what a chain of complexity. Yeah, I'll go try fixing melpa first. Markus -- For instance, on the planet Earth, man had always assumed that he was more intelligent than dolphins because he had achieved so much---the wheel, New York, wars and so on---while all the dolphins had ever done was muck about in the water having a good time. But conversely, the dolphins had always believed that they were far more intelligent than man---for precisely the same reasons. (Douglas Adams, The Hitchhikers Guide to the Galaxy.)
Re: [gentoo-portage-dev] Normaliser function for distfiles
On Mon, 2022-05-16 at 19:37 +0200, Markus Walter wrote: > Hello all, > > is it possible to do the following: after fetching a distfile portage runs > an external normaliser program specified in an ebuild before checking the > hash? > > My use case is the following: I would like to improve the gs-elpa program > and provide a precomputed overlay for melpa. However the melpa distfiles are > rebuilt everyday and cause checksum failures. However the only thing > changing are the timestamps. Hence if a normaliser program could simply set > all timestamps to some predefined value (say 1.1.1970) then this problem > should vanish. > This would require a new EAPI. We don't really want more Portage- specific hacks that are going to break for everyone not using Portage or the very specific Portage version. I'm not saying that it's not doable but I'm not convinced the added complexity is really worth the effort, especially given that this looks like a very special corner case. In the end, fixing Melpa is the right thing to do. For a start, you'd have to ensure that the "normalizer script" (or its dependencies, if you put it in the repo) is available at the time of fetching. This pretty much goes back to the problem of "fetch dependencies", and requires a major design change in Portage that reduces separation between fetching and installing that we have now. I mean, right now Portage pretty much assumes that you can do a `--fetchonly` with no extra packages necessary. The "normalizer" wouldn't be trivial either. In the end, we're talking about getting 100% consistent results on all platforms, over a reasonably long timeframe. -- Best regards, Michał Górny
Re: [gentoo-portage-dev] Normaliser function for distfiles
On Tue, May 17, 2022 at 8:32 AM Markus Walter wrote: > > On Tue, 17 May 2022 14:14:57 +0200, > Rich Freeman wrote: > > > > On Mon, May 16, 2022 at 1:37 PM Markus Walter > > wrote: > > > > > > My use case is the following: I would like to improve the gs-elpa program > > > and provide a precomputed overlay for melpa. However the melpa distfiles > > > are > > > rebuilt everyday and cause checksum failures. However the only thing > > > changing are the timestamps. Hence if a normaliser program could simply > > > set > > > all timestamps to some predefined value (say 1.1.1970) then this problem > > > should vanish. > > > > > > > Wouldn't a simpler solution be to just have an ebuild setting that > > tells the package manager to not check the timestamp? > > The timestamps are inside archive files thus changing the overall file > hash. This happens during distfile download, where some more sophisticated > replace all timestamps function would be necessary than just ignoring one > timestamp. Ah, apologies. Totally missed that. Yeah, obviously if the timestamps INSIDE the archive are changing your only solution is to edit the contents. I thought it was just the external timestamp changing (I'm not sure if portage even checks that, or if wget changes the mtime on files it downloads). -- Rich
Re: [gentoo-portage-dev] Normaliser function for distfiles
On Tue, 17 May 2022 14:14:57 +0200, Rich Freeman wrote: > > On Mon, May 16, 2022 at 1:37 PM Markus Walter > wrote: > > > > My use case is the following: I would like to improve the gs-elpa program > > and provide a precomputed overlay for melpa. However the melpa distfiles are > > rebuilt everyday and cause checksum failures. However the only thing > > changing are the timestamps. Hence if a normaliser program could simply set > > all timestamps to some predefined value (say 1.1.1970) then this problem > > should vanish. > > > > Wouldn't a simpler solution be to just have an ebuild setting that > tells the package manager to not check the timestamp? The timestamps are inside archive files thus changing the overall file hash. This happens during distfile download, where some more sophisticated replace all timestamps function would be necessary than just ignoring one timestamp. Markus -- For instance, on the planet Earth, man had always assumed that he was more intelligent than dolphins because he had achieved so much---the wheel, New York, wars and so on---while all the dolphins had ever done was muck about in the water having a good time. But conversely, the dolphins had always believed that they were far more intelligent than man---for precisely the same reasons. (Douglas Adams, The Hitchhikers Guide to the Galaxy.)
Re: [gentoo-portage-dev] Normaliser function for distfiles
On Mon, May 16, 2022 at 1:37 PM Markus Walter wrote: > > My use case is the following: I would like to improve the gs-elpa program > and provide a precomputed overlay for melpa. However the melpa distfiles are > rebuilt everyday and cause checksum failures. However the only thing > changing are the timestamps. Hence if a normaliser program could simply set > all timestamps to some predefined value (say 1.1.1970) then this problem > should vanish. > Wouldn't a simpler solution be to just have an ebuild setting that tells the package manager to not check the timestamp? -- Rich
Re: [gentoo-portage-dev] Normaliser function for distfiles
On Tue, 17 May 2022 05:39:27 +0200, Robin H. Johnson wrote: > I don't know what 'gs-elpa' & 'Melpa' are, but maybe talking to upstream > would be good here, and improving that behavior. That's actually not too bad a plan. Thanks for this somewhat obvious suggestion, which I however failed to see. I'll try that. Although reproducibility is not a tenet there, they may take a patch which improves the situation. Melpa is the largest (not totally sure) repository for Emacs add-ons. And gs-elpa is the g-sorcery integration for Gentoo. Markus -- For instance, on the planet Earth, man had always assumed that he was more intelligent than dolphins because he had achieved so much---the wheel, New York, wars and so on---while all the dolphins had ever done was muck about in the water having a good time. But conversely, the dolphins had always believed that they were far more intelligent than man---for precisely the same reasons. (Douglas Adams, The Hitchhikers Guide to the Galaxy.)
Re: [gentoo-portage-dev] Normaliser function for distfiles
On Tue, 17 May 2022 05:21:02 +0200, Zac Medico wrote: > The only usable hook that we currently have for this is FETCHCOMMAND and > RESUMCOMMAND in make.conf. You can replace them with a script that does the > normal thing and then sets the timestamp. The default values are found in > /usr/share/portage/config/make.globals. Thanks for pointing this out, but I fear this will not easily solve the problem for new users. Markus -- For instance, on the planet Earth, man had always assumed that he was more intelligent than dolphins because he had achieved so much---the wheel, New York, wars and so on---while all the dolphins had ever done was muck about in the water having a good time. But conversely, the dolphins had always believed that they were far more intelligent than man---for precisely the same reasons. (Douglas Adams, The Hitchhikers Guide to the Galaxy.)
Re: [gentoo-portage-dev] Normaliser function for distfiles
On Mon, May 16, 2022 at 07:37:40PM +0200, Markus Walter wrote: > Hello all, > > is it possible to do the following: after fetching a distfile portage runs > an external normaliser program specified in an ebuild before checking the > hash? > > My use case is the following: I would like to improve the gs-elpa program > and provide a precomputed overlay for melpa. However the melpa distfiles are > rebuilt everyday and cause checksum failures. However the only thing > changing are the timestamps. Hence if a normaliser program could simply set > all timestamps to some predefined value (say 1.1.1970) then this problem > should vanish. I don't know what 'gs-elpa' & 'Melpa' are, but maybe talking to upstream would be good here, and improving that behavior. If the file contents or non-timestamp metadata change, absolutely the timestamps should change. But otherwise, the timestamp should NOT change. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: PGP signature
Re: [gentoo-portage-dev] Normaliser function for distfiles
On 5/16/22 10:37, Markus Walter wrote: Hello all, is it possible to do the following: after fetching a distfile portage runs an external normaliser program specified in an ebuild before checking the hash? My use case is the following: I would like to improve the gs-elpa program and provide a precomputed overlay for melpa. However the melpa distfiles are rebuilt everyday and cause checksum failures. However the only thing changing are the timestamps. Hence if a normaliser program could simply set all timestamps to some predefined value (say 1.1.1970) then this problem should vanish. Thanks in advance Markus The only usable hook that we currently have for this is FETCHCOMMAND and RESUMCOMMAND in make.conf. You can replace them with a script that does the normal thing and then sets the timestamp. The default values are found in /usr/share/portage/config/make.globals. -- Thanks, Zac OpenPGP_signature Description: OpenPGP digital signature
[gentoo-portage-dev] Normaliser function for distfiles
Hello all, is it possible to do the following: after fetching a distfile portage runs an external normaliser program specified in an ebuild before checking the hash? My use case is the following: I would like to improve the gs-elpa program and provide a precomputed overlay for melpa. However the melpa distfiles are rebuilt everyday and cause checksum failures. However the only thing changing are the timestamps. Hence if a normaliser program could simply set all timestamps to some predefined value (say 1.1.1970) then this problem should vanish. Thanks in advance Markus -- For instance, on the planet Earth, man had always assumed that he was more intelligent than dolphins because he had achieved so much---the wheel, New York, wars and so on---while all the dolphins had ever done was muck about in the water having a good time. But conversely, the dolphins had always believed that they were far more intelligent than man---for precisely the same reasons. (Douglas Adams, The Hitchhikers Guide to the Galaxy.)
[gentoo-portage-dev] [PATCH] MergeProcess: propagate mtimedb["ldpath"] to parent process (bug 836375)
Use an instance of multiprocessing.Pipe to propagate mtimedb["ldpath"]
from the MergeProcess child process to the parent process. This fixes
env_update calls to avoid unnecessary regeneration of ld.so.cache in
cases where mtimedb["ldpath"] has not changed since the last call to
env_update.
Bug: https://bugs.gentoo.org/836375
---
lib/portage/dbapi/_MergeProcess.py | 18 ++
lib/portage/dbapi/vartree.py | 10 ++
2 files changed, 28 insertions(+)
diff --git a/lib/portage/dbapi/_MergeProcess.py
b/lib/portage/dbapi/_MergeProcess.py
index db3f3b105..667a5bf20 100644
--- a/lib/portage/dbapi/_MergeProcess.py
+++ b/lib/portage/dbapi/_MergeProcess.py
@@ -2,6 +2,7 @@
# Distributed under the terms of the GNU General Public License v2
import io
+import multiprocessing
import platform
import fcntl
@@ -38,6 +39,7 @@ class MergeProcess(ForkProcess):
"_dblink",
"_elog_keys",
"_locked_vdb",
+"_mtime_reader",
)
def _start(self):
@@ -113,6 +115,15 @@ class MergeProcess(ForkProcess):
self._elog_reader_fd = None
return False
+def _mtime_handler(self):
+try:
+mtimes = self._mtime_reader.recv()
+except EOFError:
+pass
+else:
+self.prev_mtimes.clear()
+self.prev_mtimes.update(mtimes)
+
def _spawn(self, args, fd_pipes, **kwargs):
"""
Extend the superclass _spawn method to perform some pre-fork and
@@ -127,6 +138,11 @@ class MergeProcess(ForkProcess):
fcntl.fcntl(elog_reader_fd, fcntl.F_GETFL) | os.O_NONBLOCK,
)
+mtime_reader, mtime_writer = multiprocessing.Pipe(duplex=False)
+fd_pipes[mtime_writer.fileno()] = mtime_writer.fileno()
+self.scheduler.add_reader(mtime_reader.fileno(), self._mtime_handler)
+self._mtime_reader = mtime_reader
+
blockers = None
if self.blockers is not None:
# Query blockers in the main process, since closing
@@ -142,6 +158,7 @@ class MergeProcess(ForkProcess):
vartree=self.vartree,
blockers=blockers,
pipe=elog_writer_fd,
+mtime_pipe=mtime_writer,
)
fd_pipes[elog_writer_fd] = elog_writer_fd
self.scheduler.add_reader(elog_reader_fd, self._elog_output_handler)
@@ -160,6 +177,7 @@ class MergeProcess(ForkProcess):
self._elog_reader_fd = elog_reader_fd
pids = super(MergeProcess, self)._spawn(args, fd_pipes, **kwargs)
os.close(elog_writer_fd)
+mtime_writer.close()
self._buf = ""
self._elog_keys = set()
# Discard messages which will be collected by the subprocess,
diff --git a/lib/portage/dbapi/vartree.py b/lib/portage/dbapi/vartree.py
index 602913862..a95d60691 100644
--- a/lib/portage/dbapi/vartree.py
+++ b/lib/portage/dbapi/vartree.py
@@ -1806,6 +1806,7 @@ class dblink:
blockers=None,
scheduler=None,
pipe=None,
+mtime_pipe=None,
):
"""
Creates a DBlink object for a given CPV.
@@ -1862,6 +1863,7 @@ class dblink:
self._device_path_map = {}
self._hardlink_merge_map = {}
self._hash_key = (self._eroot, self.mycpv)
+self._mtime_pipe = mtime_pipe
self._protect_obj = None
self._pipe = pipe
self._postinst_failure = False
@@ -2618,6 +2620,7 @@ class dblink:
writemsg_level=self._display_merge,
vardbapi=self.vartree.dbapi,
)
+self._send_mtimes(ldpath_mtimes)
unmerge_with_replacement = preserve_paths is not None
if not unmerge_with_replacement:
@@ -4243,6 +4246,12 @@ class dblink:
def _emerge_log(self, msg):
emergelog(False, msg)
+def _send_mtimes(self, mtimes):
+if self._mtime_pipe is None:
+return
+
+self._mtime_pipe.send(mtimes)
+
def treewalk(
self,
srcroot,
@@ -5274,6 +5283,7 @@ class dblink:
writemsg_level=self._display_merge,
vardbapi=self.vartree.dbapi,
)
+self._send_mtimes(prev_mtimes)
# For gcc upgrades, preserved libs have to be removed after the
# the library path has been updated.
--
2.35.1
Re: [gentoo-portage-dev] Changing the VDB format
On 4/11/2022 15:20, Sid Spry wrote: > On Mon, Apr 11, 2022, at 3:02 PM, Joshua Kinard wrote: >> >> I think json is the best format for storing the data on-disk. It's intended >> to be a data serialization format to convert data from a non-specific memory >> format to a storable on-disk format and back again, so this is a perfect use >> for it. > > Can we avoid adding another format? I find json very hard to edit by hand, > it's > good at storing lots of data in a quasi-textual format, but is strict enough > to be > obnoxious to work with. I sympathize with you here on this, json is not a format geared for editing by hand... :: looks disapprovingly at net-misc/kea :: > Can the files not be concatenated? Doing so is similar to the tar suggestion, > but would keep everything very portage-like. Have the contents assigned to > variables. I am betting someone tried this at the start but settled on the > current > scheme. Does anyone know why? (This would have to be done in bash syntax > I assume.) > > Alternatively, I think the tar suggestion is quite elegant. There's streaming > decompressors you can use from python. It adds an extra step to modify but > that could be handled transparently by a dev mode. In dev mode, leave the > files > after extraction and do not re-extract, for release mode replace the archive > with > what is on disk. Out of curiosity, what are you doing that requires manual editing of the VDB data? That data isn't, in normal scenarios, supposed to be arbitrarily edited. Throwing out a good optimization for what sounds like a niche corner-case doesn't seem like a great plan. Given json's malleable format, and especially Matt's example script, converting from json data to another format that's more conducive to manual editing in rare circumstances, then converting back, is not impossible. -- Joshua Kinard Gentoo/MIPS ku...@gentoo.org rsa6144/5C63F4E3F5C6C943 2015-04-27 177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943 "The past tempts us, the present confuses us, the future frightens us. And our lives slip away, moment by moment, lost in that vast, terrible in-between." --Emperor Turhan, Centauri Republic
Re: [gentoo-portage-dev] Changing the VDB format
On Mon, Apr 11, 2022, at 3:02 PM, Joshua Kinard wrote:
> On 3/13/2022 21:06, Matt Turner wrote:
>> The VDB uses a one-file-per-variable format. This has some
>> inefficiencies, with many file systems. For example the 'EAPI' file
>> that contains a single character will consume a 4K block on disk.
>>
>> $ cd /var/db/pkg/sys-apps/portage-3.0.30-r1/
>> $ ls -lh --block-size=1 | awk 'BEGIN { sum = 0; } { sum += $5; } END {
>> print sum }'
>> 418517
>> $ du -sh --apparent-size .
>> 413K.
>> $ du -sh .
>> 556K.
>>
>> During normal operations, portage has to read each of these 35+
>> files/package individually.
>>
>> I suggest that we change the VDB format to a commonly used format that
>> can be quickly read by portage and any other tools. Combining these
>> 35+ files into a single file with a commonly used format should:
>>
>> - speed up vdb access
>> - improve disk usage
>> - allow external tools to access VDB data more easily
>>
>> I've attached a program that prints the VDB contents of a specified
>> package in different formats: json, toml, and yaml (and also Python
>> PrettyPrinter, just because). I think it's important to keep the VDB
>> format as plain-text for ease of manipulation, so I have not
>> considered anything like sqlite.
>>
>> I expected to prefer toml, but I actually find it to be rather gross looking.
>
> Agreed, the toml output is rather "cluttered" looking.
>
>
>> I recommend json and think it is the best choice because:
>>
>> - json provides the smallest on-disk footprint
>> - json is part of Python's standard library (so is yaml, and toml will
>> be in Python 3.11)
>> - Every programming language has multiple json parsers
>> -- lots of effort has been spent making them extremely fast.
>>
>> I think we would have a significant time period for the transition. I
>> think I would include support for the new format in Portage, and ship
>> a tool with portage to switch back and forth between old and new
>> formats on-disk. Maybe after a year, drop the code from Portage to
>> support the old format?
>>
>> Thoughts?
>
> I think json is the best format for storing the data on-disk. It's intended
> to be a data serialization format to convert data from a non-specific memory
> format to a storable on-disk format and back again, so this is a perfect use
> for it.
Can we avoid adding another format? I find json very hard to edit by hand, it's
good at storing lots of data in a quasi-textual format, but is strict enough to
be
obnoxious to work with.
Can the files not be concatenated? Doing so is similar to the tar suggestion,
but would keep everything very portage-like. Have the contents assigned to
variables. I am betting someone tried this at the start but settled on the
current
scheme. Does anyone know why? (This would have to be done in bash syntax
I assume.)
Alternatively, I think the tar suggestion is quite elegant. There's streaming
decompressors you can use from python. It adds an extra step to modify but
that could be handled transparently by a dev mode. In dev mode, leave the files
after extraction and do not re-extract, for release mode replace the archive
with
what is on disk.
Sid.
Re: [gentoo-portage-dev] Changing the VDB format
On 3/13/2022 21:06, Matt Turner wrote:
> The VDB uses a one-file-per-variable format. This has some
> inefficiencies, with many file systems. For example the 'EAPI' file
> that contains a single character will consume a 4K block on disk.
>
> $ cd /var/db/pkg/sys-apps/portage-3.0.30-r1/
> $ ls -lh --block-size=1 | awk 'BEGIN { sum = 0; } { sum += $5; } END {
> print sum }'
> 418517
> $ du -sh --apparent-size .
> 413K.
> $ du -sh .
> 556K.
>
> During normal operations, portage has to read each of these 35+
> files/package individually.
>
> I suggest that we change the VDB format to a commonly used format that
> can be quickly read by portage and any other tools. Combining these
> 35+ files into a single file with a commonly used format should:
>
> - speed up vdb access
> - improve disk usage
> - allow external tools to access VDB data more easily
>
> I've attached a program that prints the VDB contents of a specified
> package in different formats: json, toml, and yaml (and also Python
> PrettyPrinter, just because). I think it's important to keep the VDB
> format as plain-text for ease of manipulation, so I have not
> considered anything like sqlite.
>
> I expected to prefer toml, but I actually find it to be rather gross looking.
Agreed, the toml output is rather "cluttered" looking.
> I recommend json and think it is the best choice because:
>
> - json provides the smallest on-disk footprint
> - json is part of Python's standard library (so is yaml, and toml will
> be in Python 3.11)
> - Every programming language has multiple json parsers
> -- lots of effort has been spent making them extremely fast.
>
> I think we would have a significant time period for the transition. I
> think I would include support for the new format in Portage, and ship
> a tool with portage to switch back and forth between old and new
> formats on-disk. Maybe after a year, drop the code from Portage to
> support the old format?
>
> Thoughts?
I think json is the best format for storing the data on-disk. It's intended
to be a data serialization format to convert data from a non-specific memory
format to a storable on-disk format and back again, so this is a perfect use
for it.
That said, I actually do like the yaml output as well, but I think the
better use-case for that would be in the form of a secondary tool that maybe
could be a part of portage's 'q' commands (qpkg, qfile, qlist, etc) to read
the JSON-formatted VDB data and export it in yaml for review. Something
like 'qvdb --yaml sys-libs/glibc-2.35-r2' to dump the VDB data to stdout
(and maybe do other tasks, but that's a discussion for another thread).
As far as support for the old format goes, I think one year is too short.
Two years is preferable, though I would not be totally opposed to as long as
three years. Adoption could probably be helped by turning this vdb.py
script into something more functional that can actually walk the current VDB
and convert it to the new chosen format and write that out to an alternate
location that a user could then transplant into /var/db/pkg after verifying it.
One other thought -- I think there should be a tuning knob in make.conf to
enable compression of the VDB's new format or not. The specific compression
format I leave up for debate (I'd say go with zstd, though), but if I am
running on a filesystem that supports native compression (e.g., ZFS), I'd
want to turn VDB compression off and let ZFS handle that at the filesystem
level. But on another system with say, XFS, I'd want to turn that on to get
some benefits, especially on older hardware that's going to be more I/O bound.
E.g., in JSON format, sys-libs/glibc-2.35-r2 clocks in at ~345KB:
# ./vdb.py --json /var/db/pkg/sys-libs/glibc-2.35-r2 > glibc.json
# ls -lh --block-size=1 glibc.json
-rw-r--r-- 1 root root 352479 Apr 11 14:53 glibc.json
# zstd glibc.json
glibc.json : 21.70% ( 344 KiB => 74.7 KiB, glibc.json.zst)
# ls -lh --block-size=1 glibc.json.zst
-rw-r--r-- 1 root root 76498 Apr 11 14:53 glibc.json.zst
(this is on a tmpfs-formatted ramdrive)
--
Joshua Kinard
Gentoo/MIPS
ku...@gentoo.org
rsa6144/5C63F4E3F5C6C943 2015-04-27
177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943
"The past tempts us, the present confuses us, the future frightens us. And
our lives slip away, moment by moment, lost in that vast, terrible in-between."
--Emperor Turhan, Centauri Republic
Re: [gentoo-portage-dev] The build system (and install layout) of Portage
On 3/17/22 10:22, Michał Górny wrote: Hi, everyone. You've probably had the opportunity to hear that a lot has changed in Python packaging since Portage's setup.py was written in 2014. There were some minor changes to keep it working since but it's time to reconsider. Long story short, distutils is strongly deprecated, setuptools deprecated most of the customizations (and we're relying heavily on customizations), PEP 517 doesn't cover our use cases exactly... and it's quite likely that sooner or later our build system will fall apart. On top of that, setuptools is going through a stage of "let's vendor a new dependency every week", so it doesn't look like a feasible long-term solution. A large part of the problem is that Portage is heavily relying on non-Pythonic idioms for installing stuff. I wonder if we can rely less on the deprecated customizations somehow. For example, the venv_data_files function in setup.py succeeds in installing a bunch of files in custom locations, and maybe we can rely more on that approach (use it not only for venv installations). -- Thanks, Zac OpenPGP_signature Description: OpenPGP digital signature
Re: [gentoo-portage-dev] The build system (and install layout) of Portage
On Thu, 2022-03-17 at 20:57 +, James Le Cuirot wrote: > On Thu, 2022-03-17 at 18:22 +0100, Michał Górny wrote: > > An alternative is to go back to using (at least partially) Makefiles or > > Meson. However, that would have the important drawback that we'd lose > > the ability to install Portage as a regular Python package (e.g. inside > > a virtualenv). > > What is the use case for doing that? I thought maybe testing, but then you can > run Portage and its unit tests in-place without installing it at all right > now. Using Portage as a library for package inspection, etc. -- i.e. mostly CI purposes. -- Best regards, Michał Górny
Re: [gentoo-portage-dev] The build system (and install layout) of Portage
On Thu, 2022-03-17 at 18:22 +0100, Michał Górny wrote: > An alternative is to go back to using (at least partially) Makefiles or > Meson. However, that would have the important drawback that we'd lose > the ability to install Portage as a regular Python package (e.g. inside > a virtualenv). What is the use case for doing that? I thought maybe testing, but then you can run Portage and its unit tests in-place without installing it at all right now. signature.asc Description: This is a digitally signed message part
[gentoo-portage-dev] The build system (and install layout) of Portage
Hi, everyone. You've probably had the opportunity to hear that a lot has changed in Python packaging since Portage's setup.py was written in 2014. There were some minor changes to keep it working since but it's time to reconsider. Long story short, distutils is strongly deprecated, setuptools deprecated most of the customizations (and we're relying heavily on customizations), PEP 517 doesn't cover our use cases exactly... and it's quite likely that sooner or later our build system will fall apart. On top of that, setuptools is going through a stage of "let's vendor a new dependency every week", so it doesn't look like a feasible long-term solution. A large part of the problem is that Portage is heavily relying on non-Pythonic idioms for installing stuff. At this point, I'd love to use flit for Portage, as that's a nice build system with upstream that understands packagers. I mean, if someone tells me they want to avoid vendoring stuff *and* creating cyclic dependencies (all other Python build systems vendor tons of stuff right now), then I can't but appreciate that very much. That said, I don't think flit is powerful enough to handle all the intricacies Portage requires, starting with the fancy way it's installing scripts to many different directories. An alternative is to go back to using (at least partially) Makefiles or Meson. However, that would have the important drawback that we'd lose the ability to install Portage as a regular Python package (e.g. inside a virtualenv). At this point, I've pretty much lost all motivation to work on it. Nevertheless, it's something that needs to be done eventually. Does anyone have some idea, motivation and will to transition Portage to another build system? -- Best regards, Michał Górny
Re: [gentoo-portage-dev] Changing the VDB format
On 14/03/2022 13.22, Fabian Groffen wrote: Hi, I've recently been thinking about this too. On 13-03-2022 18:06:21 -0700, Matt Turner wrote: The VDB uses a one-file-per-variable format. This has some inefficiencies, with many file systems. For example the 'EAPI' file that contains a single character will consume a 4K block on disk. I recommend json and think it is the best choice because: [snip] - json provides the smallest on-disk footprint - json is part of Python's standard library (so is yaml, and toml will be in Python 3.11) - Every programming language has multiple json parsers -- lots of effort has been spent making them extremely fast. I would like to suggest to use "tar". Your idea sounds very appealing and I am by no means an expert to the tar file format but https://www.gnu.org/software/tar/manual/html_node/Standard.html states """ …an archive consists of a series of file entries terminated by an end-of-archive entry, which consists of two 512 blocks of zero bytes. """ and the Wikipedia entry of 'tar' [1] states """ Each file object includes any file data, and is preceded by a 512-byte header record. The file data is written unaltered except that its length is rounded up to a multiple of 512 bytes. """ and furthermore """ The end of an archive is marked by at least two consecutive zero-filled records. """ Which sounds like a lot of overhead if no compression is involved. Not sure if this can be considered a knock out criteria for tar. - Flow
Re: [gentoo-portage-dev] Changing the VDB format
> "MT" == Matt Turner writes: MT> For example the 'EAPI' file MT> that contains a single character will consume a 4K block on disk. the sort of filesystes one expects to be used for /var all store short files either directly in the inode or in the directory entry¹. that said, Fabian’s suggestion of tar(1)ing those files sounds like a winner. a number of tools and editors allow r/w access to tar(5) files, including to the individual files therein. 1] or at least unix/linux filesystems all used too -JimC -- James Cloos OpenPGP: 0x997A9F17ED7DAEA6
