Hi Fabian, Is there anything I can do to help move this patchset forward? I'd be happy to extract the libxsimd-dev changes into a separate commit if that would make review easier, as it might be less controversial on its own.
On Mon Jan 26, 2026 at 5:06 PM CST, Fabian Grünbichler wrote: > On January 24, 2026 5:13 am, Kefu Chai wrote: >> On Fri Jan 23, 2026 at 9:03 PM CST, Fabian Grünbichler wrote: >>> one small question inline, otherwise this one LGTM >>> >>> On January 23, 2026 8:56 am, Kefu Chai wrote: >>>> Previously, cephadm's build process pulled dependencies from PyPI, >>>> which failed in environments with restricted network access. This >>>> change backports upstream improvements that enable building cephadm >>>> using system packages instead, mirroring the existing RPM packaging >>>> workflow. >>>> >>>> And the vendored Arrow pulled xsimd from GitHub, this also failed in >>>> building environment without network access. This change backports >>>> upstream improvements enabling the build to use libxsimd-dev package >>>> when building debian packages. >>>> >>>> Backported changes: >>>> - https://github.com/ceph/ceph/pull/66256 - Use system packages for >>>> cephadm bundled dependencies >>>> - https://github.com/ceph/ceph/pull/66248 - Add libxsimd-dev build >>>> dependency for vendored Arrow >>>> - https://github.com/ceph/ceph/pull/65292 - cephadm: fix building >>>> rpm-sourced cephadm zippapp on el10 >>>> >>>> Note: Test commits from these PRs are excluded from this backport. >>>> >>>> Signed-off-by: Kefu Chai <[email protected]> >>>> --- >>>> ...-issues-running-existing-cephadm-bui.patch | 66 +++++ >>>> ...root_entries-population-in-version-c.patch | 62 +++++ >>>> ...cephadm-rpm-based-builds-without-top.patch | 89 ++++++ >>>> ...d-Debian-package-support-for-bundled.patch | 261 ++++++++++++++++++ >>>> ...m-packages-for-cephadm-bundled-depen.patch | 54 ++++ >>>> ...-Use-AUTO-mode-for-xsimd-dependency-.patch | 55 ++++ >>>> ...dd-libxsimd-dev-build-dependency-for.patch | 41 +++ >>>> patches/series | 8 + >>>> 8 files changed, 636 insertions(+) >>>> create mode 100644 >>>> patches/0042-cephadm-fix-some-issues-running-existing-cephadm-bui.patch >>>> create mode 100644 >>>> patches/0043-cephadm-fix-zip_root_entries-population-in-version-c.patch >>>> create mode 100644 >>>> patches/0044-cephadm-support-cephadm-rpm-based-builds-without-top.patch >>>> create mode 100644 >>>> patches/0045-cephadm-build-Add-Debian-package-support-for-bundled.patch >>>> create mode 100644 >>>> patches/0046-debian-Use-system-packages-for-cephadm-bundled-depen.patch >>>> create mode 100644 >>>> patches/0047-cmake-BuildArrow-Use-AUTO-mode-for-xsimd-dependency-.patch >>>> create mode 100644 >>>> patches/0048-debian-control-Add-libxsimd-dev-build-dependency-for.patch >>>> >>>> diff --git >>>> a/patches/0042-cephadm-fix-some-issues-running-existing-cephadm-bui.patch >>>> b/patches/0042-cephadm-fix-some-issues-running-existing-cephadm-bui.patch >>>> new file mode 100644 >>>> index 00000000000..c6208b0db2a >>>> --- /dev/null >>>> +++ >>>> b/patches/0042-cephadm-fix-some-issues-running-existing-cephadm-bui.patch >>>> @@ -0,0 +1,66 @@ >>>> +From 2685f6c35245d033e73e16ece3e356fcead44cea Mon Sep 17 00:00:00 2001 >>>> +From: John Mulligan <[email protected]> >>>> +Date: Thu, 21 Aug 2025 12:30:55 -0400 >>>> +Subject: [PATCH 42/46] cephadm: fix some issues running existing cephadm >>>> build >>>> + tests >>>> + >>>> +As time has marched on and people changed things our tests no longer >>>> +match the expected inputs. >>>> + >>>> +Signed-off-by: John Mulligan <[email protected]> >>>> +(cherry picked from commit 31c8010faa417ca53614bd30379a9b9c0c9199de) >>>> +--- >>>> + src/cephadm/tests/build/test_cephadm_build.py | 18 ++++++++++++------ >>>> + 1 file changed, 12 insertions(+), 6 deletions(-) >>>> + >>>> +diff --git a/src/cephadm/tests/build/test_cephadm_build.py >>>> b/src/cephadm/tests/build/test_cephadm_build.py >>>> +index c2995a76d4b..9beef67ffe0 100644 >>>> +--- a/src/cephadm/tests/build/test_cephadm_build.py >>>> ++++ b/src/cephadm/tests/build/test_cephadm_build.py >>>> +@@ -34,12 +34,12 @@ CONTAINERS = { >>>> + }, >>>> + 'ubuntu-20.04': { >>>> + 'name': 'cephadm-build-test:ubuntu-20-04-py3', >>>> +- 'base_image': 'quay.io/library/ubuntu:20.04', >>>> ++ 'base_image': 'docker.io/library/ubuntu:20.04', >>>> + 'script': 'apt update && apt install -y python3-venv', >>>> + }, >>>> + 'ubuntu-22.04': { >>>> + 'name': 'cephadm-build-test:ubuntu-22-04-py3', >>>> +- 'base_image': 'quay.io/library/ubuntu:22.04', >>>> ++ 'base_image': 'docker.io/library/ubuntu:22.04', >>>> + 'script': 'apt update && apt install -y python3-venv', >>>> + }, >>>> + } >>>> +@@ -128,8 +128,8 @@ def test_cephadm_build(env, source_dir, tmp_path): >>>> + assert all('requirements_entry' in v for v in >>>> data['bundled_packages']) >>>> + assert 'zip_root_entries' in data >>>> + zre = data['zip_root_entries'] >>>> +- assert any(e.startswith('Jinja2') for e in zre) >>>> +- assert any(e.startswith('MarkupSafe') for e in zre) >>>> ++ assert any(_dist_info(e, 'Jinja2') for e in zre) >>>> ++ assert any(_dist_info(e, 'MarkupSafe') for e in zre) >>>> + assert any(e.startswith('jinja2') for e in zre) >>>> + assert any(e.startswith('markupsafe') for e in zre) >>>> + assert any(e.startswith('cephadmlib') for e in zre) >>>> +@@ -184,9 +184,15 @@ def test_cephadm_build_from_rpms(env, source_dir, >>>> tmp_path): >>>> + assert all('requirements_entry' in v for v in >>>> data['bundled_packages']) >>>> + assert 'zip_root_entries' in data >>>> + zre = data['zip_root_entries'] >>>> +- assert any(e.startswith('Jinja2') for e in zre) >>>> +- assert any(e.startswith('MarkupSafe') for e in zre) >>>> ++ assert any(_dist_info(e, 'Jinja2') for e in zre) >>>> ++ assert any(_dist_info(e, 'MarkupSafe') for e in zre) >>>> + assert any(e.startswith('jinja2') for e in zre) >>>> + assert any(e.startswith('markupsafe') for e in zre) >>>> + assert any(e.startswith('cephadmlib') for e in zre) >>>> + assert any(e.startswith('_cephadmmeta') for e in zre) >>>> ++ >>>> ++ >>>> ++def _dist_info(entry, name): >>>> ++ return ( >>>> ++ entry.startswith(entry) or entry.startswith(entry.lower()) >>>> ++ ) and (entry.endswith('.dist-info') or entry.endswith('.egg-info')) >>>> +-- >>>> +2.47.3 >>>> + >>>> diff --git >>>> a/patches/0043-cephadm-fix-zip_root_entries-population-in-version-c.patch >>>> b/patches/0043-cephadm-fix-zip_root_entries-population-in-version-c.patch >>>> new file mode 100644 >>>> index 00000000000..7a0b1a18d68 >>>> --- /dev/null >>>> +++ >>>> b/patches/0043-cephadm-fix-zip_root_entries-population-in-version-c.patch >>>> @@ -0,0 +1,62 @@ >>>> +From 2c20ae95f32a2bdf78576252f1c9171f00a2790e Mon Sep 17 00:00:00 2001 >>>> +From: Kefu Chai <[email protected]> >>>> +Date: Mon, 10 Nov 2025 12:11:08 +0800 >>>> +Subject: [PATCH 43/46] cephadm: fix zip_root_entries population in version >>>> + command >>>> + >>>> +The 'cephadm version --verbose' command was returning an empty >>>> +zip_root_entries list because it relied on the private '_files' >>>> +attribute of zipimport.zipimporter, which is not reliably populated >>>> +across Python versions. >>>> + >>>> +This commit fixes the issue by using the zipfile module to properly >>>> +read the archive contents via the loader.archive path. This ensures >>>> +that zip_root_entries is correctly populated with the root-level >>>> +directories in the zipapp. >>>> + >>>> +This fix is necessary for the cephadm build tests to properly validate >>>> +that all expected packages and modules are included in the built zipapp. >>>> + >>>> +Signed-off-by: Kefu Chai <[email protected]> >>>> +(cherry picked from commit 2c68c1496dbb7cd01bf783e31510940445040a34) >>>> +--- >>>> + src/cephadm/cephadm.py | 16 ++++++++++++---- >>>> + 1 file changed, 12 insertions(+), 4 deletions(-) >>>> + >>>> +diff --git a/src/cephadm/cephadm.py b/src/cephadm/cephadm.py >>>> +index f75aaa86dac..8d08c700868 100755 >>>> +--- a/src/cephadm/cephadm.py >>>> ++++ b/src/cephadm/cephadm.py >>>> +@@ -1688,6 +1688,7 @@ def command_version(ctx): >>>> + # type: (CephadmContext) -> int >>>> + import importlib >>>> + import zipimport >>>> ++ import zipfile >>>> + import types >>>> + >>>> + vmod: Optional[types.ModuleType] >>>> +@@ -1744,10 +1745,17 @@ def command_version(ctx): >>>> + out['bundled_packages'] = deps_info >>>> + except OSError: >>>> + pass >>>> +- files = getattr(loader, '_files', {}) >>>> +- out['zip_root_entries'] = sorted( >>>> +- {p.split('/')[0] for p in files.keys()} >>>> +- ) >>>> ++ # Use zipfile module to properly read the archive contents >>>> ++ # loader.archive contains the path to the zip file >>>> ++ try: >>>> ++ with zipfile.ZipFile(loader.archive, 'r') as zf: >>>> ++ files = zf.namelist() >>>> ++ out['zip_root_entries'] = sorted( >>>> ++ {p.split('/')[0] for p in files if p} >>>> ++ ) >>>> ++ except (OSError, zipfile.BadZipFile): >>>> ++ # Fallback to empty list if we can't read the zip >>>> ++ out['zip_root_entries'] = [] >>>> + >>>> + json.dump(out, sys.stdout, indent=2) >>>> + print() >>>> +-- >>>> +2.47.3 >>>> + >>>> diff --git >>>> a/patches/0044-cephadm-support-cephadm-rpm-based-builds-without-top.patch >>>> b/patches/0044-cephadm-support-cephadm-rpm-based-builds-without-top.patch >>>> new file mode 100644 >>>> index 00000000000..6e1b4dd2e0e >>>> --- /dev/null >>>> +++ >>>> b/patches/0044-cephadm-support-cephadm-rpm-based-builds-without-top.patch >>>> @@ -0,0 +1,89 @@ >>>> +From 5ce72178a782dfe2e80648063b025ba23d96d5d8 Mon Sep 17 00:00:00 2001 >>>> +From: John Mulligan <[email protected]> >>>> +Date: Thu, 21 Aug 2025 12:44:17 -0400 >>>> +Subject: [PATCH 44/46] cephadm: support cephadm rpm based builds without >>>> + top_level.txt >>>> + >>>> +Signed-off-by: John Mulligan <[email protected]> >>>> +(cherry picked from commit 26a499a8da339d870af193ea964368afbc84c694) >>>> +--- >>>> + src/cephadm/build.py | 56 +++++++++++++++++++++++++++++++++----------- >>>> + 1 file changed, 42 insertions(+), 14 deletions(-) >>>> + >>>> +diff --git a/src/cephadm/build.py b/src/cephadm/build.py >>>> +index 43bc58a4003..01c91fc1680 100755 >>>> +--- a/src/cephadm/build.py >>>> ++++ b/src/cephadm/build.py >>>> +@@ -434,6 +434,47 @@ def _install_rpm_deps(tempdir, config): >>>> + return dinfo >>>> + >>>> + >>>> ++def _gather_rpm_package_dirs(paths): >>>> ++ # = The easy way = >>>> ++ # the top_level.txt file can be used to determine where the python >>>> packages >>>> ++ # actually are. We need all of those and the meta-data dir (parent of >>>> ++ # top_level.txt) to be included in our zipapp >>>> ++ top_level = None >>>> ++ for path in paths: >>>> ++ if path.endswith('top_level.txt'): >>>> ++ top_level = pathlib.Path(path) >>>> ++ if top_level: >>>> ++ meta_dir = top_level.parent >>>> ++ pkg_dirs = [ >>>> ++ top_level.parent.parent / p >>>> ++ for p in top_level.read_text().splitlines() >>>> ++ ] >>>> ++ return meta_dir, pkg_dirs >>>> ++ # = The hard way = >>>> ++ # loop through the directories to find the .dist-info dir >>>> (containing the >>>> ++ # mandatory METADATA file, according to the spec) and once we know >>>> the >>>> ++ # location of dist info we find the sibling paths from the rpm >>>> listing >>>> ++ dist_info = None >>>> ++ ppaths = [] >>>> ++ for path in paths: >>>> ++ ppath = pathlib.Path(path) >>>> ++ ppaths.append(ppath) >>>> ++ if ppath.name == 'METADATA' and >>>> ppath.parent.name.endswith('.dist-info'): >>>> ++ dist_info = ppath.parent >>>> ++ break >>>> ++ if not dist_info: >>>> ++ raise ValueError('no .dist-info METADATA found') >>>> ++ if not dist_info.parent.name == 'site-packages': >>>> ++ raise ValueError( >>>> ++ 'unexpected parent directory (not site-packages):' >>>> ++ f' {dist_info.parent.name}' >>>> ++ ) >>>> ++ siblings = [ >>>> ++ p for p in ppaths if p.parent == dist_info.parent and p != >>>> dist_info >>>> ++ ] >>>> ++ return dist_info, siblings >>>> ++ >>>> ++ >>>> + def _deps_from_rpm(tempdir, config, dinfo, pkg): >>>> + # first, figure out what rpm provides a particular python lib >>>> + dist = f'python3.{sys.version_info.minor}dist({pkg})'.lower() >>>> +@@ -469,20 +510,7 @@ def _deps_from_rpm(tempdir, config, dinfo, pkg): >>>> + ['rpm', '-ql', rpmname], check=True, stdout=subprocess.PIPE >>>> + ) >>>> + paths = [l.decode('utf8') for l in res.stdout.splitlines()] >>>> +- # the top_level.txt file can be used to determine where the python >>>> packages >>>> +- # actually are. We need all of those and the meta-data dir (parent of >>>> +- # top_level.txt) to be included in our zipapp >>>> +- top_level = None >>>> +- for path in paths: >>>> +- if path.endswith('top_level.txt'): >>>> +- top_level = pathlib.Path(path) >>>> +- if not top_level: >>>> +- raise ValueError('top_level not found') >>>> +- meta_dir = top_level.parent >>>> +- pkg_dirs = [ >>>> +- top_level.parent.parent / p >>>> +- for p in top_level.read_text().splitlines() >>>> +- ] >>>> ++ meta_dir, pkg_dirs = _gather_rpm_package_dirs(paths) >>>> + meta_dest = tempdir / meta_dir.name >>>> + log.info(f"Copying {meta_dir} to {meta_dest}") >>>> + # copy the meta data directory >>>> +-- >>>> +2.47.3 >>>> + >>>> diff --git >>>> a/patches/0045-cephadm-build-Add-Debian-package-support-for-bundled.patch >>>> b/patches/0045-cephadm-build-Add-Debian-package-support-for-bundled.patch >>>> new file mode 100644 >>>> index 00000000000..94241490b64 >>>> --- /dev/null >>>> +++ >>>> b/patches/0045-cephadm-build-Add-Debian-package-support-for-bundled.patch >>>> @@ -0,0 +1,261 @@ >>>> +From a4744feb1457aa7fb3f47b42f29138682cc7d41c Mon Sep 17 00:00:00 2001 >>>> +From: Kefu Chai <[email protected]> >>>> +Date: Tue, 14 Oct 2025 21:04:42 +0800 >>>> +Subject: [PATCH 45/46] cephadm/build: Add Debian package support for >>>> bundled >>>> + dependencies >>>> + >>>> +Extends the cephadm build script to support bundling dependencies from >>>> +Debian packages in addition to pip and RPM packages. This allows building >>>> +cephadm on Debian-based distributions using system packages. >>>> + >>>> +Key changes: >>>> +- Add 'deb' to DependencyMode enum to enable Debian package mode >>>> +- Implement _setup_deb() to configure Debian dependency requirements >>>> +- Add _install_deb_deps() to orchestrate Debian package installation >>>> +- Add _gather_deb_package_dirs() to parse Debian package file listings >>>> + and locate Python package directories (handles both site-packages and >>>> + dist-packages directories used by Debian) >>>> +- Add _deps_from_deb() to extract Python dependencies from installed >>>> + Debian packages using dpkg/apt-cache tools >>>> +- Fix variable reference bug in _install_deps() (deps.mode -> >>>> config.deps_mode) >>>> + >>>> +The Debian implementation follows a similar pattern to the existing RPM >>>> +support, using dpkg-query and dpkg -L to locate installed packages and >>>> +their files, with special handling for Debian naming conventions >>>> +(e.g., PyYAML -> python3-yaml). >>>> + >>>> +Signed-off-by: Kefu Chai <[email protected]> >>>> +(cherry picked from commit 3ff9b0c24e33debe95a0a0c6b42da30be788871c) >>>> +--- >>>> + src/cephadm/build.py | 158 +++++++++++++++++++++++++++++++++++++++---- >>>> + 1 file changed, 143 insertions(+), 15 deletions(-) >>>> + >>>> +diff --git a/src/cephadm/build.py b/src/cephadm/build.py >>>> +index 01c91fc1680..53d3983de89 100755 >>>> +--- a/src/cephadm/build.py >>>> ++++ b/src/cephadm/build.py >>>> +@@ -154,6 +154,7 @@ class PipEnv(enum.Enum): >>>> + class DependencyMode(enum.Enum): >>>> + pip = enum.auto() >>>> + rpm = enum.auto() >>>> ++ deb = enum.auto() >>>> + none = enum.auto() >>>> + >>>> + >>>> +@@ -169,6 +170,8 @@ class Config: >>>> + self._setup_pip() >>>> + elif self.deps_mode == DependencyMode.rpm: >>>> + self._setup_rpm() >>>> ++ elif self.deps_mode == DependencyMode.deb: >>>> ++ self._setup_deb() >>>> + >>>> + def _setup_pip(self): >>>> + if self._maj_min == (3, 6): >>>> +@@ -180,6 +183,9 @@ class Config: >>>> + def _setup_rpm(self): >>>> + self.requirements = [InstallSpec(**v) for v in PY_REQUIREMENTS] >>>> + >>>> ++ def _setup_deb(self): >>>> ++ self.requirements = [InstallSpec(**v) for v in PY_REQUIREMENTS] >>>> ++ >>>> + >>>> + class DependencyInfo: >>>> + """Type for tracking bundled dependencies.""" >>>> +@@ -333,7 +339,9 @@ def _install_deps(tempdir, config): >>>> + return _install_pip_deps(tempdir, config) >>>> + if config.deps_mode == DependencyMode.rpm: >>>> + return _install_rpm_deps(tempdir, config) >>>> +- raise ValueError(f'unexpected deps mode: {deps.mode}') >>>> ++ if config.deps_mode == DependencyMode.deb: >>>> ++ return _install_deb_deps(tempdir, config) >>>> ++ raise ValueError(f'unexpected deps mode: {config.deps_mode}') >>>> + >>>> + >>>> + def _install_pip_deps(tempdir, config): >>>> +@@ -434,7 +442,26 @@ def _install_rpm_deps(tempdir, config): >>>> + return dinfo >>>> + >>>> + >>>> +-def _gather_rpm_package_dirs(paths): >>>> ++def _install_deb_deps(tempdir, config): >>>> ++ log.info("Installing dependencies using Debian packages") >>>> ++ dinfo = DependencyInfo(config) >>>> ++ for pkg in config.requirements: >>>> ++ log.info(f"Looking for debian package for: {pkg.name!r}") >>>> ++ _deps_from_deb(tempdir, config, dinfo, pkg.name) >>>> ++ return dinfo >>>> ++ >>>> ++ >>>> ++def _gather_package_dirs(paths, expected_parent_dir): >>>> ++ """Parse package file listing to find Python package directories. >>>> ++ >>>> ++ Args: >>>> ++ paths: List of file paths from package listing >>>> ++ expected_parent_dir: Expected parent directory name (e.g., >>>> 'site-packages' for RPM, >>>> ++ 'dist-packages' for Debian) >>>> ++ >>>> ++ Returns: >>>> ++ Tuple of (metadata_dir, package_dirs) >>>> ++ """ >>>> + # = The easy way = >>>> + # the top_level.txt file can be used to determine where the python >>>> packages >>>> + # actually are. We need all of those and the meta-data dir (parent of >>>> +@@ -453,7 +480,7 @@ def _gather_rpm_package_dirs(paths): >>>> + # = The hard way = >>>> + # loop through the directories to find the .dist-info dir >>>> (containing the >>>> + # mandatory METADATA file, according to the spec) and once we know >>>> the >>>> +- # location of dist info we find the sibling paths from the rpm >>>> listing >>>> ++ # location of dist info we find the sibling paths from the package >>>> listing >>>> + dist_info = None >>>> + ppaths = [] >>>> + for path in paths: >>>> +@@ -464,9 +491,9 @@ def _gather_rpm_package_dirs(paths): >>>> + break >>>> + if not dist_info: >>>> + raise ValueError('no .dist-info METADATA found') >>>> +- if not dist_info.parent.name == 'site-packages': >>>> ++ if dist_info.parent.name != expected_parent_dir: >>>> + raise ValueError( >>>> +- 'unexpected parent directory (not site-packages):' >>>> ++ f'unexpected parent directory (not {expected_parent_dir}):' >>>> + f' {dist_info.parent.name}' >>>> + ) >>>> + siblings = [ >>>> +@@ -475,6 +502,31 @@ def _gather_rpm_package_dirs(paths): >>>> + return dist_info, siblings >>>> + >>>> + >>>> ++def _copy_package_files(tempdir, paths, expected_parent_dir): >>>> ++ """Copy package files to the build directory. >>>> ++ >>>> ++ Args: >>>> ++ tempdir: Temporary directory to copy files to >>>> ++ paths: List of file paths from package listing >>>> ++ expected_parent_dir: Expected parent directory name per >>>> packaging convention: >>>> ++ - 'site-packages' for RPM-based distributions >>>> ++ - 'dist-packages' for Debian-based distributions >>>> ++ >>>> ++ Returns: >>>> ++ None >>>> ++ """ >>>> ++ meta_dir, pkg_dirs = _gather_package_dirs(paths, expected_parent_dir) >>>> ++ meta_dest = tempdir / meta_dir.name >>>> ++ log.info(f"Copying {meta_dir} to {meta_dest}") >>>> ++ # copy the meta data directory >>>> ++ shutil.copytree(meta_dir, meta_dest, ignore=_ignore_cephadmlib) >>>> ++ # copy all the package directories >>>> ++ for pkg_dir in pkg_dirs: >>>> ++ pkg_dest = tempdir / pkg_dir.name >>>> ++ log.info(f"Copying {pkg_dir} to {pkg_dest}") >>>> ++ shutil.copytree(pkg_dir, pkg_dest, ignore=_ignore_cephadmlib) >>>> ++ >>>> ++ >>>> + def _deps_from_rpm(tempdir, config, dinfo, pkg): >>>> + # first, figure out what rpm provides a particular python lib >>>> + dist = f'python3.{sys.version_info.minor}dist({pkg})'.lower() >>>> +@@ -510,16 +562,92 @@ def _deps_from_rpm(tempdir, config, dinfo, pkg): >>>> + ['rpm', '-ql', rpmname], check=True, stdout=subprocess.PIPE >>>> + ) >>>> + paths = [l.decode('utf8') for l in res.stdout.splitlines()] >>>> +- meta_dir, pkg_dirs = _gather_rpm_package_dirs(paths) >>>> +- meta_dest = tempdir / meta_dir.name >>>> +- log.info(f"Copying {meta_dir} to {meta_dest}") >>>> +- # copy the meta data directory >>>> +- shutil.copytree(meta_dir, meta_dest, ignore=_ignore_cephadmlib) >>>> +- # copy all the package directories >>>> +- for pkg_dir in pkg_dirs: >>>> +- pkg_dest = tempdir / pkg_dir.name >>>> +- log.info(f"Copying {pkg_dir} to {pkg_dest}") >>>> +- shutil.copytree(pkg_dir, pkg_dest, ignore=_ignore_cephadmlib) >>>> ++ # RPM-based distributions use 'site-packages' for Python packages >>>> ++ _copy_package_files(tempdir, paths, 'site-packages') >>>> ++ >>>> ++ >>>> ++def _deps_from_deb(tempdir, config, dinfo, pkg): >>>> ++ """Extract Python dependencies from Debian packages. >>>> ++ >>>> ++ Args: >>>> ++ tempdir: Temporary directory to copy package files to >>>> ++ config: Build configuration >>>> ++ dinfo: DependencyInfo instance to track dependencies >>>> ++ pkg: Python package name (e.g., 'MarkupSafe', 'Jinja2', 'PyYAML') >>>> ++ """ >>>> ++ # Convert Python package name to Debian package name >>>> ++ # Python packages are typically named python3-<lowercase-name> >>>> ++ # Handle special cases: PyYAML -> python3-yaml, MarkupSafe -> >>>> python3-markupsafe >>>> ++ pkg_lower = pkg.lower() >>>> ++ if pkg_lower == 'pyyaml': >>>> ++ deb_pkg_name = 'python3-yaml' >>>> ++ else: >>>> ++ deb_pkg_name = f'python3-{pkg_lower}' >>>> ++ >>>> ++ # First, try to find the package using apt-cache >>>> ++ # This helps verify the package exists before trying to list its >>>> files >>>> ++ try: >>>> ++ res = subprocess.run( >>>> ++ ['apt-cache', 'show', deb_pkg_name], >>>> ++ check=True, >>>> ++ stdout=subprocess.PIPE, >>>> ++ stderr=subprocess.PIPE, >>>> ++ ) >>>> ++ except subprocess.CalledProcessError: >>>> ++ # Package not found, try alternative naming >>>> ++ log.warning(f"Package {deb_pkg_name} not found via apt-cache, >>>> trying dpkg -S") >>>> ++ # Try to search for files that might belong to this package >>>> ++ # Search for the Python module in site-packages >>>> ++ search_pattern = >>>> f'/usr/lib/python3*/dist-packages/{pkg.lower()}*' >>>> ++ try: >>>> ++ res = subprocess.run( >>>> ++ ['dpkg', '-S', search_pattern], >>>> ++ check=True, >>>> ++ stdout=subprocess.PIPE, >>>> ++ stderr=subprocess.PIPE, >>>> ++ ) >>>> ++ # dpkg -S output format: "package: /path/to/file" >>>> ++ deb_pkg_name = >>>> res.stdout.decode('utf8').split(':')[0].strip() >>>> ++ except subprocess.CalledProcessError as err: >>>> ++ log.error(f"Could not find Debian package for {pkg}") >>>> ++ log.error(f"Tried: {deb_pkg_name} and pattern search") >>>> ++ sys.exit(1) >>>> ++ >>>> ++ # Get version information using dpkg-query >>>> ++ try: >>>> ++ res = subprocess.run( >>>> ++ ['dpkg-query', '-W', '-f=${Version}\\n', deb_pkg_name], >>>> ++ check=True, >>>> ++ stdout=subprocess.PIPE, >>>> ++ ) >>>> ++ version = res.stdout.decode('utf8').strip() >>>> ++ except subprocess.CalledProcessError as err: >>>> ++ log.error(f"Could not query version for package {deb_pkg_name}: >>>> {err}") >>>> ++ sys.exit(1) >>>> ++ >>>> ++ log.info(f"Debian Package: {deb_pkg_name} (version: {version})") >>>> ++ dinfo.add( >>>> ++ pkg, >>>> ++ deb_name=deb_pkg_name, >>>> ++ version=version, >>>> ++ package_source='deb', >>>> ++ ) >>>> ++ >>>> ++ # Get the list of files provided by the Debian package >>>> ++ try: >>>> ++ res = subprocess.run( >>>> ++ ['dpkg', '-L', deb_pkg_name], >>>> ++ check=True, >>>> ++ stdout=subprocess.PIPE, >>>> ++ ) >>>> ++ except subprocess.CalledProcessError as err: >>>> ++ log.error(f"Could not list files for package {deb_pkg_name}: >>>> {err}") >>>> ++ sys.exit(1) >>>> ++ >>>> ++ paths = [l.decode('utf8') for l in res.stdout.splitlines()] >>>> ++ # Debian-based distributions use 'dist-packages' for system-managed >>>> Python packages >>>> ++ # per Debian Python Policy: >>>> https://www.debian.org/doc/packaging-manuals/python-policy/ >>>> ++ _copy_package_files(tempdir, paths, 'dist-packages') >>>> + >>>> + >>>> + def generate_version_file(versioning_vars, dest): >>>> +-- >>>> +2.47.3 >>>> + >>>> diff --git >>>> a/patches/0046-debian-Use-system-packages-for-cephadm-bundled-depen.patch >>>> b/patches/0046-debian-Use-system-packages-for-cephadm-bundled-depen.patch >>>> new file mode 100644 >>>> index 00000000000..0b4daa63f79 >>>> --- /dev/null >>>> +++ >>>> b/patches/0046-debian-Use-system-packages-for-cephadm-bundled-depen.patch >>>> @@ -0,0 +1,54 @@ >>>> +From 779c9251669b65b7ed77a841f281e2d957174db2 Mon Sep 17 00:00:00 2001 >>>> +From: Kefu Chai <[email protected]> >>>> +Date: Tue, 21 Oct 2025 11:25:00 +0800 >>>> +Subject: [PATCH 46/46] debian: Use system packages for cephadm bundled >>>> + dependencies >>>> + >>>> +Configure the Debian build to use CEPHADM_BUNDLED_DEPENDENCIES=deb, >>>> +which instructs the cephadm build script to bundle dependencies from >>>> +system-installed Debian packages instead of downloading from PyPI. >>>> + >>>> +This change addresses build failures in restricted network environments >>>> +where Debian build tools do not permit internet access. By leveraging >>>> +the Debian package support added in commit 9378a2988e1, the build now >>>> +uses python3-markupsafe, python3-jinja2, and python3-yaml packages >>>> +that are already installed as build dependencies. >>>> + >>>> +This approach mirrors the existing RPM packaging workflow, ensuring >>>> +consistent behavior across different distribution package formats. >>>> + >>>> +Signed-off-by: Kefu Chai <[email protected]> >>>> +(cherry picked from commit 25680021ee2422f8b8b4075b3ab77af39126eecc) >>> >>> long term, it might be even better to not bundle at all, but add those >>> python deps as runtime dependencies? >> >> I concur with your concerns, as bundling violates Debian's phylosophy of >> avoiding "vendored" dependencies. The distro intends to control and >> track all packages. And bunlding also implies more disk space usage and >> duplications. >> >> But it's also an upstream design choice: cephadm was explicitly designed >> to be standalone. Part of cephadm'svalue is that you can just curl | >> python3 it onto a fresh system. If it requires installing 10+ Python >> packages first, you've lost that simplicity. >> >> What do you think? > > I understand that reasoning, and cephadm isn't something we are using > anyway (atm).. > > > _______________________________________________ > pve-devel mailing list > [email protected] > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
