shymega opened a new issue, #2196: URL: https://github.com/apache/buildstream/issues/2196
> [!NOTE] > This report comes out of my own personal effort to package `buildstream`/ > `buildstream-plugins` for nixpkgs (https://github.com/NixOS/nixpkgs/pull/510073) — > it isn't related to or filed on behalf of my employer. Root-causing this and > drafting this report was done with the assistance of an AI coding agent (Claude > Code); I've reviewed the analysis and reproduction myself and stand behind it. ## Summary The shared cross-VCS ("repo kind") test suite in `buildstream/_testing/_sourcetests/` (consumed by downstream source plugins via `sourcetests_collection_hook`, e.g. from `buildstream-plugins`) writes new files into the directory tree that pytest's `datafiles` fixture (from the `pytest-datafiles` plugin) copies out for each test. This assumes the copy is always writable. That assumption broke as a side effect of an intentional, correct fix in `pytest-datafiles` 3.0: it now preserves the *source* file/directory permission bits on copy (see omarkohl/pytest-datafiles#11, closed with a regression test to lock in the behaviour). If the source data — i.e. buildstream's own installed `_testing/_sourcetests` package data — ends up on disk without the write bit (for example because it was installed by a package manager that makes installed files read-only, or, as in our case, because it lives in the Nix store, where all installed files are always mode 444/dirs 555), every copy `datafiles` produces inherits that missing write bit. Any test helper that then tries to write a *new* file into that copy fails with `PermissionError`. This isn't Nix-specific in principle — it will reproduce in any environment where buildstream's installed test-data files are non-writable — but Nix's packaging convention (all store paths are always read-only) makes it 100% reproducible there. ## Where we hit it Packaging `buildstream-plugins` 2.8.0 for nixpkgs (https://github.com/apache/buildstream-plugins), whose `tests/conftest.py` pulls in the shared suite via: ```python from buildstream._testing import sourcetests_collection_hook ``` Running the resulting pytest suite against buildstream 2.8.1 (installed read-only, as all Nix store paths are) produces 48 errors, all `PermissionError`, spread across every parametrized `[bzr]`/`[git]` case in: - `_sourcetests/build_checkout.py::test_fetch_build_checkout` - `_sourcetests/fetch.py::test_fetch`, `test_fetch_cross_junction` - `_sourcetests/mirror.py::test_mirror_fetch`, `test_mirror_fetch_upstream_absent`, `test_mirror_from_includes`, `test_mirror_track_upstream_present`, `test_mirror_track_upstream_absent` - `_sourcetests/track.py::test_track`, `test_track_recurse`, `test_track_recurse_except`, `test_cross_junction`, `test_track_include`, `test_track_include_junction`, `test_track_junction_included` - `_sourcetests/workspace.py::test_open` ## Example traceback ``` request = <SubRequest 'kind' for <Function test_fetch_build_checkout[bzr-strict]>> datafiles = PosixPath('/build/source/tmp/test_fetch_build_checkout_bzr_0') @pytest.fixture(params=ALL_REPO_KINDS.keys()) def kind(request, datafiles): # Register plugins both on the toplevel project and on its junctions for project_dir in [str(datafiles), os.path.join(str(datafiles), "files", "sub-project")]: > add_plugins_conf(project_dir, request.param) .../buildstream/_testing/_sourcetests/utils.py:49: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ .../buildstream/_testing/_sourcetests/utils.py:77: in add_plugins_conf _yaml.roundtrip_dump(project_conf, project_conf_file) src/buildstream/_yaml.pyx:477: in buildstream._yaml.roundtrip_dump ??? .../lib/python3.14/contextlib.py:141: in __enter__ return next(self.gen) .../buildstream/utils.py:663: in save_file_atomic fd, tempname = tempfile.mkstemp(dir=tempdir) .../lib/python3.14/tempfile.py:354: in mkstemp return _mkstemp_inner(dir, prefix, suffix, flags, output_type) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ dir = '/build/source/tmp/test_fetch_build_checkout_bzr_0/files/sub-project' def _mkstemp_inner(dir, pre, suf, flags, output_type): ... for seq in range(TMP_MAX): name = next(names) file = _os.path.join(dir, pre + name + suf) _sys.audit("tempfile.mkstemp", file) try: > fd = _os.open(file, flags, 0o600) E PermissionError: [Errno 13] Permission denied: '/build/source/tmp/test_fetch_build_checkout_bzr_0/files/sub-project/tmpb55t9_sb' .../lib/python3.14/tempfile.py:255: PermissionError ``` The same shape of failure (write into a `datafiles`-provided directory that inherited a non-writable source) recurs across all 48 failing tests listed above — they all go through `add_plugins_conf` or an equivalent "write a new file into the copied project tree" step. ## Why this only shows up now Per omarkohl/pytest-datafiles#11, prior to `pytest-datafiles` 3.0 the `datafiles` fixture did *not* preserve source permission bits when copying (using `py.path`), so a non-writable source always produced a writable copy — masking this bug. 3.0 switched to `pathlib`/`shutil.copy`/`shutil.copytree`, which preserve mode bits by default, and that change was intentional (a regression test was added specifically to keep permissions preserved going forward). So `pytest-datafiles` is behaving exactly as intended; the bug is in the assumption made on the consuming side, inside `buildstream`'s own shared test helpers. ## Suggested fix In `add_plugins_conf` (and any other shared `_sourcetests` helper that writes into a `datafiles`-provided path), don't assume the copy is writable — e.g. `chmod` the relevant directory (or just its parent) to be owner-writable before writing into it, rather than relying on the `datafiles` copy's permissions matching the installed package data's permissions. ## Environment - `buildstream` 2.8.1 - `buildstream-plugins` 2.8.0 - `pytest-datafiles` 3.0.1 - `pytest` (via nixpkgs' `pytestCheckHook`) - Reproduced via `nixpkgs` (nix-build -A python314Packages.buildstream-plugins), where `/nix/store` paths are always installed read-only (mode 444 for files, 555 for directories) — https://github.com/NixOS/nixpkgs/pull/510073 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
