I agree that running the tests wherever practical is the best practice. I do put my time where my mouth is—I maintain 17 packages that now use the pyproject-rpm-macros, and you’ll find that in general I’ve added a lot of %check sections to packages that were previously lacking them.
However, let me point to pipx as an example where I recently gave up on maintaining an ever-shifting list of 50 tests that require interaction with PyPI (https://src.fedoraproject.org/rpms/pipx/c/55c6c9debf3039747327686a2f07dc31cd1e966b?branch=rawhide). It just wasn’t worth it to keep doing this in order to run the small minority of tests that can work offline. Even worse, upstream recently added an offline bundle of a huge number of PyPI wheels to make these tests reproducible offline, but this bundle is arch-specific with compiled binary wheels (a non-starter for including as a Source# file), and it would now require a nontrivial patch to run ANY of the tests without either the wheel bundle or a network connection. Now, running these fussy tests in CI would be great! I would accept a reasonable PR to do so. But I’m not excited about learning Yet Another YAML-Based Programming-As-Configuration Language being an absolute requirement in order to maintain packages that have impractical upstream tests like these. There are an increasing number of upstreams with test suites that are really only designed to run in upstream’s own fickle CI, and reproducing these environments could be quite a burden. Realistically, I picked up pipx and cleaned it up after it was orphaned, and I’m likely just to file a Bugzilla and orphan it again if CI is mandated. Unfortunately, I suspect there are many packagers who would choose quiet noncompliance instead. Another example is python-strictyaml. It builds its documentation using an idiosyncratic build system (see https://hitchdev.com/) that is hopelessly entangled with the idea of downloading dependencies from PyPI. An offline documentation build is therefore nearly impossible. Currently, it has no tests upstream, but if it ever adds them, they will be run via the same system and will also be effectively impossible to run offline. A third example is python-pgspecial. Most of the tests require a postgresql database, which I cannot set up in %check because this requires root privileges. This package actually seems like it would be compliant today, because it has a few tests that work without the database connection, and automatically skips the rest—but I am sure there are others where every single test requires a privileged operation. That would be the case for python-databases if it did not support SQLite. A fourth example is python-absl-py. There are a couple of “smoke tests” that I can and do run, but the core test suite requires Bazel—which, as has been discussed before, will probably never be successfully packaged for Fedora. I’m still in favor of running every test that is even vaguely practical in %check, but upstream Python packaging practices are wildly diverse (arguably, a mess) and it seems like a strongly worded SHOULD with a fallback of “trust the packager” would be a better approach than forcing packagers to build complicated CI configurations and go to great lengths to implement and debug network-connected tests they cannot reproduce locally. It’s also not clear to me why the Python guidelines should be so much stricter than the overall Fedora guidelines (https://docs.fedoraproject.org/en-US/packaging-guidelines/#_test_suites) in this area. _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
