Hello everyone,
I’m having trouble deciding on what is the best way to perform tests in
Python packages. Here are my thoughts and the solution I find most
correct, but which is kind of verbose and ugly to read. I’d be happy
about comments. TL;DR and code at the bottom.
For a Python package test to be reliable and representative of what will
be installed on a real Fedora system, I think it should be
* performed by the correct Python binary, i.e., %{python3}
* using the system-installed Python modules
* easy to read in the specfile
I found that running tests can go wrong in subtle ways:
* The wrong Python binary is used
* Python modules in the main build directory are used, not the ones
installed by the %install step – even a mix of both is possible
EXAMPLE
I’m assuming this common (for our use case unhelpful) packaging
structure where the Python modules and the test module are side-by-side
in the main directory:
package/
|- module/
|- tests/
|- setup.py
|- tox.ini
|- …
This example uses tox to call the twisted trial test runner, which can
be called as Python module or as a standalone script. It should be
similar to other test runners. Here are approaches to running tests
which are NOT working properly with this simple %check section:
%check
sed -i 's|python -m twisted\.trial|…|' tox.ini
%tox
Broken 1: tox calling `python -m twisted.trial` (upstream default)
Used Python: .tox/py/bin/python :(
sys.path[0]: cwd
Uses system modules: NO
Finds tests module: YES
Broken 2: tox calling `%{python3} -m twisted.trial`
Used Python: /usr/bin/python3
sys.path[0]: cwd
Uses system modules: NO
Finds tests module: YES
Broken 3: tox calling `trial-3`
Used Python: /usr/bin/python3
sys.path[0]: /usr/bin
Uses system modules: YES
Finds tests module: NO
A first, still fragile, solution is choosing the calling method and
PYTHONPATH in a way that the order of module discovery is “first system
modules, then cwd”:
%check
# Call test runner by its script, thus using the correct python and *avoiding*
# sys.path[0] being the cwd.
sed -i 's|python -m twisted\.trial|trial-3' tox.ini
# Prepend pwd to have access to the test module.
PYTHONPATH=%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib}:$PWD
%tox
This works, but the modules in the main directory are only shadowed.
Thus, they might interfere with the tests if, e.g., the installation
skips modules. This can be fixed by
* moving to a new directory, copying only the test module folder over, or
* removing all modules in the current working dir (except the tests).
I arbitrarily chose removing for the following solution.
TL;DR
With all previously said, I’d arrive at using the following lines for
running my Python tests:
%check
# Make sure we use the correct python and and sys.path[0] is the cwd (so the
# test module *is* found).
sed -i 's|python|%{python3}' tox.ini
# Remove all Python modules from cwd, except the test module, to avoid their
use.
find . -maxdepth 2 -name '__init__.py' -exec dirname {} \; | grep -v ^\./tests
| xargs rm -rf
# Handles populating PYTHONPATH with system paths.
%tox
Are my raised concerns valid? Are there already solutions I’m missing?
Should there be documentation/rpm macros to help with this?
Thanks for your input!
Kai
_______________________________________________
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
