Hi,
PKGBUILDs include the check() function which allows one to run tests on the
software after it has been built, but before it has been packaged.
This message proposes to add a test() function, which essentially allows to run
tests after package(), and in particular after the package has been installed
in a chroot (for these testing purposes).
This allows the packager to run tests against the final installation (or as
final as can be in a chroot). Between package() and test(), the resulting pkg
is installed in a chroot like on the regular system (i.e. /usr/bin instead of
$pkgdir/usr/bin). Everything but the pkg files and the test files is removed,
so only the installation is under test.
The idea for this came from the issue that Python packages usually don't need a
build or rather, that that build can't be tested without fully installing it.
This installation however isn't really possible since /usr/ is inaccessible for
obvious reasons. Workarounds usually involve setting the PYTHONPATH, but that's
tedious at best and ultimately unnecessary.
Because Python has an extensive auto-detection of libraries and dependencies,
the tests can be executed easily, if Python can find them. And after
installation (in /usr/lib/python*/site-packages/), the package is easily found.
This of course is in contrast to most compiled languages, who would require
lots of paths and links in order to be able to compile the tests. It shouldn't
be this way, but it is.
A slight complication arises from the fact that tests can theoretically be
anywhere in the python source code. So either makepkg figures out what is a
test, or the packager specifies that. In most cases though, all tests are in a
single directory, and the edge cases are vast, so the latter case with some
boilerplate is probably better.
In total, a PKGBUILD might look something like this (names are subject to
change):
```
build() {
cd ${pkgname}-${pkgver}/
python -m build --wheel --no-isolation
}
check() {
cd ${pkgname}-${pkgver}/
python -m pytest
}
package() {
cd ${pkgname}-${pkgver}/
python -m installer --destdir="$pkgdir" dist/*.whl
}
collect() {
cd ${pkgname}-${pkgver}/
# collects the tests and puts them into $testdir
python -m pytest --co | grep -o "[^ ]*\.py" | xargs cp -t $testdir
}
# inbetween collect() and test(), the package is installed in the chroot and
$testdir is copied
test() {
# we are in $testdir (e.g. /home/builduser)
pytest
}
```
Notice the line in collect() which is responsible to collect the tests. We
basically ask pytest what it wants to run, then filter out the files, and
lastly copy them into $testdir, which is ultimately copied over and run.
In most cases, this line might just be something like `cp tests/ $testdir/`.
------------
Makepkg
------------
This post is already quite long, so I'll make it quick with some implementation
thoughts.
Since collect() and test() always run together or don't run together (test()
needs collect() and only collect() has no effect), only a single option is
necessary to toggle whether to run the post-package tests.
The order of package() and collect() conceptually doesn't matter, but collect()
should probably run after to guarantee that it does not influence package().
Either case, the packager is responsible.
This is a pure addition, so if collect() and test() are missing, then just
nothing happens. If only collect() is implemented, that might be ignored too.
If only test() exists, then an error is probably most helpful, since no
packages are copied. Alternatively, maybe makepkg does some limited test
identification (like looking for tests/ folder). Or just nothing happens
because the packager made a mistake. Should be obvious enough.
In conclusion, two new methods are introduced to allow for better and more
precise testing of the final output, as well as easier testing in case of
Python.
The first function - collect() - collects all the tests in the codebase and
copies them into $testdir, while package() creates the pkg in $pkgdir. After
that, a chroot with all depends packages is created and the pkg installed
therein. $testdir is also copied there. Lastly, the second function - test() -
is called and executed to test the installation.
Thanks for your attention, if you have further ideas or questions, I'd like to
hear them.
If something similar has already been discussed, feel free to point out that
thread. AFAICT, it hasn't been before.
- Lamarrrk
