On 11/26/2016 02:31 PM, Ross Vandegrift wrote:
> On Sat, Nov 26, 2016 at 02:30:59AM +0100, Christian Seiler wrote:
>>> 2) Is there a common pattern for handling upstream tests that break this
>>> rule? Maybe there's an alternative to disabling them?
>>
>> If upstream tests do that, I would suggest sending a patch
>> upstream that fixes them, because especially for tests I
>> would consider this a bug.
>>
>> That said, if tests just require stuff in the home directory
>> you could set the HOME environment variable to a temporary
>> directory within the build tree before you run the tests, to
>> work around this kind of problem. Nevertheless I would consider
>> those tests buggy and would want to patch them.
>>
>> If you could give a couple of examples of what exactly you're
>> thinking of, maybe my answer could be more specific.
>
> A library service creates local sockets. The library provides a
> fallback mechanism for the socket location - first try $XDG_RUNTIME_DIR,
> second try $HOME, finally use $TMPDIR. Most of the tests unset the
> first two and go straight to TMPDIR. But to test the fallback mechanism
> itself, two tests do not.
>
> As a workaround, I disabled these. But it was suggested to instead set
> HOME=/tmp, XDG_RUNTIME_DIR=/tmp. Seems clever, but I wasn't sure if
> this was permitted.
Well, you could also do the following before running the tests (as a
bash script; how you integrate that is up to you):
cleanup() {
[ -n "$temporary_HOME" ] && rm -r "$temporary_HOME"
[ -n "$temporary_XDG_RUNTIME_DIR" ] && rm -r "$temporary_XDG_RUNTIME_DIR"
}
trap cleanup EXIT
temporary_HOME="$(mktemp -d)"
temporary_XDG_RUNTIME_DIR="$(mktemp -d)"
HOME="$temporary_HOME" XDG_RUNTIME_DIR="$temporary_XDG_RUNTIME_DIR" ./run_tests
That way you'd not be using /tmp directly (bad idea to pollute that
directly), and you'd have two different directories, to be sure that
the fallback actually works.
Also, if setting HOME is not enough (because the software reads the
home directory directly from the NSS database, e.g. /etc/passwd), then
you could use nss_wrapper for that, see https://cwrap.org/nss_wrapper.html
That was specifically designed for tests to provide a different
environment. In general CWrap is very nice for tests that integrate into
the system a bit deeper: https://cwrap.org/
Finally, some tests you may not want to execute during build time.
There are also runtime tests in Debian, called autopkgtests, and there
is automated infrastructure in place to run them regularly. Debian's
infrastructure uses LXC to isolate these tests, so in those tests you
can in fact write anywhere you want if that really is required (as
long as you declare things such as the proper isolation level and
possibly breaks-testbed). See also:
https://ci.debian.net/doc/
I would in fact recommend using some kind of autopkgtest in general,
even if you can run the unit test suite during build time - since the
autopkgtests are more related to integration testing instead of pure
functionality. (You would run different tests, obviously.)
For example, a web server package could contain unit tests that would
start the web server on localhost on a random port during build time
to see if it responds correctly to requests, whereas an autopkgtest
for the same package would test for example whether the webserver is
started properly after package installation and listens on the correct
port. The autopkgtest would have the proper isolation level specified
(isolation-container in this case) to make sure that this does not
interfere with the system the test is run on.
Regards,
Christian
