New issue 485: xdist: --boxed test crash throws away STDOUT and STDERR even if
they have data
https://bitbucket.org/hpk42/pytest/issue/485/xdist-boxed-test-crash-throws-away-stdout
Ethan John:
I'm using pytest 2.5.2 and xdist 1.10 under CentOS 6.2.
We use py.test --boxed for test isolation and resiliency to underlying crashes.
We use Cython extensively and various underlying libraries written in C, and we
were running into situations where py.test was reporting that processes crashed
(usually with signal 6, ABORT due to assertion failure) and not leaving
anything else behind with which to track down the problem.
After fixing our core file generation permissions and discovering where the
processes were hanging, it was clear in reproductions outside of py.test that
the processes were generating output that our test reports were not seeing.
Some digging into xdist's code shows the reason: If a test function crashes,
the result returned from plugin.py:forked_run_report():ff.waitfinish() does not
have a retval (because it was never returned), and therefore the
report_process_crash() method executes and simply reports the result.signal.
This is fine in the default case where py.test is run with --capture fd,
because result.err and result.out will be empty (I also think this is
incorrect, but it's nontrivial to fix given my reading of the source).
However, in the event that a user runs with --capture sys, then result.out and
result.err do indeed have useful information in them, but this data is thrown
away.
This is trivial to reproduce with a simple C file that just asserts:
{noformat}
#include <assert.h>
int main() {
assert(0);
}
{noformat}
And a degenerate test file that can be executed after compiling the above:
{noformat}
#!/usr/bin/env crpython
import os
import unittest2 as unittest
class TestAbrt(unittest.TestCase):
def testit(self):
os.execvp('./a.out', ['a.out'])
if __name__ == "__main__":
unittest.main()
{noformat}
What I would expect:
# A process crash with --boxed should not be considered a failure! It should be
a test error and relevant information like stdout and stderr should be
communicated to the user.
#* Ideally, test results should not be "all or nothing" in the world of --boxed
even if --capture is specified as "fd". It should be possible to read input
from child tests even in the event of failure, at least in the UNIX world.
Maybe --boxed should override the default --capture behavior in order to
capture as much as possible? Without xdist the --capture flag seems useful, but
with it, more advanced behavior is possible.
# If a crash continues to be classified as a failure, then any status
information collected by the test report should be included in the failure
report. Assuming that result.err and result.out are empty seems incorrect.
Thanks!
_______________________________________________
pytest-commit mailing list
[email protected]
https://mail.python.org/mailman/listinfo/pytest-commit
