On 3/6/2019 3:12 PM, Matěj Cepl wrote:
Hi,
I am a lead maintainer of Python packages in OpenSUSE and I can
see the pattern of many packagers adding blindly
python setup.py test
to %check section of our SPEC file.
I am not familiar with setup.py, so I don't know how this affects the
presence and contents of any particular files.
The problem is that if the
package doesn't use unittest (it actually uses nose, pytest or
something), it could lead to zero found tests,
Hence I don't know how unittest might be invoked in the situation you
describe nor what output you see and whether you mean 0 test file(s)
found or 0 test methods found or 0 lines of test code executed.
which pass and
Python returns exit code 0 (success) even though nothing has been
tested.
0 test methods does not mean 0 code executed in the tested module. Here
is a possible minimal test file test_mod that is better than nothing.
import mod
import unittest
class MinTest(unittest.TestCase):
def setUp(self):
self.instance = mod.MainClass()
It seems from the outside that everything is all right,
package is being tested on every build, but actually it is lie.
Unless a test covers 100% of both lines *and* logic, 'success' never
means 'everything is all right'.
Would it be possible to change unittest runner, so that when 0
tests pass, whole test suite would end up failing?
Yes, but unless a change were very narrow, and only affected the
particular situation presented, it would be a bad idea. The unittest
system is premised on 'success' rather than 'failure' being the default.
1. A test file may do better-that-nothing testing without running a test
method. See above. Calling a mininal pass a 'fail' would be wrong.
2. A test file should skip everything when running on a system that
cannot runs the tests. Several stdlib modules are OS-specific; their
test modules skip all tests on some OS. There is no OS that can run
every file in the Python test suite. Skipped test modules must not fail
the test suite.
IDLE and tkinter require graphics hardware and are then optional. IDLE
depends on idlelib and tkinter. Tkinter depends on _tkinter and tcl/tk.
Tk depends on having a graphic system, which servers and, in
particular, *nix buildbots, generally lack. Again, skipped IDLE and
tkinter test.test_x files must not fail a test suite.
I agree that labeling the result of running a single test file can be
problematical. The following could be either a 'SUCCESS' or 'FAIL',
depending on what one wanted and expected. So one should read the
detail and judge for oneself.
0:00:00 [1/1] test_idle
test_idle skipped -- No module named 'idlelib' # or tkinter or ...
test_idle skipped
== Tests result: SUCCESS ==
1 test skipped:
test_idle
Total duration: 109 ms
Tests result: SUCCESS
Unittest effectively assumes the context 'test file in test suite'.
--
Terry Jan Reedy
_______________________________________________
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
