Kit Yan Choi <k...@kychoi.org> added the comment:
I think Python does differentiate "test error" and "test failure" such that a test outcome state can be one of these: success, failure, error, skipped. One could refine these to six: expected success, unexpected success, expected failure, unexpected failure, error, skipped. For example, in the documentation for failureException: * failureException: determines which exception will be raised when the instance's assertion methods fail; test methods raising this exception will be deemed to have 'failed' rather than 'errored'. Another evidence: unittest.runner.TextTestResult, there are methods called "addSuccess", "addError", "addFailure", "addSkip", "addExpectedFailure" and "addUnexpectedSuccess". For example, this test outcome is marked as "FAILED": def test(self): x = 1 y = 2 self.assertEqual(x + y, 4) ====================================================================== FAIL: test (test_main.T) ---------------------------------------------------------------------- Traceback (most recent call last): File "test_main.py", line 9, in test self.assertEqual(x + y, 4) AssertionError: 3 != 4 But the test outcome for this test is "ERROR": def test(self): x = 1 y = 2 + z # NameError self.assertEqual(x + y, 4) ====================================================================== ERROR: test (test_main.T) ---------------------------------------------------------------------- Traceback (most recent call last): File "test_main.py", line 8, in test y = 2 + z # NameError NameError: global name 'z' is not defined The issue here being "expectedFailure" converting "error" to "success", which is not expected, and is causing decorated tests to become unmaintained. While the rest of unittest differentiates "error" and "failure", expectedFailure does not. This is either a bug in the behaviour of expectedFailure, or a bug in the documentation for not being clear on the fact that unexpected error will be considered as expected failure (which I think is wrong). ---------- _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue38296> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com