Terry J. Reedy <tjre...@udel.edu> added the comment:
This issue confuses two different testing issues. First is selection of test data before the test versus generation while testing (usually 'randomly') Second is how the function is tested. All tests test some property of the inputs and function, but it is useful to distinguish between testing that a function produces a specific expected output versus testing something more abstract or defined in terms of the inputs. Example 1: 2 + 3 = 5 (a + b = (known) c) versus 2 + 3 = 3 + 2 (a + b = b + a) and (2 + 3) + 4 = 2 + (3 + 4). Example 2: sorted([3,5,1,4]) = [1,3,4,5] versus is_sorted(sorted([random list]). The latter are what hypothesis people mean by 'property tests'. Such property tests can be done with either fixed or random pairs in unittest module tests. The test that a module compiles is a weak property test, but better than nothing. A test that that all lines of code runs at least once without an unexpected exception a stronger property test. I consider it roughly on a par with min(s) <= mean(s) <= max(s). The connection between randomized input testing, such as with hypothesis, and property tests is that with random inputs, one cannot test f(input) against an 'expected' value unless one has an alternate means of producing the expected value. It is often easier to test a parameterized equation or other properties. The problem with random input tests in not that they are 'flakey', but that they are useless unless someone is going to pay attention to failures and try to find the cause. This touches on the difference between regression testing and bug-finding tests. CPython CI is the former, and marred at that by buggy randomly failing tests. My conclusion: bug testing would likely be a good idea, but should be done separate from the CI test suite. Such testing should only be done for modules with an active maintainer who would welcome failure reports. ---------- _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue42109> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
