On Friday, March 12, 2021 at 10:47:56 AM UTC+1 Florian Apolloner wrote: > Is the current order defined in any way -- if not one should consider it > randoms anyways and as such actually randomizing should not make much of a > difference. FWIW I am fine with making it the default and outputting the > seed at the end of the run so it can reproduced. I am curious if it finds > many remaining issues. >
I would characterize it as "mostly" sorted -- I think. I believe unittest's test discovery is used by default, in conjunction with a couple large test groups that Django configures (`DiscoverRunner.reorder_by = (TestCase, SimpleTestCase)`). The unittest package's test discovery is documented as first sorting by file path: > Paths are sorted before being imported so that execution order is the same > even if the underlying file system’s ordering is not dependent on file name. https://docs.python.org/3/library/unittest.html#unittest.TestLoader.discover And within test cases, test methods are sorted by name. For example, see-- https://docs.python.org/3/library/unittest.html#unittest.TestLoader.getTestCaseNames and-- > Note: The order in which the various tests will be run is determined by > sorting the test method names with respect to the built-in ordering for > strings. https://docs.python.org/3/library/unittest.html#organizing-test-code I don't know about classes within modules. On Fri, Mar 12, 2021 at 1:53 AM Florian Apolloner <f.apollo...@gmail.com> wrote: > ... Having randomization in CI might be interesting, but then we probably > need a way to derive the salt from the PR number or so, because I think the > number of tests inside a PR is small enough that you probably do not see > isolation failures, but you really want to see progress in a PR without > random failures due to isolation. > The question of how to use seeds in conjunction with CI if randomization were the default is an interesting one (e.g. how to tell CI to repeat a particular seed). Certainly if randomization were the default, CI could still be configured to be deterministic by passing --no-random / --sorted or whatever the opt-out option would be called. But, regardless of the default, to get the full benefit I think you'd want at least one of the test runs in CI to be a random one. --Chris > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAOTb1wfbbdFbauJXCtZwa_N7F-K5dMcB-Kw4EgLiwKgL_MRCsw%40mail.gmail.com.
