> > because I think the number of tests inside a PR is small enough that you > probably do not see isolation failures, but you really want to see progress > in a PR without random failures due to isolation. >
I think it's the opposite - most isolation problems are best detected at the time they're introduced. So using a different seed for each run of a PR as it's updated will help flush out any newly authored isolation problems before merging. It's highly more likely that running tests in reverse order will catch > isolation issue for you than running them in a non-deterministic order. > There's actually a study on this, across Apache projects - see the link in https://github.com/adamchainz/pytest-reverse#history . They found that random is best, with reverse being almost as good. Reverse order fails to find three-way-plus isolation failures, e.g. in tests A, B, C where B depends on a side-effect triggered in both A and C. Both the ABC and CBA orders allow the side effect to leak into test B so the failure is not detected. We have to run it as often with as many different seeds as possible to have > an effect. The space of tests is so big that I am not sure we'd find > something otherwise (though I have to agree that is theoretical thinking I > did not try in practice). > Only two randomly sorted test runs are needed to flush out most pairwise problems. For example, imagine test B depends on a side effect of test A. Each random test run has a 50% order of running A then B, or B then A. So the expected number of test runs to see the isolation failure is 2. Yes, one could see 32 test runs in a row without the failure, but the chance of that is 1 in 2^32. For three-way isolation problems, you might expect more runs, depending on how many of the 6 possible configurations of 3 tests reveal the problem. For four-way it could be worse again. But at least such isolation problems will appear eventually, as opposed to never. On Fri, 12 Mar 2021 at 17:02, Florian Apolloner <f.apollo...@gmail.com> wrote: > Thank you for diving into how unittest discovery tests! > > On Friday, March 12, 2021 at 12:14:05 PM UTC+1 chris.j...@gmail.com wrote: > >> But, regardless of the default, to get the full benefit I think you'd >> want at least one of the test runs in CI to be a random one. >> > > This is were I somewhat disagree. We have to run it as often with as many > different seeds as possible to have an effect. The space of tests is so big > that I am not sure we'd find something otherwise (though I have to agree > that is theoretical thinking I did not try in practice). > > Cheers, > Florian > > -- > You received this message because you are subscribed to the Google Groups > "Django developers (Contributions to Django itself)" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-developers+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/4034f3db-e6ee-4d78-b140-553c73c1a170n%40googlegroups.com > <https://groups.google.com/d/msgid/django-developers/4034f3db-e6ee-4d78-b140-553c73c1a170n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Adam -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMyDDM1fk1AGcE%3Dm9zq3UrOdZ%3D2EPsstSgDOjmOzyCekdSp%2BbA%40mail.gmail.com.
