Btw while I am fine with making it default (I really do not care much either way), we certainly need a way to disable it and keep the current algorithm (even though it might not be guaranteed stable). When I develop I often run one suite and fix the errors as they occur. Having a stable order there ensures that the tests I fixed before are still fixed when the next test fails (I quite often also run with failfast there). I think having a stable test output can be useful for development. Having randomization in CI might be interesting, but then we probably need a way to derive the salt from the PR number or so, because I think the number of tests inside a PR is small enough that you probably do not see isolation failures, but you really want to see progress in a PR without random failures due to isolation. On Friday, March 12, 2021 at 10:47:56 AM UTC+1 Florian Apolloner wrote:
> Is the current order defined in any way -- if not one should consider it > randoms anyways and as such actually randomizing should not make much of a > difference. FWIW I am fine with making it the default and outputting the > seed at the end of the run so it can reproduced. I am curious if it finds > many remaining issues. > > On Thursday, March 11, 2021 at 12:15:44 AM UTC+1 chris.j...@gmail.com > wrote: > >> For those that would rather it be opt-out, remember that we can always >> make it opt-out later, once we have more experience with the feature. >> >> Regarding the suggestion to have `--random 0` mean not to use >> randomizing, I also think that could be confusing. 0 is also a valid >> integer seed, and it seems nice to have the option of passing 0 as a simple >> seed if one wants. We could use --no-random for disabling if we ever wanted >> to make the feature opt-out in the future. >> >> --Chris >> >> >> >> On Wednesday, March 10, 2021 at 4:42:08 AM UTC-8 Mariusz Felisiak wrote: >> >>> I usually agree with new features being opt-in, but perhaps this case is >>>> different? >>>> >>>> If I had tests that are breaking if executed randomly, I’d want to know >>>> about it yesterday. IOW, I’m having difficulty imagining a scenario where >>>> the user would be thankful for not activating this feature by default. So >>>> personally, I’d like to see an opt-out setting for this in settings.py. >>>> >>>> /$0.02 >>>> Fran >>>> >>> >>> `--reverse` will catch 95% of test isolation issues for you. It's >>> highly more likely that running tests in reverse order will catch isolation >>> issue for you than running them in a non-deterministic order. >>> >>> >>> -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/4ec1e24a-917b-4256-9949-6f5deded1a80n%40googlegroups.com.
