[issue31227] regrtest: reseed random with the same seed before running a test file

[STINNER Victor]

STINNER Victor  added the comment:

I didn't get a strong +1 on the issue and I'm not convinced myself by my 
approach. Moreover, Refleaks buildbots now seem to be reliable thanks to other 
fixes. For all these reasons, I close the issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31227] regrtest: reseed random with the same seed before running a test file

[STINNER Victor]

Change by STINNER Victor :


--
resolution:  -> rejected
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31227] regrtest: reseed random with the same seed before running a test file

[STINNER Victor]

STINNER Victor added the comment:

> The exact behavior depends on the order of files in directories, on string 
> hashes randomization, on address randomization, and on many other things out 
> of our control.

For hash randomization, maybe we need to generate a PYTHONHASHSEED, as tox test 
runner does.

For the filesystem: right, it's not possible to get 100% reproductible tests, 
but IMHO it's worth it to make them more reliable.

> Couldn't reseeding the PRNG just add a false promise?

I'm trying to fix random failures on the Refleaks buildbots, not to promise 
anything :-) To be honest, at this point, I don't know if it would be enough 
since I'm unable to reproduce bugs...

> The success in making tests deterministic can also narrow down the coverage 
> of the testing. Some branches that lead to failures can be never executed. 
> Our target not just making tests always success, but catch and fix even 
> pretty rare errors.

I know that it's though question, and that's why I opened this issue, to 
discuss it :-)

But I see more and more projects to get more reproductible softwares and tests:

* https://reproducible-builds.org/
* systemd big project to get more "stateless" computers, or said differently: 
to isolate better services
* containers which also want to isolate services, "stateless" containers
* etc.

Other test runners, like tox, also make efforts to get reproductible tests, 
like setting PYTHONHASHSEED.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31227] regrtest: reseed random with the same seed before running a test file

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

The PRNG is not the only source of the randomness in the tests. The exact 
behavior depends on the order of files in directories, on string hashes 
randomization, on address randomization, and on many other things out of our 
control. Couldn't reseeding the PRNG just add a false promise? The success in 
making tests deterministic can also narrow down the coverage of the testing. 
Some branches that lead to failures can be never executed. Our target not just 
making tests always success, but catch and fix even pretty rare errors.

--
nosy: +ezio.melotti, mark.dickinson, michael.foord, rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31227] regrtest: reseed random with the same seed before running a test file

[STINNER Victor]

STINNER Victor added the comment:

I'm not sure if we should use the same RNG seed for all tests, or create one 
seed per test when the option -r is used.

For example, I expect that "./python -m test -r -F test_tools" will catch a 
random bug which only occurs for a specific random seed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31227] regrtest: reseed random with the same seed before running a test file

[STINNER Victor]

STINNER Victor added the comment:

Antoine Pitrou: "If refleaks depend on the random seed, perhaps it's a bug 
worth fixing?"

I propose to change regrtest behaviour even when -R is not used, to make 
regrtest more deterministic.

Currently, when you run "./python -m test -r test_xxx test_", it's hard to 
guess the state of the RNG in test_yyy: it depends on many bytes were consumed 
by test_xxx. For example, if test_xxx is run on a buildbot, but skipped when I 
run it locally: we get a different behaviour.

I would prefer that test_yyy behaves the same when run with "./python -m test 
-r --randseed=5 test_xxx test_" (with test_xxx) and with "./python -m test 
-r --randseed=5 test_" (without test_xxx).

With my change, "./python -m test -r --randseed=5 test_ test_" 
(sequential) and "./python -m test -r --randseed=5 -j2 test_ test_" 
(parallel) runs test_yyy twice with the RNG in the same state.

Proposed change is part of a more global project to reduce side effects of 
tests, to make tests more reproductible and more "isolated".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31227] regrtest: reseed random with the same seed before running a test file

[Antoine Pitrou]

Antoine Pitrou added the comment:

If refleaks depend on the random seed, perhaps it's a bug worth fixing?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31227] regrtest: reseed random with the same seed before running a test file

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +pitrou, serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31227] regrtest: reseed random with the same seed before running a test file

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +3159

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31227] regrtest: reseed random with the same seed before running a test file

[STINNER Victor]

New submission from STINNER Victor:

Attached PR changes regrtest to reseed the random RNG before each test file. 
Use also more entropy for the seed: 2**32 (32 bits) rather than
10_000_000 (24 bits).

The change should avoid random failure of test_tools when hunting reference 
leaks: see bpo-31174.

Maybe it will also reduce false positive when hunting memory leaks, like 
bpo-31217.

--
components: Tests
messages: 300438
nosy: haypo
priority: normal
severity: normal
status: open
title: regrtest: reseed random with the same seed before running a test file
versions: Python 2.7, Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com