[issue20910] Make sleep configurable in tests

Changes by STINNER Victor : -- resolution: -> out of date status: open -> closed ___ Python tracker ___

[issue20910] Make sleep configurable in tests

STINNER Victor added the comment: I created #20964 to add the support.check_time_delta() function. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20910 ___

[issue20910] Make sleep configurable in tests

Roundup Robot added the comment: New changeset d1c80a5b42ad by Victor Stinner in branch 'default': Issue #20910: Make tests more reliable, less dependent on time http://hg.python.org/cpython/rev/d1c80a5b42ad -- nosy: +python-dev ___ Python tracker

[issue20910] Make sleep configurable in tests

Andreas Schwab added the comment: test_io.CSignalsTest.check_interrupted_write can fail in an emulated environment when the alarm races with the write call. -- nosy: +schwab ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20910

[issue20910] Make sleep configurable in tests

Andreas Schwab added the comment: Also, sometimes tests fail in non-verbose mode, but succeed in verbose mode. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20910 ___

[issue20910] Make sleep configurable in tests

Zachary Ware added the comment: Victor Stinner wrote: Is it a virtual machine or a physical machine? Was your Windows busy? Did you run tests in parallel? Physical, not really other than the tests, and I ran with -j0 (on a machine with 2 single core CPUs). I'm not sure what I think of the

[issue20910] Make sleep configurable in tests

STINNER Victor added the comment: Victor Stinner wrote: Is it a virtual machine or a physical machine? Was your Windows busy? Did you run tests in parallel? Physical, not really other than the tests, and I ran with -j0 (on a machine with 2 single core CPUs). Ah yes, I missed the -j0. I

[issue20910] Make sleep configurable in tests

New submission from STINNER Victor: Hi, I'm trying to make Python test suite reliable, but it's hard because the speed of our farm of buildbots varies a lot. The whole test suite takes less than 10 minutes on some buildbots (or event less than 5 minutes), while it took 10 hours on other

[issue20910] Make sleep configurable in tests

STINNER Victor added the comment: The test suite pass with test_sleep.patch applied on my fast Linux PC. The test must be tested on Windows (because of the clock resolution of 15.6 ms). I run the unit test using the command ./python -m test -j0 -rW. By the way, do you have an idea to simulate

[issue20910] Make sleep configurable in tests

R. David Murray added the comment: Would it be possible for regrtest to do some operations to estimate the speed and choose good values for the constants automatically? If you have the right kind of VM setup, you could simulate a slow buildbot by giving the VM a low CPU share and then running

[issue20910] Make sleep configurable in tests

STINNER Victor added the comment: Would it be possible for regrtest to do some operations to estimate the speed and choose good values for the constants automatically? It might be possible to make the parameters dynamic, for example using the system loads. But I'm not really interested to

[issue20910] Make sleep configurable in tests

STINNER Victor added the comment: The test must be tested on Windows (because of the clock resolution of 15.6 ms). The test suite pass on Windows too (PCbuild\python_d.exe -m test). -- ___ Python tracker rep...@bugs.python.org

[issue20910] Make sleep configurable in tests

Antoine Pitrou added the comment: It might be possible to make the parameters dynamic, for example using the system loads. But I'm not really interested to invest too much time on optimizing the timing parameters for buildbots. I would like to ensure that tests pass with a short test sleep

[issue20910] Make sleep configurable in tests

Zachary Ware added the comment: I got a few failures on my Windows machine running `PCbuild\python_d.exe -m test -j0 -rW` [1], only one of which looks relevant [2]: == FAIL: test_monotonic (test.test_time.TimeTestCase)

[issue20910] Make sleep configurable in tests

STINNER Victor added the comment: AssertionError: timing 608.0 ms seconds max timing 500.0 ms seconds; the clock 'monotonic' has a resolution of 15.6 ms; tolerate 1.0 ms seconds Is it a virtual machine or a physical machine? Was your Windows busy? Did you run tests in parallel? Only

[issue20910] Make sleep configurable in tests

STINNER Victor added the comment: more_reliable_tests.patch: Work-in-progress patch to make some tests more reliable with very short test sleep (I tested with 1 ns on my Linux box). -- Added file: http://bugs.python.org/file34402/more_reliable_tests.patch

[issue20910] Make sleep configurable in tests

STINNER Victor added the comment: Other tests should use check_time_delta(): test_wait_for_handle() of Lib/test/test_asyncio/test_windows_events.py test_call_later() of Lib/test/test_asyncio/test_events.py test_run_until_complete() of Lib/test/test_asyncio/test_events.py --

[issue20910] Make sleep configurable in tests

Antoine Pitrou added the comment: I don't like these patches at all. It is not obvious that making sleep times configurable improves things at all, and it's one more complication in the test suite that people will have to care about. -- nosy: +neologix

[issue20910] Make sleep configurable in tests

Ned Deily added the comment: I agree with Antoine. There is never going to be a right value for sleep: the behavior of the buildbots can vary so much from run to run, especially if there are multiple VMs running on one host. If anything, we should be working to remove as many sleep

[issue20910] Make sleep configurable in tests

Serhiy Storchaka added the comment: How much this patch speeds up testing? Especially interesting results for medium-speed buildbots (about a hour). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20910

[issue20910] Make sleep configurable in tests

STINNER Victor added the comment: I don't like these patches at all. It is not obvious that making sleep times configurable improves things at all, and it's one more complication in the test suite that people will have to care about. The default timings should choosen to work on most

[issue20910] Make sleep configurable in tests

STINNER Victor added the comment: How much this patch speeds up testing? Especially interesting results for medium-speed buildbots (about a hour). Quick benchmark, I only ran the test once. Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz (4 cores with HT: 8 logical cores) and 12 GB of RAM. I tested

[issue20910] Make sleep configurable in tests

Nick Coghlan added the comment: I've sped up wait for something to happen tests in other contexts and found the best solution to be a polling loop so that success is as fast as possible and failure may be slow. For example, rather than just having a 10 second timeout, it's often better to have

[issue20910] Make sleep configurable in tests

STINNER Victor added the comment: I've sped up wait for something to happen tests in other contexts and found the best solution to be a polling loop so that success is as fast as possible and failure may be slow. more_reliable_tests.patch tries to fix tests to not depend on timings. But it is

[issue20910] Make sleep configurable in tests

Raymond Hettinger added the comment: Some tests use time.sleep() as a basic synchronization primitive, because other synchronization primitive cannot be used. For example, lock_tests.py tests locks and so cannot use lock in its tests. The problem is that slow buildbots require long sleep. It