Bug#854084: binaryornot: Non-determistically FTBFS due to unreliable timing in tests
Vincent Bernat wrote: > I don't want to spend my Debian time fixing random test suites. Well, I wasn't asking you to :) What do upstream say on this matter? > […] Maybe there is a bug in hypothesis. Right, just one example how it might not be local to binaryornot's test suite. Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `-
Bug#854084: binaryornot: Non-determistically FTBFS due to unreliable timing in tests
❦ 5 février 2017 09:22 +1300, Chris Lamb : >> I have slowed down my own machine in a such ways tests are now taking 6s >> to execute. I cannot reproduce this. > > Interesting, the Reproducible Builds framework can reproduce it, and I > doubt that build node is _that_ slow! > > > https://tests.reproducible-builds.org/debian/logs/unstable/amd64/binaryornot_0.4.0-1.build2.log.gz 1.14 seconds to get 2 random 512-byte string. Maybe there is another problem. First, I thought that maybe the machine is running on low entropy and for some reason, /dev/random is used. But hypothesis uses a seed and no random data after that. Maybe there is a bug in hypothesis. I don't want to spend my Debian time fixing random test suites. I want to keep the test suite as is as it is the best tool for me to detect regressions on new version. I am not one to agree with the fact that test suite should run in all environments, even reasonable ones (like the 1-CPU one Santiago tested). -- The ripest fruit falls first. -- William Shakespeare, "Richard II" signature.asc Description: PGP signature
Bug#854084: binaryornot: Non-determistically FTBFS due to unreliable timing in tests
Hi Vincent, > I have slowed down my own machine in a such ways tests are now taking 6s > to execute. I cannot reproduce this. Interesting, the Reproducible Builds framework can reproduce it, and I doubt that build node is _that_ slow! https://tests.reproducible-builds.org/debian/logs/unstable/amd64/binaryornot_0.4.0-1.build2.log.gz > (otherwise, we have to disable all timing related tests). IMHO tests should be entirely deterministic (in the Martin Fowler TDD tradition) so these are not therefore actually "tests" and thus should be disabled... but that might be just me. Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org / chris-lamb.co.uk `-
Bug#854084: binaryornot: Non-determistically FTBFS due to unreliable timing in tests
Control: severity -1 normal <#secure method=pgpmime mode=sign> ❦ 4 février 2017 09:25 +1300, Chris Lamb : > […] > > == > ERROR: test_never_crashes (tests.test_check.TestDetectionProperties) > -- > Traceback (most recent call last): > File "«BUILDDIR»/tests/test_check.py", line 180, in test_never_crashes > def test_never_crashes(self, data): > File "/usr/lib/python2.7/dist-packages/hypothesis/core.py", line 438, in > wrapped_test > HealthCheck.too_slow, > File "/usr/lib/python2.7/dist-packages/hypothesis/core.py", line 306, in > fail_health_check > raise FailedHealthCheck(message) > FailedHealthCheck: Data generation is extremely slow: Only produced 10 > valid examples in 0.33 seconds (0 invalid ones and 1 exceeded maximum size). > Try decreasing size of the data you're generating (with e.g.average_size or > max_leaves parameters). > See https://hypothesis.readthedocs.io/en/latest/healthchecks.html for more > information about this. If you want to disable just this health check, add > HealthCheck.too_slow to the suppress_health_check settings for this test. > > -- > Ran 43 tests in 0.557s I have slowed down my own machine in a such ways tests are now taking 6s to execute. I cannot reproduce this. I need to slow it down such that tests take 20s to get this error (and it says 10 valid examples in 1.30 seconds and the test suite took 2.9s). Runing the tests in a loop without CPU limitation (tests run in 1.88 seconds) doesn't trigger the bug. I don't intend disabling this test which seems important because there is not enough CPU to run the test (even if this is not a timing-related test). Like for memory and disk, I understand that test suites have some expectation on how much CPU the environment should provide (otherwise, we have to disable all timing related tests). -- "You have been in Afghanistan, I perceive." -- Sir Arthur Conan Doyle, "A Study in Scarlet"
Bug#854084: binaryornot: Non-determistically FTBFS due to unreliable timing in tests
Source: binaryornot
Version: 0.4.0-1
Severity: serious
Justification: fails to build from source
User: reproducible-bui...@lists.alioth.debian.org
Usertags: ftbfs
X-Debbugs-Cc: reproducible-b...@lists.alioth.debian.org
Dear Maintainer,
binaryornot's testsuite appears to use method timing/benchmarking in
such a way that it will non-deterministically FTBFS:
[…]
==
ERROR: test_never_crashes (tests.test_check.TestDetectionProperties)
--
Traceback (most recent call last):
File "«BUILDDIR»/tests/test_check.py", line 180, in test_never_crashes
def test_never_crashes(self, data):
File "/usr/lib/python2.7/dist-packages/hypothesis/core.py", line 438, in
wrapped_test
HealthCheck.too_slow,
File "/usr/lib/python2.7/dist-packages/hypothesis/core.py", line 306, in
fail_health_check
raise FailedHealthCheck(message)
FailedHealthCheck: Data generation is extremely slow: Only produced 10 valid
examples in 0.33 seconds (0 invalid ones and 1 exceeded maximum size). Try
decreasing size of the data you're generating (with e.g.average_size or
max_leaves parameters).
See https://hypothesis.readthedocs.io/en/latest/healthchecks.html for more
information about this. If you want to disable just this health check, add
HealthCheck.too_slow to the suppress_health_check settings for this test.
--
Ran 43 tests in 0.557s
FAILED (errors=1, expected failures=1)
Test failed:
error: Test failed:
E: pybuild pybuild:283: test: plugin distutils failed with: exit code=1:
python2.7 setup.py test
dh_auto_test: pybuild --test -i python{version} -p 2.7 returned exit code 13
debian/rules:7: recipe for target 'build' failed
make: *** [build] Error 25
dpkg-buildpackage: error: debian/rules build gave error exit status 2
[…]
The full build log is attached.
Regards,
--
,''`.
: :' : Chris Lamb
`. `'` la...@debian.org / chris-lamb.co.uk
`-
binaryornot.0.4.0-1.unstable.amd64.log.txt.gz
Description: Binary data
