Processed: Re: Bug#854084: binaryornot: Non-determistically FTBFS due to unreliable timing in tests

2017-02-04 Thread Debian Bug Tracking System 
Processing control commands:

> severity -1 normal
Bug #854084 [src:binaryornot] binaryornot: Non-determistically FTBFS due to 
unreliable timing in tests
Severity set to 'normal' from 'serious'

-- 
854084: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=854084
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems

Bug#854084: binaryornot: Non-determistically FTBFS due to unreliable timing in tests

2017-02-04 Thread Vincent Bernat 
Control: severity -1 normal

<#secure method=pgpmime mode=sign>
 ❦  4 février 2017 09:25 +1300, Chris Lamb  :

>   […]
>   
>   ==
>   ERROR: test_never_crashes (tests.test_check.TestDetectionProperties)
>   --
>   Traceback (most recent call last):
> File "«BUILDDIR»/tests/test_check.py", line 180, in test_never_crashes
>   def test_never_crashes(self, data):
> File "/usr/lib/python2.7/dist-packages/hypothesis/core.py", line 438, in 
> wrapped_test
>   HealthCheck.too_slow,
> File "/usr/lib/python2.7/dist-packages/hypothesis/core.py", line 306, in 
> fail_health_check
>   raise FailedHealthCheck(message)
>   FailedHealthCheck: Data generation is extremely slow: Only produced 10 
> valid examples in 0.33 seconds (0 invalid ones and 1 exceeded maximum size). 
> Try decreasing size of the data you're generating (with e.g.average_size or 
> max_leaves parameters).
>   See https://hypothesis.readthedocs.io/en/latest/healthchecks.html for more 
> information about this. If you want to disable just this health check, add 
> HealthCheck.too_slow to the suppress_health_check settings for this test.
>   
>   --
>   Ran 43 tests in 0.557s

I have slowed down my own machine in a such ways tests are now taking 6s
to execute. I cannot reproduce this. I need to slow it down such that
tests take 20s to get this error (and it says 10 valid examples in 1.30
seconds and the test suite took 2.9s). Runing the tests in a loop
without CPU limitation (tests run in 1.88 seconds) doesn't trigger the
bug.

I don't intend disabling this test which seems important because there
is not enough CPU to run the test (even if this is not a timing-related
test). Like for memory and disk, I understand that test suites have some
expectation on how much CPU the environment should provide (otherwise,
we have to disable all timing related tests).
-- 
"You have been in Afghanistan, I perceive."
-- Sir Arthur Conan Doyle, "A Study in Scarlet"

Bug#854084: binaryornot: Non-determistically FTBFS due to unreliable timing in tests

2017-02-03 Thread Chris Lamb 
Source: binaryornot
Version: 0.4.0-1
Severity: serious
Justification: fails to build from source
User: reproducible-bui...@lists.alioth.debian.org
Usertags: ftbfs
X-Debbugs-Cc: reproducible-b...@lists.alioth.debian.org

Dear Maintainer,

binaryornot's testsuite appears to use method timing/benchmarking in
such a way that it will non-deterministically FTBFS:

  […]
  
  ==
  ERROR: test_never_crashes (tests.test_check.TestDetectionProperties)
  --
  Traceback (most recent call last):
File "«BUILDDIR»/tests/test_check.py", line 180, in test_never_crashes
  def test_never_crashes(self, data):
File "/usr/lib/python2.7/dist-packages/hypothesis/core.py", line 438, in 
wrapped_test
  HealthCheck.too_slow,
File "/usr/lib/python2.7/dist-packages/hypothesis/core.py", line 306, in 
fail_health_check
  raise FailedHealthCheck(message)
  FailedHealthCheck: Data generation is extremely slow: Only produced 10 valid 
examples in 0.33 seconds (0 invalid ones and 1 exceeded maximum size). Try 
decreasing size of the data you're generating (with e.g.average_size or 
max_leaves parameters).
  See https://hypothesis.readthedocs.io/en/latest/healthchecks.html for more 
information about this. If you want to disable just this health check, add 
HealthCheck.too_slow to the suppress_health_check settings for this test.
  
  --
  Ran 43 tests in 0.557s
  
  FAILED (errors=1, expected failures=1)
  Test failed: 
  error: Test failed: 
  E: pybuild pybuild:283: test: plugin distutils failed with: exit code=1: 
python2.7 setup.py test 
  dh_auto_test: pybuild --test -i python{version} -p 2.7 returned exit code 13
  debian/rules:7: recipe for target 'build' failed
  make: *** [build] Error 25
  dpkg-buildpackage: error: debian/rules build gave error exit status 2

  […]

The full build log is attached.


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-


binaryornot.0.4.0-1.unstable.amd64.log.txt.gz
Description: Binary data