Bug#851722: [Python-modules-team] Bug#851722: django-pipeline: FTBFS randomly (failing tests)

[Brian May]

Brian May  writes:

> Ok, I got a suggestion to run the failing test by itself. It fails
> everytime. As reported in the upstream bug report.

Unblock request sent: #857429
-- 
Brian May 

Bug#851722: [Python-modules-team] Bug#851722: django-pipeline: FTBFS randomly (failing tests)

[Brian May]

Brian May  writes:

> Santiago Vila  writes:
>
>> So I think forwarding the bug would make sense. Just tell upstream the
>> truth, namely, that you have received a bug saying the package fails
>> to build randomly.
>
> See https://github.com/jazzband/django-pipeline/issues/622

Ok, I got a suggestion to run the failing test by itself. It fails
everytime. As reported in the upstream bug report.
-- 
Brian May 

Bug#851722: [Python-modules-team] Bug#851722: django-pipeline: FTBFS randomly (failing tests)

[Brian May]

Santiago Vila  writes:

> So I think forwarding the bug would make sense. Just tell upstream the
> truth, namely, that you have received a bug saying the package fails
> to build randomly.

See https://github.com/jazzband/django-pipeline/issues/622
-- 
Brian May 

Bug#851722: [Python-modules-team] Bug#851722: django-pipeline: FTBFS randomly (failing tests)

[Santiago Vila]

On Sun, 22 Jan 2017, Brian May wrote:

> Are you able to reproduce with the following patch? This will show what
> order the tests are being run in:
> [...]
> 
> diff --git a/debian/rules b/debian/rules
> index dc35e19..f53ec96 100755
> --- a/debian/rules
> +++ b/debian/rules
> @@ -21,4 +21,4 @@ override_dh_installchangelogs:
> dh_installchangelogs -- HISTORY.rst
>  
>  override_dh_auto_test:
> -   PYBUILD_SYSTEM=custom PYBUILD_TEST_ARGS="PYTHONPATH=. python{version} 
> /usr/bin/django-admin test --settings=tests.settings" dh_auto_test
> +   PYBUILD_SYSTEM=custom PYBUILD_TEST_ARGS="PYTHONPATH=. python{version} 
> /usr/bin/django-admin test -v3 --settings=tests.settings" dh_auto_test

Sorry for the late reply.

I built this package 200 times yesterday, on eight different virtual
machines, and this time it never failed.

Unfortunately, a recent build in one of the reproducible builds
autobuilders failed with the same error message I got in the past:

https://tests.reproducible-builds.org/debian/rbuild/testing/armhf/django-pipeline_1.6.8-2.rbuild.log

which means the bug is most likely not fixed.


In cases like this one there are basically two ways of approaching the
problem:

The difficult one is to build the package many times and expect it to
fail in your system. This may or may not happen.

The alternate way (maybe not so difficult, but it depends) is to
carefully examine the code and the build log and try to guess how it
may happen. This is not necessarily easier than the first method, but
I would recommend giving this method a try at least.


Because in my case it always failed in one machine and never in the
others, I suspect that the problem may be related with filesystem
ordering.


> I am reluctant to forward upstream a bug I cannot personally reproduce.

That's usually the right thing to do, because we don't want to bother
upstream with problems which are not real bugs.

In this case, however, the problem happened in one of my autobuilders,
a single-cpu machine using sbuild, but also in one of the reproducible
builds autobuilders, which are multi-core machines using pbuilder.

Because those autobuilders are completely independent and unrelated,
I really believe this is a bug in the package and not a misconfiguration
in any of the autobuilders.

So I think forwarding the bug would make sense. Just tell upstream the
truth, namely, that you have received a bug saying the package fails
to build randomly.

Maybe the author can try the "second method" to debug this (examine
the code and the build log and try to guess how it may happen).

Thanks.

Bug#851722: [Python-modules-team] Bug#851722: django-pipeline: FTBFS randomly (failing tests)

[Brian May]

Hello All,

I would appreciate any help with this bug. While I have ideas on the
cause, I cannot reproduce it; and I am reluctant to forward upstream a
bug I cannot personally reproduce.

Thanks.


Santiago Vila  writes:

> Package: src:django-pipeline
> Version: 1.6.8-2
> Severity: important
>
> Dear maintainer:
>
> I tried to build this package in stretch with "dpkg-buildpackage -A"
> but it failed:
>
> 
> [...]
>  debian/rules build-indep
> dh build-indep --with python2,python3,sphinxdoc --buildsystem=pybuild
>dh_testdir -i -O--buildsystem=pybuild
>dh_update_autotools_config -i -O--buildsystem=pybuild
>dh_auto_configure -i -O--buildsystem=pybuild
> I: pybuild base:184: python2.7 setup.py config 
> running config
> I: pybuild base:184: python3.5 setup.py config 
> running config
>debian/rules override_dh_auto_build
> make[1]: Entering directory '/<>'
> dh_auto_build
> I: pybuild base:184: /usr/bin/python setup.py build 
> running build
>
> [... snipped ...]
>
> default_collector.clear()
>   File "/<>/pipeline/collector.py", line 23, in clear
> dirs, files = self.storage.listdir(path)
>   File "/usr/lib/python2.7/dist-packages/django/core/files/storage.py", line 
> 399, in listdir
> for entry in os.listdir(path):
> OSError: [Errno 2] No such file or directory: '/<>/tests/static'
>
> ==
> ERROR: test_pipeline_enabled_and_found 
> (tests.tests.test_views.ServeStaticViewsTest)
> --
> Traceback (most recent call last):
>   File "/<>/tests/tests/test_views.py", line 24, in setUp
> default_collector.clear()
>   File "/<>/pipeline/collector.py", line 23, in clear
> dirs, files = self.storage.listdir(path)
>   File "/usr/lib/python2.7/dist-packages/django/core/files/storage.py", line 
> 399, in listdir
> for entry in os.listdir(path):
> OSError: [Errno 2] No such file or directory: '/<>/tests/static'
>
> ==
> ERROR: test_pipeline_enabled_and_not_found 
> (tests.tests.test_views.ServeStaticViewsTest)
> --
> Traceback (most recent call last):
>   File "/<>/tests/tests/test_views.py", line 24, in setUp
> default_collector.clear()
>   File "/<>/pipeline/collector.py", line 23, in clear
> dirs, files = self.storage.listdir(path)
>   File "/usr/lib/python2.7/dist-packages/django/core/files/storage.py", line 
> 399, in listdir
> for entry in os.listdir(path):
> OSError: [Errno 2] No such file or directory: '/<>/tests/static'
>
> --
> Ran 88 tests in 0.931s
>
> FAILED (errors=8, skipped=15)
> Creating test database for alias 'default'...
> Destroying test database for alias 'default'...
> E: pybuild pybuild:276: test: plugin custom failed with: exit code=1: 
> PYTHONPATH=. python2.7 /usr/bin/django-admin test --settings=tests.settings
> dh_auto_test: pybuild --test -i python{version} -p 2.7 returned exit code 13
> debian/rules:24: recipe for target 'override_dh_auto_test' failed
> make[1]: *** [override_dh_auto_test] Error 25
> make[1]: Leaving directory '/<>'
> debian/rules:11: recipe for target 'build-indep' failed
> make: *** [build-indep] Error 2
> dpkg-buildpackage: error: debian/rules build-indep gave error exit status 2
> 
>
> This is just how the build ends, not necessarily the relevant part.
>
> I've put several build logs here:
>
> https://people.debian.org/~sanvila/build-logs/django-pipeline/
>
> If this is really a bug in one of the build-depends, please use reassign and 
> affects,
> so that this is still visible in the page for this package.
>
> The bug should be reproducible with sbuild on a single CPU virtual machine,
> provided you try enough times (as the failure happens randomly).
>
> Thanks.
>
> ___
> Python-modules-team mailing list
> python-modules-t...@lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/python-modules-team
-- 
Brian May 

Bug#851722: [Python-modules-team] Bug#851722: django-pipeline: FTBFS randomly (failing tests)

[Brian May]

Santiago Vila  writes:

> I tried to build this package in stretch with "dpkg-buildpackage -A"
> but it failed:

I am not able to reproduce this :-(

I am guessing it might be related to different order the tests are being
run in or something. Which can be non-determinstic - depends on order
files are listed in filesystem.

Are you able to reproduce with the following patch? This will show what
order the tests are being run in:


diff --git a/debian/rules b/debian/rules
index dc35e19..f53ec96 100755
--- a/debian/rules
+++ b/debian/rules
@@ -21,4 +21,4 @@ override_dh_installchangelogs:
dh_installchangelogs -- HISTORY.rst
 
 override_dh_auto_test:
-   PYBUILD_SYSTEM=custom PYBUILD_TEST_ARGS="PYTHONPATH=. python{version} 
/usr/bin/django-admin test --settings=tests.settings" dh_auto_test
+   PYBUILD_SYSTEM=custom PYBUILD_TEST_ARGS="PYTHONPATH=. python{version} 
/usr/bin/django-admin test -v3 --settings=tests.settings" dh_auto_test


Thanks!
-- 
Brian May