Bug#854496: python-qtpy: FTBFS randomly (failing tests)

[Ghislain Vaillant]

On Mon, 13 Mar 2017 21:34:23 + Ghislain Vaillant  wrote:
> On Mon, 2017-03-13 at 22:19 +0100, Santiago Vila wrote:
> > I would consider disabling the test suite (maybe to enable it again
> > after the release of Debian 9).
> 
> I believe you are right, let's disable the tests for Stretch and re-
> enable them in Buster using pytest-xvfb. I'll keep the autopkgtests
> alive though since deb-ci seems to work.

I have tagged and pushed an update of the packaging with the tests
dropped [1]. I also validated the build on debomatic [2].

Could you sponsor it please?

[1] https://anonscm.debian.org/git/python-modules/packages/python-qtpy.git
[2] 
http://debomatic-amd64.debian.net/distribution#unstable/python-qtpy/1.2.1-2/buildlog

Thanks,
Ghis

Bug#854496: python-qtpy: FTBFS randomly (failing tests)

[Ghislain Vaillant]

On Mon, 2017-03-13 at 22:19 +0100, Santiago Vila wrote:
> On Mon, Mar 13, 2017 at 09:40:14AM +, Ghislain Vaillant wrote:
> > Could you try the following debdiff, please?
> 
> The proposed patch does not fix the FTBFS problem.
> See attach (but I also tried several more times).

Apart from the -a option, I don't have any other ideas besides using
pytest-xvfb.

> In case you want to reproduce this for yourself, I wrote a description
> of my build environment here:
> 
> https://people.debian.org/~sanvila/my-building-environment.txt
> 
> I would consider disabling the test suite (maybe to enable it again
> after the release of Debian 9).

I believe you are right, let's disable the tests for Stretch and re-
enable them in Buster using pytest-xvfb. I'll keep the autopkgtests
alive though since deb-ci seems to work.

Ghis

Bug#854496: python-qtpy: FTBFS randomly (failing tests)

[Santiago Vila]

On Mon, Mar 13, 2017 at 09:40:14AM +, Ghislain Vaillant wrote:
> Could you try the following debdiff, please?

The proposed patch does not fix the FTBFS problem.
See attach (but I also tried several more times).

In case you want to reproduce this for yourself, I wrote a description
of my build environment here:

https://people.debian.org/~sanvila/my-building-environment.txt

I would consider disabling the test suite (maybe to enable it again
after the release of Debian 9).

Thanks.

python-qtpy_1.2.1-2_amd64-20170313T210329Z.gz
Description: application/gzip

Bug#854496: python-qtpy: FTBFS randomly (failing tests)

[Ghislain Vaillant]

On Mon, 2017-03-13 at 21:41 +0100, Santiago Vila wrote:
> On Sat, Mar 11, 2017 at 10:47:36PM +, Ghislain Vaillant wrote:
> 
> > Could you point me to the specific commit or portion of the discussion
> > where the fix is described, please? I'd be happy to prepare a debdiff
> > for you to apply to check whether this fix you mention works with
> > src:python-qtpy.
> 
> This is the most relevant message from Bug #848063.
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=848063#64

src:python-qtpy does not use SDL, so I don't think this is the fix we
are looking for.

> In this case, just adding -a did not seem to fix the problem.

Could you try the debdiff I sent you nonetheless please?

Thanks,
Ghis

Bug#854496: python-qtpy: FTBFS randomly (failing tests)

[Santiago Vila]

On Sat, Mar 11, 2017 at 10:47:36PM +, Ghislain Vaillant wrote:

> Could you point me to the specific commit or portion of the discussion
> where the fix is described, please? I'd be happy to prepare a debdiff
> for you to apply to check whether this fix you mention works with
> src:python-qtpy.

This is the most relevant message from Bug #848063.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=848063#64

In this case, just adding -a did not seem to fix the problem.

Thanks.

Bug#854496: python-qtpy: FTBFS randomly (failing tests)

[Ghislain Vaillant]

On Sun, 2017-03-12 at 19:08 +, Ghislain Vaillant wrote:
> On 11/03/17 22:47, Ghislain Vaillant wrote:
> > On Sat, 2017-03-11 at 21:47 +0100, Santiago Vila wrote:
> > > BTW: The ri-li package used to fail 95% of the time in my autobuilders
> > > and also "often" in reproducible builds autobuilders. Somebody found
> > > a way to make this failure rate to decrease dramatically, maybe
> > > it may give you a hint.
> > 
> > Could you point me to the specific commit or portion of the discussion
> > where the fix is described, please? I'd be happy to prepare a debdiff
> > for you to apply to check whether this fix you mention works with
> > src:python-qtpy.
> 
> The only recent commit from src:ri-li that looks applicable to
> src:python-qtpy is the use of the additional `-a` flag for the
> xvfb-run calls. I can apply it and provide you with a debdiff.

Hi Santiago,

Could you try the following debdiff, please?

Thanks,
Ghis
diff -Nru python-qtpy-1.2.1/debian/changelog python-qtpy-1.2.1/debian/changelog
--- python-qtpy-1.2.1/debian/changelog	2017-01-22 08:09:05.0 +
+++ python-qtpy-1.2.1/debian/changelog	2017-03-13 09:23:53.0 +
@@ -1,3 +1,12 @@
+python-qtpy (1.2.1-2) unstable; urgency=medium
+
+  * Call xvfb-run with the -a option.
+Reason: fixes random segfaults happening during the tests
+Thanks to Santiago Vila for reporting (Closes: #854496)
+  * Run autopkgtests for all supported Python versions
+
+ -- Ghislain Antony Vaillant   Mon, 13 Mar 2017 09:23:53 +
+
 python-qtpy (1.2.1-1) unstable; urgency=medium
 
   * New upstream release
diff -Nru python-qtpy-1.2.1/debian/rules python-qtpy-1.2.1/debian/rules
--- python-qtpy-1.2.1/debian/rules	2017-01-22 08:09:05.0 +
+++ python-qtpy-1.2.1/debian/rules	2017-03-13 09:23:53.0 +
@@ -12,5 +12,5 @@
 
 override_dh_auto_test:
 ifeq (,$(findstring nocheck,$(DEB_BUILD_PROFILES)))
-	xvfb-run dh_auto_test
+	xvfb-run -a dh_auto_test
 endif
diff -Nru python-qtpy-1.2.1/debian/tests/control python-qtpy-1.2.1/debian/tests/control
--- python-qtpy-1.2.1/debian/tests/control	2017-01-22 08:09:05.0 +
+++ python-qtpy-1.2.1/debian/tests/control	2017-03-13 09:23:53.0 +
@@ -1,5 +1,25 @@
-Test-Command: cp -a ./qtpy/tests $AUTOPKGTEST_TMP ; cd $AUTOPKGTEST_TMP ; xvfb-run py.test
-Depends: python-qtpy, python-pytest, xauth, xvfb
+Test-Command: set -e
+ ; cp -r qtpy/tests "$AUTOPKGTEST_TMP"
+ ; for py in $(pyversions -r 2>/dev/null)
+ ; do cd "$AUTOPKGTEST_TMP"
+ ; echo "Testing with $py:"
+ ; xvfb-run -a $py -m pytest -v
+ ; done
+Depends: python-all,
+ python-qtpy,
+ python-pytest,
+ xauth,
+ xvfb
 
-Test-Command: cp -a ./qtpy/tests $AUTOPKGTEST_TMP ; cd $AUTOPKGTEST_TMP ; xvfb-run py.test-3
-Depends: python3-qtpy, python3-pytest, xauth, xvfb
+Test-Command: set -e
+ ; cp -r qtpy/tests "$AUTOPKGTEST_TMP"
+ ; for py in $(py3versions -r 2>/dev/null)
+ ; do cd "$AUTOPKGTEST_TMP"
+ ; echo "Testing with $py:"
+ ; xvfb-run -a $py -m pytest -v
+ ; done
+Depends: python3-all,
+ python3-qtpy,
+ python3-pytest,
+ xauth,
+ xvfb

Bug#854496: python-qtpy: FTBFS randomly (failing tests)

[Ghislain Vaillant]

On 11/03/17 22:47, Ghislain Vaillant wrote:

On Sat, 2017-03-11 at 21:47 +0100, Santiago Vila wrote:

BTW: The ri-li package used to fail 95% of the time in my autobuilders
and also "often" in reproducible builds autobuilders. Somebody found
a way to make this failure rate to decrease dramatically, maybe
it may give you a hint.


Could you point me to the specific commit or portion of the discussion
where the fix is described, please? I'd be happy to prepare a debdiff
for you to apply to check whether this fix you mention works with
src:python-qtpy.


The only recent commit from src:ri-li that looks applicable to
src:python-qtpy is the use of the additional `-a` flag for the
xvfb-run calls. I can apply it and provide you with a debdiff.

Ghis

Bug#854496: python-qtpy: FTBFS randomly (failing tests)

[Ghislain Vaillant]

On Sat, 2017-03-11 at 21:47 +0100, Santiago Vila wrote:
> On Sat, Mar 11, 2017 at 08:17:18AM +, Ghislain Vaillant wrote:
> 
> > A (potential) fix for this would be to try out testing with pytest-xvfb
> > (which I have recently packaged, so only in unstable for now), instead of
> > calling xvfb-run manually.
> > 
> > pytest-xvfb is supposed to provide a more robust setup for tests requiring
> > xvfb. I have had successful results with packages such as
> > src:spyder-unittest. After the freeze (since pytest-xvfb will not be
> > available in Stretch), I intend to switch both src:python-qtpy and
> > src:python-qtawesome to using pytest-xvfb for testing.
> > 
> > We'll see whether this addresses the problem you reported. Are you happy
> > with this plan?
> 
> It depends. This is a FTBFS problem, so it should be fixed in stretch.

I agree.

> So, before trying what you describe I would ask Release Managers if
> they would allow pytest-xvfb in stretch as a way to fix this bug.

Keep in mind this is only an hypothesis. It might be worth confirming
that using pytest-xvfb effectively solves the problem before requesting
an unblock.

Another issue is that pytest-xvfb being a new Python package, it has
only been packaged for Python 3. So, we would have to either drop
testing for Python 2 or introduce a Python 2 binary package for pytest-
xvfb.

> BTW: The ri-li package used to fail 95% of the time in my autobuilders
> and also "often" in reproducible builds autobuilders. Somebody found
> a way to make this failure rate to decrease dramatically, maybe
> it may give you a hint.

Could you point me to the specific commit or portion of the discussion
where the fix is described, please? I'd be happy to prepare a debdiff
for you to apply to check whether this fix you mention works with
src:python-qtpy.

Cheers,
Ghis

Bug#854496: python-qtpy: FTBFS randomly (failing tests)

[Santiago Vila]

On Sat, Mar 11, 2017 at 08:17:18AM +, Ghislain Vaillant wrote:

> A (potential) fix for this would be to try out testing with pytest-xvfb
> (which I have recently packaged, so only in unstable for now), instead of
> calling xvfb-run manually.
> 
> pytest-xvfb is supposed to provide a more robust setup for tests requiring
> xvfb. I have had successful results with packages such as
> src:spyder-unittest. After the freeze (since pytest-xvfb will not be
> available in Stretch), I intend to switch both src:python-qtpy and
> src:python-qtawesome to using pytest-xvfb for testing.
> 
> We'll see whether this addresses the problem you reported. Are you happy
> with this plan?

It depends. This is a FTBFS problem, so it should be fixed in stretch.

So, before trying what you describe I would ask Release Managers if
they would allow pytest-xvfb in stretch as a way to fix this bug.

BTW: The ri-li package used to fail 95% of the time in my autobuilders
and also "often" in reproducible builds autobuilders. Somebody found
a way to make this failure rate to decrease dramatically, maybe
it may give you a hint.

Thanks.

Bug#854496: python-qtpy: FTBFS randomly (failing tests)

[Ghislain Vaillant]

Hi Santiago,

A (potential) fix for this would be to try out testing with pytest-xvfb 
(which I have recently packaged, so only in unstable for now), instead 
of calling xvfb-run manually.


pytest-xvfb is supposed to provide a more robust setup for tests 
requiring xvfb. I have had successful results with packages such as 
src:spyder-unittest. After the freeze (since pytest-xvfb will not be 
available in Stretch), I intend to switch both src:python-qtpy and 
src:python-qtawesome to using pytest-xvfb for testing.


We'll see whether this addresses the problem you reported. Are you happy 
with this plan?


Cheers,
Ghis

Bug#854496: python-qtpy: FTBFS randomly (failing tests)

[Ghislain Vaillant]

On Mon, 2017-02-13 at 19:42 +0100, Santiago Vila wrote:
> > As far as this bug is concerned, all I can do is tag it for help, since
> > I have no clue how to make any sort of progress. If yourself or someone
> > else finds a fix for it, I will happily incorporate the corresponding
> > patches.
> 
> A closer look tells me that this is not really random.
> 
> Instead, it always fails on the fast machines I have, and never on the
> slower ones. When it fails, we can see this in the build log:
> 
> tests/test_uic.py Aborted
> 
> so I tried "ulimit -c unlimited" before "dpkg-buildpackage -uc -us -b"
> and this was the result:
> 
> https://people.debian.org/~sanvila/python-qtpy/python-qtpy-1.2.1.build.tar.gz
> 
> This is the whole build tree, including the "core" file that was generated
> by the python interpreter, so in addition to the help tag, I would also:
> 
> a) disable tests/test_uic.py until further notice, as we don't know
> why it fails.

Well, it sounds like an environment related issue. The fact that it
builds correctly both on the builders and locally, and tests fine on
both debci (which runs the very same test suite) and upstream travis
hints me towards this conclusion.

Also, looking at the current logs in reproducible builds, the execution
of the tests is aborted (timeout?), not marked as FAILED, which is a
sign that the failure of the tests is triggered by something external.

Actually, the termination of the tests does not systematically happen
at `tests/test_uic.py` (at `tests/test_patch_qheaderview.py` on amd64,
at `tests/test_patch_qcombobox.py` on arm64). So, the solution you
proposed would not solve what is happening on reproducible builds.

> b) forward the bug upstream so that the author investigates about it.

Done. Upstream is clueless, unsurprisingly.

Bug#854496: python-qtpy: FTBFS randomly (failing tests)

[Santiago Vila]

> As far as this bug is concerned, all I can do is tag it for help, since
> I have no clue how to make any sort of progress. If yourself or someone
> else finds a fix for it, I will happily incorporate the corresponding
> patches.

A closer look tells me that this is not really random.

Instead, it always fails on the fast machines I have, and never on the
slower ones. When it fails, we can see this in the build log:

tests/test_uic.py Aborted

so I tried "ulimit -c unlimited" before "dpkg-buildpackage -uc -us -b"
and this was the result:

https://people.debian.org/~sanvila/python-qtpy/python-qtpy-1.2.1.build.tar.gz

This is the whole build tree, including the "core" file that was generated
by the python interpreter, so in addition to the help tag, I would also:

a) disable tests/test_uic.py until further notice, as we don't know
why it fails.

b) forward the bug upstream so that the author investigates about it.

Thanks.

Bug#854496: python-qtpy: FTBFS randomly (failing tests)

[Ghislain Vaillant]

control: tags -1 + help

On Mon, 2017-02-13 at 18:03 +0100, Santiago Vila wrote:
> > > However, there is another package involving xvfb-run which always fail
> > > for me. Can you reproduce this bug?
> > > 
> > > https://bugs.debian.org/848063
> > > 
> > > The maintainer downgraded it to important, but it fails for me 100% of
> > > the time, so it's not really very random (for me at least).
> > 
> > I built it successfully with sbuild on my local machine. I also did a
> > successful trial run on debomatic:
> > 
> > http://debomatic-i386.debian.net/distribution#unstable/ri-li/2.0.1+ds-4/buildlog
> > 
> > I understand Markus' frustration considering the package used to build
> > fine before, still builds ok now (works on my sbuild and on dom) but
> > fails with your setup, with no obvious clue as where the failure could
> > originate from.
> 
> Well, I hope you can understand my frustration too: ri-li fails for me
> 97% of the time (which is almost always), on different machines, and
> it seems to fail always as well in the reproducible builds autobuilders.

Absolutely. I don't question your motivations here, don't get me wrong.


> In such conditions the least I would expect from Markus is to accept
> the offer I made (i.e. free access to a machine where this happens always).

Even if you were to give me access to the machine where the src:python-
qtpy failures happen randomly, I don't know what I would be able to do
with it. Perhaps Markus is feeling the same, who knows.


As far as this bug is concerned, all I can do is tag it for help, since
I have no clue how to make any sort of progress. If yourself or someone
else finds a fix for it, I will happily incorporate the corresponding
patches.

Cheers,
Ghis

Bug#854496: python-qtpy: FTBFS randomly (failing tests)

[Santiago Vila]

Package: src:python-qtpy
Version: 1.2.1-1
Severity: important

Dear maintainer:

I tried to build this package in stretch with "dpkg-buildpackage -A"
but it failed:


[...]
 debian/rules build-indep
dh build-indep --with python2,python3 --buildsystem=pybuild
   dh_testdir -i -O--buildsystem=pybuild
   dh_update_autotools_config -i -O--buildsystem=pybuild
   dh_autoreconf -i -O--buildsystem=pybuild
   dh_auto_configure -i -O--buildsystem=pybuild
I: pybuild base:184: python2.7 setup.py config 
running config
I: pybuild base:184: python3.5 setup.py config 
running config
   dh_auto_build -i -O--buildsystem=pybuild
I: pybuild base:184: /usr/bin/python setup.py build 
running build
running build_py

[... snipped ...]

copying qtpy/QtTest.py -> /<>/.pybuild/pythonX.Y_3.5/build/qtpy
copying qtpy/QtDesigner.py -> /<>/.pybuild/pythonX.Y_3.5/build/qtpy
copying qtpy/_version.py -> /<>/.pybuild/pythonX.Y_3.5/build/qtpy
copying qtpy/QtWebEngineWidgets.py -> 
/<>/.pybuild/pythonX.Y_3.5/build/qtpy
copying qtpy/QtPrintSupport.py -> 
/<>/.pybuild/pythonX.Y_3.5/build/qtpy
copying qtpy/QtCore.py -> /<>/.pybuild/pythonX.Y_3.5/build/qtpy
copying qtpy/__init__.py -> /<>/.pybuild/pythonX.Y_3.5/build/qtpy
copying qtpy/compat.py -> /<>/.pybuild/pythonX.Y_3.5/build/qtpy
copying qtpy/QtGui.py -> /<>/.pybuild/pythonX.Y_3.5/build/qtpy
copying qtpy/QtSvg.py -> /<>/.pybuild/pythonX.Y_3.5/build/qtpy
copying qtpy/QtMultimedia.py -> 
/<>/.pybuild/pythonX.Y_3.5/build/qtpy
copying qtpy/py3compat.py -> /<>/.pybuild/pythonX.Y_3.5/build/qtpy
copying qtpy/QtWidgets.py -> /<>/.pybuild/pythonX.Y_3.5/build/qtpy
creating /<>/.pybuild/pythonX.Y_3.5/build/qtpy/_patch
copying qtpy/_patch/qheaderview.py -> 
/<>/.pybuild/pythonX.Y_3.5/build/qtpy/_patch
copying qtpy/_patch/__init__.py -> 
/<>/.pybuild/pythonX.Y_3.5/build/qtpy/_patch
copying qtpy/_patch/qcombobox.py -> 
/<>/.pybuild/pythonX.Y_3.5/build/qtpy/_patch
   debian/rules override_dh_auto_test
make[1]: Entering directory '/<>'
xvfb-run dh_auto_test
I: pybuild pybuild:212: cp -r /<>/qtpy/tests 
/<>/.pybuild/pythonX.Y_2.7/build
I: pybuild base:184: cd /<>/.pybuild/pythonX.Y_2.7/build; 
python2.7 -m pytest 
= test session starts ==
platform linux2 -- Python 2.7.13, pytest-3.0.5, py-1.4.32, pluggy-0.4.0

PyQt4: not installed
PyQt5: PyQt: 5.7 - Qt: 5.7.1
PySide: not installed

rootdir: /<>, inifile: 
collected 8 items

tests/test_main.py .
tests/test_patch_qcombobox.py ..
tests/test_patch_qheaderview.py .
tests/test_qtmultimedia.py .
tests/test_uic.py Aborted
E: pybuild pybuild:276: test: plugin distutils failed with: exit code=134: cd 
/<>/.pybuild/pythonX.Y_2.7/build; python2.7 -m pytest 
dh_auto_test: pybuild --test --test-pytest -i python{version} -p 2.7 returned 
exit code 13
debian/rules:15: recipe for target 'override_dh_auto_test' failed
make[1]: *** [override_dh_auto_test] Error 25
make[1]: Leaving directory '/<>'
debian/rules:11: recipe for target 'build-indep' failed
make: *** [build-indep] Error 2
dpkg-buildpackage: error: debian/rules build-indep gave error exit status 2


This is just how the build ends, not necessarily the relevant part.

I've put several build logs here:

https://people.debian.org/~sanvila/build-logs/python-qtpy/

If this is really a bug in one of the build-depends, please use reassign and 
affects,
so that this is still visible in the page for this package.

The bug should be reproducible with sbuild on a single CPU virtual machine,
provided you try enough times (as the failure happens randomly).

Thanks.