Bug#943369: numpy: test failures do not fail build

2019-10-23 Thread Michael Hudson-Doyle
Source: numpy
Version: 1:1.17.2-1
Severity: important
Tags: patch

Dear Maintainer,

Dear Maintainer,

The tests are run during build like this:

python$$v -c "import sys ; sys.path.insert(0, 
'./tmp/usr/lib/python3/dist-packages/') ; import numpy; numpy.test(verbose=5)"

but numpy.test returns a boolean True/False to indicate whether it
passed or failed, not raising an exception or anything else that would
cause python to exit with a non-zero code. Simple, untested, patch
attached.

Cheers,
mwh

-- System Information:
Debian Release: buster/sid
  APT prefers eoan
  APT policy: (500, 'eoan'), (400, 'eoan-proposed')
Architecture: amd64 (x86_64)

Kernel: Linux 5.3.0-18-generic (SMP w/8 CPU cores)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_WARN, TAINT_OOT_MODULE
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), 
LANGUAGE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
>From f7405ec04b68a80cba5d0bc83188a6262e8ada38 Mon Sep 17 00:00:00 2001
From: Michael Hudson-Doyle 
Date: Thu, 24 Oct 2019 12:46:38 +1300
Subject: [PATCH] make sure failing tests fail the build

---
 debian/rules | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/debian/rules b/debian/rules
index b4381f5..2c50338 100755
--- a/debian/rules
+++ b/debian/rules
@@ -116,9 +116,9 @@ ifeq (,$(findstring nocheck,$(DEB_BUILD_OPTIONS)))
set -e; for v in $(PY3VERS) ; do \
cd debian ; \
echo "-- running tests for "$$v" plain --" ; \
-   python$$v -c "import sys ; sys.path.insert(0, 
'./tmp/usr/lib/python3/dist-packages/') ; import numpy; numpy.test(verbose=5)" 
; \
+   python$$v -c "import sys ; sys.path.insert(0, 
'./tmp/usr/lib/python3/dist-packages/') ; import numpy; 
sys.exit(numpy.test(verbose=5))" ; \
echo "-- running tests for "$$v" debug --" ; \
-   python$$v-dbg -c "import sys ; sys.path.insert(0, 
'./tmp/usr/lib/python3/dist-packages/') ; import numpy; numpy.test(verbose=5)" 
; \
+   python$$v-dbg -c "import sys ; sys.path.insert(0, 
'./tmp/usr/lib/python3/dist-packages/') ; import numpy; sys.exit(not 
numpy.test(verbose=5))" ; \
cd .. ; \
done
 endif
-- 
2.20.1



Bug#943369: numpy: test failures do not fail build

2020-04-22 Thread Sandro Tosi
control: tags -1 + wontfix

> The tests are run during build like this:
>
> python$$v -c "import sys ; sys.path.insert(0, 
> './tmp/usr/lib/python3/dist-packages/') ; import numpy; numpy.test(verbose=5)"
>
> but numpy.test returns a boolean True/False to indicate whether it
> passed or failed, not raising an exception or anything else that would
> cause python to exit with a non-zero code. Simple, untested, patch
> attached.

sadly it is not that simple.

With just the recent upload of numpy/1.8.3 there are 4 release
architectures where tests are failing:

https://github.com/numpy/numpy/issues/16046
https://github.com/numpy/numpy/issues/16047
https://github.com/numpy/numpy/issues/16048
https://github.com/numpy/numpy/issues/16049

If i enable build failures for test failures, it means the new version
of numpy will never transition to testing until those tests failures
get fixes; given the relative small footprint of those archs, it's
unlikely this will be a timely process, so it could pass weeks, if not
months before they get fixed, if ever.

This is not an acceptable process.

We could skip the failing tests, but then at every new upload we're
have to remove the skip instructions and re-assess what tests are
failing: it will just be extra work and obtain the same result as of
now.

For all that's said above, i'm tagging this but as wontfix, but leave
it open for others to express their opinion

Regards,
-- 
Sandro "morph" Tosi
My website: http://sandrotosi.me/
Me at Debian: http://wiki.debian.org/SandroTosi
Twitter: https://twitter.com/sandrotosi