Control: tags -1 + patch Gilles Filippini a écrit le 06/04/2020 à 14:23 : > Drew Parsons a écrit le 06/04/2020 à 05:08 : >> On 2020-04-06 09:56, Drew Parsons wrote: >>> On 2020-04-06 01:48, Gilles Filippini wrote: >>>> Drew Parsons a écrit le 05/04/2020 à 18:57 : >>>>> >>>>> Another option is to create an environment variable to force h5py to >>>>> load the mpi version even when run in a serial environment without >>>>> mpirun. Easy enough to set up, though I'm interested to see if "mpirun >>>>> -n 1 dh_auto_build" or a variation of that is viable. Maybe >>>>> %: >>>>> mpirun -n 1 dh $@ --with python3 --buildsystem=pybuild >>>> >>>> This, way the test cases run against python3.7 is OK, but it fails >>>> against python3.8 with: >>>> >>>> I: pybuild base:217: cd >>>> /build/bitshuffle-z2ZvpN/bitshuffle-0.3.5/.pybuild/cpython3_3.8_bitshuffle/build; >>>> >>>> python3.8 -m unittest discover -v >>>> [pinibrem15:43725] OPAL ERROR: Unreachable in file ext3x_client.c at >>>> line 112 >>>> *** An error occurred in MPI_Init_thread >>>> *** on a NULL communicator >>>> *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, >>>> *** and potentially your MPI job) >>>> [pinibrem15:43725] Local abort before MPI_INIT completed completed >>>> successfully, but am not able to aggregate error messages, and not able >>>> to guarantee that all other processes were killed! >>>> E: pybuild pybuild:352: test: plugin distutils failed with: exit code=1: >>>> cd >>>> /build/bitshuffle-z2ZvpN/bitshuffle-0.3.5/.pybuild/cpython3_3.8_bitshuffle/build; >>>> >>>> python3.8 -m unittest discover -v >>>> dh_auto_test: error: pybuild --test -i python{version} -p "3.7 3.8" >>>> returned exit code 13 >>>> >>>> But the HDF5 error is no more present with python3.7. So it seems a good >>>> point. >>> >>> >>> Strange again. I would have expected the same behaviour in python3.8 >>> and python3.7, whether successful or unsuccessful. >> >> >> Putting dh into mpirun seems to be interfering with process spawning. >> Once MPI is initialised (for the python3.7 test) it's not reinitialised >> for the python3.8 and so it's in a bad state for the test. Something >> like that. >> >> It's only in the tests where h5py is invoked that we get the problems. >> This variant works, applying mpirun separately for each test run: >> >> override_dh_auto_test: >> set -e; \ >> for py in `py3versions -s -v`; do \ >> mpirun -n 1 pybuild --test -i python{version} -p $$py; \ >> done >> >> (could use mpirun -n $(NPROC) for real mpi testing). > > Yes, it works! \o/ > >> Do we want to use this as a solution? Or would you prefer an environment >> variable that h5py can check to allow mpi invocation on a serial process? > > I let this decision up to you. Whatever you choose it deserve a bit fat > note in README.Debian. > >> Note that this means bitshuffle as built now is expressly tied in with >> hdf5-mpi and h5py-mpi (this seems intentional by debian/rules and >> debian/control, though the Build-Depends must be updated to >> python3-h5py-mpi). It's a separate question whether it's desirable to >> also support a hdf5-serial build of bitshuffle. Likewise we need to >> think about what we want to happen when bitshuffle is invoked in a >> serial process. > > I'll let that to the bitshuffle maintainer. I'll propose a patch to fix > the current FTBFS, sticking on the mpi flavour to be conservative vs > bitshuffle's previous builds.
Here it is. _g.
diff -Nru bitshuffle-0.3.5/debian/changelog bitshuffle-0.3.5/debian/changelog --- bitshuffle-0.3.5/debian/changelog 2019-12-01 19:03:38.000000000 +0100 +++ bitshuffle-0.3.5/debian/changelog 2020-04-06 14:47:10.000000000 +0200 @@ -1,3 +1,21 @@ +bitshuffle (0.3.5-3.1) UNRELEASED; urgency=medium + + * Non-maintainer upload. + + [ Drew Parsons <dpars...@debian.org>, Gilles Filippini <p...@debian.org> ] + * Closes: #955456 + - fix-deprecated.patch: fix test_h5filter.py and test_h5plugin.py + to open files with flag 'w' when required + - Build-Depends: python3-h5py-mpi to force using the mpi flavour + of h5py + - override_dh_auto_test: + - Run the tests via mpirun so that h5py knows it has to invoke its + mpi implementation + - Launch the tests for each python version separatly to permit MPI + initialization at each run + + -- Gilles Filippini <p...@debian.org> Mon, 06 Apr 2020 14:47:10 +0200 + bitshuffle (0.3.5-3) unstable; urgency=medium * don't use -march=native when building the package diff -Nru bitshuffle-0.3.5/debian/control bitshuffle-0.3.5/debian/control --- bitshuffle-0.3.5/debian/control 2019-12-01 19:03:38.000000000 +0100 +++ bitshuffle-0.3.5/debian/control 2020-04-06 14:46:51.000000000 +0200 @@ -11,7 +11,7 @@ # , libopenmpi-dev , openmpi-bin , python3-setuptools - , python3-h5py + , python3-h5py-mpi , quilt , cmake , pkg-config diff -Nru bitshuffle-0.3.5/debian/patches/fix-deprecated.patch bitshuffle-0.3.5/debian/patches/fix-deprecated.patch --- bitshuffle-0.3.5/debian/patches/fix-deprecated.patch 1970-01-01 01:00:00.000000000 +0100 +++ bitshuffle-0.3.5/debian/patches/fix-deprecated.patch 2020-04-06 14:46:44.000000000 +0200 @@ -0,0 +1,53 @@ +Index: bitshuffle-0.3.5/bitshuffle/tests/test_h5filter.py +=================================================================== +--- bitshuffle-0.3.5.orig/bitshuffle/tests/test_h5filter.py ++++ bitshuffle-0.3.5/bitshuffle/tests/test_h5filter.py +@@ -23,7 +23,7 @@ class TestFilter(unittest.TestCase): + dtype = np.int64 + data = np.arange(shape[0]) + fname = "tmp_test_filters.h5" +- f = h5py.File(fname) ++ f = h5py.File(fname, 'w') + h5.create_dataset(f, b"range", shape, dtype, chunks, + filter_pipeline=(32008, 32000), + filter_flags=(h5z.FLAG_MANDATORY, h5z.FLAG_MANDATORY), +@@ -43,7 +43,7 @@ class TestFilter(unittest.TestCase): + dtype = np.int64 + data = np.arange(shape[0]) + fname = "tmp_test_filters.h5" +- f = h5py.File(fname) ++ f = h5py.File(fname, 'w') + h5.create_dataset(f, b"range", shape, dtype, chunks, + filter_pipeline=(32008, 32000), + filter_flags=(h5z.FLAG_MANDATORY, h5z.FLAG_MANDATORY), +@@ -65,7 +65,7 @@ class TestFilter(unittest.TestCase): + dtype = np.int64 + data = np.arange(shape[0]) + fname = "tmp_test_filters.h5" +- f = h5py.File(fname) ++ f = h5py.File(fname, 'w') + h5.create_dataset(f, b"range", shape, dtype, chunks, + filter_pipeline=(32008,), + filter_flags=(h5z.FLAG_MANDATORY,), +Index: bitshuffle-0.3.5/bitshuffle/tests/test_h5plugin.py +=================================================================== +--- bitshuffle-0.3.5.orig/bitshuffle/tests/test_h5plugin.py ++++ bitshuffle-0.3.5/bitshuffle/tests/test_h5plugin.py +@@ -34,7 +34,7 @@ class TestFilterPlugins(unittest.TestCas + dtype = np.int64 + data = np.arange(shape[0]) + fname = "tmp_test_filters.h5" +- f = h5py.File(fname) ++ f = h5py.File(fname, 'w') + tid = h5t.py_create(dtype, logical=1) + sid = h5s.create_simple(shape, shape) + # +@@ -72,7 +72,7 @@ class TestFilterPlugins(unittest.TestCas + # return + # # Does not appear to be supported by h5py. + # fname = "tmp_test_h5py_hl.h5" +- # f = h5py.File(fname) ++ # f = h5py.File(fname, 'w') + # f.create_dataset("range", np.arange(1024, dtype=np.int64), + # compression=32008) + diff -Nru bitshuffle-0.3.5/debian/patches/series bitshuffle-0.3.5/debian/patches/series --- bitshuffle-0.3.5/debian/patches/series 2019-12-01 19:03:38.000000000 +0100 +++ bitshuffle-0.3.5/debian/patches/series 2020-04-06 14:46:44.000000000 +0200 @@ -1,2 +1,3 @@ hdf5-old-api.patch change-build-flags.patch +fix-deprecated.patch diff -Nru bitshuffle-0.3.5/debian/rules bitshuffle-0.3.5/debian/rules --- bitshuffle-0.3.5/debian/rules 2019-08-20 19:29:38.000000000 +0200 +++ bitshuffle-0.3.5/debian/rules 2020-04-06 14:46:58.000000000 +0200 @@ -24,5 +24,11 @@ %: dh $@ --with python3 --buildsystem=pybuild +override_dh_auto_test: + set -e; \ + for py in `py3versions -s -v`; do \ + mpirun -n 1 pybuild --test -i python{version} -p $$py; \ + done + #override_dh_auto_clean: # dh_auto_clean
signature.asc
Description: OpenPGP digital signature