Re: WIP: Update of math/py-numpy to 1.16.5
On 2020/04/01 21:30, Paul Irofti wrote: > Here is a final diff that incorporates jca's patch and sthen's. OK? OK with me. > > Index: Makefile > === > RCS file: /cvs/ports/math/py-numpy/Makefile,v > retrieving revision 1.57 > diff -u -p -u -p -r1.57 Makefile > --- Makefile 10 Mar 2020 03:31:45 - 1.57 > +++ Makefile 1 Apr 2020 18:28:16 - > @@ -2,11 +2,10 @@ > > COMMENT= fast array and numeric programming library for Python > > -MODPY_EGG_VERSION= 1.14.6 > +MODPY_EGG_VERSION= 1.16.5 > DISTNAME=numpy-${MODPY_EGG_VERSION} > PKGNAME= py-${DISTNAME} > EXTRACT_SUFX=.zip > -REVISION=2 > > CATEGORIES= math devel > > @@ -15,7 +14,8 @@ HOMEPAGE= https://www.numpy.org/ > # BSD > PERMIT_PACKAGE = Yes > > -WANTLIB= blas lapack m pthread ${MODFORTRAN_WANTLIB} > ${MODPY_WANTLIB} > +WANTLIB += ${MODFORTRAN_WANTLIB} ${MODPY_WANTLIB} > +WANTLIB += blas cblas lapack m pthread > > MODULES= lang/python \ > fortran > @@ -27,13 +27,14 @@ MODPY_PI =Yes > MODPY_SETUPTOOLS=Yes > MODPY_DISTUTILS_BUILDARGS = --fcompiler=gnu95 > > -TEST_DEPENDS=devel/py-nose${MODPY_FLAVOR} \ > - devel/py-tz${MODPY_FLAVOR} > - > +TEST_DEPENDS=${FULLPKGNAME}:${FULLPKGPATH} \ > + devel/py-test${MODPY_FLAVOR} \ > + devel/py-tz${MODPY_FLAVOR} > MODFORTRAN_COMPILER =gfortran > > BUILD_DEPENDS = ${MODFORTRAN_BUILD_DEPENDS} > -LIB_DEPENDS=math/lapack \ > +LIB_DEPENDS= math/cblas \ > + math/lapack \ > ${MODFORTRAN_LIB_DEPENDS} > > # Cython 0.28.* uses __attribute__((optimize("Os"))) which > @@ -54,7 +55,7 @@ pre-build: > cp -f ${WRKSRC}/numpy/distutils/site.cfg ${WRKSRC}/site.cfg > > do-test: > - cd ${WRKDIR} && ${MAKE_ENV} PYTHONPATH=${WRKINST}${MODPY_SITEPKG} > ${MODPY_BIN} -c \ > + cd ${WRKDIR} && ${MAKE_ENV} ${MODPY_BIN} -c \ > 'import numpy ; numpy.test()' > > .include > Index: distinfo > === > RCS file: /cvs/ports/math/py-numpy/distinfo,v > retrieving revision 1.13 > diff -u -p -u -p -r1.13 distinfo > --- distinfo 26 Oct 2018 21:09:31 - 1.13 > +++ distinfo 1 Apr 2020 18:28:16 - > @@ -1,2 +1,2 @@ > -SHA256 (numpy-1.14.6.zip) = ElDt9vbEPh14I/CWdBa8GCWLsnHcU2KY6w6gCp5FuAo= > -SIZE (numpy-1.14.6.zip) = 4906096 > +SHA256 (numpy-1.16.5.zip) = i7RS2U6WSzEiBbDeEjjdcgnaRSNDZTqyFLXWgXgOegw= > +SIZE (numpy-1.16.5.zip) = 5138208 > Index: patches/patch-numpy_core_include_numpy_npy_common_h > === > RCS file: > /cvs/ports/math/py-numpy/patches/patch-numpy_core_include_numpy_npy_common_h,v > retrieving revision 1.6 > diff -u -p -u -p -r1.6 patch-numpy_core_include_numpy_npy_common_h > --- patches/patch-numpy_core_include_numpy_npy_common_h 30 Jun 2018 > 21:49:33 - 1.6 > +++ patches/patch-numpy_core_include_numpy_npy_common_h 1 Apr 2020 > 18:28:16 - > @@ -1,13 +1,8 @@ > $OpenBSD: patch-numpy_core_include_numpy_npy_common_h,v 1.6 2018/06/30 > 21:49:33 daniel Exp $ > > -(1) optimize function attributes were added in gcc 4.4. Apply a fix for the > warnings. But > -the next release of numpy should fix this in a better way. See: > -https://github.com/numpy/numpy/issues/5451 > +XXX recheck powerpc, is this still needed? > > -Gets rid of these warnings from the build: > -numpy/core/src/multiarray/lowlevel_strided_loops.c.src:120: warning: > 'optimize' attribute directive ignored > - > -(2) py-numpy only checks for expl to determine whether extended-precision > +py-numpy only checks for expl to determine whether extended-precision > support is present. since we don't have it yet; it implements > it's own. however, on alpha, powerpc, it declared functions with > types that conflict with C99 (double for *l), therefore failed. > @@ -15,36 +10,7 @@ types that conflict with C99 (double for > Index: numpy/core/include/numpy/npy_common.h > --- numpy/core/include/numpy/npy_common.h.orig > +++ numpy/core/include/numpy/npy_common.h > -@@ -20,6 +20,7 @@ > - #endif > - #endif > - > -+#if defined(__GNUC__) && (__GNUC__ > 4 || (__GNUC__ == 4 && (__GNUC_MINOR__ > > 4))) > - /* > - * gcc does not unroll even with -O3 > - * use with care, unrolling on modern cpus rarely speeds things up > -@@ -30,13 +31,20 @@ > - #else > - #define NPY_GCC_UNROLL_LOOPS > - #endif > -+#else > -+#define NPY_GCC_UNROLL_LOOPS > -+#endif > - > -+#if defined(__GNUC__) && (__GNUC__ > 4 || (__GNUC__ == 4 && (__GNUC_MINOR__ > > 4))) > - /* highest gcc optimization level, enabled autovectorizer */ > - #ifdef HAVE_ATTRIBUTE_OPTIMIZE_OPT_3 > - #define NPY_GCC_OPT_3
Re: WIP: Update of math/py-numpy to 1.16.5
On Sun, Mar 29, 2020 at 03:38:33PM +0100, Stuart Henderson wrote: > On 2020/03/29 11:49, Christian Weisgerber wrote: > > On 2020-03-28, Stuart Henderson wrote: > > > > >> > Can somenoe commit the numpy update as well then? > > >> > > >> I am about to start a test bulk build on i386 with this. > > > > > > No build problems relating to the numpy update. > > > > Meanwhile, the existing port (1.14.6p2) errored out in my latest bulk > > build: > > > > numpy/core/src/multiarray/temp_elide.c:87:10: fatal error: execinfo.h: No > > such file or directory > > #include > > ^~~~ > > compilation terminated. > > > > Please check that the updated port doesn't haphazardly pick up > > execinfo too. > > > > -- > > Christian "naddy" Weisgerber na...@mips.inka.de > > > > It will need this or an alternative. (Unless we can either get the > backtrace functions into libc, or remove the libexecinfo port ;-) Here is a final diff that incorporates jca's patch and sthen's. OK? Index: Makefile === RCS file: /cvs/ports/math/py-numpy/Makefile,v retrieving revision 1.57 diff -u -p -u -p -r1.57 Makefile --- Makefile10 Mar 2020 03:31:45 - 1.57 +++ Makefile1 Apr 2020 18:28:16 - @@ -2,11 +2,10 @@ COMMENT= fast array and numeric programming library for Python -MODPY_EGG_VERSION= 1.14.6 +MODPY_EGG_VERSION= 1.16.5 DISTNAME= numpy-${MODPY_EGG_VERSION} PKGNAME= py-${DISTNAME} EXTRACT_SUFX= .zip -REVISION= 2 CATEGORIES=math devel @@ -15,7 +14,8 @@ HOMEPAGE= https://www.numpy.org/ # BSD PERMIT_PACKAGE = Yes -WANTLIB= blas lapack m pthread ${MODFORTRAN_WANTLIB} ${MODPY_WANTLIB} +WANTLIB += ${MODFORTRAN_WANTLIB} ${MODPY_WANTLIB} +WANTLIB += blas cblas lapack m pthread MODULES= lang/python \ fortran @@ -27,13 +27,14 @@ MODPY_PI = Yes MODPY_SETUPTOOLS= Yes MODPY_DISTUTILS_BUILDARGS = --fcompiler=gnu95 -TEST_DEPENDS= devel/py-nose${MODPY_FLAVOR} \ - devel/py-tz${MODPY_FLAVOR} - +TEST_DEPENDS= ${FULLPKGNAME}:${FULLPKGPATH} \ + devel/py-test${MODPY_FLAVOR} \ + devel/py-tz${MODPY_FLAVOR} MODFORTRAN_COMPILER = gfortran BUILD_DEPENDS = ${MODFORTRAN_BUILD_DEPENDS} -LIB_DEPENDS=math/lapack \ +LIB_DEPENDS= math/cblas \ + math/lapack \ ${MODFORTRAN_LIB_DEPENDS} # Cython 0.28.* uses __attribute__((optimize("Os"))) which @@ -54,7 +55,7 @@ pre-build: cp -f ${WRKSRC}/numpy/distutils/site.cfg ${WRKSRC}/site.cfg do-test: - cd ${WRKDIR} && ${MAKE_ENV} PYTHONPATH=${WRKINST}${MODPY_SITEPKG} ${MODPY_BIN} -c \ + cd ${WRKDIR} && ${MAKE_ENV} ${MODPY_BIN} -c \ 'import numpy ; numpy.test()' .include Index: distinfo === RCS file: /cvs/ports/math/py-numpy/distinfo,v retrieving revision 1.13 diff -u -p -u -p -r1.13 distinfo --- distinfo26 Oct 2018 21:09:31 - 1.13 +++ distinfo1 Apr 2020 18:28:16 - @@ -1,2 +1,2 @@ -SHA256 (numpy-1.14.6.zip) = ElDt9vbEPh14I/CWdBa8GCWLsnHcU2KY6w6gCp5FuAo= -SIZE (numpy-1.14.6.zip) = 4906096 +SHA256 (numpy-1.16.5.zip) = i7RS2U6WSzEiBbDeEjjdcgnaRSNDZTqyFLXWgXgOegw= +SIZE (numpy-1.16.5.zip) = 5138208 Index: patches/patch-numpy_core_include_numpy_npy_common_h === RCS file: /cvs/ports/math/py-numpy/patches/patch-numpy_core_include_numpy_npy_common_h,v retrieving revision 1.6 diff -u -p -u -p -r1.6 patch-numpy_core_include_numpy_npy_common_h --- patches/patch-numpy_core_include_numpy_npy_common_h 30 Jun 2018 21:49:33 - 1.6 +++ patches/patch-numpy_core_include_numpy_npy_common_h 1 Apr 2020 18:28:16 - @@ -1,13 +1,8 @@ $OpenBSD: patch-numpy_core_include_numpy_npy_common_h,v 1.6 2018/06/30 21:49:33 daniel Exp $ -(1) optimize function attributes were added in gcc 4.4. Apply a fix for the warnings. But -the next release of numpy should fix this in a better way. See: -https://github.com/numpy/numpy/issues/5451 +XXX recheck powerpc, is this still needed? -Gets rid of these warnings from the build: -numpy/core/src/multiarray/lowlevel_strided_loops.c.src:120: warning: 'optimize' attribute directive ignored - -(2) py-numpy only checks for expl to determine whether extended-precision +py-numpy only checks for expl to determine whether extended-precision support is present. since we don't have it yet; it implements it's own. however, on alpha, powerpc, it declared functions with types that conflict with C99 (double for *l), therefore failed. @@ -15,36 +10,7 @@ types that conflict with C99 (double for Index: numpy/core/include/numpy/npy_common.h ---
Re: WIP: Update of math/py-numpy to 1.16.5
On Sun, Mar 29, 2020 at 03:38:33PM +0100, Stuart Henderson wrote: > On 2020/03/29 11:49, Christian Weisgerber wrote: > > On 2020-03-28, Stuart Henderson wrote: > > > > >> > Can somenoe commit the numpy update as well then? > > >> > > >> I am about to start a test bulk build on i386 with this. > > > > > > No build problems relating to the numpy update. > > > > Meanwhile, the existing port (1.14.6p2) errored out in my latest bulk > > build: > > > > numpy/core/src/multiarray/temp_elide.c:87:10: fatal error: execinfo.h: No > > such file or directory > > #include > > ^~~~ > > compilation terminated. > > > > Please check that the updated port doesn't haphazardly pick up > > execinfo too. > > > > -- > > Christian "naddy" Weisgerber na...@mips.inka.de > > > > It will need this or an alternative. (Unless we can either get the > backtrace functions into libc, or remove the libexecinfo port ;-) I would encourage any solution. What we have now is a total mess :-( > Index: patches/patch-numpy_core_setup_common_py > === > RCS file: patches/patch-numpy_core_setup_common_py > diff -N patches/patch-numpy_core_setup_common_py > --- /dev/null 1 Jan 1970 00:00:00 - > +++ patches/patch-numpy_core_setup_common_py 29 Mar 2020 14:36:51 - > @@ -0,0 +1,14 @@ > +$OpenBSD$ > + > +Index: numpy/core/setup_common.py > +--- numpy/core/setup_common.py.orig > numpy/core/setup_common.py > +@@ -111,7 +111,7 @@ OPTIONAL_STDFUNCS = ["expm1", "log1p", "acosh", "asinh > + "rint", "trunc", "exp2", "log2", "hypot", "atan2", "pow", > + "copysign", "nextafter", "ftello", "fseeko", > + "strtoll", "strtoull", "cbrt", "strtold_l", "fallocate", > +-"backtrace", "madvise"] > ++"madvise"] > + > + > + OPTIONAL_HEADERS = [ > -- Antoine
Re: WIP: Update of math/py-numpy to 1.16.5
On 2020/03/29 11:49, Christian Weisgerber wrote: > On 2020-03-28, Stuart Henderson wrote: > > >> > Can somenoe commit the numpy update as well then? > >> > >> I am about to start a test bulk build on i386 with this. > > > > No build problems relating to the numpy update. > > Meanwhile, the existing port (1.14.6p2) errored out in my latest bulk > build: > > numpy/core/src/multiarray/temp_elide.c:87:10: fatal error: execinfo.h: No > such file or directory > #include > ^~~~ > compilation terminated. > > Please check that the updated port doesn't haphazardly pick up > execinfo too. > > -- > Christian "naddy" Weisgerber na...@mips.inka.de > It will need this or an alternative. (Unless we can either get the backtrace functions into libc, or remove the libexecinfo port ;-) Index: patches/patch-numpy_core_setup_common_py === RCS file: patches/patch-numpy_core_setup_common_py diff -N patches/patch-numpy_core_setup_common_py --- /dev/null 1 Jan 1970 00:00:00 - +++ patches/patch-numpy_core_setup_common_py29 Mar 2020 14:36:51 - @@ -0,0 +1,14 @@ +$OpenBSD$ + +Index: numpy/core/setup_common.py +--- numpy/core/setup_common.py.orig numpy/core/setup_common.py +@@ -111,7 +111,7 @@ OPTIONAL_STDFUNCS = ["expm1", "log1p", "acosh", "asinh + "rint", "trunc", "exp2", "log2", "hypot", "atan2", "pow", + "copysign", "nextafter", "ftello", "fseeko", + "strtoll", "strtoull", "cbrt", "strtold_l", "fallocate", +-"backtrace", "madvise"] ++"madvise"] + + + OPTIONAL_HEADERS = [
Re: WIP: Update of math/py-numpy to 1.16.5
On 2020-03-28, Stuart Henderson wrote: >> > Can somenoe commit the numpy update as well then? >> >> I am about to start a test bulk build on i386 with this. > > No build problems relating to the numpy update. Meanwhile, the existing port (1.14.6p2) errored out in my latest bulk build: numpy/core/src/multiarray/temp_elide.c:87:10: fatal error: execinfo.h: No such file or directory #include ^~~~ compilation terminated. Please check that the updated port doesn't haphazardly pick up execinfo too. -- Christian "naddy" Weisgerber na...@mips.inka.de
Re: WIP: Update of math/py-numpy to 1.16.5
On 2020/03/27 12:16, Stuart Henderson wrote: > On 2020/03/24 19:30, Paul Irofti wrote: > > Can somenoe commit the numpy update as well then? > > I am about to start a test bulk build on i386 with this. No build problems relating to the numpy update.
Re: WIP: Update of math/py-numpy to 1.16.5
On 2020/03/24 19:30, Paul Irofti wrote: > Can somenoe commit the numpy update as well then? I am about to start a test bulk build on i386 with this.
Re: WIP: Update of math/py-numpy to 1.16.5
On Fri, Mar 13, 2020 at 02:40:05PM +0100, Jeremie Courreges-Anglas wrote: > On Fri, Mar 13 2020, Paul Irofti wrote: > > On Wed, Mar 11, 2020 at 08:58:27PM +0100, Martin Reindl wrote: > >> Am 11.03.20 um 18:53 schrieb Theo Buehler: > >> > On Wed, Mar 11, 2020 at 04:12:56AM +0100, Jeremie Courreges-Anglas wrote: > >> >> On Tue, Mar 10 2020, Theo Buehler wrote: > >> >>> On Tue, Mar 10, 2020 at 06:35:04PM +0100, Jeremie Courreges-Anglas > >> >>> wrote: > >> On Mon, Mar 09 2020, Stuart Henderson wrote: > >> > On 2020/03/09 10:42, Theo Buehler wrote: > >> >> On Mon, Jan 13, 2020 at 12:50:32PM +, Stuart Henderson wrote: > >> >>> 2/3 through a bulk build and I see that this breaks scipy (missing > >> >>> symbols, > >> >>> blas/cblas-related) so needs a bit more work, but I think it's > >> >>> generally > >> >>> along the right lines. > >> >> > >> >> Not sure if this provides any useful clue, but py-numpy doesn't > >> >> build at > >> >> all on sparc64 with this diff, also due to missing blas/cblas > >> >> symbols: > >> > > >> > You'll probably see the same on amd64 with USE_LLD=no. > >> > >> I managed to build scipy with no changes on amd64, so I'm not sure > >> what > >> the problem is on this arch (did not try with USE_LLD=No). > >> > >> However I took a look at the issue reported by tb on sparc64. > >> > >> --8<-- > >> creating /tmp/tmpKcZ0cd/tmp > >> creating /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd > >> compile options: '-I/usr/local/include -I/usr/include -c' > >> cc: /tmp/tmpKcZ0cd/source.c > >> cc /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o -L/usr/local/lib -lcblas -o > >> /tmp/tmpKcZ0cd/a.out > >> /usr/local/lib/libcblas.so.1.0: undefined reference to `ztbsv_' > >> /usr/local/lib/libcblas.so.1.0: undefined reference to `dasum_' > >> > >> [...] > >> > >> /usr/local/lib/libcblas.so.1.0: undefined reference to `zsymm_' > >> /usr/local/lib/libcblas.so.1.0: undefined reference to `ztrsm_' > >> /usr/local/lib/libcblas.so.1.0: undefined reference to `sswap_' > >> collect2: error: ld returned 1 exit status > >> cc /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o -L/usr/local/lib -lblas -o > >> /tmp/tmpKcZ0cd/a.out > >> /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o: In function `main': > >> source.c:(.text.startup+0xdc): undefined reference to `cblas_ddot' > >> collect2: error: ld returned 1 exit status > >> -->8-- > >> > >> libcblas.so doesn't depend on libblas.so so missing symbols are to be > >> expected if one links with -lcblas instead of -lcblas -lblas. The > >> second linking test fails because libblas.so doesn't provide cblas > >> symbols. > >> >>> > >> >>> Thanks, this makes sense. But why does this work with ld.lld? > >> >> > >> >> ld.lld doesn't bother checking that all symbols in libcblas.so can be > >> >> resolved, ld.bfd does. This means that if you link against a library > >> >> that references a bogus symbol or lacks some library interdependency > >> >> (DT_NEEDED) you only get a crash at run time. > >> >> > >> >> On amd64, using the testcase from numpy: > >> >> > >> >> --8<-- > >> >> russell /tmp$ cat r.c > >> >> #include > >> >> int main(int argc, const char *argv[]) > >> >> { > >> >> double a[4] = {1,2,3,4}; > >> >> double b[4] = {5,6,7,8}; > >> >> return cblas_ddot(4, a, 1, b, 1) > 10; > >> >> } > >> >> russell /tmp$ cc -I/usr/local/include r.c -L/usr/local/lib -lcblas > >> >> russell /tmp$ ./a.out > >> >> a.out:/usr/local/lib/libcblas.so.1.0: undefined symbol 'ddot_' > >> >> ld.so: a.out: lazy binding failed! > >> >> Killed > >> >> -->8-- > >> >> > >> >> I suspect Stuart hit a similar problem with this numpy update and scipy. > >> >> > >> >> Using -fuse-ld=bfd in the testcase above would result in the same errors > >> >> as in your log. > >> > > >> > I see. Thank you very much for your explanations. > >> > > >> > FWIW your cblas diff is ok tb (also tested on macppc). > >> > > >> > >> Also tested on arm64, with numpy-1.16.6: > >> 42 failed, 7268 passed, 93 skipped, 168 deselected, 12 xfailed, 1 > >> xpassed, 1 warnings > >> > >> I think this is ready to go into the tree with the cblas diff on top? > >> The update to 1.16.6 is straightforward if you want to stick to 1.16.5 now. > > > > Can you please commit this? Thank you! > > By popular demand I have committed the cblas diff. I haven't looked > closely at the numpy update, you folks probably know better. Great, thank you! Can somenoe commit the numpy update as well then? Paul
Re: WIP: Update of math/py-numpy to 1.16.5
On Fri, Mar 13 2020, Paul Irofti wrote: > On Wed, Mar 11, 2020 at 08:58:27PM +0100, Martin Reindl wrote: >> Am 11.03.20 um 18:53 schrieb Theo Buehler: >> > On Wed, Mar 11, 2020 at 04:12:56AM +0100, Jeremie Courreges-Anglas wrote: >> >> On Tue, Mar 10 2020, Theo Buehler wrote: >> >>> On Tue, Mar 10, 2020 at 06:35:04PM +0100, Jeremie Courreges-Anglas wrote: >> On Mon, Mar 09 2020, Stuart Henderson wrote: >> > On 2020/03/09 10:42, Theo Buehler wrote: >> >> On Mon, Jan 13, 2020 at 12:50:32PM +, Stuart Henderson wrote: >> >>> 2/3 through a bulk build and I see that this breaks scipy (missing >> >>> symbols, >> >>> blas/cblas-related) so needs a bit more work, but I think it's >> >>> generally >> >>> along the right lines. >> >> >> >> Not sure if this provides any useful clue, but py-numpy doesn't build >> >> at >> >> all on sparc64 with this diff, also due to missing blas/cblas symbols: >> > >> > You'll probably see the same on amd64 with USE_LLD=no. >> >> I managed to build scipy with no changes on amd64, so I'm not sure what >> the problem is on this arch (did not try with USE_LLD=No). >> >> However I took a look at the issue reported by tb on sparc64. >> >> --8<-- >> creating /tmp/tmpKcZ0cd/tmp >> creating /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd >> compile options: '-I/usr/local/include -I/usr/include -c' >> cc: /tmp/tmpKcZ0cd/source.c >> cc /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o -L/usr/local/lib -lcblas -o >> /tmp/tmpKcZ0cd/a.out >> /usr/local/lib/libcblas.so.1.0: undefined reference to `ztbsv_' >> /usr/local/lib/libcblas.so.1.0: undefined reference to `dasum_' >> >> [...] >> >> /usr/local/lib/libcblas.so.1.0: undefined reference to `zsymm_' >> /usr/local/lib/libcblas.so.1.0: undefined reference to `ztrsm_' >> /usr/local/lib/libcblas.so.1.0: undefined reference to `sswap_' >> collect2: error: ld returned 1 exit status >> cc /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o -L/usr/local/lib -lblas -o >> /tmp/tmpKcZ0cd/a.out >> /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o: In function `main': >> source.c:(.text.startup+0xdc): undefined reference to `cblas_ddot' >> collect2: error: ld returned 1 exit status >> -->8-- >> >> libcblas.so doesn't depend on libblas.so so missing symbols are to be >> expected if one links with -lcblas instead of -lcblas -lblas. The >> second linking test fails because libblas.so doesn't provide cblas >> symbols. >> >>> >> >>> Thanks, this makes sense. But why does this work with ld.lld? >> >> >> >> ld.lld doesn't bother checking that all symbols in libcblas.so can be >> >> resolved, ld.bfd does. This means that if you link against a library >> >> that references a bogus symbol or lacks some library interdependency >> >> (DT_NEEDED) you only get a crash at run time. >> >> >> >> On amd64, using the testcase from numpy: >> >> >> >> --8<-- >> >> russell /tmp$ cat r.c >> >> #include >> >> int main(int argc, const char *argv[]) >> >> { >> >> double a[4] = {1,2,3,4}; >> >> double b[4] = {5,6,7,8}; >> >> return cblas_ddot(4, a, 1, b, 1) > 10; >> >> } >> >> russell /tmp$ cc -I/usr/local/include r.c -L/usr/local/lib -lcblas >> >> russell /tmp$ ./a.out >> >> a.out:/usr/local/lib/libcblas.so.1.0: undefined symbol 'ddot_' >> >> ld.so: a.out: lazy binding failed! >> >> Killed >> >> -->8-- >> >> >> >> I suspect Stuart hit a similar problem with this numpy update and scipy. >> >> >> >> Using -fuse-ld=bfd in the testcase above would result in the same errors >> >> as in your log. >> > >> > I see. Thank you very much for your explanations. >> > >> > FWIW your cblas diff is ok tb (also tested on macppc). >> > >> >> Also tested on arm64, with numpy-1.16.6: >> 42 failed, 7268 passed, 93 skipped, 168 deselected, 12 xfailed, 1 >> xpassed, 1 warnings >> >> I think this is ready to go into the tree with the cblas diff on top? >> The update to 1.16.6 is straightforward if you want to stick to 1.16.5 now. > > Can you please commit this? Thank you! By popular demand I have committed the cblas diff. I haven't looked closely at the numpy update, you folks probably know better. -- jca | PGP : 0x1524E7EE / 5135 92C1 AD36 5293 2BDF DDCC 0DFA 74AE 1524 E7EE
Re: WIP: Update of math/py-numpy to 1.16.5
On Wed, Mar 11, 2020 at 08:58:27PM +0100, Martin Reindl wrote: > Am 11.03.20 um 18:53 schrieb Theo Buehler: > > On Wed, Mar 11, 2020 at 04:12:56AM +0100, Jeremie Courreges-Anglas wrote: > >> On Tue, Mar 10 2020, Theo Buehler wrote: > >>> On Tue, Mar 10, 2020 at 06:35:04PM +0100, Jeremie Courreges-Anglas wrote: > On Mon, Mar 09 2020, Stuart Henderson wrote: > > On 2020/03/09 10:42, Theo Buehler wrote: > >> On Mon, Jan 13, 2020 at 12:50:32PM +, Stuart Henderson wrote: > >>> 2/3 through a bulk build and I see that this breaks scipy (missing > >>> symbols, > >>> blas/cblas-related) so needs a bit more work, but I think it's > >>> generally > >>> along the right lines. > >> > >> Not sure if this provides any useful clue, but py-numpy doesn't build > >> at > >> all on sparc64 with this diff, also due to missing blas/cblas symbols: > > > > You'll probably see the same on amd64 with USE_LLD=no. > > I managed to build scipy with no changes on amd64, so I'm not sure what > the problem is on this arch (did not try with USE_LLD=No). > > However I took a look at the issue reported by tb on sparc64. > > --8<-- > creating /tmp/tmpKcZ0cd/tmp > creating /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd > compile options: '-I/usr/local/include -I/usr/include -c' > cc: /tmp/tmpKcZ0cd/source.c > cc /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o -L/usr/local/lib -lcblas -o > /tmp/tmpKcZ0cd/a.out > /usr/local/lib/libcblas.so.1.0: undefined reference to `ztbsv_' > /usr/local/lib/libcblas.so.1.0: undefined reference to `dasum_' > > [...] > > /usr/local/lib/libcblas.so.1.0: undefined reference to `zsymm_' > /usr/local/lib/libcblas.so.1.0: undefined reference to `ztrsm_' > /usr/local/lib/libcblas.so.1.0: undefined reference to `sswap_' > collect2: error: ld returned 1 exit status > cc /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o -L/usr/local/lib -lblas -o > /tmp/tmpKcZ0cd/a.out > /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o: In function `main': > source.c:(.text.startup+0xdc): undefined reference to `cblas_ddot' > collect2: error: ld returned 1 exit status > -->8-- > > libcblas.so doesn't depend on libblas.so so missing symbols are to be > expected if one links with -lcblas instead of -lcblas -lblas. The > second linking test fails because libblas.so doesn't provide cblas > symbols. > >>> > >>> Thanks, this makes sense. But why does this work with ld.lld? > >> > >> ld.lld doesn't bother checking that all symbols in libcblas.so can be > >> resolved, ld.bfd does. This means that if you link against a library > >> that references a bogus symbol or lacks some library interdependency > >> (DT_NEEDED) you only get a crash at run time. > >> > >> On amd64, using the testcase from numpy: > >> > >> --8<-- > >> russell /tmp$ cat r.c > >> #include > >> int main(int argc, const char *argv[]) > >> { > >> double a[4] = {1,2,3,4}; > >> double b[4] = {5,6,7,8}; > >> return cblas_ddot(4, a, 1, b, 1) > 10; > >> } > >> russell /tmp$ cc -I/usr/local/include r.c -L/usr/local/lib -lcblas > >> russell /tmp$ ./a.out > >> a.out:/usr/local/lib/libcblas.so.1.0: undefined symbol 'ddot_' > >> ld.so: a.out: lazy binding failed! > >> Killed > >> -->8-- > >> > >> I suspect Stuart hit a similar problem with this numpy update and scipy. > >> > >> Using -fuse-ld=bfd in the testcase above would result in the same errors > >> as in your log. > > > > I see. Thank you very much for your explanations. > > > > FWIW your cblas diff is ok tb (also tested on macppc). > > > > Also tested on arm64, with numpy-1.16.6: > 42 failed, 7268 passed, 93 skipped, 168 deselected, 12 xfailed, 1 > xpassed, 1 warnings > > I think this is ready to go into the tree with the cblas diff on top? > The update to 1.16.6 is straightforward if you want to stick to 1.16.5 now. Can you please commit this? Thank you!
Re: WIP: Update of math/py-numpy to 1.16.5
On Fri, Jan 10, 2020 at 04:00:10PM +, Stuart Henderson wrote: [...] > I also removed part of patch-numpy_core_include_numpy_npy_common_h > that was dealing with gcc-<4.4 which we don't have to worry about > (the gfortran module uses gcc for C as well as Fortran, so it will > always be built with 4.4+ for us). I left the second part in but > we could do with testing powerpc with that file removed completely > (I added an XXX). I believe this second part is still needed. Without it, there are lots of warnings of this type: /usr/include/math.h:425:32: note: expected 'long double *' but argument is of type 'npy_longdouble *' {aka 'double *'} On Tue, Mar 10, 2020 at 06:41:22PM +0100, Jeremie Courreges-Anglas wrote: [...] > Here's an updated diff for numpy-1.16.5, for convenience I decided to > drop the hard requirements I had on cblas>=1.1 (WANTLIB) / > math/cblas>=1.0p7 (LIB_DEPENDS). Somebody with a better knowledge of powerpc should probably take a look: Both flavors build, but the regress tests abort due to a bus error in the multiarray code between 18% and 19% in (trace below). This is new. The 14.0.6 tests only had a handful (9?) unexpected failures (also with jca@'s cblas diff). It's probably unrelated, but there are new warnings from dragon4.c that are of the same kind as the ones fixed by the patch sthen mentioned, e.g.: numpy/core/src/multiarray/dragon4.c:3160:1: note: in expansion of macro 'make_dragon4_typefuncs' make_dragon4_typefuncs(LongDouble, npy_longdouble, NPY_LONGDOUBLE_BINFMT_NAME) ^~ numpy/core/src/multiarray/dragon4.c:2373:48: note: expected 'npy_float64 *' {aka 'double *'} but argument is of type 'npy_longdouble *' {aka 'long double *'} Dragon4_Scratch *scratch, npy_float64 *value, Dragon4_Options *opt) ~^ And here's the start of the trace of the bus error during tests. The python3 version is almost the same. #0 0xd79f731c in _contig_cast_cfloat_to_cdouble () from /usr/obj/pobj/py-numpy-1.16.5/fake-powerpc/usr/local/lib/python2.7/site-packages/numpy/core/_multiarray_umath.so #1 0xd797be04 in raw_array_assign_array () from /usr/obj/pobj/py-numpy-1.16.5/fake-powerpc/usr/local/lib/python2.7/site-packages/numpy/core/_multiarray_umath.so #2 0xd797c55c in PyArray_AssignArray () from /usr/obj/pobj/py-numpy-1.16.5/fake-powerpc/usr/local/lib/python2.7/site-packages/numpy/core/_multiarray_umath.so #3 0xd798b2bc in PyArray_CastToType () from /usr/obj/pobj/py-numpy-1.16.5/fake-powerpc/usr/local/lib/python2.7/site-packages/numpy/core/_multiarray_umath.so #4 0xd7aa8908 in PyUFunc_GenericFunction () from /usr/obj/pobj/py-numpy-1.16.5/fake-powerpc/usr/local/lib/python2.7/site-packages/numpy/core/_multiarray_umath.so #5 0xd7aa9648 in ufunc_generic_call () from /usr/obj/pobj/py-numpy-1.16.5/fake-powerpc/usr/local/lib/python2.7/site-packages/numpy/core/_multiarray_umath.so #6 0xa9513200 in PyObject_Call () from /usr/local/lib/libpython2.7.so.0.0 #7 0xa9513ca4 in PyObject_CallFunctionObjArgs () from /usr/local/lib/libpython2.7.so.0.0 #8 0xd7a48a00 in PyArray_GenericBinaryFunction () from /usr/obj/pobj/py-numpy-1.16.5/fake-powerpc/usr/local/lib/python2.7/site-packages/numpy/core/_multiarray_umath.so #9 0xd7a49640 in array_add () from /usr/obj/pobj/py-numpy-1.16.5/fake-powerpc/usr/local/lib/python2.7/site-packages/numpy/core/_multiarray_umath.so #10 0xa950cfe4 in binary_op1 () from /usr/local/lib/libpython2.7.so.0.0 ...
Re: WIP: Update of math/py-numpy to 1.16.5
Am 11.03.20 um 18:53 schrieb Theo Buehler: > On Wed, Mar 11, 2020 at 04:12:56AM +0100, Jeremie Courreges-Anglas wrote: >> On Tue, Mar 10 2020, Theo Buehler wrote: >>> On Tue, Mar 10, 2020 at 06:35:04PM +0100, Jeremie Courreges-Anglas wrote: On Mon, Mar 09 2020, Stuart Henderson wrote: > On 2020/03/09 10:42, Theo Buehler wrote: >> On Mon, Jan 13, 2020 at 12:50:32PM +, Stuart Henderson wrote: >>> 2/3 through a bulk build and I see that this breaks scipy (missing >>> symbols, >>> blas/cblas-related) so needs a bit more work, but I think it's generally >>> along the right lines. >> >> Not sure if this provides any useful clue, but py-numpy doesn't build at >> all on sparc64 with this diff, also due to missing blas/cblas symbols: > > You'll probably see the same on amd64 with USE_LLD=no. I managed to build scipy with no changes on amd64, so I'm not sure what the problem is on this arch (did not try with USE_LLD=No). However I took a look at the issue reported by tb on sparc64. --8<-- creating /tmp/tmpKcZ0cd/tmp creating /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd compile options: '-I/usr/local/include -I/usr/include -c' cc: /tmp/tmpKcZ0cd/source.c cc /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o -L/usr/local/lib -lcblas -o /tmp/tmpKcZ0cd/a.out /usr/local/lib/libcblas.so.1.0: undefined reference to `ztbsv_' /usr/local/lib/libcblas.so.1.0: undefined reference to `dasum_' [...] /usr/local/lib/libcblas.so.1.0: undefined reference to `zsymm_' /usr/local/lib/libcblas.so.1.0: undefined reference to `ztrsm_' /usr/local/lib/libcblas.so.1.0: undefined reference to `sswap_' collect2: error: ld returned 1 exit status cc /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o -L/usr/local/lib -lblas -o /tmp/tmpKcZ0cd/a.out /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o: In function `main': source.c:(.text.startup+0xdc): undefined reference to `cblas_ddot' collect2: error: ld returned 1 exit status -->8-- libcblas.so doesn't depend on libblas.so so missing symbols are to be expected if one links with -lcblas instead of -lcblas -lblas. The second linking test fails because libblas.so doesn't provide cblas symbols. >>> >>> Thanks, this makes sense. But why does this work with ld.lld? >> >> ld.lld doesn't bother checking that all symbols in libcblas.so can be >> resolved, ld.bfd does. This means that if you link against a library >> that references a bogus symbol or lacks some library interdependency >> (DT_NEEDED) you only get a crash at run time. >> >> On amd64, using the testcase from numpy: >> >> --8<-- >> russell /tmp$ cat r.c >> #include >> int main(int argc, const char *argv[]) >> { >> double a[4] = {1,2,3,4}; >> double b[4] = {5,6,7,8}; >> return cblas_ddot(4, a, 1, b, 1) > 10; >> } >> russell /tmp$ cc -I/usr/local/include r.c -L/usr/local/lib -lcblas >> russell /tmp$ ./a.out >> a.out:/usr/local/lib/libcblas.so.1.0: undefined symbol 'ddot_' >> ld.so: a.out: lazy binding failed! >> Killed >> -->8-- >> >> I suspect Stuart hit a similar problem with this numpy update and scipy. >> >> Using -fuse-ld=bfd in the testcase above would result in the same errors >> as in your log. > > I see. Thank you very much for your explanations. > > FWIW your cblas diff is ok tb (also tested on macppc). > Also tested on arm64, with numpy-1.16.6: 42 failed, 7268 passed, 93 skipped, 168 deselected, 12 xfailed, 1 xpassed, 1 warnings I think this is ready to go into the tree with the cblas diff on top? The update to 1.16.6 is straightforward if you want to stick to 1.16.5 now. -m
Re: WIP: Update of math/py-numpy to 1.16.5
On Wed, Mar 11, 2020 at 04:12:56AM +0100, Jeremie Courreges-Anglas wrote: > On Tue, Mar 10 2020, Theo Buehler wrote: > > On Tue, Mar 10, 2020 at 06:35:04PM +0100, Jeremie Courreges-Anglas wrote: > >> On Mon, Mar 09 2020, Stuart Henderson wrote: > >> > On 2020/03/09 10:42, Theo Buehler wrote: > >> >> On Mon, Jan 13, 2020 at 12:50:32PM +, Stuart Henderson wrote: > >> >> > 2/3 through a bulk build and I see that this breaks scipy (missing > >> >> > symbols, > >> >> > blas/cblas-related) so needs a bit more work, but I think it's > >> >> > generally > >> >> > along the right lines. > >> >> > >> >> Not sure if this provides any useful clue, but py-numpy doesn't build at > >> >> all on sparc64 with this diff, also due to missing blas/cblas symbols: > >> > > >> > You'll probably see the same on amd64 with USE_LLD=no. > >> > >> I managed to build scipy with no changes on amd64, so I'm not sure what > >> the problem is on this arch (did not try with USE_LLD=No). > >> > >> However I took a look at the issue reported by tb on sparc64. > >> > >> --8<-- > >> creating /tmp/tmpKcZ0cd/tmp > >> creating /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd > >> compile options: '-I/usr/local/include -I/usr/include -c' > >> cc: /tmp/tmpKcZ0cd/source.c > >> cc /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o -L/usr/local/lib -lcblas -o > >> /tmp/tmpKcZ0cd/a.out > >> /usr/local/lib/libcblas.so.1.0: undefined reference to `ztbsv_' > >> /usr/local/lib/libcblas.so.1.0: undefined reference to `dasum_' > >> > >> [...] > >> > >> /usr/local/lib/libcblas.so.1.0: undefined reference to `zsymm_' > >> /usr/local/lib/libcblas.so.1.0: undefined reference to `ztrsm_' > >> /usr/local/lib/libcblas.so.1.0: undefined reference to `sswap_' > >> collect2: error: ld returned 1 exit status > >> cc /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o -L/usr/local/lib -lblas -o > >> /tmp/tmpKcZ0cd/a.out > >> /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o: In function `main': > >> source.c:(.text.startup+0xdc): undefined reference to `cblas_ddot' > >> collect2: error: ld returned 1 exit status > >> -->8-- > >> > >> libcblas.so doesn't depend on libblas.so so missing symbols are to be > >> expected if one links with -lcblas instead of -lcblas -lblas. The > >> second linking test fails because libblas.so doesn't provide cblas > >> symbols. > > > > Thanks, this makes sense. But why does this work with ld.lld? > > ld.lld doesn't bother checking that all symbols in libcblas.so can be > resolved, ld.bfd does. This means that if you link against a library > that references a bogus symbol or lacks some library interdependency > (DT_NEEDED) you only get a crash at run time. > > On amd64, using the testcase from numpy: > > --8<-- > russell /tmp$ cat r.c > #include > int main(int argc, const char *argv[]) > { > double a[4] = {1,2,3,4}; > double b[4] = {5,6,7,8}; > return cblas_ddot(4, a, 1, b, 1) > 10; > } > russell /tmp$ cc -I/usr/local/include r.c -L/usr/local/lib -lcblas > russell /tmp$ ./a.out > a.out:/usr/local/lib/libcblas.so.1.0: undefined symbol 'ddot_' > ld.so: a.out: lazy binding failed! > Killed > -->8-- > > I suspect Stuart hit a similar problem with this numpy update and scipy. > > Using -fuse-ld=bfd in the testcase above would result in the same errors > as in your log. I see. Thank you very much for your explanations. FWIW your cblas diff is ok tb (also tested on macppc).
Re: WIP: Update of math/py-numpy to 1.16.5
On Tue, Mar 10 2020, Theo Buehler wrote: > On Tue, Mar 10, 2020 at 06:35:04PM +0100, Jeremie Courreges-Anglas wrote: >> On Mon, Mar 09 2020, Stuart Henderson wrote: >> > On 2020/03/09 10:42, Theo Buehler wrote: >> >> On Mon, Jan 13, 2020 at 12:50:32PM +, Stuart Henderson wrote: >> >> > 2/3 through a bulk build and I see that this breaks scipy (missing >> >> > symbols, >> >> > blas/cblas-related) so needs a bit more work, but I think it's generally >> >> > along the right lines. >> >> >> >> Not sure if this provides any useful clue, but py-numpy doesn't build at >> >> all on sparc64 with this diff, also due to missing blas/cblas symbols: >> > >> > You'll probably see the same on amd64 with USE_LLD=no. >> >> I managed to build scipy with no changes on amd64, so I'm not sure what >> the problem is on this arch (did not try with USE_LLD=No). >> >> However I took a look at the issue reported by tb on sparc64. >> >> --8<-- >> creating /tmp/tmpKcZ0cd/tmp >> creating /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd >> compile options: '-I/usr/local/include -I/usr/include -c' >> cc: /tmp/tmpKcZ0cd/source.c >> cc /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o -L/usr/local/lib -lcblas -o >> /tmp/tmpKcZ0cd/a.out >> /usr/local/lib/libcblas.so.1.0: undefined reference to `ztbsv_' >> /usr/local/lib/libcblas.so.1.0: undefined reference to `dasum_' >> >> [...] >> >> /usr/local/lib/libcblas.so.1.0: undefined reference to `zsymm_' >> /usr/local/lib/libcblas.so.1.0: undefined reference to `ztrsm_' >> /usr/local/lib/libcblas.so.1.0: undefined reference to `sswap_' >> collect2: error: ld returned 1 exit status >> cc /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o -L/usr/local/lib -lblas -o >> /tmp/tmpKcZ0cd/a.out >> /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o: In function `main': >> source.c:(.text.startup+0xdc): undefined reference to `cblas_ddot' >> collect2: error: ld returned 1 exit status >> -->8-- >> >> libcblas.so doesn't depend on libblas.so so missing symbols are to be >> expected if one links with -lcblas instead of -lcblas -lblas. The >> second linking test fails because libblas.so doesn't provide cblas >> symbols. > > Thanks, this makes sense. But why does this work with ld.lld? ld.lld doesn't bother checking that all symbols in libcblas.so can be resolved, ld.bfd does. This means that if you link against a library that references a bogus symbol or lacks some library interdependency (DT_NEEDED) you only get a crash at run time. On amd64, using the testcase from numpy: --8<-- russell /tmp$ cat r.c #include int main(int argc, const char *argv[]) { double a[4] = {1,2,3,4}; double b[4] = {5,6,7,8}; return cblas_ddot(4, a, 1, b, 1) > 10; } russell /tmp$ cc -I/usr/local/include r.c -L/usr/local/lib -lcblas russell /tmp$ ./a.out a.out:/usr/local/lib/libcblas.so.1.0: undefined symbol 'ddot_' ld.so: a.out: lazy binding failed! Killed -->8-- I suspect Stuart hit a similar problem with this numpy update and scipy. Using -fuse-ld=bfd in the testcase above would result in the same errors as in your log. >> >> I think the way forward is to make libcblas.so depend on libblas.so >> (this is what you get eg on Debian). It would probably make sense to do >> the same with lapack. >> >> With the following cblas diff I can build py-numpy-1.16.5 on amd64 and >> sparc64. cc'ing Steven. >> >> ok? >> >> >> Index: Makefile >> === >> RCS file: /cvs/ports/math/cblas/Makefile,v >> retrieving revision 1.20 >> diff -u -p -r1.20 Makefile >> --- Makefile 12 Jul 2019 20:47:40 - 1.20 >> +++ Makefile 10 Mar 2020 16:53:46 - >> @@ -5,8 +5,8 @@ COMMENT= C interface to the BLAS library >> VERSION=1.0 >> DISTNAME= cblas >> PKGNAME=${DISTNAME}-${VERSION} >> -REVISION= 6 >> -SHARED_LIBS=cblas 1.0 >> +REVISION= 7 >> +SHARED_LIBS=cblas 1.1 >> >> CATEGORIES= math >> >> Index: files/Makefile >> === >> RCS file: /cvs/ports/math/cblas/files/Makefile,v >> retrieving revision 1.1.1.1 >> diff -u -p -r1.1.1.1 Makefile >> --- files/Makefile 2 Oct 2006 21:58:25 - 1.1.1.1 >> +++ files/Makefile 10 Mar 2020 16:53:46 - >> @@ -42,5 +42,6 @@ cblas_dgemm.c cblas_sdsdot.c >> cblas_dgemv.c cblas_sgbmv.c cblas_zhemv.c \ >> cblas_dger.ccblas_sgemm.c cblas_zher.c \ >> cblas_dnrm2.c cblas_sgemv.c cblas_zher2.c >> +LDADD=-lblas -lgfortran >> >> .include >> Index: pkg/PLIST >> === >> RCS file: /cvs/ports/math/cblas/pkg/PLIST,v >> retrieving revision 1.4 >> diff -u -p -r1.4 PLIST >> --- pkg/PLIST16 Mar 2015 18:07:49 - 1.4 >> +++ pkg/PLIST10 Mar 2020 16:53:46 - >> @@ -1,6 +1,6 @@ >> @comment $OpenBSD: PLIST,v 1.4 2015/03/16 18:07:49 naddy Exp $ >> include/cblas.h >> include/cblas_f77.h >> -lib/libcblas.a >> +@static-lib lib/libcblas.a
Re: WIP: Update of math/py-numpy to 1.16.5
On Tue, Mar 10, 2020 at 06:35:04PM +0100, Jeremie Courreges-Anglas wrote: > On Mon, Mar 09 2020, Stuart Henderson wrote: > > On 2020/03/09 10:42, Theo Buehler wrote: > >> On Mon, Jan 13, 2020 at 12:50:32PM +, Stuart Henderson wrote: > >> > 2/3 through a bulk build and I see that this breaks scipy (missing > >> > symbols, > >> > blas/cblas-related) so needs a bit more work, but I think it's generally > >> > along the right lines. > >> > >> Not sure if this provides any useful clue, but py-numpy doesn't build at > >> all on sparc64 with this diff, also due to missing blas/cblas symbols: > > > > You'll probably see the same on amd64 with USE_LLD=no. > > I managed to build scipy with no changes on amd64, so I'm not sure what > the problem is on this arch (did not try with USE_LLD=No). > > However I took a look at the issue reported by tb on sparc64. > > --8<-- > creating /tmp/tmpKcZ0cd/tmp > creating /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd > compile options: '-I/usr/local/include -I/usr/include -c' > cc: /tmp/tmpKcZ0cd/source.c > cc /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o -L/usr/local/lib -lcblas -o > /tmp/tmpKcZ0cd/a.out > /usr/local/lib/libcblas.so.1.0: undefined reference to `ztbsv_' > /usr/local/lib/libcblas.so.1.0: undefined reference to `dasum_' > > [...] > > /usr/local/lib/libcblas.so.1.0: undefined reference to `zsymm_' > /usr/local/lib/libcblas.so.1.0: undefined reference to `ztrsm_' > /usr/local/lib/libcblas.so.1.0: undefined reference to `sswap_' > collect2: error: ld returned 1 exit status > cc /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o -L/usr/local/lib -lblas -o > /tmp/tmpKcZ0cd/a.out > /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o: In function `main': > source.c:(.text.startup+0xdc): undefined reference to `cblas_ddot' > collect2: error: ld returned 1 exit status > -->8-- > > libcblas.so doesn't depend on libblas.so so missing symbols are to be > expected if one links with -lcblas instead of -lcblas -lblas. The > second linking test fails because libblas.so doesn't provide cblas > symbols. Thanks, this makes sense. But why does this work with ld.lld? > > I think the way forward is to make libcblas.so depend on libblas.so > (this is what you get eg on Debian). It would probably make sense to do > the same with lapack. > > With the following cblas diff I can build py-numpy-1.16.5 on amd64 and > sparc64. cc'ing Steven. > > ok? > > > Index: Makefile > === > RCS file: /cvs/ports/math/cblas/Makefile,v > retrieving revision 1.20 > diff -u -p -r1.20 Makefile > --- Makefile 12 Jul 2019 20:47:40 - 1.20 > +++ Makefile 10 Mar 2020 16:53:46 - > @@ -5,8 +5,8 @@ COMMENT= C interface to the BLAS library > VERSION= 1.0 > DISTNAME=cblas > PKGNAME= ${DISTNAME}-${VERSION} > -REVISION=6 > -SHARED_LIBS= cblas 1.0 > +REVISION=7 > +SHARED_LIBS= cblas 1.1 > > CATEGORIES= math > > Index: files/Makefile > === > RCS file: /cvs/ports/math/cblas/files/Makefile,v > retrieving revision 1.1.1.1 > diff -u -p -r1.1.1.1 Makefile > --- files/Makefile2 Oct 2006 21:58:25 - 1.1.1.1 > +++ files/Makefile10 Mar 2020 16:53:46 - > @@ -42,5 +42,6 @@ cblas_dgemm.c cblas_sdsdot.c > cblas_dgemv.c cblas_sgbmv.c cblas_zhemv.c \ > cblas_dger.ccblas_sgemm.c cblas_zher.c \ > cblas_dnrm2.c cblas_sgemv.c cblas_zher2.c > +LDADD=-lblas -lgfortran > > .include > Index: pkg/PLIST > === > RCS file: /cvs/ports/math/cblas/pkg/PLIST,v > retrieving revision 1.4 > diff -u -p -r1.4 PLIST > --- pkg/PLIST 16 Mar 2015 18:07:49 - 1.4 > +++ pkg/PLIST 10 Mar 2020 16:53:46 - > @@ -1,6 +1,6 @@ > @comment $OpenBSD: PLIST,v 1.4 2015/03/16 18:07:49 naddy Exp $ > include/cblas.h > include/cblas_f77.h > -lib/libcblas.a > +@static-lib lib/libcblas.a > @lib lib/libcblas.so.${LIBcblas_VERSION} > -lib/libcblas_p.a > +@static-lib lib/libcblas_p.a > > > -- > jca | PGP : 0x1524E7EE / 5135 92C1 AD36 5293 2BDF DDCC 0DFA 74AE 1524 E7EE >
Re: WIP: Update of math/py-numpy to 1.16.5
On Tue, Mar 10 2020, Jeremie Courreges-Anglas wrote: > On Mon, Mar 09 2020, Stuart Henderson wrote: >> On 2020/03/09 10:42, Theo Buehler wrote: >>> On Mon, Jan 13, 2020 at 12:50:32PM +, Stuart Henderson wrote: >>> > 2/3 through a bulk build and I see that this breaks scipy (missing >>> > symbols, >>> > blas/cblas-related) so needs a bit more work, but I think it's generally >>> > along the right lines. >>> >>> Not sure if this provides any useful clue, but py-numpy doesn't build at >>> all on sparc64 with this diff, also due to missing blas/cblas symbols: >> >> You'll probably see the same on amd64 with USE_LLD=no. > > I managed to build scipy with no changes on amd64, so I'm not sure what > the problem is on this arch (did not try with USE_LLD=No). > > However I took a look at the issue reported by tb on sparc64. > > --8<-- > creating /tmp/tmpKcZ0cd/tmp > creating /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd > compile options: '-I/usr/local/include -I/usr/include -c' > cc: /tmp/tmpKcZ0cd/source.c > cc /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o -L/usr/local/lib -lcblas -o > /tmp/tmpKcZ0cd/a.out > /usr/local/lib/libcblas.so.1.0: undefined reference to `ztbsv_' > /usr/local/lib/libcblas.so.1.0: undefined reference to `dasum_' > > [...] > > /usr/local/lib/libcblas.so.1.0: undefined reference to `zsymm_' > /usr/local/lib/libcblas.so.1.0: undefined reference to `ztrsm_' > /usr/local/lib/libcblas.so.1.0: undefined reference to `sswap_' > collect2: error: ld returned 1 exit status > cc /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o -L/usr/local/lib -lblas -o > /tmp/tmpKcZ0cd/a.out > /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o: In function `main': > source.c:(.text.startup+0xdc): undefined reference to `cblas_ddot' > collect2: error: ld returned 1 exit status > -->8-- > > libcblas.so doesn't depend on libblas.so so missing symbols are to be > expected if one links with -lcblas instead of -lcblas -lblas. The > second linking test fails because libblas.so doesn't provide cblas > symbols. > > I think the way forward is to make libcblas.so depend on libblas.so > (this is what you get eg on Debian). It would probably make sense to do > the same with lapack. > > With the following cblas diff I can build py-numpy-1.16.5 on amd64 and > sparc64. cc'ing Steven. > > ok? Here's an updated diff for numpy-1.16.5, for convenience I decided to drop the hard requirements I had on cblas>=1.1 (WANTLIB) / math/cblas>=1.0p7 (LIB_DEPENDS). Index: Makefile === RCS file: /cvs/ports/math/py-numpy/Makefile,v retrieving revision 1.57 diff -u -p -r1.57 Makefile --- Makefile10 Mar 2020 03:31:45 - 1.57 +++ Makefile10 Mar 2020 17:06:13 - @@ -2,11 +2,10 @@ COMMENT= fast array and numeric programming library for Python -MODPY_EGG_VERSION= 1.14.6 +MODPY_EGG_VERSION= 1.16.5 DISTNAME= numpy-${MODPY_EGG_VERSION} PKGNAME= py-${DISTNAME} EXTRACT_SUFX= .zip -REVISION= 2 CATEGORIES=math devel @@ -15,7 +14,8 @@ HOMEPAGE= https://www.numpy.org/ # BSD PERMIT_PACKAGE = Yes -WANTLIB= blas lapack m pthread ${MODFORTRAN_WANTLIB} ${MODPY_WANTLIB} +WANTLIB += ${MODFORTRAN_WANTLIB} ${MODPY_WANTLIB} +WANTLIB += blas cblas lapack m pthread MODULES= lang/python \ fortran @@ -27,13 +27,14 @@ MODPY_PI = Yes MODPY_SETUPTOOLS= Yes MODPY_DISTUTILS_BUILDARGS = --fcompiler=gnu95 -TEST_DEPENDS= devel/py-nose${MODPY_FLAVOR} \ - devel/py-tz${MODPY_FLAVOR} - +TEST_DEPENDS= ${FULLPKGNAME}:${FULLPKGPATH} \ + devel/py-test${MODPY_FLAVOR} \ + devel/py-tz${MODPY_FLAVOR} MODFORTRAN_COMPILER = gfortran BUILD_DEPENDS = ${MODFORTRAN_BUILD_DEPENDS} -LIB_DEPENDS=math/lapack \ +LIB_DEPENDS= math/cblas \ + math/lapack \ ${MODFORTRAN_LIB_DEPENDS} # Cython 0.28.* uses __attribute__((optimize("Os"))) which Index: distinfo === RCS file: /cvs/ports/math/py-numpy/distinfo,v retrieving revision 1.13 diff -u -p -r1.13 distinfo --- distinfo26 Oct 2018 21:09:31 - 1.13 +++ distinfo10 Mar 2020 17:06:13 - @@ -1,2 +1,2 @@ -SHA256 (numpy-1.14.6.zip) = ElDt9vbEPh14I/CWdBa8GCWLsnHcU2KY6w6gCp5FuAo= -SIZE (numpy-1.14.6.zip) = 4906096 +SHA256 (numpy-1.16.5.zip) = i7RS2U6WSzEiBbDeEjjdcgnaRSNDZTqyFLXWgXgOegw= +SIZE (numpy-1.16.5.zip) = 5138208 Index: patches/patch-numpy_core_include_numpy_npy_common_h === RCS file: /cvs/ports/math/py-numpy/patches/patch-numpy_core_include_numpy_npy_common_h,v retrieving revision 1.6 diff -u -p -r1.6 patch-numpy_core_include_numpy_npy_common_h --- patches/patch-numpy_core_include_numpy_npy_common_h 30 Jun 2018 21:49:33 - 1.6 +++
Re: WIP: Update of math/py-numpy to 1.16.5
On Mon, Mar 09 2020, Stuart Henderson wrote: > On 2020/03/09 10:42, Theo Buehler wrote: >> On Mon, Jan 13, 2020 at 12:50:32PM +, Stuart Henderson wrote: >> > 2/3 through a bulk build and I see that this breaks scipy (missing symbols, >> > blas/cblas-related) so needs a bit more work, but I think it's generally >> > along the right lines. >> >> Not sure if this provides any useful clue, but py-numpy doesn't build at >> all on sparc64 with this diff, also due to missing blas/cblas symbols: > > You'll probably see the same on amd64 with USE_LLD=no. I managed to build scipy with no changes on amd64, so I'm not sure what the problem is on this arch (did not try with USE_LLD=No). However I took a look at the issue reported by tb on sparc64. --8<-- creating /tmp/tmpKcZ0cd/tmp creating /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd compile options: '-I/usr/local/include -I/usr/include -c' cc: /tmp/tmpKcZ0cd/source.c cc /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o -L/usr/local/lib -lcblas -o /tmp/tmpKcZ0cd/a.out /usr/local/lib/libcblas.so.1.0: undefined reference to `ztbsv_' /usr/local/lib/libcblas.so.1.0: undefined reference to `dasum_' [...] /usr/local/lib/libcblas.so.1.0: undefined reference to `zsymm_' /usr/local/lib/libcblas.so.1.0: undefined reference to `ztrsm_' /usr/local/lib/libcblas.so.1.0: undefined reference to `sswap_' collect2: error: ld returned 1 exit status cc /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o -L/usr/local/lib -lblas -o /tmp/tmpKcZ0cd/a.out /tmp/tmpKcZ0cd/tmp/tmpKcZ0cd/source.o: In function `main': source.c:(.text.startup+0xdc): undefined reference to `cblas_ddot' collect2: error: ld returned 1 exit status -->8-- libcblas.so doesn't depend on libblas.so so missing symbols are to be expected if one links with -lcblas instead of -lcblas -lblas. The second linking test fails because libblas.so doesn't provide cblas symbols. I think the way forward is to make libcblas.so depend on libblas.so (this is what you get eg on Debian). It would probably make sense to do the same with lapack. With the following cblas diff I can build py-numpy-1.16.5 on amd64 and sparc64. cc'ing Steven. ok? Index: Makefile === RCS file: /cvs/ports/math/cblas/Makefile,v retrieving revision 1.20 diff -u -p -r1.20 Makefile --- Makefile12 Jul 2019 20:47:40 - 1.20 +++ Makefile10 Mar 2020 16:53:46 - @@ -5,8 +5,8 @@ COMMENT=C interface to the BLAS library VERSION= 1.0 DISTNAME= cblas PKGNAME= ${DISTNAME}-${VERSION} -REVISION= 6 -SHARED_LIBS= cblas 1.0 +REVISION= 7 +SHARED_LIBS= cblas 1.1 CATEGORIES=math Index: files/Makefile === RCS file: /cvs/ports/math/cblas/files/Makefile,v retrieving revision 1.1.1.1 diff -u -p -r1.1.1.1 Makefile --- files/Makefile 2 Oct 2006 21:58:25 - 1.1.1.1 +++ files/Makefile 10 Mar 2020 16:53:46 - @@ -42,5 +42,6 @@ cblas_dgemm.c cblas_sdsdot.c cblas_dgemv.c cblas_sgbmv.c cblas_zhemv.c \ cblas_dger.ccblas_sgemm.c cblas_zher.c \ cblas_dnrm2.c cblas_sgemv.c cblas_zher2.c +LDADD=-lblas -lgfortran .include Index: pkg/PLIST === RCS file: /cvs/ports/math/cblas/pkg/PLIST,v retrieving revision 1.4 diff -u -p -r1.4 PLIST --- pkg/PLIST 16 Mar 2015 18:07:49 - 1.4 +++ pkg/PLIST 10 Mar 2020 16:53:46 - @@ -1,6 +1,6 @@ @comment $OpenBSD: PLIST,v 1.4 2015/03/16 18:07:49 naddy Exp $ include/cblas.h include/cblas_f77.h -lib/libcblas.a +@static-lib lib/libcblas.a @lib lib/libcblas.so.${LIBcblas_VERSION} -lib/libcblas_p.a +@static-lib lib/libcblas_p.a -- jca | PGP : 0x1524E7EE / 5135 92C1 AD36 5293 2BDF DDCC 0DFA 74AE 1524 E7EE
Re: WIP: Update of math/py-numpy to 1.16.5
On 2020/03/09 10:42, Theo Buehler wrote: > On Mon, Jan 13, 2020 at 12:50:32PM +, Stuart Henderson wrote: > > 2/3 through a bulk build and I see that this breaks scipy (missing symbols, > > blas/cblas-related) so needs a bit more work, but I think it's generally > > along the right lines. > > Not sure if this provides any useful clue, but py-numpy doesn't build at > all on sparc64 with this diff, also due to missing blas/cblas symbols: You'll probably see the same on amd64 with USE_LLD=no.
Re: WIP: Update of math/py-numpy to 1.16.5
On Mon, Jan 13, 2020 at 12:50:32PM +, Stuart Henderson wrote: > 2/3 through a bulk build and I see that this breaks scipy (missing symbols, > blas/cblas-related) so needs a bit more work, but I think it's generally > along the right lines. Not sure if this provides any useful clue, but py-numpy doesn't build at all on sparc64 with this diff, also due to missing blas/cblas symbols: ===> py-numpy-1.16.5 depends on: g95->=8,<9 -> g95-8.3.0p4 ===> py-numpy-1.16.5 depends on: python->=2.7,<2.8 -> python-2.7.17p1 ===> py-numpy-1.16.5 depends on: py-setuptools->=39.0.1v0 -> py-setuptools-41.6.0v0 ===> py-numpy-1.16.5 depends on: gcc->=8,<9 -> gcc-8.3.0p4 ===> py-numpy-1.16.5 depends on: unzip-* -> unzip-6.0p12 ===> py-numpy-1.16.5 depends on: cblas-* -> cblas-1.0p6 ===> py-numpy-1.16.5 depends on: lapack-* -> lapack-3.8.0p1 ===> py-numpy-1.16.5 depends on: gcc-libs->=8,<9 -> gcc-libs-8.3.0p4 ===> Verifying specs: gfortran>=8 python2.7 blas cblas lapack m pthread gfortran>=8 ===> found gfortran.8.0 python2.7.0.0 blas.2.1 cblas.1.0 lapack.7.1 m.10.1 pthread.26.1 ===> Checking files for py-numpy-1.16.5 `/usr/ports/distfiles/numpy-1.16.5.zip' is up to date. >> (SHA256) numpy-1.16.5.zip: OK ===> Extracting for py-numpy-1.16.5 ===> Patching for py-numpy-1.16.5 ===> Applying OpenBSD patch patch-numpy_core_include_numpy_npy_common_h Hmm... Looks like a unified diff to me... The text leading up to this was: -- |$OpenBSD: patch-numpy_core_include_numpy_npy_common_h,v 1.6 2018/06/30 21:49:33 daniel Exp $ | |XXX recheck powerpc, is this still needed? | |py-numpy only checks for expl to determine whether extended-precision |support is present. since we don't have it yet; it implements |it's own. however, on alpha, powerpc, it declared functions with |types that conflict with C99 (double for *l), therefore failed. | |Index: numpy/core/include/numpy/npy_common.h |--- numpy/core/include/numpy/npy_common.h.orig |+++ numpy/core/include/numpy/npy_common.h -- Patching file numpy/core/include/numpy/npy_common.h using Plan A... Hunk #1 succeeded at 320. done ===> Ignoring patchfile patch-numpy_core_include_numpy_npy_common_h.orig ===> Applying OpenBSD patch patch-numpy_distutils_command_build_src_py Hmm... Looks like a unified diff to me... The text leading up to this was: -- |$OpenBSD: patch-numpy_distutils_command_build_src_py,v 1.3 2018/06/30 21:49:33 daniel Exp $ | |fix build of other packages (e.g. py-scipy) in some cases (e.g. when |WRKOBJDIR has a trailing slash) | |Index: numpy/distutils/command/build_src.py |--- numpy/distutils/command/build_src.py.orig |+++ numpy/distutils/command/build_src.py -- Patching file numpy/distutils/command/build_src.py using Plan A... Hunk #1 succeeded at 370. done ===> Ignoring patchfile patch-numpy_distutils_command_build_src_py.orig ===> Applying OpenBSD patch patch-numpy_distutils_fcompiler_gnu_py Hmm... Looks like a unified diff to me... The text leading up to this was: -- |$OpenBSD: patch-numpy_distutils_fcompiler_gnu_py,v 1.2 2018/06/30 21:49:33 daniel Exp $ | |Causes segmentation fault on powerpc when building py-scipy. | |See discussion at: |https://github.com/numpy/numpy/issues/5451 | |Index: numpy/distutils/fcompiler/gnu.py |--- numpy/distutils/fcompiler/gnu.py.orig |+++ numpy/distutils/fcompiler/gnu.py -- Patching file numpy/distutils/fcompiler/gnu.py using Plan A... Hunk #1 succeeded at 245. done ===> Ignoring patchfile patch-numpy_distutils_fcompiler_gnu_py.orig ===> Applying OpenBSD patch patch-numpy_distutils_site_cfg Hmm... Looks like a unified diff to me... The text leading up to this was: -- |$OpenBSD: patch-numpy_distutils_site_cfg,v 1.2 2009/02/16 10:10:09 eric Exp $ |--- numpy/distutils/site.cfg.orig Fri Feb 13 15:41:03 2009 |+++ numpy/distutils/site.cfg Fri Feb 13 15:41:47 2009 -- (Creating file numpy/distutils/site.cfg...) Patching file numpy/distutils/site.cfg using Plan A... Empty context always matches. Hunk #1 succeeded at 1. done ===> Compiler link: gcc -> /usr/local/bin/egcc ===> Compiler link: cc -> /usr/local/bin/egcc ===> Compiler link: gfortran -> /usr/local/bin/egfortran ===> Compiler link: c++ -> /usr/bin/c++ ===> Generating configure for py-numpy-1.16.5 ===> Configuring for py-numpy-1.16.5 ===> Building for py-numpy-1.16.5 cp -f /usr/ports/pobj/py-numpy-1.16.5/numpy-1.16.5/numpy/distutils/site.cfg /usr/ports/pobj/py-numpy-1.16.5/numpy-1.16.5/site.cfg Running from numpy source directory. running egg_info creating numpy.egg-info writing numpy.egg-info/PKG-INFO writing top-level names to numpy.egg-info/top_level.txt writing dependency_links to numpy.egg-info/dependency_links.txt writing entry points to numpy.egg-info/entry_points.txt writing manifest file 'numpy.egg-info/SOURCES.txt' reading manifest file
Re: WIP: Update of math/py-numpy to 1.16.5
2/3 through a bulk build and I see that this breaks scipy (missing symbols, blas/cblas-related) so needs a bit more work, but I think it's generally along the right lines. -- Sent from a phone, apologies for poor formatting. On 10 January 2020 16:00:43 Stuart Henderson wrote: On 2019/11/02 02:55, Kurt Mosiejczuk wrote: I got this as far as compiling with flang, running its tests mostly sucessfully, and packaging. I'm about to start traveling tomorrow so I won't get to testing all the consumers right away. I figured I'd send this out in case someone wants to throw it through a bulk or pick up where I left off. Here's a diff based on Kurt's but reinstating gfortran as compiler. I'm not sure what the compiler change was originally done for, perhaps it was just to avoid the problem with gcc flags that jca already fixed in python/2.7? I also removed part of patch-numpy_core_include_numpy_npy_common_h that was dealing with gcc-<4.4 which we don't have to worry about (the gfortran module uses gcc for C as well as Fortran, so it will always be built with 4.4+ for us). I left the second part in but we could do with testing powerpc with that file removed completely (I added an XXX). Builds ok (py2+3) on amd64, tests are in good shape: py2: 10 failed, 7173 passed, 159 skipped, 166 deselected, 12 xfailed, 1 xpassed, 3 warnings in 265.80 seconds py3: 10 failed, 7266 passed, 83 skipped, 167 deselected, 12 xfailed, 1 xpassed, 4 warnings in 234.12 seconds I think it would be fair to ask for at least a bulk build on one fast arch, plus builds of NumPy itself and at least some kind of runtime testing on a couple others, but we are still at a time in the release cycle where we can check build logs on other arches and get things fixed, and expect to get some feedback from users if there are runtime problems before things get baked into a release, so we don't need to go overboard on this. Index: Makefile === RCS file: /cvs/ports/math/py-numpy/Makefile,v retrieving revision 1.56 diff -u -p -r1.56 Makefile --- Makefile18 Nov 2019 04:10:31 - 1.56 +++ Makefile10 Jan 2020 15:46:06 - @@ -2,11 +2,10 @@ COMMENT=fast array and numeric programming library for Python -MODPY_EGG_VERSION= 1.14.6 +MODPY_EGG_VERSION= 1.16.5 DISTNAME= numpy-${MODPY_EGG_VERSION} PKGNAME=py-${DISTNAME} EXTRACT_SUFX= .zip -REVISION= 1 CATEGORIES= math devel @@ -17,7 +16,8 @@ MAINTAINER= Daniel Dickman RCS file: /cvs/ports/math/py-numpy/patches/patch-numpy_core_include_numpy_npy_common_h,v retrieving revision 1.6 diff -u -p -r1.6 patch-numpy_core_include_numpy_npy_common_h --- patches/patch-numpy_core_include_numpy_npy_common_h 30 Jun 2018 21:49:33 - 1.6 +++ patches/patch-numpy_core_include_numpy_npy_common_h 10 Jan 2020 15:46:06 - @@ -1,13 +1,8 @@ $OpenBSD: patch-numpy_core_include_numpy_npy_common_h,v 1.6 2018/06/30 21:49:33 daniel Exp $ -(1) optimize function attributes were added in gcc 4.4. Apply a fix for the warnings. But -the next release of numpy should fix this in a better way. See: -https://github.com/numpy/numpy/issues/5451 +XXX recheck powerpc, is this still needed? -Gets rid of these warnings from the build: -numpy/core/src/multiarray/lowlevel_strided_loops.c.src:120: warning: 'optimize' attribute directive ignored - -(2) py-numpy only checks for expl to determine whether extended-precision +py-numpy only checks for expl to determine whether extended-precision support is present. since we don't have it yet; it implements it's own. however, on alpha, powerpc, it declared functions with types that conflict with C99 (double for *l), therefore failed. @@ -15,36 +10,7 @@ types that conflict with C99 (double for Index: numpy/core/include/numpy/npy_common.h --- numpy/core/include/numpy/npy_common.h.orig +++ numpy/core/include/numpy/npy_common.h -@@ -20,6 +20,7 @@ - #endif - #endif - -+#if defined(__GNUC__) && (__GNUC__ > 4 || (__GNUC__ == 4 && (__GNUC_MINOR__ > 4))) - /* - * gcc does not unroll even with -O3 - * use with care, unrolling on modern cpus rarely speeds things up -@@ -30,13 +31,20 @@ - #else - #define NPY_GCC_UNROLL_LOOPS - #endif -+#else -+#define NPY_GCC_UNROLL_LOOPS -+#endif - -+#if defined(__GNUC__) && (__GNUC__ > 4 || (__GNUC__ == 4 && (__GNUC_MINOR__ > 4))) - /* highest gcc optimization level, enabled autovectorizer */ - #ifdef HAVE_ATTRIBUTE_OPTIMIZE_OPT_3 - #define NPY_GCC_OPT_3 __attribute__((optimize("O3"))) - #else - #define NPY_GCC_OPT_3 - #endif -+#else -+#define NPY_GCC_OPT_3 -+#endif - - /* compile target attributes */ - #if defined HAVE_ATTRIBUTE_TARGET_AVX && defined HAVE_LINK_AVX -@@ -336,14 +344,8 @@ typedef unsigned char npy_bool; +@@ -320,14 +320,8 @@ typedef unsigned char npy_bool; #define NPY_FALSE 0 #define NPY_TRUE 1 Index: patches/patch-numpy_distutils_command_build_src_py
Re: WIP: Update of math/py-numpy to 1.16.5
On 2019/11/02 02:55, Kurt Mosiejczuk wrote: > I got this as far as compiling with flang, running its tests mostly > sucessfully, and packaging. I'm about to start traveling tomorrow > so I won't get to testing all the consumers right away. I figured I'd > send this out in case someone wants to throw it through a bulk or pick > up where I left off. Here's a diff based on Kurt's but reinstating gfortran as compiler. I'm not sure what the compiler change was originally done for, perhaps it was just to avoid the problem with gcc flags that jca already fixed in python/2.7? I also removed part of patch-numpy_core_include_numpy_npy_common_h that was dealing with gcc-<4.4 which we don't have to worry about (the gfortran module uses gcc for C as well as Fortran, so it will always be built with 4.4+ for us). I left the second part in but we could do with testing powerpc with that file removed completely (I added an XXX). Builds ok (py2+3) on amd64, tests are in good shape: py2: 10 failed, 7173 passed, 159 skipped, 166 deselected, 12 xfailed, 1 xpassed, 3 warnings in 265.80 seconds py3: 10 failed, 7266 passed, 83 skipped, 167 deselected, 12 xfailed, 1 xpassed, 4 warnings in 234.12 seconds I think it would be fair to ask for at least a bulk build on one fast arch, plus builds of NumPy itself and at least some kind of runtime testing on a couple others, but we are still at a time in the release cycle where we can check build logs on other arches and get things fixed, and expect to get some feedback from users if there are runtime problems before things get baked into a release, so we don't need to go overboard on this. Index: Makefile === RCS file: /cvs/ports/math/py-numpy/Makefile,v retrieving revision 1.56 diff -u -p -r1.56 Makefile --- Makefile18 Nov 2019 04:10:31 - 1.56 +++ Makefile10 Jan 2020 15:46:06 - @@ -2,11 +2,10 @@ COMMENT= fast array and numeric programming library for Python -MODPY_EGG_VERSION= 1.14.6 +MODPY_EGG_VERSION= 1.16.5 DISTNAME= numpy-${MODPY_EGG_VERSION} PKGNAME= py-${DISTNAME} EXTRACT_SUFX= .zip -REVISION= 1 CATEGORIES=math devel @@ -17,7 +16,8 @@ MAINTAINER= Daniel Dickman https://github.com/numpy/numpy/issues/5451 +XXX recheck powerpc, is this still needed? -Gets rid of these warnings from the build: -numpy/core/src/multiarray/lowlevel_strided_loops.c.src:120: warning: 'optimize' attribute directive ignored - -(2) py-numpy only checks for expl to determine whether extended-precision +py-numpy only checks for expl to determine whether extended-precision support is present. since we don't have it yet; it implements it's own. however, on alpha, powerpc, it declared functions with types that conflict with C99 (double for *l), therefore failed. @@ -15,36 +10,7 @@ types that conflict with C99 (double for Index: numpy/core/include/numpy/npy_common.h --- numpy/core/include/numpy/npy_common.h.orig +++ numpy/core/include/numpy/npy_common.h -@@ -20,6 +20,7 @@ - #endif - #endif - -+#if defined(__GNUC__) && (__GNUC__ > 4 || (__GNUC__ == 4 && (__GNUC_MINOR__ > 4))) - /* - * gcc does not unroll even with -O3 - * use with care, unrolling on modern cpus rarely speeds things up -@@ -30,13 +31,20 @@ - #else - #define NPY_GCC_UNROLL_LOOPS - #endif -+#else -+#define NPY_GCC_UNROLL_LOOPS -+#endif - -+#if defined(__GNUC__) && (__GNUC__ > 4 || (__GNUC__ == 4 && (__GNUC_MINOR__ > 4))) - /* highest gcc optimization level, enabled autovectorizer */ - #ifdef HAVE_ATTRIBUTE_OPTIMIZE_OPT_3 - #define NPY_GCC_OPT_3 __attribute__((optimize("O3"))) - #else - #define NPY_GCC_OPT_3 - #endif -+#else -+#define NPY_GCC_OPT_3 -+#endif - - /* compile target attributes */ - #if defined HAVE_ATTRIBUTE_TARGET_AVX && defined HAVE_LINK_AVX -@@ -336,14 +344,8 @@ typedef unsigned char npy_bool; +@@ -320,14 +320,8 @@ typedef unsigned char npy_bool; #define NPY_FALSE 0 #define NPY_TRUE 1 Index: patches/patch-numpy_distutils_command_build_src_py === RCS file: /cvs/ports/math/py-numpy/patches/patch-numpy_distutils_command_build_src_py,v retrieving revision 1.3 diff -u -p -r1.3 patch-numpy_distutils_command_build_src_py --- patches/patch-numpy_distutils_command_build_src_py 30 Jun 2018 21:49:33 - 1.3 +++ patches/patch-numpy_distutils_command_build_src_py 10 Jan 2020 15:46:06 - @@ -6,7 +6,7 @@ WRKOBJDIR has a trailing slash) Index: numpy/distutils/command/build_src.py --- numpy/distutils/command/build_src.py.orig +++ numpy/distutils/command/build_src.py -@@ -372,6 +372,7 @@ class build_src(build_ext.build_ext): +@@ -370,6 +370,7 @@ class build_src(build_ext.build_ext): #incl_dirs.append(self.build_src) build_dir = os.path.join(*([self.build_src]\
Re: WIP: Update of math/py-numpy to 1.16.5
> On Nov 4, 2019, at 6:18 AM, Kurt Mosiejczuk wrote: > > On Sun, Nov 03, 2019 at 10:06:52PM +0100, Marc Espie wrote: >> On Mon, Nov 04, 2019 at 02:32:03AM +0900, Daniel Dickman wrote: clang doesn't error out, gcc does. Why does numpy use gcc? > >>> Because that's what upstream recommends in their docs. > >>> 'Note that NumPy is developed mainly using GNU compilers. Compilers >>> from other vendors such as Intel, Absoft, Sun, NAG, Compaq, Vast, >>> Portland, Lahey, HP, IBM, Microsoft are only supported in the form >>> of community feedback, and may not work out of the box. GCC 4.x (and >>> later) compilers are recommended.' > >>> See: https://numpy.org/devdocs/user/building.html > >> That's probably not a good enough reason, especially considering how >> many security improvements are in base clang as opposed to base gcc... > > That page may be out of date, too. It says Python 2.7 is supported, > whereas the 1.17.x release no longer support Python 2.7. Kurt, please feel to take it up with upstream not me. > > --Kurt >
Re: WIP: Update of math/py-numpy to 1.16.5
> On Nov 4, 2019, at 6:06 AM, Marc Espie wrote: > > On Mon, Nov 04, 2019 at 02:32:03AM +0900, Daniel Dickman wrote: >>> clang doesn't error out, gcc does. >>> Why does numpy use gcc? >> >> Because that’s what upstream recommends in their docs. >> >> “Note that NumPy is developed mainly using GNU compilers. Compilers from >> other vendors such as Intel, Absoft, Sun, NAG, Compaq, Vast, Portland, >> Lahey, HP, IBM, Microsoft are only supported in the form of community >> feedback, and may not work out of the box. GCC 4.x (and later) compilers are >> recommended.” >> >> See: https://numpy.org/devdocs/user/building.html > > That's probably not a good enough reason, especially considering how many > security improvements are in base clang as opposed to base gcc... Marc please take it up with upstream not me.
Re: WIP: Update of math/py-numpy to 1.16.5
On Sun, Nov 03 2019, Marc Espie wrote: > On Mon, Nov 04, 2019 at 02:32:03AM +0900, Daniel Dickman wrote: >> > clang doesn't error out, gcc does. >> > Why does numpy use gcc? >> >> Because that’s what upstream recommends in their docs. >> >> “Note that NumPy is developed mainly using GNU compilers. Compilers >> from other vendors such as Intel, Absoft, Sun, NAG, Compaq, Vast, >> Portland, Lahey, HP, IBM, Microsoft are only supported in the form of >> community feedback, and may not work out of the box. GCC 4.x (and >> later) compilers are recommended.” >> >> See: https://numpy.org/devdocs/user/building.html > > That's probably not a good enough reason, especially considering how many > security improvements are in base clang as opposed to base gcc... hmmpf. I initially thought that maybe the failure was due to /usr/bin/gcc being used instead of /usr/bin/cc. I was wrong. numpy uses gfortran from ports/lang/gcc, so it makes perfect sense for some gcc-ish option parser to bail out when passed a bogus -Oprout option. And indeed math/py-numpy fails the same way with the previous python2 package on an amd64 system without /usr/bin/gcc. I'm not willing to spend more time on this but I'm not keen on seeing others waste time on this either. Sorry for bringing this up without actually checking in the first place. -- jca | PGP : 0x1524E7EE / 5135 92C1 AD36 5293 2BDF DDCC 0DFA 74AE 1524 E7EE
Re: WIP: Update of math/py-numpy to 1.16.5
On Mon, Nov 04, 2019 at 02:32:03AM +0900, Daniel Dickman wrote: > > clang doesn't error out, gcc does. > > Why does numpy use gcc? > > Because that’s what upstream recommends in their docs. > > “Note that NumPy is developed mainly using GNU compilers. Compilers from > other vendors such as Intel, Absoft, Sun, NAG, Compaq, Vast, Portland, Lahey, > HP, IBM, Microsoft are only supported in the form of community feedback, and > may not work out of the box. GCC 4.x (and later) compilers are recommended.” > > See: https://numpy.org/devdocs/user/building.html That's probably not a good enough reason, especially considering how many security improvements are in base clang as opposed to base gcc...
Re: WIP: Update of math/py-numpy to 1.16.5
Now that the python issues seem to be solved, just one comment on the proposed diff: > -MODFORTRAN_COMPILER =gfortran > +MODFORTRAN_COMPILER =flang numpy developers basically say to use gnu compilers, otherwise you’re on your own. And numpy’s fortran compiler can’t be switched in isolation. We’d have to do a switch across all the forward and reverse dependencies in the tree at the same time.
Re: WIP: Update of math/py-numpy to 1.16.5
> clang doesn't error out, gcc does. > Why does numpy use gcc? Because that’s what upstream recommends in their docs. “Note that NumPy is developed mainly using GNU compilers. Compilers from other vendors such as Intel, Absoft, Sun, NAG, Compaq, Vast, Portland, Lahey, HP, IBM, Microsoft are only supported in the form of community feedback, and may not work out of the box. GCC 4.x (and later) compilers are recommended.” See: https://numpy.org/devdocs/user/building.html
Re: WIP: Update of math/py-numpy to 1.16.5
Ok with the REVISION bump. -- Sent from a phone, apologies for poor formatting. On 2 November 2019 22:49:21 Kurt Mosiejczuk wrote: On Sat, Nov 02, 2019 at 10:16:03PM +0100, Jeremie Courreges-Anglas wrote: To stay on the safe side, here's a simple diff for lang/python/2.7 which unbreaks numpy and maybe others. clang doesn't error out, gcc does. Why does numpy use gcc? Index: Makefile === RCS file: /cvs/ports/lang/python/2.7/Makefile,v retrieving revision 1.63 diff -u -p -r1.63 Makefile --- Makefile1 Nov 2019 17:40:08 - 1.63 +++ Makefile2 Nov 2019 21:13:16 - @@ -11,6 +11,7 @@ SHARED_LIBS = python2.7 0.0 VERSION_SPEC = >=2.7,<2.8 CONFIGURE_ARGS += --with-ensurepip=no +CONFIGURE_ENV += ac_cv_opt_olimit_ok=no PORTROACH = limit:^2\.7 This works for me. Shouldn't we bump REVISION though? --Kurt
Re: WIP: Update of math/py-numpy to 1.16.5
On 2019/11/02 10:40, Daniel Dickman wrote: > > > > On Nov 2, 2019, at 10:33 AM, Kurt Mosiejczuk wrote: > > > > On Sat, Nov 02, 2019 at 07:03:57AM -0400, Daniel Dickman wrote: > > > On Nov 2, 2019, at 2:56 AM, Kurt Mosiejczuk wrote: > >> > 'So the existing py-numpy is unhappy with the 2.7 update and it > >>> seems to be clashes between clang and gcc. > > > >> Should the 2.7 python update be backed out? Or is there some smaller > >> fix for the existing numpy version? > > > > Perhaps it should be backed out. That will also be rather complicated > > though. > > Rolling back to previous and cranking EPOCH doesn’t do the trick? With EPOCH bumped, the version string used in dependencies no longer matches. It's not as much work to fix as I first feared (thanks to PKGSPEC), but it still would mean that pkg_add will have to merge a huge update set, which is usually best avoided if possible.
Re: WIP: Update of math/py-numpy to 1.16.5
On Sat, Nov 02 2019, Kurt Mosiejczuk wrote: > So the existing py-numpy is unhappy with the 2.7 update and it seems to > be clashes between clang and gcc. > > This is the last version of py-numpy to work with Python 2.7. 1.17.x will > be Python 3.5+ only. To stay on the safe side, here's a simple diff for lang/python/2.7 which unbreaks numpy and maybe others. clang doesn't error out, gcc does. Why does numpy use gcc? Index: Makefile === RCS file: /cvs/ports/lang/python/2.7/Makefile,v retrieving revision 1.63 diff -u -p -r1.63 Makefile --- Makefile1 Nov 2019 17:40:08 - 1.63 +++ Makefile2 Nov 2019 21:13:16 - @@ -11,6 +11,7 @@ SHARED_LIBS = python2.7 0.0 VERSION_SPEC = >=2.7,<2.8 CONFIGURE_ARGS += --with-ensurepip=no +CONFIGURE_ENV += ac_cv_opt_olimit_ok=no PORTROACH = limit:^2\.7 -- jca | PGP : 0x1524E7EE / 5135 92C1 AD36 5293 2BDF DDCC 0DFA 74AE 1524 E7EE
Re: WIP: Update of math/py-numpy to 1.16.5
> On Nov 2, 2019, at 10:33 AM, Kurt Mosiejczuk wrote: > > On Sat, Nov 02, 2019 at 07:03:57AM -0400, Daniel Dickman wrote: > On Nov 2, 2019, at 2:56 AM, Kurt Mosiejczuk wrote: >> 'So the existing py-numpy is unhappy with the 2.7 update and it >>> seems to be clashes between clang and gcc. > >> Should the 2.7 python update be backed out? Or is there some smaller >> fix for the existing numpy version? > > Perhaps it should be backed out. That will also be rather complicated > though. Rolling back to previous and cranking EPOCH doesn’t do the trick?
Re: WIP: Update of math/py-numpy to 1.16.5
> On Nov 2, 2019, at 2:56 AM, Kurt Mosiejczuk wrote: > > So the existing py-numpy is unhappy with the 2.7 update and it seems to > be clashes between clang and gcc. Should the 2.7 python update be backed out? Or is there some smaller fix for the existing numpy version? Anyway can you share more details of the problem? I am traveling and probably can’t get to a box for at least another week. > > This is the last version of py-numpy to work with Python 2.7. 1.17.x will > be Python 3.5+ only. Even if 1.17 supported python2, it can’t go in right now due to avx issues that I thought I’d solved but apparently haven’t and am I’m still trying to debug. > > I got this as far as compiling with flang, running its tests mostly > sucessfully, and packaging. I'm about to start traveling tomorrow I am also traveling at the moment and not back for a week which is why I think we should back out the python update. > so I won't get to testing all the consumers right away. I figured I'd > send this out in case someone wants to throw it through a bulk or pick > up where I left off. The challenge with numpy is we really need to test that the direct consumers package *and* we want to do runtime tests as well (ie test the consumers of those consumers). That’s a lot of stuff to build which is why I don’t update numpy very often and try to do it carefully. Going to 1.16.5 as a fix for python, without testing numpy in a bulk, preferably on multiple archs is asking for trouble it may just work but it also might cause more problems. I wouldn’t go that route in a rush. > > (cc maintainer) > > --Kurt > > Index: Makefile > === > RCS file: /cvs/ports/math/py-numpy/Makefile,v > retrieving revision 1.55 > diff -u -p -r1.55 Makefile > --- Makefile7 Jul 2019 14:58:04 -1.55 > +++ Makefile2 Nov 2019 06:50:17 - > @@ -2,11 +2,10 @@ > > COMMENT=fast array and numeric programming library for Python > > -MODPY_EGG_VERSION=1.14.6 > +MODPY_EGG_VERSION=1.16.5 > DISTNAME=numpy-${MODPY_EGG_VERSION} > PKGNAME=py-${DISTNAME} > EXTRACT_SUFX=.zip > -REVISION=1 > > CATEGORIES=math devel > > @@ -17,7 +16,10 @@ MAINTAINER=Daniel Dickman # BSD > PERMIT_PACKAGE =Yes > > -WANTLIB=blas lapack m pthread ${MODFORTRAN_WANTLIB} ${MODPY_WANTLIB} > +COMPILER=base-clang ports-gcc base-gcc > + > +WANTLIB=blas lapack m pthread ${MODFORTRAN_WANTLIB} \ > +${MODPY_WANTLIB} > > MODULES=lang/python \ > fortran > @@ -27,12 +29,12 @@ FLAVOR?= > > MODPY_PI =Yes > MODPY_SETUPTOOLS=Yes > -MODPY_DISTUTILS_BUILDARGS = --fcompiler=gnu95 > > -TEST_DEPENDS=devel/py-nose${MODPY_FLAVOR} \ > -devel/py-tz${MODPY_FLAVOR} > +TEST_DEPENDS=${FULLPKGNAME}:${FULLPKGPATH} \ > +devel/py-test${MODPY_FLAVOR} \ > +devel/py-tz${MODPY_FLAVOR} > > -MODFORTRAN_COMPILER =gfortran > +MODFORTRAN_COMPILER =flang > > BUILD_DEPENDS = ${MODFORTRAN_BUILD_DEPENDS} > LIB_DEPENDS=math/lapack \ > @@ -55,8 +57,8 @@ CFLAGS +=-D CYTHON_SMALL_CODE= > pre-build: > cp -f ${WRKSRC}/numpy/distutils/site.cfg ${WRKSRC}/site.cfg > > -do-test: fake > -cd ${WRKDIR} && ${MAKE_ENV} PYTHONPATH=${WRKINST}${MODPY_SITEPKG} > ${MODPY_BIN} -c \ > +do-test: > +cd ${WRKDIR} && ${MAKE_ENV} ${MODPY_BIN} -c \ > 'import numpy ; numpy.test()' > > .include > Index: distinfo > === > RCS file: /cvs/ports/math/py-numpy/distinfo,v > retrieving revision 1.13 > diff -u -p -r1.13 distinfo > --- distinfo26 Oct 2018 21:09:31 -1.13 > +++ distinfo2 Nov 2019 06:50:17 - > @@ -1,2 +1,2 @@ > -SHA256 (numpy-1.14.6.zip) = ElDt9vbEPh14I/CWdBa8GCWLsnHcU2KY6w6gCp5FuAo= > -SIZE (numpy-1.14.6.zip) = 4906096 > +SHA256 (numpy-1.16.5.zip) = i7RS2U6WSzEiBbDeEjjdcgnaRSNDZTqyFLXWgXgOegw= > +SIZE (numpy-1.16.5.zip) = 5138208 > Index: pkg/PLIST > === > RCS file: /cvs/ports/math/py-numpy/pkg/PLIST,v > retrieving revision 1.12 > diff -u -p -r1.12 PLIST > --- pkg/PLIST30 Jun 2018 21:49:33 -1.12 > +++ pkg/PLIST2 Nov 2019 06:50:17 - > @@ -5,6 +5,8 @@ lib/python${MODPY_VERSION}/site-packages > lib/python${MODPY_VERSION}/site-packages/numpy-${MODPY_EGG_VERSION}-py${MODPY_VERSION}.egg-info/PKG-INFO > lib/python${MODPY_VERSION}/site-packages/numpy-${MODPY_EGG_VERSION}-py${MODPY_VERSION}.egg-info/SOURCES.txt > lib/python${MODPY_VERSION}/site-packages/numpy-${MODPY_EGG_VERSION}-py${MODPY_VERSION}.egg-info/dependency_links.txt > +lib/python${MODPY_VERSION}/site-packages/numpy-${MODPY_EGG_VERSION}-py${MODPY_VERSION}.egg-info/entry_points.txt > +lib/python${MODPY_VERSION}/site-packages/numpy-${MODPY_EGG_VERSION}-py${MODPY_VERSION}.egg-info/not-zip-safe >