[Numpy-discussion] Issue with setup_requires and 1.16 release candidates
Hi all, Back in December I started getting failures in continuous integration as well as reports of failures from users of installation issues for a couple of packages. The problem can be easily reproduced in a Docker container with: FROM ubuntu:16.04 RUN apt-get update RUN apt-get install -y python3 python3-dev python3-pip python3-wheel RUN pip3 install fast-histogram RUN python3 -c 'import fast_histogram' Doing this results in the following traceback: ImportError: No module named 'numpy.core._multiarray_umath' Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python3.5/dist-packages/fast_histogram/__init__.py", line 1, in from .histogram import * File "/usr/local/lib/python3.5/dist-packages/fast_histogram/histogram.py", line 7, in from ._histogram_core import (_histogram1d, ImportError: numpy.core.multiarray failed to import I've seen similar issues with other packages too. The key is that the fast-histogram package defines: setup_requires=['numpy'] in the setup.py (since the package has a C extension that uses the Numpy C API) and numpy is needed before the install_requires dependencies are installed: https://github.com/astrofrog/fast-histogram/blob/master/setup.py Now this normally works fine, but the issues I saw started when the first 1.16 RC was made available, when installing into an environment in which numpy is not already installed. My understanding is that setup_requires is honored by easy_install (even if installing the main package with pip), which doesn't ignore pre-releases. Thus, the package is built against the 1.16 RC but then 1.15 is installed due to: install_requires=['numpy'] which is honored by pip. I think that the correct solution is to make sure that: [build-system] requires = ["setuptools", "wheel", "numpy"] is added to the pyproject.toml file (as per PEP 518). This then works properly with recent versions of pip (>10 I think). I think removing setup_requires then makes sense because it'd be better to have an error with older versions of pip that numpy is not installed rather than having the package built against the wrong version. I know this is a temporary issue in the sense that it will go away once the final version of 1.16 is out, but I just wanted to share this as a heads-up in case you get reports of issues from other people, and also to check whether there are any other solutions/workarounds to be aware of? (to avoid a similar situation in future). Thanks, Tom ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Issue with setup_requires and 1.16 release candidates
On Sun, Jan 6, 2019 at 12:45 PM Thomas Robitaille < thomas.robitai...@gmail.com> wrote: > Hi all, > > Back in December I started getting failures in continuous integration > as well as reports of failures from users of installation issues for a > couple of packages. The problem can be easily reproduced in a Docker > container with: > >FROM ubuntu:16.04 >RUN apt-get update >RUN apt-get install -y python3 python3-dev python3-pip python3-wheel >RUN pip3 install fast-histogram >RUN python3 -c 'import fast_histogram' > > Doing this results in the following traceback: > >ImportError: No module named 'numpy.core._multiarray_umath' >Traceback (most recent call last): > File "", line 1, in > File > "/usr/local/lib/python3.5/dist-packages/fast_histogram/__init__.py", > line 1, in >from .histogram import * > File > "/usr/local/lib/python3.5/dist-packages/fast_histogram/histogram.py", > line 7, in >from ._histogram_core import (_histogram1d, >ImportError: numpy.core.multiarray failed to import > > I've seen similar issues with other packages too. The key is that the > fast-histogram package defines: > >setup_requires=['numpy'] > > in the setup.py (since the package has a C extension that uses the > Numpy C API) and numpy is needed before the install_requires > dependencies are installed: > >https://github.com/astrofrog/fast-histogram/blob/master/setup.py > > Now this normally works fine, but the issues I saw started when the > first 1.16 RC was made available, when installing into an environment > in which numpy is not already installed. > > My understanding is that setup_requires is honored by easy_install > (even if installing the main package with pip), which doesn't ignore > pre-releases. Thus, the package is built against the 1.16 RC but then > 1.15 is installed due to: > >install_requires=['numpy'] > > which is honored by pip. > Oh fun. Thanks for explaining this! > I think that the correct solution is to make sure that: > >[build-system] >requires = ["setuptools", "wheel", "numpy"] > > is added to the pyproject.toml file (as per PEP 518). To clarify: you're talking about fast-histogram's pyproject.toml file right? This then works > properly with recent versions of pip (>10 I think). I think removing > setup_requires then makes sense because it'd be better to have an > error with older versions of pip that numpy is not installed rather > than having the package built against the wrong version. > This is a pretty hacky solution, needed because pip doesn't support setup_requires, but yes it should work and I can't think of a better way. Cheers, Ralf > I know this is a temporary issue in the sense that it will go away > once the final version of 1.16 is out, but I just wanted to share this > as a heads-up in case you get reports of issues from other people, and > also to check whether there are any other solutions/workarounds to be > aware of? (to avoid a similar situation in future). > > Thanks, > Tom > ___ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Issue with setup_requires and 1.16 release candidates
Hi Ralf, On Sun, 6 Jan 2019 at 22:06, Ralf Gommers wrote: > > > > On Sun, Jan 6, 2019 at 12:45 PM Thomas Robitaille > wrote: >> >> Hi all, >> >> Back in December I started getting failures in continuous integration >> as well as reports of failures from users of installation issues for a >> couple of packages. The problem can be easily reproduced in a Docker >> container with: >> >>FROM ubuntu:16.04 >>RUN apt-get update >>RUN apt-get install -y python3 python3-dev python3-pip python3-wheel >>RUN pip3 install fast-histogram >>RUN python3 -c 'import fast_histogram' >> >> Doing this results in the following traceback: >> >>ImportError: No module named 'numpy.core._multiarray_umath' >>Traceback (most recent call last): >> File "", line 1, in >> File >> "/usr/local/lib/python3.5/dist-packages/fast_histogram/__init__.py", >> line 1, in >>from .histogram import * >> File >> "/usr/local/lib/python3.5/dist-packages/fast_histogram/histogram.py", >> line 7, in >>from ._histogram_core import (_histogram1d, >>ImportError: numpy.core.multiarray failed to import >> >> I've seen similar issues with other packages too. The key is that the >> fast-histogram package defines: >> >>setup_requires=['numpy'] >> >> in the setup.py (since the package has a C extension that uses the >> Numpy C API) and numpy is needed before the install_requires >> dependencies are installed: >> >>https://github.com/astrofrog/fast-histogram/blob/master/setup.py >> >> Now this normally works fine, but the issues I saw started when the >> first 1.16 RC was made available, when installing into an environment >> in which numpy is not already installed. >> >> My understanding is that setup_requires is honored by easy_install >> (even if installing the main package with pip), which doesn't ignore >> pre-releases. Thus, the package is built against the 1.16 RC but then >> 1.15 is installed due to: >> >>install_requires=['numpy'] >> >> which is honored by pip. > > > Oh fun. Thanks for explaining this! > >> >> I think that the correct solution is to make sure that: >> >>[build-system] >>requires = ["setuptools", "wheel", "numpy"] >> >> is added to the pyproject.toml file (as per PEP 518). > > > To clarify: you're talking about fast-histogram's pyproject.toml file right? Yes that's right. Just for the record, here's the pull request to fast-histogram to implement the fix: https://github.com/astrofrog/fast-histogram/pull/33 Cheers, Tom > >> This then works >> properly with recent versions of pip (>10 I think). I think removing >> setup_requires then makes sense because it'd be better to have an >> error with older versions of pip that numpy is not installed rather >> than having the package built against the wrong version. > > > This is a pretty hacky solution, needed because pip doesn't support > setup_requires, but yes it should work and I can't think of a better way. > > Cheers, > Ralf > > >> >> I know this is a temporary issue in the sense that it will go away >> once the final version of 1.16 is out, but I just wanted to share this >> as a heads-up in case you get reports of issues from other people, and >> also to check whether there are any other solutions/workarounds to be >> aware of? (to avoid a similar situation in future). >> >> Thanks, >> Tom >> ___ >> NumPy-Discussion mailing list >> NumPy-Discussion@python.org >> https://mail.python.org/mailman/listinfo/numpy-discussion > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
