Re: [Distutils] pip can't find header file for extension module, but `python setup.py install` works fine

2015-06-02 Thread Erik Bray
On Tue, Jun 2, 2015 at 3:41 AM, Robert Collins
 wrote:
> On 2 June 2015 at 02:51, Erik Bray  wrote:
>> On Sun, May 31, 2015 at 5:07 PM, AJ Friend  wrote:
>>> Hi,
>>>
>>> I'm trying to write a new `setup.py` file for an extension module to
>>> wrap a C library (https://github.com/cvxgrp/scs).
>>>
>>> The current `setup.py` file imports numpy. I'm trying to delay that
>>> import statement until setuptools has a chance to install numpy if
>>> it's not already installed. I'm trying to do that with this bit of
>>> code:
>>>
>>> from setuptools.command.build_ext import build_ext as _build_ext
>>> class build_ext(_build_ext):
>>> def finalize_options(self):
>>> _build_ext.finalize_options(self)
>>> # Prevent numpy from thinking it is still in its setup process:
>>> __builtins__.__NUMPY_SETUP__ = False
>>> import numpy
>>> self.include_dirs += ext['include_dirs'] + [numpy.get_include()]
>>>
>>> Running `python setup.py install` seems to work fine on my OSX
>>> machine, but when I run `pip install .` in the directory with
>>> `setup.py`, I get a clang error that it can't find one of the header
>>> files.
>>>
>>> Any idea why that would be happening? Could it have anything to do
>>> with the relative path I'm giving for the include directories?
>>>
>>> Also, I had trouble finding good documentation on subclassing
>>> build_ext. Does anyone know if setting self.include_dirs overwrites or
>>> appends to the include_dirs attribute of an Extension object defined
>>> later in setup.py?
>>>
>>> For the curious, my current attempt at setup.py is
>>> athttps://github.com/ajfriend/scs/blob/setup2/python/setup.py. The
>>> original can be found in the same directory.
>>>
>>> More generally, since I'm new to python packaging, I'm not sure how
>>> well or correctly I've written my `setup.py` file. Any feedback on
>>> doing things correctly would be appreciated.
>>
>> Hi AJ,
>>
>> For a lot of things in Python packaging there is not, sadly, One Right
>> Way to Do It.  Your setup.py looks okay though.
>>
>> You may want to have a look at the get_numpy_include_path utility here:
>> https://github.com/astropy/astropy-helpers/blob/7ee7e543641759ed1ee2b691bba1378cec76a001/astropy_helpers/utils.py#L66
>>
>> It's similar to what you're already doing, but maybe a little more
>> 'robust'.  In particular, I think the reload of the numpy module may
>> be important.
>
> It is, because what numpy is doing is importing from its own tree
> before setup has completed - which is in general unsafe.
>
> And numpy may be in process depending on exactly what setuptools
> setup_requires easy-install code has done, which can lead to the
> symptoms you see. This should be better once we get pip interpreting
> setup_requires for you, but a perhaps shorter term fix would be to
> factor out the distutils support glue from numpy (which seems pretty
> self contained) into a out of package file - either adjacent to
> setup.py, or, if numpy is willing to make the dependency on setuptools
> a hard dep (which I recommend), then as a separate package which can
> be setup_require'd.
>
> That would avoid the bootstrapping complexity, resulting in numpy not
> being imported until it has actually been installed, and things should
> be rosy for folk downstream of numpy from that point on.

That might not be a bad idea.  This is more or less what we did for
Astropy with astropy-helpers (this was done also so that other
software distributions in the Astropy affiliated package ecosystem
could take advantage of it).

I think maybe splitting out much of the stuff in numpy.distutils into
its own package (though unfortunately it would need a new name since I
don't think we could make numpy into a namespace package) might help.
Though I'm not sure how that would work for numpy.get_includes().
Would it just figure out where the include files *would* be installed
to if one were installing Numpy into the currently active environment?

Erik
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


Re: [Distutils] pip can't find header file for extension module, but `python setup.py install` works fine

2015-06-02 Thread Robert Collins
On 2 June 2015 at 02:51, Erik Bray  wrote:
> On Sun, May 31, 2015 at 5:07 PM, AJ Friend  wrote:
>> Hi,
>>
>> I'm trying to write a new `setup.py` file for an extension module to
>> wrap a C library (https://github.com/cvxgrp/scs).
>>
>> The current `setup.py` file imports numpy. I'm trying to delay that
>> import statement until setuptools has a chance to install numpy if
>> it's not already installed. I'm trying to do that with this bit of
>> code:
>>
>> from setuptools.command.build_ext import build_ext as _build_ext
>> class build_ext(_build_ext):
>> def finalize_options(self):
>> _build_ext.finalize_options(self)
>> # Prevent numpy from thinking it is still in its setup process:
>> __builtins__.__NUMPY_SETUP__ = False
>> import numpy
>> self.include_dirs += ext['include_dirs'] + [numpy.get_include()]
>>
>> Running `python setup.py install` seems to work fine on my OSX
>> machine, but when I run `pip install .` in the directory with
>> `setup.py`, I get a clang error that it can't find one of the header
>> files.
>>
>> Any idea why that would be happening? Could it have anything to do
>> with the relative path I'm giving for the include directories?
>>
>> Also, I had trouble finding good documentation on subclassing
>> build_ext. Does anyone know if setting self.include_dirs overwrites or
>> appends to the include_dirs attribute of an Extension object defined
>> later in setup.py?
>>
>> For the curious, my current attempt at setup.py is
>> athttps://github.com/ajfriend/scs/blob/setup2/python/setup.py. The
>> original can be found in the same directory.
>>
>> More generally, since I'm new to python packaging, I'm not sure how
>> well or correctly I've written my `setup.py` file. Any feedback on
>> doing things correctly would be appreciated.
>
> Hi AJ,
>
> For a lot of things in Python packaging there is not, sadly, One Right
> Way to Do It.  Your setup.py looks okay though.
>
> You may want to have a look at the get_numpy_include_path utility here:
> https://github.com/astropy/astropy-helpers/blob/7ee7e543641759ed1ee2b691bba1378cec76a001/astropy_helpers/utils.py#L66
>
> It's similar to what you're already doing, but maybe a little more
> 'robust'.  In particular, I think the reload of the numpy module may
> be important.

It is, because what numpy is doing is importing from its own tree
before setup has completed - which is in general unsafe.

And numpy may be in process depending on exactly what setuptools
setup_requires easy-install code has done, which can lead to the
symptoms you see. This should be better once we get pip interpreting
setup_requires for you, but a perhaps shorter term fix would be to
factor out the distutils support glue from numpy (which seems pretty
self contained) into a out of package file - either adjacent to
setup.py, or, if numpy is willing to make the dependency on setuptools
a hard dep (which I recommend), then as a separate package which can
be setup_require'd.

That would avoid the bootstrapping complexity, resulting in numpy not
being imported until it has actually been installed, and things should
be rosy for folk downstream of numpy from that point on.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig