I have faced an issue with setuptools (and even plain distutils) in the past that has some resemblance to this.
There was a PyPI package that contained C extensions and headers, and another python package that had more C extensions, but depended on the location of the headers on the first package to be built. Was it numpy and scipy? Anyway zope.proxy and zope.container had the same issue, as did Zope's Acquisition and ExtensionClass. The issue can be clearly summarized with the `setup.py` snippet below, which I found at [1]: from distutils.core import setup, Extension import numpy # define the extension module cos_module_np = Extension('cos_module_np', sources=['cos_module_np.c'], include_dirs=[numpy.get_include()]) # run the setup setup(ext_modules=[cos_module_np]) [1] https://scipy-lectures.github.io/advanced/interfacing_with_c/interfacing_with_c.html Semantically, it's clear that numpy is both a `setup_requires` and an `install_requires` for the `setup.py` above. It's needed both at build time for the header locations and at run time for dynamically linking / module importing. However, even if the snippet above were using `setuptools` there'd be no way of passing `setup_requires` to the `setup()` call above, the script will have failed with an `ImportError` long before that, in the `include_dirs` part. The way the zope packages solved this issue was horribly crude and unsatisfying, but effective: they vendored the header files (and sometimes whole extensions) of the dependency packages. First by svn:extensions, and then by outright copying, when the `git` wave came around. The simplest and most backward compatible fix for setuptools I could think of would be for setup() keywords to have lazy evaluation: In setuptools, some `setup()` keywords accept parameters in multiple formats. For example `entry_points` accepts both a dictionary and a .ini formatted string [2]. [2] https://pythonhosted.org/setuptools/setuptools.html#dynamic-discovery-of-services-and-plugins If `setup()` keywords were to accept callables as well, then we could delay importing dependency packages until `setup_requires` where already satisfied, like this: from setuptools import setup, Extension def ext_modules(): import numpy # define the extension module cos_module_np = Extension('cos_module_np', sources=['cos_module_np.c'], include_dirs=[numpy.get_include()]) return [cos_module_np] # run the setup setup(setup_requires=['numpy'], ext_modules=ext_modules) Cheers, Leo On Friday, January 23, 2015, Ben Finney <ben+pyt...@benfinney.id.au> wrote: > Ben Finney <ben+pyt...@benfinney.id.au> writes: > > > Marius Gedminas <mar...@gedmin.as> writes: > > > > > 2. Implement metadata extraction using custom command classes[*] and > > > setup_requires. > > > > This seems to be the only one which might be feasible (and without, > > needless duplication of information, which is part of the whole point of > > this exercise). I will learn more and try it. > > Okay, the Setuptools ‘egg_info’ command is rather hostile to extension, > but less hostile than Setuptools entry points. I have cobbled together a > solution. > > The packaging code for version 2.0.4 of ‘python-daemon’: > > * Has no Setuptools entry points. These are not the right tool for the > job, it seems. > > * Adds a custom Setuptools command ‘write_version_info’ to parse the > Changelog and write the version info metadata file. > > * Minor hack: Uses a custom ‘egg_info’ command that will recognise and > run sub-commands. (Why doesn't every top-level command do this without > needing to be taught?) > > * Major hack: Re-binds class names in order to update their base class, > after dynamically importing ‘docutils’. (If there was a way to ensure > Docutils was installed before running any of the commands, this would > not be needed.) > > Thanks for all the assistance understanding the limitations and quirks > of Distutils and Setuptools. I'm not very happy with this solution, but > hopefully it is modular enough that it can be changed, in some happy > future when Python's packaging features are more tractable. > > -- > \ “Programs must be written for people to read, and only | > `\ incidentally for machines to execute.” —Abelson & Sussman, | > _o__) _Structure and Interpretation of Computer Programs_ | > Ben Finney > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig >
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig