On 17 May 2015 06:19, "Chris Barker" <chris.bar...@noaa.gov> wrote: > indeed -- but it does have a bunch of python-specific features....it was built around the need to combine python with other systems. > >> That makes it an interesting alternative to pip on the package >> *consumption* side for data analysts, but it isn't currently a good >> fit for any of pip's other use cases (e.g. one of the scenarios I'm >> personally most interested in is that pip is now part of the >> Fedora/RHEL/CentOS build pipeline for Python based RPM packages - we >> universally recommend using "pip install" in the %install phase over >> using "setup.py install" directly) > > > hmm -- conda generally uses "setup.py install" in its build scripts. And it doesn't use pip install because it wants to handle the downloading and dependencies itself (in fact, turning OFF setuptools dependency handling is an annoyance..) > > So I'm not sure why pip is needed here -- would it be THAT much harder to build rpms of python packages if it didn't exist? (I do see why you wouldn't want to use conda to build rpms..)
We switched to recommending pip to ensure that the Fedora (et al) build toolchain can be updated to emit & handle newer Python metadata standards just by upgrading pip. For example, it means that system installed packages on modern Fedora installations should (at least in theory) provide full PEP 376 installation metadata with the installer reported as the system package manager. The conda folks (wastefully, in my view) are still attempting to compete directly with pip upstream, instead of delegating to it from their build scripts as an abstraction layer that helps hide the complexity of the Python packaging ecosystem. > But while _maybe_ if conda had been around 5 years earlier we could have not bothered with wheel, No, we couldn't, as conda doesn't work as well for system integrators. > I'm not proposing that we drop it -- just that we push pip and wheel a bit farther to broaden the supported user-base. I can't stop you working on something I consider a deep rabbithole, but why not just recommend the use of conda, and only pubish sdists on PyPI? conda needs more users and contributors seeking better integration with the PyPA tooling, and minimising the non-productive competition. The web development folks targeting Linux will generally be in a position to build from source (caching the resulting wheel file, or perhaps an entire container image). Also, assuming Fedora's experiment with language specific repos goes well ( https://fedoraproject.org/wiki/Env_and_Stacks/Projects/LanguageSpecificRepositories), we may see other distros replicating that model of handling the wheel creation task on behalf of their users. It's also worth noting that one of my key intended use cases for metadata extensions is to publish platform specific external dependencies in the upstream project metadata, which would get us one step closer to fully automated repackaging into policy compliant redistributor packages. >> Binary wheels already work for Python packages that have been >> developed with cross-platform maintainability and deployability taken >> into account as key design considerations (including pure Python >> wheels, where the binary format just serves as an installation >> accelerator). That category just happens to exclude almost all >> research and data analysis software, because it excludes the libraries >> at the bottom of that stack > > > It doesn't quite exclude those -- just makes it harder. And while depending on Fortran, etc, is pretty unique to the data analysis stack, stuff like libpng, libcurl, etc, etc, isn't -- non-system libs are not a rare thing. The rare thing is having two packages which are tightly coupled to the ABI of a given external dependency. That's a generally bad idea because it causes exactly these kinds of problems with independent distribution of prebuilt components. The existence of tight ABI coupling between components both gives the scientific Python stack a lot of its power, *and* makes it almost as hard to distribute in binary form as native GUI applications. >> It's also the case that when you *are* doing your own system >> integration, wheels are a powerful tool for caching builds, > conda does this nicely as well :-) I"m not tlrying to argue, at all, that binary wheels are useless, jsu that they could be a bit more useful. A PEP 426 metadata extension proposal for describing external binary dependencies would certainly be a welcome addition. That's going to be a common need for automated repackaging tools, even if we never find a practical way to take advantage of it upstream. >> > Ah -- here is a key point -- because of that, we DO support binary packages >> >> > on PyPi -- but only for Windows and OS-X.. I'm just suggesting we find a way >> > to extend that to pacakges that require a non-system non-python dependency. >> >> At the point you're managing arbitrary external binary dependencies, >> you've lost all the constraints that let us get away with doing this >> for extension modules without adequate metadata, and are back to >> trying to solve the same arbitrary ABI problem that exists on Linux. > > > I still don't get that -- any binary extension needs to match the ABI of the python is it used with -- a shared lib is the same problem. No, it's not, because we don't have python.org defining the binary ABI for anything except CPython itself. That's what constrains the target ABIs for extension modules on Windows & Mac OS X to a feasible number evolving at a feasible rate. A large part of what *defines* a platform is making decisions about the ABI to publish & target. Linux distros, nix, conda do that for everything they redistribute. I assume chocolatey does as well (I'm not sure if the Mac systems do prebuilt binaries) >> The line is drawn at ABI compatibility management. We're able to fuzz >> that line a little bit in the case of Windows and Mac OS X extension >> modules because we have the python.org CPython releases to act as an >> anchor for the ABI definition. >> >> We don't have that at all on other *nix platforms, and we don't have >> it on Windows and Mac OS X either once we move beyond the CPython C >> ABI (which encompasses the underlying platform ABI) > > > Showing my ignorance here -- what else is there we want to support (fortran ABI maybe?) Every single external binary dependency where the memory layout may be exposed to other extension modules when loaded into the same process becomes an ABI compatibility concern, with structs changing sizes so they don't fit in the allocated space any more being one of the most common issues. The stable ABI PEP has some additional background on that: https://www.python.org/dev/peps/pep-0384/ >> We *might* be able to get to the point of being able to describe >> platform ABIs well enough to allow public wheels for arbitrary >> platforms, > > That would be cool -- but not what I'm talking about here. I'm only talking about the ABIs we already describe. Except you're not, as we don't currently describe the ABIs for arbitrary external dependencies at all. > Anyway -- I've take a lot of my time (and a bunch of others on this list). And where ai think we are at is: > > * No one else seems to think it's worth trying to extend the PyPa ecosystem a bit more to better support dynamic libs. (except _maybe_ Enthought?) I know Donald is keen to see this, and a lot of ideas become more feasible if(/when?) PyPI gets an integrated wheel build farm. At that point, we can use the "centre of gravity" approach by letting the build farm implicitly determine the "standard" version of particular external dependencies, even if we can't communicate those versions effectively in the metadata. I'm also interested in metadata extensions to describe external binary dependencies, but I'm not sure that can be done sensibly in a platform independent way (in which case, it would be better to let various communities define their own extensions). > * I still think it can be done with minimal changes, and hacked in to do the proof of concept I'm still not clear on what "it" is. I've been pointing out how hard it is to do this right in the general case, but I get the impression you're actually more interested in the narrower case of defining a "SciPy ABI" that encompasses selected third party binary dependencies. That's a more attainable goal, but NumFocus or the SciPy community would likely be a better audience for that discussion than distutils-sig. > * But I'm not sure it's something that's going to get to the top of my ToDo list anyway -- I can get my needs met with conda anyway. My real production work is deep in the SciPy stack. > * So I may or may not move my ideas forward -- if I do, I'll be back with questions and maybe a more concrete proposal some day.... If I'm correct that your underlying notion is "It would be nice if there was an agreed SciPy ABI we could all build wheels against, and a way of publishing the external dependency manifest in the wheel metadata so it could be checked for consistency at installation time", that's a far more tractable problem than the "arbitrary binary dependencies" one. Attempting to solve the latter is what I believe leads to reinventing something functionally equivalent to conda, while I expect attempting to solve the former is likely to be more of a political battle than a technical one :) Cheers, Nick.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
