On Mon, Apr 13, 2015 at 12:56 PM, Chris Barker <chris.bar...@noaa.gov> wrote: > NOTE: I don't work for any of the companies involved -- just a somewhat > frustrated user... And someone that has been trying for years to make things > easier for OS-X users. > >>> I’m not sure what (3) means exactly. What is a “normal” Python, do you >>> modify Python in a way that breaks the ABI but which isn’t reflected in the >>> standard ABI tag? >> >> >> It could be multiple things. The most obvious one is that generally. >> cross-platforms python distributions will try to be "relocatable" (i.e. the >> whole installation can be moved and still work). This means they require >> python itself to be built a special way. Strictly speaking, it is not an ABI >> issue, but the result is the same though: you can't use libraries from >> anaconda or canopy on top of a normal python > > > But why not? -- at least for Anaconda, it's because those libraries likely > have non-python dependencies, which are expected to be installed in a > particular way. And really, this is not particular to Anaconda/Canopy at > all. Python itself has no answer for this issue, and eggs and wheels don't > help. Well, maybe kinda sorta they do, but in a clunky/ugly way: in order to > build a binary wheel with non-python dependencies (let's say something like > libjpeg, for instance), you need to either: > - assume that libjpeg is installed in a "standard" place -- really no > solution at all (at least outside of linux) > - statically link it > - ship the dynamic lib with the package > > For the most part, the accepted solution for OS-X has been to statically > link, but: > > - it's a pain to do. The gnu toolchain really likes to use dynamic linking, > and building a static lib that will run on a > maybe-older-than-the-build-system machine is pretty tricky. > > - now we end up with multiple copies of the same lib in the python install. > There are a handful of libs that are used a LOT. Maybe there is no real > downside -- disk space and memory are cheap these days, but it sure feels > ugly. And I have yet to feel comfortable with having multiple versions of > the same lib linked into one python instance -- I can't say I've seen a > problem, but it makes me nervous. > > On Windows, the choices are the same, except that: It is so much harder to > build many of the "standard" open source libs that package authors are more > likely to do it for folks, and you do get the occasional "dll hell" issues. > > I had a plan to make some binary wheels for OS-X that were not really python > packages, but actually just bundled up libs, so that other wheels could > depend on them. OS-X does allow linking to relative paths, so this should > have been doable, but I never got anyone else to agree this was a good idea, > and I never found the roundtoits anyway. And it doesn't really fit into the > PyPi, pip, wheel, etc. philosphy to have dependencies that are platform > dependent and even worse, build-dependent. > > Meanwhile, conda was chugging along and getting a lot of momentum in the > Scientific community. And the core thing here is that conda was designed > from the ground up to support essentially anything, This means is supports > python packages that depend on non-python packages, but also supports > packages that have nothing to do with python (Perl, command line tools, what > have you...) > > So I have been focusing on conda lately. > > Which brings me back to the question: should the python tools (i.e. wheel) > be extended to support more use-cases, specifically non-python dependencies? > Or do we just figure that that's a problem better solved by projects with a > larger scope (i.e. rpm, deb, conda, canopy). > > I'm on the fence here. I mostly care about Python, and I think we're pretty > darn close with allowing wheel to support the non-python dependencies, which > would allow us all to "simply pip install" pretty much anything -- that > would be cool. But maybe it's a bit of a slippery slope, and if we go there, > we'll end up re-writing conda. > > BTW, while you can't generally install a conda package in/for another > python, you can generally install a wheel in a conda python....There are a > few issues with pip/setuptools trying to resolve dependencies while not > knowing about conda packages, but it does mostly work. > > Not sure that helped the discussion -- but I've been wrestling with this for > a while, so thought I'd get my thoughts out there.
I've always thought of wheel as solving only the Python-specific problem. Providing relocatable Python-specific packaging without trying to solve the intractable problem of non-Python dependencies. The strategy works best the more you are targeting "Python" as your platform and not a specific OS or distribution - sometimes it works well, other times not at all. Obviously if you need a specific build of PostgreSQL wheel isn't going to help you. With enough hacks you could make it work but are we ready to "pip install kde"? I don't think so. Personally I'm happy to let other tools solve the problem of C-level virtualenv. It's been suggested you could have a whole section in your Python package that said "by the way, RedHat package x, or Debian package y, or Gentoo package z", or use a separate package equivalency mapping as a level of indirection. I don't think this would be very good either. Instead, if you are doing system-level stuff, you should just use a system-level or user-level packaging tool that can easily re-package Python packages such as conda, rpm, deb, etc. _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig