On Sat, May 7, 2016 at 6:51 AM, Nick Coghlan <ncogh...@gmail.com> wrote:
> On 7 May 2016 01:55, "Chris Barker" <chris.bar...@noaa.gov> wrote: > > So my point is about scope-creep -- if you want the PyPa tools to solve > all these problems, then you are re-inventing conda -- better to simply > adopt conda (or fork it and fix issues that I'm sure are there....) > > conda doesn't solve these problems either - it solves the *end user* > problem for data analysts (install the Python library they want to use), > I really need to make this clear -- there is NOTHING "data analyst" specific about these problems -- they do come up more in the computational programming domain, but there are any number of other application that have the same problems (pyQT has come up in this conversation, yes?) -- and we're finding conda to be a good solution for our web development needs, too -- a conda environment is kinda like a lighter-weight, platform independent docker container. And of course, there is more an more data analysis going on behind web services these days too -- any python developer is going to run into these issues eventually... > by ignoring the system integrator one (interoperate with the system > integrator's existing platform level package management systems, of which > we all tend to have our own with no plans for consolidation) > > That's by design, though - conda was specifically created as a language > independent cross-platform platform, not as a cross-platform plugin > management system for Python runtimes. > exactly! > For a long time I was convinced the existence of conda and Linux > containers as end user tools meant we wouldn't ever need to solve these > problems at the language runtime layer, but it's since become clear to me > that there's significant value in solving these problems in a way that > doesn't care about how you obtained your Python runtime. > yup -- that would be the scope creep I'm referring too :-) But now I'm confused about what problems we're talking about. From above: """ interoperate with the system integrator's existing platform level package management systems """ you mean rpm, .deb, homebrew, etc? distutils did have (does it still) bdist_rpm (and, for that matter bdist_wininst) -- which I see as an attempt to inter-operate with the system integrator's platform. But I think it turns out that this is a hopeless task -- there are just too many different systems to consider and maintain -- much better to let the external systems handle that. And it got really ugly when you wanted to mingle virtualenv with rpm, etc.... The "solution" to that is to actually not do it -- use Docker or Conda if you want an isolated environment. What all this means to me is that we really do need to keep the building separate from the packaging -- so that the building tools can be used / supported by the systems integrator. But people shouldn't be using rpm to manage their system, and expect to install binary wheels... The trick is that the pip/wheel system is really handy -- certainly for Windows and OS-X that don't provide a standard system package manager. It is nice to be able to "just pip install" various packages. But the problem is that it is a half way solution -- it only works well for python-centered packages -- pure python is trivial of course, but custom written C extensions work great too -- but as soon as you get external dependencies, it all goes to heck. We can improve that -- defining the manylinux platform is a great step, but to really get there, there needs to be a way to handle third party libs (which we could probably do with the existing tools by establishing conventions), but then you get to non-python support tools: Fortran compilers, other language run times, all sorts of stuff. Oh, and then we want isolated environments, and they need to isolate these other non-python tools, so .... I think it's WAY beyond the current PEP under discussion, but we really should map out a "where we'd like to get" plan. "significant value in solving these problems in a way that doesn't care about how you obtained your Python runtime." I don't get this -- I don't think it's possible to not care about the python run-time -- we've only gotten this far because most folks have more or less standardized on the python.org builds as the standard python build. Even when you get your python elsewhere (Anaconda, for instance) they take pains to keep it compatible. I think we should count on python itself being managed by the same package manager as the packages -- or at least conforming to a standard (which I think we have with teh ABI tags, yes?) BTW, I only recently learned that there are more "other" python builds out there than I thought. Apparently the Visual Effect folks are on the habit of building their own python2.7 on Windows, so they can use a newer MS compiler, and then build their extensions to be compatible (all so you can use modern C++ in the extension code). I don't know that they are careful about setting the ABI tags, but they could be. > For a long time I was convinced the existence of conda and Linux containers as end user > tools meant we wouldn't ever need to solve these problems at the language runtime layer to an extent that is true -- but it does mean a proliferation of options, and as I think you pointed out earlier, that sucks because you need someone to support all those options -- for instance, most conda packages are built by third parties now - not the original package author. This is (very slowly) starting to change (Ned Batchelder recently asked about building conda packages for coverage...). and of course, the same is true with homebrew, and rpm, and deb, and.... So yeah, it would be nice to have ONE thing that python package developers need to do to make their work available to everyone. So I'll leave off with this: If there is even going to be one way for python package developers to provide binary packages for their users -- that system will need to support all (most) of what conda supports. Every feature in conda was added because it was needed. And THAT's what I've been meaning every time i say "reimpliment conda" -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig