On Tue, May 19, 2015 at 10:58 AM, Paul Moore <p.f.mo...@gmail.com> wrote:
> Sure. Doesn't have to be the same way, but the user experience has to > be the same. absolutely. > But maybe that's not going to cut it -- in a way, we are headed there now, > > with a contingent of people porting pypi packages to conda. So far it's > > various subsets of the scientific community, but if we could get a few > web > > developers to join in... > > Unless project owners switch to providing conda packages, isn't there > always going to be a lag? If a new version of lxml comes out, how long > must I wait for "the conda folks" to release a package for it? > who knows? -- but it is currently a light lift to update a conda package to a new version, once the original is built -- and we've got handy scripts and CI systems that will push an updated version of the binaries as soon as an updated version of the build script is pushed. It's a short step to automate looking for new versions on PyPi and automatically updadating the conda pacakges -- though there would need to be hand-intervention for whenever a update broke the build script... Of course, the ideal is for package maintainers to push conda pacakges themselves -- which is why the more-than-one-system to support is unfortunate. On the other hand, there is one plus side -- if the package maintainer doesn't push to PyPi, it's easier for a third party to take on that role -- see pychecker, or, for that matter numpy and scipy -- on pipy, but not binaries for Windows. But you can get them on binstar (or Anaconda, or...) > hmm -- that's the interesting technical question -- conda works at a > higher > > level than pip -- it CAN manage python itself -- I'm not sure it is HAS > to, > > but that's how it is usually used, and the idea is to provide a complete > > environment, which does include python itself. > > Yes. But I don't want to use Anaconda Python, Same reason - how long > do I wait for the new release of Python to be available in Anaconda? > There's currently no Python 3.5 alpha for example... you can grab and build the latest Python3.5 inside a conda environment just as well. Or are you using python.org builds for alpha versions, too? Oh, and as a conda environment sits at a higher level than python, it's actually easier to set up an environment specifically for a particular version of python. And anyone could put up a conda package of Python3.5 Alpha as well --- once the build script is written, it's pretty easy. But again -- teh more than one way to do it problem. If conda/binstar is good enough to replace pip/PyPI, there's no reason > for pip/PyPI to still exist. So in effect binstar *becomes* PyPI. > yup. > There's an element of evangelisation going on here - you're > (effectively) asking what it'd take to persuade me to use conda in > place of pip. I'm playing hard to get, a little, because I see no > specific benefits to me in using conda, so I don't see why I should > accept any loss at all, in the absence of a benefit to justify it. > I take no position here -- I'm playing around with ideas as to how we can move the community toward a better future -- I'm not trying to advocate any particular solution, but trying to figure out what solution we may want to pursue -- quite specifically which solution I'm going to put my personal energy toward. We may want to look back at a thread on this list where Travis Oliphant talks about why he built conda, etc. (I can't find it now -- maybe someone with better google-fu than me can. It think it was a thread on this list, probably about a year ago) or read his Blog Post: http://technicaldiscovery.blogspot.com/2013/12/why-i-promote-conda.html One of the key points is that when they started building conda -- pip+wheel where not mature, and the core folks behind them didn't want to support what was needed (dynamic libs, etc) -- and still don't. My biggest worry is that at some point, "if you want numpy/scipy, you > should use conda" becomes an explicit benefit of conda, That is EXACTLY what the explicit benefit of conda is. I think we'll get binary wheels for numpy and scipy up on PyPi before too long, but the rest of the more complex stuff is not going to be there. > and pip/PyPI > users get abandoned by the scientific community. They kind of already have -- it's been a long time, and a lot of work by only a couple folks to try to get binary wheels up on PyPi for Windows and OS-X > If that happens, I'd > rather see the community rally behind conda than see a split. But I > hope that's not the way things end up going. we'll see. But look at Travis' post -- pip+wheel simply does not support the full needs of the full scientific user. If we want a "one ring to rule them all", then it'll have to be conda -- or something a lot like it. On the other hand, I think pip+wheel+PyPi (or maybe just the community around it) can be extended a bit to at least support all the truly python focused stuff -- which I think would be pretty worthwhile it itself. This is the key point. The decision was made to "bless" pip as the > official Python package manager. Should we revisit that decision? I'm not sure I want to be the one to bring that up ;-) > If > not, then how do we ensure that pip (and the surrounding > infrastructure) handles the needs of the *whole* Python community? If > the authors of scientific extensions for Python abandon pip for conda, > then pip isn't supporting that part of the community properly. But > conversely, if the scientific community doesn't look to address their > issues within the pip/wheel infrastructure, how can we do anything to > avoid a rift? > well -- I think the problem is that while SOME of the needs of scientific community can be addressed within the pip-wheel infrastructure, they all can't all be addressed there. And (I wish I could find that thread), I'm pretty sure Travis said that before he started conda, he talked to the PyPa folks (before it was called that), and told that he'd be best off going off and building something new -- pip+wheel were just getting started, and were not going to support what he needed -- certainly not anytime soon. > I'd like to think so. The goal of pip is to be the baseline Python > package manager. We're expecting Linux distributions to build their > system packages via wheel, why can't Anaconda? Part of the problem > here, to my mind, is that it's *very* hard for the outsider to > separate out (Ana)conda-as-a-platform versus conda-as-a-tool, versus > conda-as-a-distribution-format. > absolutely. When you say "build their system packages via wheel" -- what does that mean? and why wheel, rather than, say, pip + setuptools? you can put whatever you want in a conda build script -- the current standard practice baseline for python packages is: $PYTHON setup.py install it's that simple. And I don't know if this is what it actually does, but essentially the conda package is all the stuff that that script added to the environment. So if changing that invocation to use pip would get us some better meta data or what have you, then by all means, we should change that standard of practice. (but it seems like an odd thing to have to use the package manager to build the package correctly -- shouldn't' that be distutils or setuptools' job? > * Toss out your current setup > > * Install Anaconda (or miniconda) > > * Switch from virtualenv to conda environments > > * re-install all your dependencies > > Yeah, that's hopeless. And worse still is the possibility that a pure > Python user might have to do that just to gain access to a particular > package. > exactly. I keep saying this, and I ought to ask - is there *any* likelihood > that a package would formally abandon any attempt to provide binary > distributions for Windows (or OSX, or whatever) except in conda > format? absolutely -- probably not the core major packages -- but I think that's already the case with a number of more domain specific packages (including my stuff -- OK, I have maybe three users outside my organization now, but still..) And numpy and scipy don't yet have binary wheels for Windows up -- though that is being worked on. So PyPI users will be told "install from source yourself or > switch to conda". If there's no intention for that to ever happen, Sadly, we are already there for minor packages, at least. Oh wait, not so minor -- the geospatial stack is not well supported on PyPi. I don't think there are pynetcdf or pyhdf binaries, etc... On the other hand, some domain specifc stuff is being support, like scikit-leaern, for instance: http://scikit-learn.org/stable/install.html#install-official-release lot of the "conda vs pip" discussion is less relevant. At the moment, > there are projects like numpy that don't distribute Windows wheels on > PyPI, but Christoph Gohlke has most of them available, yes, but in a form that is not redistributable on PyPi... > and in general > such projects seem to be aiming to move to wheels. So there aren't any > practical cases of "conda-only" packages. I'm not sure about "conda-only", but not pip-installable is all too common. > > And for even that to work, we need a way for everything installable by > pip > > to be installable within that conda environment -- which we could > probably > > achieve. > > See above - without some serious resource on the conda side, that > "everything" is unlikely. conda can provide a full, pretty standard, python environment -- why do you think "everything" is unlikely? > I do think that we could push > > pip+pypi+wheel a little further to better support at least the > > python-centric stuff -- i.e. third party libs, which would get us a lot > > farther. > > Agreed. Understanding the *actual* problem here is important though > (see my other post about clarifying why dynamic linking is so > important). > yes -- I don't know that that's answered yet -- but the third party dependency problem is real -- whether it is addressed by supporting dynamic linking, or by making it easier to find, build, distribute compatible static libs for package maintainers to use is still up in the air. > And again, it's not just the scipy stack -- there is stuff like image > > manipulation packages, etc, that could be better handled. > > Well, (for example) Pillow provides wheels with no issue, so I'm not > sure what you're thinking of here? the PIllow folks have figured it out -- and are doing the work -- but it took years, and we had a lot of pain building binaries for OS-X of PIL during that time. I think dynamic libs would be a good thing for packages like PIL, but maybe static is fine (I presume they are doing static now...) > And the geospatial packages are a mess, too - is that "scientific"? -- I > > don't know, but it's the "new hotness" in web development. > > I can't really comment on that one, as I've never used them. Does that > make me untrendy? :-) > Absolutely! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
