Re: [Distutils] pip merges wheel
we might do different defaults for each platform On Mar 19, 2013 8:33 PM, "Glyph" wrote: > On Mar 19, 2013, at 5:00 PM, Daniel Holth wrote: > > It's possible to upload broken wheels. I don't want "I had to find the > disable flag" to be anyone's first impression. > > It's possible to upload broken sdists, too. > > Trust me, Windows' users (who do not have C compilers) impression of pip > _could not_ get any worse. > > -glyph > > ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip merges wheel
On Mar 19, 2013, at 5:00 PM, Daniel Holth wrote: > It's possible to upload broken wheels. I don't want "I had to find the > disable flag" to be anyone's first impression. > It's possible to upload broken sdists, too. Trust me, Windows' users (who do not have C compilers) impression of pip _could not_ get any worse. -glyph ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip merges wheel
It's possible to upload broken wheels. I don't want "I had to find the disable flag" to be anyone's first impression. On Mar 19, 2013 7:40 PM, "Glyph" wrote: > > On Mar 18, 2013, at 4:52 PM, Nick Coghlan wrote: > > > On Mon, Mar 18, 2013 at 4:39 PM, Barry Warsaw wrote: > >> On Mar 18, 2013, at 04:13 PM, Nick Coghlan wrote: > >> > >>> Eventually I expect pip will grow a "--wheel-only" option to run it in > >>> strict "installer only" mode, but the ecosystem is a long way from > >>> supporting that being a useful option (especially since there are some > >>> cases which will still require falling back to the "build from source" > >>> model). > >> > >> If that's the end goal, then it should be the default now. > > > > No, user experience is king. Right now, defaulting to wheel-only would > > be an awful user experience (because you wouldn't be able to install > > anything), as well as being completely backwards incompatible with the > > current behaviour (because everything would break). > > But it could default to wheels-and-other-things right now without breaking > anything, right? What's the rationale for not just preferring wheels if > they're available? > > -glyph > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip merges wheel
On Mar 18, 2013, at 4:52 PM, Nick Coghlan wrote: > On Mon, Mar 18, 2013 at 4:39 PM, Barry Warsaw wrote: >> On Mar 18, 2013, at 04:13 PM, Nick Coghlan wrote: >> >>> Eventually I expect pip will grow a "--wheel-only" option to run it in >>> strict "installer only" mode, but the ecosystem is a long way from >>> supporting that being a useful option (especially since there are some >>> cases which will still require falling back to the "build from source" >>> model). >> >> If that's the end goal, then it should be the default now. > > No, user experience is king. Right now, defaulting to wheel-only would > be an awful user experience (because you wouldn't be able to install > anything), as well as being completely backwards incompatible with the > current behaviour (because everything would break). But it could default to wheels-and-other-things right now without breaking anything, right? What's the rationale for not just preferring wheels if they're available? -glyph ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip merges wheel
On Mon, Mar 18, 2013 at 5:28 PM, Daniel Holth wrote: > I do understand the confusion. Binary package formats have more than > one use. Coincidentally we have implemented the slightly different > "cache compiles" and "distribute software" features using the same > format. It might help if you can imagine that "pip wheel" produces a > different format than "python setup.py bdist_wheel upload". I think it helps more to just imagine that it produces the same format, but for different reasons. That particular imagining has the virtue of being accurate :) We do need to do some work on better explaining that wheel has two major use cases, and that these two use cases correspond to "pip wheel" (for caching your own local builds of an sdist) and "./setup.py bdist_wheel upload" for publication of pre-built binaries via PyPI. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip merges wheel
I do understand the confusion. Binary package formats have more than one use. Coincidentally we have implemented the slightly different "cache compiles" and "distribute software" features using the same format. It might help if you can imagine that "pip wheel" produces a different format than "python setup.py bdist_wheel upload". Daniel ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip merges wheel
On Mon, Mar 18, 2013 at 4:48 PM, Lennart Regebro wrote: > On Mon, Mar 18, 2013 at 3:13 PM, Marcus Smith wrote: >> 1) pip is *currently* very much a build tool in that it build/installs from >> source archives, but I understand the new model is for pip to eventually be >> working with pre-built wheels much of the time, with no build system >> required. > > I used the word "build tool" in the packaging summit, that was the wrong word. > My point is that is currently a tool to *install* distributions as > opposed to *create* distributions. I think it's best if it is kept > that way. I'd rather see the wheel command go into another tool used > to build distributions (and perhaps upload them to the cheeseshop). No, that's not the intended use case of "pip wheel". It's intended for use as a multi-stage installation tool. Stage 1: install from sdist on a staging server which has development dependencies installed Stage 2: upload to a private PyPI index you control (perhaps just a directory published internally over HTTP) Stage 3: install from wheel on production servers without development dependencies For actual development use, it makes a lot more sense to install the wheel project and just use ./setup.py bdist_wheel. Cheers, Nick. > > [ I overheard the proposal to call such a tool "pup". ;-) ] > > //Lennart > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip merges wheel
On Mon, Mar 18, 2013 at 4:39 PM, Barry Warsaw wrote: > On Mar 18, 2013, at 04:13 PM, Nick Coghlan wrote: > >>Eventually I expect pip will grow a "--wheel-only" option to run it in >>strict "installer only" mode, but the ecosystem is a long way from >>supporting that being a useful option (especially since there are some >>cases which will still require falling back to the "build from source" >>model). > > If that's the end goal, then it should be the default now. No, user experience is king. Right now, defaulting to wheel-only would be an awful user experience (because you wouldn't be able to install anything), as well as being completely backwards incompatible with the current behaviour (because everything would break). Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip merges wheel
On Mon, Mar 18, 2013 at 3:13 PM, Marcus Smith wrote: > 1) pip is *currently* very much a build tool in that it build/installs from > source archives, but I understand the new model is for pip to eventually be > working with pre-built wheels much of the time, with no build system > required. I used the word "build tool" in the packaging summit, that was the wrong word. My point is that is currently a tool to *install* distributions as opposed to *create* distributions. I think it's best if it is kept that way. I'd rather see the wheel command go into another tool used to build distributions (and perhaps upload them to the cheeseshop). [ I overheard the proposal to call such a tool "pup". ;-) ] //Lennart ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip merges wheel
On Mar 18, 2013, at 7:39 PM, Barry Warsaw wrote: > On Mar 18, 2013, at 04:13 PM, Nick Coghlan wrote: > >> Eventually I expect pip will grow a "--wheel-only" option to run it in >> strict "installer only" mode, but the ecosystem is a long way from >> supporting that being a useful option (especially since there are some >> cases which will still require falling back to the "build from source" >> model). > > If that's the end goal, then it should be the default now. > > -Barry > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig --wheel-only as the default now would make approximately 3 things installable from PyPI, one of which is wheel itself. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip merges wheel
On Mar 18, 2013, at 04:13 PM, Nick Coghlan wrote: >Eventually I expect pip will grow a "--wheel-only" option to run it in >strict "installer only" mode, but the ecosystem is a long way from >supporting that being a useful option (especially since there are some >cases which will still require falling back to the "build from source" >model). If that's the end goal, then it should be the default now. -Barry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip merges wheel
On Mar 18, 2013, at 02:16 PM, Lennart Regebro wrote: >I still think it is unfortunate that we are starting to extend pip to >be a tool for developers to create distributions. It would be better >of pip was kept as an install tool, and we added the utilities for >creating distributions separate. +1. Doesn't this violate Nick's mission for killing `setup.py install` which IIUC is motivated by wanting to separate building and installing? I really want pip to just be about installing and use other tools to build source and binary distributions. -Barry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip merges wheel
On Mon, Mar 18, 2013 at 3:51 PM, Eric V. Smith wrote: > On 3/18/2013 5:16 PM, Lennart Regebro wrote: >> On Sat, Mar 16, 2013 at 5:06 PM, Daniel Holth wrote: >>> Earlier today we merged the existing wheel branch into mainline pip. >>> This adds opt-in wheel install support (built into pip, "pip install >>> --use-wheel ...") and the convenient "pip wheel ..." command for >>> creating the wheels you need. >> >> I still think it is unfortunate that we are starting to extend pip to >> be a tool for developers to create distributions. It would be better >> of pip was kept as an install tool, and we added the utilities for >> creating distributions separate. > > I completely agree. And those tools to build distributions could be > downloaded separately. As Marcus already noted, we cannot achieve this nirvana until the primary distribution format for Python software is something other than source archives. pip's support of the wheel format is a necessary step on that path - it's already a build system *because* our current "installation" command is "./setup.py install", and handling that command invokes the build system. Eventually I expect pip will grow a "--wheel-only" option to run it in strict "installer only" mode, but the ecosystem is a long way from supporting that being a useful option (especially since there are some cases which will still require falling back to the "build from source" model). Cheers, Nick. > > -- > Eric. > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip merges wheel
On 3/18/2013 5:16 PM, Lennart Regebro wrote: > On Sat, Mar 16, 2013 at 5:06 PM, Daniel Holth wrote: >> Earlier today we merged the existing wheel branch into mainline pip. >> This adds opt-in wheel install support (built into pip, "pip install >> --use-wheel ...") and the convenient "pip wheel ..." command for >> creating the wheels you need. > > I still think it is unfortunate that we are starting to extend pip to > be a tool for developers to create distributions. It would be better > of pip was kept as an install tool, and we added the utilities for > creating distributions separate. I completely agree. And those tools to build distributions could be downloaded separately. -- Eric. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip merges wheel
On Mon, Mar 18, 2013 at 3:13 PM, Marcus Smith wrote: >> The other option is of course that we start adding all sorts of >> development commands to pip, such as build, test, sdist etc. But I do >> think it's the wrong place. > > > I've thought of that too, but that's a discussion for another day or > parallel universe. The meta-build hooks are definitely a topic for post-metadata-2.0 :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip merges wheel
> I still think it is unfortunate that we are starting to extend pip to > be a tool for developers to create distributions. It would be better > of pip was kept as an install tool, and we added the utilities for > creating distributions separate. > I understand where you're coming from, but a few thoughts: 1) pip is *currently* very much a build tool in that it build/installs from source archives, but I understand the new model is for pip to eventually be working with pre-built wheels much of the time, with no build system required. 2) the motivation for "pip wheel" is *not* really for building single wheels for the project you're developing. For that use case, I agree it makes more sense conceptually to install the "wheel" package and use it's "bdist_wheel" setuptools extension (i.e. "python setup.py bdist_wheel" ) 3) the real motivation for "pip wheel" (which is a builder convenience tool) is to help people *install* from wheels *now* given that pypi won't be full of wheels for a time to come. This allows people to quickly and easily get all their source archive dependencies converted with a single command (if you're using requirements files) and start gaining the benefits of wheel with very little fiddling. 4) even when pypi is full of wheels, I can imagine people wanting to build certain dependencies with different build options, and "pip wheel" could help with that. 5) I can imagine "pip wheel" disappearing at some point down the road. We'll have to see, but until then, "pip wheel" is going to very critical IMO in getting people familiar with and actually using wheel-based installs. > The other option is of course that we start adding all sorts of > development commands to pip, such as build, test, sdist etc. But I do > think it's the wrong place. > I've thought of that too, but that's a discussion for another day or parallel universe. Marcus ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip merges wheel
On Sat, Mar 16, 2013 at 5:06 PM, Daniel Holth wrote: > Earlier today we merged the existing wheel branch into mainline pip. > This adds opt-in wheel install support (built into pip, "pip install > --use-wheel ...") and the convenient "pip wheel ..." command for > creating the wheels you need. I still think it is unfortunate that we are starting to extend pip to be a tool for developers to create distributions. It would be better of pip was kept as an install tool, and we added the utilities for creating distributions separate. The other option is of course that we start adding all sorts of development commands to pip, such as build, test, sdist etc. But I do think it's the wrong place. //Lennart ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip merges wheel
On Sun, Mar 17, 2013 at 8:38 PM, Marcus Smith wrote:
> the pip docs have a cookbook entry now for the wheel support
> http://www.pip-installer.org/en/latest/cookbook.html#building-and-installing-wheels
>
> the usage reference is up to date as well
> http://www.pip-installer.org/en/latest/usage.html
Great news!
Given Glyph's questions, It may be a good idea to add a "publishing
wheels" entry to the cookbook, explaining:
1. After building the wheel, publishing it to PyPI means users on a
matching system can download and install it without building it
2. The current caveats on handling C extensions that are *wrappers*
around shared libraries/DLLs that are expected to be installed on the
target system, rather than CPython-only C extensions. (Specifically,
bundled DLLs on Windows should work just fine, but depending on system
binaries on *nix systems can cause problems if other users have the
right SO version installed, and bundling on *nix systems may require
tweaking of the environment to find the bundled SO instead of only
looking in the system paths)
3. Users currently need to add a "--use-wheel" flag to enable the
wheel support. It is expected that using wheels will become the
default behaviour in the feature, but we want it to be opt in until it
has seen some more widespread real word usage.
Cheers,
Nick.
>
> Marcus
>
>
> On Sat, Mar 16, 2013 at 5:06 PM, Daniel Holth wrote:
>>
>> Earlier today we merged the existing wheel branch into mainline pip.
>> This adds opt-in wheel install support (built into pip, "pip install
>> --use-wheel ...") and the convenient "pip wheel ..." command for
>> creating the wheels you need.
>>
>> "pip wheel ..." uses the wheel reference implementation ("pip install
>> wheel") to compile a dependency tree as .whl archives. Used together
>> with "pip install --use-wheel ..." it provides a powerful way to speed
>> up repeated installs and reap other good packaging benefits. I've been
>> using this code in production for months and it works well.
>>
>> I am now a pip maintainer. We are committed to offering excellent
>> wheel support in pip including a good way to produce and consume the
>> format. In the future we will likely refactor the code to offer the
>> same features with more distlib and less setuptools but this change
>> will be mostly transparent to the end user. Once everyone is
>> comfortable with the format we will move towards installing wheels by
>> default when they are available instead of requiring the --use-wheel
>> flag.
>>
>> Enjoy! Please share your experiences with the new feature. The most
>> common issue is that you must install distribute >= 0.6.34 for
>> everything to work well.
>>
>> Daniel Holth
>> ___
>> Distutils-SIG maillist - Distutils-SIG@python.org
>> http://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
> ___
> Distutils-SIG maillist - Distutils-SIG@python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip merges wheel
On 18 March 2013 09:08, Glyph wrote: > My understanding is that in order to achieve this nirvana, what we must do > is: (Daniel may wish to chime in with more details) > A twisted developer, on each supported Windows configuration, must 'pip > install wheel; pip wheel ./Twisted' and place that build artifact on PyPI. To create the wheel, just run pip wheel , as you say on each supported Windows configuration. And it doesn't have to just be Windows - if you care to, builds for other platforms can also be created the same way. This may save on the "you need the following dev packages installed" type of FAQs. > Make our dependencies do the same thing. If the dependencies don't, install from source will still work, so it's probably not crucial for pure Python dependencies. > Tell our users to do 'pip install --use-wheel twisted' > > I have two questions: first, is this sequence of steps accurate, See above, but basically yes. > and if it > is, why is '--use-wheel' not just the default? Does this option just mean > '--please-work --no-really' or is there some functional change in behavior > in using wheels that might cause problems? It simply reflects that the wheel format and pip's support of it is relatively new, and so somewhat experimental. There's no functional change other than the fact that you get a wheel install if one's available - or at least there shouldn't be :-) In due course, I expect --use-wheel to become the default. We may offer a --no-wheel option at that stage, if it seems useful. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip merges wheel
On Mar 16, 2013, at 5:06 PM, Daniel Holth wrote: > Earlier today we merged the existing wheel branch into mainline pip. > This adds opt-in wheel install support (built into pip, "pip install > --use-wheel ...") and the convenient "pip wheel ..." command for > creating the wheels you need. Hi! I don't really understand all the issues that lead to the creation of the Wheel format, and I don't care to. But, I _very_ much would like some of the advertised benefits, such as being able to install binary packages on Windows. Well, _I_ don't care about installing binary packages on Windows. But I want users of Twisted on Windows to be able to 'pip install twisted' and just get a working install without having to learn about MSVCRT versions and the various miseries thereof. I am quite excited about the possibility that such a situation might be near our grasp. My understanding is that in order to achieve this nirvana, what we must do is: A twisted developer, on each supported Windows configuration, must 'pip install wheel; pip wheel ./Twisted' and place that build artifact on PyPI. Make our dependencies do the same thing. Tell our users to do 'pip install --use-wheel twisted' I have two questions: first, is this sequence of steps accurate, and if it is, why is '--use-wheel' not just the default? Does this option just mean '--please-work --no-really' or is there some functional change in behavior in using wheels that might cause problems? Thanks a lot! -glyph___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip merges wheel
the pip docs have a cookbook entry now for the wheel support
http://www.pip-installer.org/en/latest/cookbook.html#building-and-installing-wheels
the usage reference is up to date as well
http://www.pip-installer.org/en/latest/usage.html
Marcus
On Sat, Mar 16, 2013 at 5:06 PM, Daniel Holth wrote:
> Earlier today we merged the existing wheel branch into mainline pip.
> This adds opt-in wheel install support (built into pip, "pip install
> --use-wheel ...") and the convenient "pip wheel ..." command for
> creating the wheels you need.
>
> "pip wheel ..." uses the wheel reference implementation ("pip install
> wheel") to compile a dependency tree as .whl archives. Used together
> with "pip install --use-wheel ..." it provides a powerful way to speed
> up repeated installs and reap other good packaging benefits. I've been
> using this code in production for months and it works well.
>
> I am now a pip maintainer. We are committed to offering excellent
> wheel support in pip including a good way to produce and consume the
> format. In the future we will likely refactor the code to offer the
> same features with more distlib and less setuptools but this change
> will be mostly transparent to the end user. Once everyone is
> comfortable with the format we will move towards installing wheels by
> default when they are available instead of requiring the --use-wheel
> flag.
>
> Enjoy! Please share your experiences with the new feature. The most
> common issue is that you must install distribute >= 0.6.34 for
> everything to work well.
>
> Daniel Holth
> ___
> Distutils-SIG maillist - Distutils-SIG@python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
