[Distutils] Call for PEP author/champion: Bundling pip with CPython installers
Based on the recent discussions, I now plan to reject the pip bootstrapping-on-first-invocation approach described in PEP 439 in favour of a new PEP that proposes: * bundling the latest version of pip with the CPython binary installers for Mac OS X and Windows for all future CPython releases (including maintenance releases) * aligns the proposal with the Python 3.4 release schedule by noting that CPython changes must be completed by the first 3.4 beta, while pip changes must be completed by the first 3.4 release candidate. * ensuring that, for Python 3.4, python3 and python3.4 are available for command line invocation of Python, even on Windows * ensuring that the bundled pip, for Python 3.4, ensures pip, pip3 and pip3.4 are available for command line invocation of Python, even on Windows * ensuring that the bundled pip is able to upgrade/downgrade itself independent of the CPython release cycle * ensuring that pip is automatically available in virtual environments created with pyvenv * adding a get-pip.py script to Tools/scripts in the CPython repo for bootstrapping the latest pip release from PyPI into custom CPython source builds Note that there are still open questions to be resolved, which is why an author/champion is needed: * what guidance, if any, should we provide to Linux distro packagers? * how should maintenance updates handle the presence of an existing pip installation? Automatically upgrade older versions to the bundled version, while leaving newer versions alone? Force installation of the bundled version? Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On Jul 13, 2013, at 10:58 PM, Nick Coghlan wrote:
On 14 July 2013 12:46, Donald Stufft don...@stufft.io wrote:
I'm sure I've seen people say other things that have made me think are you
expecting the pip maintainers to make that change? in the various threads,
so I doubt this list is definitive.
The other big one is the one you noted about pip *not* offering a stable
API, *but* exposing an apparently stable API to introspection. Introspection
currently tells me that pip exports *at least* 32 public names (and this is
without checking for public submodules that aren't implicitly imported by
pip/__init__.py):
import pip; public = set(k for k, v in pip.__dict__.items() if not
k.startswith('_') and (not hasattr(v, __name__) or hasattr(v,
__module__) or v.__name__.startswith(pip.))); print(len(public))
32
If pip really has no stable public API, then it should properly indicate
this under introspection (if it already uses relative imports correctly,
then the easiest ways to achieve that are to just shove everything under a
pip._impl subpackage or shuffle it sideways into a _pip package).
Pip does not use relative imports. Is simply documenting the fact there is no
public API enough? Pushing everything into a _impl or _pip directory makes me
nervous because that's a lot of code churn (and I know there are people using
those APIs, and while they aren't technically stable it feels like moving
things around just for the sake of an _ in the name is unfriendly to those
people.
Either the existing APIs are moved to a different name, or they get declared
stable and pip switches to internally forked APIs any time a backwards
incompatible change is needed for refactoring purposes (see
runpy._run_module_as_main for an example of needing to do this in the
standard library). I've had to directly deal with too many issues arising
from getting this wrong in the past for me to endorse bundling of a module
that doesn't follow this practice with CPython - if introspection indicates
an API is public, then it's public and subject to all standard library
backwards compatibility guarantees, or else we take the pain *once* and
explicitly mark it private by adding a leading underscore rather than leaving
it in limbo (contextlib._GeneratorContextManager is a standard library
example of the latter approach - it used to lack the leading underscore,
suggesting it was a public API when it's really just an implementation detail
of contextlib.contextmanager).
Respectfully, I disagree. Pip is not going in to the stdlib, and as such should
not be subject to the same API stability policies as the stdlib. If the PyPA
team wants to break the API every release, that is their call as the subject
matter experts. Pip is not being included as a library at all. What should be
subject to compat is the defined command line interface, because pip is a CLI
tool. Independently of this discussion I've already been talking to the PyPA
team about what they want to consider a stable API, but that is a discussion to
be had over in pip-land, not here and not now. This new category of bundled
for your convenience but still external applications will need new standards,
and we should be clear about them for sure, but I think this is going too far
and puts undue burden on the PyPA team. Remember the end goal is simply to get
an installer in the hands of users easier.
--Noah
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On 14 July 2013 16:19, Noah Kantrowitz n...@coderanger.net wrote: On Jul 13, 2013, at 10:58 PM, Nick Coghlan wrote: On 14 July 2013 12:46, Donald Stufft don...@stufft.io wrote: Either the existing APIs are moved to a different name, or they get declared stable and pip switches to internally forked APIs any time a backwards incompatible change is needed for refactoring purposes (see runpy._run_module_as_main for an example of needing to do this in the standard library). I've had to directly deal with too many issues arising from getting this wrong in the past for me to endorse bundling of a module that doesn't follow this practice with CPython - if introspection indicates an API is public, then it's public and subject to all standard library backwards compatibility guarantees, or else we take the pain *once* and explicitly mark it private by adding a leading underscore rather than leaving it in limbo (contextlib._GeneratorContextManager is a standard library example of the latter approach - it used to lack the leading underscore, suggesting it was a public API when it's really just an implementation detail of contextlib.contextmanager). Respectfully, I disagree. Pip is not going in to the stdlib, and as such should not be subject to the same API stability policies as the stdlib. If the PyPA team wants to break the API every release, that is their call as the subject matter experts. Pip is not being included as a library at all. What should be subject to compat is the defined command line interface, because pip is a CLI tool. Independently of this discussion I've already been talking to the PyPA team about what they want to consider a stable API, but that is a discussion to be had over in pip-land, not here and not now. This new category of bundled for your convenience but still external applications will need new standards, and we should be clear about them for sure, but I think this is going too far and puts undue burden on the PyPA team. Remember the end goal is simply to get an installer in the hands of users easier. I would also be fine with a solution where import pip issues a warning about API instability if sys.argv[0] indicates the main executable is something other than the pip CLI. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On 14 July 2013 16:34, Nick Coghlan ncogh...@gmail.com wrote: On 14 July 2013 16:19, Noah Kantrowitz n...@coderanger.net wrote: On Jul 13, 2013, at 10:58 PM, Nick Coghlan wrote: On 14 July 2013 12:46, Donald Stufft don...@stufft.io wrote: Either the existing APIs are moved to a different name, or they get declared stable and pip switches to internally forked APIs any time a backwards incompatible change is needed for refactoring purposes (see runpy._run_module_as_main for an example of needing to do this in the standard library). I've had to directly deal with too many issues arising from getting this wrong in the past for me to endorse bundling of a module that doesn't follow this practice with CPython - if introspection indicates an API is public, then it's public and subject to all standard library backwards compatibility guarantees, or else we take the pain *once* and explicitly mark it private by adding a leading underscore rather than leaving it in limbo (contextlib._GeneratorContextManager is a standard library example of the latter approach - it used to lack the leading underscore, suggesting it was a public API when it's really just an implementation detail of contextlib.contextmanager). Respectfully, I disagree. Pip is not going in to the stdlib, and as such should not be subject to the same API stability policies as the stdlib. If the PyPA team wants to break the API every release, that is their call as the subject matter experts. Pip is not being included as a library at all. What should be subject to compat is the defined command line interface, because pip is a CLI tool. Independently of this discussion I've already been talking to the PyPA team about what they want to consider a stable API, but that is a discussion to be had over in pip-land, not here and not now. This new category of bundled for your convenience but still external applications will need new standards, and we should be clear about them for sure, but I think this is going too far and puts undue burden on the PyPA team. Remember the end goal is simply to get an installer in the hands of users easier. I would also be fine with a solution where import pip issues a warning about API instability if sys.argv[0] indicates the main executable is something other than the pip CLI. Oops, meant to add a point to https://github.com/pypa/pip/issues/1052 for that one. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On Jul 14, 2013, at 1:58 AM, Nick Coghlan ncogh...@gmail.com wrote: Either the existing APIs are moved to a different name, or they get declared stable and pip switches to internally forked APIs any time a backwards incompatible change is needed for refactoring purposes (see runpy._run_module_as_main for an example of needing to do this in the standard library). I've had to directly deal with too many issues arising from getting this wrong in the past for me to endorse bundling of a module that doesn't follow this practice with CPython - if introspection indicates an API is public, then it's public and subject to all standard library backwards compatibility guarantees, or else we take the pain *once* and explicitly mark it private by adding a leading underscore rather than leaving it in limbo (contextlib._GeneratorContextManager is a standard library example of the latter approach - it used to lack the leading underscore, suggesting it was a public API when it's really just an implementation detail of contextlib.contextmanager). Mere documentation of public vs private generally doesn't cut it, as too many people use dir(), help() and inspect() rather than the published docs to explore APIs. The only general exception I'm aware of is test packages, including the standard library's test package, and for those you can make the case that having test or tests as a name segment is just as clear an indicator of something being private as at least one name segment starting with a leading underscore. I really this is a fairly big ask for the pip maintainers, but I *don't* consider Oh, don't use our module API, it isn't stable to be an adequate answer for something that is bundled with the standard installers. Beyond that, I don't mind if the answer is to declare the 1.5 API stable or to sprinkle underscore where appropriate or moving everything to a private package - the documentation and the naming conventions just need to be consistent in their private vs public distinctions (although your points do suggest heavily that the right answer is to accept the burden of backwards compatibility for all APIs currently marked public, and move towards the introduction of appropriate private APIs over time through refactoring). I agree with Noah here. In my eyes pip is either an external project with it's own polices on backwards compatibility and governance or it's part of the standard library and under the domain of Python core. I'm completely against moving it into the standard library for all the reasons I've given in the past. Maybe I'm reading too much into this but one of my primary fears here is that including pip with the Python distribution is going to lead to Python core dictating to pip what pip must do with itself. Now I don't mean to say that Python core should have no sway over pip either as they are officially blessing it and including it as part of the official releases. If I am reading too much into it then I apologize. I just want to make sure that the boundaries between the governance of Python and pip are clearly defined and the expectations on both sides are laid out and agreed upon before it happens. And I think this raises a good point about how the two projects are going to interact. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Best practices for optional C extensions
I'd like to add a C extension to speed up a small bit of code in a package (Tornado), but make it optional both for compatibility with non-cpython implementations and for ease of installation on systems without a C compiler available. Ideally any user who runs pip install tornado on a system capable of compiling extensions would get the extensions; if this capability cannot be detected automatically I'd prefer the opt-out case to be the one that requires non-default arguments. Are there any packages that provide a good example to follow for this? PEP 426 uses c-accelerators as an example of an extra, but it's unclear how this would work (based on the equivalent setuptools feature). There doesn't appear to be a way to know what extras are requested at build time. If the extra required a package like cython then you could build the extension whenever that package is present, but what about hand-written extensions? Extras are also opt-in instead of opt-out, so I'd have to recommend that most people use pip install tornado[fast] instead of pip install tornado (with tornado[slow] available as an option for limited environments). Thanks, -Ben ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Best practices for optional C extensions
I'd like to add a C extension to speed up a small bit of code in a package (Tornado), but make it optional both for compatibility with non-cpython implementations and for ease of installation on systems without a C compiler available. Ideally any user who runs pip install tornado on a system capable of compiling extensions would get the extensions; if this capability cannot be detected automatically I'd prefer the opt-out case to be the one that requires non-default arguments. Are there any packages that provide a good example to follow for this? PEP 426 uses c-accelerators as an example of an extra, but it's unclear how this would work (based on the equivalent setuptools feature). There doesn't appear to be a way to know what extras are requested at build time. If the extra required a package like cython then you could build the extension whenever that package is present, but what about hand-written extensions? Extras are also opt-in instead of opt-out, so I'd have to recommend that most people use pip install tornado[fast] instead of pip install tornado (with tornado[slow] available as an option for limited environments). Thanks, -Ben ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On 14 July 2013 16:43, Donald Stufft don...@stufft.io wrote: I just want to make sure that the boundaries between the governance of Python and pip are clearly defined and the expectations on both sides are laid out and agreed upon before it happens. And I think this raises a good point about how the two projects are going to interact. Agreed, I think the boundaries need to be clear. If something installed by default is *only* support code for a bundled application, then it should either adhere to the standard library's backwards compatibility policies (by appropriately marking private APIs as private), or else it should issue a warning when imported by any other application. Either of those options sounds good to me. However, I consider expecting people to just know (or to look at documentation to determine) which provided modules are public or private without adhering to standard naming conventions or providing an explicit runtime warning to be unreasonable. (and yes, if pip goes down the runtime warning path, we should probably look into providing a runtime warning for at least the test namespace and possibly even the idlelib namespace, too) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On Jul 14, 2013, at 3:01 AM, Nick Coghlan ncogh...@gmail.com wrote: On 14 July 2013 16:43, Donald Stufft don...@stufft.io wrote: I just want to make sure that the boundaries between the governance of Python and pip are clearly defined and the expectations on both sides are laid out and agreed upon before it happens. And I think this raises a good point about how the two projects are going to interact. Agreed, I think the boundaries need to be clear. If something installed by default is *only* support code for a bundled application, then it should either adhere to the standard library's backwards compatibility policies (by appropriately marking private APIs as private), or else it should issue a warning when imported by any other application. Either of those options sounds good to me. However, I consider expecting people to just know (or to look at documentation to determine) which provided modules are public or private without adhering to standard naming conventions or providing an explicit runtime warning to be unreasonable. (and yes, if pip goes down the runtime warning path, we should probably look into providing a runtime warning for at least the test namespace and possibly even the idlelib namespace, too) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia Yea I forget to talk about the *actual* change that prompted that email when I started feeling dictated to which touched upon one of my fears in this process :) I'm not against either renaming or emitting a warning. I was actually asking if just documenting the fact would be ok because I fear bugs from the code churn that renaming would cause :) I think we'd need to rename things because emitting a warning is an all or nothing ordeal and we've had requests to make certain parts of the API public for Chef and other tools like it. A question that certainly raises in my mind though is standard library's backwards compatibility policies. What affect does this have on *actual* public API exposed from pip? Does it mean we cannot break compatibility for them until Python 4.x? That sounds very onerous for something that is installed in a way that allows easy upgrade and downgrading separately from Python to match the version requirements of someone using that library. Pip has it's own versions and develops at it's own speed. I think it would be reasonable for the pip maintainers to be asked to declare a public API (even if that's None) using the naming scheme or an import warning and declare a backwards compatibility policy for pip itself so that people can know what to expect from pip. I do not however, believe it is reasonable to bind pip to the same policy that CPython uses nor the same schedule. (If you weren't suggesting that I apologize). - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On 14 July 2013 17:13, Donald Stufft don...@stufft.io wrote: I think it would be reasonable for the pip maintainers to be asked to declare a public API (even if that's None) using the naming scheme or an import warning and declare a backwards compatibility policy for pip itself so that people can know what to expect from pip. I do not however, believe it is reasonable to bind pip to the same policy that CPython uses nor the same schedule. (If you weren't suggesting that I apologize). The main elements of CPython's backwards compatibility policy that I consider relevant are: * Use leading underscores to denote private APIs with no backwards compatibility guarantees * Be conservative with deprecating public APIs that aren't fundamentally broken * Use DeprecationWarning to give at least one (pip) release notice of an upcoming backwards incompatible change We *are* sometimes quite aggressive with deprecation and removal even in the standard library - we removed contextlib.nested from Python 3.2 as a problematic bug magnet well before I came up with the contextlib.ExitStack API as a less error prone replacement in Python 3.3. It's only when it comes to core syntax and builtin behaviour that we're likely to hit issues that simply don't have a sensible deprecation strategy, so we decide we have to live with them indefinitely. That said, I think the answer to this discussion also affects the answer to whether or not CPython maintenance releases should update to newer versions of pip: if pip chooses to adopt a faster deprecation cycle than CPython, then our maintenance releases shouldn't bundle updated versions. Instead, they should follow the policy: * if this is a new major release, or the first maintenance release to bundle pip, bundle the latest available version of pip * otherwise, bundle the same version of pip as the previous release This would mean we'd be asking the pip team to help out by providing security releases for the bundled version, so we can get that without breaking the public API that's available by default. On the other hand, if the pip team are willing to use long deprecation cycles then we can just bundle the updated versions and not worry about security releases (I'd prefer that, but it only works if the pip team are willing to put up with keeping old APIs around for a couple of years before killing them off once the affected CPython branches go into security fix only mode). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
I do think pip is pretty conservative about backwards compat other than the security related changes I've been doing. I think we can find the middle ground that lets things work smoothly here :). I was just making sure that we wernt going to have to keep things around for really long times like python 4 ;) On Jul 14, 2013, at 3:35 AM, Nick Coghlan ncogh...@gmail.com wrote: On 14 July 2013 17:13, Donald Stufft don...@stufft.io wrote: I think it would be reasonable for the pip maintainers to be asked to declare a public API (even if that's None) using the naming scheme or an import warning and declare a backwards compatibility policy for pip itself so that people can know what to expect from pip. I do not however, believe it is reasonable to bind pip to the same policy that CPython uses nor the same schedule. (If you weren't suggesting that I apologize). The main elements of CPython's backwards compatibility policy that I consider relevant are: * Use leading underscores to denote private APIs with no backwards compatibility guarantees * Be conservative with deprecating public APIs that aren't fundamentally broken * Use DeprecationWarning to give at least one (pip) release notice of an upcoming backwards incompatible change We *are* sometimes quite aggressive with deprecation and removal even in the standard library - we removed contextlib.nested from Python 3.2 as a problematic bug magnet well before I came up with the contextlib.ExitStack API as a less error prone replacement in Python 3.3. It's only when it comes to core syntax and builtin behaviour that we're likely to hit issues that simply don't have a sensible deprecation strategy, so we decide we have to live with them indefinitely. That said, I think the answer to this discussion also affects the answer to whether or not CPython maintenance releases should update to newer versions of pip: if pip chooses to adopt a faster deprecation cycle than CPython, then our maintenance releases shouldn't bundle updated versions. Instead, they should follow the policy: * if this is a new major release, or the first maintenance release to bundle pip, bundle the latest available version of pip * otherwise, bundle the same version of pip as the previous release This would mean we'd be asking the pip team to help out by providing security releases for the bundled version, so we can get that without breaking the public API that's available by default. On the other hand, if the pip team are willing to use long deprecation cycles then we can just bundle the updated versions and not worry about security releases (I'd prefer that, but it only works if the pip team are willing to put up with keeping old APIs around for a couple of years before killing them off once the affected CPython branches go into security fix only mode). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Best practices for optional C extensions
On 13 July 2013 14:14, Ben Darnell b...@bendarnell.com wrote: I'd like to add a C extension to speed up a small bit of code in a package (Tornado), but make it optional both for compatibility with non-cpython implementations and for ease of installation on systems without a C compiler available. Ideally any user who runs pip install tornado on a system capable of compiling extensions would get the extensions; if this capability cannot be detected automatically I'd prefer the opt-out case to be the one that requires non-default arguments. Are there any packages that provide a good example to follow for this? I believe that coverage has an optional C extension like this. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On 14 July 2013 17:50, Donald Stufft don...@stufft.io wrote: I do think pip is pretty conservative about backwards compat other than the security related changes I've been doing. I think we can find the middle ground that lets things work smoothly here :). I was just making sure that we wernt going to have to keep things around for really long times like python 4 ;) Even most of the standard library isn't that conservative - it usually only happens when we can't find a sensible place to hook up DeprecationWarning. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On Jul 14, 2013, at 12:35 AM, Nick Coghlan wrote: On 14 July 2013 17:13, Donald Stufft don...@stufft.io wrote: I think it would be reasonable for the pip maintainers to be asked to declare a public API (even if that's None) using the naming scheme or an import warning and declare a backwards compatibility policy for pip itself so that people can know what to expect from pip. I do not however, believe it is reasonable to bind pip to the same policy that CPython uses nor the same schedule. (If you weren't suggesting that I apologize). The main elements of CPython's backwards compatibility policy that I consider relevant are: * Use leading underscores to denote private APIs with no backwards compatibility guarantees * Be conservative with deprecating public APIs that aren't fundamentally broken * Use DeprecationWarning to give at least one (pip) release notice of an upcoming backwards incompatible change We *are* sometimes quite aggressive with deprecation and removal even in the standard library - we removed contextlib.nested from Python 3.2 as a problematic bug magnet well before I came up with the contextlib.ExitStack API as a less error prone replacement in Python 3.3. It's only when it comes to core syntax and builtin behaviour that we're likely to hit issues that simply don't have a sensible deprecation strategy, so we decide we have to live with them indefinitely. That said, I think the answer to this discussion also affects the answer to whether or not CPython maintenance releases should update to newer versions of pip: if pip chooses to adopt a faster deprecation cycle than CPython, then our maintenance releases shouldn't bundle updated versions. Instead, they should follow the policy: * if this is a new major release, or the first maintenance release to bundle pip, bundle the latest available version of pip * otherwise, bundle the same version of pip as the previous release This would mean we'd be asking the pip team to help out by providing security releases for the bundled version, so we can get that without breaking the public API that's available by default. On the other hand, if the pip team are willing to use long deprecation cycles then we can just bundle the updated versions and not worry about security releases (I'd prefer that, but it only works if the pip team are willing to put up with keeping old APIs around for a couple of years before killing them off once the affected CPython branches go into security fix only mode). If I can surmise your worry here, it is that people will open an interactive terminal, import pip, reflect out the classes/methods/etc, see that despite being mentioned no-where in the Python or pip documentation the methods and classes don't start with an underscore, and thus conclude that this is a stable API to build against? I agree that conventions are good, but I have to say this sounds like a bit of a stretch and certainly anyone complaining that their undocumented API that they only found via reflection (or reading the pip source) was broken basically gets what they deserve. The point I was trying to make is that a major shift in thinking is needed here. pip is not part of CPython, regardless of this bundling neither this mailing list nor the CPython team will have any control (aside from the nuclear option that the CPython team can elect to stop bundling pip). If you think it would be good for the code-health of pip to be clearer about what their public API is, I will suppor t that all the way and in fact have an open ticket against pip to that effect already, but that is something for the pip team to decide. This does very much mean that the CPython team is not just backing the pip codebase, but the PyPA/pip team. I think the past few years have shown them deserving of this trust, and they should be allowed to run things as they see fit. These lines get blurry since several people move back and forth between CPython and PyPA (and distutils and PyPI, etc) hats, so I think this must be stated clearly up front that what the CPython team thinks is reasonable for an API policy will be nothing more than a recommendation from very knowledgable colleagues and will be given the appropriate consideration and respect it deserves based on that. Hopefully that makes my point-of-view a little clearer. --Noah ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On 14 Jul 2013 18:24, Noah Kantrowitz n...@coderanger.net wrote: On Jul 14, 2013, at 12:35 AM, Nick Coghlan wrote: On 14 July 2013 17:13, Donald Stufft don...@stufft.io wrote: I think it would be reasonable for the pip maintainers to be asked to declare a public API (even if that's None) using the naming scheme or an import warning and declare a backwards compatibility policy for pip itself so that people can know what to expect from pip. I do not however, believe it is reasonable to bind pip to the same policy that CPython uses nor the same schedule. (If you weren't suggesting that I apologize). The main elements of CPython's backwards compatibility policy that I consider relevant are: * Use leading underscores to denote private APIs with no backwards compatibility guarantees * Be conservative with deprecating public APIs that aren't fundamentally broken * Use DeprecationWarning to give at least one (pip) release notice of an upcoming backwards incompatible change We *are* sometimes quite aggressive with deprecation and removal even in the standard library - we removed contextlib.nested from Python 3.2 as a problematic bug magnet well before I came up with the contextlib.ExitStack API as a less error prone replacement in Python 3.3. It's only when it comes to core syntax and builtin behaviour that we're likely to hit issues that simply don't have a sensible deprecation strategy, so we decide we have to live with them indefinitely. That said, I think the answer to this discussion also affects the answer to whether or not CPython maintenance releases should update to newer versions of pip: if pip chooses to adopt a faster deprecation cycle than CPython, then our maintenance releases shouldn't bundle updated versions. Instead, they should follow the policy: * if this is a new major release, or the first maintenance release to bundle pip, bundle the latest available version of pip * otherwise, bundle the same version of pip as the previous release This would mean we'd be asking the pip team to help out by providing security releases for the bundled version, so we can get that without breaking the public API that's available by default. On the other hand, if the pip team are willing to use long deprecation cycles then we can just bundle the updated versions and not worry about security releases (I'd prefer that, but it only works if the pip team are willing to put up with keeping old APIs around for a couple of years before killing them off once the affected CPython branches go into security fix only mode). If I can surmise your worry here, it is that people will open an interactive terminal, import pip, reflect out the classes/methods/etc, see that despite being mentioned no-where in the Python or pip documentation the methods and classes don't start with an underscore, and thus conclude that this is a stable API to build against? I agree that conventions are good, but I have to say this sounds like a bit of a stretch and certainly anyone complaining that their undocumented API that they only found via reflection (or reading the pip source) was broken basically gets what they deserve. The point I was trying to make is that a major shift in thinking is needed here. pip is not part of CPython, regardless of this bundling neither this mailing list nor the CPython team will have any control (aside from the nuclear option that the CPython team can elect to stop bundling pip). If you think it would be good for the code-health of pip to be clearer about what their public API is, I will suppor t that all the way and in fact have an open ticket against pip to that effect already, but that is something for the pip team to decide. This does very much mean that the CPython team is not just backing the pip codebase, but the PyPA/pip team. I think the past few years have shown them deserving of this trust, and they should be allowed to run things as they see fit. These lines get blurry since several people move back and forth between CPython and PyPA (and distutils and PyPI, etc) hats, so I think this must be stated clearly up front that what the CPython team thinks is reasonable for an API policy will be nothing more than a recommendation from very knowledgable colleagues and will be given the appropriate consideration and respect it deserves based on that. Hopefully that makes my point-of-view a little clearer. I started a thread on python-dev proposing strengthened wording in PEP 8 regarding marking of private interfaces, but beyond that, yes, I now agree that this isn't a blocker for bundling pip with CPython. Cheers, Nick. --Noah ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Another conversation starter - pip documentation in the Python docs
I don't think there is any doubt that we need to document to at least some extent how to use pip, in the Python documentation for 3.4. The obvious place is Installing Python Modules as a chapter to itself. What is less clear to me is how much to document - just basic pip install XXX, or the whole pip command set, or somewhere in between? Should there be some type of refer to www.pip-installer.org for the complete documentation reference? Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Another conversation starter - pip documentation in the Python docs
On 14 July 2013 21:09, Paul Moore p.f.mo...@gmail.com wrote: I don't think there is any doubt that we need to document to at least some extent how to use pip, in the Python documentation for 3.4. The obvious place is Installing Python Modules as a chapter to itself. What is less clear to me is how much to document - just basic pip install XXX, or the whole pip command set, or somewhere in between? Should there be some type of refer to www.pip-installer.org for the complete documentation reference? This is the hole https://python-packaging-user-guide.readthedocs.org/en/latest/ is supposed to fill - once it's ready (i.e. things have stabilised sufficiently , then I'd like to replace the Installing Python Modules and Distributing Python Modules sections for 2.7 and 3.3 with some *very* abbreviated quick start guides that then reference that site. The 3.3 changes would then carry over into 3.4. I spent some time at PyCon AU talking to Matthew Iverson ( https://bitbucket.org/Ivoz/python-packaging-user-guide) who I believe Marcus was hoping to get back to that after pip 1.4 was out the door, but anyone on the PyPA list on BitBucket actually has full access to accept pull requests, etc. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Another conversation starter - pip documentation in the Python docs
On 14 July 2013 12:42, Nick Coghlan ncogh...@gmail.com wrote: This is the hole https://python-packaging-user-guide.readthedocs.org/en/latest/ is supposed to fill - once it's ready (i.e. things have stabilised sufficiently , then I'd like to replace the Installing Python Modules and Distributing Python Modules sections for 2.7 and 3.3 with some *very* abbreviated quick start guides that then reference that site. The 3.3 changes would then carry over into 3.4. Hmm, OK. I've no problem with that (although I do find the packaging guide pretty hard to get into for an end user who only wants to *use* packages, not *create* them, but that's a separate issue for me to address by providing some pull requests). I was more thinking in terms of your quick start guides. I think we should explain *in the core documentation* how to (a) install a new package, (b) uninstall a package, (c) list what is installed and (d) upgrade pip itself. That translates to the pip install, uninstall, and list commands at a minimum. I could offer some text, if that's the way you want to go with this. How about if I provide a new (short) document called something like Python package management and we work out how to integrate it into the docs as things settle down? Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping))
On 13 July 2013 10:05, Paul Moore p.f.mo...@gmail.com wrote: How robust is the process of upgrading pip using itself? Specifically on Windows, where these things typically seem less reliable. OK, I just did some tests. On Windows, pip install -U pip FAILS. The reason for the failure is simple enough to explain - the pip.exe wrapper is held open by the OS while it's in use, so that the upgrade cannot replace it. The result is a failed upgrade and a partially installed new version of pip. In practice, the exe stubs are probably added fairly late in the install (at least when installing from sdist, with a wheel that depends on the order of the files in the wheel), so it's probably only a little bit broken, but a little bit broken is still broken :-( On the other hand, python -m pip install -U pip works fine because it avoids the exe wrappers. There's a lot of scope for user confusion and frustration in all this. For standalone pip I've tended to recommend don't do that - manually uninstall and reinstall pip, or recreate your virtualenv. It's not nice, but it's effective. That sort of advice isn't going to be realistic for a pip bundled with CPython. Does anyone have any suggestions? Paul. PS In better news, apart from this issue, pip upgrades of pip and setuptools seem fine. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Another conversation starter - pip documentation in the Python docs
On 14 July 2013 23:01, Paul Moore p.f.mo...@gmail.com wrote: On 14 July 2013 12:42, Nick Coghlan ncogh...@gmail.com wrote: This is the hole https://python-packaging-user-guide.readthedocs.org/en/latest/ is supposed to fill - once it's ready (i.e. things have stabilised sufficiently , then I'd like to replace the Installing Python Modules and Distributing Python Modules sections for 2.7 and 3.3 with some *very* abbreviated quick start guides that then reference that site. The 3.3 changes would then carry over into 3.4. Hmm, OK. I've no problem with that (although I do find the packaging guide pretty hard to get into for an end user who only wants to *use* packages, not *create* them, but that's a separate issue for me to address by providing some pull requests). I was more thinking in terms of your quick start guides. I think we should explain *in the core documentation* how to (a) install a new package, (b) uninstall a package, (c) list what is installed and (d) upgrade pip itself. That translates to the pip install, uninstall, and list commands at a minimum. I could offer some text, if that's the way you want to go with this. How about if I provide a new (short) document called something like Python package management and we work out how to integrate it into the docs as things settle down? That sounds great - so far it's mostly just been myself and Marcus thinking about it (mostly Marcus, to be honest, along with a couple of folks that submitted pull requests and BitBucket issues), and it keeps getting bumped down the todo list by other things. I think we're getting closer to having something stable enough to document clearly, though - with distribute merged back into setuptools and pip not far away, the bootstrapping seems to be the only remaining slightly messy part (since the 3.4 discussions aren't relevant to the user guide as yet). As far as your first point goes, I agree the Installation Tutorial part should probably come first and definitely needs more content. I did just accept a pull request earlier that at least makes that page more than just a list of headings (see https://python-packaging-user-guide.readthedocs.org/en/latest/installation_tutorial.html- courtesy of https://bitbucket.org/alexjeffburke) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Expectations on how pip needs to change for Python 3.4
On Sun, Jul 14, 2013 at 4:23 AM, Noah Kantrowitz n...@coderanger.netwrote: On Jul 14, 2013, at 12:35 AM, Nick Coghlan wrote: On 14 July 2013 17:13, Donald Stufft don...@stufft.io wrote: I think it would be reasonable for the pip maintainers to be asked to declare a public API (even if that's None) using the naming scheme or an import warning and declare a backwards compatibility policy for pip itself so that people can know what to expect from pip. I do not however, believe it is reasonable to bind pip to the same policy that CPython uses nor the same schedule. (If you weren't suggesting that I apologize). The main elements of CPython's backwards compatibility policy that I consider relevant are: * Use leading underscores to denote private APIs with no backwards compatibility guarantees * Be conservative with deprecating public APIs that aren't fundamentally broken * Use DeprecationWarning to give at least one (pip) release notice of an upcoming backwards incompatible change We *are* sometimes quite aggressive with deprecation and removal even in the standard library - we removed contextlib.nested from Python 3.2 as a problematic bug magnet well before I came up with the contextlib.ExitStack API as a less error prone replacement in Python 3.3. It's only when it comes to core syntax and builtin behaviour that we're likely to hit issues that simply don't have a sensible deprecation strategy, so we decide we have to live with them indefinitely. That said, I think the answer to this discussion also affects the answer to whether or not CPython maintenance releases should update to newer versions of pip: if pip chooses to adopt a faster deprecation cycle than CPython, then our maintenance releases shouldn't bundle updated versions. Instead, they should follow the policy: * if this is a new major release, or the first maintenance release to bundle pip, bundle the latest available version of pip * otherwise, bundle the same version of pip as the previous release This would mean we'd be asking the pip team to help out by providing security releases for the bundled version, so we can get that without breaking the public API that's available by default. On the other hand, if the pip team are willing to use long deprecation cycles then we can just bundle the updated versions and not worry about security releases (I'd prefer that, but it only works if the pip team are willing to put up with keeping old APIs around for a couple of years before killing them off once the affected CPython branches go into security fix only mode). If I can surmise your worry here, it is that people will open an interactive terminal, import pip, reflect out the classes/methods/etc, see that despite being mentioned no-where in the Python or pip documentation the methods and classes don't start with an underscore, and thus conclude that this is a stable API to build against? Yes, and to make that statement even stronger: all of that happening with a freshly installed copy of Python with no external packages installed. Nick's worry stems from experience (which I have also had) where people simply don't check docs as to whether something is public in the stdlib and so it ends up being considered such by users to the point that we feel obliged to support it. I agree that conventions are good, but I have to say this sounds like a bit of a stretch and certainly anyone complaining that their undocumented API that they only found via reflection (or reading the pip source) was broken basically gets what they deserve. That's what we typically say for older modules, especially when we are cranky. =) But it doesn't stop the wingeing and bug reports. Luckily I think we have gotten better about this. The point I was trying to make is that a major shift in thinking is needed here. pip is not part of CPython, regardless of this bundling neither this mailing list nor the CPython team will have any control (aside from the nuclear option that the CPython team can elect to stop bundling pip). If you think it would be good for the code-health of pip to be clearer about what their public API is, I will suppor t that all the way and in fact have an open ticket against pip to that effect already, but that is something for the pip team to decide. This does very much mean that the CPython team is not just backing the pip codebase, but the PyPA/pip team. I think the past few years have shown them deserving of this trust, and they should be allowed to run things as they see fit. These lines get blurry since several people move back and forth between CPython and PyPA (and distutils and PyPI, etc) hats, so I think this must be stated clearly up front that what the CPython team thinks is reasonable for an API policy will be nothing more than a recommendation from very knowledgable colleagues and will be given the appropriate consideration and respect it deserves based
Re: [Distutils] Best practices for optional C extensions
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 07/13/2013 09:14 AM, Ben Darnell wrote:
I'd like to add a C extension to speed up a small bit of code in a
package (Tornado), but make it optional both for compatibility with
non-cpython implementations and for ease of installation on systems
without a C compiler available. Ideally any user who runs pip
install tornado on a system capable of compiling extensions would get
the extensions; if this capability cannot be detected automatically
I'd prefer the opt-out case to be the one that requires non-default
arguments. Are there any packages that provide a good example to
follow for this?
PEP 426 uses c-accelerators as an example of an extra, but it's
unclear how this would work (based on the equivalent setuptools
feature). There doesn't appear to be a way to know what extras are
requested at build time. If the extra required a package like cython
then you could build the extension whenever that package is present,
but what about hand-written extensions? Extras are also opt-in
instead of opt-out, so I'd have to recommend that most people use pip
install tornado[fast] instead of pip install tornado (with
tornado[slow] available as an option for limited environments).
zope.interface subclasses the 'build_ext' command so:
- % -
from distutils.command.build_ext import build_ext
from distutils.errors import CCompilerError
from distutils.errors import DistutilsExecError
from distutils.errors import DistutilsPlatformError
class optional_build_ext(build_ext):
Allow the building of C extensions to fail.
def run(self):
try:
build_ext.run(self)
except DistutilsPlatformError as e:
self._unavailable(e)
def build_extension(self, ext):
try:
build_ext.build_extension(self, ext)
except (CCompilerError, DistutilsExecError) as e:
self._unavailable(e)
def _unavailable(self, e):
print('*' * 80)
print(WARNING:
An optional code optimization (C extension) could not be compiled.
Optimizations for this package will not be available!)
print()
print(e)
print('*' * 80)
- % -
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software Excellence by Designhttp://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/
iEYEARECAAYFAlHitjoACgkQ+gerLs4ltQ7SJQCgrhgN58g9ztFPEkFAOM49Wu4p
RpQAoLnboKietjTx3eXho1kyRvH3r2uN
=qWRK
-END PGP SIGNATURE-
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Call for PEP author/champion: Bundling pip with CPython installers
On 15 July 2013 00:28, Brett Cannon br...@python.org wrote: On Sun, Jul 14, 2013 at 2:13 AM, Nick Coghlan ncogh...@gmail.com wrote: Based on the recent discussions, I now plan to reject the pip bootstrapping-on-first-invocation approach described in PEP 439 in favour of a new PEP that proposes: [SNIP] * ensuring that, for Python 3.4, python3 and python3.4 are available for command line invocation of Python, even on Windows Can I ask why this is part of the PEP? Mostly because pip3 makes no sense without it. If we *don't* bring Windows into conformance with the way other platforms handle parallel Python 2 and Python 3 installs, then the PEP should document an explicit rationale for not doing it (the py launcher doesn't count in that regard, since the real purpose of that is to handle shebang lines through file associations, with the direct command line usage as an added bonus). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Call for PEP author/champion: Bundling pip with CPython installers
On Sun, Jul 14, 2013 at 2:13 AM, Nick Coghlan ncogh...@gmail.com wrote: Based on the recent discussions, I now plan to reject the pip bootstrapping-on-first-invocation approach described in PEP 439 in favour of a new PEP that proposes: [SNIP] * ensuring that, for Python 3.4, python3 and python3.4 are available for command line invocation of Python, even on Windows Can I ask why this is part of the PEP? ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Call for PEP author/champion: Bundling pip with CPython installers
On 14 July 2013 02:13, Nick Coghlan ncogh...@gmail.com wrote: Based on the recent discussions, I now plan to reject the pip bootstrapping-on-first-invocation approach described in PEP 439 in favour of a new PEP that proposes: * bundling the latest version of pip with the CPython binary installers for Mac OS X and Windows for all future CPython releases (including maintenance releases) * aligns the proposal with the Python 3.4 release schedule by noting that CPython changes must be completed by the first 3.4 beta, while pip changes must be completed by the first 3.4 release candidate. * ensuring that, for Python 3.4, python3 and python3.4 are available for command line invocation of Python, even on Windows * ensuring that the bundled pip, for Python 3.4, ensures pip, pip3 and pip3.4 are available for command line invocation of Python, even on Windows * ensuring that the bundled pip is able to upgrade/downgrade itself independent of the CPython release cycle * ensuring that pip is automatically available in virtual environments created with pyvenv * adding a get-pip.py script to Tools/scripts in the CPython repo for bootstrapping the latest pip release from PyPI into custom CPython source builds Note that there are still open questions to be resolved, which is why an author/champion is needed: I've a bunch of contacts in various distros still - I've not championed a PEP before but I would be happy to take this on. * what guidance, if any, should we provide to Linux distro packagers? * how should maintenance updates handle the presence of an existing pip installation? There are various distro packaging specific ways of handling this. Hard requirements, recommends, obsoleting the standalone package and providing it virtually as part of Automatically upgrade older versions to the bundled version, while leaving newer versions alone? Force installation of the bundled version? My personal experience is that forcing the bundled version to OS with strong in-built packaging (Linux, BSD, other *NIX) is likely to meet with some resistance. I can certainly talk with some people, my instinct is it's likely to be only bundle with installers, allow optional install as part of the cPython build which can then be subpackaged/ignored for seperate pip/bundled as distros so desire. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] [issue154] file protection group/world writeable on egg-info files since 0.7x
New submission from richard: since 0.7.x, there are some issues in the tarball setuptools.egg-info directory...namely, 1. the file protection of the files is group/world writeable (should be group/world read-only) 2. there are additional .orig files 3. there is an additional directory EGG-INFO containing an obsolete copy of PKG-INFO. This should be trivial to fix in the source tree. extract below from the 0.8 tarball: drwxrwxrwx 0/0 0 2013-07-05 19:12 setuptools-0.8/setuptools.egg-info/ -rw-rw-rw- 0/0 654 2013-07-05 19:12 setuptools-0.8/setuptools.egg-info/dependency_links.txt drwxrwxrwx 0/0 0 2013-07-05 19:12 setuptools-0.8/setuptools.egg-info/EGG-INFO/ -rw-rw-rw- 0/0 153 2013-07-02 17:30 setuptools-0.8/setuptools.egg-info/EGG-INFO/PKG-INFO -rw-rw-rw- 0/02773 2013-07-05 19:12 setuptools-0.8/setuptools.egg-info/entry_points.txt -rw-rw-rw- 0/02773 2013-07-03 14:12 setuptools-0.8/setuptools.egg-info/entry_points.txt.orig -rw-rw-rw- 0/0 45823 2013-07-05 19:12 setuptools-0.8/setuptools.egg-info/PKG-INFO -rw-rw-rw- 0/0 186 2013-07-05 19:12 setuptools-0.8/setuptools.egg-info/requires.txt -rw-rw-rw- 0/0 186 2013-07-03 14:12 setuptools-0.8/setuptools.egg-info/requires.txt.orig -rw-rw-rw- 0/03663 2013-07-05 19:12 setuptools-0.8/setuptools.egg-info/SOURCES.txt -rw-rw-rw- 0/0 49 2013-07-05 19:12 setuptools-0.8/setuptools.egg-info/top_level.txt -rw-rw-rw- 0/0 2 2013-07-02 17:48 setuptools-0.8/setuptools.egg-info/zip-safe -- messages: 737 nosy: richard priority: bug status: unread title: file protection group/world writeable on egg-info files since 0.7x ___ Setuptools tracker setupto...@bugs.python.org http://bugs.python.org/setuptools/issue154 ___ ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping))
From: Paul Moore On 13 July 2013 10:05, Paul Moore p.f.mo...@gmail.com wrote: How robust is the process of upgrading pip using itself? Specifically on Windows, where these things typically seem less reliable. OK, I just did some tests. On Windows, pip install -U pip FAILS. The reason for the failure is simple enough to explain - the pip.exe wrapper is held open by the OS while it's in use, so that the upgrade cannot replace it. The result is a failed upgrade and a partially installed new version of pip. In practice, the exe stubs are probably added fairly late in the install (at least when installing from sdist, with a wheel that depends on the order of the files in the wheel), so it's probably only a little bit broken, but a little bit broken is still broken :-( On the other hand, python -m pip install -U pip works fine because it avoids the exe wrappers. There's a lot of scope for user confusion and frustration in all this. For standalone pip I've tended to recommend don't do that - manually uninstall and reinstall pip, or recreate your virtualenv. It's not nice, but it's effective. That sort of advice isn't going to be realistic for a pip bundled with CPython. Does anyone have any suggestions? Unless I misunderstand how the exe wrappers work (they're all the same code that looks for a .py file by the same name?) it may be easiest to somehow mark them as non-vital, such that failing to update them does not fail the installer. Maybe detect that it can't be overwritten, compare the contents/hash with the new one, and only fail if it's changed (with an instruction to use 'python -m...')? Spawning a separate process to do the install is probably no good, since you'd have to kill the original one which is going to break command line output. MoveFileEx (with its copy-on-reboot flag) is off the table, since it requires elevation and a reboot. But I think that's the only supported API for doing a deferred copy. If Windows was opening .exes with FILE_SHARE_DELETE then it would be possible to delete the exe and create a new one by the same name, but I doubt that will work and in any case could not be assumed to never change. So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed. Cheers, Steve ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Another conversation starter - pip documentation in the Python docs
Hmm, OK. I've no problem with that (although I do find the packaging guide pretty hard to get into for an end user who only wants to *use* packages, not *create* them there's a section marked Installation Tutorial. Someone wanting to install packages should be able to get into that, once it actually has content : ) I think we should explain *in the core documentation* How about if I provide a new (short) document called something like Python package management IMO, we should put the energy into one comprehensive document (the new user guide) and link out to it, not duplicate effort into the core docs. The Installing Python Modules/Distributing Python Modules documents should be integrated into the new User Guide. It will be confusing and anti-zen for a user to come upon those documents and have to reconcile it with the user guide Marcus ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping))
On 14 July 2013 17:45, Steve Dower steve.do...@microsoft.com wrote: So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed. Thanks. That sounds annoyingly complex (pretty much as I expected, though). My feeling is that I'd like to remove the exe wrapper altogether, and use a .py file. I need to check what issues there might be with that before recommending it, though. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping))
On Jul 14, 2013, at 1:03 PM, Paul Moore p.f.mo...@gmail.com wrote: On 14 July 2013 17:45, Steve Dower steve.do...@microsoft.com wrote: So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed. Thanks. That sounds annoyingly complex (pretty much as I expected, though). My feeling is that I'd like to remove the exe wrapper altogether, and use a .py file. I need to check what issues there might be with that before recommending it, though. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig Wouldn't a .py file make the command `pip.py`` and not ``pip`` ? - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping))
On Jul 14, 2013, at 9:45 AM, Steve Dower wrote: From: Paul Moore On 13 July 2013 10:05, Paul Moore p.f.mo...@gmail.com wrote: How robust is the process of upgrading pip using itself? Specifically on Windows, where these things typically seem less reliable. OK, I just did some tests. On Windows, pip install -U pip FAILS. The reason for the failure is simple enough to explain - the pip.exe wrapper is held open by the OS while it's in use, so that the upgrade cannot replace it. The result is a failed upgrade and a partially installed new version of pip. In practice, the exe stubs are probably added fairly late in the install (at least when installing from sdist, with a wheel that depends on the order of the files in the wheel), so it's probably only a little bit broken, but a little bit broken is still broken :-( On the other hand, python -m pip install -U pip works fine because it avoids the exe wrappers. There's a lot of scope for user confusion and frustration in all this. For standalone pip I've tended to recommend don't do that - manually uninstall and reinstall pip, or recreate your virtualenv. It's not nice, but it's effective. That sort of advice isn't going to be realistic for a pip bundled with CPython. Does anyone have any suggestions? Unless I misunderstand how the exe wrappers work (they're all the same code that looks for a .py file by the same name?) it may be easiest to somehow mark them as non-vital, such that failing to update them does not fail the installer. Maybe detect that it can't be overwritten, compare the contents/hash with the new one, and only fail if it's changed (with an instruction to use 'python -m...')? Spawning a separate process to do the install is probably no good, since you'd have to kill the original one which is going to break command line output. MoveFileEx (with its copy-on-reboot flag) is off the table, since it requires elevation and a reboot. But I think that's the only supported API for doing a deferred copy. If Windows was opening .exes with FILE_SHARE_DELETE then it would be possible to delete the exe and create a new one by the same name, but I doubt that will work and in any case could not be assumed to never change. So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed. The usual way to do this is just move the existing executable to pip.exe.deleteme or something, and then write out the new one. Then on every startup (or maybe some level of special case for just pip upgrades?) try to unlink *.deleteme. Not the simplest system ever, but it gets the job done. --Noah ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping))
On Sun, Jul 14, 2013 at 1:12 PM, Noah Kantrowitz n...@coderanger.net wrote: On Jul 14, 2013, at 9:45 AM, Steve Dower wrote: From: Paul Moore On 13 July 2013 10:05, Paul Moore p.f.mo...@gmail.com wrote: How robust is the process of upgrading pip using itself? Specifically on Windows, where these things typically seem less reliable. OK, I just did some tests. On Windows, pip install -U pip FAILS. The reason for the failure is simple enough to explain - the pip.exe wrapper is held open by the OS while it's in use, so that the upgrade cannot replace it. The result is a failed upgrade and a partially installed new version of pip. In practice, the exe stubs are probably added fairly late in the install (at least when installing from sdist, with a wheel that depends on the order of the files in the wheel), so it's probably only a little bit broken, but a little bit broken is still broken :-( On the other hand, python -m pip install -U pip works fine because it avoids the exe wrappers. There's a lot of scope for user confusion and frustration in all this. For standalone pip I've tended to recommend don't do that - manually uninstall and reinstall pip, or recreate your virtualenv. It's not nice, but it's effective. That sort of advice isn't going to be realistic for a pip bundled with CPython. Does anyone have any suggestions? Unless I misunderstand how the exe wrappers work (they're all the same code that looks for a .py file by the same name?) it may be easiest to somehow mark them as non-vital, such that failing to update them does not fail the installer. Maybe detect that it can't be overwritten, compare the contents/hash with the new one, and only fail if it's changed (with an instruction to use 'python -m...')? Spawning a separate process to do the install is probably no good, since you'd have to kill the original one which is going to break command line output. MoveFileEx (with its copy-on-reboot flag) is off the table, since it requires elevation and a reboot. But I think that's the only supported API for doing a deferred copy. If Windows was opening .exes with FILE_SHARE_DELETE then it would be possible to delete the exe and create a new one by the same name, but I doubt that will work and in any case could not be assumed to never change. So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed. The usual way to do this is just move the existing executable to pip.exe.deleteme or something, and then write out the new one. Then on every startup (or maybe some level of special case for just pip upgrades?) try to unlink *.deleteme. Not the simplest system ever, but it gets the job done. I accidentally only emailed Paul earlier, but why can't we upgrade the pip module with the exe and then replace the process (using something in the os.exec* family) with `python -m pip update-exe` which could then succeed since the OS isn't holding onto the exe file? I could be missing something entirely obvious since I haven't developed (directly) on or for Windows in at least 5 years. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping))
On Jul 14, 2013, at 10:31 AM, Ian Cordasco wrote: On Sun, Jul 14, 2013 at 1:12 PM, Noah Kantrowitz n...@coderanger.net wrote: On Jul 14, 2013, at 9:45 AM, Steve Dower wrote: From: Paul Moore On 13 July 2013 10:05, Paul Moore p.f.mo...@gmail.com wrote: How robust is the process of upgrading pip using itself? Specifically on Windows, where these things typically seem less reliable. OK, I just did some tests. On Windows, pip install -U pip FAILS. The reason for the failure is simple enough to explain - the pip.exe wrapper is held open by the OS while it's in use, so that the upgrade cannot replace it. The result is a failed upgrade and a partially installed new version of pip. In practice, the exe stubs are probably added fairly late in the install (at least when installing from sdist, with a wheel that depends on the order of the files in the wheel), so it's probably only a little bit broken, but a little bit broken is still broken :-( On the other hand, python -m pip install -U pip works fine because it avoids the exe wrappers. There's a lot of scope for user confusion and frustration in all this. For standalone pip I've tended to recommend don't do that - manually uninstall and reinstall pip, or recreate your virtualenv. It's not nice, but it's effective. That sort of advice isn't going to be realistic for a pip bundled with CPython. Does anyone have any suggestions? Unless I misunderstand how the exe wrappers work (they're all the same code that looks for a .py file by the same name?) it may be easiest to somehow mark them as non-vital, such that failing to update them does not fail the installer. Maybe detect that it can't be overwritten, compare the contents/hash with the new one, and only fail if it's changed (with an instruction to use 'python -m...')? Spawning a separate process to do the install is probably no good, since you'd have to kill the original one which is going to break command line output. MoveFileEx (with its copy-on-reboot flag) is off the table, since it requires elevation and a reboot. But I think that's the only supported API for doing a deferred copy. If Windows was opening .exes with FILE_SHARE_DELETE then it would be possible to delete the exe and create a new one by the same name, but I doubt that will work and in any case could not be assumed to never change. So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed. The usual way to do this is just move the existing executable to pip.exe.deleteme or something, and then write out the new one. Then on every startup (or maybe some level of special case for just pip upgrades?) try to unlink *.deleteme. Not the simplest system ever, but it gets the job done. I accidentally only emailed Paul earlier, but why can't we upgrade the pip module with the exe and then replace the process (using something in the os.exec* family) with `python -m pip update-exe` which could then succeed since the OS isn't holding onto the exe file? I could be missing something entirely obvious since I haven't developed (directly) on or for Windows in at least 5 years. Unfortunately windows doesn't actually offer the equivalent of a POSIX exec(). The various functions in os don't actually replace the current process, they just create a new one and terminate the old one. This means the controlling terminal would see the pip process as ended, so it makes showing output difficult at best. --Noah ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping))
On Jul 14, 2013, at 10:39 AM, Noah Kantrowitz wrote: On Jul 14, 2013, at 10:31 AM, Ian Cordasco wrote: On Sun, Jul 14, 2013 at 1:12 PM, Noah Kantrowitz n...@coderanger.net wrote: On Jul 14, 2013, at 9:45 AM, Steve Dower wrote: From: Paul Moore On 13 July 2013 10:05, Paul Moore p.f.mo...@gmail.com wrote: How robust is the process of upgrading pip using itself? Specifically on Windows, where these things typically seem less reliable. OK, I just did some tests. On Windows, pip install -U pip FAILS. The reason for the failure is simple enough to explain - the pip.exe wrapper is held open by the OS while it's in use, so that the upgrade cannot replace it. The result is a failed upgrade and a partially installed new version of pip. In practice, the exe stubs are probably added fairly late in the install (at least when installing from sdist, with a wheel that depends on the order of the files in the wheel), so it's probably only a little bit broken, but a little bit broken is still broken :-( On the other hand, python -m pip install -U pip works fine because it avoids the exe wrappers. There's a lot of scope for user confusion and frustration in all this. For standalone pip I've tended to recommend don't do that - manually uninstall and reinstall pip, or recreate your virtualenv. It's not nice, but it's effective. That sort of advice isn't going to be realistic for a pip bundled with CPython. Does anyone have any suggestions? Unless I misunderstand how the exe wrappers work (they're all the same code that looks for a .py file by the same name?) it may be easiest to somehow mark them as non-vital, such that failing to update them does not fail the installer. Maybe detect that it can't be overwritten, compare the contents/hash with the new one, and only fail if it's changed (with an instruction to use 'python -m...')? Spawning a separate process to do the install is probably no good, since you'd have to kill the original one which is going to break command line output. MoveFileEx (with its copy-on-reboot flag) is off the table, since it requires elevation and a reboot. But I think that's the only supported API for doing a deferred copy. If Windows was opening .exes with FILE_SHARE_DELETE then it would be possible to delete the exe and create a new one by the same name, but I doubt that will work and in any case could not be assumed to never change. So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed. The usual way to do this is just move the existing executable to pip.exe.deleteme or something, and then write out the new one. Then on every startup (or maybe some level of special case for just pip upgrades?) try to unlink *.deleteme. Not the simplest system ever, but it gets the job done. I accidentally only emailed Paul earlier, but why can't we upgrade the pip module with the exe and then replace the process (using something in the os.exec* family) with `python -m pip update-exe` which could then succeed since the OS isn't holding onto the exe file? I could be missing something entirely obvious since I haven't developed (directly) on or for Windows in at least 5 years. Unfortunately windows doesn't actually offer the equivalent of a POSIX exec(). The various functions in os don't actually replace the current process, they just create a new one and terminate the old one. This means the controlling terminal would see the pip process as ended, so it makes showing output difficult at best. Check that, maybe I'm wrong, does anyone know if the P_OVERLAY flag unlocks the original binary? /me drags out a windows VM … --Noah ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Another conversation starter - pip documentation in the Python docs
On 14 July 2013 18:02, Marcus Smith qwc...@gmail.com wrote: there's a section marked Installation Tutorial. Someone wanting to install packages should be able to get into that, once it actually has content : ) Yes, I see that. I'm not sure I like the up-front decide between user or system or virtualenv presentation, though. I'm working on something I prefer, see what you think when it's done. yea, don't take the current install tutorial TOC that seriously. The user/global/virtualenv content came from a recent merge Nick referred to above. We just need knowledgeable people to get in there and start working/changing OK... But remember that most users are consumers of packages, not creators of them. The packaging guide should reflect that Like Nick said, we can put the Installation Tutorial above the Packaging Tutorial. Neither tutorial should be considered Advanced IMO. The tutorials should be crisp and fast. Also, we *could* put a Quickstart above both tutorials, that just lists the frequent commands with one-liner descriptions, but we have to be really careful that it doesn't end up duping the tutorials, which are also intended to be quick as well. If not, let's just get a user-only starter page in the Python docs and leave the packaging guide as the more comprehensive documentation it can refer to). anything that results in feeling like the user guide needs a user-only starter page in Python docs means we're doing it wrong IMO : ) I admit the title itself concerns me: Python Packaging User Guide, like it should have the word Installation in it. Python Installation and Packaging User Guide? (its s long though) Marcus ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Another conversation starter - pip documentation in the Python docs
I believe Marcus was hoping to get back to that after pip 1.4 was out the door, but anyone on the PyPA list on BitBucket actually has full access to accept pull requests, etc. yes, the time has come to get this thing going or risk getting a yet another dead document rep, which it's starting to get. Marcus ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping))
On Jul 14, 2013, at 10:43 AM, Noah Kantrowitz wrote: On Jul 14, 2013, at 10:39 AM, Noah Kantrowitz wrote: On Jul 14, 2013, at 10:31 AM, Ian Cordasco wrote: On Sun, Jul 14, 2013 at 1:12 PM, Noah Kantrowitz n...@coderanger.net wrote: On Jul 14, 2013, at 9:45 AM, Steve Dower wrote: From: Paul Moore On 13 July 2013 10:05, Paul Moore p.f.mo...@gmail.com wrote: How robust is the process of upgrading pip using itself? Specifically on Windows, where these things typically seem less reliable. OK, I just did some tests. On Windows, pip install -U pip FAILS. The reason for the failure is simple enough to explain - the pip.exe wrapper is held open by the OS while it's in use, so that the upgrade cannot replace it. The result is a failed upgrade and a partially installed new version of pip. In practice, the exe stubs are probably added fairly late in the install (at least when installing from sdist, with a wheel that depends on the order of the files in the wheel), so it's probably only a little bit broken, but a little bit broken is still broken :-( On the other hand, python -m pip install -U pip works fine because it avoids the exe wrappers. There's a lot of scope for user confusion and frustration in all this. For standalone pip I've tended to recommend don't do that - manually uninstall and reinstall pip, or recreate your virtualenv. It's not nice, but it's effective. That sort of advice isn't going to be realistic for a pip bundled with CPython. Does anyone have any suggestions? Unless I misunderstand how the exe wrappers work (they're all the same code that looks for a .py file by the same name?) it may be easiest to somehow mark them as non-vital, such that failing to update them does not fail the installer. Maybe detect that it can't be overwritten, compare the contents/hash with the new one, and only fail if it's changed (with an instruction to use 'python -m...')? Spawning a separate process to do the install is probably no good, since you'd have to kill the original one which is going to break command line output. MoveFileEx (with its copy-on-reboot flag) is off the table, since it requires elevation and a reboot. But I think that's the only supported API for doing a deferred copy. If Windows was opening .exes with FILE_SHARE_DELETE then it would be possible to delete the exe and create a new one by the same name, but I doubt that will work and in any case could not be assumed to never change. So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed. The usual way to do this is just move the existing executable to pip.exe.deleteme or something, and then write out the new one. Then on every startup (or maybe some level of special case for just pip upgrades?) try to unlink *.deleteme. Not the simplest system ever, but it gets the job done. I accidentally only emailed Paul earlier, but why can't we upgrade the pip module with the exe and then replace the process (using something in the os.exec* family) with `python -m pip update-exe` which could then succeed since the OS isn't holding onto the exe file? I could be missing something entirely obvious since I haven't developed (directly) on or for Windows in at least 5 years. Unfortunately windows doesn't actually offer the equivalent of a POSIX exec(). The various functions in os don't actually replace the current process, they just create a new one and terminate the old one. This means the controlling terminal would see the pip process as ended, so it makes showing output difficult at best. Check that, maybe I'm wrong, does anyone know if the P_OVERLAY flag unlocks the original binary? /me drags out a windows VM … Ignore my ignoring, with os.execl command flow does return back to the controlling terminal process (the new process continues in the background) and with os.spawnl(os.P_OVERLAY, 'python-2') I just get a segfault on 3.3. Yay for not completely misremembering, boo for this being so complicated. --Noah signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping))
On Sun, Jul 14, 2013 at 2:10 PM, Noah Kantrowitz n...@coderanger.net wrote: On Jul 14, 2013, at 10:43 AM, Noah Kantrowitz wrote: On Jul 14, 2013, at 10:39 AM, Noah Kantrowitz wrote: On Jul 14, 2013, at 10:31 AM, Ian Cordasco wrote: On Sun, Jul 14, 2013 at 1:12 PM, Noah Kantrowitz n...@coderanger.net wrote: On Jul 14, 2013, at 9:45 AM, Steve Dower wrote: From: Paul Moore On 13 July 2013 10:05, Paul Moore p.f.mo...@gmail.com wrote: How robust is the process of upgrading pip using itself? Specifically on Windows, where these things typically seem less reliable. OK, I just did some tests. On Windows, pip install -U pip FAILS. The reason for the failure is simple enough to explain - the pip.exe wrapper is held open by the OS while it's in use, so that the upgrade cannot replace it. The result is a failed upgrade and a partially installed new version of pip. In practice, the exe stubs are probably added fairly late in the install (at least when installing from sdist, with a wheel that depends on the order of the files in the wheel), so it's probably only a little bit broken, but a little bit broken is still broken :-( On the other hand, python -m pip install -U pip works fine because it avoids the exe wrappers. There's a lot of scope for user confusion and frustration in all this. For standalone pip I've tended to recommend don't do that - manually uninstall and reinstall pip, or recreate your virtualenv. It's not nice, but it's effective. That sort of advice isn't going to be realistic for a pip bundled with CPython. Does anyone have any suggestions? Unless I misunderstand how the exe wrappers work (they're all the same code that looks for a .py file by the same name?) it may be easiest to somehow mark them as non-vital, such that failing to update them does not fail the installer. Maybe detect that it can't be overwritten, compare the contents/hash with the new one, and only fail if it's changed (with an instruction to use 'python -m...')? Spawning a separate process to do the install is probably no good, since you'd have to kill the original one which is going to break command line output. MoveFileEx (with its copy-on-reboot flag) is off the table, since it requires elevation and a reboot. But I think that's the only supported API for doing a deferred copy. If Windows was opening .exes with FILE_SHARE_DELETE then it would be possible to delete the exe and create a new one by the same name, but I doubt that will work and in any case could not be assumed to never change. So unless the exe wrapper is changing with each version, I think the best way of handling this is to not force them to be replaced when they have not changed. The usual way to do this is just move the existing executable to pip.exe.deleteme or something, and then write out the new one. Then on every startup (or maybe some level of special case for just pip upgrades?) try to unlink *.deleteme. Not the simplest system ever, but it gets the job done. I accidentally only emailed Paul earlier, but why can't we upgrade the pip module with the exe and then replace the process (using something in the os.exec* family) with `python -m pip update-exe` which could then succeed since the OS isn't holding onto the exe file? I could be missing something entirely obvious since I haven't developed (directly) on or for Windows in at least 5 years. Unfortunately windows doesn't actually offer the equivalent of a POSIX exec(). The various functions in os don't actually replace the current process, they just create a new one and terminate the old one. This means the controlling terminal would see the pip process as ended, so it makes showing output difficult at best. Check that, maybe I'm wrong, does anyone know if the P_OVERLAY flag unlocks the original binary? /me drags out a windows VM … Ignore my ignoring, with os.execl command flow does return back to the controlling terminal process (the new process continues in the background) and with os.spawnl(os.P_OVERLAY, 'python-2') I just get a segfault on 3.3. Yay for not completely misremembering, boo for this being so complicated. I expected I was wrong, but I appreciate you looking at it to be certain. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Another conversation starter - pip documentation in the Python docs
I admit the title itself concerns me: Python Packaging User Guide, like it should have the word Installation in it. Python Installation and Packaging User Guide? (its s long though) other ideas: Python Installation and Packaging Guide Python Installation and Distribution Guide ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] buildout/setuptools/distribute unhelpful error message (0.7.x issue?)
On 13/07/2013 17:37, Jason R. Coombs wrote: It looks like something is trying to install Setuptools 0.7.2, possibly with a temporary version of distribute or one that's not visible by default in your Python environment. That would have been buildout's bootstrap.py. Jim, is this fixed in the latest 1 and 2 bootstrap.pys? When you get that error message, I suggest you upgrade away from distribute. The problem is that this is the stable python-pkg-resources in Debian, so likely Ubuntu to. This problem has the potential to affect many many users so I hope a sensible story is being developed to deal with it... The easiest way to do this if you have distribute installed is to 'easy_install -U distribute', which will grab distribute 0.7.3 and install setuptools=0.7. What's the OS-packages plan for dealing with this? I would consider doing the above to an OS-installed package to be pretty anti-social... I hope that helps. Please report back if that doesn't get you going. I fixed it before you replied, but it was pretty horrific: aptitude remove python-pkg-resources aptitude remove python-pkg-resources python-bzrlib python-pygments aptitude remove python-pkg-resources bzr python-bzrlib python-pygments aptitude remove python-pkg-resources bzr bzr-svn python-docutils python-bzrlib python-pygments ...which removed a lot of packages. I'm in a position where I can manage that, but I'd imagine a civilian user could end up in a lot of trouble. Also, don't forget the weird and crappy setuptools versions that Apple will have baked in to the various Pythons that ship with the various Mac OS's. While this merge is a good thing, it's causing a lot of pain from what I've heard and experienced. Chris Regards, Jason -Original Message- From: Distutils-SIG [mailto:distutils-sig- bounces+jaraco=jaraco@python.org] On Behalf Of Chris Withers Sent: Sunday, 07 July, 2013 03:09 To: distutils sig Subject: [Distutils] buildout/setuptools/distribute unhelpful error message (0.7.x issue?) Hi All, What is this exception trying to tell me? Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools- 0.7.2.tar.gz Extracting in /tmp/tmpJNVsOY Now working in /tmp/tmpJNVsOY/setuptools-0.7.2 Building a Setuptools egg in /tmp/tmpBLZGeg /tmp/tmpBLZGeg/setuptools-0.7.2-py2.6.egg Traceback (most recent call last): File bootstrap.py, line 91, in module pkg_resources.working_set.add_entry(path) File /usr/lib/python2.6/dist-packages/pkg_resources.py, line 451, in add_entry self.add(dist, entry, False) File /usr/lib/python2.6/dist-packages/pkg_resources.py, line 542, in add self._added_new(dist) File /usr/lib/python2.6/dist-packages/pkg_resources.py, line 705, in _added_new callback(dist) File /usr/lib/python2.6/dist-packages/pkg_resources.py, line 2727, in lambda add_activation_listener(lambda dist: dist.activate()) File /usr/lib/python2.6/dist-packages/pkg_resources.py, line 2227, in activate self.insert_on(path) File /usr/lib/python2.6/dist-packages/pkg_resources.py, line 2328, in insert_on with distribute. Found one at %s % str(self.location)) ValueError: A 0.7-series setuptools cannot be installed with distribute. Found one at /tmp/tmpBLZGeg/setuptools-0.7.2-py2.6.egg I don't see any distribute in there, and I don't know where it might be... $ python2.6 Python 2.6.8 (unknown, Jan 29 2013, 10:05:44) [GCC 4.7.2] on linux2 Type help, copyright, credits or license for more information. import setuptools Traceback (most recent call last): File stdin, line 1, in module ImportError: No module named setuptools cheers, Chris -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig -- Simplistix - Content Management, Batch Processing Python Consulting - http://www.simplistix.co.uk ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping))
On 14 July 2013 18:06, Donald Stufft don...@stufft.io wrote: Wouldn't a .py file make the command `pip.py`` and not ``pip`` ? Not if .py is a registered extension. What I can't remember is whether it needs to be in PATHEXT (which it isn't by default). The big problem here is that the behaviour isn't very well documented (if at all) so the various command shells act subtly differently. That's why I want to test, and why it won't be a 5-minute job to do so... But the various replace the exe afterwards hacks sound awfully complicated to me - particularly as pip doesn't control the exes in the first place, they are part of the setuptools console script entry point infrastructure. My strong preference here is to remove the current use of setuptools entry points, simply because I don't think the problem is solvable while pip doesn't control the exe management at all. That's a non-trivial change, but longer term maybe the best. Question for Nick, Brett and any other core devs around: Would python-dev be willing to include in the stdlib some sort of package for managing exe-wrappers? I don't really want pip to manage exe wrappers any more than I like setuptools doing so. Maybe the existing launcher can somehow double up in that role? Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Another conversation starter - pip documentation in the Python docs
On 14 July 2013 18:49, Marcus Smith qwc...@gmail.com wrote: I admit the title itself concerns me: Python Packaging User Guide, like it should have the word Installation in it. Python Installation and Packaging User Guide? (its s long though) I like Package Management Guide. Strong hints of managing existing 3rd party packages, which is good because it's what the majority want, but general enough to allow for including managing the build distribution of your own packages. BTW, I'm sick and tired of agonising every time I use the word package over whether I should be correct and use distribution. Can the guide just come right out and bless the occasionally-ambiguous but commonly-used dual nature of the word package? If not, can people start actually *using* distribution consistently for what pip downloads and installs, so I can find a few more examples for me to copy when I end up with awkward phrases like distributing your distribution...? (You'd never believe English was my native language, would you? :-)) Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Another conversation starter - pip documentation in the Python docs
BTW, I'm sick and tired of agonising every time I use the word package over whether I should be correct and use distribution. Can the guide just come right out and bless the occasionally-ambiguous but commonly-used dual nature of the word package? If not, can people start actually *using* distribution consistently for what pip downloads and installs, so I can find a few more examples for me to copy when I end up with awkward phrases like distributing your distribution...? (You'd never believe English was my native language, would you? :-)) I hear you. I feel the same agony. It think we should use the word Distribution, but it's hard to compete when PyPI uses Package. Nick, what do we do? : ) Marcus ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Another conversation starter - pip documentation in the Python docs
Donald: thoughts on changing our use of Package on pypi to Distribution? (except for the title of course) If we're not going to do that, we should explain and bless the double use of Package and drop using Distribution in any docs. Our fundamental concepts shouldn't be confusing and conflicted. On Sun, Jul 14, 2013 at 1:05 PM, Marcus Smith qwc...@gmail.com wrote: BTW, I'm sick and tired of agonising every time I use the word package over whether I should be correct and use distribution. Can the guide just come right out and bless the occasionally-ambiguous but commonly-used dual nature of the word package? If not, can people start actually *using* distribution consistently for what pip downloads and installs, so I can find a few more examples for me to copy when I end up with awkward phrases like distributing your distribution...? (You'd never believe English was my native language, would you? :-)) I hear you. I feel the same agony. It think we should use the word Distribution, but it's hard to compete when PyPI uses Package. Nick, what do we do? : ) Marcus ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Another conversation starter - pip documentation in the Python docs
On Jul 14, 2013, at 5:22 PM, Marcus Smith qwc...@gmail.com wrote: Donald: thoughts on changing our use of Package on pypi to Distribution? (except for the title of course) If we're not going to do that, we should explain and bless the double use of Package and drop using Distribution in any docs. Our fundamental concepts shouldn't be confusing and conflicted. On Sun, Jul 14, 2013 at 1:05 PM, Marcus Smith qwc...@gmail.com wrote: BTW, I'm sick and tired of agonising every time I use the word package over whether I should be correct and use distribution. Can the guide just come right out and bless the occasionally-ambiguous but commonly-used dual nature of the word package? If not, can people start actually *using* distribution consistently for what pip downloads and installs, so I can find a few more examples for me to copy when I end up with awkward phrases like distributing your distribution...? (You'd never believe English was my native language, would you? :-)) I hear you. I feel the same agony. It think we should use the word Distribution, but it's hard to compete when PyPI uses Package. Nick, what do we do? : ) Marcus ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig PyPI will eventually move to using the definitions as defined in PEP426 http://www.python.org/dev/peps/pep-0426/#supporting-definitions . - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Another conversation starter - pip documentation in the Python docs
no reason not to change Package to Distribution now, right? this (http://docs.python.org/2/install/index.html) already uses the Distribution word. which means changing sentences like There are currently 32660 packages here. to There are currently 32660 distributions here. and the main table listing would use the word Distribution On Sun, Jul 14, 2013 at 2:24 PM, Donald Stufft don...@stufft.io wrote: On Jul 14, 2013, at 5:22 PM, Marcus Smith qwc...@gmail.com wrote: Donald: thoughts on changing our use of Package on pypi to Distribution? (except for the title of course) If we're not going to do that, we should explain and bless the double use of Package and drop using Distribution in any docs. Our fundamental concepts shouldn't be confusing and conflicted. On Sun, Jul 14, 2013 at 1:05 PM, Marcus Smith qwc...@gmail.com wrote: BTW, I'm sick and tired of agonising every time I use the word package over whether I should be correct and use distribution. Can the guide just come right out and bless the occasionally-ambiguous but commonly-used dual nature of the word package? If not, can people start actually *using* distribution consistently for what pip downloads and installs, so I can find a few more examples for me to copy when I end up with awkward phrases like distributing your distribution...? (You'd never believe English was my native language, would you? :-)) I hear you. I feel the same agony. It think we should use the word Distribution, but it's hard to compete when PyPI uses Package. Nick, what do we do? : ) Marcus ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig PyPI will eventually move to using the definitions as defined in PEP426 http://www.python.org/dev/peps/pep-0426/#supporting-definitions . - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Another conversation starter - pip documentation in the Python docs
On Jul 14, 2013, at 5:29 PM, Marcus Smith qwc...@gmail.com wrote: no reason not to change Package to Distribution now, right? this (http://docs.python.org/2/install/index.html) already uses the Distribution word. which means changing sentences like There are currently 32660 packages here. to There are currently 32660 distributions here. and the main table listing would use the word Distribution That's 32600 projects, There are almost 200k distributions. And the main table listing would be Releases or Project Releases. On Sun, Jul 14, 2013 at 2:24 PM, Donald Stufft don...@stufft.io wrote: On Jul 14, 2013, at 5:22 PM, Marcus Smith qwc...@gmail.com wrote: Donald: thoughts on changing our use of Package on pypi to Distribution? (except for the title of course) If we're not going to do that, we should explain and bless the double use of Package and drop using Distribution in any docs. Our fundamental concepts shouldn't be confusing and conflicted. On Sun, Jul 14, 2013 at 1:05 PM, Marcus Smith qwc...@gmail.com wrote: BTW, I'm sick and tired of agonising every time I use the word package over whether I should be correct and use distribution. Can the guide just come right out and bless the occasionally-ambiguous but commonly-used dual nature of the word package? If not, can people start actually *using* distribution consistently for what pip downloads and installs, so I can find a few more examples for me to copy when I end up with awkward phrases like distributing your distribution...? (You'd never believe English was my native language, would you? :-)) I hear you. I feel the same agony. It think we should use the word Distribution, but it's hard to compete when PyPI uses Package. Nick, what do we do? : ) Marcus ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig PyPI will eventually move to using the definitions as defined in PEP426 http://www.python.org/dev/peps/pep-0426/#supporting-definitions . - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Another conversation starter - pip documentation in the Python docs
although some of the uses could be argued as needing Project e.g Package documentation is hosted on its own domain On Sun, Jul 14, 2013 at 2:29 PM, Marcus Smith qwc...@gmail.com wrote: no reason not to change Package to Distribution now, right? this (http://docs.python.org/2/install/index.html) already uses the Distribution word. which means changing sentences like There are currently 32660 packages here. to There are currently 32660 distributions here. and the main table listing would use the word Distribution On Sun, Jul 14, 2013 at 2:24 PM, Donald Stufft don...@stufft.io wrote: On Jul 14, 2013, at 5:22 PM, Marcus Smith qwc...@gmail.com wrote: Donald: thoughts on changing our use of Package on pypi to Distribution? (except for the title of course) If we're not going to do that, we should explain and bless the double use of Package and drop using Distribution in any docs. Our fundamental concepts shouldn't be confusing and conflicted. On Sun, Jul 14, 2013 at 1:05 PM, Marcus Smith qwc...@gmail.com wrote: BTW, I'm sick and tired of agonising every time I use the word package over whether I should be correct and use distribution. Can the guide just come right out and bless the occasionally-ambiguous but commonly-used dual nature of the word package? If not, can people start actually *using* distribution consistently for what pip downloads and installs, so I can find a few more examples for me to copy when I end up with awkward phrases like distributing your distribution...? (You'd never believe English was my native language, would you? :-)) I hear you. I feel the same agony. It think we should use the word Distribution, but it's hard to compete when PyPI uses Package. Nick, what do we do? : ) Marcus ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig PyPI will eventually move to using the definitions as defined in PEP426 http://www.python.org/dev/peps/pep-0426/#supporting-definitions . - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Another conversation starter - pip documentation in the Python docs
That's 32600 projects, There are almost 200k distributions. And the main table listing would be Releases or Project Releases. ok, got it. anything I do from here on will follow this. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Another conversation starter - pip documentation in the Python docs
On 14 July 2013 22:36, Marcus Smith qwc...@gmail.com wrote: That's 32600 projects, There are almost 200k distributions. And the main table listing would be Releases or Project Releases. ok, got it. anything I do from here on will follow this. That's much better. I apologise, I hadn't noticed that PEP 426 included terminology definitions. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping))
On 15 Jul 2013 05:44, Paul Moore p.f.mo...@gmail.com wrote: On 14 July 2013 18:06, Donald Stufft don...@stufft.io wrote: Wouldn't a .py file make the command `pip.py`` and not ``pip`` ? Not if .py is a registered extension. What I can't remember is whether it needs to be in PATHEXT (which it isn't by default). The big problem here is that the behaviour isn't very well documented (if at all) so the various command shells act subtly differently. That's why I want to test, and why it won't be a 5-minute job to do so... But the various replace the exe afterwards hacks sound awfully complicated to me - particularly as pip doesn't control the exes in the first place, they are part of the setuptools console script entry point infrastructure. My strong preference here is to remove the current use of setuptools entry points, simply because I don't think the problem is solvable while pip doesn't control the exe management at all. That's a non-trivial change, but longer term maybe the best. Question for Nick, Brett and any other core devs around: Would python-dev be willing to include in the stdlib some sort of package for managing exe-wrappers? I don't really want pip to manage exe wrappers any more than I like setuptools doing so. Maybe the existing launcher can somehow double up in that role? Not sure it fits the launcher, but having something along those lines in the stdlib makes sense (especially in the context of a pip bundling PEP). Another option we may want to consider is an actual msi installer for pip (I'm not sure that would actually help, but it's worth looking into), as well as investigating what other self-updating Windows apps (like Firefox) do to handle this problem. Cheers, Nick. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping))
On Jul 14, 2013, at 3:06 PM, Nick Coghlan wrote: On 15 Jul 2013 05:44, Paul Moore p.f.mo...@gmail.com wrote: On 14 July 2013 18:06, Donald Stufft don...@stufft.io wrote: Wouldn't a .py file make the command `pip.py`` and not ``pip`` ? Not if .py is a registered extension. What I can't remember is whether it needs to be in PATHEXT (which it isn't by default). The big problem here is that the behaviour isn't very well documented (if at all) so the various command shells act subtly differently. That's why I want to test, and why it won't be a 5-minute job to do so... But the various replace the exe afterwards hacks sound awfully complicated to me - particularly as pip doesn't control the exes in the first place, they are part of the setuptools console script entry point infrastructure. My strong preference here is to remove the current use of setuptools entry points, simply because I don't think the problem is solvable while pip doesn't control the exe management at all. That's a non-trivial change, but longer term maybe the best. Question for Nick, Brett and any other core devs around: Would python-dev be willing to include in the stdlib some sort of package for managing exe-wrappers? I don't really want pip to manage exe wrappers any more than I like setuptools doing so. Maybe the existing launcher can somehow double up in that role? Not sure it fits the launcher, but having something along those lines in the stdlib makes sense (especially in the context of a pip bundling PEP). Another option we may want to consider is an actual msi installer for pip (I'm not sure that would actually help, but it's worth looking into), as well as investigating what other self-updating Windows apps (like Firefox) do to handle this problem. They do the exec a helper executable that replaces the original approach, which works fine for non-console apps since there isn't the problem of the shell getting confused :-/ --Noah ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Executable wrappers and upgrading pip (Was: Current status of PEP 439 (pip boostrapping))
On 14 July 2013 23:09, Noah Kantrowitz n...@coderanger.net wrote: Another option we may want to consider is an actual msi installer for pip (I'm not sure that would actually help, but it's worth looking into), as well as investigating what other self-updating Windows apps (like Firefox) do to handle this problem. They do the exec a helper executable that replaces the original approach, which works fine for non-console apps since there isn't the problem of the shell getting confused :-/ Generally, I don't think that going down the route of MSIs is a good move. They aren't a good fit for this problem. Apart from anything else, they won't support installing into a virtualenv. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Another conversation starter - pip documentation in the Python docs
On 15 Jul 2013 07:45, Paul Moore p.f.mo...@gmail.com wrote: On 14 July 2013 22:36, Marcus Smith qwc...@gmail.com wrote: That's 32600 projects, There are almost 200k distributions. And the main table listing would be Releases or Project Releases. ok, got it. anything I do from here on will follow this. That's much better. I apologise, I hadn't noticed that PEP 426 included terminology definitions. They were necessary for me to keep them straight in my own head. The original set were a bit odd, but I think the project/release/distribution/archive split has ended up in a reasonable place after a few iterations (it's still open to revisions if anything seems too awkward, though). I don't think the ambiguity of package will ever go away entirely, but we can justify that to some degree by noting that distributions mainly serve to get Python packages and modules installed and available for import. Cheers, Nick. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
