Re: [Distutils] Comments on PEP 426

[Tres Seaver]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 08/30/2013 09:57 PM, Daniel Holth wrote:
> One of the most important packaging insights to understand is that 
> distutils is in fact worse than setuptools. It is badly outdated, 
> poorly extensible, and too complex to ever re-implement in a 
> compatible way. It is not better before the monkey patching. In a 
> perfect world it should be removed from the standard library too 
> except that removing distutils would be too impractical.



> If you can believe that distutils, not setuptools, is the problem we 
> are trying to recover from, then the monkeypatching strategy may make 
> more sense.

Amen!


Tres.
- -- 
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlIhYiIACgkQ+gerLs4ltQ52SwCcDd/Nt8ETGsLIpxbiksYiayfO
59gAn1VxLMvZq4akWuzJsdCNUlBUUxGh
=DFtl
-END PGP SIGNATURE-

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Retiring mirror g.pypi.python.org

[Noah Kantrowitz]

DNS and LB config have been updated and should take effect over the next day or 
so.

--Noah

On Aug 30, 2013, at 11:01 AM, Ralph Bean wrote:

> The admins at g.pypi.python.org/mirror.rit.edu have decided they no
> longer have the resources to maintain their mirror.  They've already
> taken down the content and would like to be removed from any indexes
> out there (such as www.pypi-mirrors.org).
> 
> -Ralph
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> http://mail.python.org/mailman/listinfo/distutils-sig



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Comments on PEP 426

[Daniel Holth]

On Fri, Aug 30, 2013 at 7:18 PM, Antoine Pitrou  wrote:
> Nick Coghlan  gmail.com> writes:
>>
>> On 30 Aug 2013 17:23, "Paul Moore"  gmail.com> wrote:
>> >
>> > On 30 August 2013 00:08, Nick Coghlan  gmail.com> wrote:
>> >>
>> >> We also need to officially bless pip's trick of forcing the use of
> setuptools for distutils based setup.py files.
>> >
>> >
>> > Do we? What does official blessing imply? We've managed for years
> without the trick being "official"...
>> >
>> > The main reason it is currently used is to allow setup.py install to
> specify --record, so that we can get the list of installed files. If
> distutils added a --record flag, for example, I don't believe we'd need the
> hack at all. (Obviously, we'd still need setuptools so we could use wheel to
> build wheels, but that's somewhat different as it's a new feature). Maybe a
> small distutils patch is better than blessing setuptools here?
>>
>> A distutils patch won't help with Python 2.7 or 3.3. The purpose of
> blessing the substitution is to decouple the update cycle of the build
> system from the update cycle of the standard library.
>
> It sounds like a nasty hack. What you call "substitution" is actually monkey
> patching,
> right? (edit: apparently it is pre-loading setuptools, which probably does
> the monkey patching by itself)
>
> This is crazy. We removed packaging from the stdlib because it wasn't
> "good enough", and now we would "bless the substitution" (aka silent runtime
> monkeypatching) of distutils with setuptools, a third-party library whose 
> stdlib
> inclusion has always been widely refused by the community (for many reasons)?
>
> pip can do what it likes, but blessing such behaviour officially sounds
> completely
> backwards.
>
> Regards
>
> Antoine.

One of the most important packaging insights to understand is that
distutils is in fact worse than setuptools. It is badly outdated,
poorly extensible, and too complex to ever re-implement in a
compatible way. It is not better before the monkey patching. In a
perfect world it should be removed from the standard library too
except that removing distutils would be too impractical.

The new strategy is to standardize just the install (how to install a
wheel binary package after it has already been built) and runtime for
eventual standard library inclusion. These are simple enough that they
can be documented and re-implemented adequately. Distutils and
setuptools will both be equally discouraged as legacy build tools.

*Hopefully* a dominant 80% simpler-use-cases distutils replacement
will emerge along with a more complicated 20% "scipy" distutils
replacement but neither will necessarily need to be in the standard
library; more complicated build+install tools that can deal with both
sdists and wheels will be able to grab the necessary build tool as
part of the build.

If you can believe that distutils, not setuptools, is the problem we
are trying to recover from, then the monkeypatching strategy may make
more sense.

We are also making setuptools optional *at run time* even for packages
that need to be built under setuptools.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Retiring mirror g.pypi.python.org

[Ralph Bean]

The admins at g.pypi.python.org/mirror.rit.edu have decided they no
longer have the resources to maintain their mirror.  They've already
taken down the content and would like to be removed from any indexes
out there (such as www.pypi-mirrors.org).

-Ralph
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP453 - Explicit bootstrapping of pip in Python installations

[Nick Coghlan]

On 31 Aug 2013 02:02, "Paul Moore"  wrote:
>
>
> On 30 August 2013 14:44, Donald Stufft  wrote:
>>
>> The Windows and OSX installers distributed by Python.org will
automatically
>> attempt to run ``python -m getpip`` by default however the ``make
install``
>> and ``make altinstall`` commands of the source distribution will not.
>
>
> Presumably the uninstaller components of these installers should
similarly uninstall pip before uninstalling Python. Would something like a
"python -m getpip uninstall" command be worthwhile to support this? (That's
probably a question for the authors of the installers to answer, as I don't
know if such a command would be needed - I suspect that the Windows MSI
installer just records what files it installs and uninstalls them, so it
may not be needed there).
>
> But otherwise +1 on this PEP.

Yeah, a command to uninstall pip and everything it installed would be
desirable. I wouldn't consider it a *blocker* for inclusion (since anyone
bootstrapping pip manually already has this problem), but it would be a
nice one to solve (and having it on getpip neatly avoids the issues with
pip uninstalling itself on Windows).

For the other questions people asked:

* yes, pyvenv should gain a "--without-pip" option

* the only change to the existing get-pip.py bootstrap script should be to
check for "getpip" and use it if available, and otherwise continue on with
the legacy bootstrap mechanism.

* I agree with Donald that getpip should be an ordinary standard library
Python module

* To allow releases to be recreated exactly from the source tarball, I
agree with Donald that we need to actually include the relevant wheel files
inside the CPython source tree. Retrieving them automatically at build time
would be nice, but that unfortunately creates ugly version reproducibility
issues, and also requires that the build process be patched before it can
run on a trusted build server with no network access.

* If we have to bundle setuptools as well, so be it. However, we should
document that it is *not* guaranteed to be available, so projects that need
it should declare the appropriate dependencies and mention "pip install
setuptools" in the appropriate places to ensure it is available

* a "--wheel-only" option for both getpip and pyvenv might be interesting,
if pip gets to a point where it only needs setuptools for source builds and
not installing from wheels. However, for now, I think the PEP should just
assume setuptools will be bundled and installed as a pip dependency. I'm OK
with that - Guido was amenable to adding setuptools to the standard library
back in 2.5 as a case of practicality beating purity. (In retrospect, that
would have just meant setuptools suffered from the same "can't readily be
updated in existing Python versions" problem as distutils, so it's probably
a net positive that PJE eventually decided not to continue with the idea)

Cheers,
Nick.

>
> Paul
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Comments on PEP 426

[Antoine Pitrou]

Nick Coghlan  gmail.com> writes:
> 
> On 30 Aug 2013 17:23, "Paul Moore"  gmail.com> wrote:
> >
> > On 30 August 2013 00:08, Nick Coghlan  gmail.com> wrote:
> >>
> >> We also need to officially bless pip's trick of forcing the use of
setuptools for distutils based setup.py files.
> >
> >
> > Do we? What does official blessing imply? We've managed for years
without the trick being "official"...
> >
> > The main reason it is currently used is to allow setup.py install to
specify --record, so that we can get the list of installed files. If
distutils added a --record flag, for example, I don't believe we'd need the
hack at all. (Obviously, we'd still need setuptools so we could use wheel to
build wheels, but that's somewhat different as it's a new feature). Maybe a
small distutils patch is better than blessing setuptools here?
> 
> A distutils patch won't help with Python 2.7 or 3.3. The purpose of
blessing the substitution is to decouple the update cycle of the build
system from the update cycle of the standard library.

It sounds like a nasty hack. What you call "substitution" is actually monkey
patching, 
right? (edit: apparently it is pre-loading setuptools, which probably does
the monkey patching by itself)

This is crazy. We removed packaging from the stdlib because it wasn't
"good enough", and now we would "bless the substitution" (aka silent runtime 
monkeypatching) of distutils with setuptools, a third-party library whose stdlib
inclusion has always been widely refused by the community (for many reasons)?

pip can do what it likes, but blessing such behaviour officially sounds
completely 
backwards.

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP453 - Explicit bootstrapping of pip in Python installations

[Brett Cannon]

On Fri, Aug 30, 2013 at 9:44 AM, Donald Stufft  wrote:

> [SNIP]
> Pre-installation
> 
>
> During the installation of Python from Python.org ``python -m getpip``
> should
> be executed. Leaving people using the Windows or OSX installers with a
> working
> copy of pip once the installation has completed.


"should be executed, leaving".


> The exact method of this is
> left up to the maintainers of the installers however if the bootstrapping
> is
> optional it should be opt out rather than opt in.
>

"installers, however"
"opt-in", "opt-out"


>
> The Windows and OSX installers distributed by Python.org will automatically
> attempt to run ``python -m getpip`` by default however the ``make install``
> and ``make altinstall`` commands of the source distribution will not.
>

Is the plan to leave getpip entirely out of the source distribution or to
have it checked into hg.python.org/cpython?


>
> Keeping the pip bootstrapping as a separate step for make based
> installations should minimize the changes CPython redistributors need to
> make to their build processes. Avoiding the layer of indirection through
> make for the getpip invocation also ensures those installing from a custom
> source build can easily force an offline installation of pip, install it
> from a private index server, or skip installing pip entirely.
>
>
> Python Virtual Environments
> ===
>
> Python 3.3 included a standard library approach to virtual Python
> environments
> through the ``venv`` module. Since it's release it has become clear that
> very
> few users have been willing to use this feature in part due to the lack of
> an installer present by default inside of the virtual environment. They
> have
> instead opted to continue using the ``virtualenv`` package which *does*
> include
> pip installed by default.
>
> To make the ``venv`` more useful to users it will be modified to issue the
> pip bootstrap by default inside of the new environment while creating it.
> This
> will allow people the same convenience inside of the virtual environment as
> this PEP provides outside of it as well as bringing the ``venv`` module
> closer
> to feature parity with the external ``virtualenv`` package making it a more
> suitable replacement.
>

What about a --without-pip option?


>
>
> Recommendations for Downstream Distributors
> ===
>
> A common source of Python installations are through downstream distributors
> such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_,
> OSX
> package managers [#homebrew]_, or python specific tools [#conda]_. In
> order to
> provide a consistent, user friendly experience to all users of Python
> regardless of how they attained Python this PEP recommends and asks that
> downstream distributors:
>
> * Ensure that whenever Python is installed pip is also installed.
>
>   * This may take the form of separate with dependencies on each either so
> that
> installing the python package installs the pip package and installing
> the
> pip package installs the Python package.
>
> * Do not remove the bundled copy of pip.
>
>   * This is required for offline installation of pip into a virtual
> environment.
>   * This is similar to the existing ``virtualenv`` package for which many
> downstream distributors have already made exception to the common
> "debundling" policy.
>   * This does mean that if ``pip`` needs to be updated due to a security
> issue, so does the bundled version in the ``getpip`` bootstrap module
>
> * Migrate build systems to utilize `pip`_ and `Wheel`_ instead of directly
>   using ``setup.py``.
>
>   * This will ensure that downstream packages can utilize the new formats
> which
> will not have a ``setup.py`` easier.
>
> * Ensure that all features of this PEP continue to work with any
> modifications
>   made.
>
>   * Online installation of the latest version of pip into a global or
> virtual
> python environment using ``python -m getpip``.
>   * Offline installation of the bundled version of pip into a global or
> virtual
> python environment using ``python -m getpip``.
>   * ``pip install --upgrade pip`` in a global installation should not
> affect
> any already created virtual environments.
>   * ``pip install --upgrade pip`` in a virtual environment should not
> affect
> the global installation.
>
>
> Policies & Governance
> =
>
> The maintainers of the bundled software and the CPython core team will work
> together in order to address the needs of both. The bundled software will
> still
> remain external to CPython and this PEP does not include CPython subsuming
> the
> responsibilities or decisions of the bundled software. This PEP aims to
> decrease the burden on end users wanting to use third party packages and
> the
> decisions inside it are pragmatic ones that represent the trust that the
> Python community has placed in the authors and ma

Re: [Distutils] PEP453 - Explicit bootstrapping of pip in Python installations

[Marcus Smith]

There's no mention of setuptools.
I guess the handling of that dependency (or not handling it up front, if
pip is refactored such that setuptools just becomes a dependency when
building) is considered an implementation detail of get-pip?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP453 - Explicit bootstrapping of pip in Python installations

[Marcus Smith]

> This should specify if it is ever expected to be kept in the CPython repo
> (and thus distributed in the source tarball) or if it will simply be
> bundled in the installers (and thus never included with the source).
>
> Nick may say differently (he's much more "in tune" with what files are
> owned by which processes) but I'd expect that the ``getpip`` module will be
> included as a normal part of the standard library and with it a tarball (or
> a Wheel) containing the sources of the bundled software.
>

will getpip also replace the current role of "get-pip.py" (available in
pip's master branch), i.e. will getpip be available for download and be
mentioned in the pip install instructions for python's that don't have it?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Distributable binary with dependencies

[PJ Eby]

On Mon, Aug 26, 2013 at 2:25 PM, bharath ravi kumar
 wrote:
> Carl, Eby,
>
> Thanks for taking time to suggest various alternatives. Considering that the
> deployment hosts are identical in every as[ect, the approach of moving
> virtualenv's with packages pip-installed at build time appears the simplest,
> low-overhead  approach that can be implemented without hacking the
> environment or resorting to custom scripts. I'll go ahead with that  option.

What hacking the environment or custom scripts?

I'm confused, because AFAIK there are actually more steps to
pip-install a virtualenv and copy it to different machines, than there
are involved in using easy_install to create a portable installation.
In both cases, you end up with a directory to archive and copy, so the
only difference is in the commands used to build that directory, and
the layout of the directory afterwards.

Perhaps you misunderstood my post as meaning that you had to run
easy_install on the target system?

(I don't have any particular stake in what you do for your own system,
but I'm curious, both for the future reference of folks reading this
thread by way of Googling this question, and in case there is
something for me to learn or that I'm mistaken about, in relation
either to pip/virtualenv or your use case.  And certainly if you are
more familiar with pip+virtualenv, that would actually be sufficient
reason in this case to use it.  But I'd prefer future readers of this
thread not to be under an erroneous impression that easy_install
involves more steps, scripts, or environment changes in order to
implement this use case.  Thanks.)
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP453 - Explicit bootstrapping of pip in Python installations

[Donald Stufft]

On Aug 30, 2013, at 12:32 PM, Marcus Smith  wrote:

> 
> 
> > This should specify if it is ever expected to be kept in the CPython repo 
> > (and thus distributed in the source tarball) or if it will simply be 
> > bundled in the installers (and thus never included with the source).
> 
> Nick may say differently (he's much more "in tune" with what files are owned 
> by which processes) but I'd expect that the ``getpip`` module will be 
> included as a normal part of the standard library and with it a tarball (or a 
> Wheel) containing the sources of the bundled software.
> 
> will getpip also replace the current role of "get-pip.py" (available in pip's 
> master branch), i.e. will getpip be available for download and be mentioned 
> in the pip install instructions for python's that don't have it?
> 

Pip will probably need to maintain get-pip.py for any Python version it 
supports that Doesn't have the getpip module.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP453 - Explicit bootstrapping of pip in Python installations

[Donald Stufft]

On Aug 30, 2013, at 12:12 PM, Marcus Smith  wrote:

> ok, got it.  so, theoretically if the pip/setuptools relationship stays what 
> it is at the moment, get-pip would end up including a bundled setuptools to 
> fulfill the "Not all users will have network access to PyPI"

Yes.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP453 - Explicit bootstrapping of pip in Python installations

[Marcus Smith]

ok, got it.  so, theoretically if the pip/setuptools relationship stays
what it is at the moment, get-pip would end up including a bundled
setuptools to fulfill the "Not all users will have network access to PyPI"


On Fri, Aug 30, 2013 at 9:07 AM, Donald Stufft  wrote:

>
> On Aug 30, 2013, at 12:04 PM, Marcus Smith  wrote:
>
> > There's no mention of setuptools.
> > I guess the handling of that dependency (or not handling it up front, if
> pip is refactored such that setuptools just becomes a dependency when
> building) is considered an implementation detail of get-pip?
>
>
> It's not mentioned by name but instead it's said:
>
> An additional module called ``getpip`` will be added to the standard
> library
> whose purpose is to install pip and any of its dependencies into the
> appropriate location (most commonly site-packages).
>
> Setuptools would be the "and any of its dependencies". I want to make
> setuptools not depend on setuptools but I added that as an escape clause
> incase that proves to be hard to actually do. The PEP itself specifies only
> pip because I'm considering setuptools an implementation detail of pip (one
> that can be replaced eventually or removed from it's "special" status).
>
> -
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP453 - Explicit bootstrapping of pip in Python installations

[Donald Stufft]

On Aug 30, 2013, at 11:39 AM, Brett Cannon  wrote:

> What about a --without-pip option?

Added this and the grammar mistakes you noted (I suck at this english thing~).

> This should specify if it is ever expected to be kept in the CPython repo 
> (and thus distributed in the source tarball) or if it will simply be bundled 
> in the installers (and thus never included with the source).

Nick may say differently (he's much more "in tune" with what files are owned by 
which processes) but I'd expect that the ``getpip`` module will be included as 
a normal part of the standard library and with it a tarball (or a Wheel) 
containing the sources of the bundled software.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP453 - Explicit bootstrapping of pip in Python installations

[Paul Moore]

On 30 August 2013 14:44, Donald Stufft  wrote:

> The Windows and OSX installers distributed by Python.org will automatically
> attempt to run ``python -m getpip`` by default however the ``make install``
> and ``make altinstall`` commands of the source distribution will not.
>

Presumably the uninstaller components of these installers should similarly
uninstall pip before uninstalling Python. Would something like a "python -m
getpip uninstall" command be worthwhile to support this? (That's probably a
question for the authors of the installers to answer, as I don't know if
such a command would be needed - I suspect that the Windows MSI installer
just records what files it installs and uninstalls them, so it may not be
needed there).

But otherwise +1 on this PEP.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP453 - Explicit bootstrapping of pip in Python installations

[Donald Stufft]

On Aug 30, 2013, at 12:04 PM, Marcus Smith  wrote:

> There's no mention of setuptools. 
> I guess the handling of that dependency (or not handling it up front, if pip 
> is refactored such that setuptools just becomes a dependency when building) 
> is considered an implementation detail of get-pip?


It's not mentioned by name but instead it's said:

An additional module called ``getpip`` will be added to the standard library
whose purpose is to install pip and any of its dependencies into the
appropriate location (most commonly site-packages).

Setuptools would be the "and any of its dependencies". I want to make 
setuptools not depend on setuptools but I added that as an escape clause incase 
that proves to be hard to actually do. The PEP itself specifies only pip 
because I'm considering setuptools an implementation detail of pip (one that 
can be replaced eventually or removed from it's "special" status).

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP453 - Explicit bootstrapping of pip in Python installations

[Donald Stufft]

On Aug 30, 2013, at 9:44 AM, Donald Stufft  wrote:

> [snip]

Available online at http://www.python.org/dev/peps/pep-0453/

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] PEP453 - Explicit bootstrapping of pip in Python installations

[Donald Stufft]

Abstract


This PEP proposes the inclusion of a method for explicitly bootstrapping
`pip`_ as the default package manager for Python. It also proposes that the
distributions of Python available via Python.org will automatically run this
explicit bootstrapping method and a recommendation to third party
redistributors of Python to also provide pip by default (in a way reasonable
for their distributions).

This PEP does *not* propose the inclusion of pip itself in the standard
library.


Proposal


This PEP proposes the inclusion of a ``getpip`` bootstrapping module in
Python 3.4, as well as in the upcoming maintenance releases of Python 2.7
and Python 3.3.


Rationale
=

Installing a third party package into a freshly installed Python requires first
installing the package manager. This requires users ahead of time to know what
the package manager is, where to get them from, and how to install them. The
effect of this is that these external projects are required to either blindly
assume the user already has the package manager installed, needs to duplicate
the instructions and tell their users how to install the package manager, or
completely forgo the use of dependencies to ease installation concerns for
their users.

All of the available options have their own drawbacks.

If a project simply assumes a user already has the tooling then they get a
confusing error message when the installation command doesn't work. Some
operating may ease this pain by providing a global hook that looks for commands
that don't exist and suggest an OS package they can install to make the command
work.

If a project chooses to duplicate the installation instructions and tell their
users how to install the package manager before telling them how to install
their own project then whenever these instructions need updates they need
updating by every project that has duplicated them. This will inevitably not
happen in every case leaving many different instructions on how to install it
many of them broken or less than optimal. These additional instructions might
also confuse users who try to install the package manager a second time
thinking that it's part of the instructions of installing the project.

The problem of stale instructions can be alleviated by referencing `pip's
own bootstrapping instructions
`__, but the user
experience involved still isn't good (especially on Windows, where
downloading and running a Python script with the default OS configuration is
significantly more painful than downloading and running a binary executable
or installer).

The projects that have decided to forgo dependencies all together are forced
to either duplicate the efforts of other projects by inventing their own
solutions to problems or are required to simply include the other projects
in their own source trees. Both of these options present their own problems
either in duplicating maintenance work across the ecosystem or potentially
leaving users vulnerable to security issues because the included code or
duplicated efforts are not automatically updated when upstream releases a new
version.

By providing the package manager by default it will be easier for users trying
to install these third party packages as well as easier for the people
distributing them as they no longer need to pick the lesser evil. This will
become more important in the future as the Wheel_ package format does not have
a built in "installer" in the form of ``setup.py`` so users wishing to install
a Wheel package will need an installer even in the simple case.

Reducing the burden of actually installing a third party package should also
decrease the pressure to add every useful module to the standard library. This
will allow additions to the standard library to focus more on why Python should
have a particular tool out of the box instead of needing to use the difficulty
in installing a package as justification for inclusion.


Explicit Bootstrapping
==

An additional module called ``getpip`` will be added to the standard library
whose purpose is to install pip and any of its dependencies into the
appropriate location (most commonly site-packages). It will expose a single
callable named ``bootstrap()`` as well as offer direct execution via
``python -m getpip``. Options for installing it such as index server,
installation location (``--user``, ``--root``, etc) will also be available
to enable different installation schemes.

It is believed that users will want the most recent versions available to be
installed so that they can take advantage of the new advances in packaging.
Since any particular version of Python has a much longer staying power than
a version of pip in order to satisfy a user's desire to have the most recent
version the bootstrap will contact PyPI, find the latest version, download it,
and then install it. This process is security sensitive, difficult to get
rig

Re: [Distutils] Comments on PEP 426

[Oscar Benjamin]

On 30 August 2013 13:49, Daniel Holth  wrote:
> On Fri, Aug 30, 2013 at 7:54 AM, Oscar Benjamin  
> wrote:
>>
>> I just tried to install bento to test it out and:
>>
>> $ pip install bento
>> Downloading/unpacking bento
>>   Downloading bento-0.1.1.tar.gz (582kB): 582kB downloaded
>>   Running setup.py egg_info for package bento
>> Installing collected packages: bento
>>   Running setup.py install for bento
>>   Could not find .egg-info directory in install record for bento
>> Cleaning up...
>> Exception:
>> Traceback (most recent call last):
>>   File "Q:\tools\Python27\lib\site-packages\pip\basecommand.py", line
>> 134, in main
>> status = self.run(options, args)
>>   File "Q:\tools\Python27\lib\site-packages\pip\commands\install.py",
>> line 241, in run
>> requirement_set.install(install_options, global_options,
>> root=options.root_path)
>>   File "Q:\tools\Python27\lib\site-packages\pip\req.py", line 1298, in 
>> install
>> requirement.install(install_options, global_options, *args, **kwargs)
>>   File "Q:\tools\Python27\lib\site-packages\pip\req.py", line 668, in install
>> os.remove(record_filename)
>> WindowsError: [Error 32] The process cannot access the file because it
>> is being used by another process:
>> 'c:\\docume~1\\enojb\\locals~1\\temp\\pip-aae65s-record\\install-record.txt'
>>
>> Storing complete log in c:/Documents and Settings/enojb\pip\pip.log
>>
>> I tried deleting the mentioned file but just got the same error
>> message again. Is that a bento/pip/setuptools bug? I notice that the
>> bento docs don't mention pip on the installation page:
>> http://cournape.github.io/Bento/html/install.html
>>
>> Here's the appropriate version information:
>>
>> $ pip --version
>> pip 1.4.1 from q:\tools\python27\lib\site-packages (python 2.7)
>> $ python --version
>> Python 2.7.5
>> $ python -c 'import setuptools; print(setuptools.__version__)'
>> 1.1
>>
>> (I just very carefully updated pip/setuptools based on Paul's previous
>> instructions).
>>
>> The bento setup.py uses bento's own setup() command:
>> https://github.com/cournape/Bento/blob/master/setup.py
>
> It looks like you cannot install bento itself using pip on Windows. It
> might be a Windows bug "WindowsError: [Error 32] The process cannot
> access the file because it is being used by another process:". It's a
> little better on Linux (it gets installed) but I don't think Bento is
> really meant to be installed in this way.

I't's a bug in pip. The file in question is opened by pip a few lines
above. The particular code path is called because the else
logger.warn() clause gets triggered (i.e. where it says "## FIX ME"
:) )

f = open(record_filename)
for line in f:
line = line.strip()
if line.endswith('.egg-info'):
egg_info_dir = prepend_root(line)
break
else:
logger.warn('Could not find .egg-info directory in
install record for %s' % self)
## FIXME: put the record somewhere
## FIXME: should this be an error?
return
f.close()
new_lines = []
f = open(record_filename)
for line in f:
filename = line.strip()
if os.path.isdir(filename):
filename += os.path.sep

new_lines.append(make_path_relative(prepend_root(filename),
egg_info_dir))
f.close()
f = open(os.path.join(egg_info_dir, 'installed-files.txt'), 'w')
f.write('\n'.join(new_lines)+'\n')
f.close()
finally:
if os.path.exists(record_filename):
os.remove(record_filename)
os.rmdir(temp_location)

The error comes from the os.remove line 2nd from bottom. The file was
opened in the top line. The logger.warn code path returns without
closing the file. If I add f.close() just before return then I get:

$ pip install bento
Downloading/unpacking bento
  Downloading bento-0.1.1.tar.gz (582kB): 582kB downloaded
  Running setup.py egg_info for package bento
Installing collected packages: bento
  Running setup.py install for bento
  Could not find .egg-info directory in install record for bento
Successfully installed bento
Cleaning up...

It's probably better to use the with statement though.


Oscar
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Rejecting PEP 439: implicit pip bootstrapping

[Nick Coghlan]

Donald's PEP 453 for explicit pip bootstrapping (by inclusion of a
"getpip" module in the standard library, automatically executed during
installation by the CPython binary installers) is now available on
python.org.

Accordingly, I am now formally rejecting the implicit bootstrapping
proposal that Richard put together for PEP 439. The explicit
bootstrapping proposal in PEP 453 should provide a more reliable
experience, both on initial installation and when upgrading both pip
and CPython.

Discussions on PEP 453 will likely be split - here on distutils-sig to
ensure the proposal is acceptable to the pip developers, and on
python-dev to ensure it is acceptable to the CPython team.

As a proposal with a direct impact on CPython, final pronouncement on
PEP 453 will take place on python-dev rather than here.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Comments on PEP 426

[Nick Coghlan]

On 30 August 2013 22:26, Paul Moore  wrote:
> On 30 August 2013 12:54, Oscar Benjamin  wrote:
>>
>> setup.py that would satisfy those commands. It could be as easy as:
>>
>> if sys.argv[1] == 'bdist_wheel':
>> sys.exit(subprocess.call(['bentomaker', 'build_wheel'])
>>
>> or whatever.
>
>
> OK, inspiration moment for me. It had never occurred to me that a project
> could do something like that, and now I see it explained, I understand the
> various discussions about standardising command line APIs much better.
> Thanks!

If you look at bento's setup.py it's even simpler:

from bento.distutils.monkey_patch import setup
setup()

>> I tried deleting the mentioned file but just got the same error
>> message again. Is that a bento/pip/setuptools bug? I notice that the
>> bento docs don't mention pip on the installation page:
>> http://cournape.github.io/Bento/html/install.html
>>
>
> I've never yet got bento working for me. I have assumed that some aspect of
> Windows, Python 3.x, pip or oddities of my setup are not supported yet, and
> have simply left it as "not for prime time yet" (based on the 0.1.1 version
> number) in my mind. So it's always remained as the "obvious example of an
> alternative build system" while still being theoretical.

Could just be a Windows incompatibility - *nix will let another
process open the same file quite happily.

> I'd love to see something concrete though, as these debates tend to get
> remarkably abstract at times :-)

d2to1 is actually my "go to" example of an alternate build system that
people are already using. A setup.py style shim for a setup.cfg based
project. Although d2to1 may just embed the metadata directly in
setup.py - I haven't actually looked into it in detail.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Comments on PEP 426

[Daniel Holth]

On Fri, Aug 30, 2013 at 7:54 AM, Oscar Benjamin
 wrote:
> On 29 August 2013 16:49, Paul Moore  wrote:
>> On 29 August 2013 16:02, Oscar Benjamin  wrote:
>>>On 29 August 2013 15:30, Antoine Pitrou  wrote:
>> [...]
 (after all, it's just "python setup.py build_bdist", or something :-))
>>>
>>> The point is that pip and other packaging tools will use 'python
>>> setup.py ...' to do all the building and wheel making and so on.
>>> However the required interface that setup.py should expose is not
>>> documented anywhere and is essentially implementation defined where
>>> the implementation is the setup() function from a recent version of
>>> setuptools. In the interest of standardising the required parts of
>>> existing practice the required subset of this interface should be
>>> documented.
>>
>> Specifically, the command is
>>
>> python setup.py bdist_wheel
>>
>> But that requires the wheel project and setuptools to be installed,
>> and we're not going to require all users to have those available.
>>
>> Also, other projects can build wheels with different commands/interfaces:
>> * distlib says put all your built files in a set of directories then
>> do wheel.build(paths=path_mapping) - no setup.py needed at all
>> * pip says pip wheel requirement (but that uses setuptools/wheel under the 
>> hood)
>> * bento might do something completely different
>
> Yes, but whatever is used if the required interface from setup.py is
> documented it's easy enough for a distribution author to create a
> setup.py that would satisfy those commands. It could be as easy as:
>
> if sys.argv[1] == 'bdist_wheel':
> sys.exit(subprocess.call(['bentomaker', 'build_wheel'])
>
> or whatever. Then, if bento is build-required (or distributed in the
> sdist), 'pip wheel' would work right? Bento could even ship/generate
> setup.py files for bento-using distributions to use (I assume
> 'bentomaker sdist' does actually do this but I got an error installing
> bento; see below...).
>
> However, right now it's not clear exactly what the command line
> interface would need to be e.g.: Should setup.py process any optional
> arguments? How should it know what filename to give the wheel and what
> directory to put it in? Should the setup.py assume that its current
> working directory is the VCS checkout or unpacked sdist directory?
> Will pip et al. infer sucess/failure from the return code? Who is
> supposed to be responsible for any cleanup if necessary?
>
>> The whole question of standardising the command line API for building
>> (sdists and) wheels is being avoided at the moment, as it's going to
>> be another long debate (setup.py is too closely associated with
>> distutils and/or setuptools for some people).
>
> Yes but rather than try to think of something better I'm just
> suggesting to document what is *already* required, with some guarantee
> of backward compatibility that will be respected in the future. Even
> if wheels become commonplace and are used by the most significant
> projects there will still be a need to build some distributions from
> source e.g. because the authors didn't build a wheel for your
> architecture, or the user/author prefer to make build-time
> optimisations etc.
>
>> AIUI, we're sort of moving towards the "official" command line API
>> being pip's (so "pip wheel XXX") but that's not a complete answer as
>> currently pip internally just uses the setup.py command line, and the
>> intention is to decouple the two so that alternative build tools (like
>> bento, I guess) get a look in. It's all a bit vague at the moment,
>> though, because nobody has even looked at what alternative build tools
>> might even be needed.
>>
>> I could have this completely wrong, though - we're trying very hard to
>> keep the work in small chunks, and building is not one of those chunks
>> yet.
>
> Leaving the build infrastructure alone for now seems reasonable to me.
> However if a static target is created for third-party build tools then
> there could be more progress on that front.
>
> I just tried to install bento to test it out and:
>
> $ pip install bento
> Downloading/unpacking bento
>   Downloading bento-0.1.1.tar.gz (582kB): 582kB downloaded
>   Running setup.py egg_info for package bento
> Installing collected packages: bento
>   Running setup.py install for bento
>   Could not find .egg-info directory in install record for bento
> Cleaning up...
> Exception:
> Traceback (most recent call last):
>   File "Q:\tools\Python27\lib\site-packages\pip\basecommand.py", line
> 134, in main
> status = self.run(options, args)
>   File "Q:\tools\Python27\lib\site-packages\pip\commands\install.py",
> line 241, in run
> requirement_set.install(install_options, global_options,
> root=options.root_path)
>   File "Q:\tools\Python27\lib\site-packages\pip\req.py", line 1298, in install
> requirement.install(install_options, global_options, *args, **kwargs)
>   File "Q:\tools\Python27\lib\site-packages\pip\req.py", line 668

Re: [Distutils] PEP 449 -- Removal of the PyPI Mirror Auto Discovery and Naming Scheme

[Donald Stufft]

On Aug 29, 2013, at 9:33 PM, Richard Jones  wrote:

> Hi all,
> 
> Given that I believe all outstanding issues with PEP 449 < 
> http://python.org/dev/peps/pep-0449/> have been resolved I will accept it in 
> its current form (last modified August 16) so the immediate changes may be 
> made and publicity of the change can be started.
> 
> My thanks to everyone involved in trying to make mirroring work, and also for 
> everyone who has contributed to getting this PEP to an acceptable state.
> 

Awesome, thanks!

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Comments on PEP 426

[Paul Moore]

On 30 August 2013 12:54, Oscar Benjamin  wrote:

> setup.py that would satisfy those commands. It could be as easy as:
>
> if sys.argv[1] == 'bdist_wheel':
> sys.exit(subprocess.call(['bentomaker', 'build_wheel'])
>
> or whatever.


OK, inspiration moment for me. It had never occurred to me that a project
could do something like that, and now I see it explained, I understand the
various discussions about standardising command line APIs much better.
Thanks!


> I tried deleting the mentioned file but just got the same error
> message again. Is that a bento/pip/setuptools bug? I notice that the
> bento docs don't mention pip on the installation page:
> http://cournape.github.io/Bento/html/install.html
>
>
I've never yet got bento working for me. I have assumed that some aspect of
Windows, Python 3.x, pip or oddities of my setup are not supported yet, and
have simply left it as "not for prime time yet" (based on the 0.1.1 version
number) in my mind. So it's always remained as the "obvious example of an
alternative build system" while still being theoretical.

I'd love to see something concrete though, as these debates tend to get
remarkably abstract at times :-)

Paul.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Comments on PEP 426

[Oscar Benjamin]

On 29 August 2013 16:49, Paul Moore  wrote:
> On 29 August 2013 16:02, Oscar Benjamin  wrote:
>>On 29 August 2013 15:30, Antoine Pitrou  wrote:
> [...]
>>> (after all, it's just "python setup.py build_bdist", or something :-))
>>
>> The point is that pip and other packaging tools will use 'python
>> setup.py ...' to do all the building and wheel making and so on.
>> However the required interface that setup.py should expose is not
>> documented anywhere and is essentially implementation defined where
>> the implementation is the setup() function from a recent version of
>> setuptools. In the interest of standardising the required parts of
>> existing practice the required subset of this interface should be
>> documented.
>
> Specifically, the command is
>
> python setup.py bdist_wheel
>
> But that requires the wheel project and setuptools to be installed,
> and we're not going to require all users to have those available.
>
> Also, other projects can build wheels with different commands/interfaces:
> * distlib says put all your built files in a set of directories then
> do wheel.build(paths=path_mapping) - no setup.py needed at all
> * pip says pip wheel requirement (but that uses setuptools/wheel under the 
> hood)
> * bento might do something completely different

Yes, but whatever is used if the required interface from setup.py is
documented it's easy enough for a distribution author to create a
setup.py that would satisfy those commands. It could be as easy as:

if sys.argv[1] == 'bdist_wheel':
sys.exit(subprocess.call(['bentomaker', 'build_wheel'])

or whatever. Then, if bento is build-required (or distributed in the
sdist), 'pip wheel' would work right? Bento could even ship/generate
setup.py files for bento-using distributions to use (I assume
'bentomaker sdist' does actually do this but I got an error installing
bento; see below...).

However, right now it's not clear exactly what the command line
interface would need to be e.g.: Should setup.py process any optional
arguments? How should it know what filename to give the wheel and what
directory to put it in? Should the setup.py assume that its current
working directory is the VCS checkout or unpacked sdist directory?
Will pip et al. infer sucess/failure from the return code? Who is
supposed to be responsible for any cleanup if necessary?

> The whole question of standardising the command line API for building
> (sdists and) wheels is being avoided at the moment, as it's going to
> be another long debate (setup.py is too closely associated with
> distutils and/or setuptools for some people).

Yes but rather than try to think of something better I'm just
suggesting to document what is *already* required, with some guarantee
of backward compatibility that will be respected in the future. Even
if wheels become commonplace and are used by the most significant
projects there will still be a need to build some distributions from
source e.g. because the authors didn't build a wheel for your
architecture, or the user/author prefer to make build-time
optimisations etc.

> AIUI, we're sort of moving towards the "official" command line API
> being pip's (so "pip wheel XXX") but that's not a complete answer as
> currently pip internally just uses the setup.py command line, and the
> intention is to decouple the two so that alternative build tools (like
> bento, I guess) get a look in. It's all a bit vague at the moment,
> though, because nobody has even looked at what alternative build tools
> might even be needed.
>
> I could have this completely wrong, though - we're trying very hard to
> keep the work in small chunks, and building is not one of those chunks
> yet.

Leaving the build infrastructure alone for now seems reasonable to me.
However if a static target is created for third-party build tools then
there could be more progress on that front.

I just tried to install bento to test it out and:

$ pip install bento
Downloading/unpacking bento
  Downloading bento-0.1.1.tar.gz (582kB): 582kB downloaded
  Running setup.py egg_info for package bento
Installing collected packages: bento
  Running setup.py install for bento
  Could not find .egg-info directory in install record for bento
Cleaning up...
Exception:
Traceback (most recent call last):
  File "Q:\tools\Python27\lib\site-packages\pip\basecommand.py", line
134, in main
status = self.run(options, args)
  File "Q:\tools\Python27\lib\site-packages\pip\commands\install.py",
line 241, in run
requirement_set.install(install_options, global_options,
root=options.root_path)
  File "Q:\tools\Python27\lib\site-packages\pip\req.py", line 1298, in install
requirement.install(install_options, global_options, *args, **kwargs)
  File "Q:\tools\Python27\lib\site-packages\pip\req.py", line 668, in install
os.remove(record_filename)
WindowsError: [Error 32] The process cannot access the file because it
is being used by another process:
'c:\\docume~1\\enojb\\locals~1\\temp\\pip-aae65s-record\\in

Re: [Distutils] Comments on PEP 426

[Nick Coghlan]

On 30 Aug 2013 17:23, "Paul Moore"  wrote:
>
> On 30 August 2013 00:08, Nick Coghlan  wrote:
>>
>> We also need to officially bless pip's trick of forcing the use of
setuptools for distutils based setup.py files.
>
>
> Do we? What does official blessing imply? We've managed for years without
the trick being "official"...
>
> The main reason it is currently used is to allow setup.py install to
specify --record, so that we can get the list of installed files. If
distutils added a --record flag, for example, I don't believe we'd need the
hack at all. (Obviously, we'd still need setuptools so we could use wheel
to build wheels, but that's somewhat different as it's a new feature).
Maybe a small distutils patch is better than blessing setuptools here?

A distutils patch won't help with Python 2.7 or 3.3. The purpose of
blessing the substitution is to decouple the update cycle of the build
system from the update cycle of the standard library.

(This is a lesson I note even RH acknowledged for RHEL when we first
released the Red Hat Developer Toolset)

It's not just the record flag - it's also about supporting wheel generation
and eventually the metadata 2.0 formats and features.

> Forcing setuptools does break some builds. From when I last tried, for
instance, I believe that cx_Oracle won't build with setuptools forced.
Whose issue should that be? Pip's, setuptools', or cx_Oracle's? (Note that
I'm *not* saying that this is a showstopper, just trying to clarify the
intent here).

Officially blessing replacing distutils with setuptools would strongly push
this case in the direction of being a setuptools bug (since setuptools
generally aims to be a drop in replacement for distutils), without ruling
out a cx_Oracle bug (e.g. if the problem is incompatible monkey patching of
distutils).

pip would be cleared of any blame.

>
> Note - I'm not against blessing pip's hack. But I'd like it to be clear
what such a blessing *means* and why just leaving it as an internal
implementation detail isn't sufficient.

My aim would be to ensure all build systems are free to substitute
setuptools for distutils to get support for new distribution standards on
old versions of Python.

Cheers,
Nick.
>
> Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Comments on PEP 426

[Vinay Sajip]

Nick Coghlan  gmail.com> writes:

> We also need to officially bless pip's trick of forcing the use of
> setuptools for distutils based setup.py files.

What does official blessing mean in practice? Is pip to be wedded to
setuptools forever, or might it one day have the possibility of being
implemented differently? If so, why should any comment be made about what is
an implementation detail?

Regards,

Vinay Sajip

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Comments on PEP 426

[Paul Moore]

On 30 August 2013 00:08, Nick Coghlan  wrote:

> We also need to officially bless pip's trick of forcing the use of
> setuptools for distutils based setup.py files.


Do we? What does official blessing imply? We've managed for years without
the trick being "official"...

The main reason it is currently used is to allow setup.py install to
specify --record, so that we can get the list of installed files. If
distutils added a --record flag, for example, I don't believe we'd need the
hack at all. (Obviously, we'd still need setuptools so we could use wheel
to build wheels, but that's somewhat different as it's a new feature).
Maybe a small distutils patch is better than blessing setuptools here?

Forcing setuptools does break some builds. From when I last tried, for
instance, I believe that cx_Oracle won't build with setuptools forced.
Whose issue should that be? Pip's, setuptools', or cx_Oracle's? (Note that
I'm *not* saying that this is a showstopper, just trying to clarify the
intent here).

Note - I'm not against blessing pip's hack. But I'd like it to be clear
what such a blessing *means* and why just leaving it as an internal
implementation detail isn't sufficient.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig