Re: New, updated pip coming
On Feb 01, 2016, at 03:01 PM, Piotr Ożarowski wrote: >Did you consider creating whl files at install time¹ rather than pip's >build time? This way whl can be regenerated whenever one of needed >packages is updated and there's no need to rebuild pip package. > >[¹] using dpkg (file) triggers (/usr/share/doc/dpkg-dev/triggers.txt.gz) I did think about it, and maybe it's where we want to end up, but I didn't go this way right now because of the added complexity. Well, maybe "different" complexity, since none of this stuff is simple. ;) What we have now is better than what we had before, but once I get virtualenv fixed and uploaded, I think I'll come back around to looking at what you suggest. (Certainly, patches welcome in the meantime.) Fortunately, I believe we have all the tools in place (e.g. dirtbike) to generate the wheels at install time. It does mean some extra run-time dependencies, but that's probably fine. Cheers, -Barry
Re: New, updated pip coming
Hi Barry, Did you consider creating whl files at install time¹ rather than pip's build time? This way whl can be regenerated whenever one of needed packages is updated and there's no need to rebuild pip package. [¹] using dpkg (file) triggers (/usr/share/doc/dpkg-dev/triggers.txt.gz) -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
Re: New, updated pip coming
On Fri, Jan 29, 2016 at 18:28:53 -0500, Barry Warsaw wrote: > I don't actually know whether Built-Using *does* anything useful[*], but IWBNI > (maybe) some kind of notification or auto-rebuild happened. I don't think it > does, but Donald's right. It's the same problem that Go or statically linked > C/C++ has. > ATM there's no automatic rebuild for out of date built-using. Its main use is to force the archive software to keep the source packages listed as Built-Using in the archive, to comply with GPL requirements. Cheers, Julien -- Julien Cristau Logilab http://www.logilab.fr/ Informatique scientifique & gestion de connaissances
Re: New, updated pip coming
On Jan 30, 2016, at 11:33 AM, Robert Collins wrote: > If a new requests package is built, uploaded to the archive and >apt-get installed on my machine, and I then do: >virtualenv test >. test/bin/activate >pip install foobar > ^--- what version of requests will be used by this in-virtualenv >invocation of pip? > >>From your description I expect it to be the version of requests that was >dpkg installed at the time the pip deb was built? Correct. On Jan 29, 2016, at 05:45 PM, Donald Stufft wrote: >These wheel files that represent pip's dependencies are generated when >python-pip is being built from the versions of the packages in Debian's >repositories and the python-pip package has a Built-Using header to indicate >which dependencies have been rebundled inside of pip through this >process. This has essentially recreated static linking but for Python. It is >my understanding that all of the wheels for pip's dependencies will exist >inside of the python-pip-whl .deb. Yep. I don't actually know whether Built-Using *does* anything useful[*], but IWBNI (maybe) some kind of notification or auto-rebuild happened. I don't think it does, but Donald's right. It's the same problem that Go or statically linked C/C++ has. >On top of that, virtualenv and venv will both need wheels that it needs to >install for pip, setuptools, and in the case of virtualenv, wheel. I'm not >sure what the plan is for those. I plan to use the same technique for virtualenv/pyvenv dependencies, although they will probably Depend on python-pip-whl to just get those for free. TBH, I haven't spent a ton of time on those higher level tools, just because of how much work it was to sort out pip. But I'll look at those soon. Cheers, -Barry [*] There are some plans to honor this field in Ubuntu, where Build-Depends may not need to go through the same main-inclusion-report (MIR) process, but Built-Using packages would. It's all currently speculative, so I don't have any more details right now. pgpYCju0beL_D.pgp Description: OpenPGP digital signature
Re: New, updated pip coming
> On Jan 29, 2016, at 5:33 PM, Robert Collins wrote: > > I am confused. Here's my understanding of things... > > - Pip doesn't need wheels at all - its vendoring technique doesn't use wheels. > - Virtualenv needs to install pip, wheel, setuptools when it makes a > new environment, and it does that by some oogly code > - venv might do the same thing, but I haven't check its actual implementation > - the pip-whl package was used to give virtualenv a thing to use to > install pip when making a virtualenv > > If that aligns with your understanding, I'll put it down to the prose > you wrote rather than us actually having a different worldview; OTOH > if it doesn't, I'd like to find out where I'm wrong, so as to improve > my understanding. > > With my understanding in mind - Can I just check something? > > If a new requests package is built, uploaded to the archive and > apt-get installed on my machine, and I then do: > virtualenv test > . test/bin/activate > pip install foobar > ^--- what version of requests will be used by this in-virtualenv > invocation of pip? Debian doesn’t want to utilize the bundled libraries in pip for various reasons (some I agree with, some I don't, but suffice it to say it's a requirement) but there is recognition that pip is a bit special here and can't just be debundled in the standard way. In conjunction with Barry, pip has some basic support for debundling by deleting the contents of pip/_vendor/ except for pip/_vendor/__init__.py, creating wheel files for all of pip's dependencies, and then making a small patch to pip/_vendor/__init__.py to turn on the debundling feature and tell it where those wheel files live. Pip will then add all of those wheel files to the front of sys.path. This gives us a few things: * Pip will *always* use the versions of those libraries from the wheels, regardless of what is installed. This makes it so you can't break yourself with ``pip install requests`` in any environment (virtual or otherwise). * Debian gets to have all of the code in pip come from the correct source packages. These wheel files that represent pip's dependencies are generated when python-pip is being built from the versions of the packages in Debian's repositories and the python-pip package has a Built-Using header to indicate which dependencies have been rebundled inside of pip through this process. This has essentially recreated static linking but for Python. It is my understanding that all of the wheels for pip's dependencies will exist inside of the python-pip-whl .deb. On top of that, virtualenv and venv will both need wheels that it needs to install for pip, setuptools, and in the case of virtualenv, wheel. I'm not sure what the plan is for those. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail
Re: New, updated pip coming
I am confused. Here's my understanding of things...
- Pip doesn't need wheels at all - its vendoring technique doesn't use wheels.
- Virtualenv needs to install pip, wheel, setuptools when it makes a
new environment, and it does that by some oogly code
- venv might do the same thing, but I haven't check its actual implementation
- the pip-whl package was used to give virtualenv a thing to use to
install pip when making a virtualenv
If that aligns with your understanding, I'll put it down to the prose
you wrote rather than us actually having a different worldview; OTOH
if it doesn't, I'd like to find out where I'm wrong, so as to improve
my understanding.
With my understanding in mind - Can I just check something?
If a new requests package is built, uploaded to the archive and
apt-get installed on my machine, and I then do:
virtualenv test
. test/bin/activate
pip install foobar
^--- what version of requests will be used by this in-virtualenv
invocation of pip?
>From your description I expect it to be the version of requests that
was dpkg installed at the time the pip deb was built?
-Rob
On 30 January 2016 at 08:28, Barry Warsaw wrote:
> TL;DR By the end of today, I expect to upload pip 8.0.2 to unstable, finally
> bringing us up to the latest version.
>
> It's been a long slog, with many people helping out. I hope that all the hard
> work done to get us here means it will be much, much easier to track new pip,
> virtualenv, and pyvenv updates going forward. And now, details!
>
> pip requires the use of PEP 427 wheels, which are essentially zip files of
> importable, pure-Python, bilingual code, with a little bit of metadata. Put a
> wheel on sys.path and Python can import from it. pip and friends use wheels
> because it ensures that they can't break themselves when their dependent
> packages are updated. E.g. if pip depends on requests, and you `pip install
> --upgrade requests` it's possible the new version would be incompatible with
> pip. Then you'd have a broken pip on your hands. Remember that you can use
> pip to install pre-release packages too.
>
> So by putting the wheels on sys.path *before* anything else, even if you pip
> install an incompatible requests, it won't break pip itself.
>
> pip "vendorizes" (i.e. bundles) the wheels of its dependencies with known good
> versions, but this breaks Debian policy, so we have to unbundle these.
> Several upstreams use this technique, with requests and urllib3 as probably
> the most popular examples. In all these cases, we have to remove the bundled
> wheels and hope that the versions of their dependencies in Debian remain
> compatible. It's a chore, but that's our job as distro developers, and it's
> the best option given the conflicting requirements.
>
> Fortunately, we can create our own wheels and use them with pip and friends.
> Until now, we had to build the wheels at the time that the dependent packages
> where built, and we had to have a corresponding foo-whl binary package in the
> archives, because we could only use setup.py's bdist_wheel command (which
> comes with python{,3}-wheel) to create these. With 15 run time dependencies
> as of pip 7.1.2, this has become untenable, and was a huge reason why Debian's
> pip has lagged so far behind upstream. (That, and a few ITPs for new
> dependencies. ;)
>
> Along comes dirtbike. https://github.com/paulproteus/dirtbike
>
> While it has been much discussed and attempted over the years, it would be
> really nice to have a tool that turned installed Python packages into wheels.
> I think we discussed such a tool for Debian at last year's (two years ago?)
> Debian BOF at Pycon. Anyway, Asheesh Laroia finally took hold of the, ahem,
> wheel, and wrote the initial implementation, which he called dirtbike. Since
> then I've been enhancing it to work with several oddball cases in the Debian
> archive, and now it's good enough to turn pip's dependencies into wheels.
>
> Yesterday I uploaded the last of the new pip dependencies (modulo any
> surprises in 8.0.2 ;), and dirtbike so now all the pieces are in place. What
> pip now does during its own build process, is turn its installed dependencies
> into .whl files and puts them in the policy defined location. This allows us
> to eliminate all those scattered -whl binary packages, and just
> Build-Depend/Built-Using in pip to install the standard Python packages,
> rewheel them, and put them in a python-pip-whl package. Going forward, this
> will make life so much easier, because only python-pip needs to know anything
> about wheels, even if new dependencies are added.
>
> I have everything working locally. I need to update the master branch for
> pip, which involves many curse words hurled at git-dpm (and that's a post for
> another day!). I also have to update my local branch to 8.0.2, but I'm
> assuming there won't be any new surprises there.
>
> Eventually, I'll be using the same techique for virtualenv and pyvenv, but not
> ri
New, updated pip coming
TL;DR By the end of today, I expect to upload pip 8.0.2 to unstable, finally
bringing us up to the latest version.
It's been a long slog, with many people helping out. I hope that all the hard
work done to get us here means it will be much, much easier to track new pip,
virtualenv, and pyvenv updates going forward. And now, details!
pip requires the use of PEP 427 wheels, which are essentially zip files of
importable, pure-Python, bilingual code, with a little bit of metadata. Put a
wheel on sys.path and Python can import from it. pip and friends use wheels
because it ensures that they can't break themselves when their dependent
packages are updated. E.g. if pip depends on requests, and you `pip install
--upgrade requests` it's possible the new version would be incompatible with
pip. Then you'd have a broken pip on your hands. Remember that you can use
pip to install pre-release packages too.
So by putting the wheels on sys.path *before* anything else, even if you pip
install an incompatible requests, it won't break pip itself.
pip "vendorizes" (i.e. bundles) the wheels of its dependencies with known good
versions, but this breaks Debian policy, so we have to unbundle these.
Several upstreams use this technique, with requests and urllib3 as probably
the most popular examples. In all these cases, we have to remove the bundled
wheels and hope that the versions of their dependencies in Debian remain
compatible. It's a chore, but that's our job as distro developers, and it's
the best option given the conflicting requirements.
Fortunately, we can create our own wheels and use them with pip and friends.
Until now, we had to build the wheels at the time that the dependent packages
where built, and we had to have a corresponding foo-whl binary package in the
archives, because we could only use setup.py's bdist_wheel command (which
comes with python{,3}-wheel) to create these. With 15 run time dependencies
as of pip 7.1.2, this has become untenable, and was a huge reason why Debian's
pip has lagged so far behind upstream. (That, and a few ITPs for new
dependencies. ;)
Along comes dirtbike. https://github.com/paulproteus/dirtbike
While it has been much discussed and attempted over the years, it would be
really nice to have a tool that turned installed Python packages into wheels.
I think we discussed such a tool for Debian at last year's (two years ago?)
Debian BOF at Pycon. Anyway, Asheesh Laroia finally took hold of the, ahem,
wheel, and wrote the initial implementation, which he called dirtbike. Since
then I've been enhancing it to work with several oddball cases in the Debian
archive, and now it's good enough to turn pip's dependencies into wheels.
Yesterday I uploaded the last of the new pip dependencies (modulo any
surprises in 8.0.2 ;), and dirtbike so now all the pieces are in place. What
pip now does during its own build process, is turn its installed dependencies
into .whl files and puts them in the policy defined location. This allows us
to eliminate all those scattered -whl binary packages, and just
Build-Depend/Built-Using in pip to install the standard Python packages,
rewheel them, and put them in a python-pip-whl package. Going forward, this
will make life so much easier, because only python-pip needs to know anything
about wheels, even if new dependencies are added.
I have everything working locally. I need to update the master branch for
pip, which involves many curse words hurled at git-dpm (and that's a post for
another day!). I also have to update my local branch to 8.0.2, but I'm
assuming there won't be any new surprises there.
Eventually, I'll be using the same techique for virtualenv and pyvenv, but not
right away.
Huge thanks also go to Donald Stufft, who always is so helpful with various
technical details, and works really hard to understand our needs and balance
them with upstream's.
Finally, we need to update Python policy to describe the new wheel policy.
Fortunately, this is much better because it removes constraints on other
packages, and reduces the number of -whl binary packages in the distro. Yes,
I will update those packages that already ship -whls, or file bugs, but only
once I'm sure everything is working correctly.
Below is the policy diff, which I'll commit to the python-defaults bzr.
As always, once everything's uploaded, please do let me know via this list or
BTS, if you notice any problems with pip.
Cheers,
-Barry
=== modified file 'debian/python-policy.sgml'
--- debian/python-policy.sgml 2016-01-24 06:01:19 +
+++ debian/python-policy.sgml 2016-01-29 16:49:59 +
@@ -458,13 +458,12 @@
http://legacy.python.org/dev/peps/pep-0427/";
name="PEP 427">
- defines a built-package format called "wheels", which is a zip
- format archive containing Python code and a "dist-info" metadata
+ defines a built-package format called "wheels", a zip
+ archive containing Python
