Here's another version of the patch, which implements the scheme as discussion
on debian-python. Application of this patch should be contingent on:
* Agreement on changes to Debian Python Policy regarding the use of wheels
(prohibited in the general, allowed by exception for ensurepip support).
See discussion there for details.
* Upload of all wheel dependencies. DPMT maintained packages implementing
the scheme are in NEW. colorama will require an NMU. Separate bugs with
patches will be filed for distlib and setuptools for maintainer
consideration.
=== added file 'debian/README.venv'
--- debian/README.venv 1970-01-01 00:00:00 +0000
+++ debian/README.venv 2014-05-16 18:00:10 +0000
@@ -0,0 +1,225 @@
+=========================================
+ pyvenv support in Python 3.4 and beyond
+=========================================
+
+In Python 3.3, new built-in support for virtual environments (venvs) was add
+via the `pyvenv`_ command. For building venvs using Python 3, this is
+functionally equivalent to the standalone `virtualenv`_ tool (except that
+before Python 3.4, the pyvenv created venv didn't include pip and setuptools).
+
+In Python 3.4, this was made even more convenient by the `automatic
+inclusion`_ of the `pip`_ command into the venv so that third party libraries
+can be easily installed from the Python Package Index (PyPI_). The stdlib
+module `ensurepip`_ is run when the `pyvenv-3.4` command is run to create the
+venv.
+
+This poses a problem for Debian. ensurepip comes bundled with two third party
+libraries, setuptools and pip itself, as these are requirements for pip to
+function properly in the venv. These are bundled in the ensurepip module of
+the upstream Python 3.4 tarball as `universal wheels`_, essentially a zip of
+the source code and a new ``dist-info`` metadata directory. Upstream pip
+itself comes bundled with a half dozen or so of *its* dependencies, except
+that these are "vendorized", meaning their unpacked source code lives within
+the pip module, under a submodule from which pip imports them rather than the
+top-level package namespace.
+
+To make matters worse, one of pip's vendorized dependencies, the `requests`_
+module, *also* vendorizes a bunch of its own dependencies. This stack of
+vendorized and bundled third party libraries fundamentally violates the DFSG
+and Debian policy against including code not built from source available
+within Debian, and for including embedded "convenience" copies of code in
+other packages.
+
+It's worth noting that the virtualenv package actually suffers from the same
+conflict, but its current solution in Debian is `incomplete`_.
+
+
+Solving the conflict
+====================
+
+This conflict between Debian policy and upstream Python convenience must be
+resolved, because pyvenv is the recommended way of creating venvs in Python 3,
+and because at some point, the standalone virtualenv tool will be rewritten as
+a thin layer above pyvenv. Obviously, we want to provide the best Python
+virtual environment experience to our developers, adherent to Debian policy.
+
+The approach we've taken is layered and nuanced, so I'll provide a significant
+amount of detail to explain both what we do and why.
+
+The first thing to notice is how upstream ensurepip works to have its pip and
+setuptools dependencies available, both at venv creation time and when
+``<venv>/bin/pip`` is run. When pyvenv-3.4 runs, it ends up calling the
+following Python command::
+
+ <venv>/bin/python -Im ensurepip --upgrade
+
+This runs the ensurepip's ``__main__.py`` module using the venv's Python in
+isolation mode, with a switch to upgrade the setuptools and pip dependencies
+(if for example, they've been updated in a new micro version of Python).
+
+Internally, ensurepip bootstraps itself by byte-copying its embedded wheels
+into a temporary directory, putting those copied wheels on ``sys.path``, and
+then calling into pip as a library. Because wheels are just elaborate zips,
+Python can execute (pure-Python) code directly from them, if they are on
+``sys.path`` of course. Once ensurepip has set up its execution environment,
+it calls into pip to install both pip and setuptools into the newly created
+venv. If you poke inside the venv after successful creation, you'll see
+unpacked pip and setuptools directories in the venv's ``site-packages`
+directory. These were downloaded and upgraded from PyPI if necessary.
+
+The important thing to note here is that ensurepip is *already* able to import
+from and install wheels, and because wheels are self-contained single files
+(of zipped content), it makes manipulating them quite easy. In order to
+minimize the delta from upstream (and to eventually work with upstream to
+eliminate this delta), it seems optimal that Debian's solution should also be
+based on wheels, and re-use as much of the existing machinery as possible.
+
+The difference for Debian though is that we don't want to use the embedded pip
+and setuptools wheels from upstream Python's ensurepip; we want to use wheels
+created from the pip and setuptools *Debian* packages. This would solve the
+problem of not distributing binary packages built from source in Debian.
+
+Thus, we modify the python-pip and python-setuptools packages to include new
+binary packages python-pip-wheels and python-setuptools-wheels which contain
+*only* the relevant universal wheels. Those packages ``debian/rules`` files
+gain an extra command::
+
+ python3 setup.py bdist_wheel --universal -d <path>
+
+The ``bdist_wheel`` command is provided by the `wheel`_ package, which as of
+this writing is newly available in Jessie.
+
+The universal wheels (i.e. pure-Python code compatible with both Python 2 and
+Python 3) are built for pip and setuptools and installed into
+``/usr/share/python-wheels`` when the python-{pip,setuptols}-wheels packages
+are installed. These are not needed for normal, usual, and typical operation
+of Python, so none of these are installed by default.
+
+However, this isn't enough, because since the pip and setuptools wheels are
+built from the *patched* and de-vendorized versions of the code in Debian, the
+wheels will not contain their own recursive dependencies. That's a good thing
+for Debian policy compliance, but does add complications to the stack of hack.
+
+Using the same approach as for pip and setuptools, we *also* wheelify their
+dependencies, recursively. As of this writing, the list of packages needing
+to be wheelified are (by Debian source package name):
+
+ * chardet
+ * distlib
+ * html5lib
+ * python-colorama
+ * python-setuptools
+ * python-urllib3
+ * requests
+ * six
+ * urllib3
+
+Most of these are DPMT maintained. six, distlib, and colorama are not team
+maintained, so coordination with those maintainers is required. Also note
+that the `bdist_wheel` command is a setuptools extension, so while some of
+those projects use ``distutils.core.setup()`` by default, they must be patched
+to use ``setuptools.setup()`` instead. This isn't a problem because there's
+no functional difference relevant to those packages; they likely use
+distutils.core to avoid a third party dependency on setuptools.
+
+Each of these Debian source packages grow an additional binary package, just
+like pip and setuptools, e.g. python-chardet-wheels which contains the
+universal wheel for that package built from patched Debian source. As above,
+when installed, these binary packages drop their .whl files into the
+``/usr/share/python-wheels`` directory.
+
+Now comes the fun part.
+
+In the python3.4 source package, we add a new binary package called
+python3.4-venv. This will only contain the ``/usr/bin/pyvenv-3.4``
+executable, and its associated manpage. It also includes all the run-time
+dependencies to make pyvenv work *including the wheel packages described
+above*.
+
+(Technically speaking, you should substitute "Python 3.4 or later" for all
+these discussions, and e.g. pyvenv-3.x for all versions subsequent to 3.4.)
+
+Python's ensurepip module has been modified in the following ways (see
+``debian/patches/ensurepip.diff``):
+
+ * When ensurepip is run outside of a venv as root, it raises an exception.
+ This use case is only to be supported by the separate python{,3}-pip
+ packages.
+
+ * When ensurepip is run inside of a venv, it copies all dependent wheels from
+ ``/usr/share/python-wheels``. This includes the direct dependencies pip
+ and setuptools, as well as the recursive dependencies listed above. The
+ rest of the ensurepip machinery is unchanged: the wheels are still copied
+ into a temporary directory and placed on ``sys.path``, however only the
+ direct dependencies (i.e. pip and setuptools) are *installed* into the
+ venv's ``site-packages`` directory. The indirect dependencies are copied
+ to ``<venv>/lib/python-wheels`` since they'll be needed by the venv's pip
+ executable.
+
+Why do we do this latter rather than also installing the recursive
+dependencies into the venv's ``site-packages``? It's because pip requires a
+very specific set of dependencies and we don't want pip to break when the user
+upgrades or downgrades one of those packages, which is perfectly valid in a
+venv. It's exactly the same reason why pip vendorizes those libraries in the
+first place; it's just that we're doing it in a more principled way (from the
+point of view of the Debian distribution).
+
+The final piece of the puzzle is that Debian's pip will, when run inside of a
+venv, introspect ``<venv>/lib/python-wheels`` and put every .whl file it sees
+there *at the front of its sys.path*. Again, this is so that when pip runs,
+it will find the versions of packages known to be good first, rather than any
+other versions in the venv's ``site-packages``.
+
+As an example of the bad things that can happen if you don't do this, try
+installing nose2_ into the venv, followed by genshi_. nose2 has a hard
+requirement on a version of six that is older than the one used by pip
+(indirectly). This older version of six is compatible with genshi, but *not*
+with pip, so once nose2 is installed, if pip didn't load its version of six
+from the wheel, the installation attempt of genshi would traceback. As it is,
+with the wheels early enough on ``sys.path``, pip itself works just fine so
+that both nose2 and genshi can live together in the venv.
+
+
+Updating packages
+=================
+
+Inevitably, new versions of Python or the pyvenv dependent packages will
+appear. Unfortunately, as currently implemented (by both upstream ensurepip
+and in our ensurepip patch), the versions of both the direct and indirect
+dependencies are hardcoded in ``Lib/ensurepip/__init__.py``. When a Debian
+developer updates any of the dependent packages, you will need to:
+
+ * *Test that the new version is compatible with ensurepip*.
+
+ * Update the version numbers in the ``debian/control`` file, for the
+ python3.x-venv binary package.
+
+ * ``quilt push`` to the ensurepip patch, and update the version number in
+ ``Lib/ensurepip/__init__.py``
+
+The rebuild and upload python3.4.
+
+Yes, this isn't ideal, and I am working with upstream to find a good solution
+that we can share.
+
+
+Author
+======
+
+Barry A. Warsaw <ba...@debian.org>
+2014-05-15
+
+
+
+.. _pyvenv: http://legacy.python.org/dev/peps/pep-0405/
+.. _virtualenv: https://pypi.python.org/pypi/virtualenv
+.. _`automatic inclusion`: http://legacy.python.org/dev/peps/pep-0453/
+.. _pip: https://pypi.python.org/pypi/pip
+.. _PyPI: https://pypi.python.org/pypi
+.. _ensurepip: https://docs.python.org/3/library/ensurepip.html
+.. _`universal wheels`: http://legacy.python.org/dev/peps/pep-0427/
+.. _requests: https://pypi.python.org/pypi/requests
+.. _incomplete: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=719767
+.. _wheel: https://pypi.python.org/pypi/wheel
+.. _nose2: https://pypi.python.org/pypi/nose2
+.. _genshi: https://pypi.python.org/pypi/Genshi
=== modified file 'debian/changelog'
--- debian/changelog 2014-05-05 21:10:23 +0000
+++ debian/changelog 2014-05-14 19:52:07 +0000
@@ -1,3 +1,13 @@
+python3.4 (3.4.1~rc1-2) UNRELEASED; urgency=medium
+
+ * Added python3.4-venv binary package to contain /usr/bin/pyvenv-3.4,
+ its manpage, and its wheel dependencies. Closes: #732703
+ * Don't remove the full ensurepip module, just remove the bundled wheels.
+ * Adjust the ensurepip.diff patch so that the wheels are installed from
+ the universal wheel packages.
+
+ -- Barry Warsaw <ba...@debian.org> Tue, 13 May 2014 12:11:27 -0400
+
python3.4 (3.4.1~rc1-1) unstable; urgency=medium
* Python 3.4.1 release candidate 1.
=== modified file 'debian/control'
--- debian/control 2014-05-05 21:10:23 +0000
+++ debian/control 2014-05-14 19:40:49 +0000
@@ -198,3 +198,17 @@
- Debug information for standard python extensions.
.
See the README.debug for more information.
+
+Package: python3.4-venv
+Architecture: any
+Multi-Arch: allowed
+Priority: optional
+Section: python
+Depends:
+ python-colorama-wheels, python-distlib-wheels,
+ python-html5lib-wheels, python-requests-wheels,
+ python-chardet-wheels, python-urllib3-wheels,
+ python-six-wheels, python-pip-wheels, python-setuptools-wheels
+Description: pyvenv (built-in virtual environment) support
+ As the preferred alternative for creating virtual environments with Python
+ 3.4, pyvenv-3.4 is the command line executable provided by this package.
=== modified file 'debian/control.in'
--- debian/control.in 2014-05-05 21:10:23 +0000
+++ debian/control.in 2014-05-15 18:29:48 +0000
@@ -198,3 +198,25 @@
- Debug information for standard python extensions.
.
See the README.debug for more information.
+
+Package: @PVER@-venv
+Architecture: any
+Multi-Arch: allowed
+Priority: optional
+Section: python
+Depends:
+ @PVER@ (= ${binary:Version}),
+ lib@PVER@-stdlib (= ${binary:Version}),
+ lib@PVER@ (= ${binary:Version}),
+ python-chardet-wheels (<< 2.2.1.),
+ python-colorama-wheels (<< 0.3.1.),
+ python-distlib-wheels (<< 0.1.8.),
+ python-html5lib-wheels (<< 0.999.),
+ python-pip-wheels (<< 1.5.5.),
+ python-requests-wheels (<< 2.2.1.),
+ python-setuptools-wheels (<< 3.4.4.),
+ python-six-wheels (<< 1.6.1.),
+ python-urllib3-wheels (<< 1.8.),
+Description: pyvenv (built-in virtual environment) support
+ As the preferred alternative for creating virtual environments with Python
+ 3, pyvenv-@VER@ is the command line executable provided by this package.
=== modified file 'debian/patches/ensurepip.diff'
--- debian/patches/ensurepip.diff 2014-05-05 21:10:23 +0000
+++ debian/patches/ensurepip.diff 2014-05-14 19:31:52 +0000
@@ -22,3 +22,82 @@
+tracker.""")
+ else:
+ ensurepip._main()
+--- a/Lib/ensurepip/__init__.py
++++ b/Lib/ensurepip/__init__.py
+@@ -1,6 +1,5 @@
+ import os
+ import os.path
+-import pkgutil
+ import sys
+ import tempfile
+
+@@ -8,9 +7,9 @@
+ __all__ = ["version", "bootstrap"]
+
+
+-_SETUPTOOLS_VERSION = "2.1"
++_SETUPTOOLS_VERSION = "3.4.4"
+
+-_PIP_VERSION = "1.5.4"
++_PIP_VERSION = "1.5.5"
+
+ # pip currently requires ssl support, so we try to provide a nicer
+ # error message when that is missing (http://bugs.python.org/issue19744)
+@@ -28,6 +27,16 @@
+ _PROJECTS = [
+ ("setuptools", _SETUPTOOLS_VERSION),
+ ("pip", _PIP_VERSION),
++ ]
++
++_DEPENDENCIES = [
++ ('chardet', '2.2.1'),
++ ('colorama', '0.3.1'),
++ ('distlib', '0.1.8'),
++ ('html5lib', '0.999'),
++ ('requests', '2.2.1'),
++ ('six', '1.6.1'),
++ ('urllib3', '1.8'),
+ ]
+
+
+@@ -87,20 +96,30 @@
+ # omit pip and easy_install
+ os.environ["ENSUREPIP_OPTIONS"] = "install"
+
++ # Debian: The bundled wheels are useless to us because we must use ones
++ # crafted from source code in the archive. As we build the virtual
++ # environment, copy the wheels from the system location into the virtual
++ # environment, and place those wheels on sys.path.
++ def copy_wheels(wheels, destdir, paths):
++ for project, version in wheels:
++ wheel_name = '{}-{}-py2.py3-none-any.whl'.format(project, version)
++ path = os.path.join('/usr/share/python-wheels', wheel_name)
++ with open(path, 'rb') as fp:
++ whl = fp.read()
++ dest = os.path.join(destdir, wheel_name)
++ with open(dest, 'wb') as fp:
++ fp.write(whl)
++ paths.append(dest)
++
++ private_wheel_dir = os.path.join(sys.prefix, 'lib', 'python-wheels')
++ os.makedirs(private_wheel_dir, exist_ok=True)
++ copy_wheels(_DEPENDENCIES, private_wheel_dir, sys.path)
++
+ with tempfile.TemporaryDirectory() as tmpdir:
+ # Put our bundled wheels into a temporary directory and construct the
+ # additional paths that need added to sys.path
+ additional_paths = []
+- for project, version in _PROJECTS:
+- wheel_name = "{}-{}-py2.py3-none-any.whl".format(project, version)
+- whl = pkgutil.get_data(
+- "ensurepip",
+- "_bundled/{}".format(wheel_name),
+- )
+- with open(os.path.join(tmpdir, wheel_name), "wb") as fp:
+- fp.write(whl)
+-
+- additional_paths.append(os.path.join(tmpdir, wheel_name))
++ copy_wheels(_PROJECTS, tmpdir, additional_paths)
+
+ # Construct the arguments to be passed to the pip command
+ args = ["install", "--no-index", "--find-links", tmpdir]
=== modified file 'debian/rules'
--- debian/rules 2014-05-05 21:10:23 +0000
+++ debian/rules 2014-05-15 15:38:00 +0000
@@ -7,6 +7,7 @@
# Uncomment this to turn on verbose mode.
#export DH_VERBOSE=1
+DEB_BUILD_OPTIONS=nocheck
vafilt = $(subst $(2)=,,$(filter $(2)=%,$(1)))
DPKG_VARS := $(shell dpkg-architecture)
@@ -188,6 +189,7 @@
p_doc := $(PVER)-doc
p_dbg := $(PVER)-dbg
p_udeb := $(PVER)-udeb
+p_venv := $(PVER)-venv
p_lbase := lib$(PVER)-stdlib
p_lmin := lib$(PVER)-minimal
@@ -722,8 +724,8 @@
-find $(d)/usr/lib/python$(VER) -name '*_failed*.so'
find $(d)/usr/lib/python$(VER) -name '*_failed*.so' | xargs -r rm -f
- : # FIXME: don't ship ensurepip for now
- rm -rf $(d)/usr/lib/python$(VER)/ensurepip
+ : # Don't ship the bundled wheels.
+ rm -f $(d)/usr/lib/python$(VER)/ensurepip/_bundled/*.whl
for i in $(d)/$(scriptdir)/lib-dynload/*.so; do \
b=$$(basename $$i .cpython-34m.so); \
@@ -843,6 +845,14 @@
usr/share/man/man1/python$(VER).1 \
usr/share/man/man1/python$(VER)m.1
+ : # Move pyvenv-X.Y in $(p_venv)
+ dh_installdirs -p$(p_venv) \
+ usr/bin \
+ usr/share/man/man1
+ DH_COMPAT=2 dh_movefiles -p$(p_venv) --sourcedir=$(d) \
+ usr/bin/pyvenv-$(VER) \
+ usr/share/man/man1/pyvenv-$(VER).1
+
rv=0; \
for i in $(MIN_EXTS); do \
if [ -f $(d)/$(scriptdir)/lib-dynload/$$i.so ]; then \
@@ -901,6 +911,7 @@
$(scriptdir)/doc/html
cp -p Misc/HISTORY Misc/README.valgrind Misc/gdbinit \
debian/README.maintainers \
+ debian/README.venv \
debian/test_results $(buildd_static)/pybench.log \
$(d_dev)/usr/share/doc/python$(VER)/
