Hi all,
We need to make a decision about the packaging module in Python 3.3.
Please read this message and breathe deeply before replying :)
[Sorry this ends up being so long; Tarek, Georg, Guido, I hope you
have the time to read it.]
Let me first summarize the history of packaging in the standard
library. (Please correct if I misremember something; this email
expresses my opinion and I did not talk with Tarek or Alexis before
writing it.) Two years ago the distutils2 (hereafter d2) project was
started outside of the stdlib, to allow for fast-paced changes, releases
and testing before merging it back. Changes in distutils were reverted
to go back to misfeature-for-misfeature compatibility (but not
bug-for-bug: bug fixes are allowed, unless we know for sure everyone is
depending on them or working around them).
Tarek’s first hope was to have something that could be included in
2.7 and 3.2, but these deadlines came too fast. At one point near the
start of 2011 (didn’t find the email) there was a discussion with Martin
about adding support for the stable ABI or parallel builds to distutils,
in which Tarek and I opposed adding this new feature to distutils as per
the freeze policy, and Martin declared he was not willing to work
outside of the standard library. We (d2 developers and python-dev) then
quickly agreed that distutils2 would be merged back after the release of
3.2, which was done. There was no PEP requested for this addition,
maybe because this was not a fully new module but an improvement of an
existing one with real-world-tested features, or maybe just because
nobody thought about the process. In retrospect, a PEP would have
really helped define the scope of the changes and the roadmap for packaging.
Real life caused contributors to come and go, and the primary
maintainer (Tarek at first, me since last December) to be at times very
busy (like me these last three months), with the result that packaging
is in my opinion just not ready. Many big and small things need more
work: the three packaging PEPs implemented in d2 have small flaws or
missing pieces (I’m not repeating the list here to avoid spawning
subthreads) that need to be addressed, we’ve started to get feedback
from users and developers only recently (pip and buildout devs since
last PyCon for example) the public Python API of d2 is far from great,
the implementation is of very unequal quality, important features have
patches that are not fully ready (—and I do acknowledge that I am the
blocker for reviews on many of them—), the compiler system has not been
revised, tests are not all clear and robust, some of the bdist commands
need to be removed, a new bdist format needs to be designed, etc.
With beta coming, a way to deal with that unfortunate situation needs
to be found. We could (a) grant an exception to packaging to allow
changes after beta1; (b) keep packaging as it is now under a provisional
status, with due warnings that many things are expected to change; (c)
remove the unstable parts and deliver a subset that works (proposed by
Tarek to the Pyramid author on distutils-sig); (d) not release packaging
as part of Python 3.3 (I think that was also suggested on distutils-sig
last month).
I don’t think (a) would give us enough time; we really want a few
months (and releases) to hash out the API (most notably with the pip and
buildout developers) and clean the bdist situation. Likewise (c) would
require developer (my) time that is currently in short supply. (b) also
requires time and would make development harder, not to mention probable
user pain. This leaves (d), after long reflection, as my preferred
choice, even though I disliked the idea at first (and I fully expect
Tarek to feel the same way).
I’d like to stress that this is not as bad as it appears at first.
We (I) will have to craft reassuring wording to explain why 3.3b1 does
not include packaging any more, but I believe that it would be worse for
our users (code-wise and PR-wise) to deliver a half-finished version in
3.3 rather than to retract it and wait for 3.4. And if we keep in mind
that many people are still using a 2.x version, releasing in 3.3 or 3.4
makes no change for them: the standalone releases on PyPI will keep coming.
Developer-wise, this would *not* mean that the considerable effort
that went into porting and merging, and the really appreciated patches
from developers such as Vinay, would have been in vain: even if
packaging is removed from the main repo (or just from the release
systems), there would be a clone to continue development, or the module
would be added back right after the 3.3 release, or we would develop in
the d2 repo and merge it back when it’s ready—this is really an
implementation detail for the decision; my point is that the work will
not be lost.
Thanks for reading; please express your opinion (especially Tarek as
d2 project lead, Georg as RM and Guido as BDFL).
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
