Re: [Distutils] buildout bootstrap.py doesn't work on Sabayon Linux with system python
On Thu, Aug 22, 2013 at 09:07:52PM -0700, Benedek Zoltan wrote:
> I've downloaded bootstrap.py and tried to initialize with system python:
>
> sabd1@sab /home/buildout $ wget
> http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py
FWIW that's an old version. You should be using one of
wget http://downloads.buildout.org/1/bootstrap.py
wget http://downloads.buildout.org/2/bootstrap.py
to get a bootstrap for zc.buildout 1.7.x or 2.2, respectively.
If you don't know which one you want, use 2.
> sabd1@sab /home/buildout $ python bootstrap.py
> Downloading
> http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg
>
> Traceback (most recent call last):
>
...
> pkg_resources.VersionConflict: (setuptools 0.6c11
> (/tmp/tmpzJN6Tt/setuptools-0.6c11-py2.7.egg),
> Requirement.parse('setuptools>=0.7'))
>
> I know, it works with virtualenv, but with system python is this expected
> behavior?
Basically, yes. At least it's what I've come to expect.
Here's my fool-proof method of setting up buildouts on the brave new
post-setuptools-0.7 world:
virtualenv python
python/bin/pip install -U setuptools
python/bin/python bootstrap.py
bin/buildout
You'll notice that I upgrade setuptools in the virtualenv I created.
That's because I'm using python-virtualenv from Ubuntu, and it installs
an old copy of distribute in the virtualenv by default. Then bootstrap
tries to upgrade it to the newest distribute, which is a shim that
depends on new setuptools, but bootstrap's upgrader isn't smart enough
to go and fetch dependencies so it all breaks down in the same error you've
seen. Upgrading setuptools with pip avoids this failure.
Marius Gedminas
--
Beware of bugs in the above code; I have only proved it correct, not tried it.
-- Donald Knuth
signature.asc
Description: Digital signature
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
On 23 August 2013 05:47, Chris Barker - NOAA Federal wrote: > On Thu, Aug 22, 2013 at 8:17 PM, Nick Coghlan wrote: > > > numpy-1.7.1-cp27-cp22m-win32.whl > > numpy-1.7.1-cp27-cp22m-win32-sse.whl > > numpy-1.7.1-cp27-cp22m-win32-sse2.whl > > numpy-1.7.1-cp27-cp22m-win32-sse3.whl > > I'm still confused -- how would "pip install numpy" know which of > these to install? > The cp27/cp22m/win32 bits are selected automatically based on target platform (it's the "tags" mechanism, see the Wheel PEP) Nick is suggesting that the version with no sse will always be selected by default, but the user can supply some sort of command line argument saying effectively "if you have a sse2 variant, give me that one instead". Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
On 23 Aug, 2013, at 5:17, Nick Coghlan wrote: > > That said, I'm considering the idea of adding a "variant" field to the > compatibility tags for wheel 1.1, along the lines of what Oscar > Benjamin suggested earlier. By default, installers would only find > wheels without a variant defined, but users could opt in to looking > for particular variants. The meaning of the variants field would be > entirely distribution specific. Numpy, for example, could publish: > > numpy-1.7.1-cp27-cp22m-win32.whl > numpy-1.7.1-cp27-cp22m-win32-sse.whl > numpy-1.7.1-cp27-cp22m-win32-sse2.whl > numpy-1.7.1-cp27-cp22m-win32-sse3.whl > > The only restrictions on the variant tags would be: > 1. must be ASCII > 2. must not contain '.' or '-' characters > > You could even go to the extent of using hashes as variant tags. Is adding variants necessary? Numpy use runtime selection for picking the most appropriate extension code (heck, AFAIK recent versions of GCC can even compile multiple variants of functions and pick the right one automaticly). A variant field like this introduces a new failure mode: uses that install packages on one machine and then copy the entire tree to another one where the software crashes hard because the other machine is older and doesn't support some CPU features. That said, I have no experience with directly using SSE as most of my code either doesn't vector calculations or isn't anywhere close to the performance limitations of using naive code (CPU's are too fast ;-) Regards, Ronald ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
On 23 Aug, 2013, at 0:52, Paul Moore wrote:
> On 22 August 2013 23:08, Chris Barker - NOAA Federal
> wrote:
> I want to give it a shot for OS-X -- no one seems to want to maintian
> bdist_mpkg, and it's time to move forward...
>
> My impression is that the architecture and "fat binary" stuff on OSX is the
> bit that may bite you. I know little or nothing about OSX, but I'm sure if
> you try and report on how you get on, the people on the list will be able to
> help you get things sorted and we will be able to get any dark corners ironed
> out.
I don't really expect problems on OSX, I've used binary eggs in the past and
those work just fine. Wheels seem to be simular enough to eggs to not expect
problems there either.
The one thing that might be problematic later on is that distutils (and hence
setuptools and eggs) uses labels for sets of architectures in fat binaries,
while wheels can describe those directly. That is, distutils uses "intel" as
the architecture string for the set {i386, x86_64}, while wheel can use
something like "darwin_i386.darwin_x86_64" (through PEP 425, the actual value
may be different as this is based on a light rereading of the pep). That is an
optional difference and can be ignored for now.
Note that fat binaries are not at all problematic from the point of view of
installing a wheel, a fat binary is a single file that happens to work on
multiple CPU archectures. Creating a structure that would allow for wheels
that support both 32-bit and 64-bit Windows is harder because you'd have two
.pyd files that obviously cannot have the same path in the filesystem or wheel
archive (but easily solved by another level of indirection, such as ".pyext"
directory that can contain extension files whose name is the result of
distutils.util.get_platform()).
Ronald
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
On Aug 23, 2013, at 12:47 AM, Chris Barker - NOAA Federal wrote: > On Thu, Aug 22, 2013 at 8:17 PM, Nick Coghlan wrote: > >> numpy-1.7.1-cp27-cp22m-win32.whl >> numpy-1.7.1-cp27-cp22m-win32-sse.whl >> numpy-1.7.1-cp27-cp22m-win32-sse2.whl >> numpy-1.7.1-cp27-cp22m-win32-sse3.whl > > I'm still confused -- how would "pip install numpy" know which of > these to install? Most likely pip would just ignore anything that has a variant it doesn't understand. So it would only see the first one as an available download. However an alternative installer that understands the variants could use the additional information to select one of the SSE optimized downloads. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
On Thu, Aug 22, 2013 at 8:17 PM, Nick Coghlan wrote: > numpy-1.7.1-cp27-cp22m-win32.whl > numpy-1.7.1-cp27-cp22m-win32-sse.whl > numpy-1.7.1-cp27-cp22m-win32-sse2.whl > numpy-1.7.1-cp27-cp22m-win32-sse3.whl I'm still confused -- how would "pip install numpy" know which of these to install? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] buildout bootstrap.py doesn't work on Sabayon Linux with system python
Hi,
I've downloaded bootstrap.py and tried to initialize with system python:
sabd1@sab /home/buildout $ wget
http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py
Warning: wildcards not supported in HTTP.
--2013-08-23 06:40:52--
http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py
Resolving svn.zope.org... 74.84.203.155
Connecting to svn.zope.org|74.84.203.155|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/x-python]
Saving to: ‘bootstrap.py’
[ <=> ] 10,107
41.2KB/s in 0.2s
2013-08-23 06:40:53 (41.2 KB/s) - ‘bootstrap.py’ saved [10107]
sabd1@sab /home/buildout $ python bootstrap.py
Downloading
http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg
Traceback (most recent call last):
File "bootstrap.py", line 258, in
ws.require(requirement)
File "/tmp/tmpzJN6Tt/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 666,
in require
File "/tmp/tmpzJN6Tt/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 569,
in resolve
pkg_resources.VersionConflict: (setuptools 0.6c11
(/tmp/tmpzJN6Tt/setuptools-0.6c11-py2.7.egg),
Requirement.parse('setuptools>=0.7'))
I know, it works with virtualenv, but with system python is this expected
behavior?
Thanks
Zoltan Benedek___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
On Aug 22, 2013, at 11:17 PM, Nick Coghlan wrote: > - qualified yes for publication on PyPI (i.e. "there may still be > rough edges, so don't be too surprised if this still has flaws at this > point, especially on OS X and Linux") PyPI won't even accept binary Wheels for Linux or OSX at the moment. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
On 23 August 2013 08:03, Chris Barker - NOAA Federal wrote: > On Thu, Aug 22, 2013 at 9:37 AM, Paul Moore wrote: > >> That is essentially possible now. >> >> 1. Go to Christoph Gohlke's website and download his bdist_wininst >> installers for numpy and scipy. > > > > Exactly. And when all this settles down, hopefully Christoph, and > others, will put up binary wheels and we'll have one stop installing > that supports virtualenv, and pypi discover of "good enough" binary > wheels. Right - this is exactly my ambition. Have pip+wheel+virtualenv provide a "good enough" out-of-the-box experience, but not necessarily support a fully optimised experience for any given platform. There are too many different possible full stack integration technologies for it to make sense for us to try to supersede them all - instead, I'd like to provide an 80% cross-platform solution that "plays well enough with others" to cover the remaining 20%. The "others" then includes things like zc.buildout, conda, Linux distro package managers, Microsoft System Centre, automated configuration management tools, PaaS providers like OpenShift and Heroku, container technologies like Docker and the underpinnings of OpenShift. In this space, the goal of the pip+wheel ecosystem will be to make it possible to either use a command like "pip install zc.buildout" or "pip install conda" to bootstrap a cross-platform toolchain, or else to use a platform specific toolchain (like pyp2rpm) to *consume* the upstream packages and produce nice, policy compliant, packages automatically. Managing arbitrary binary dependencies adds a lot of complexity for a capability that I believe the majority of Python projects don't need. Even the ones that can use it (like scientific tools) can often provide a "good enough" fallback option that will fit within the constraints of the draft metadata 2.0 standards. > My point is that is may not be wise to try to support the more complex > builds -- they ARE complex, and trying to suppor it with an auto tool > is a bit much. This is where I think "pip install conda" shows a lot of promise as a good, cross-platform solution, at least in the scientific space. However, the trade-off it makes is that the hash-based packaging system means you *always* pin your dependencies when building a package, with all the downsides that entails (mainly in increasing the complexity of security updates). It's just that in the scientific space, easily reproducing a particular software stack will often trump the desire to make security updates easy to deploy with minimal impact on other components. That said, I'm considering the idea of adding a "variant" field to the compatibility tags for wheel 1.1, along the lines of what Oscar Benjamin suggested earlier. By default, installers would only find wheels without a variant defined, but users could opt in to looking for particular variants. The meaning of the variants field would be entirely distribution specific. Numpy, for example, could publish: numpy-1.7.1-cp27-cp22m-win32.whl numpy-1.7.1-cp27-cp22m-win32-sse.whl numpy-1.7.1-cp27-cp22m-win32-sse2.whl numpy-1.7.1-cp27-cp22m-win32-sse3.whl The only restrictions on the variant tags would be: 1. must be ASCII 2. must not contain '.' or '-' characters You could even go to the extent of using hashes as variant tags. > Oscar wrote: >> And actually 'pip wheel numpy' works fine for me on Windows with MinGW > installed. > > Good start, but the bigger issue is that 'pip install' finds that wheel... > > I'm still confused as to the state of all this -- are the tools ready > for project to start posting wheels so that pip can find them? As others have noted: - definite yes for building your own wheels for internal use (including simple build caching to speed up virtualenv creation) - qualified yes for publication on PyPI (i.e. "there may still be rough edges, so don't be too surprised if this still has flaws at this point, especially on OS X and Linux") Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
On Thu, Aug 22, 2013 at 3:52 PM, Paul Moore wrote: > On 22 August 2013 23:08, Chris Barker - NOAA Federal > My impression is that the architecture and "fat binary" stuff on OSX is the > bit that may bite you. exactly. > I know little or nothing about OSX, but I'm sure if > you try and report on how you get on, the people on the list will be able to > help you get things sorted and we will be able to get any dark corners > ironed out. yup -- we'll never figure it out if no one uses it... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
On 22 August 2013 23:08, Chris Barker - NOAA Federal wrote: > I want to give it a shot for OS-X -- no one seems to want to maintian > bdist_mpkg, and it's time to move forward... > My impression is that the architecture and "fat binary" stuff on OSX is the bit that may bite you. I know little or nothing about OSX, but I'm sure if you try and report on how you get on, the people on the list will be able to help you get things sorted and we will be able to get any dark corners ironed out. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
On Thu, Aug 22, 2013 at 9:37 AM, Paul Moore wrote: > That is essentially possible now. > > 1. Go to Christoph Gohlke's website and download his bdist_wininst > installers for numpy and scipy. Exactly. And when all this settles down, hopefully Christoph, and others, will put up binary wheels and we'll have one stop installing that supports virtualenv, and pypi discover of "good enough" binary wheels. My point is that is may not be wise to try to support the more complex builds -- they ARE complex, and trying to suppor it with an auto tool is a bit much. Oscar wrote: > And actually 'pip wheel numpy' works fine for me on Windows with MinGW installed. Good start, but the bigger issue is that 'pip install' finds that wheel... I'm still confused as to the state of all this -- are the tools ready for project to start posting wheels so that pip can find them? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
On Aug 22, 2013, at 6:08 PM, Chris Barker - NOAA Federal wrote: > but there is little point for a pure-python project. -- pip install > works fine form source for those.. This isn't a true, a pure python wheel is still great. It's an order of magnitude faster with less moving parts and static metadata. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
> C extensions are not ready for use yet anywhere other than on Windows > (Linux in particular has architecture/ABI questions to resolve).. > Projects that use script wrappers probably shouldn't expect wheels to > manage these seamlessly yet (things are still in flux in that area). > "Advanced" features like post-install scripts don't work yet. > The rest is likely fine. > > If you have a simple pure-Python project, or one with a > straightforward C extension, you should be fine. but there is little point for a pure-python project. -- pip install works fine form source for those.. as for a straightforward C extension -- does that only work for Windows now (as above)? I want to give it a shot for OS-X -- no one seems to want to maintian bdist_mpkg, and it's time to move forward... > And it's hard to determine > if anyone is doing things like that internally. Yup -- I'll probably keep it internal at first as well. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
And sent form Paul Moore -- accidentally got off-list (my fault): On 22 August 2013 18:47, Chris Barker - NOAA Federal wrote: > I'm still confused as to the state of all this -- are the tools ready > for projects to start posting wheels so that pip can find them? Basically, the answer is "maybe". Projects that do will still very much be early adopters with all the risks that entails - things could change, users may need help when things don't go quite as expected, etc. C extensions are not ready for use yet anywhere other than on Windows (Linux in particular has architecture/ABI questions to resolve).. Projects that use script wrappers probably shouldn't expect wheels to manage these seamlessly yet (things are still in flux in that area). "Advanced" features like post-install scripts don't work yet. The rest is likely fine. If you have a simple pure-Python project, or one with a straightforward C extension, you should be fine. And of course, the more people that start publishing wheels, the more we'll get feedback on how things are working, and the faster things will settle down. Organisations maintaining internal (or even public) package indexes hosting "unofficial" wheels for projects they have an interest in is probably more realistic at this stage. But I know of no such public indexes (the nearest is Christoph Gohlke's site, but that is not structured to be a PyPI-style index, so it's not usable directly by pip even if he were to switch to wheels). And it's hard to determine if anyone is doing things like that internally. -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
On 22 August 2013 16:33, Chris Barker - NOAA Federal wrote: > On Thu, Aug 22, 2013 at 6:52 AM, Oscar Benjamin > wrote: > >> I'm pretty sure the current Windows installer just doesn't bother with >> BLAS/LAPACK libraries. Maybe it will become possible to expose them >> via a separate wheel-distributed PyPI name one day. > > Well, the rule of thumb with Windows binaries is that you bundle in > (usually via static linking) all the libs you need -- numpy could have > a semi-optimized LAPACK or not, and the user shouldn't know the > difference at install time. But the trick in this case is that numpy > is used by itself, but also widely used with external C and Fortran > that might want LAPACK. (including scipy, in fact...) > > But maybe this is all too much to bite off for pip and wheels. If we > could get to a state where "pip install numpy" and "pip install scipy" > would do something reasonable, if not optimized, I think that would be > great! Agreed. And actually 'pip wheel numpy' works fine for me on Windows with MinGW installed. (I don't even need to patch distutils because numpy.distutils fixed the MinGW bug three years ago!). There's no BLAS/LAPACK support but I assume it has the appropriate SSE2 build which is basically what the win32 superpack gives you. > And it's really not a big deal to say: > > If you want an optimized LAPACK, etc. for your system, you need to do > something special/ by hand/ etc... > > "something special" may be as simple as "download > numpy_optimized_just_for_this_machine.whl and install it with pip. Exactly. Speaking personally, I do all my real computation on Linux clusters managed by scientific software professionals who have hand-tuned and pre-built a bewildering array of alternative BLAS/LAPACK setups and numpy etc. to go with. For me having numpy on Windows is just for developing, debugging etc. so hard-core optimisation isn't important. Oscar ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
On 22 August 2013 16:33, Chris Barker - NOAA Federal wrote: > But maybe this is all too much to bite off for pip and wheels. If we > could get to a state where "pip install numpy" and "pip install scipy" > would do something reasonable, if not optimized, I think that would be > great! And it's really not a big deal to say: > That is essentially possible now. 1. Go to Christoph Gohlke's website and download his bdist_wininst installers for numpy and scipy. 2. Make sure you have pip 1.4+, setuptools and wheel installed (you only need wheel for the wheel convert step) 3. wheel convert numpy-*.exe; wheel convert scipy-*.exe 4. pip install numpy*.whl scipy*.whl You need to manually choose the right wininst installers for your architecture (or you can grab and wheel convert them all and let the architecture tags choose the right ones). I don't believe that Christoph's installers need a postinstall step, so the architecture/instruction set issues don't apply. But I've only visually checked and done some minimal use tests myself. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
On Thu, Aug 22, 2013 at 6:52 AM, Oscar Benjamin wrote: > I'm pretty sure the current Windows installer just doesn't bother with > BLAS/LAPACK libraries. Maybe it will become possible to expose them > via a separate wheel-distributed PyPI name one day. Well, the rule of thumb with Windows binaries is that you bundle in (usually via static linking) all the libs you need -- numpy could have a semi-optimized LAPACK or not, and the user shouldn't know the difference at install time. But the trick in this case is that numpy is used by itself, but also widely used with external C and Fortran that might want LAPACK. (including scipy, in fact...) But maybe this is all too much to bite off for pip and wheels. If we could get to a state where "pip install numpy" and "pip install scipy" would do something reasonable, if not optimized, I think that would be great! And it's really not a big deal to say: If you want an optimized LAPACK, etc. for your system, you need to do something special/ by hand/ etc... "something special" may be as simple as "download numpy_optimized_just_for_this_machine.whl and install it with pip. All that being said -- OS-X has a moderately complex binary set, what with universal binaries, so maybe we can have a bit more meta-data about the architecture supported. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 22 August 2013 16:04, Nick Coghlan wrote: > The next step is up to the pip folks - if they think adopting distlib > wholesale makes sense for them, fine, I have no direct say in that. If > they decide to make a "piplib" instead, to expose a public API for an > updated version of pip's own infrastructure (perhaps derived from > distlib), that's fine by me, too. > For what it's worth, we currently have a vendored copy of distlib bundled into pip but (a) it's pretty out of date now and (b) we only make minimal use of it - in particular we do not use it for any of the wheel support at the moment. I don't have any feel for what we might do going forward - I suspect we'll wait until the dust settles a bit on the whole issue in distutils-sig before trying to make a decision. For virtualenv, I have a longer-term plan to switch to bundling pip and setuptools wheels instead of sdists, but again I don't plan on rushing into a decision on how I'll do that. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On 23 August 2013 00:19, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: >> When I made that suggestion, I misunderstood your plans for distlib. >> If pip are only adopting a subset of it, they can't use the same name, >> or people will get confused. > > I can certainly see that there are ways to avoid confusion. But never mind, > I see that you've made your decision. I would have hoped for a more > transparent decision process, but that's probably due to my slowness of > uptake. The only decision I've made is to stop saying "distlib is the future", since that is now in doubt, and I certainly don't have the time available or expertise needed to review all the APIs that have been added to it (I thought it was just the four distutils2 interfaces that almost made it into Python 3.3 and that all your experimental interfaces were in distil, not distlib. While there was plenty of evidence to indicate I was wrong in that belief, I wasn't paying proper attention to it and it didn't properly register until it came up in this thread). The next step is up to the pip folks - if they think adopting distlib wholesale makes sense for them, fine, I have no direct say in that. If they decide to make a "piplib" instead, to expose a public API for an updated version of pip's own infrastructure (perhaps derived from distlib), that's fine by me, too. The only absolute in this space relates to the default installation toolchain: it *will* be pip. Unlike setutptools as a build system, I consider easy_install irredeemable as an installer (from a social perspective), and there's no way I would ever inflict yet another change of recommended client on the community. Given that, any future changes to the core infrastructure will be heavily influenced by the technical choices of the pip developers. Other tools will exist around that, since packaging is a complex topic where "one size" really doesn't fit all, but pip will be the centrepiece. Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Nick Coghlan gmail.com> writes: > standard library is a mistake - the PyXML debacle shows us that. If > the API is different (even if that means a strict subset), then it > needs a different name. I'm not really hung up about a specific name - what's in a name? > It has nothing to do with code quality, and everything to do with > being able to explain the migration plan to people. I *can* say to Code quality is pertinent when it's the subtext behind "experimental". > them "pip is going to cherry pick parts of distlib and potentially > other libraries and make them available as 'piplib', which will be > installed automatically when you install pip". At the moment, I no > longer feel I can say "distlib will become the reference > implementation". Note that there's also the bootstrapping issue with > having pip depend on an external library: having the core library *in > pip* makes that problem go away. > When I made that suggestion, I misunderstood your plans for distlib. > If pip are only adopting a subset of it, they can't use the same name, > or people will get confused. I can certainly see that there are ways to avoid confusion. But never mind, I see that you've made your decision. I would have hoped for a more transparent decision process, but that's probably due to my slowness of uptake. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
On 22 August 2013 12:57, Vinay Sajip wrote: >> I think that the installer ships variants for each architecture and >> decides at install time which to place on the target system. If that's >> the case then would it be possible for a wheel to ship all variants so >> that a post-install script could sort it out (rename/delete) after the >> wheel is installed? > > It's not just about the architecture on the target system, it's also about > e.g. what libraries are installed on the target system. Files like > numpy/__config__.py and numpy/distutils/__config__.py are created at build > time, based on local conditions, and those files would then be written to > the wheel. On the installation machine, the environment may not be > compatible with those configurations computed on the build machine. Those > are the things I was talking about which may need moving from build-time to > run-time computations. I'm pretty sure the current Windows installer just doesn't bother with BLAS/LAPACK libraries. Maybe it will become possible to expose them via a separate wheel-distributed PyPI name one day. That would help since they're currently not very easy to setup/build on Windows but the same sse etc. issues would apply to them as well. For now just leaving out BLAS/LAPACK is probably okay. apt-get doesn't bother to install them for numpy either (on Ubuntu). It will set them up properly if you explicitly ask for them though. Oscar ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Disclaimer: everything I say below about pip is ultimately up to the pip devs. I'm just pointing out what I think makes sense, and my reading of Donald's comments means that I expect he would feel the same way. On 22 August 2013 17:22, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: >> What that means though, is we need *something else* that indicates the >> common core that people can assume will always be available. It's this > > If that "something else" you're thinking of is something that is supposed to > live in the stdlib, then I see no reason why a subset of distlib couldn't be > that something else, since stdlib changes are not on the table for 3.4. I > certainly have never envisaged that distlib would be adopted wholesale into > the stdlib (if at all) without peer review and any changes coming out of that. I *can't* tell people "we're going to migrate to distlib as the reference packaging infrastructure implementation" when I really mean "we're going to migrate to some as yet undefined subset of distlib, so 'import distlib' won't be the long term answer". Incorporating only a subset of an existing published API into the standard library is a mistake - the PyXML debacle shows us that. If the API is different (even if that means a strict subset), then it needs a different name. >> common core which pip will need to factor out to remove their >> dependency on setuptools, rather than adopting distlib wholesale, >> experimental features and all. > > I honestly think you're making a bit too much of the "experimental" label > here, even though it is a label that I use myself. For me, that label is > most appropriate for the extended metadata that I collect from PyPI and > which is the basis for distlib's smarter dependency resolution. > > If your concerns are about instability due to experimental features (and I > quite understand the importance of stability in packaging), then there's > nothing stopping anyone doing a technical review of distlib to see what any > actual risks are. Indeed, I've invited such review from day one. It has nothing to do with code quality, and everything to do with being able to explain the migration plan to people. I *can* say to them "pip is going to cherry pick parts of distlib and potentially other libraries and make them available as 'piplib', which will be installed automatically when you install pip". At the moment, I no longer feel I can say "distlib will become the reference implementation". Note that there's also the bootstrapping issue with having pip depend on an external library: having the core library *in pip* makes that problem go away. >> Currently, pip doesn't expose a programmatic API. I suggested to >> Donald that it may make sense to start exposing one as "piplib". The > > I think this would be a mistake, and it seems a little early to make this > sort of decision. You've given me to understand that pip could at some > future point use (some subset of) distlib under the covers, with > compatibility maintained at the CLI level. If that is still the case, then I > don't see much value in having two lib layers. When I made that suggestion, I misunderstood your plans for distlib. If pip are only adopting a subset of it, they can't use the same name, or people will get confused. I now think it makes more sense for pip to migrate to a more tightly constrained public library API, that doesn't go beyond the documented metadata standards (except for legacy compatibility reasons). > Like setuptools, pip has done sterling service, but it's not a codebase I'd > like to see become the basis for our long-term packaging infrastructure. I > don't mean to offend anybody by saying that - it's just software that has > grown organically over time and IMO there will be technical debt to repay if > we go down the route of exposing bits of it as Python APIs. I don't expect the contents of piplib to match the contents of the existing pip module. This is about enabling a gradual refactoring over to a cleaner core library with a public API, rather than a big bang migration to an alternative solution like distlib. > It certainly feels like you're side-lining distlib, or planning to, whether > or not that's the message you're intending to send. No matter :-) I don't currently believe your plans for distlib and my plans for the standard library software distribution support (whether directly in the standard library or via the pip bootstrapping) are compatible. If I am correct, then distlib remains extemely valuable in that scenario, but the nature of its role changes. I want a completely barebones "absolutely no features that aren't covered by an Accepted PEP" library as a candidate for future inclusion. Such a library *cannot* be particularly useful at this point in time, because the metadata 2.0 PEPs are nowhere near complete enough for that. By contrast, you understandably wish for distlib to be useful *now*, which means running ahead of the standardisation proc
Re: [Distutils] State of the wheel spec (Was: How to handle launcher script importability?)
On Thu, Aug 22, 2013 at 4:24 AM, Paul Moore wrote: > On 21 August 2013 23:20, Vinay Sajip wrote: >> >> Nick Coghlan gmail.com> writes: >> >> > Um, the current wheel spec uses PEP 345 + setuptools metadata only. If >> distlib is expecting PEP 426 metadata in wheel files, it is not compliant >> with >> the spec. >> >> I can certainly rectify that - I was possibly confused by the fact that >> the >> latest wheel implementation writes pydist.json to the wheel (though the >> Wheel- >> Version in WHEEL is still 1.0). > > > Conversely, of course, there's no mention in the wheel spec that setuptools > metadata (specifically entry-points.txt) should be present. Which is why I > mentioned that the wheel spec might need a review/update to clarify this (if > we want to ensure that any necessary script metadata is guaranteed to be > present in compliant wheels). pydist.json is in there in order to have an implementation / old metadata converter to inform PEP 426 development. I added the "generator" tag to PEP 426 to deal with the problem of detecting pydist.json that conform to obsolete drafts of the spec. In the meantime the stable metadata is what setuptools supports inside .dist-info directories which is similar to old-draft key/value PEP 426, including Provides-Extra: etc. Apart from the upcoming wrapper scripts generation, the basic wheel install step shouldn't need to read any of the setuptools/PEP 345/426 metadata at all. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distutils.util.get_platform() - Linux vs Windows
> -Original Message- > From: Ronald Oussoren [mailto:ronaldousso...@mac.com] > Sent: Thursday, August 22, 2013 12:21 PM > To: Ferencik, Samuel: Markets (PRG) > Cc: chris.bar...@noaa.gov; distutils-sig@python.org > Subject: Re: [Distutils] distutils.util.get_platform() - Linux vs Windows > > > On 20 Aug, 2013, at 18:00, samuel.feren...@barclays.com wrote: > >>> -Original Message- >>> From: Chris Barker - NOAA Federal [mailto:chris.bar...@noaa.gov] >>> Sent: Tuesday, August 20, 2013 5:47 PM >>> To: Ferencik, Samuel: Markets (PRG) >>> Cc: distutils-sig@python.org >>> Subject: Re: [Distutils] distutils.util.get_platform() - Linux vs Windows >>> >>> On Mon, Aug 19, 2013 at 11:15 PM, wrote: >>> What does your 'uname -m' return? >>> >>> x86_64 >>> Is it possible you're really running a 32-bit Python on a *32-bit* OS X kernel? [http://superuser.com/q/161195] >>> >>> nope -- I am quite deliberately running a 32 bit Python on my 64 bit >>> OS (I have some custom code C++ I"m using that is not yet 64 bit >>> safe). >> >> That's strange. I'm on Python 3.3.1, and it seems to me that get_platform() >> derives the value from uname for OS X, similar to Linux. >> >>(osname, host, release, version, machine) = os.uname() >>... >>elif osname[:6] == "darwin": >>import _osx_support, distutils.sysconfig >>osname, release, machine = _osx_support.get_platform_osx( >>distutils.sysconfig.get_config_vars(), >>osname, release, machine) >>return "%s-%s-%s" % (osname, release, machine) >> >> so I would expect "uname -m" to be in line with get_plaform(). But maybe I'm >> misreading that... Also, I don't have access to the _osx_support source code. > > _osx_support is a pure python module in the stdlib, the source is in the > usual > location. Of course it is. I don't know where I was looking. Basically, get_platform_osx() overrides the value of 'machine' passed in. So in distutils.util.get_platform() it looks like it's doing a similar thing as for Linux (uname) but it then throws it away and lets _osx_support.get_platform_osx() do its own thing. > The behavior on OSX is quite intentional and ensures that disutils binary > archive > names correctly reflect the use of fat binaries and the minimal supported OSX > release. > > The only thing that might need change is the name of the supported > architectures, > the wheel spec has a better way to indicate multiple executable architectures > than > making up names for every set of architectures that we care to support, but > to be > honest I haven't had time yet to fully ingest the spec and work out if is > completely > useful for fat binaries on OSX. > >> >>> return value is wrong on Linux and correct on Windows, right? >>> >>> no -- I'm saying that it's right on Windows (and OS-X), but wrong on Linux. >> >> I think you have misread my sentence, and we actually agree here. >> >> What's the next action? Report a Python bug? (That's a cultural question; I'm >> new to Python.) > > http://bugs.python.org/ Thanks, I'll report one. Sam ___ This message is for information purposes only, it is not a recommendation, advice, offer or solicitation to buy or sell a product or service nor an official confirmation of any transaction. It is directed at persons who are professionals and is not intended for retail customer use. Intended for recipient only. This message is subject to the terms at: www.barclays.com/emaildisclaimer. For important disclosures, please see: www.barclays.com/salesandtradingdisclaimer regarding market commentary from Barclays Sales and/or Trading, who are active market participants; and in respect of Barclays Research, including disclosures relating to specific issuers, please see http://publicresearch.barclays.com. ___ ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
Oscar Benjamin gmail.com> writes: > I think that the installer ships variants for each architecture and > decides at install time which to place on the target system. If that's > the case then would it be possible for a wheel to ship all variants so > that a post-install script could sort it out (rename/delete) after the > wheel is installed? It's not just about the architecture on the target system, it's also about e.g. what libraries are installed on the target system. Files like numpy/__config__.py and numpy/distutils/__config__.py are created at build time, based on local conditions, and those files would then be written to the wheel. On the installation machine, the environment may not be compatible with those configurations computed on the build machine. Those are the things I was talking about which may need moving from build-time to run-time computations. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
On 21 August 2013 22:22, Paul Moore wrote: > On 21 August 2013 22:13, Nick Coghlan wrote: >> >> Wheel is a suitable replacement for bdist_wininst (although anything that >> needs install hooks will have to wait for wheel 1.1, which will support >> metadata 2.0). It's just not a replacement for what hashdist and conda let >> you do when you care more about reproducibility than you do about security >> updates. > > OK, that's a good statement - wheels as a better bdist_wininst is all I want > to be able to promote (and yes, if you need post-install hooks, wait for > wheel 1.1). Okay, so going back to my earlier question... Oscar asked: > BTW is there any reason for numpy et al not to start distributing > wheels now? Is any part of the wheel > specification/tooling/infrastructure not complete yet? the answer is basically yes to both questions. The pip+PyPI+wheel infrastructure is not yet able to satisfy numpy's needs as the wheel spec doesn't give sufficiently fine-grained architecture information and there's no way to monkey-patch the installation process in order to do what the current installers do. It seems to me that the ideal solution for numpy is not really the post-install script but a way to distribute wheels appropriate to the given CPU. Bundling the different binaries in one installer makes sense for an installer that is manually downloaded by a user but not for one that is automatically downloaded. There's a pure Python script here that seems to be able to obtain the appropriate information: https://code.google.com/p/numexpr/source/browse/numexpr/cpuinfo.py?r=ac92866e7929df669ca5e4e050179cd7448798f0 $ python cpuinfo.py CPU information: CPUInfoBase__get_nbits=32 getNCPUs=2 has_mmx has_sse2 is_32bit is_Core2 is_Intel is_i686 So perhaps numpy could upload multiple wheels: numpy-1.7.1-cp27-cp22m-win32.whl numpy-1.7.1-cp27-cp22m-win32_sse.whl numpy-1.7.1-cp27-cp22m-win32_sse2.whl numpy-1.7.1-cp27-cp22m-win32_sse3.whl Then ordinary pip would just install the win32 wheel but "fancypip" could install the one with the right level of sse2 support. Or is there perhaps a way that a distribution like numpy could depend on another distribution that finds CPU information and informs or hooks into pip etc. so that pip would be able to gain this support in an extensible way? Oscar ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distutils.util.get_platform() - Linux vs Windows
On 20 Aug, 2013, at 18:00, samuel.feren...@barclays.com wrote: >> -Original Message- >> From: Chris Barker - NOAA Federal [mailto:chris.bar...@noaa.gov] >> Sent: Tuesday, August 20, 2013 5:47 PM >> To: Ferencik, Samuel: Markets (PRG) >> Cc: distutils-sig@python.org >> Subject: Re: [Distutils] distutils.util.get_platform() - Linux vs Windows >> >> On Mon, Aug 19, 2013 at 11:15 PM, wrote: >> >>> What does your 'uname -m' return? >> >> x86_64 >> >>> Is it possible you're really running a 32-bit >>> Python on a *32-bit* OS X kernel? [http://superuser.com/q/161195] >> >> nope -- I am quite deliberately running a 32 bit Python on my 64 bit >> OS (I have some custom code C++ I"m using that is not yet 64 bit >> safe). > > That's strange. I'm on Python 3.3.1, and it seems to me that get_platform() > derives the value from uname for OS X, similar to Linux. > >(osname, host, release, version, machine) = os.uname() >... >elif osname[:6] == "darwin": >import _osx_support, distutils.sysconfig >osname, release, machine = _osx_support.get_platform_osx( >distutils.sysconfig.get_config_vars(), >osname, release, machine) >return "%s-%s-%s" % (osname, release, machine) > > so I would expect "uname -m" to be in line with get_plaform(). But maybe I'm > misreading that... Also, I don't have access to the _osx_support source code. _osx_support is a pure python module in the stdlib, the source is in the usual location. The behavior on OSX is quite intentional and ensures that disutils binary archive names correctly reflect the use of fat binaries and the minimal supported OSX release. The only thing that might need change is the name of the supported architectures, the wheel spec has a better way to indicate multiple executable architectures than making up names for every set of architectures that we care to support, but to be honest I haven't had time yet to fully ingest the spec and work out if is completely useful for fat binaries on OSX. > >> >>> return value is wrong on Linux and correct on >>> Windows, right? >> >> no -- I'm saying that it's right on Windows (and OS-X), but wrong on Linux. > > I think you have misread my sentence, and we actually agree here. > > What's the next action? Report a Python bug? (That's a cultural question; I'm > new to Python.) http://bugs.python.org/ Ronald ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] distutils.util.get_platform() - Linux vs Windows
On 20 Aug, 2013, at 8:15, samuel.feren...@barclays.com wrote: >> -Original Message- >> From: Chris Barker - NOAA Federal [mailto:chris.bar...@noaa.gov] >> Sent: Monday, August 19, 2013 7:13 PM >> To: Ferencik, Samuel: Markets (PRG) >> Cc: distutils-sig@python.org >> Subject: Re: [Distutils] distutils.util.get_platform() - Linux vs Windows >> >> On Fri, Aug 16, 2013 at 2:18 AM, wrote: >>> It seems distutils.util.get_platform() semantically differs on Windows and >>> Linux. >>> >>> Windows: the return value is derived from the architecture of the >>> *interpreter*, hence for 32-bit Python running on 64-bit Windows >>> get_platform() = 'win32' (32-bit). >>> >>> Linux: the return value is derived from the architecture of the *OS*, hence >>> for 32-bit Python running on 64-bit Linux get_platform() = 'linux-x86_64' >>> (64-bit). >>> >>> Is this intentional? >> >> This seems just plain wrong to me. >> >> For the record, running a 32 bit Python on a 64 bit OS_X box: >> >> In [5]: distutils.util.get_platform() >> Out[5]: 'macosx-10.6-i386' >> >> which is the answer I want. >> >> -Chris > > Chris, > > What does your 'uname -m' return? Is it possible you're really running a > 32-bit > Python on a *32-bit* OS X kernel? [http://superuser.com/q/161195] disutils.util.get_platform() on OSX returns the "architecture" supported by the current binary. I get: :>>> distutils.util.get_platform() 'macosx-10.8-intel' This means that Python was build for a deployment target of 10.8 (that is, the binary runs on OSX 10.8 or later) and supports the 'intel' set of architecures (i386 and x86_64). Ronald ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Am 20.08.2013 19:39, schrieb PJ Eby: I thought that at one point you (Thomas) had come up with a way to load modules into memory from a zipfile without needing to extract them. Was that you? If so, how did that work out? To give a definite answer, after thinking it over: It works, for quite some extensions. The main problem is this: If it does NOT work (process crashes) there is no way to find out why. It is nearly impossible to debug because you end up with all the machine code from the extensions/dlls mapped into the process and the debugger has no info about it. Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] State of the wheel spec (Was: How to handle launcher script importability?)
On 21 August 2013 23:20, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: > > > Um, the current wheel spec uses PEP 345 + setuptools metadata only. If > distlib is expecting PEP 426 metadata in wheel files, it is not compliant > with > the spec. > > I can certainly rectify that - I was possibly confused by the fact that the > latest wheel implementation writes pydist.json to the wheel (though the > Wheel- > Version in WHEEL is still 1.0). Conversely, of course, there's no mention in the wheel spec that setuptools metadata (specifically entry-points.txt) should be present. Which is why I mentioned that the wheel spec might need a review/update to clarify this (if we want to ensure that any necessary script metadata is guaranteed to be present in compliant wheels). Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] State of the wheel spec (Was: How to handle launcher script importability?)
Nick Coghlan gmail.com> writes: > However, the pydist.json that wheel currently writes is in the > category of "arbitrary additional metadata in the dist-info > directory", since the metadata 2.0 spec is still far from stable. You can perhaps see why that could cause confusion - was that mentioned somewhere on this list? It certainly seems odd to add a pydist.json there if it's not needed; the natural thing to do is to assume that if it's there, it's usable. Unfortunately, the pydist.json that it currently writes is not conformant to the latest version of the PEP (though it passes schema validation) :-( Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
Nick Coghlan gmail.com> writes: > I previously thought distlib was going to be the repository for the > agreed, stable, "this is going to happen" stuff. It's OK that I was > wrong - I think you're right that somewhere is needed as an > experimental location to show some of the *possibilities* of the new > metadata, and to seed ideas for making it into the eventual standard > base that people can assume is readily available. It's not just about completely new, experimental stuff. For example, the resources functionality isn't completely new territory. The PyPI interfacing is (IMO) a saner API than the one in distutils2. A better Windows story (for when launcher support when py.exe can't be used) is also not rocket science. > What that means though, is we need *something else* that indicates the > common core that people can assume will always be available. It's this If that "something else" you're thinking of is something that is supposed to live in the stdlib, then I see no reason why a subset of distlib couldn't be that something else, since stdlib changes are not on the table for 3.4. I certainly have never envisaged that distlib would be adopted wholesale into the stdlib (if at all) without peer review and any changes coming out of that. > common core which pip will need to factor out to remove their > dependency on setuptools, rather than adopting distlib wholesale, > experimental features and all. I honestly think you're making a bit too much of the "experimental" label here, even though it is a label that I use myself. For me, that label is most appropriate for the extended metadata that I collect from PyPI and which is the basis for distlib's smarter dependency resolution. If your concerns are about instability due to experimental features (and I quite understand the importance of stability in packaging), then there's nothing stopping anyone doing a technical review of distlib to see what any actual risks are. Indeed, I've invited such review from day one. > Currently, pip doesn't expose a programmatic API. I suggested to > Donald that it may make sense to start exposing one as "piplib". The I think this would be a mistake, and it seems a little early to make this sort of decision. You've given me to understand that pip could at some future point use (some subset of) distlib under the covers, with compatibility maintained at the CLI level. If that is still the case, then I don't see much value in having two lib layers. Like setuptools, pip has done sterling service, but it's not a codebase I'd like to see become the basis for our long-term packaging infrastructure. I don't mean to offend anybody by saying that - it's just software that has grown organically over time and IMO there will be technical debt to repay if we go down the route of exposing bits of it as Python APIs. It certainly feels like you're side-lining distlib, or planning to, whether or not that's the message you're intending to send. No matter :-) Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
