Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Nick Coghlan]

On 17 August 2016 at 15:21, Nick Coghlan  wrote:
> While rebranding eggs (or a close derivative) as "pypath entries"
> could help make them more self-explanatory (especially when it comes
> to explaining how the zipped form differs from wheel files), doing so
> wouldn't actually give us any more architectural flexibility in
> practice than the last option.

That said, if folks *did* want to go down this path, then an important
use case to consider is "single archive applications" suitable for use
in services like AWS Lambda and Azure Cloud Functions, where you get a
language runtime to play with, but need to provide everything you want
to run in that environment as a single pre-bundled archive. Similar
problems also show up when building rich client applications for
Windows, Mac OS X, Android and iOS.

This means you want tooling that targets the direct dependency
bundling model, rather than the runtime environment configuration
model favoured by pip and virtualenv.

So this particular challenge *shouldn't* be framed as just a backwards
compatibility and migration concern for buildout et al - the "all
inclusive application bundle" approach is still an important use case,
even though it isn't the lowest common denominator cross-platform
dependency specification, retrieval and installation problem handled
by the default toolchain.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Nick Coghlan]

On 17 August 2016 at 01:15, Leonardo Rochael Almeida
 wrote:
> PS: In the buildout community, we never really understood the impetus for
> replacing egg as a format, which is really not all that complex and not all
> the different from wheel as it stands, instead of just formalizing it in a
> PEP. We got the feeling that, in the movement of getting rid of setuptools
> as an "installation" dependency, we ended up throwing out the baby with the
> bathwater...

The majority of the problems with eggs are due to the fact that they
were designed to operate *by default* as the plugin management system
for Chandler. buildout uses them in a similar way, so they're a good
fit there.

However, they're *not* a good fit for other cases, hence
"--single-version-externally-managed" and all of the other ways in
which pip switches the defaults to be more developer centric, and less
deployment focused. Most of the problems people ran into with the
defaults were actually related to the way easy_install and
pkg_resources worked rather than due to the egg format itself, but the
lack of clear documentation and specifications meant that the broader
adoption process for programmatic dependency management in general
needed to be based on a more modular reimplementation of similar ideas
with more explicit interoperability interfaces, rather than adopting
the setuptools/easy_install/pkg_resources implementation of those
ideas wholesale.

This means that the situation we've managed to get to now is that we
have wheels as a satisfactory replacement for the "binary
distribution" use case, but we haven't even *tried* to replace eggs
for the "standalone path entry" use case.

That gives us a few options for possible ways forward:

- define a new egg-inspired "*.pypath" format for directories and zip
files that are designed to be placed directly on sys.path, rather than
installed
- just bless eggs as the recommended standalone importable format, and
offer a wheel2egg conversion utility and perhaps an "--as-egg" install
option in pip
- as with the previous, but bake the wheel2egg conversion into
setuptools/easy_install rather than into pip

At least for now, I think that last option probably makes the most
sense, as it moves easy_install, buildout, and other egg based tools
into a similar position to conda, PyPM and the Linux distros: a
downstream community with its own binary distribution format, but
associated tooling to also make it easy for participants in that
ecosystem to consume pre-built wheel files if they want to.

While rebranding eggs (or a close derivative) as "pypath entries"
could help make them more self-explanatory (especially when it comes
to explaining how the zipped form differs from wheel files), doing so
wouldn't actually give us any more architectural flexibility in
practice than the last option.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Future of setuptools and buildout

[Paul Moore]

On 16 August 2016 at 17:58, Leonardo Rochael Almeida
 wrote:
> Questions:
>
> 1. Is the future outlined above considered likely?

I can't speak for setuptools, but certainly pip is intended to be the
canonical installer for Python. It's not impossible for competing
tools like the download/install features in setuptools to remain, but
like you say, I'd expect the trend to be against them.

> 2. If this future arrives, what package should buildout be ported to use?

That's really for buildout to decide.

> It should provide:
>
>  * The ability to locate packages and their versions, resolve their
> dependencies, and download them, whether they're sdists or wheels (or eggs,
> while we still have them)

pip will do this (no support for eggs exists or will be added, but
sdists and wheels yes). Note that pip should be run as a command line
tool - we don't support use of pip's internal API from an existing
Python process. If an in-process API is important, though, it wouldn't
be impossible to discuss adding that.

>  * Install each package separately into it's own directory that, if added to
> `sys.path`, allows this same package (or perhaps another one) to read
> information from entry points so that buildout can locate its own extensions
> and install console scripts.

That's not an installer feature. So pip doesn't provide anything
specific for that. Rather, that's a runtime support facility that is
tied to Python's import mechanisms. Setuptools (or rather,
pkg_resources?) may well continue to provide that, but laying out the
files present in a wheel in the format you need to support your
requirements would have to be added - maybe setuptools would be
interested in providing such a facility, I don't know. But it's not
exactly hard, at the most basic level you're probably only talking
about unzipping the wheel and writing a bit of metadata based on
what's specified in the wheel. Heck, a "wheel to egg" function
shouldn't be that hard to write, and then you could simply use the
existing runtime egg support code.

However, I don't believe that mandating that a single package should
provide both requirements is necessary, or helpful. The two features
are different - one of the features of setuptools that many people
(myself included) disliked, was the fact that it mixed "install time"
and "run time" features in one package, making it hard to avoid having
install time capabilities in a runtime-only environment.

Having said this, if someone were to propose, design and standardise a
*runtime* format for whatever you're hinting at as "install into its
own directory ... read information", and get it accepted as an
installation layout standard, then it might make sense to add support
to pip for installing files using that layout (analogous to the
"--target" flag for pip install). Runtime support would still have to
come from elsewhere, though. The design ideas for that spec may even
be based on the ideas developed for egg. But the key focus is moving
away from adhoc, implementation-defined standards, to properly
specified and documented standards. So step 1 would be to get such a
standard accepted, *then* look at proposing pip add it as an
alternative installation layout. (And I'd recommend not calling that
standard "egg" format - like it or not, the historical baggage
associated with that name will make it harder than you'd like to get
adoption of a new standard).

Hope this helps,
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Future of setuptools and buildout

[Donald Stufft]

> On Aug 16, 2016, at 12:58 PM, Leonardo Rochael Almeida  
> wrote:
> 
> Spinning this off into its own thread...
> 
> On 16 August 2016 at 13:10, Donald Stufft  > wrote:
> 
>> On Aug 16, 2016, at 11:15 AM, Leonardo Rochael Almeida > > wrote:
>> 
>> Specifically, buildout right now has setuptools as its only dependence, and 
>> buildout uses it to locate, download and build sdist dependencies, or 
>> directly download and use .egg dependencies, which are important for Windows 
>> platforms in the communities that use buildout and depend on compiled 
>> extensions.
> 
> 
> Ah, I knew I was forgetting something. I think we should hold off on 
> preventing egg uploads until setuptools can download and install wheels.
> 
> Which brings us to a question that I'm meaning to ask for a while.
> 
> It looks like we're close to removing all mentions of setuptools in pip. When 
> this happens, it looks like pressure is going to start to mount on setuptools 
> to drop the ability to install packages and limit itself on being just a 
> build tool.
> 
> It makes sense that it should be so, considering that there would be no 
> incentive to keep around two distinct implementations of how to locate, 
> download and install stuff, one in pip (or whatever library pip uses for 
> locating and installing packages) and another in setuptools.

So one of our end goals here (and why we’re so stuck on standards for 
everything, and not ad hoc things) is to make it entirely possible to have 
multiple distinct implementations of any part of the tool chain without having 
to try and copy each other’s features/implementation details to work. There’s 
one standard for what is correct and as long as you follow that, then you 
should work.

With that in mind, I certainly hope to get to a point where at the very least, 
setuptools make the conscious choice to continue to implement these features, 
rather than needing to do it because pip depends on it. Whether it makes sense 
for setuptools to continue to support them or not is something that the 
setuptools devs (particularly Jason) is going to be better suited to answer, 
but for my own opinion I hope we can get to a point that setuptools is just a 
build tool.

> 
> In this future, I can imagine setuptools will simply complain loudly when the 
> packages it requires to do its work are not installed yet, perhaps with a 
> plugin mechanism for allowing another package to automatically resolve and 
> installing packages, (like today setuptools-git extends setuptools to 
> recognize files from git to be included in a package).
> 
> And people will promptly write a pip implementation of this plugin (or 
> whatever it is that pip will use as a library to do it)... thereby inverting 
> the dependency between setuptools and pip.
> 
> When this future arrives, buildout will be out of a mechanism for locating, 
> downloading and installing packages (not counting old versions of setuptools, 
> of course).
> 
> Questions:
> 
> 1. Is the future outlined above considered likely?

I suspect that maybe, at some point it might be, but I think that setuptools 
will likely keep it’s support for these things for a good long while to avoid 
breaking things for people. At least until a satisfactory answer for what 
someone who is currently using setuptools should use emerges.

> 
> 2. If this future arrives, what package should buildout be ported to use? It 
> should provide:
> 
>  * The ability to locate packages and their versions, resolve their 
> dependencies, and download them, whether they're sdists or wheels (or eggs, 
> while we still have them)
>  * Install each package separately into it's own directory that, if added to 
> `sys.path`, allows this same package (or perhaps another one) to read 
> information from entry points so that buildout can locate its own extensions 
> and install console scripts.
> 


So you have a few choices, right now you can use pip in a subprocess (pip does 
not expose a programmatic API) or continue to use setuptools.

One of the things we’ve been trying to do with pip is instead of 
adding/exposing parts of pip via a programmatic API, is to instead extract them 
out into their own dedicated libraries that then pip consumes. This has the 
benefit that setuptools can depend on those things, instead of pip itself, but 
also that other people can come along and piece meal use the pieces that make 
sense for them. If someone has a great new idea for a pip-killer that does 
everything better, then they can do that, and reuse these libraries to help 
make their thing better right out of the starting gate.

So we could work on that effort too, I think the main thing holding it back is 
both developer time, as well as solid use cases so we can design the API around 
them. 

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/

Re: [Distutils] Future of setuptools and buildout

[Daniel Holth]

There are two proposals to add pluggable build systems to Python sdists,
and one of them will probably be implemented. You add a pyproject.toml to
the root of your sdist, which the installer uses to install dependencies
that setup.py itself needs to run. Second, you also tell pip which build
system you are using, once you have done that, setup.py is no longer
necessary - instead, pip could install and then invoke flit for example.

This isn't quite the situation you've outlined as setuptools will likely
continue to have all its features. Instead, packages that don't use
setuptools will also proliferate. It causes about the same problem for
buildout without breaking older packages.

In this scenario buildout will need to be able to install wheels however it
wishes to do so because that is the output of the pluggable build system. I
don't have advice for you on the implementation, except perhaps it could
call out to pip.

On Tue, Aug 16, 2016 at 12:59 PM Leonardo Rochael Almeida <
leoroch...@gmail.com> wrote:

> Spinning this off into its own thread...
>
> On 16 August 2016 at 13:10, Donald Stufft  wrote:
>
>>
>> On Aug 16, 2016, at 11:15 AM, Leonardo Rochael Almeida <
>> leoroch...@gmail.com> wrote:
>>
>> Specifically, buildout right now has setuptools as its only dependence,
>> and buildout uses it to locate, download and build sdist dependencies, or
>> directly download and use .egg dependencies, which are important for
>> Windows platforms in the communities that use buildout and depend on
>> compiled extensions.
>>
>>
>>
>> Ah, I knew I was forgetting something. I think we should hold off on
>> preventing egg uploads until setuptools can download and install wheels.
>>
>
> Which brings us to a question that I'm meaning to ask for a while.
>
> It looks like we're close to removing all mentions of setuptools in pip.
> When this happens, it looks like pressure is going to start to mount on
> setuptools to drop the ability to install packages and limit itself on
> being just a build tool.
>
> It makes sense that it should be so, considering that there would be no
> incentive to keep around two distinct implementations of how to locate,
> download and install stuff, one in pip (or whatever library pip uses for
> locating and installing packages) and another in setuptools.
>
> In this future, I can imagine setuptools will simply complain loudly when
> the packages it requires to do its work are not installed yet, perhaps with
> a plugin mechanism for allowing another package to automatically resolve
> and installing packages, (like today setuptools-git extends setuptools to
> recognize files from git to be included in a package).
>
> And people will promptly write a pip implementation of this plugin (or
> whatever it is that pip will use as a library to do it)... thereby
> inverting the dependency between setuptools and pip.
>
> When this future arrives, buildout will be out of a mechanism for
> locating, downloading and installing packages (not counting old versions of
> setuptools, of course).
>
> Questions:
>
> 1. Is the future outlined above considered likely?
>
> 2. If this future arrives, what package should buildout be ported to use?
> It should provide:
>
>  * The ability to locate packages and their versions, resolve their
> dependencies, and download them, whether they're sdists or wheels (or eggs,
> while we still have them)
>  * Install each package separately into it's own directory that, if added
> to `sys.path`, allows this same package (or perhaps another one) to read
> information from entry points so that buildout can locate its own
> extensions and install console scripts.
>
> Regards,
>
> Leo
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 16, 2016, at 12:58 PM, Daniel Holth  wrote:
> 
> On Tue, Aug 16, 2016 at 12:50 PM Ian Cordasco  > wrote:
> So perhaps I'm missing something, but why are we talking about trying
> to shoehorn a legacy design into Wheel? Why wouldn't we leave
> bdist_egg alone and start trying to find a better way to replace it?
> We could avoid over-complicating Wheel and we could spend time
> polishing whatever we come up with.
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org 
> 
> https://mail.python.org/mailman/listinfo/distutils-sig 
> 
> 
> It is because wheel is almost identical to egg, but the name 'newegg' was 
> already in use by a major online retailer. Possibly adding the one missing 
> egg feature to wheel would save someone some time. Maybe you would want to 
> give 'extracted wheel in its own directory' a different name, or no name, to 
> avoid confusion.

Installing a wheel to a single directory is fine. It’s no longer a wheel at 
that point, it is an installed distribution.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 16, 2016, at 12:46 PM, Daniel Holth  wrote:
> 
> On Tue, Aug 16, 2016 at 12:15 PM Donald Stufft  > wrote:
>> On Aug 16, 2016, at 11:51 AM, Brett Cannon > > wrote:
>> 
>> One thing to remember is that Windows can't read tar files natively while it 
>> can for zip files. Now you can easily download tools on Windows to read tar 
>> files and thanks to Bash on Windows you even have it included once you turn 
>> that feature on.
> 
> This is true, but I think that using .tar.gz by default still makes sense 
> because it’s still the vast bulk of what people actually release to PyPI. So 
> it represents the status quo and switching to zip as the default would break 
> a lot of things.
> 
>> 
>> The other point is we have a zip importer in Python but not a .tar.gz one. I 
>> don't know how often anyone actually downloads a zip file directly from PyPI 
>> and then tack it on to their sys.path for importing, but that is currently 
>> possible.
> 
> A sdist is not an acceptable format for adding to sys.path. While in many, 
> simple cases, it will “just work”, that’s more of an implementation detail 
> than anything else. There are many projects which simply do not run or error 
> out if you do this. I don’t think worrying about something that sort of 
> works, sometimes, is a big deal.
> 
>> 
>> I doubt either of these points are important enough to continue to support 
>> zip files for sdists, but I just wanted to point it out. At worst this is 
>> something to think about if we ever do formalize the sdist format and come 
>> up with a custom file extension.
> 
> If we make a sdist 2.0 with a new format, I do think it makes sense to make 
> it a zipfile like wheel already is (which reduces the internal formats down 
> from 2 to 1), not for the reasons above though, just for consistency with 
> wheel.
> 
> ZIP is a fantastically designed file format. JAR, IPA (iPhone), all are ZIP 
> files. The only thing I sometimes wonder about for wheel is whether it would 
> be worth the trouble to support greater compression with something like 
> .zip.xz (an uncompressed ZIP inside a wrapper, possibly another ZIP) since 
> ZIP compresses each file individually and does not compress its metadata. But 
> most packages are quite small.

I don’t think it would be worth it. Right now there’s only 1 optional c library 
required (zlib) to install from wheels. If we deprecate all of the other things 
then there’s only 1 optional c library required to install from sdists too 
(zlib). At best it would make sense to use gzip, but since we’re already use 
ZIP_DEFLATE I’m not sure the extra complexity is worth it.

> 
> ZIP's greatest weakness is also its greatest strength, as ZIP allows random 
> access to its members and very fast access to its metadata. This is why it 
> makes sense to have a ZIP importer but not a .tar.gz importer.


—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Paul Moore]

On 16 August 2016 at 17:49, Ian Cordasco  wrote:
> So perhaps I'm missing something, but why are we talking about trying
> to shoehorn a legacy design into Wheel? Why wouldn't we leave
> bdist_egg alone and start trying to find a better way to replace it?
> We could avoid over-complicating Wheel and we could spend time
> polishing whatever we come up with.

+1

Wheel should remain a distribution format (with running from wheel,
zipped or unzipped, an intentional but unsupported benefit - get-pip
and virtualenv use the feature, but it should be treated as "use at
your own risk").

Egg as distribution format is deprecated and we shouldn't look to
encourage it, but egg as runtime format is fine if people want/need it
(I don't see the value myself, but the work needed to modify something
like buildout to move away from it probably isn't worth it).
Downloading a wheel and converting it to a runtime egg format is
perfectly possible. Whether buildout's preference for not adding
dependencies is sufficient to push setuptools to add support for that
route, I can't say. But I'm perfectly OK with saying that in due
course, we'll stop supporting eggs for distribution (e.g., in terms of
hosting on PyPI) and then provide a period for the likes of
buildout/setuptools to decide how to migrate. If they choose not to,
then fine - we have to decide whether buildout is important enough to
unilaterally block the change. But I don't think we should rush on
this - let's just publish the statement of intent, and then give
projects like buildout the time they need to react.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Future of setuptools and buildout

[Leonardo Rochael Almeida]

Spinning this off into its own thread...

On 16 August 2016 at 13:10, Donald Stufft  wrote:

>
> On Aug 16, 2016, at 11:15 AM, Leonardo Rochael Almeida <
> leoroch...@gmail.com> wrote:
>
> Specifically, buildout right now has setuptools as its only dependence,
> and buildout uses it to locate, download and build sdist dependencies, or
> directly download and use .egg dependencies, which are important for
> Windows platforms in the communities that use buildout and depend on
> compiled extensions.
>
>
>
> Ah, I knew I was forgetting something. I think we should hold off on
> preventing egg uploads until setuptools can download and install wheels.
>

Which brings us to a question that I'm meaning to ask for a while.

It looks like we're close to removing all mentions of setuptools in pip.
When this happens, it looks like pressure is going to start to mount on
setuptools to drop the ability to install packages and limit itself on
being just a build tool.

It makes sense that it should be so, considering that there would be no
incentive to keep around two distinct implementations of how to locate,
download and install stuff, one in pip (or whatever library pip uses for
locating and installing packages) and another in setuptools.

In this future, I can imagine setuptools will simply complain loudly when
the packages it requires to do its work are not installed yet, perhaps with
a plugin mechanism for allowing another package to automatically resolve
and installing packages, (like today setuptools-git extends setuptools to
recognize files from git to be included in a package).

And people will promptly write a pip implementation of this plugin (or
whatever it is that pip will use as a library to do it)... thereby
inverting the dependency between setuptools and pip.

When this future arrives, buildout will be out of a mechanism for locating,
downloading and installing packages (not counting old versions of
setuptools, of course).

Questions:

1. Is the future outlined above considered likely?

2. If this future arrives, what package should buildout be ported to use?
It should provide:

 * The ability to locate packages and their versions, resolve their
dependencies, and download them, whether they're sdists or wheels (or eggs,
while we still have them)
 * Install each package separately into it's own directory that, if added
to `sys.path`, allows this same package (or perhaps another one) to read
information from entry points so that buildout can locate its own
extensions and install console scripts.

Regards,

Leo
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Daniel Holth]

On Tue, Aug 16, 2016 at 12:50 PM Ian Cordasco 
wrote:

> So perhaps I'm missing something, but why are we talking about trying
> to shoehorn a legacy design into Wheel? Why wouldn't we leave
> bdist_egg alone and start trying to find a better way to replace it?
> We could avoid over-complicating Wheel and we could spend time
> polishing whatever we come up with.
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig


It is because wheel is almost identical to egg, but the name 'newegg' was
already in use by a major online retailer. Possibly adding the one missing
egg feature to wheel would save someone some time. Maybe you would want to
give 'extracted wheel in its own directory' a different name, or no name,
to avoid confusion.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Ian Cordasco]

So perhaps I'm missing something, but why are we talking about trying
to shoehorn a legacy design into Wheel? Why wouldn't we leave
bdist_egg alone and start trying to find a better way to replace it?
We could avoid over-complicating Wheel and we could spend time
polishing whatever we come up with.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Daniel Holth]

On Tue, Aug 16, 2016 at 12:43 PM Alex Grönholm 
wrote:

> 16.08.2016, 19:37, Daniel Holth kirjoitti:
>
> On Tue, Aug 16, 2016 at 12:06 PM Donald Stufft  wrote:
>
>>
>> On Aug 16, 2016, at 8:50 AM, Daniel Holth  wrote:
>>
>> Wheel should be updated to support the egg use case before egg is
>> removed. IIUC this would mostly mean officially supporting 'unzipped wheel'
>> as a thing you can add to PYTHONPATH, possibly with some additional
>> restrictions for the specific wheel. We could go a little further and
>> officially support zipped wheels "if zip safe". We could implement
>> wheel2egg to complement egg2wheel?
>>
>>
>>
>> I don’t think Wheel should officially supported unzip wheels as a thing
>> you can add to PYTHONPATH nor do I think we should officially support
>> zipped wheels being added to PYTHONPATH. Neither of those things are going
>> to work universally and setuptools has gross heuristics to try and figure
>> out when they will and won’t work (which regularly break or report
>> inaccurately). Wheel is improved by remaining focused on being a format for
>> distributing and installed via an installer, not one that tries to do all
>> of the things like Egg did.
>>
>
> So this is how I envision "installing a wheel to its own directory".
>
> 1. Unzip the wheel into its own directory.
> 2. If there are any nested *.data/purelib, platlib, make sure they are
> also unzipped into the root of the unzipped wheel instead of their archive
> paths.
> 3. If there are scripts and you want them, install those too.
>
> If 2 and 3 don't apply to you, you are done almost before you've started.
> What's missing?
>
> How does one go about installing console_scripts this way?
>

No difference. Read entry_points.txt and generate script wrappers for all
of the listed console_scripts. In buildout's case it adds all of the
dependencies to sys.path in the generated console_script wrapper.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Daniel Holth]

On Tue, Aug 16, 2016 at 12:15 PM Donald Stufft  wrote:

> On Aug 16, 2016, at 11:51 AM, Brett Cannon  wrote:
>
> One thing to remember is that Windows can't read tar files natively while
> it can for zip files. Now you can easily download tools on Windows to read
> tar files and thanks to Bash on Windows you even have it included once you
> turn that feature on.
>
>
> This is true, but I think that using .tar.gz by default still makes sense
> because it’s still the vast bulk of what people actually release to PyPI.
> So it represents the status quo and switching to zip as the default would
> break a lot of things.
>
>
> The other point is we have a zip importer in Python but not a .tar.gz one. I
> don't know how often anyone actually downloads a zip file directly from
> PyPI and then tack it on to their sys.path for importing, but that is
> currently possible.
>
>
> A sdist is not an acceptable format for adding to sys.path. While in many,
> simple cases, it will “just work”, that’s more of an implementation detail
> than anything else. There are many projects which simply do not run or
> error out if you do this. I don’t think worrying about something that sort
> of works, sometimes, is a big deal.
>
>
> I doubt either of these points are important enough to continue to support
> zip files for sdists, but I just wanted to point it out. At worst this is
> something to think about if we ever do formalize the sdist format and come
> up with a custom file extension.
>
>
> If we make a sdist 2.0 with a new format, I do think it makes sense to
> make it a zipfile like wheel already is (which reduces the internal formats
> down from 2 to 1), not for the reasons above though, just for consistency
> with wheel.
>

ZIP is a fantastically designed file format. JAR, IPA (iPhone), all are ZIP
files. The only thing I sometimes wonder about for wheel is whether it
would be worth the trouble to support greater compression with something
like .zip.xz (an uncompressed ZIP inside a wrapper, possibly another ZIP)
since ZIP compresses each file individually and does not compress its
metadata. But most packages are quite small.

ZIP's greatest weakness is also its greatest strength, as ZIP allows random
access to its members and very fast access to its metadata. This is why it
makes sense to have a ZIP importer but not a .tar.gz importer.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Alex Grönholm]

16.08.2016, 19:37, Daniel Holth kirjoitti:
On Tue, Aug 16, 2016 at 12:06 PM Donald Stufft > wrote:




On Aug 16, 2016, at 8:50 AM, Daniel Holth mailto:dho...@gmail.com>> wrote:

Wheel should be updated to support the egg use case before egg is
removed. IIUC this would mostly mean officially supporting
'unzipped wheel' as a thing you can add to PYTHONPATH, possibly
with some additional restrictions for the specific wheel. We
could go a little further and officially support zipped wheels
"if zip safe". We could implement wheel2egg to complement egg2wheel?



I don’t think Wheel should officially supported unzip wheels as a
thing you can add to PYTHONPATH nor do I think we should
officially support zipped wheels being added to PYTHONPATH.
Neither of those things are going to work universally and
setuptools has gross heuristics to try and figure out when they
will and won’t work (which regularly break or report
inaccurately). Wheel is improved by remaining focused on being a
format for distributing and installed via an installer, not one
that tries to do all of the things like Egg did.


So this is how I envision "installing a wheel to its own directory".

1. Unzip the wheel into its own directory.
2. If there are any nested *.data/purelib, platlib, make sure they are 
also unzipped into the root of the unzipped wheel instead of their 
archive paths.

3. If there are scripts and you want them, install those too.

If 2 and 3 don't apply to you, you are done almost before you've 
started. What's missing?



How does one go about installing console_scripts this way?



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Daniel Holth]

On Tue, Aug 16, 2016 at 12:06 PM Donald Stufft  wrote:

>
> On Aug 16, 2016, at 8:50 AM, Daniel Holth  wrote:
>
> Wheel should be updated to support the egg use case before egg is removed.
> IIUC this would mostly mean officially supporting 'unzipped wheel' as a
> thing you can add to PYTHONPATH, possibly with some additional restrictions
> for the specific wheel. We could go a little further and officially support
> zipped wheels "if zip safe". We could implement wheel2egg to complement
> egg2wheel?
>
>
>
> I don’t think Wheel should officially supported unzip wheels as a thing
> you can add to PYTHONPATH nor do I think we should officially support
> zipped wheels being added to PYTHONPATH. Neither of those things are going
> to work universally and setuptools has gross heuristics to try and figure
> out when they will and won’t work (which regularly break or report
> inaccurately). Wheel is improved by remaining focused on being a format for
> distributing and installed via an installer, not one that tries to do all
> of the things like Egg did.
>

So this is how I envision "installing a wheel to its own directory".

1. Unzip the wheel into its own directory.
2. If there are any nested *.data/purelib, platlib, make sure they are also
unzipped into the root of the unzipped wheel instead of their archive paths.
3. If there are scripts and you want them, install those too.

If 2 and 3 don't apply to you, you are done almost before you've started.
What's missing?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 16, 2016, at 11:51 AM, Brett Cannon  wrote:
> 
> One thing to remember is that Windows can't read tar files natively while it 
> can for zip files. Now you can easily download tools on Windows to read tar 
> files and thanks to Bash on Windows you even have it included once you turn 
> that feature on.

This is true, but I think that using .tar.gz by default still makes sense 
because it’s still the vast bulk of what people actually release to PyPI. So it 
represents the status quo and switching to zip as the default would break a lot 
of things.

> 
> The other point is we have a zip importer in Python but not a .tar.gz one. I 
> don't know how often anyone actually downloads a zip file directly from PyPI 
> and then tack it on to their sys.path for importing, but that is currently 
> possible.

A sdist is not an acceptable format for adding to sys.path. While in many, 
simple cases, it will “just work”, that’s more of an implementation detail than 
anything else. There are many projects which simply do not run or error out if 
you do this. I don’t think worrying about something that sort of works, 
sometimes, is a big deal.

> 
> I doubt either of these points are important enough to continue to support 
> zip files for sdists, but I just wanted to point it out. At worst this is 
> something to think about if we ever do formalize the sdist format and come up 
> with a custom file extension.

If we make a sdist 2.0 with a new format, I do think it makes sense to make it 
a zipfile like wheel already is (which reduces the internal formats down from 2 
to 1), not for the reasons above though, just for consistency with wheel.


—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 16, 2016, at 11:15 AM, Leonardo Rochael Almeida  
> wrote:
> 
> Specifically, buildout right now has setuptools as its only dependence, and 
> buildout uses it to locate, download and build sdist dependencies, or 
> directly download and use .egg dependencies, which are important for Windows 
> platforms in the communities that use buildout and depend on compiled 
> extensions.


Ah, I knew I was forgetting something. I think we should hold off on preventing 
egg uploads until setuptools can download and install wheels.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Donald Stufft]

> On Aug 16, 2016, at 8:50 AM, Daniel Holth  wrote:
> 
> Wheel should be updated to support the egg use case before egg is removed. 
> IIUC this would mostly mean officially supporting 'unzipped wheel' as a thing 
> you can add to PYTHONPATH, possibly with some additional restrictions for the 
> specific wheel. We could go a little further and officially support zipped 
> wheels "if zip safe". We could implement wheel2egg to complement egg2wheel?


I don’t think Wheel should officially supported unzip wheels as a thing you can 
add to PYTHONPATH nor do I think we should officially support zipped wheels 
being added to PYTHONPATH. Neither of those things are going to work 
universally and setuptools has gross heuristics to try and figure out when they 
will and won’t work (which regularly break or report inaccurately). Wheel is 
improved by remaining focused on being a format for distributing and installed 
via an installer, not one that tries to do all of the things like Egg did.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Brett Cannon]

+1 on the whole idea. Trying to continue to nudge the community towards
more standardized approaches in packaging is always a good thing. :) I only
have one data point in relation to sdists file formats.

On Mon, 15 Aug 2016 at 12:09 Donald Stufft  wrote:

> [SNIP]
>
> Looking at numbers again, the current number of projects for each file ext
> are:
>
> * .tar.gz: 444,338
> * .zip: 58,774
> * .tar.bz2: 3,265
> * .tgz: 217
> * Everything Else: 0
>

One thing to remember is that Windows can't read tar files natively while
it can for zip files. Now you can easily download tools on Windows to read
tar files and thanks to Bash on Windows you even have it included once you
turn that feature on.

The other point is we have a zip importer in Python but not a .tar.gz one. I
don't know how often anyone actually downloads a zip file directly from
PyPI and then tack it on to their sys.path for importing, but that is
currently possible.

I doubt either of these points are important enough to continue to support
zip files for sdists, but I just wanted to point it out. At worst this is
something to think about if we ever do formalize the sdist format and come
up with a custom file extension.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Leonardo Rochael Almeida]

Thanks Daniel,

A few corrections and considerations below:

On 16 August 2016 at 09:50, Daniel Holth  wrote:

> Wheel should be updated to support the egg use case before egg is removed.
> IIUC this would mostly mean officially supporting 'unzipped wheel' as a
> thing you can add to PYTHONPATH, possibly with some additional restrictions
> for the specific wheel. We could go a little further and officially support
> zipped wheels "if zip safe". We could implement wheel2egg to complement
> egg2wheel?
>

Specifically, buildout right now has setuptools as its only dependence, and
buildout uses it to locate, download and build sdist dependencies, or
directly download and use .egg dependencies, which are important for
Windows platforms in the communities that use buildout and depend on
compiled extensions.

It also uses setuptools for reading entry points of packages for internal
use (buildout recipes and extensions), and for figuring out console scripts
to install in "bin"

(this last part is actually the job of the `zc.recipe.egg` recipe, which is
buildout's pip replacement that predates pip by a few years, but it's the
only recipe that is developed in the same repo as buildout itself, so we'll
treat them as one for this discussion).

This also means that so far buildout has been missing out on wheels. It
hasn't been much of a problem since almost all packages also have an sdist
that can be turned into an egg by setuptools, but if the trend of
sdist-less wheels increases, this could quickly become a problem.

It also means that, buildout is missing out on manylinux...

A wheel2egg might be nice, but unless it's integrated into setuptools
proper, it would mean adding another dependency to buildout and the
developers would rather depend on a single library for talking to PyPI and
making packages appear as "directories addable to `sys.path`.

The main reason wheel did not support these use cases from the beginning is
> that it was quicker not to. Also, it is a bit confusing for 'unzipped
> directory in the (egg/wheel) format', 'archive', and 'plugin you add to
> PYTHONPATH' to all have the same name 'eggs'.
>
> We would need to work with the buildout developers on this. Buildout uses
> a flat directory full of unzipped eggs
>

Not necessarily unzipped. Although buildout unzips all sdists and eggs it
installs, if it's in the "eggs" directory, has the right name (e.g.
suds_jurko-0.6-py2.7.egg), and can be added to sys.path, that's all
buildout cares about, but if it's not yet there, buildout will use
setuptools to install it as an "unzipped egg" (there used to be a switch to
install eggs as zipped, or rather, as unzipped, zipped being the default,
but nobody liked the eggs zipped so they threw that foot gun away).


> , not exactly the uploading to pypi case, and when installing
> console_scripts, makes sure the necessary eggs are added to sys.path in the
> script itself. The exact mechanism varies depending on the version of
> buildout used.
>

This hasn't changed for a while. The mechanism is, basically, to create a
script in "bin/" that appends all (recursive) dependencies of the script
from the "eggs" directory into `sys.path` then call the entry point
declared in the `console_scripts`.

The buildout configuration can specify both the name that the console
scripts will be installed, and also specify some code to be run immediately
before the entry point,  which is useful for pre-setting command line
arguments, for example. All of this is done using setuptools APIs.

This and the fact that buildout has a plethora of extensions and powerful
declarative configuration language that allows for dependency management
between different "parts" of the configuration (e.g. coordinating ports and
addresses between services like Varnish and Plone, or adding the paths of
the script and configuation file from Plone into a supervisord
configuration) means that, like many things coming out of the brilliant
mind of Jim Fulton, it's considered overly complex by many, but those who
use it would find it rather difficult to replace if it's no longer
functional.

zope.interface is an important package with up-to-date eggs.
> https://pypi.python.org/pypi/zope.interface
>

And it's used by Plone and Pyramid, to mention two mainstream communities
that would be affected by a lack of binary eggs on Windows, for example.

To sumarize, if we could convince setuptools to treat wheels on PyPI for
download and installation the same way it treats eggs, or if we could
replace setuptools with something that did, then buildout would be onboard
the deprecation of eggs.

Cheers,

PS: In the buildout community, we never really understood the impetus for
replacing egg as a format, which is really not all that complex and not all
the different from wheel as it stands, instead of just formalizing it in a
PEP. We got the feeling that, in the movement of getting rid of setuptools
as an "installation" dependency, we ended up throwing out the baby with the

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Daniel Holth]

Wheel should be updated to support the egg use case before egg is removed.
IIUC this would mostly mean officially supporting 'unzipped wheel' as a
thing you can add to PYTHONPATH, possibly with some additional restrictions
for the specific wheel. We could go a little further and officially support
zipped wheels "if zip safe". We could implement wheel2egg to complement
egg2wheel?

The main reason wheel did not support these use cases from the beginning is
that it was quicker not to. Also, it is a bit confusing for 'unzipped
directory in the (egg/wheel) format', 'archive', and 'plugin you add to
PYTHONPATH' to all have the same name 'eggs'.

We would need to work with the buildout developers on this. Buildout uses a
flat directory full of unzipped eggs, not exactly the uploading to pypi
case, and when installing console_scripts, makes sure the necessary eggs
are added to sys.path in the script itself. The exact mechanism varies
depending on the version of buildout used.

zope.interface is an important package with up-to-date eggs.
https://pypi.python.org/pypi/zope.interface

On Tue, Aug 16, 2016 at 5:41 AM Paul Moore  wrote:

> On 15 August 2016 at 20:09, Donald Stufft  wrote:
> >
> > First off, we currently allow people to upload sdist, bdist_wheel,
> bdist_egg,
> > bdist_dmg, bdist_dumb, bdist_msi, bdist_rpm, and bdist_wininst. However
> I think
> > that we should try to get rid of support for most of these. Just for
> reference
> > currently the number of files uploaded for each type of file looks like:
> >
> > * sdist: 506,585
> > * bdist_wheel: 81,207
> > * bdist_egg: 48,282
> > * bdist_wininst: 14,002
> > * bdist_dumb: 5,502
> > * bdist_msi: 497
> > * bdist_rpm: 464
> > * bdist_dmg: 45
> >
> > Out of all of these, I think that we can easily remove bdist_dmg,
> bdist_rpm,
> > and bdist_dumb. I also believe that there is a strong case for removing
> > bdist_msi and bdist_wininst. I also think we should consider removing
> > bdist_egg.
>
> Overall +1 from me for this.
>
> Some thoughts:
>
> 1. It would be nice to provide some transition process for the
> bdist_wininst case, as these are automatically convertible (with some
> caveats) to wheel using "wheel convert". I'm not suggesting that we do
> an auto-convert, but a way of getting the message out to projects that
> use wininst that there is a migration path might be worth it (although
> it's equally possible that the projects in question are all
> unmaintained, in which case there's not much point in bothering).
> 2. Do we understand the remaining use cases for eggs? AIUI, buildout
> uses it, and I get the impression that there are other specialist
> groups that still use the egg format (the plone community?). While I
> don't think having two binary distribution formats is good for the
> ecosystem (it's confusing for users, and splits effort), I think we
> should be make sure we have those use cases covered (or at least have
> a plan on how we handle the situation) before deprecating the egg
> format.
>
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Paul Moore]

On 15 August 2016 at 20:09, Donald Stufft  wrote:
>
> First off, we currently allow people to upload sdist, bdist_wheel, bdist_egg,
> bdist_dmg, bdist_dumb, bdist_msi, bdist_rpm, and bdist_wininst. However I 
> think
> that we should try to get rid of support for most of these. Just for reference
> currently the number of files uploaded for each type of file looks like:
>
> * sdist: 506,585
> * bdist_wheel: 81,207
> * bdist_egg: 48,282
> * bdist_wininst: 14,002
> * bdist_dumb: 5,502
> * bdist_msi: 497
> * bdist_rpm: 464
> * bdist_dmg: 45
>
> Out of all of these, I think that we can easily remove bdist_dmg, bdist_rpm,
> and bdist_dumb. I also believe that there is a strong case for removing
> bdist_msi and bdist_wininst. I also think we should consider removing
> bdist_egg.

Overall +1 from me for this.

Some thoughts:

1. It would be nice to provide some transition process for the
bdist_wininst case, as these are automatically convertible (with some
caveats) to wheel using "wheel convert". I'm not suggesting that we do
an auto-convert, but a way of getting the message out to projects that
use wininst that there is a migration path might be worth it (although
it's equally possible that the projects in question are all
unmaintained, in which case there's not much point in bothering).
2. Do we understand the remaining use cases for eggs? AIUI, buildout
uses it, and I get the impression that there are other specialist
groups that still use the egg format (the plone community?). While I
don't think having two binary distribution formats is good for the
ecosystem (it's confusing for users, and splits effort), I think we
should be make sure we have those use cases covered (or at least have
a plan on how we handle the situation) before deprecating the egg
format.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig