Re: [Distutils] Builders vs Installers
On 26 Mar 2013 21:08, "Daniel Holth" wrote: > > > Yea, it's totally about keywords and that's just not an example of a larger problem (like embedding little mini json documents) and what we need is another competing standard all because of a legacy file format for a file that barely anything uses right now (which makes it the ideal time _to_ replace it, before it starts being actively used in a widespread fashion). > > We need approximately five fields: > > Name > Version > Provides-Extra > Requires-Dist > Setup-Requires-Dist > > the rest are useless, never need to be parsed by anyone, or are > already sent to pypi as a dict. > > We need the environment markers language. > > We need the requirements specifiers >= 4.0.0, < 9. > > Define the JSON serialization and we'll have this format converted in > 50 lines of code or less. It's that easy. I've already defined it for the post install hook design, and will now be rewriting the PEP to use that as the base format. As added bonuses, it will allow 2.0 metadata to live alongside 1.1 metadata (due to a different file name), be easier to explain to readers of the PEP and allow us to fix some clumsy legacy naming. When we last considered this question, we were still trying to keep the metadata 1.3 changes minimal to avoid delaying the addition of wheel support to pip. That issue has since been solved more expediently by allowing metadata 1.1 in wheel files. The addition of the post install hook is the other major relevant change, and that's the one which means we need to define a structured metadata format regardless of the on-disk format. It all adds up to it making far more sense to just switch the format to JSON for 2.0 rather than persisting with ad hoc attempts to use a key-value multidict for structured data storage. Cheers, Nick. P.S. I forgot LAX has free wi-fi now :) > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
> Yea, it's totally about keywords and that's just not an example of a larger > problem (like embedding little mini json documents) and what we need is > another competing standard all because of a legacy file format for a file > that barely anything uses right now (which makes it the ideal time _to_ > replace it, before it starts being actively used in a widespread fashion). We need approximately five fields: Name Version Provides-Extra Requires-Dist Setup-Requires-Dist the rest are useless, never need to be parsed by anyone, or are already sent to pypi as a dict. We need the environment markers language. We need the requirements specifiers >= 4.0.0, < 9. Define the JSON serialization and we'll have this format converted in 50 lines of code or less. It's that easy. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Mar 26, 2013, at 10:49 PM, Daniel Holth wrote:
> On Tue, Mar 26, 2013 at 9:12 PM, PJ Eby wrote:
>> On Tue, Mar 26, 2013 at 8:33 PM, Donald Stufft wrote:
>>> As far as I can tell the only things that even use PKG-INFO is
>>> setuptools/distribute and we want to phase them out of existence anyways.
>>
>> The only thing setuptools uses it for is to find out the version of a
>> package in the case where an .egg-info directory or filename doesn't
>> have a version in its filename... which normally only happens in the
>> "setup.py develop" case. So no need to keep it around on my account.
>> ;-)
>>
>> (Some tools do check for the *existence* of a PKG-INFO, like PyPI's
>> sdist upload validation, and the various egg formats require a file
>> *named* PKG-INFO, but AFAIK nothing commonly used out there actually
>> *reads* PKG-INFO or gives a darn about its contents, except for that
>> version usecase mentioned above.)
>> ___
>> Distutils-SIG maillist - Distutils-SIG@python.org
>> http://mail.python.org/mailman/listinfo/distutils-sig
>
> It will be OK. Take a deep breath and laugh at the idea that
> string.rsplit(', ', 1) on a useless field that's probably already
> posted as a dict to pypi should be considered a serious threat to the
> future of packaging. If you didn't laugh you can write Metadata 3.0 /
> define the JSON serialization and we'll write metadata.json into the
> .dist-info directory. It's not the end of the world, it is the
> beginning.
Yea, it's totally about keywords and that's just not an example of a larger
problem (like embedding little mini json documents) and what we need is another
competing standard all because of a legacy file format for a file that barely
anything uses right now (which makes it the ideal time _to_ replace it, before
it starts being actively used in a widespread fashion).
-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Tue, Mar 26, 2013 at 9:12 PM, PJ Eby wrote:
> On Tue, Mar 26, 2013 at 8:33 PM, Donald Stufft wrote:
>> As far as I can tell the only things that even use PKG-INFO is
>> setuptools/distribute and we want to phase them out of existence anyways.
>
> The only thing setuptools uses it for is to find out the version of a
> package in the case where an .egg-info directory or filename doesn't
> have a version in its filename... which normally only happens in the
> "setup.py develop" case. So no need to keep it around on my account.
> ;-)
>
> (Some tools do check for the *existence* of a PKG-INFO, like PyPI's
> sdist upload validation, and the various egg formats require a file
> *named* PKG-INFO, but AFAIK nothing commonly used out there actually
> *reads* PKG-INFO or gives a darn about its contents, except for that
> version usecase mentioned above.)
> ___
> Distutils-SIG maillist - Distutils-SIG@python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
It will be OK. Take a deep breath and laugh at the idea that
string.rsplit(', ', 1) on a useless field that's probably already
posted as a dict to pypi should be considered a serious threat to the
future of packaging. If you didn't laugh you can write Metadata 3.0 /
define the JSON serialization and we'll write metadata.json into the
.dist-info directory. It's not the end of the world, it is the
beginning.
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Tue, Mar 26, 2013 at 8:33 PM, Donald Stufft wrote: > As far as I can tell the only things that even use PKG-INFO is > setuptools/distribute and we want to phase them out of existence anyways. The only thing setuptools uses it for is to find out the version of a package in the case where an .egg-info directory or filename doesn't have a version in its filename... which normally only happens in the "setup.py develop" case. So no need to keep it around on my account. ;-) (Some tools do check for the *existence* of a PKG-INFO, like PyPI's sdist upload validation, and the various egg formats require a file *named* PKG-INFO, but AFAIK nothing commonly used out there actually *reads* PKG-INFO or gives a darn about its contents, except for that version usecase mentioned above.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Mar 26, 2013, at 7:50 PM, Nick Coghlan wrote:
> On Wed, Mar 27, 2013 at 8:01 AM, Donald Stufft wrote:
>> Hopefully this will be included in .dist-info and in every package so we*
>> can pretend PKF-INFO doesn't exist ;)
>
> The key-value format is actually easier for hand editing and covers
> most cases. The extension format allows embedded JSON for more complex
> cases. As an on-disk format, it's isomorphic to JSON, so I don't
> actually plan to propose changing it.
I disagree.
- These files are used for tools to exchange data, so "hand editable" shouldn't
be a primary concern.
- There are a number of current fields where the current format is *not* enough
and one off psuedo formats have had to be added
- `Keywords: dog puppy voting election` - A list masquerading as a string,
this one needs field.split() to actually parse it
- `Project-URL: Bug, Issue Tracker,
http://bitbucket.org/tarek/distribute/issues/` - A dictionary masquerading as a
list of strings, this one needs {key.strip(): value.strip() for key, value in
[x.rsplit(", ", 1) for x in field]}
- Any of the fields can contain arbitrary content, previously Description had
specialized handling for this which it has now been moved to the payload
section, but all the same issues there affect other fields.
- The Extension field name using ExtensionName/ActualKey to kludge a nested
dictionary
- The ExtensionName/json is a horrible kludge why are we nesting a format
inside of a format instead of just using a format that supports everything we
could want?
As far as I can tell the only things that even use PKG-INFO is
setuptools/distribute and we want to phase them out of existence anyways. The
only other thing I can think of is Wheel which can either a) be updated to a
different format it's new enough there's not much need to worry about legacy
support or b) generate the METADATA file just for Wheels.
TBH I'd like it if my name was removed as author of the PEP, I only briefly
touched the versioning section and I do not agree with the decision to continue
using PKG-INFO and do not want my name attached to a PEP that advocates it.
>
> Where we *do* need JSON-compatible metadata, though, is as an easy to
> pass around in-memory data structure for use in APIs. In particular,
> metadata 2.0 will be defining this format (and how to convert it
> to/from the key/value format) so that the signature of the
> post-install hook can be:
>
>def post_install_hook(installed, previous=None):
>...
>
> "installed" will be a string-keyed metadata dictionary for the
> distribution that was just installed, containing only dicts, lists and
> strings as values.
> "previous" will be the metadata for the version of the distribution
> that was previously installed, if any.
>
> Cheers,
> Nick.
>
> P.S. And now I'm leaving for the airport to fly home to Australia - no
> more replies from me for a couple of days :)
>
> --
> Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Wed, Mar 27, 2013 at 8:01 AM, Donald Stufft wrote: > Hopefully this will be included in .dist-info and in every package so we* > can pretend PKF-INFO doesn't exist ;) The key-value format is actually easier for hand editing and covers most cases. The extension format allows embedded JSON for more complex cases. As an on-disk format, it's isomorphic to JSON, so I don't actually plan to propose changing it. Where we *do* need JSON-compatible metadata, though, is as an easy to pass around in-memory data structure for use in APIs. In particular, metadata 2.0 will be defining this format (and how to convert it to/from the key/value format) so that the signature of the post-install hook can be: def post_install_hook(installed, previous=None): ... "installed" will be a string-keyed metadata dictionary for the distribution that was just installed, containing only dicts, lists and strings as values. "previous" will be the metadata for the version of the distribution that was previously installed, if any. Cheers, Nick. P.S. And now I'm leaving for the airport to fly home to Australia - no more replies from me for a couple of days :) -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP439 and backward compat / easy_install / distlib
On Wed, Mar 27, 2013 at 6:42 AM, Erik Bray wrote: > I pretty much agree with you on all of this, but I don't think the > question should be ignored either--avoiding this question is one of > the things that got previous packaging reform efforts into trouble. > Though the agreement to treat "build" and "installation" as two > different stories mitigates the issue this time around. In any case > it's sort of off topic for this thread so I'll bring it up again > elsewhen. One thing I see as a possible short-term solution is to > still rely on some version of distutils as a build tool *only*. But > it would still be nice to have some easy way to standardize "in-place" > installation regardless of how extension modules get built. That's exactly the interim solution I have in mind: for the moment, the "archive system" will be "python setup.py sdist" in an appropriate location and the "build system" will be "python setup.py bdist_wheel". Both will be modelled on pip's current behaviour when installing from sdists - the difference will be in the explicit invocation of the separate steps, rather than handling the whole chain with "setup.py install". Longer term I want to make setup.py optional even for source installs, but that requires further enhancements to the metadata. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Mar 26, 2013, at 6:47 PM, Erik Bray wrote: > I'm with you on that--I much prefer YAML (which is a superset of > JSON!) but we don't even have that in the stdlib and it's not worth > bikeshedding over to me. YAML is great for human editable. I don't think is has much value over JSON for machine oriented data. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Tue, Mar 26, 2013 at 5:21 PM, Daniel Holth wrote: > I want to poke myself in the eye every time I have to edit json by hand. > Especially the description field. I'm with you on that--I much prefer YAML (which is a superset of JSON!) but we don't even have that in the stdlib and it's not worth bikeshedding over to me. > On Mar 26, 2013 5:17 PM, "Donald Stufft" wrote: >> >> >> On Mar 26, 2013, at 5:12 PM, Daniel Holth wrote: >> >> > I am -1 on renaming anything unless it solves a technical problem. >> > Forever after we will have to explain "well, it used to be called X, >> > now it's called Y..." >> > >> > On Tue, Mar 26, 2013 at 5:01 PM, Erik Bray >> > wrote: >> >> On Tue, Mar 26, 2013 at 4:08 PM, PJ Eby wrote: >> >>> On Tue, Mar 26, 2013 at 3:03 PM, Daniel Holth >> >>> wrote: >> I think PKG-INFO is a highly human-editable format. >> >>> >> >>> That doesn't mean you necessarily want to edit it yourself; notably, >> >>> there will likely be some redundancy between the description in the >> >>> file and other files like the README. >> >>> >> >>> Also, today one of the key use cases people have for custom code in >> >>> setup.py is to pull the package version from a __version__ attribute >> >>> in a module. (Which is evil, of course, but people do it anyway.) >> >>> >> >>> But it might be worth adding a setuptools feature to pull metadata >> >>> from PKG-INFO (or DIST-INFO) instead of generating a new one, to see >> >>> what people think of using PKG-INFO first, other files second. In >> >>> principle, one could reduce a setup.py to just "from setuptools import >> >>> setup_distinfo; setup_distinfo()" or some such. >> >> >> >> In other words, using d2to1 and only for `setup.py egg_info` (only not >> >> egg_info but whatever we're doing instead to generate the metadata ;) >> >> >> >> Erik >> > ___ >> > Distutils-SIG maillist - Distutils-SIG@python.org >> > http://mail.python.org/mailman/listinfo/distutils-sig >> >> >> Rename it and make it JSON instead of the homebrew* format! >> >> * Yes techincally it's based on a real format, but that format doesn't >> support all the things it needs so there are hackishly added extensions >> added to it. >> >> - >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 >> DCFA >> > ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Mar 26, 2013, at 5:56 PM, Daniel Holth wrote: > We will have a standard json version of metadata 2. > > On Mar 26, 2013 5:24 PM, "Donald Stufft" wrote: > > On Mar 26, 2013, at 5:21 PM, Daniel Holth wrote: > >> I want to poke myself in the eye every time I have to edit json by hand. >> Especially the description field. >> >> On Mar 26, 2013 5:17 PM, "Donald Stufft" wrote: >> >> On Mar 26, 2013, at 5:12 PM, Daniel Holth wrote: >> >> > I am -1 on renaming anything unless it solves a technical problem. >> > Forever after we will have to explain "well, it used to be called X, >> > now it's called Y..." >> > >> > On Tue, Mar 26, 2013 at 5:01 PM, Erik Bray wrote: >> >> On Tue, Mar 26, 2013 at 4:08 PM, PJ Eby wrote: >> >>> On Tue, Mar 26, 2013 at 3:03 PM, Daniel Holth wrote: >> I think PKG-INFO is a highly human-editable format. >> >>> >> >>> That doesn't mean you necessarily want to edit it yourself; notably, >> >>> there will likely be some redundancy between the description in the >> >>> file and other files like the README. >> >>> >> >>> Also, today one of the key use cases people have for custom code in >> >>> setup.py is to pull the package version from a __version__ attribute >> >>> in a module. (Which is evil, of course, but people do it anyway.) >> >>> >> >>> But it might be worth adding a setuptools feature to pull metadata >> >>> from PKG-INFO (or DIST-INFO) instead of generating a new one, to see >> >>> what people think of using PKG-INFO first, other files second. In >> >>> principle, one could reduce a setup.py to just "from setuptools import >> >>> setup_distinfo; setup_distinfo()" or some such. >> >> >> >> In other words, using d2to1 and only for `setup.py egg_info` (only not >> >> egg_info but whatever we're doing instead to generate the metadata ;) >> >> >> >> Erik >> > ___ >> > Distutils-SIG maillist - Distutils-SIG@python.org >> > http://mail.python.org/mailman/listinfo/distutils-sig >> >> >> Rename it and make it JSON instead of the homebrew* format! >> >> * Yes techincally it's based on a real format, but that format doesn't >> support all the things it needs so there are hackishly added extensions >> added to it. >> >> - >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >> > > > > So don't edit it by hand, nobody edits PKG-INFO by hand. PKG-INFO (and the > would be replacement) are for tools. Archiver can create it however the > package author wants to, could be setup.py sdist, could be bentomaker sdist, > could be totallyradpackagemaker create. It's a data exchange format not the > API for developers or end users. > > - > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > Hopefully this will be included in .dist-info and in every package so we* can pretend PKF-INFO doesn't exist ;) * The proverbial we. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
We will have a standard json version of metadata 2. On Mar 26, 2013 5:24 PM, "Donald Stufft" wrote: > > On Mar 26, 2013, at 5:21 PM, Daniel Holth wrote: > > I want to poke myself in the eye every time I have to edit json by hand. > Especially the description field. > On Mar 26, 2013 5:17 PM, "Donald Stufft" wrote: > >> >> On Mar 26, 2013, at 5:12 PM, Daniel Holth wrote: >> >> > I am -1 on renaming anything unless it solves a technical problem. >> > Forever after we will have to explain "well, it used to be called X, >> > now it's called Y..." >> > >> > On Tue, Mar 26, 2013 at 5:01 PM, Erik Bray >> wrote: >> >> On Tue, Mar 26, 2013 at 4:08 PM, PJ Eby wrote: >> >>> On Tue, Mar 26, 2013 at 3:03 PM, Daniel Holth >> wrote: >> I think PKG-INFO is a highly human-editable format. >> >>> >> >>> That doesn't mean you necessarily want to edit it yourself; notably, >> >>> there will likely be some redundancy between the description in the >> >>> file and other files like the README. >> >>> >> >>> Also, today one of the key use cases people have for custom code in >> >>> setup.py is to pull the package version from a __version__ attribute >> >>> in a module. (Which is evil, of course, but people do it anyway.) >> >>> >> >>> But it might be worth adding a setuptools feature to pull metadata >> >>> from PKG-INFO (or DIST-INFO) instead of generating a new one, to see >> >>> what people think of using PKG-INFO first, other files second. In >> >>> principle, one could reduce a setup.py to just "from setuptools import >> >>> setup_distinfo; setup_distinfo()" or some such. >> >> >> >> In other words, using d2to1 and only for `setup.py egg_info` (only not >> >> egg_info but whatever we're doing instead to generate the metadata ;) >> >> >> >> Erik >> > ___ >> > Distutils-SIG maillist - Distutils-SIG@python.org >> > http://mail.python.org/mailman/listinfo/distutils-sig >> >> >> Rename it and make it JSON instead of the homebrew* format! >> >> * Yes techincally it's based on a real format, but that format doesn't >> support all the things it needs so there are hackishly added extensions >> added to it. >> >> - >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 >> DCFA >> >> > > So don't edit it by hand, nobody edits PKG-INFO by hand. PKG-INFO (and the > would be replacement) are for tools. Archiver can create it however the > package author wants to, could be setup.py sdist, could be bentomaker > sdist, could be totallyradpackagemaker create. It's a data exchange format > not the API for developers or end users. > > - > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Mar 26, 2013, at 5:21 PM, Daniel Holth wrote: > I want to poke myself in the eye every time I have to edit json by hand. > Especially the description field. > > On Mar 26, 2013 5:17 PM, "Donald Stufft" wrote: > > On Mar 26, 2013, at 5:12 PM, Daniel Holth wrote: > > > I am -1 on renaming anything unless it solves a technical problem. > > Forever after we will have to explain "well, it used to be called X, > > now it's called Y..." > > > > On Tue, Mar 26, 2013 at 5:01 PM, Erik Bray wrote: > >> On Tue, Mar 26, 2013 at 4:08 PM, PJ Eby wrote: > >>> On Tue, Mar 26, 2013 at 3:03 PM, Daniel Holth wrote: > I think PKG-INFO is a highly human-editable format. > >>> > >>> That doesn't mean you necessarily want to edit it yourself; notably, > >>> there will likely be some redundancy between the description in the > >>> file and other files like the README. > >>> > >>> Also, today one of the key use cases people have for custom code in > >>> setup.py is to pull the package version from a __version__ attribute > >>> in a module. (Which is evil, of course, but people do it anyway.) > >>> > >>> But it might be worth adding a setuptools feature to pull metadata > >>> from PKG-INFO (or DIST-INFO) instead of generating a new one, to see > >>> what people think of using PKG-INFO first, other files second. In > >>> principle, one could reduce a setup.py to just "from setuptools import > >>> setup_distinfo; setup_distinfo()" or some such. > >> > >> In other words, using d2to1 and only for `setup.py egg_info` (only not > >> egg_info but whatever we're doing instead to generate the metadata ;) > >> > >> Erik > > ___ > > Distutils-SIG maillist - Distutils-SIG@python.org > > http://mail.python.org/mailman/listinfo/distutils-sig > > > Rename it and make it JSON instead of the homebrew* format! > > * Yes techincally it's based on a real format, but that format doesn't > support all the things it needs so there are hackishly added extensions added > to it. > > - > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > So don't edit it by hand, nobody edits PKG-INFO by hand. PKG-INFO (and the would be replacement) are for tools. Archiver can create it however the package author wants to, could be setup.py sdist, could be bentomaker sdist, could be totallyradpackagemaker create. It's a data exchange format not the API for developers or end users. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Mar 26, 2013, at 5:12 PM, Daniel Holth wrote: > I am -1 on renaming anything unless it solves a technical problem. > Forever after we will have to explain "well, it used to be called X, > now it's called Y..." > > On Tue, Mar 26, 2013 at 5:01 PM, Erik Bray wrote: >> On Tue, Mar 26, 2013 at 4:08 PM, PJ Eby wrote: >>> On Tue, Mar 26, 2013 at 3:03 PM, Daniel Holth wrote: I think PKG-INFO is a highly human-editable format. >>> >>> That doesn't mean you necessarily want to edit it yourself; notably, >>> there will likely be some redundancy between the description in the >>> file and other files like the README. >>> >>> Also, today one of the key use cases people have for custom code in >>> setup.py is to pull the package version from a __version__ attribute >>> in a module. (Which is evil, of course, but people do it anyway.) >>> >>> But it might be worth adding a setuptools feature to pull metadata >>> from PKG-INFO (or DIST-INFO) instead of generating a new one, to see >>> what people think of using PKG-INFO first, other files second. In >>> principle, one could reduce a setup.py to just "from setuptools import >>> setup_distinfo; setup_distinfo()" or some such. >> >> In other words, using d2to1 and only for `setup.py egg_info` (only not >> egg_info but whatever we're doing instead to generate the metadata ;) >> >> Erik > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig Rename it and make it JSON instead of the homebrew* format! * Yes techincally it's based on a real format, but that format doesn't support all the things it needs so there are hackishly added extensions added to it. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
I want to poke myself in the eye every time I have to edit json by hand. Especially the description field. On Mar 26, 2013 5:17 PM, "Donald Stufft" wrote: > > On Mar 26, 2013, at 5:12 PM, Daniel Holth wrote: > > > I am -1 on renaming anything unless it solves a technical problem. > > Forever after we will have to explain "well, it used to be called X, > > now it's called Y..." > > > > On Tue, Mar 26, 2013 at 5:01 PM, Erik Bray > wrote: > >> On Tue, Mar 26, 2013 at 4:08 PM, PJ Eby wrote: > >>> On Tue, Mar 26, 2013 at 3:03 PM, Daniel Holth > wrote: > I think PKG-INFO is a highly human-editable format. > >>> > >>> That doesn't mean you necessarily want to edit it yourself; notably, > >>> there will likely be some redundancy between the description in the > >>> file and other files like the README. > >>> > >>> Also, today one of the key use cases people have for custom code in > >>> setup.py is to pull the package version from a __version__ attribute > >>> in a module. (Which is evil, of course, but people do it anyway.) > >>> > >>> But it might be worth adding a setuptools feature to pull metadata > >>> from PKG-INFO (or DIST-INFO) instead of generating a new one, to see > >>> what people think of using PKG-INFO first, other files second. In > >>> principle, one could reduce a setup.py to just "from setuptools import > >>> setup_distinfo; setup_distinfo()" or some such. > >> > >> In other words, using d2to1 and only for `setup.py egg_info` (only not > >> egg_info but whatever we're doing instead to generate the metadata ;) > >> > >> Erik > > ___ > > Distutils-SIG maillist - Distutils-SIG@python.org > > http://mail.python.org/mailman/listinfo/distutils-sig > > > Rename it and make it JSON instead of the homebrew* format! > > * Yes techincally it's based on a real format, but that format doesn't > support all the things it needs so there are hackishly added extensions > added to it. > > - > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
I am -1 on renaming anything unless it solves a technical problem. Forever after we will have to explain "well, it used to be called X, now it's called Y..." On Tue, Mar 26, 2013 at 5:01 PM, Erik Bray wrote: > On Tue, Mar 26, 2013 at 4:08 PM, PJ Eby wrote: >> On Tue, Mar 26, 2013 at 3:03 PM, Daniel Holth wrote: >>> I think PKG-INFO is a highly human-editable format. >> >> That doesn't mean you necessarily want to edit it yourself; notably, >> there will likely be some redundancy between the description in the >> file and other files like the README. >> >> Also, today one of the key use cases people have for custom code in >> setup.py is to pull the package version from a __version__ attribute >> in a module. (Which is evil, of course, but people do it anyway.) >> >> But it might be worth adding a setuptools feature to pull metadata >> from PKG-INFO (or DIST-INFO) instead of generating a new one, to see >> what people think of using PKG-INFO first, other files second. In >> principle, one could reduce a setup.py to just "from setuptools import >> setup_distinfo; setup_distinfo()" or some such. > > In other words, using d2to1 and only for `setup.py egg_info` (only not > egg_info but whatever we're doing instead to generate the metadata ;) > > Erik ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Tue, Mar 26, 2013 at 4:08 PM, PJ Eby wrote: > On Tue, Mar 26, 2013 at 3:03 PM, Daniel Holth wrote: >> I think PKG-INFO is a highly human-editable format. > > That doesn't mean you necessarily want to edit it yourself; notably, > there will likely be some redundancy between the description in the > file and other files like the README. > > Also, today one of the key use cases people have for custom code in > setup.py is to pull the package version from a __version__ attribute > in a module. (Which is evil, of course, but people do it anyway.) > > But it might be worth adding a setuptools feature to pull metadata > from PKG-INFO (or DIST-INFO) instead of generating a new one, to see > what people think of using PKG-INFO first, other files second. In > principle, one could reduce a setup.py to just "from setuptools import > setup_distinfo; setup_distinfo()" or some such. In other words, using d2to1 and only for `setup.py egg_info` (only not egg_info but whatever we're doing instead to generate the metadata ;) Erik ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Tue, Mar 26, 2013 at 2:48 PM, Nick Coghlan wrote: > On Tue, Mar 26, 2013 at 3:15 PM, PJ Eby wrote: >> More specifically, I was hoping to move the discussion forward on >> nailing down some of the details that still need specifying in a PEP >> somewhere, to finish out what the "new world" of packaging will look >> like. > > I'm deliberately trying to postpone some of those decisions - one of > the reasons distutils2 foundered is because it tried to solve > everything at once, and that's just too big a topic. > > So, *right now*, my focus is on making it possible to systematically > decouple building from installing, so that running "setup.py install" > on a production system becomes as bizarre an idea as running "make > install". > > As we move further back in the tool chain, I want to follow the lead > of the most widely deployed package management systems (i.e. Debian > control files and RPM SPEC files) and provide appropriate configurable > hooks for *invoking* archivers and builders, allowing developers to > choose their own tools, so long as those tools correctly emit > standardised formats understood by the rest of the Python packaging > ecosystem. Right--what we really need here is something akin to the debian/rules file, only not a shell script :) I like the hook idea. It's the "so long as those tools correctly emit standardised formats" that's the problem. > In the near term, however, these hooks will still be based on setup.py > (specifically setuptools rather than raw distutils, so we can update > older versions of Python). That pretty much eases the concerns I brought up in the "backwards compat" thread. >> But, if we support "either you have a setup.cfg specifying your >> archiver, or a PKG-INFO so an archiver isn't needed", then that would >> probably cover all the bases, actually. > > Yeah, you're probably right that we will need to support something > else in addition to the PKG-INFO file. A PKG-INFO.in could work, > though, rather than a completely independent format like setup.cfg. > That way we could easily separate a source checkout/tarball (with > PKG-INFO.in) from an sdist (with PKG-INFO) from a wheel (with a named > .dist-info directory). I'm partly in favor of just saying, "there should be a PKG-INFO in your version control to be considered a valid python distribution". Intermediate formats like a setup.cfg or Nick's JSON format seem kind of unnecessary to me--why have two different formats to describe the same thing? In cases where the metadata needs to be mutated somehow--such as attaching revision numbers to the version--some sort of PKG-INFO.in like you suggest would be great. But I don't see why it should have a different format from PKG-INFO itself. I'd think it would just be a superset of the metadata format but with support for hooks. Basically akin to what d2to1 does with setup.cfg, but without the unnecessarily different-looking intermediate format (I do agree that a JSON format would allow a much greater degree of expression and flexibility, but I'm not entirely sure how I feel about having one file format that generates an entirely different file format). > (For consistency, we may want to rename PKG-INFO to DIST-INFO in sdist > 2.0, though) +1 Thanks, Erik ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP439 and backward compat / easy_install / distlib
On Tue, Mar 26, 2013 at 12:46 PM, Daniel Holth wrote: > On Tue, Mar 26, 2013 at 11:45 AM, Erik Bray wrote: >> On Mon, Mar 25, 2013 at 1:23 PM, Daniel Holth wrote: >>> My vision for the setuptools deprecation process is that distutils >>> rides into the sunset with it. In this future eventually bugs in >>> setuptools will be solved by porting setup.py to one of (X, Y, Z) >>> which haven't necessarily been invented yet. >> >> That would be nice (really!) but what are you proposing replace it for >> building packages with heavy reliance on C extensions? Because for >> that one use case (and perhaps that alone) it works "pretty okay" for >> most cases. I don't want to start seeing an infinite number of ways >> to configure and build extension modules. The great thing about using >> distutils (or some variant) for this is that if I had the source for a >> package I could just `./setup.py build` and it would "just work" for >> all but the most complex cases (SciPy for example). >> >> I don't want to have a situation where some projects are using bento >> and others are using scons and some are using waf and others are using >> autoconf, etc, etc. It's fine if a few projects have their own >> special needs for build toolchains and I've been saying all along that >> building should be separate from installing, and it should be easier >> to drop in one's own build system. >> >> Another thing that setuptools provides that currently works "pretty >> well" with extension modules is `./setup.py develop`. It calls >> `setup.py build_ext --inplace` to make extension modules importable. >> Any build system for extension modules needs to be able to do >> something similar to support in-place install functionality like >> `setup.py develop`, `pip install -e`, etc. > > It's true that de-standardization of the build process has its own > problems. As a consolation you don't have to do it as often because we > have binary packages. It's hard to say how much fragmentation will > happen, but distutils Extension() is an awful way to compile things! > It has very little to recommend it apart from that it's there. For > example, no parallel builds and no partial re-compiles based on what's > changed. The trouble is that we know the packages on the cheeseshop > mostly work but it's harder to count the stuff that avoids the > cheeseshop because setuptools and setup.py wasn't a good solution. > > The example I've had to deal with recently is pycairo. They already > use waf to compile their Python extension and don't use setup.py at > all, so I argue that de-standardization has already happened. Whatever > very easy (for pycairo) thing we can do make them "pip install"-able > again is a plus. > > Even my own simple bcrypt wrapper "cryptacular" would compile an > assembler file if I knew how to make that work with distutils. It > doesn't and it's a little slower than it would have been if I had a > decent build tool. > > I suspect at least 80% of packages will use some simple thing that > comes with Python, two third party build tools will dominate, and we > will discover interesting things that just weren't possible before. At > least if someone wants to improve packaging we can make it easy for > them to try without having to ask distutils-sig. I pretty much agree with you on all of this, but I don't think the question should be ignored either--avoiding this question is one of the things that got previous packaging reform efforts into trouble. Though the agreement to treat "build" and "installation" as two different stories mitigates the issue this time around. In any case it's sort of off topic for this thread so I'll bring it up again elsewhen. One thing I see as a possible short-term solution is to still rely on some version of distutils as a build tool *only*. But it would still be nice to have some easy way to standardize "in-place" installation regardless of how extension modules get built. Erik P.S. pycairo does have a setup.py which worked for me, but the installation instructions say it's "unsupported", though I don't see the waf script doing anything enormously different from it. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Tue, Mar 26, 2013 at 3:03 PM, Daniel Holth wrote: > I think PKG-INFO is a highly human-editable format. That doesn't mean you necessarily want to edit it yourself; notably, there will likely be some redundancy between the description in the file and other files like the README. Also, today one of the key use cases people have for custom code in setup.py is to pull the package version from a __version__ attribute in a module. (Which is evil, of course, but people do it anyway.) But it might be worth adding a setuptools feature to pull metadata from PKG-INFO (or DIST-INFO) instead of generating a new one, to see what people think of using PKG-INFO first, other files second. In principle, one could reduce a setup.py to just "from setuptools import setup_distinfo; setup_distinfo()" or some such. No matter what, though, there's going to be some redundancy with the rest of the project. Some people use revision control tags or other automated tags in their versioning, and it's *precisely* these projects that most need raw source builds. Maybe DIST-INFO shouldn't strictly be a PEP 426-conformant file, but rather, a file that allows some additional metadata to be specified via hooks. That way, you could list your version hook, your readme-generation hook, etc. in it, and then the output gets used to generate the final PKG-INFO. So, call it PKG-INFO.in (as Nick said), or BUILD-INFO, or something like that, add a list of "metadata hooks", and presto: no redundancy in the file, so people can check it into source control, and minimal duplication with your build tool. (Presumably, if you use Bento, your BUILD-INFO file would just list the Bento hook and nothing else, if all the other data comes from Bento's .info file.) Heck, in the minimalist case, you could pretend that a missing BUILD-INFO was there and contained a hook that runs setup.py to troll for the metadata, stopping once setup() is called. ;-) And now it's (mostly) backward compatible. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Tue, Mar 26, 2013 at 2:48 PM, Nick Coghlan wrote: > On Tue, Mar 26, 2013 at 3:15 PM, PJ Eby wrote: >> More specifically, I was hoping to move the discussion forward on >> nailing down some of the details that still need specifying in a PEP >> somewhere, to finish out what the "new world" of packaging will look >> like. > > I'm deliberately trying to postpone some of those decisions - one of > the reasons distutils2 foundered is because it tried to solve > everything at once, and that's just too big a topic. > > So, *right now*, my focus is on making it possible to systematically > decouple building from installing, so that running "setup.py install" > on a production system becomes as bizarre an idea as running "make > install". > > As we move further back in the tool chain, I want to follow the lead > of the most widely deployed package management systems (i.e. Debian > control files and RPM SPEC files) and provide appropriate configurable > hooks for *invoking* archivers and builders, allowing developers to > choose their own tools, so long as those tools correctly emit > standardised formats understood by the rest of the Python packaging > ecosystem. > > In the near term, however, these hooks will still be based on setup.py > (specifically setuptools rather than raw distutils, so we can update > older versions of Python). > >> OTOH, I suppose a case could be made for checking PKG-INFO into source >> control along with the rest of your code, in which case the problem >> disappears entirely: there'd be no such thing as a "raw" source in >> that case. >> >> The downside, though, is that there's a small but vocal contingent >> that believes checking generated files into source control is a sign >> of ultimate evil, so it probably won't be a *popular* choice. >> >> But, if we support "either you have a setup.cfg specifying your >> archiver, or a PKG-INFO so an archiver isn't needed", then that would >> probably cover all the bases, actually. > > Yeah, you're probably right that we will need to support something > else in addition to the PKG-INFO file. A PKG-INFO.in could work, > though, rather than a completely independent format like setup.cfg. > That way we could easily separate a source checkout/tarball (with > PKG-INFO.in) from an sdist (with PKG-INFO) from a wheel (with a named > .dist-info directory). > > (For consistency, we may want to rename PKG-INFO to DIST-INFO in sdist > 2.0, though) > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia With Metadata 2.0 it's pretty feasible that sdists will have a trustworthy PKG-INFO at their root since there can be a lot less changing of requires-dist based on whether you are on win32 (perhaps occasionally still changing based on things we forgot to put into environment marker variables). It would not be surprising to see them also grow a full .dist-info directory (with an unfortunate copy of PKG-INFO, named METADATA) just like sdists tend to contain .egg-info directories. You might always regenerate the file anyway as long as you're running the package's build system. I think PKG-INFO is a highly human-editable format. My hypothetical sdist archiver would validate PKG-INFO instead of regenerating it. It should be clear that I am also in the deliberately postpone as much as possible camp. Daniel ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Tue, Mar 26, 2013 at 3:15 PM, PJ Eby wrote: > More specifically, I was hoping to move the discussion forward on > nailing down some of the details that still need specifying in a PEP > somewhere, to finish out what the "new world" of packaging will look > like. I'm deliberately trying to postpone some of those decisions - one of the reasons distutils2 foundered is because it tried to solve everything at once, and that's just too big a topic. So, *right now*, my focus is on making it possible to systematically decouple building from installing, so that running "setup.py install" on a production system becomes as bizarre an idea as running "make install". As we move further back in the tool chain, I want to follow the lead of the most widely deployed package management systems (i.e. Debian control files and RPM SPEC files) and provide appropriate configurable hooks for *invoking* archivers and builders, allowing developers to choose their own tools, so long as those tools correctly emit standardised formats understood by the rest of the Python packaging ecosystem. In the near term, however, these hooks will still be based on setup.py (specifically setuptools rather than raw distutils, so we can update older versions of Python). > OTOH, I suppose a case could be made for checking PKG-INFO into source > control along with the rest of your code, in which case the problem > disappears entirely: there'd be no such thing as a "raw" source in > that case. > > The downside, though, is that there's a small but vocal contingent > that believes checking generated files into source control is a sign > of ultimate evil, so it probably won't be a *popular* choice. > > But, if we support "either you have a setup.cfg specifying your > archiver, or a PKG-INFO so an archiver isn't needed", then that would > probably cover all the bases, actually. Yeah, you're probably right that we will need to support something else in addition to the PKG-INFO file. A PKG-INFO.in could work, though, rather than a completely independent format like setup.cfg. That way we could easily separate a source checkout/tarball (with PKG-INFO.in) from an sdist (with PKG-INFO) from a wheel (with a named .dist-info directory). (For consistency, we may want to rename PKG-INFO to DIST-INFO in sdist 2.0, though) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A new, experimental packaging tool: distil
Paul Moore gmail.com> writes: > A couple of immediate points. I tried "distil install distribute pip > wheel" which failed, because distribute requires 2to3 to be run as > part of setup.py (no real surprise there). But distil *did* partially > install wheel, leaving a broken installation (there was no METADATA > filein wheel's dist-info directory). I had to manually delete what had > been installed of the 3 projects. I'd suggest that distil needs to > roll back anything it did after a failed install. Another problem with distribute is that you can't install it directly off PyPI with distil, because it does stuff in setup.py in a post-installation step. You will need to use the special wheel I created [1], as I mentioned in my initial post where I showed how to bootstrap pip (or rather, linked to a Gist that shows it being done). Regards, Vinay Sajip [1] https://bitbucket.org/vinay.sajip/distribute3/downloads/ ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Builders vs Installers
On Tue, Mar 26, 2013 at 9:15 AM, PJ Eby wrote: > On balance, I think I lean towards just having a simple way to specify > your chosen archiver, so that installing from source checkouts and > dumps is possible. I just find it annoying that you have to have > *two* files in your checkout, one to say what tool you're using, and > another one to configure it. Ah, you have uncovered part of the cunning plan behind Setup-Requires-Dist and the metadata extension system in 2.0+: once we have the basic hooks in place, then we should be able to embed the config settings for the archivers and builders in the main metadata, without the installer needing to understand the details. Allowing embedded json also supports almost arbitrarily complex config options. For metadata 2.0, however, I'm thinking we should retain the distutils-based status quo for the archive hook and the build hook: Archive hook: python setup.py sdist Environment: current working directory = root dir of source checkout/unpacked tarball Build hook: python setup.py bdist_wheel Environment: current working directory = root dir of unpacked sdist The install tool would then pick up the files from their default output locations. Installing from a checkout/tarball would go through the full daisy chain (make sdist, make wheel from sdist, install the wheel), and installing from sdist would also build the intermediate wheel file. The only entry points inspired hook in 2.0 would be the post-install hook I have discussed previously (and will write up properly in PEP 426 later this week). In theory, we could have separate dependencies for the "make sdist" and "make wheel" parts of the chain, but that seems to add complexity without adequate justification to me. The runtime vs setup split is necessary so that you don't need a build chain on your deployment targets, but it seems comparatively harmless to install the archiver onto a dedicated build system even if you're only building from sdists (particularly when I expect most Python-specific tools to continue to follow the model of handling both archiving and building, rather than having completely separate tools for the two steps). > (What'd be nice is if you could just somehow detect files like > bento.info and setup.cfg and thereby detect what archiver to use. But > that would have limited extensibility unless there was a standard > naming convention for the files, or a standardized format for at least > the first line in the file or something like that, so you could > identify the needed tool.) Yeah, I plan to use future releases of the 2.x metadata to define hooks for this. We can also start experimenting in 2.0 through entry points and the structured metadata format I will be defining for the post install hook. (Daniel has an entry points extension PEP mostly written, we just haven't got around to publishing it yet...) In the meantime, formalising the "setup.py sdist" and "setup.py bdist_wheel" invocations should provide a useful stepping stone to a setup.py-is-optional future. Cheers, Nick. > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP439 and backward compat / easy_install / distlib
On Tue, Mar 26, 2013 at 12:46 PM, Daniel Holth wrote: > I suspect at least 80% of packages will use some simple thing that > comes with Python, two third party build tools will dominate, and we > will discover interesting things that just weren't possible before. At > least if someone wants to improve packaging we can make it easy for > them to try without having to ask distutils-sig. ...and for that matter, without them having to monkeypatch distutils, scrape HTML from the PyPI UI, and sandbox-execute other people's setup.py files. ;-) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP439 and backward compat / easy_install / distlib
On Tue, Mar 26, 2013 at 11:45 AM, Erik Bray wrote: > On Mon, Mar 25, 2013 at 1:23 PM, Daniel Holth wrote: >> My vision for the setuptools deprecation process is that distutils >> rides into the sunset with it. In this future eventually bugs in >> setuptools will be solved by porting setup.py to one of (X, Y, Z) >> which haven't necessarily been invented yet. > > That would be nice (really!) but what are you proposing replace it for > building packages with heavy reliance on C extensions? Because for > that one use case (and perhaps that alone) it works "pretty okay" for > most cases. I don't want to start seeing an infinite number of ways > to configure and build extension modules. The great thing about using > distutils (or some variant) for this is that if I had the source for a > package I could just `./setup.py build` and it would "just work" for > all but the most complex cases (SciPy for example). > > I don't want to have a situation where some projects are using bento > and others are using scons and some are using waf and others are using > autoconf, etc, etc. It's fine if a few projects have their own > special needs for build toolchains and I've been saying all along that > building should be separate from installing, and it should be easier > to drop in one's own build system. > > Another thing that setuptools provides that currently works "pretty > well" with extension modules is `./setup.py develop`. It calls > `setup.py build_ext --inplace` to make extension modules importable. > Any build system for extension modules needs to be able to do > something similar to support in-place install functionality like > `setup.py develop`, `pip install -e`, etc. It's true that de-standardization of the build process has its own problems. As a consolation you don't have to do it as often because we have binary packages. It's hard to say how much fragmentation will happen, but distutils Extension() is an awful way to compile things! It has very little to recommend it apart from that it's there. For example, no parallel builds and no partial re-compiles based on what's changed. The trouble is that we know the packages on the cheeseshop mostly work but it's harder to count the stuff that avoids the cheeseshop because setuptools and setup.py wasn't a good solution. The example I've had to deal with recently is pycairo. They already use waf to compile their Python extension and don't use setup.py at all, so I argue that de-standardization has already happened. Whatever very easy (for pycairo) thing we can do make them "pip install"-able again is a plus. Even my own simple bcrypt wrapper "cryptacular" would compile an assembler file if I knew how to make that work with distutils. It doesn't and it's a little slower than it would have been if I had a decent build tool. I suspect at least 80% of packages will use some simple thing that comes with Python, two third party build tools will dominate, and we will discover interesting things that just weren't possible before. At least if someone wants to improve packaging we can make it easy for them to try without having to ask distutils-sig. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP439 and backward compat / easy_install / distlib
On Tue, Mar 26, 2013 at 11:21 AM, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 03/26/2013 05:28 AM, Ronald Oussoren wrote: >> >> On 25 Mar, 2013, at 19:16, PJ Eby wrote: >>> >>> >>> Also, as far as detecting the need for setuptools, I think that can >>> be done just by noticing whether the PKG-INFO included in an sdist >>> is metadata 2.0 or not. If it is, then setuptools should be >>> explicitly declared as a build-time dependency, otherwise it's not >>> needed. If it's an older metadata version, then you probably need >>> setuptools. >> >> Is it even necessary to automaticly install setuptools? >> Setuptools-using package are supposed to use ez_setup.py, or >> distribute_setup.py for distribute, to ensure that the setuptools >> package is available during setup. > > No, they are not. That usage was for bootstrapping in an era when > setuptools was not widely presetn. Most packages have *removed* those > files today. I still use distribute_setup.py very regularly. I'm dealing with scientific users, mostly on Macs or Windows who barely even know what version of Python they have installed (or even what distribution of Python--python.org/macports/homebrew/etc.) much less that they need some variant of setuptools to install a large percentage of packages out there. Sometimes they do have setuptools installed but it's an outdated version, or they didn't install it properly, or something to that effect. I don't think "downloading the installer" should be a side effect of running the installation either, but until this mess is cleaned up it's a necessary evil. Yes, making things easier for users who don't know what they're doing is a legitimate use case. Erik ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP439 and backward compat / easy_install / distlib
On Tue, Mar 26, 2013 at 11:54 AM, Erik Bray wrote: > I don't think "downloading the installer" should be a side effect of > running the installation either, but until this mess is cleaned up > it's a necessary evil. Yes, making things easier for users who don't > know what they're doing is a legitimate use case. I should clarify--when I write "until this mess is cleaned up" what I really mean is, "as soon as most packages have wheels built for them for a wide range of platforms". Then I don't really see it as an issue :) Erik ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP439 and backward compat / easy_install / distlib
On 26 Mar, 2013, at 16:21, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 03/26/2013 05:28 AM, Ronald Oussoren wrote: >> >> On 25 Mar, 2013, at 19:16, PJ Eby wrote: >>> >>> >>> Also, as far as detecting the need for setuptools, I think that can >>> be done just by noticing whether the PKG-INFO included in an sdist >>> is metadata 2.0 or not. If it is, then setuptools should be >>> explicitly declared as a build-time dependency, otherwise it's not >>> needed. If it's an older metadata version, then you probably need >>> setuptools. >> >> Is it even necessary to automaticly install setuptools? >> Setuptools-using package are supposed to use ez_setup.py, or >> distribute_setup.py for distribute, to ensure that the setuptools >> package is available during setup. > > No, they are not. That usage was for bootstrapping in an era when > setuptools was not widely presetn. Most packages have *removed* those > files today. I didn't know that, all my project still include the bootstrap code to make it easier to install them in a fresh build of python. The distribute docs still mention that you should use distribute_setup.py (their version of ez_setup.py) in your project. Ronald ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP439 and backward compat / easy_install / distlib
On Mon, Mar 25, 2013 at 1:23 PM, Daniel Holth wrote: > My vision for the setuptools deprecation process is that distutils > rides into the sunset with it. In this future eventually bugs in > setuptools will be solved by porting setup.py to one of (X, Y, Z) > which haven't necessarily been invented yet. That would be nice (really!) but what are you proposing replace it for building packages with heavy reliance on C extensions? Because for that one use case (and perhaps that alone) it works "pretty okay" for most cases. I don't want to start seeing an infinite number of ways to configure and build extension modules. The great thing about using distutils (or some variant) for this is that if I had the source for a package I could just `./setup.py build` and it would "just work" for all but the most complex cases (SciPy for example). I don't want to have a situation where some projects are using bento and others are using scons and some are using waf and others are using autoconf, etc, etc. It's fine if a few projects have their own special needs for build toolchains and I've been saying all along that building should be separate from installing, and it should be easier to drop in one's own build system. Another thing that setuptools provides that currently works "pretty well" with extension modules is `./setup.py develop`. It calls `setup.py build_ext --inplace` to make extension modules importable. Any build system for extension modules needs to be able to do something similar to support in-place install functionality like `setup.py develop`, `pip install -e`, etc. Erik ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP439 and backward compat / easy_install / distlib
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 03/26/2013 05:28 AM, Ronald Oussoren wrote: > > On 25 Mar, 2013, at 19:16, PJ Eby wrote: >> >> >> Also, as far as detecting the need for setuptools, I think that can >> be done just by noticing whether the PKG-INFO included in an sdist >> is metadata 2.0 or not. If it is, then setuptools should be >> explicitly declared as a build-time dependency, otherwise it's not >> needed. If it's an older metadata version, then you probably need >> setuptools. > > Is it even necessary to automaticly install setuptools? > Setuptools-using package are supposed to use ez_setup.py, or > distribute_setup.py for distribute, to ensure that the setuptools > package is available during setup. No, they are not. That usage was for bootstrapping in an era when setuptools was not widely presetn. Most packages have *removed* those files today. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlFRvPAACgkQ+gerLs4ltQ6XhgCgknMlM9drnL5KJKSvoEcuoKqw 60gAn1QyyUersaUdKXbJrpnJuu3AXkzz =i63/ -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 updated
Made some progress on the wheel signature system that fills my design requirements of being key-centric and emphatically not GPG. It turns out RSA signature verification is just pow(signature, pubkey.e, pubkey.n) and some hashing. You would be able to use "openssl genrsa -out private.pem 2048" to generate the private key, "openssl dgst -sha256 -sign private.pem -binary < partial_jws_blob" to do the actual signature, and use key fingerprints (the same 32-byte length as literal Ed25519 public keys) when asking for "something signed with a particular key or keys". RSA, while producing slower and bigger signatures than the elliptic curve Ed25519, would be more palatable to some by being a more conservative choice and you would be able to use openssl for key management. The idea of "multiple signatures / no key revocation" would be limited to "we don't have tuf yet" installs of things like pip or tuf itself, once tuf was available more complex trust delegation would be available and more subtle attacks could be detected. The idea is to have a security system with a tiny implementation when you do not have, want or need something more complex. On Mon, Mar 25, 2013 at 11:55 PM, Richard Jones wrote: > Hi all, > > I've updated PEP 439 to note the outcome of the recent discussion > regarding setuptools dependencies and a couple of other minor things. > > The changes are viewable here: > http://hg.python.org/peps/diff/0d57c70eff91/pep-0439.txt > > > Richard > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP439 and backward compat / easy_install / distlib
> Just assuming that every sdist with old metadata requires setuptools would > work, although it will be strange to see that some packages @work that use > plain disutils suddenly seem to require setuptool :-) pip does this already, importing setuptools before running any setup.py >> These improved installers will target both 2.7 and 3.4. I do understand that >> some people feel it is harder to say "manually download the installer and >> then install what you want" rather than "manually download and install the >> package you want". > > No me. I'm glad to see that the hard work by everyone working in the > packaging space is coming to fruition. Infrastructure work is almost never > glamorous, and work on Python's packaging system appears to be more stressful > than average. You said it. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP439 and backward compat / easy_install / distlib
On 26 Mar, 2013, at 13:27, Daniel Holth wrote: > stall themselves, they can just mention setup-requires and the installer > grabs the necessary setuptools. > > > > > That can only be done for sdists with 2.0 metadata, sdists for older > > versions don't have a setup-requires in their metadata. This is not just > > for installing, if you want to use setuptools in your setup.py you'll have > > to make sure it is installed in your setup.py, and with the current version > > of the packaging tools this means you have to use something like > > ez_setup.py or tell users to install setuptools themselves. > > Yes, which is why we propose to assume Setup-Requires-Dist: setuptools if > Metadata-Version < 2.0. Then a no-op ez_setup.py can be added to sys.modules > before setup.py runs and the installer will have a lot more control over that > side effect. Just because I'm curious, is that control needed to make sure that a new enough version of setuptools gets used (e.g. one that supports modern features, instead of the 2 year old version that is mentioned in ez_setup.py for $SOME_OLD_PACKAGE)? Just assuming that every sdist with old metadata requires setuptools would work, although it will be strange to see that some packages @work that use plain disutils suddenly seem to require setuptool :-) > > > These improved installers will target both 2.7 and 3.4. I do understand that > some people feel it is harder to say "manually download the installer and > then install what you want" rather than "manually download and install the > package you want". No me. I'm glad to see that the hard work by everyone working in the packaging space is coming to fruition. Infrastructure work is almost never glamorous, and work on Python's packaging system appears to be more stressful than average. Ronald ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP439 and backward compat / easy_install / distlib
On Mar 26, 2013 7:26 AM, "Ronald Oussoren" wrote: > > > On 26 Mar, 2013, at 12:06, Daniel Holth wrote: > > > > > On Mar 26, 2013 5:28 AM, "Ronald Oussoren" wrote: > > > > > > > > > On 25 Mar, 2013, at 19:16, PJ Eby wrote: > > > > > > > > > > > > Also, as far as detecting the need for setuptools, I think that can be > > > > done just by noticing whether the PKG-INFO included in an sdist is > > > > metadata 2.0 or not. If it is, then setuptools should be explicitly > > > > declared as a build-time dependency, otherwise it's not needed. If > > > > it's an older metadata version, then you probably need setuptools. > > > > > > Is it even necessary to automaticly install setuptools? Setuptools-using package are supposed to use ez_setup.py, or distribute_setup.py for distribute, to ensure that the setuptools package is available during setup. Although I must admit that I have no idea how many packages still do this instead of assuming that users will have installed setuptools anyway. > > > > > > Ronald > > > > > > > We really really really want to get rid of ez_setup. It is considered by many to be the example of something that should not happen as a side effect of running a build script. > > That can't be helped with the current tool versions, distutils in current release of python doesn't support setup-requires and hence the only way to use setuptools is by using ez_setup.py. Ez_setup.py will still have to be present for python 2.7 users that want to use "python setup.py ...", and hence can be assumed to be present for now. I'm all for adding support for metadata 2.0 to the stdlib for Python 3.4, that way ez_setup.py can be phased out in the long run. > > > > > When packages no longer have to install themselves, they can just mention setup-requires and the installer grabs the necessary setuptools. > > > That can only be done for sdists with 2.0 metadata, sdists for older versions don't have a setup-requires in their metadata. This is not just for installing, if you want to use setuptools in your setup.py you'll have to make sure it is installed in your setup.py, and with the current version of the packaging tools this means you have to use something like ez_setup.py or tell users to install setuptools themselves. Yes, which is why we propose to assume Setup-Requires-Dist: setuptools if Metadata-Version < 2.0. Then a no-op ez_setup.py can be added to sys.modules before setup.py runs and the installer will have a lot more control over that side effect. These improved installers will target both 2.7 and 3.4. I do understand that some people feel it is harder to say "manually download the installer and then install what you want" rather than "manually download and install the package you want". > And with some luck a large subset of packages will ship wheels in the future, that way the installer doesn't even have to look at sdists. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] PEP 439 updated
Hi all, I've updated PEP 439 to note the outcome of the recent discussion regarding setuptools dependencies and a couple of other minor things. The changes are viewable here: http://hg.python.org/peps/diff/0d57c70eff91/pep-0439.txt Richard ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A new, experimental packaging tool: distil
Paul Moore gmail.com> writes: > Yes, now it just installs coverage.exe. (No coverage-3.3.exe? Not that > it bothers me, I don't use the version-specific script wrappers > anyway). > So distil (or is it distlib?) uses metadata from www.red-dove.com as > well as PyPI? That's a bit of a surprise. I presume this is a > short-term fix, what's the longer-term plan for getting such metadata > onto PyPI? Yes, it's a short-term fix because otherwise it would be no better than pip in the dependency resolution department: download each dist, run egg_info, look for dependencies, download them, rinse and repeat. I'd love to get this metadata onto PyPI, but that depends on the PyPI folks + for the metadata to be accepted as a format. For it to be proven as a useful format, it needs wider exposure ... so that's what I'm hoping for. No doubt the wider exposure will lead to improvements. You can think of the red-dove.com location as just a sort of unofficial early version of what could be on PyPI, if the relevant people agree it's useful. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP439 and backward compat / easy_install / distlib
On 26 Mar, 2013, at 12:06, Daniel Holth wrote: > > On Mar 26, 2013 5:28 AM, "Ronald Oussoren" wrote: > > > > > > On 25 Mar, 2013, at 19:16, PJ Eby wrote: > > > > > > > > > Also, as far as detecting the need for setuptools, I think that can be > > > done just by noticing whether the PKG-INFO included in an sdist is > > > metadata 2.0 or not. If it is, then setuptools should be explicitly > > > declared as a build-time dependency, otherwise it's not needed. If > > > it's an older metadata version, then you probably need setuptools. > > > > Is it even necessary to automaticly install setuptools? Setuptools-using > > package are supposed to use ez_setup.py, or distribute_setup.py for > > distribute, to ensure that the setuptools package is available during > > setup. Although I must admit that I have no idea how many packages still > > do this instead of assuming that users will have installed setuptools > > anyway. > > > > Ronald > > > > We really really really want to get rid of ez_setup. It is considered by many > to be the example of something that should not happen as a side effect of > running a build script. That can't be helped with the current tool versions, distutils in current release of python doesn't support setup-requires and hence the only way to use setuptools is by using ez_setup.py. Ez_setup.py will still have to be present for python 2.7 users that want to use "python setup.py ...", and hence can be assumed to be present for now. I'm all for adding support for metadata 2.0 to the stdlib for Python 3.4, that way ez_setup.py can be phased out in the long run. > > When packages no longer have to install themselves, they can just mention > setup-requires and the installer grabs the necessary setuptools. > That can only be done for sdists with 2.0 metadata, sdists for older versions don't have a setup-requires in their metadata. This is not just for installing, if you want to use setuptools in your setup.py you'll have to make sure it is installed in your setup.py, and with the current version of the packaging tools this means you have to use something like ez_setup.py or tell users to install setuptools themselves. And with some luck a large subset of packages will ship wheels in the future, that way the installer doesn't even have to look at sdists. Ronald ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP439 and backward compat / easy_install / distlib
On Mar 26, 2013 5:28 AM, "Ronald Oussoren" wrote: > > > On 25 Mar, 2013, at 19:16, PJ Eby wrote: > > > > > > Also, as far as detecting the need for setuptools, I think that can be > > done just by noticing whether the PKG-INFO included in an sdist is > > metadata 2.0 or not. If it is, then setuptools should be explicitly > > declared as a build-time dependency, otherwise it's not needed. If > > it's an older metadata version, then you probably need setuptools. > > Is it even necessary to automaticly install setuptools? Setuptools-using package are supposed to use ez_setup.py, or distribute_setup.py for distribute, to ensure that the setuptools package is available during setup. Although I must admit that I have no idea how many packages still do this instead of assuming that users will have installed setuptools anyway. > > Ronald > We really really really want to get rid of ez_setup. It is considered by many to be the example of something that should not happen as a side effect of running a build script. When packages no longer have to install themselves, they can just mention setup-requires and the installer grabs the necessary setuptools. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A new, experimental packaging tool: distil
On 26 March 2013 10:57, Vinay Sajip wrote: > Ah. The metadata (see [1] for an example) mentions "coverage-2.7" as a > script, as > it was built on 2.7. That shouldn't really be in the metadata - there should > be a > single declaration, which is used by distlib/distil to create version-specific > aliases. > > I've now removed it from the metadata from 3.6, you could try again using > > distil install "coverage (3.6)" > > to make sure you pick up the version I changed. Yes, now it just installs coverage.exe. (No coverage-3.3.exe? Not that it bothers me, I don't use the version-specific script wrappers anyway). > [1] http://www.red-dove.com/pypi/projects/C/coverage/package-3.6b3.json So distil (or is it distlib?) uses metadata from www.red-dove.com as well as PyPI? That's a bit of a surprise. I presume this is a short-term fix, what's the longer-term plan for getting such metadata onto PyPI? Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A new, experimental packaging tool: distil
Paul Moore gmail.com> writes: > I installed coverage into an empty virtuwlenv based on Python 3.3. > > It installed coverage-2.7 and coverage2 executables into Scripts. Why > 2.7? Where did it get the idea that this was a Python 2.7 > installation? I ran distil with the python 3.3 that is installed in > the virtualenv. Ah. The metadata (see [1] for an example) mentions "coverage-2.7" as a script, as it was built on 2.7. That shouldn't really be in the metadata - there should be a single declaration, which is used by distlib/distil to create version-specific aliases. I've now removed it from the metadata from 3.6, you could try again using distil install "coverage (3.6)" to make sure you pick up the version I changed. Regards, Vinay Sajip [1] http://www.red-dove.com/pypi/projects/C/coverage/package-3.6b3.json ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A new, experimental packaging tool: distil
Paul Moore gmail.com> writes: > A couple of immediate points. I tried "distil install distribute pip > wheel" which failed, because distribute requires 2to3 to be run as > part of setup.py (no real surprise there). But distil *did* partially > install wheel, leaving a broken installation (there was no METADATA > filein wheel's dist-info directory). I had to manually delete what had > been installed of the 3 projects. I'd suggest that distil needs to > roll back anything it did after a failed install. There is code in distil to roll back when installation fails, but there could be a bug which prevents it kicking in. I'll investigate. Distil does invoke 2to3 automatically if the metadata indicates it, but the metadata for distribute might be wrong if it was built on a 2.x system. I'll investigate this. > Secondly, when there is a C extension in the distribution (on Windows) > the install fails even though I have Visual C installed. This is > because cl.exe is not on my PATH - distil should do the same detection > of the location of Visual C as distutils does. The install does work > if cl.exe is on PATH - presumably, though, it doesn't check that it is > the *right* cl.exe (2010 for Python 3.3, 2008 for 2007, etc). Also > distil doesn't deal with packages with optional C extensions - but > again, that's a case of a "too complex" setup.py (and I'm glad it > picks the option of installing the C extension in that case, and not > just the pure Python version). distil could certainly be improved in this area, but the documentation [1] mentions that C builds should be run in a Visual Studio command window. The checking for the right version of Visual Studio is for a little later, but it's on my list. > PS I'm not entirely happy with the default of installing to the user > packages directory. 99.9% of my time, I'm installing into a > virtualenv, and this default is very wrong - as the installed packages > will "infect" all of my virtualenvs. In what way does "distil -e install distname" fall short of your expectations? If you have a venv activated, it should install in there - does it not do this? Regards, Vinay Sajip [1] http://pythonhosted.org/distil/installing.html#distributions-which-include-c-extensions ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A new, experimental packaging tool: distil
On 26 March 2013 08:54, Vinay Sajip wrote: > I would welcome any feedback you could give regarding distil/distlib. There is > of course a lot more testing to do, but I consider these initial findings to > be > promising, and worth sharing. If you find any problems, you can raise issues > at [5]. One other (slight) oddity. I installed coverage into an empty virtuwlenv based on Python 3.3. It installed coverage-2.7 and coverage2 executables into Scripts. Why 2.7? Where did it get the idea that this was a Python 2.7 installation? I ran distil with the python 3.3 that is installed in the virtualenv. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A new, experimental packaging tool: distil
Philippe Ombredanne nexb.com> writes: > I see that you are using a pattern similar to the virtualenv.py > script, embedding other code as a compressed byte array. > > I am in general fine with the approach, though I feel a bit uncomfy > with this approach creeping in as "the" way to bootstrap things with > one single file for core distribution-related tools. It's not a particularly new approach - it's just that way because it makes things easier for the user. If I had used a more conventional approach, I'm not sure as many people would be willing to try it. Like virtualenv, it's a tool that cannot rely on the presence of existing installation tools. > Would anyone know of a better way to package things in a single > python-executable bootstrapping script file without obfuscating the > source contents in compressed/encoded/obfuscated byte arrays? It's only obfuscated as a side-effect - the other way would be to put all your code in a single module - not much fun to maintain, that way. But if someone has a better way, that would certainly be of interest. > Also, in your code calling this binary payload STUFF feels a tad > scary: this is arbitrary code that I cannot see nor inspect before > running. Would you find it more trustworthy if it was called TRUST_ME_ITS_SAFE? ;-) Remember, it's just a Python script running without system privileges. > I would not want to run unknown STUFFs on my machine ... and even more > so since the corresponding sources are not available publicly yet in a > source repo. The absence of a public source repo is a red herring. If you want to inspect the code, the time taken to add a pdb breakpoint after the .zip write, and to unzip the file to a folder of your choice (or to add code to distil.py to do this), is trivial compared to the time you would spend doing the actual code inspection. The code is open to inspection, but I'd hope that most users focus on whether the tool has useful qualities, how it could be used to move packaging forwards, what it demonstrates about distlib etc. Have you inspected setuptools or pip code to verify that they are safe? As well as everything you've ever downloaded from PyPI, which might or might not be exactly the same as what's shown in a project's public VCS repo? > At the minimum, getting some comments or explicit variable names the > virtualenv way on what this payload is would help IMHO: > > "STUFF = """ > eJyEm1OMLlC3Zb+ybbtO1Snbtm3brjpl27Zt27Zt27b6Tye3b9J9k37YK9kv+2FmPMxkrC0vBQKKCgA > AIAGIUeqCCqFgEh/4AACRBQCAD8AFGFs4OVtbGNLpGRoYWdnbOTrTObk7GdnZmlqY0dq7qyhDAUDqZP > .." > While virtualenv has a number of discrete files, I have just one zip file containing distlib, CLI support code and distil code - that's a lot of files, so I'm not sure a comment would be all that helpful. What would it really tell you? What "STUFF" is really saying, to most users, is "stuff you don't need to care about the details of". For the security-conscious, a mere comment from a potentially untrusted source is no substitute for that unzip + time-consuming code inspection. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A new, experimental packaging tool: distil
On 26 March 2013 08:54, Vinay Sajip wrote: > I would welcome any feedback you could give regarding distil/distlib. There is > of course a lot more testing to do, but I consider these initial findings to > be > promising, and worth sharing. If you find any problems, you can raise issues > at [5]. A couple of immediate points. I tried "distil install distribute pip wheel" which failed, because distribute requires 2to3 to be run as part of setup.py (no real surprise there). But distil *did* partially install wheel, leaving a broken installation (there was no METADATA filein wheel's dist-info directory). I had to manually delete what had been installed of the 3 projects. I'd suggest that distil needs to roll back anything it did after a failed install. Secondly, when there is a C extension in the distribution (on Windows) the install fails even though I have Visual C installed. This is because cl.exe is not on my PATH - distil should do the same detection of the location of Visual C as distutils does. The install does work if cl.exe is on PATH - presumably, though, it doesn't check that it is the *right* cl.exe (2010 for Python 3.3, 2008 for 2007, etc). Also distil doesn't deal with packages with optional C extensions - but again, that's a case of a "too complex" setup.py (and I'm glad it picks the option of installing the C extension in that case, and not just the pure Python version). But other than these niggles, it's impressively effective so far :-) Paul. PS I'm not entirely happy with the default of installing to the user packages directory. 99.9% of my time, I'm installing into a virtualenv, and this default is very wrong - as the installed packages will "infect" all of my virtualenvs. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A new, experimental packaging tool: distil
On Tue, Mar 26, 2013 at 9:54 AM, Vinay Sajip wrote: > I've created a new tool called distil which I'm using to experiment with > packaging functionality. Thanks for doing this, I think it's a good way forward. //Lennart ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A new, experimental packaging tool: distil
On 26 March 2013 09:49, Philippe Ombredanne wrote: > Would anyone know of a better way to package things in a single > python-executable bootstrapping script file without obfuscating the > source contents in compressed/encoded/obfuscated byte arrays? Packaging as a zip file is a good way - but on Windows the file needs to be named xxx.py (which is surprising, to say the least :-)) for the relevant file association to be triggered (and on Unix, a #! line needs to be prepended). Windows users could define additional associations (pyz and pywz) for "zipped Python applications". Maybe the Python installer should include these in 3.4+, to improve the visibility of this approach. Paul. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A new, experimental packaging tool: distil
On Tue, Mar 26, 2013 at 9:54 AM, Vinay Sajip wrote: > I've created a new tool called distil which I'm using to experiment with > packaging functionality. Nice! > * Very simple deployment - just copy distil.py[1] to a location on your path, > optionally naming it to distil on POSIX platforms. There's no need to > install > distlib - it's all included. I see that you are using a pattern similar to the virtualenv.py script, embedding other code as a compressed byte array. See how virtualenv.py is turning out to be lately: https://github.com/pypa/virtualenv/blob/11ccab2698274f0e10b72da863f9efb73cf1a9aa/virtualenv.py#L1937 I am in general fine with the approach, though I feel a bit uncomfy with this approach creeping in as "the" way to bootstrap things with one single file for core distribution-related tools. Would anyone know of a better way to package things in a single python-executable bootstrapping script file without obfuscating the source contents in compressed/encoded/obfuscated byte arrays? Also, in your code calling this binary payload STUFF feels a tad scary: this is arbitrary code that I cannot see nor inspect before running. I would not want to run unknown STUFFs on my machine ... and even more so since the corresponding sources are not available publicly yet in a source repo. At the minimum, getting some comments or explicit variable names the virtualenv way on what this payload is would help IMHO: "STUFF = """ eJyEm1OMLlC3Zb+ybbtO1Snbtm3brjpl27Zt27Zt27b6Tye3b9J9k37YK9kv+2FmPMxkrC0vBQKKCgA AIAGIUeqCCqFgEh/4AACRBQCAD8AFGFs4OVtbGNLpGRoYWdnbOTrTObk7GdnZmlqY0dq7qyhDAUDqZP .." -- Cordially Philippe Ombredanne ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A new, experimental packaging tool: distil
Paul Moore gmail.com> writes: > Interesting! Is the source available anywhere? (Not distil.py, but the Not in a public VCS repo - I'm not soliciting contributions, as I'm still experimenting with a few features. But it'd be nice if the packaging-savvy readers here played with it just as users, and gave some feedback from that perspective. > source of the big chunk of embedded zipfile that appears to contain > the bulk of the functionality...) That zipfile contains distlib, some CLI support code and a packager.py, which contains the distil core functionality. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP439 and backward compat / easy_install / distlib
On 25 Mar, 2013, at 19:16, PJ Eby wrote: > > > Also, as far as detecting the need for setuptools, I think that can be > done just by noticing whether the PKG-INFO included in an sdist is > metadata 2.0 or not. If it is, then setuptools should be explicitly > declared as a build-time dependency, otherwise it's not needed. If > it's an older metadata version, then you probably need setuptools. Is it even necessary to automaticly install setuptools? Setuptools-using package are supposed to use ez_setup.py, or distribute_setup.py for distribute, to ensure that the setuptools package is available during setup. Although I must admit that I have no idea how many packages still do this instead of assuming that users will have installed setuptools anyway. Ronald ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A new, experimental packaging tool: distil
On 26 March 2013 08:54, Vinay Sajip wrote: > I've created a new tool called distil which I'm using to experiment with > packaging functionality. Interesting! Is the source available anywhere? (Not distil.py, but the source of the big chunk of embedded zipfile that appears to contain the bulk of the functionality...) No big deal if not, I can easily enough unpack the data from distil.py Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] A new, experimental packaging tool: distil
I've created a new tool called distil which I'm using to experiment with packaging functionality. Overview It's based on distlib and has IMO some interesting features. With it, one can: * Install projects from PyPI and wheels (see PEP 427). Distil does not invoke setup.py, so projects that do significant computation in setup.py may not be installable by distil. However, a large number of projects on PyPI *can* be installed, and dependencies are detected, downloaded and installed. For those distributions that absolutely *have* to run setup.py, distil can create wheels using pip as a helper, and then install from those wheels. * Optionally upgrade installed distributions, whether installed by distil or installed by pip. * Uninstall distributions installed by distil or pip. * Build source distributions in .tar.gz, .tar.bz2, and .zip formats. * Build binary distributions in wheel format. These can be pure-Python, or have C libraries and extensions. Support for Cython and Fortran (using f2py) is possible, though currently distil cannot install Cython or Numpy directly because of how they use setup.py. * Run tests on built distributions. * Register projects on PyPI. * Upload distributions to PyPI. * Upload documentation to http://pythonhosted.org/. * Display dependencies of a distribution - either as a list of what would be downloaded (and a suggested download order), or in Graphviz format suitable for conversion to an image. Getting started is simple (documentation is at [2]): * Very simple deployment - just copy distil.py[1] to a location on your path, optionally naming it to distil on POSIX platforms. There's no need to install distlib - it's all included. * Uses either a system Python or one in a virtual environment, but by default installs to the user site rather than system Python library locations. * Offers tab-completion and abbreviation of commands and parameters on Bash-compatible shells. Logically, packaging activities can be divided into a number of categories or roles: * Archiver - builds source distributions from a source tree * Builder - builds binary distributions from source * Installer - installs source or binary distributions This version of distil incorporates (for convenience) all of the above roles. There is a school of thought which says that that these roles should be fulfilled by separate programs, and that's fine for production quality tools - it's just more convenient for now to have everything in one package for an experimental tool like distil. Actual Improvements --- Despite the fact that distil is in an alpha stage of development and has received no real-world exposure like the existing go-to packaging tools, it does offer some improvements over them: * Dependency resolution can be performed without downloading any distributions. Unlike e.g. pip, you are told which additional dependencies will be downloaded and installed, before any download occurs. * Better information is stored for uninstallation. This allows better feedback to be given to users during uninstallation. * Dependency checking is done during uninstallation. Say you've installed a distribution A, which pulled in dependencies B and C. If you request an uninstallation of B (or C), distil will complain that you can't do this because A needs it. When you uninstall A, you are offered the option to uninstall B and C as well (assuming you didn't install something else that depends on B or C, after installing A). * By default, installation is to the user site and not to the system Python, so you shouldn't need to invoke sudo to install distributions for personal use which are not for specific projects/virtual environments. * There's no need to "install" distil - the exact same script will run with any system Python or any venv (subject to Python version constraints of 2.6, 2.7, 3.2 or greater). Bootstrapping pip - I've used distil to bootstrap pip, then used that pip to install other stuff. I created a fresh PEP 405 venv with nothing in it, used distil to install a wheel[3] for my distribute fork which runs on Python 2.x and 3.x, then used distil to install pip from PyPI. Finally, to test pip, I installed SQLAlchemy (using pip) from PyPI. See [4] for the transcript. I would welcome any feedback you could give regarding distil/distlib. There is of course a lot more testing to do, but I consider these initial findings to be promising, and worth sharing. If you find any problems, you can raise issues at [5]. Regards, Vinay Sajip [1] https://bitbucket.org/vinay.sajip/docs-distil/downloads/distil.py [2] https://pythonhosted.org/distil/ [3] https://bitbucket.org/vinay.sajip/distribute3/downloads/ [4] https://gist.github.com/vsajip/5243936 [5] https://bitbucket.org/vinay.sajip/distlib/issues/new ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mai
