Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
Paul Moore gmail.com> writes: > Understood - that's why I suggested that distlib reach a point where > it's stable as an external package and supported on (some) older > versions. I'm hoping for an experience more like unittest2 than > packaging/distutils2. Currently, distlib runs on Python 2.6, 2.7 and 3.2+, so there shouldn't be any concerns about support on those older Python versions. Regards, Vinay Sajip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
Chris Jerdonek gmail.com> writes: > Maybe this is already stated somewhere, but is there a plan for when > distlib will be brought into the repository? Is there a reason not to > do it now? It seems it would have more visibility that way (e.g. > people could see it as part of the development version of the online > docs and it would be in check-ins as are PEP edits), and its status > relative to Python would be clearer. There's no firm plan other than it being on the 3.4 wishlist, but IMO there's no rush to do it right now. AFAICT interested parties can be in the loop even though the project is actively developed on BitBucket rather than hg.python.org: remember that the current stakeholders in terms of packaging tools are not all in python-dev and may find it easier to contribute on BitBucket. (I often get comments from Daniel Holth in response to distlib commits I push to BitBucket.) I periodically sync the BitBucket repo with hg.python.org/distlib. The docs are also readily accessible on readthedocs.org (the first two Google results for distlib are those docs and the project on BitBucket). While not integrated with docs.python.org, I don't think that the distlib docs suffer inordinately from being on readthedocs.org - for one thing, I've enabled Disqus comments, allowing feedback on the docs - not that I've had any ;-) Closer integration should perhaps happen, but not until the project has matured a little. Regards, Vinay Sajip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
M.-A. Lemburg egenix.com> writes: > The suggestion to have the metadata available on PyPI doesn't > have anything to do with security. > > It's about being able to determine compatibility and select the > right distribution file for download. The metadata also helps in > creating dependency graphs, which are useful for a lot of things. FYI, distlib already uses metadata in this way. Note that: 1. Dependency information is locked away in setup.py files, e.g in kwargs to the setup() call. 2. PyPI does not provide access to this metadata. So I have had to write code to extract what information I could out of setup.py setup() and into metadata that can be used. I have, for now, hosted it on my own server, but it can be used for e.g. determining complete dependency graphs without ever actually downloading any distributions. Distlib provides access to it: $ python finddeps.py pyramid 12 dists, elapsed time: 4.35 secs Download order: download setuptools (0.6c11) [for pyramid, zope.interface, zope.deprecation] download PasteDeploy (1.5.0) [for pyramid] download repoze.lru (0.6) [for pyramid] download venusian (1.0a7) [for pyramid] download MarkupSafe (0.15) [for Mako] download translationstring (1.1) [for pyramid] download Chameleon (2.11) [for pyramid] download WebOb (1.2.3) [for pyramid] download zope.interface (4.0.3) [for pyramid] download Mako (0.7.3) [for pyramid] download zope.deprecation (4.0.2) [for pyramid] download pyramid (1.4) In the above, nothing was actually downloaded except the metadata. It's just indicating an intuitive ordering for download based on a topological sort of the dependency graph - for pyramid, in this case. Regards, Vinay Sajip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Wed, Feb 20, 2013 at 5:30 AM, M.-A. Lemburg wrote: > The wording in the PEP alienates the egg format by defining > an incompatible new standard for the location of the metadata > file: This isn't a problem, because there's not really a use case at the moment for eggs to include a PEP 426-format metadata file, and if they ever do, it ought to be called METADATA, so that pkg_resources will know to read it if there are no egg-format dependencies listed. Setuptools also doesn't care what format PKG-INFO is in, as it only ever reads the "Version:" field, and that only in the case of in-development source packages. > It's easy to upgrade distribute and distutils to write > metadata 1.2 format, simply by changing the version in the > PKG-INFO files. As soon as distutils does it, setuptools will do it, because setuptools delegates metadata writing to distutils. So there's no "alienation" here. What will need to happen at some point is for pkg_resources to implement support for PEP 426-style version requirements, which I haven't tried to fully wrap my head around as yet. I'm hoping that there are simple textual substitutions (e.g. regexes) that can be done to convert them to pkg_resources requirements. If need be, I'll swipe whatever's needed from distlib. ;-) In the meantime, eggs aren't actually going anywhere, since wheels aren't actually trying to replace all of their use cases. And since the new metadata and binary formats don't actually add much new functionality over what eggs already do, eggs wouldn't lose much by not being able to use the same metadata, anyway. ;-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On 20.02.2013 00:16, Daniel Holth wrote:
> On Tue, Feb 19, 2013 at 5:10 PM, M.-A. Lemburg wrote:
>
>> On 19.02.2013 23:01, Daniel Holth wrote:
>>> On Tue, Feb 19, 2013 at 4:34 PM, M.-A. Lemburg wrote:
>>>
On 19.02.2013 14:40, Nick Coghlan wrote:
> On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg
>> wrote:
>> * PEP 426 doesn't include any mention of the egg distribution format,
>> even though it's the most popular distribution format at the moment.
>> It should at least include the location of the metadata file
>> in eggs (EGG-INFO/PKG-INFO) and egg installations
>> (/EGG-INFO/PKG-INFO).
>
> "Other tools involved in Python distribution may also use this format."
>
> The egg format has never been, and never will be, officially endorsed
> by python-dev. The wheel format is the standard format for binary
> distribution, and PEP 376 defines the standard location for metadata
> on installed distributions.
Oh, come on, Nick, that's just silly. setuptools was included in stdlib
for a short while, so the above is simply wrong. Eggs are the most
widely used binary distribution format for Python package on PyPI:
# wc *files.csv
25585 25598 1431013 2013-02-19-egg-files.csv
46194640 236694 2013-02-19-exe-files.csv
254 255 13402 2013-02-19-msi-files.csv
104691 104853 5251962 2013-02-19-tar-gz-files.csv
24 241221 2013-02-19-whl-files.csv
17937 18022 905913 2013-02-19-zip-files.csv
153110 153392 7840205 total
(based on todays PyPI stats)
It doesn't really help ignoring realities... and I'm saying
that as one of the core devs who got setuptools kicked out of
the stdlib again.
--
Marc-Andre Lemburg
eGenix.com
>>>
>>> The wheel philosophy is that it should be supported by both python-dev
>> and
>>> setuptools and that you should feel happy about using setuptools if you
>>> like it whether or not python-dev (currently) endorses that. If you are
>>> using setuptools (distribute's pkg_resources) then you can use both at
>> the
>>> same time.
>>>
>>> Distribute, distutils and setuptools' problems have not been well
>>> understood which I think is why there has been a need to discredit
>>> setuptools by calling it non-standard. It is the defacto standard. If
>> your
>>> packages have dependencies there is no other choice. Wheel tries to solve
>>> the real problem by allowing you to build a package with setuptools while
>>> giving the end-user the choice of installing setuptools or not.
>>>
>>> Of course eggs are the most popular right now. The wheel format is very
>>> egg-like while avoiding some of egg's problems. See the comparison in the
>>> PEP or read the story on wheel's rtfd. The wheel project includes tools
>> to
>>> losslessly convert eggs or bdist_wininst to wheel.
>>
>> That's all fine, but it doesn't explain the refusal to add the
>> documentation of the location of the PKG-INFO file in eggs ?
>
>
> It would just be a sentence, I wouldn't have a problem with it but I also
> don't see why it would be necessary. Even setuptools doesn't touch the file
> usually. Right now distribute's pkg_resources currently only understands
> Requires-Dist if it is inside a .dist-info directory.
Perhaps I'm not clear enough. I'll try again :-)
The wording in the PEP alienates the egg format by defining
an incompatible new standard for the location of the metadata
file:
"""
There are three standard locations for these metadata files:
* the PKG-INFO file included in the base directory of Python source
distribution archives (as
created by the distutils sdist command)
* the {distribution}-{version}.dist-info/METADATA file in a wheel binary
distribution archive (as
described in PEP 425, or a later version of that specification)
* the {distribution}-{version}.dist-info/METADATA files in a local Python
installation database (as
described in PEP 376, or a later version of that specification)
"""
It's easy to upgrade distribute and distutils to write
metadata 1.2 format, simply by changing the version in the
PKG-INFO files.
These addition are necessary to fix the above and also include
the standard location of the metadata for pip and distutils installations:
* the EGG-INFO/PKG-INFO file in an egg binary distribution archive (as created
by the distribute
bdist_egg command)
* the {distribution}-{version}.egg/EGG-INFO/PKG-INFO file in an installed egg
distribution archive
* the {distribution}-{version}.egg-info/PKG-INFO file for packages installed
with "pip install" or
distribute's "python setup.py install"
* the {distribution}-{version}.egg-info file for packages installed with
distutils' "python setup.py
install"
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Feb 20 2013)
>>> Python Projects, Consulting and Support ... http://www.egenix.com/
>>> mx
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On 20.02.2013 03:37, Paul Moore wrote: > On 20 February 2013 00:54, Fred Drake wrote: >> I'd posit that anything successful will no longer need to be added to >> the standard library, to boot. Packaging hasn't done well there. > > distlib may be the exception, though. Packaging tools are somewhat > unique because of the chicken and egg issue involved in having a > packaging tool with external dependencies - who installs your > dependencies for you? So enabling technology (library code to perform > packaging-related tasks, particularly in support of standardised > formats) could be better available from the stdlib. > >> I'd rather see a successful packaging story develop than bundle it into the >> standard library. The later just isn't that interesting any more. > > Bundling too early is a bad idea though. distlib is developing fast > and to do so it needs (1) a development cycle independent of python's > and (2) compatibility and ease of use with earlier versions of Python > (the latter is also critical for adoption in place of custom code in > packaging tools). > > Aiming for an accelerated level of development targeting inclusion in > Python 3.4 is plausible, though. MAL pointed out that agreeing > standards but not offering tools to support them in the stdlib is > risky, as people have no incentive to adopt those standards. We've got > 6 months or more until 3.4 feature freeze, let's not make any decision > too soon, though. I'm fine with not adding distlib to Python 3.4. The point I wanted to make was that there has to be an reference implementation of the PEP that tool makers can use to avoid reinventing the wheel over and over again (each with its own set of problems). If distlib implements the PEP, then it just needs to be mentioned there as a suitable reference implementation. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 20 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
> On Wednesday, February 20, 2013 at 2:48 AM, Chris Jerdonek wrote: > > I meant that bringing distlib into http://hg.python.org/cpython/ would > give it more visibility to core devs and others that already keep an > eye on python-checkins (the mailing list). And I think seeing the > Sphinx-processed docs integrated and cross-referenced with > http://docs.python.org/dev/ will help people understand better what > has been done and how it fits in with the rest of CPython -- which I > think would be useful to the community. It may also encourage > involvement (e.g. by being part of the main tracker). On Tue, Feb 19, 2013 at 11:53 PM, Donald Stufft wrote: > On the other hand it makes contributing to it more annoying since it > does not have pull requests, unless it was just a mirror. Maybe just the finished/production-ready pieces could be added as they are ready, with the main development happening outside. My understanding of distlib is that it's a collection of independent, bite-sized pieces of functionality, which could lend itself well to such a process. --Chris ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On 20 February 2013 04:07, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 02/19/2013 09:37 PM, Paul Moore wrote: >> On 20 February 2013 00:54, Fred Drake wrote: >>> I'd posit that anything successful will no longer need to be added >>> to the standard library, to boot. Packaging hasn't done well >>> there. >> >> distlib may be the exception, though. Packaging tools are somewhat >> unique because of the chicken and egg issue involved in having a >> packaging tool with external dependencies - who installs your >> dependencies for you? So enabling technology (library code to perform >> packaging-related tasks, particularly in support of standardised >> formats) could be better available from the stdlib. > > The big blocker there is that users can't rely on having it in the stdlib > until after they drop Python < 3.4 (assuming accelearated absorption) or > even 3.5. Understood - that's why I suggested that distlib reach a point where it's stable as an external package and supported on (some) older versions. I'm hoping for an experience more like unittest2 than packaging/distutils2. But we shouldn't be blocking new stdlib features just because they won't be available in older Python versions... Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Wednesday, February 20, 2013 at 2:48 AM, Chris Jerdonek wrote: > On Tue, Feb 19, 2013 at 3:16 PM, Daniel Holth (mailto:dho...@gmail.com)> wrote: > > Sorry, Chris must have meant http://hg.python.org/distlib/ . I was > > struggling to imagine a world where that is more visible than something on > > bitbucket. > > > > > I meant that bringing distlib into http://hg.python.org/cpython/ would > give it more visibility to core devs and others that already keep an > eye on python-checkins (the mailing list). And I think seeing the > Sphinx-processed docs integrated and cross-referenced with > http://docs.python.org/dev/ will help people understand better what > has been done and how it fits in with the rest of CPython -- which I > think would be useful to the community. It may also encourage > involvement (e.g. by being part of the main tracker). > > On the other hand it makes contributing to it more annoying since it does not have pull requests, unless it was just a mirror. > In asking about the "plan" for doing this, I was thinking of the > following remark by Nick: > > On Tue, Feb 19, 2013 at 5:40 AM, Nick Coghlan (mailto:ncogh...@gmail.com)> wrote: > > On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg > (mailto:m...@egenix.com)> wrote: > > > > > > Hmm, what is distlib and where does it live ? > > > > As part of the post-mortem of packaging's removal from Python 3.3, > > several subcomponents were identified as stable and useful. distlib is > > those subcomponents extracted into a separate repository by Vinay > > Sajip. > > > > It will be proposed as the standard library infrastructure for > > building packaging related tools, while distutils will become purely a > > build system and have nothing to do with installing software directly > > (except perhaps on developer machines). > > > > > My question was basically whether there was a tentative plan for when > it (or completed parts of it) will be proposed (e.g. when a certain > amount of functionality is completed, etc). It's better not to do > this at the last minute if 3.4 is the plan (as I think was attempted > with packaging but for 3.3). > > On Tue, Feb 19, 2013 at 6:40 PM, Steven D'Aprano (mailto:st...@pearwood.info)> wrote: > > > > I keep hearing people say that the stdlib is not important, but I don't > > think > > that is true. There are lots of people who have problems with anything not > > in > > the standard library. > > > > - Beginners often have difficulty (due to inexperience, lack of confidence > > or > > knowledge) in *finding*, let alone installing and using, packages that > > aren't > > in the standard library. > > > > - To people in the Linux world, adding anything outside of your distro's > > packaging system is a nuisance. No matter how easy your packaging library > > makes it, you now have two sorts of packages: first-class packages that > > your distro will automatically update for you, and second-class ones that > > aren't. > > > > - People working in restrictive corporate systems often have to jump through > > flaming hoops before installing software. > > > > > I would also add that for people new to writing Python modules and > that want to distribute them, it's hard to evaluate what they are > "supposed" to use (distutils, setuptools, distribute, bento, etc). > Just a day or two ago, this exact question was asked on the Distutils > mailing list with subject "Confusion of a hobby programmer." Code not > being in the standard library creates an extra mental hurdle to > overcome. > > I agree that eventually the stdlib needs standard tooling to work with the future (™) but until that future is in use adding it to the stdlib feels premature to me. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Tue, Feb 19, 2013 at 3:16 PM, Daniel Holth wrote: > Sorry, Chris must have meant http://hg.python.org/distlib/ . I was > struggling to imagine a world where that is more visible than something on > bitbucket. I meant that bringing distlib into http://hg.python.org/cpython/ would give it more visibility to core devs and others that already keep an eye on python-checkins (the mailing list). And I think seeing the Sphinx-processed docs integrated and cross-referenced with http://docs.python.org/dev/ will help people understand better what has been done and how it fits in with the rest of CPython -- which I think would be useful to the community. It may also encourage involvement (e.g. by being part of the main tracker). In asking about the "plan" for doing this, I was thinking of the following remark by Nick: On Tue, Feb 19, 2013 at 5:40 AM, Nick Coghlan wrote: > On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg wrote: >> >> Hmm, what is distlib and where does it live ? > > As part of the post-mortem of packaging's removal from Python 3.3, > several subcomponents were identified as stable and useful. distlib is > those subcomponents extracted into a separate repository by Vinay > Sajip. > > It will be proposed as the standard library infrastructure for > building packaging related tools, while distutils will become purely a > build system and have nothing to do with installing software directly > (except perhaps on developer machines). My question was basically whether there was a tentative plan for when it (or completed parts of it) will be proposed (e.g. when a certain amount of functionality is completed, etc). It's better not to do this at the last minute if 3.4 is the plan (as I think was attempted with packaging but for 3.3). On Tue, Feb 19, 2013 at 6:40 PM, Steven D'Aprano wrote: > > I keep hearing people say that the stdlib is not important, but I don't > think > that is true. There are lots of people who have problems with anything not > in > the standard library. > > - Beginners often have difficulty (due to inexperience, lack of confidence > or > knowledge) in *finding*, let alone installing and using, packages that > aren't > in the standard library. > > - To people in the Linux world, adding anything outside of your distro's > packaging system is a nuisance. No matter how easy your packaging library > makes it, you now have two sorts of packages: first-class packages that > your distro will automatically update for you, and second-class ones that > aren't. > > - People working in restrictive corporate systems often have to jump through > flaming hoops before installing software. I would also add that for people new to writing Python modules and that want to distribute them, it's hard to evaluate what they are "supposed" to use (distutils, setuptools, distribute, bento, etc). Just a day or two ago, this exact question was asked on the Distutils mailing list with subject "Confusion of a hobby programmer." Code not being in the standard library creates an extra mental hurdle to overcome. --Chris ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/19/2013 09:37 PM, Paul Moore wrote: > On 20 February 2013 00:54, Fred Drake wrote: >> I'd posit that anything successful will no longer need to be added >> to the standard library, to boot. Packaging hasn't done well >> there. > > distlib may be the exception, though. Packaging tools are somewhat > unique because of the chicken and egg issue involved in having a > packaging tool with external dependencies - who installs your > dependencies for you? So enabling technology (library code to perform > packaging-related tasks, particularly in support of standardised > formats) could be better available from the stdlib. The big blocker there is that users can't rely on having it in the stdlib until after they drop Python < 3.4 (assuming accelearated absorption) or even 3.5. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlEkTAsACgkQ+gerLs4ltQ6VBgCePncI4cX7a8NEN6lj98CVBdAE UTUAnA+6zo8Zjmp6T4n0uL804PnHHvZ8 =OT8w -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On 20/02/13 11:54, Fred Drake wrote: On Tue, Feb 19, 2013 at 6:19 PM, Donald Stufft wrote: Let's not add anything to the stdlib till it has real world usage. Doing otherwise is putting the cart before the horse. I'd posit that anything successful will no longer need to be added to the standard library, to boot. Packaging hasn't done well there. I'd rather see a successful packaging story develop than bundle it into the standard library. The later just isn't that interesting any more. I keep hearing people say that the stdlib is not important, but I don't think that is true. There are lots of people who have problems with anything not in the standard library. - Beginners often have difficulty (due to inexperience, lack of confidence or knowledge) in *finding*, let alone installing and using, packages that aren't in the standard library. - To people in the Linux world, adding anything outside of your distro's packaging system is a nuisance. No matter how easy your packaging library makes it, you now have two sorts of packages: first-class packages that your distro will automatically update for you, and second-class ones that aren't. - People working in restrictive corporate systems often have to jump through flaming hoops before installing software. Packages in the stdlib are a no-brainer. Anything outside the stdlib has additional barriers to use, even if installing them is as simple as "some-package-manager install spam.py". For the avoidance of doubt, this is *not* a veiled request for "everything" to be in the stdlib, since that is impractical and stupid, just a reminder that the stdlib is still important and that no matter how easy packaging becomes, it will never be as easy as having something already there. -- Steven ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On 20 February 2013 00:54, Fred Drake wrote: > I'd posit that anything successful will no longer need to be added to > the standard library, to boot. Packaging hasn't done well there. distlib may be the exception, though. Packaging tools are somewhat unique because of the chicken and egg issue involved in having a packaging tool with external dependencies - who installs your dependencies for you? So enabling technology (library code to perform packaging-related tasks, particularly in support of standardised formats) could be better available from the stdlib. > I'd rather see a successful packaging story develop than bundle it into the > standard library. The later just isn't that interesting any more. Bundling too early is a bad idea though. distlib is developing fast and to do so it needs (1) a development cycle independent of python's and (2) compatibility and ease of use with earlier versions of Python (the latter is also critical for adoption in place of custom code in packaging tools). Aiming for an accelerated level of development targeting inclusion in Python 3.4 is plausible, though. MAL pointed out that agreeing standards but not offering tools to support them in the stdlib is risky, as people have no incentive to adopt those standards. We've got 6 months or more until 3.4 feature freeze, let's not make any decision too soon, though. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Tue, Feb 19, 2013 at 6:19 PM, Donald Stufft wrote: > Let's not add anything to the stdlib till it has real world usage. Doing > otherwise is putting the cart before the horse. I'd posit that anything successful will no longer need to be added to the standard library, to boot. Packaging hasn't done well there. I'd rather see a successful packaging story develop than bundle it into the standard library. The later just isn't that interesting any more. -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Tuesday, February 19, 2013 at 6:16 PM, Daniel Holth wrote: > Sorry, Chris must have meant http://hg.python.org/distlib/ . I was struggling > to imagine a world where that is more visible than something on bitbucket. > Half the comments have been about putting something in stdlib right away, > something that I do not understand at all as a voracious PyPI or > https://crate.io/ user. Let's not add anything to the stdlib till it has real world usage. Doing otherwise is putting the cart before the horse. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Tue, Feb 19, 2013 at 5:10 PM, M.-A. Lemburg wrote: > On 19.02.2013 23:01, Daniel Holth wrote: > > On Tue, Feb 19, 2013 at 4:34 PM, M.-A. Lemburg wrote: > > > >> On 19.02.2013 14:40, Nick Coghlan wrote: > >>> On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg > wrote: > * PEP 426 doesn't include any mention of the egg distribution format, > even though it's the most popular distribution format at the moment. > It should at least include the location of the metadata file > in eggs (EGG-INFO/PKG-INFO) and egg installations > (/EGG-INFO/PKG-INFO). > >>> > >>> "Other tools involved in Python distribution may also use this format." > >>> > >>> The egg format has never been, and never will be, officially endorsed > >>> by python-dev. The wheel format is the standard format for binary > >>> distribution, and PEP 376 defines the standard location for metadata > >>> on installed distributions. > >> > >> Oh, come on, Nick, that's just silly. setuptools was included in stdlib > >> for a short while, so the above is simply wrong. Eggs are the most > >> widely used binary distribution format for Python package on PyPI: > >> > >> # wc *files.csv > >> 25585 25598 1431013 2013-02-19-egg-files.csv > >>46194640 236694 2013-02-19-exe-files.csv > >> 254 255 13402 2013-02-19-msi-files.csv > >> 104691 104853 5251962 2013-02-19-tar-gz-files.csv > >> 24 241221 2013-02-19-whl-files.csv > >> 17937 18022 905913 2013-02-19-zip-files.csv > >> 153110 153392 7840205 total > >> > >> (based on todays PyPI stats) > >> > >> It doesn't really help ignoring realities... and I'm saying > >> that as one of the core devs who got setuptools kicked out of > >> the stdlib again. > >> > >> -- > >> Marc-Andre Lemburg > >> eGenix.com > >> > > > > The wheel philosophy is that it should be supported by both python-dev > and > > setuptools and that you should feel happy about using setuptools if you > > like it whether or not python-dev (currently) endorses that. If you are > > using setuptools (distribute's pkg_resources) then you can use both at > the > > same time. > > > > Distribute, distutils and setuptools' problems have not been well > > understood which I think is why there has been a need to discredit > > setuptools by calling it non-standard. It is the defacto standard. If > your > > packages have dependencies there is no other choice. Wheel tries to solve > > the real problem by allowing you to build a package with setuptools while > > giving the end-user the choice of installing setuptools or not. > > > > Of course eggs are the most popular right now. The wheel format is very > > egg-like while avoiding some of egg's problems. See the comparison in the > > PEP or read the story on wheel's rtfd. The wheel project includes tools > to > > losslessly convert eggs or bdist_wininst to wheel. > > That's all fine, but it doesn't explain the refusal to add the > documentation of the location of the PKG-INFO file in eggs ? It would just be a sentence, I wouldn't have a problem with it but I also don't see why it would be necessary. Even setuptools doesn't touch the file usually. Right now distribute's pkg_resources currently only understands Requires-Dist if it is inside a .dist-info directory. Sorry, Chris must have meant http://hg.python.org/distlib/ . I was struggling to imagine a world where that is more visible than something on bitbucket. Half the comments have been about putting something in stdlib right away, something that I do not understand at all as a voracious PyPI or https://crate.io/ user. You should secure both the metadata and the packages but it doesn't have anything to do with the serialization format. Wheel is designed so that the metadata is at the end of the .zip and can be retrieved with an HTTP partial request. The feature may or may not be useful. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On 19.02.2013 23:01, Daniel Holth wrote: > On Tue, Feb 19, 2013 at 4:34 PM, M.-A. Lemburg wrote: > >> On 19.02.2013 14:40, Nick Coghlan wrote: >>> On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg wrote: * PEP 426 doesn't include any mention of the egg distribution format, even though it's the most popular distribution format at the moment. It should at least include the location of the metadata file in eggs (EGG-INFO/PKG-INFO) and egg installations (/EGG-INFO/PKG-INFO). >>> >>> "Other tools involved in Python distribution may also use this format." >>> >>> The egg format has never been, and never will be, officially endorsed >>> by python-dev. The wheel format is the standard format for binary >>> distribution, and PEP 376 defines the standard location for metadata >>> on installed distributions. >> >> Oh, come on, Nick, that's just silly. setuptools was included in stdlib >> for a short while, so the above is simply wrong. Eggs are the most >> widely used binary distribution format for Python package on PyPI: >> >> # wc *files.csv >> 25585 25598 1431013 2013-02-19-egg-files.csv >>46194640 236694 2013-02-19-exe-files.csv >> 254 255 13402 2013-02-19-msi-files.csv >> 104691 104853 5251962 2013-02-19-tar-gz-files.csv >> 24 241221 2013-02-19-whl-files.csv >> 17937 18022 905913 2013-02-19-zip-files.csv >> 153110 153392 7840205 total >> >> (based on todays PyPI stats) >> >> It doesn't really help ignoring realities... and I'm saying >> that as one of the core devs who got setuptools kicked out of >> the stdlib again. >> >> -- >> Marc-Andre Lemburg >> eGenix.com >> > > The wheel philosophy is that it should be supported by both python-dev and > setuptools and that you should feel happy about using setuptools if you > like it whether or not python-dev (currently) endorses that. If you are > using setuptools (distribute's pkg_resources) then you can use both at the > same time. > > Distribute, distutils and setuptools' problems have not been well > understood which I think is why there has been a need to discredit > setuptools by calling it non-standard. It is the defacto standard. If your > packages have dependencies there is no other choice. Wheel tries to solve > the real problem by allowing you to build a package with setuptools while > giving the end-user the choice of installing setuptools or not. > > Of course eggs are the most popular right now. The wheel format is very > egg-like while avoiding some of egg's problems. See the comparison in the > PEP or read the story on wheel's rtfd. The wheel project includes tools to > losslessly convert eggs or bdist_wininst to wheel. That's all fine, but it doesn't explain the refusal to add the documentation of the location of the PKG-INFO file in eggs ? > I am confident distlib can thrive outside of the standard library! Why the > rush to kill it before its prime? Who's trying to kill distlib ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 19 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On 19.02.2013 14:40, Nick Coghlan wrote: > On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg wrote: >> On 19.02.2013 11:28, Nick Coghlan wrote: >>> On Tue, Feb 19, 2013 at 7:37 PM, M.-A. Lemburg wrote: On 17.02.2013 11:11, Nick Coghlan wrote: I'm not against modernizing the format, but given that version 1.2 has been out for around 8 years now, without much following, I think we need to make the implementation bit a requirement before accepting the PEP. >>> >>> It is being implemented in distlib, and the (short!) appendix to the >>> PEP itself shows how to read the format using the standard library's >>> email module. >> >> Hmm, what is distlib and where does it live ? > > As part of the post-mortem of packaging's removal from Python 3.3, > several subcomponents were identified as stable and useful. distlib is > those subcomponents extracted into a separate repository by Vinay > Sajip. > > It will be proposed as the standard library infrastructure for > building packaging related tools, while distutils will become purely a > build system and have nothing to do with installing software directly > (except perhaps on developer machines). Shouldn't those details be mentioned in the PEP ? >> The PEP only shows how to parse the RFC822-style format used by the >> metadata. That's not what I was referring to. >> >> If a tools wants to support metadata 2.0, it has to support all >> the complicated stuff as well, i.e. handle the requires fields, >> the environment markers and version comparisons/sorting. > > Which is what distutils2 can be used for now, and what distlib will > provide without the unwanted build system infrastructure in > distutils2. > >>> v2.0 is designed to fix many of the issues that prevented the adoption >>> of v1.2, including tweaks to the standardised version scheme and the >>> addition of a formal extension mechanism to avoid the ad hoc >>> extensions that occurred with earlier metadata versions. >> >> Some more notes: >> >> * I find it confusing that we now have two version schemes, >> one defined in PEP 426 (hidden in the middle of the document) >> and one in PEP 386. It would be better to amend or replace >> PEP 386, since that's where you look for Python version strings. > > You can't understand version specifiers without understanding the sort > order defined for the version scheme, so documenting them separately > is just a recipe for confusion. PEP 386 defines both. The point here is that the version scheme goes far beyond the metadata format and is complicated enough to warrant a separate PEP. > I plan to mark PEP 386 as Withdrawn when I accept this PEP, as the > sorting scheme it defines is broken, and the distutils changes > proposed in that PEP are never going to happen. Hmm, Tarek is the author, so only he can withdraw the PEP, AFAIK. >> * PEP 426 doesn't include any mention of the egg distribution format, >> even though it's the most popular distribution format at the moment. >> It should at least include the location of the metadata file >> in eggs (EGG-INFO/PKG-INFO) and egg installations >> (/EGG-INFO/PKG-INFO). > > "Other tools involved in Python distribution may also use this format." > > The egg format has never been, and never will be, officially endorsed > by python-dev. The wheel format is the standard format for binary > distribution, and PEP 376 defines the standard location for metadata > on installed distributions. See my other reply. >> >> Not sure whether related or not, I also think it would be a good idea >> to make the metadata file available on PyPI for download (could be sent >> there when registering the package release). The register command >> only posts the data as 1.0 metadata, but includes fields from >> metadata 1.1. PyPI itself only displays part of the data. > > It's not related, but I plan to propose the adoption of TUF (with GPG > based signatures) for PyPI's end-to-end security solution, and the > conversion of the metadata files to JSON for distribution through > TUF's metadata support. (Donald Stufft already wrote PEP 426 <-> JSON > bidirectional converter as part of an unrelated experiment) Why convert the metadata format you are defining in PEP 426 to yet another format when it can be uploaded as file straight to PyPI ? TUF doesn't have anything to do with that, agreed ;-) >> It would be useful to have the metadata readily available for >> inspection on PyPI without having to download one of the >> distribution files first. > > Indeed, which is a large part of why TUF (aka The Update Framework: > https://www.updateframework.com/) is such an interesting security > solution. The suggestion to have the metadata available on PyPI doesn't have anything to do with security. It's about being able to determine compatibility and select the right distribution file for download. The metadata also helps in creating dependency graphs, which are useful for a lot of things. -- Marc-Andre Lemburg e
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Tue, Feb 19, 2013 at 4:34 PM, M.-A. Lemburg wrote: > On 19.02.2013 14:40, Nick Coghlan wrote: > > On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg wrote: > >> * PEP 426 doesn't include any mention of the egg distribution format, > >> even though it's the most popular distribution format at the moment. > >> It should at least include the location of the metadata file > >> in eggs (EGG-INFO/PKG-INFO) and egg installations > >> (/EGG-INFO/PKG-INFO). > > > > "Other tools involved in Python distribution may also use this format." > > > > The egg format has never been, and never will be, officially endorsed > > by python-dev. The wheel format is the standard format for binary > > distribution, and PEP 376 defines the standard location for metadata > > on installed distributions. > > Oh, come on, Nick, that's just silly. setuptools was included in stdlib > for a short while, so the above is simply wrong. Eggs are the most > widely used binary distribution format for Python package on PyPI: > > # wc *files.csv > 25585 25598 1431013 2013-02-19-egg-files.csv >46194640 236694 2013-02-19-exe-files.csv > 254 255 13402 2013-02-19-msi-files.csv > 104691 104853 5251962 2013-02-19-tar-gz-files.csv > 24 241221 2013-02-19-whl-files.csv > 17937 18022 905913 2013-02-19-zip-files.csv > 153110 153392 7840205 total > > (based on todays PyPI stats) > > It doesn't really help ignoring realities... and I'm saying > that as one of the core devs who got setuptools kicked out of > the stdlib again. > > -- > Marc-Andre Lemburg > eGenix.com > The wheel philosophy is that it should be supported by both python-dev and setuptools and that you should feel happy about using setuptools if you like it whether or not python-dev (currently) endorses that. If you are using setuptools (distribute's pkg_resources) then you can use both at the same time. Distribute, distutils and setuptools' problems have not been well understood which I think is why there has been a need to discredit setuptools by calling it non-standard. It is the defacto standard. If your packages have dependencies there is no other choice. Wheel tries to solve the real problem by allowing you to build a package with setuptools while giving the end-user the choice of installing setuptools or not. Of course eggs are the most popular right now. The wheel format is very egg-like while avoiding some of egg's problems. See the comparison in the PEP or read the story on wheel's rtfd. The wheel project includes tools to losslessly convert eggs or bdist_wininst to wheel. I am confident distlib can thrive outside of the standard library! Why the rush to kill it before its prime? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On 19.02.2013 14:40, Nick Coghlan wrote: > On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg wrote: >> * PEP 426 doesn't include any mention of the egg distribution format, >> even though it's the most popular distribution format at the moment. >> It should at least include the location of the metadata file >> in eggs (EGG-INFO/PKG-INFO) and egg installations >> (/EGG-INFO/PKG-INFO). > > "Other tools involved in Python distribution may also use this format." > > The egg format has never been, and never will be, officially endorsed > by python-dev. The wheel format is the standard format for binary > distribution, and PEP 376 defines the standard location for metadata > on installed distributions. Oh, come on, Nick, that's just silly. setuptools was included in stdlib for a short while, so the above is simply wrong. Eggs are the most widely used binary distribution format for Python package on PyPI: # wc *files.csv 25585 25598 1431013 2013-02-19-egg-files.csv 46194640 236694 2013-02-19-exe-files.csv 254 255 13402 2013-02-19-msi-files.csv 104691 104853 5251962 2013-02-19-tar-gz-files.csv 24 241221 2013-02-19-whl-files.csv 17937 18022 905913 2013-02-19-zip-files.csv 153110 153392 7840205 total (based on todays PyPI stats) It doesn't really help ignoring realities... and I'm saying that as one of the core devs who got setuptools kicked out of the stdlib again. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 19 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Tue, Feb 19, 2013 at 2:28 AM, Nick Coghlan wrote: > On Tue, Feb 19, 2013 at 7:37 PM, M.-A. Lemburg wrote: >> On 17.02.2013 11:11, Nick Coghlan wrote: >> I'm not against modernizing the format, but given that version 1.2 >> has been out for around 8 years now, without much following, >> I think we need to make the implementation bit a requirement >> before accepting the PEP. > > It is being implemented in distlib, and the (short!) appendix to the > PEP itself shows how to read the format using the standard library's > email module. Maybe this is already stated somewhere, but is there a plan for when distlib will be brought into the repository? Is there a reason not to do it now? It seems it would have more visibility that way (e.g. people could see it as part of the development version of the online docs and it would be in check-ins as are PEP edits), and its status relative to Python would be clearer. --Chris > > v2.0 is designed to fix many of the issues that prevented the adoption > of v1.2, including tweaks to the standardised version scheme and the > addition of a formal extension mechanism to avoid the ad hoc > extensions that occurred with earlier metadata versions. > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Tue, Feb 19, 2013 at 3:25 PM, Paul Moore wrote: > On 19 February 2013 13:40, Nick Coghlan wrote: > >> If a tools wants to support metadata 2.0, it has to support all > >> the complicated stuff as well, i.e. handle the requires fields, > >> the environment markers and version comparisons/sorting. > > > > Which is what distutils2 can be used for now, and what distlib will > > provide without the unwanted build system infrastructure in > > distutils2. > > One particular item in Metadata 2.0 which needs to be addressed is > requirements specifiers, as used in (for example) Requires-Dist. These > take forms like "DistName (>= ver)". There are a number of potential > issues here: > > 1. This differs from the pkg_resources format (which omits the > parentheses). Having 2 subtly different formats is not a good idea in > the long term. At the moment, pkg_resources format is used in pip > requirements (on the command line and in requirement files) as well as > in setuptools/distribute and possibly elsewhere. > 2. There is currently no code that I am aware of that implements this > spec. I don't believe distlib does (yet - give Vinay 5 minutes and who > knows? :-)), pkg_resources as I said implements a different format, > and distutils2, apart from being a big dependency to cover just this > point, only matches the version (not the project name) and presumably > does so using the distutils2 version ordering (which is incompatible > with Metadata 2.0). > 3. The format is fiddly to parse and process - nobody is likely to > implement it for themselves without a library that does it (especially > not when using pkg_resources is so easy). > > The PEP needs a rationale as to why the pkg_resources format wasn't > used, and should suggest a migration path for projects which currently > use the pkg_resources style (such as pip's requirements files). > > Paul We did add full support for Metadata 2.0 Requires-Dist in pkg_resources. The fiddling happens here: https://bitbucket.org/tarek/distribute/src/tip/pkg_resources.py?at=default#cl-2469 The code removes the parenthesis and adds == when no operator was specified to convert us back to the pkg_resources format. The rest of pkg_resources carries on as before. Wheel installs would not pull in dependencies otherwise. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On 19 February 2013 20:36, Donald Stufft wrote: > 2. There is currently no code that I am aware of that implements this > spec. I don't believe distlib does (yet - give Vinay 5 minutes and who > knows? :-)), pkg_resources as I said implements a different format, > and distutils2, apart from being a big dependency to cover just this > point, only matches the version (not the project name) and presumably > does so using the distutils2 version ordering (which is incompatible > with Metadata 2.0). > > It should be alongside the version stuff, atleast it was in distutils2. Yes, sorry - distlib does include code for this: https://distlib.readthedocs.org/en/latest/tutorial.html#matching-versions-against-constraints I was looking for the term "requirement" which is not used, and missed it. (Either that or Vinay's been borrowing the time machine keys again!) Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Tuesday, February 19, 2013 at 3:25 PM, Paul Moore wrote: > On 19 February 2013 13:40, Nick Coghlan (mailto:ncogh...@gmail.com)> wrote: > > > If a tools wants to support metadata 2.0, it has to support all > > > the complicated stuff as well, i.e. handle the requires fields, > > > the environment markers and version comparisons/sorting. > > > > > > > > > Which is what distutils2 can be used for now, and what distlib will > > provide without the unwanted build system infrastructure in > > distutils2. > > > > > One particular item in Metadata 2.0 which needs to be addressed is > requirements specifiers, as used in (for example) Requires-Dist. These > take forms like "DistName (>= ver)". There are a number of potential > issues here: > > 1. This differs from the pkg_resources format (which omits the > parentheses). Having 2 subtly different formats is not a good idea in > the long term. At the moment, pkg_resources format is used in pip > requirements (on the command line and in requirement files) as well as > in setuptools/distribute and possibly elsewhere. > 2. There is currently no code that I am aware of that implements this > spec. I don't believe distlib does (yet - give Vinay 5 minutes and who > knows? :-)), pkg_resources as I said implements a different format, > and distutils2, apart from being a big dependency to cover just this > point, only matches the version (not the project name) and presumably > does so using the distutils2 version ordering (which is incompatible > with Metadata 2.0). > > It should be alongside the version stuff, atleast it was in distutils2. > 3. The format is fiddly to parse and process - nobody is likely to > implement it for themselves without a library that does it (especially > not when using pkg_resources is so easy). > > The PEP needs a rationale as to why the pkg_resources format wasn't > used, and should suggest a migration path for projects which currently > use the pkg_resources style (such as pip's requirements files). > > Paul > ___ > Python-Dev mailing list > Python-Dev@python.org (mailto:Python-Dev@python.org) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On 19 February 2013 13:40, Nick Coghlan wrote: >> If a tools wants to support metadata 2.0, it has to support all >> the complicated stuff as well, i.e. handle the requires fields, >> the environment markers and version comparisons/sorting. > > Which is what distutils2 can be used for now, and what distlib will > provide without the unwanted build system infrastructure in > distutils2. One particular item in Metadata 2.0 which needs to be addressed is requirements specifiers, as used in (for example) Requires-Dist. These take forms like "DistName (>= ver)". There are a number of potential issues here: 1. This differs from the pkg_resources format (which omits the parentheses). Having 2 subtly different formats is not a good idea in the long term. At the moment, pkg_resources format is used in pip requirements (on the command line and in requirement files) as well as in setuptools/distribute and possibly elsewhere. 2. There is currently no code that I am aware of that implements this spec. I don't believe distlib does (yet - give Vinay 5 minutes and who knows? :-)), pkg_resources as I said implements a different format, and distutils2, apart from being a big dependency to cover just this point, only matches the version (not the project name) and presumably does so using the distutils2 version ordering (which is incompatible with Metadata 2.0). 3. The format is fiddly to parse and process - nobody is likely to implement it for themselves without a library that does it (especially not when using pkg_resources is so easy). The PEP needs a rationale as to why the pkg_resources format wasn't used, and should suggest a migration path for projects which currently use the pkg_resources style (such as pip's requirements files). Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg wrote: > On 19.02.2013 11:28, Nick Coghlan wrote: >> On Tue, Feb 19, 2013 at 7:37 PM, M.-A. Lemburg wrote: >>> On 17.02.2013 11:11, Nick Coghlan wrote: >>> I'm not against modernizing the format, but given that version 1.2 >>> has been out for around 8 years now, without much following, >>> I think we need to make the implementation bit a requirement >>> before accepting the PEP. >> >> It is being implemented in distlib, and the (short!) appendix to the >> PEP itself shows how to read the format using the standard library's >> email module. > > Hmm, what is distlib and where does it live ? As part of the post-mortem of packaging's removal from Python 3.3, several subcomponents were identified as stable and useful. distlib is those subcomponents extracted into a separate repository by Vinay Sajip. It will be proposed as the standard library infrastructure for building packaging related tools, while distutils will become purely a build system and have nothing to do with installing software directly (except perhaps on developer machines). > The PEP only shows how to parse the RFC822-style format used by the > metadata. That's not what I was referring to. > > If a tools wants to support metadata 2.0, it has to support all > the complicated stuff as well, i.e. handle the requires fields, > the environment markers and version comparisons/sorting. Which is what distutils2 can be used for now, and what distlib will provide without the unwanted build system infrastructure in distutils2. >> v2.0 is designed to fix many of the issues that prevented the adoption >> of v1.2, including tweaks to the standardised version scheme and the >> addition of a formal extension mechanism to avoid the ad hoc >> extensions that occurred with earlier metadata versions. > > Some more notes: > > * I find it confusing that we now have two version schemes, > one defined in PEP 426 (hidden in the middle of the document) > and one in PEP 386. It would be better to amend or replace > PEP 386, since that's where you look for Python version strings. You can't understand version specifiers without understanding the sort order defined for the version scheme, so documenting them separately is just a recipe for confusion. I plan to mark PEP 386 as Withdrawn when I accept this PEP, as the sorting scheme it defines is broken, and the distutils changes proposed in that PEP are never going to happen. > * PEP 426 doesn't include any mention of the egg distribution format, > even though it's the most popular distribution format at the moment. > It should at least include the location of the metadata file > in eggs (EGG-INFO/PKG-INFO) and egg installations > (/EGG-INFO/PKG-INFO). "Other tools involved in Python distribution may also use this format." The egg format has never been, and never will be, officially endorsed by python-dev. The wheel format is the standard format for binary distribution, and PEP 376 defines the standard location for metadata on installed distributions. > > Not sure whether related or not, I also think it would be a good idea > to make the metadata file available on PyPI for download (could be sent > there when registering the package release). The register command > only posts the data as 1.0 metadata, but includes fields from > metadata 1.1. PyPI itself only displays part of the data. It's not related, but I plan to propose the adoption of TUF (with GPG based signatures) for PyPI's end-to-end security solution, and the conversion of the metadata files to JSON for distribution through TUF's metadata support. (Donald Stufft already wrote PEP 426 <-> JSON bidirectional converter as part of an unrelated experiment) > It would be useful to have the metadata readily available for > inspection on PyPI without having to download one of the > distribution files first. Indeed, which is a large part of why TUF (aka The Update Framework: https://www.updateframework.com/) is such an interesting security solution. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On 19.02.2013 11:28, Nick Coghlan wrote: > On Tue, Feb 19, 2013 at 7:37 PM, M.-A. Lemburg wrote: >> On 17.02.2013 11:11, Nick Coghlan wrote: >> I'm not against modernizing the format, but given that version 1.2 >> has been out for around 8 years now, without much following, >> I think we need to make the implementation bit a requirement >> before accepting the PEP. > > It is being implemented in distlib, and the (short!) appendix to the > PEP itself shows how to read the format using the standard library's > email module. Hmm, what is distlib and where does it live ? The PEP only shows how to parse the RFC822-style format used by the metadata. That's not what I was referring to. If a tools wants to support metadata 2.0, it has to support all the complicated stuff as well, i.e. handle the requires fields, the environment markers and version comparisons/sorting. > v2.0 is designed to fix many of the issues that prevented the adoption > of v1.2, including tweaks to the standardised version scheme and the > addition of a formal extension mechanism to avoid the ad hoc > extensions that occurred with earlier metadata versions. Some more notes: * I find it confusing that we now have two version schemes, one defined in PEP 426 (hidden in the middle of the document) and one in PEP 386. It would be better to amend or replace PEP 386, since that's where you look for Python version strings. * PEP 426 doesn't include any mention of the egg distribution format, even though it's the most popular distribution format at the moment. It should at least include the location of the metadata file in eggs (EGG-INFO/PKG-INFO) and egg installations (/EGG-INFO/PKG-INFO). Not sure whether related or not, I also think it would be a good idea to make the metadata file available on PyPI for download (could be sent there when registering the package release). The register command only posts the data as 1.0 metadata, but includes fields from metadata 1.1. PyPI itself only displays part of the data. It would be useful to have the metadata readily available for inspection on PyPI without having to download one of the distribution files first. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 19 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Tue, Feb 19, 2013 at 7:37 PM, M.-A. Lemburg wrote: > On 17.02.2013 11:11, Nick Coghlan wrote: > I'm not against modernizing the format, but given that version 1.2 > has been out for around 8 years now, without much following, > I think we need to make the implementation bit a requirement > before accepting the PEP. It is being implemented in distlib, and the (short!) appendix to the PEP itself shows how to read the format using the standard library's email module. v2.0 is designed to fix many of the issues that prevented the adoption of v1.2, including tweaks to the standardised version scheme and the addition of a formal extension mechanism to avoid the ad hoc extensions that occurred with earlier metadata versions. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On 17.02.2013 11:11, Nick Coghlan wrote: > FYI > > > -- Forwarded message -- > From: Nick Coghlan > Date: Sun, Feb 17, 2013 at 8:10 PM > Subject: PEP 426 is now the draft spec for distribution metadata 2.0 > To: "DistUtils mailing list\"\"" > > > The latest draft of PEP 426 is up at http://www.python.org/dev/peps/pep-0426/ > > Major changes since the last draft: > > 1. Metadata-Version is 2.0 rather than 1.3, and the field now has the > same major.minor semantics as are defined for wheel versions in PEP > 427 (i.e. if a tool sees a major version number it doesn't recognise, > it should give up rather than trying to guess what to do with it, > while it's OK to process a higher minor version) > > 2. The "Private-Version" field is added, and several examples are > given showing how to use it in conjunction with translated public > versions when a project wants to use a version numbering scheme that > the standard installation tools won't understand. > > 3. The environment markers section more clearly covers the need to > handle parentheses (this was mentioned in the text, but not the > pseudo-grammar), and the fields which support those markers have an > explicit cross-reference to that section of the spec. > > 4. Leading/trailing whitespace and date based versioning are > explicitly excluded from valid public versions > > 5. Version compatibility statistics are provided for this PEP relative > to PEP 386 (the analysis script has been added to the PEP repo if > anyone wants to check my numbers) > > 6. I have reclaimed BDFL-Delegate status (with Guido's approval) > > Since getting wheel support into pip no longer depends on this version > of the metadata spec, I plan to leave it open for comment for another > week, and then accept it next weekend if I don't hear any significant > objections. Overall, I think the meta data for Python packages is getting too complicated. Without a support module in the stdlib implementing the required parsing, evaluation and formatting mechanisms needed to analyze and write the format, I'm -1 on adding yet another format version on top of the stack. At the moment, discussing yet another version update is mostly academic, since not even version 1.2 has been picked up by the tools yet: distutils still writes version 1.1 meta data and doesn't even understand 1.2 meta data. The only tool in wide spread use that understands part of the 1.2 data is setuptools/distribute, but it can only understand the Requires-Dist field of that version of the spec (only because the 1.1 Requires field was deprecated) and interprets a Provides-Extra field which isn't even standard. All other 1.2 fields are ignored. setuptools/distribute still writes 1.1 meta-data. I've never seen environment markers being used or supported in the wild. I'm not against modernizing the format, but given that version 1.2 has been out for around 8 years now, without much following, I think we need to make the implementation bit a requirement before accepting the PEP. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 19 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
