Re: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440)
On 20 June 2013 23:07, Nick Coghlan ncogh...@gmail.com wrote: New versions of PEP 426 and PEP 440 are up on python.org: PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ PEP 440 (version spec): http://www.python.org/dev/peps/pep-0440/ (as before, not including them inline due to sheer length) The bulk of the changes are in this commit: http://hg.python.org/peps/rev/de305af601fa (there are a few minor tweaks in subsequent commits to the PEPs repo) PEP 426 has now been updated based on the feedback on this thread (and to handle some todo items that were pending anyway): http://hg.python.org/peps/rev/3f733fe7c06c * Adds implementation_name and implementation_version marker variables * Expands on the expected use cases for supports_environments * References wheel and warehouse from legacy metadata appendix * Drops most of the Sphinx notes (moving some to Rejected Features) I forgot to mention it in the commit message, but it also expands on some of the possible use cases for metadata extensions, such as a build tool storing the default build options for wheel creation. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440)
On 23 June 2013 08:05, Nick Coghlan ncogh...@gmail.com wrote: PEP 426 has now been updated based on the feedback on this thread (and to handle some todo items that were pending anyway): http://hg.python.org/peps/rev/3f733fe7c06c Sorry I didn't think of this sooner, but I don't see in the PEP a brief summary of precisely what metadata is required. From the Core Metadata section I infer that a project with *only* Name and Version (and Metadata-Version, which will be supplied by the build tool) is valid. That's fine, but notably more permissive than earlier versions of the spec. One thing I see is Index servers MAY require that this field be provided for some metadata (e.g., Summary) which is fine, I guess, but implies that somewhere, the rules for PyPI should be documented. Also, is it true that build tools MAY require certain fields to be provided? I'm pretty sure setup.py sdist does at present. My logic here is that previously I would go to the metadata PEP to check precisely what fields I need when building my project. Now, I'm not sure where I'd go. (Note that most of my projects are internal only, so it's actually the extra distutils/setuptools requirements that interest me more than those of PyPI...) I can imagine the answer would be the documentation of the individual tool - but in reality said documentation often isn't present (possibly because earlier metadata PEPs documented this information centrally) Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440)
On 23 Jun 2013 21:09, Paul Moore p.f.mo...@gmail.com wrote: On 23 June 2013 08:05, Nick Coghlan ncogh...@gmail.com wrote: PEP 426 has now been updated based on the feedback on this thread (and to handle some todo items that were pending anyway): http://hg.python.org/peps/rev/3f733fe7c06c Sorry I didn't think of this sooner, but I don't see in the PEP a brief summary of precisely what metadata is required. From the Core Metadata section I infer that a project with *only* Name and Version (and Metadata-Version, which will be supplied by the build tool) is valid. That's fine, but notably more permissive than earlier versions of the spec. That's deliberate - the mandatory fields are just those where dependency analysis can't work if they're missing. One thing I see is Index servers MAY require that this field be provided for some metadata (e.g., Summary) which is fine, I guess, but implies that somewhere, the rules for PyPI should be documented. I think I'll just move the summary back up into the core metadata. You can't make a sane index server without it, and it used to be mandatory, so maybe mandatory, maybe not just makes things more complicated than they need to be rather than simplifying anything. Also, is it true that build tools MAY require certain fields to be provided? I'm pretty sure setup.py sdist does at present. My logic here is that previously I would go to the metadata PEP to check precisely what fields I need when building my project. Now, I'm not sure where I'd go. (Note that most of my projects are internal only, so it's actually the extra distutils/setuptools requirements that interest me more than those of PyPI...) I can imagine the answer would be the documentation of the individual tool - but in reality said documentation often isn't present (possibly because earlier metadata PEPs documented this information centrally) I expect tools to come into line with PEP 426 on that front over time. Those that still generate legacy metadata will likely follow the legacy requirements, and for tools with stronger opinions about what should be provided, it will be up to the authors to document that properly. Contributions to the Python packaging user guide (a shared introduction to setuptools and pip) are also more than welcome (see http://python-packaging-user-guide.readthedocs.org). Marcus has made a good start based on the distribute guide (with less opinionated hyperbole), and I expect it to settle down to something quite usable once pip 1.4 is released and it can simply direct people towards setuptools 0.7+ as the default build tool (plus wheel for bdist_wheel support) and pip 1.4+ as the default installer (including support for installation from wheels, and automatic use of cached local builds for installation from source). We really don't want end users needing to read PEPs just to get started. Cheers, Nick. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440)
On 23 June 2013 13:53, Nick Coghlan ncogh...@gmail.com wrote: I think I'll just move the summary back up into the core metadata. You can't make a sane index server without it, and it used to be mandatory, so maybe mandatory, maybe not just makes things more complicated than they need to be rather than simplifying anything. Yes, it's the maybe mandatory, depending on your tools state that's annoying. If summary was the only one of them, then this change makes sense. Thanks. Older metadata versions made things like Download-URL mandatory, which was a pain for internal projects that didn't actually *have* a download URL, just a file on the software server or something (and no, file URLs don't count :-)) I guess they all become optional under Metadata 2.0 (as per the Changes from Previous Versions appendix). I expect tools to come into line with PEP 426 on that front over time. Those that still generate legacy metadata will likely follow the legacy requirements, and for tools with stronger opinions about what should be provided, it will be up to the authors to document that properly. OK, that's cool. As tools move towards *validating* to the 2.0 spec, the problem goes away. It's just that there's no way to make older tools validate to the new spec, so there's a transition here which we have to live with (as opposed to *generating* metadata 2.0, which can be handled by converters). Contributions to the Python packaging user guide (a shared introduction to setuptools and pip) are also more than welcome (see http://python-packaging-user-guide.readthedocs.org). Marcus has made a good start based on the distribute guide (with less opinionated hyperbole), and I expect it to settle down to something quite usable once pip 1.4 is released and it can simply direct people towards setuptools 0.7+ as the default build tool (plus wheel for bdist_wheel support) and pip 1.4+ as the default installer (including support for installation from wheels, and automatic use of cached local builds for installation from source). Yes, I should have a look at that. We really don't want end users needing to read PEPs just to get started. Agreed, but again there's not much else to work with at the moment :-) Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440)
Nick Coghlan ncoghlan at gmail.com writes: New versions of PEP 426 and PEP 440 are up on python.org: Thanks for the update; I'm in the process of updating distlib to provide support for the updated spec. The tip version of distlib provides good coverage for all except the very latest updates to the spec, and they should be included soon. It's a very minor point, but I noticed that all the keys of the main metadata dictionary and all sub-dictionaries are lower case with underscore, except for the project_urls dictionary which has keys like Home, Documentation etc. Is there any reason for this stylistic departure in just this part of the spec? Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440)
On Sun, Jun 23, 2013 at 1:52 PM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote:
Nick Coghlan ncoghlan at gmail.com writes:
* References wheel and warehouse from legacy metadata appendix
It may also be worth mentioning that:
* distlib supports converting legacy metadata to the new format; it's as
simple as:
from distlib.metadata import Metadata
metadata = Metadata(path='PKG-INFO')
metadata.write(path='pymeta.json')
* Metadata extracted from setup.py, which includes full dependency
information, is available through distlib's locate function. This
includes setuptools' newest format for extras, which has support for
for environment markers. For example:
from distlib.locators import locate
dist = locate('setuptools')
dist.metadata.write('pymeta.json')
Then pymeta.json contains (some fields omitted for brevity, but the
dependency stuff is there):
{
version: 0.7.4,
name: setuptools,
metadata_version: 2.0,
run_may_require: [
{
environment: sys_platform=='win32',
dependencies: [
wincertstore (== 0.1)
],
extra: ssl
},
{
environment: sys_platform=='win32' and python_version=='2.4',
dependencies: [
ctypes (== 1.0.2)
],
extra: ssl
},
{
dependencies: [
certifi (== 0.0.8)
],
extra: certs
},
{
environment: python_version in '2.4, 2.5',
dependencies: [
ssl (== 1.16)
],
extra: ssl
}
],
extras: [
certs,
ssl
],
}
I would welcome feedback from anyone who cares to clone/sync the distlib
repository. A release of distlib with this functionality will occur once
incorporation of the latest spec changes occurs, and I'm comfortable that
distlib is reasonably stable following these changes. Additional testing by
others can speed up getting to that point :-)
Regards,
Vinay Sajip
Here is the setuptools extras+conditionals format. It just separates
them with a colon:
extras_require = {
ssl:sys_platform=='win32': wincertstore==0.1,
ssl:sys_platform=='win32' and python_version=='2.4': ctypes==1.0.2,
ssl:python_version in '2.4, 2.5':ssl==1.16,
certs: certifi==0.0.8,
},
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440)
Daniel Holth dholth at gmail.com writes: Here is the setuptools extras+conditionals format. It just separates them with a colon: Thanks, but I've already seen that. The example I gave showed how that was migrated to the PEP 426 format as pymeta.json :-) Did I miss anything? Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440)
No I just forgot the syntax so I looked it up. On Jun 23, 2013 7:58 PM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Daniel Holth dholth at gmail.com writes: Here is the setuptools extras+conditionals format. It just separates them with a colon: Thanks, but I've already seen that. The example I gave showed how that was migrated to the PEP 426 format as pymeta.json :-) Did I miss anything? Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440)
On 21 June 2013 00:24, Nick Coghlan ncogh...@gmail.com wrote: How close am I to reality here? Close, except that this is explicitly an ecosystem first plan, with the standard library playing catch up later. That's the big reason I sought (and received) agreement at this year's language summit for us to discuss and approve packaging related PEPs right here on distutils-sig rather than having to repost them to python-dev for ratification. The only must have on the list for 3.4 is the pip bootstrapping, and Richard and I hope to get that squared away at the PyCon AU sprints next month. Thanks. My remaining concern is the chicken and egg question of how this much richer metadata will get populated - things like dependency metadata is only as useful as the accuracy of the information. Some of it can be translated from existing legacy formats (notably setuptools) and hopefully once Appendix A is completed, we'll have a better understanding of how that will work, and how complete it will be. As regards the actual PEP contents, my thoughts are basically: - I'm +1 on the PEP as a whole. Most of it seems to be to be entirely reasonable and an important step forward - I don't have much personal use for the more complex items like semantic dependencies, extras, hooks, etc, but I see no particular problems with them - Thinking about what I *do* use, the most glaring omission is a specific supports Python 3 indicator. I know it's available in the classifiers, and it may have already been discussed, but is there realistically any benefit in formalising this? The environment markers section covers this somewhat, but is it within the remit of the PEP to recommend how (or even when) packages should declare what major versions of Python they support? (And should installers be warning about attempts to install packages that don't support the relevant Python major version?) Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440)
On 21 June 2013 12:35, Nick Coghlan ncogh...@gmail.com wrote: The quick compatibility check is actually part of the wheel file naming specification (it's covered by the compatibility tags defined in PEP 425). Yes, I was thinking more of the sdist side - and also of how a pure Python package can specify whether separate wheels are needed for Python 2 and Python 3, for example. There's a wheel-specific setup.cfg extension for this, but should it be in the metadata spec? For PEP 426, I originally retained the separate Requires-Python field, but eventually realised it was redundant given the supports_environments field (http://www.python.org/dev/peps/pep-0426/#supports-environments) and the python_version and python_full_version variables in environment markers. It gets a little clumsy if you want to express a Python version constraint *and* a platform constraint (due to the implicit or), but the current approach does have the virtue of handling cases where version requirements differ between platforms. For example, if you need atomic renaming capabilities, then they've been available on POSIX systems since pretty much forever, but on Windows you need Python 3.3 so you can have access to os.replace. In the current PEP, that might look like: supports_environments: [sys_platform != 'win32' and python_version = 2.6, sys_platform == 'win32' and python_version = 3.3] Yes, but that's very much the power user end of the spectrum, as you mentioned before. If your package was source compatible between 2.6, 2.7 and 3.3+ you might say: supports_environments: [python_version == 2.6, python_version == 2.7, python_version = 3.3] More often than not, my thinking is more at the level of did I take the time to make the code dual-version compatible - things like putting brackets round print arguments. Or simply will I accept bug reports for both Python 2 and Python 3, even though I only really test on Python3. The supports_environments tag seems too precise for that sort of very broad brush approach. However, looking at the PEP, I realise I didn't actually update the remaining text to reflect that realisation - I just deleted the old text that was no longer applicable. I'll add an example that shows using supports_environments to specify a minimum Python version in a way installation tools are expected to check. I'll also make it clearer that supports_environments describes what the *sdist* supports. Generating a wheel file may limit it further (as indicated by the compatibility tags). That sounds good. I'd suggest examples like twisted (doesn't support Python 3 yet), virtualenv (supports Python 2.6 or greater, including 3.x) and maybe something that is purely for Python 3. I'd also like to see a paragraph explaining that installers SHOULD at least respect python version specs (even if they don't handle the full environment marker spec). I'm not sure how practical it would be to require that sort of partial support, though, which is my main reservation about using the environment spec for this. (I think respecting language version specs is important, but the full environment spec is no more than a nice to have). Also, it seems to me that wheel could use the Python version spec to generate appropriate compatibility tags. Again, that's not possible for tags in their full generality, but is possible for Python version only tags. And again, I'd like to see something saying that wheel generators should do this (otherwise, there's a duplication somewhere as this information would need to be specified elsewhere in a generator-specific manner, like the setup.cfg extension I mentioned above). Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440)
On 21 Jun, 2013, at 14:21, Paul Moore p.f.mo...@gmail.com wrote: On 21 June 2013 12:35, Nick Coghlan ncogh...@gmail.com wrote: The quick compatibility check is actually part of the wheel file naming specification (it's covered by the compatibility tags defined in PEP 425). Yes, I was thinking more of the sdist side - and also of how a pure Python package can specify whether separate wheels are needed for Python 2 and Python 3, for example. There's a wheel-specific setup.cfg extension for this, but should it be in the metadata spec? No, but it should be in the spec/documentation for the tool that builds sdists or wheels. That is, when metadata 2.0 support is added to distutils/setuptools/... it should document how the metadata is configured. Ronald ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440)
On 21 Jun, 2013, at 1:24, Nick Coghlan ncogh...@gmail.com wrote: 1. Python setup.py sdist and python setup.py dist_info will be changed to generate pymeta.json files. But that will be for Python 3.4 only (there's a big problem if this doesn't make it into 3.4...). Unless there's a distutils backport or similar. The initial vehicles for migration will be Daniel's bdist_wheel command and upgrades to setuptools. Standard library support may not land for 3.4, but we will at least have the pip bootstrapping mechanism in place (which in turn will allow retrieval of setuptools). What would be needed to land support for 3.4? Is it just a matter of pushing distlib to the stdlib and updating distutils (add a way to configure 2.0 metadata, add bdist_wheel and update sdist), or are there other other issues? Well, other than finalizing the PEP and its implementation in distlib? Ronald ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440)
On 21 June 2013 14:30, Ronald Oussoren ronaldousso...@mac.com wrote: Yes, I was thinking more of the sdist side - and also of how a pure Python package can specify whether separate wheels are needed for Python 2 and Python 3, for example. There's a wheel-specific setup.cfg extension for this, but should it be in the metadata spec? No, but it should be in the spec/documentation for the tool that builds sdists or wheels. That is, when metadata 2.0 support is added to distutils/setuptools/... it should document how the metadata is configured. So a package author creates a package that only works with Python 3, builds a sdist using a tool-specific option that states this, and that data isn't recorded in the sdist so that whoever uses the sdist to build a wheel has to find that out elsewhere and re-specify it to build a wheel? That sounds wrong, unless I'm misunderstanding you... Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440)
On 22 June 2013 00:10, Paul Moore p.f.mo...@gmail.com wrote: On 21 June 2013 14:30, Ronald Oussoren ronaldousso...@mac.com wrote: Yes, I was thinking more of the sdist side - and also of how a pure Python package can specify whether separate wheels are needed for Python 2 and Python 3, for example. There's a wheel-specific setup.cfg extension for this, but should it be in the metadata spec? No, but it should be in the spec/documentation for the tool that builds sdists or wheels. That is, when metadata 2.0 support is added to distutils/setuptools/... it should document how the metadata is configured. So a package author creates a package that only works with Python 3, builds a sdist using a tool-specific option that states this, and that data isn't recorded in the sdist so that whoever uses the sdist to build a wheel has to find that out elsewhere and re-specify it to build a wheel? That sounds wrong, unless I'm misunderstanding you... This is a build problem. The metadata 2.0 spec *deliberately* avoids trying to solve build problems, it merely leaves the door open to attempting to solve them in the future. It *may* make sense for a build tool to check the supports_environment flags for implied options when building a wheel, but that's up to the build tool developer. Designing a metabuild system that is actually better than the setup.py based command line API is an incredibly hard problem, and I'm not yet convinced it's a problem we really need to solve. Killing setup.py install? Sure, that definitely needs to happen and decoupling the intergrator's choice of install tool from the developer's choice of build tool is the main aim of metadata 2.0. Killing setup.py dist_info/sdist/bdist_wheel/test, though? I'm *not* convinced that needs to happen, since there's no way to decouple of project's build system from itself and projects like d2to1 show that creating a setup.py based shim for an alternate build system doesn't need to be *that* painful, particularly once there are well documented standard archive formats to target. Even if a setup.py replacement on the build side does happen eventually, it's a long way down the priority list. In the meantime, once sdist 2.0 is officially defined in a PEP, build tools will be able to define their own metadata extensions to record whatever they want in the sdist files. They'll also be free to add arbitrary files to the dist-info directory and have them propagate through the wheel files and all the way to the installation database. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440)
On 21 June 2013 15:52, Nick Coghlan ncogh...@gmail.com wrote: In the meantime, once sdist 2.0 is officially defined in a PEP, build tools will be able to define their own metadata extensions to record whatever they want in the sdist files. They'll also be free to add arbitrary files to the dist-info directory and have them propagate through the wheel files and all the way to the installation database. Ah, OK. I see how that would work now. Thanks for the clarification. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440)
On Thu, Jun 20, 2013 at 9:07 AM, Nick Coghlan ncogh...@gmail.com wrote: New versions of PEP 426 and PEP 440 are up on python.org: PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ PEP 440 (version spec): http://www.python.org/dev/peps/pep-0440/ (as before, not including them inline due to sheer length) The bulk of the changes are in this commit: http://hg.python.org/peps/rev/de305af601fa (there are a few minor tweaks in subsequent commits to the PEPs repo) There have been several significant changes to the main metadata spec in PEP 426: * Added a basic jsonschema based description of the format * The abbreviated metadata notion is gone, replaced by included documents. The metadata can now list the names of additional files included in the distinfo directory for the long description, the license and the change history. The contents are no longer embedded directly in the metadata (but will be extracted and made available by PyPI, with the markup format being determined from the file extension, just as it is by sites like GitHub) * The dependency system has been redesigned under the name Semantic dependencies (as they now allow a distribution to better say *why* a dependency is needed, rather than just saying I need this with almost no capacity to say why). There are now five kinds of dependencies (:meta:, :run:, :test:, :build: and :dev:) and they're integrated into the extras system so you can easily say you want to install some, none or all of them. * The * notation is added to extras to make it easier to say install all dependencies, along with the -extra_name notation to exclude the dependencies for specific extras * The - notation is added to extras to make it easier to install *just* a distribution's dependencies, without installing the distribution itself. * Install hooks are now a distinct concept from the still-hypothetical metabuild system, and place more constraints on their expected handling (installation tools are also explicitly permitted to refuse to run them, but doing so is required to fail noisily rather than silently appearing to succeed) The most significant change to PEP 440 is to the handling of pre-releases: whether or not pre-releases should be accepted by default is now determined solely by whether or not there is a stable release that *also* satisfies the version specifier. Reference a pre-release (or not) now has no effect on whether pre-releases are considered viable candidates. Pre-releases are now accepted if: * they're already installed * there's no other available option * the installation tool is told specifically to consider them Other less significant changes to PEP 426 include: * Longer introduction giving more context for the nature and purpose of the metadata * Separated various other things out into appendices * Various tweaks to definitions (including the Release tweak from PEP 440, and switching source archive to refer to the original raw source, while using sdist for the Python specific format with the extra metadata) * Blanket permission for distribution related online services to impose additional restrictions to protect the integrity of the service (such as additional length limits beyond those stated in the PEP). * Explicitly require UTF-8 encoded JSON for serialisation * build_label renamed to source_label * version_url renamed to source_url * tightened up the validation rules for various fields (including RFC references where appropriate). Many of these new rules are things projects already comply with (because not complying doesn't work properly). Including them in the spec is about giving PyPI clear guidance to enforce them at point of upload, rather than leaving it to installation tools to try to sort out later. * a few more additions to the Rejected Features appendix (notably, my rationale for *not* requiring the install hooks to accept arbitrary keyword arguments) The other PEP 440 changes are also relatively minor: - what were previously called releases are now final releases, freeing up release as a general term for any kind of release (developmental release, pre-release, final release, post release). - source references are now direct references and can also refer to prebuilt wheel files - automated tools (especially index servers) are given a lot of freedom to be opinionated about the appropriate uses for direct references - a few tweaks to the security guidelines for direct references - pytz/Olson database version translation is explicitly discussed (add a leading 0., convert the trailing letter to an incrementing number) - tighter rules for what characters are allowed in a source label The only major remaining open item is the addition of guidelines in Appendix A for converting legacy metadata to metadata 2.0. I consider the rest of the spec stable at this point, unless anyone picks up on a glaring hole in the latest draft that Daniel,
Re: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440)
On 20 June 2013 23:36, Daniel Holth dho...@gmail.com wrote: On Thu, Jun 20, 2013 at 9:07 AM, Nick Coghlan ncogh...@gmail.com wrote: The only major remaining open item is the addition of guidelines in Appendix A for converting legacy metadata to metadata 2.0. I consider the rest of the spec stable at this point, unless anyone picks up on a glaring hole in the latest draft that Daniel, Donald and I missed. Cheers, Nick. FYI the tip revision of wheel (bdist_wheel, https://bitbucket.org/dholth/wheel/) produces pymeta.json that is almost up-to-date with this version of the spec, except that it still inlines the description rather than saving it to a separate file. bdist_wheel has always worked by converting setuptools metadata to PEP metadata. Yeah, pointing at the relevant code in the wheel implementation as one approach that is known to work reasonably well may be the most expedient way of handling that. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440)
I still think the testing part of the interchange format between software publication and integration tools is underspecified. The dependencies alone will not be sufficient to allow the running of tests in many cases. Or am i missing something? best and thanks for your good work, holger On Thu, Jun 20, 2013 at 23:07 +1000, Nick Coghlan wrote: New versions of PEP 426 and PEP 440 are up on python.org: PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ PEP 440 (version spec): http://www.python.org/dev/peps/pep-0440/ (as before, not including them inline due to sheer length) The bulk of the changes are in this commit: http://hg.python.org/peps/rev/de305af601fa (there are a few minor tweaks in subsequent commits to the PEPs repo) There have been several significant changes to the main metadata spec in PEP 426: * Added a basic jsonschema based description of the format * The abbreviated metadata notion is gone, replaced by included documents. The metadata can now list the names of additional files included in the distinfo directory for the long description, the license and the change history. The contents are no longer embedded directly in the metadata (but will be extracted and made available by PyPI, with the markup format being determined from the file extension, just as it is by sites like GitHub) * The dependency system has been redesigned under the name Semantic dependencies (as they now allow a distribution to better say *why* a dependency is needed, rather than just saying I need this with almost no capacity to say why). There are now five kinds of dependencies (:meta:, :run:, :test:, :build: and :dev:) and they're integrated into the extras system so you can easily say you want to install some, none or all of them. * The * notation is added to extras to make it easier to say install all dependencies, along with the -extra_name notation to exclude the dependencies for specific extras * The - notation is added to extras to make it easier to install *just* a distribution's dependencies, without installing the distribution itself. * Install hooks are now a distinct concept from the still-hypothetical metabuild system, and place more constraints on their expected handling (installation tools are also explicitly permitted to refuse to run them, but doing so is required to fail noisily rather than silently appearing to succeed) The most significant change to PEP 440 is to the handling of pre-releases: whether or not pre-releases should be accepted by default is now determined solely by whether or not there is a stable release that *also* satisfies the version specifier. Reference a pre-release (or not) now has no effect on whether pre-releases are considered viable candidates. Pre-releases are now accepted if: * they're already installed * there's no other available option * the installation tool is told specifically to consider them Other less significant changes to PEP 426 include: * Longer introduction giving more context for the nature and purpose of the metadata * Separated various other things out into appendices * Various tweaks to definitions (including the Release tweak from PEP 440, and switching source archive to refer to the original raw source, while using sdist for the Python specific format with the extra metadata) * Blanket permission for distribution related online services to impose additional restrictions to protect the integrity of the service (such as additional length limits beyond those stated in the PEP). * Explicitly require UTF-8 encoded JSON for serialisation * build_label renamed to source_label * version_url renamed to source_url * tightened up the validation rules for various fields (including RFC references where appropriate). Many of these new rules are things projects already comply with (because not complying doesn't work properly). Including them in the spec is about giving PyPI clear guidance to enforce them at point of upload, rather than leaving it to installation tools to try to sort out later. * a few more additions to the Rejected Features appendix (notably, my rationale for *not* requiring the install hooks to accept arbitrary keyword arguments) The other PEP 440 changes are also relatively minor: - what were previously called releases are now final releases, freeing up release as a general term for any kind of release (developmental release, pre-release, final release, post release). - source references are now direct references and can also refer to prebuilt wheel files - automated tools (especially index servers) are given a lot of freedom to be opinionated about the appropriate uses for direct references - a few tweaks to the security guidelines for direct references - pytz/Olson database version translation is explicitly discussed (add a leading 0., convert the trailing letter to an incrementing number) - tighter rules for what characters are
Re: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440)
On 21 Jun 2013 00:36, holger krekel hol...@merlinux.eu wrote: I still think the testing part of the interchange format between software publication and integration tools is underspecified. The dependencies alone will not be sufficient to allow the running of tests in many cases. Or am i missing something? If setup.py test works for a distribution today, it will still work under PEP 426. Anything more is deliberately deferred (as noted in the PEP). Cheers, Nick. best and thanks for your good work, holger On Thu, Jun 20, 2013 at 23:07 +1000, Nick Coghlan wrote: New versions of PEP 426 and PEP 440 are up on python.org: PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ PEP 440 (version spec): http://www.python.org/dev/peps/pep-0440/ (as before, not including them inline due to sheer length) The bulk of the changes are in this commit: http://hg.python.org/peps/rev/de305af601fa (there are a few minor tweaks in subsequent commits to the PEPs repo) There have been several significant changes to the main metadata spec in PEP 426: * Added a basic jsonschema based description of the format * The abbreviated metadata notion is gone, replaced by included documents. The metadata can now list the names of additional files included in the distinfo directory for the long description, the license and the change history. The contents are no longer embedded directly in the metadata (but will be extracted and made available by PyPI, with the markup format being determined from the file extension, just as it is by sites like GitHub) * The dependency system has been redesigned under the name Semantic dependencies (as they now allow a distribution to better say *why* a dependency is needed, rather than just saying I need this with almost no capacity to say why). There are now five kinds of dependencies (:meta:, :run:, :test:, :build: and :dev:) and they're integrated into the extras system so you can easily say you want to install some, none or all of them. * The * notation is added to extras to make it easier to say install all dependencies, along with the -extra_name notation to exclude the dependencies for specific extras * The - notation is added to extras to make it easier to install *just* a distribution's dependencies, without installing the distribution itself. * Install hooks are now a distinct concept from the still-hypothetical metabuild system, and place more constraints on their expected handling (installation tools are also explicitly permitted to refuse to run them, but doing so is required to fail noisily rather than silently appearing to succeed) The most significant change to PEP 440 is to the handling of pre-releases: whether or not pre-releases should be accepted by default is now determined solely by whether or not there is a stable release that *also* satisfies the version specifier. Reference a pre-release (or not) now has no effect on whether pre-releases are considered viable candidates. Pre-releases are now accepted if: * they're already installed * there's no other available option * the installation tool is told specifically to consider them Other less significant changes to PEP 426 include: * Longer introduction giving more context for the nature and purpose of the metadata * Separated various other things out into appendices * Various tweaks to definitions (including the Release tweak from PEP 440, and switching source archive to refer to the original raw source, while using sdist for the Python specific format with the extra metadata) * Blanket permission for distribution related online services to impose additional restrictions to protect the integrity of the service (such as additional length limits beyond those stated in the PEP). * Explicitly require UTF-8 encoded JSON for serialisation * build_label renamed to source_label * version_url renamed to source_url * tightened up the validation rules for various fields (including RFC references where appropriate). Many of these new rules are things projects already comply with (because not complying doesn't work properly). Including them in the spec is about giving PyPI clear guidance to enforce them at point of upload, rather than leaving it to installation tools to try to sort out later. * a few more additions to the Rejected Features appendix (notably, my rationale for *not* requiring the install hooks to accept arbitrary keyword arguments) The other PEP 440 changes are also relatively minor: - what were previously called releases are now final releases, freeing up release as a general term for any kind of release (developmental release, pre-release, final release, post release). - source references are now direct references and can also refer to prebuilt wheel files - automated tools (especially index servers) are given a lot of freedom to be
Re: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440)
On Thu, Jun 20, 2013 at 4:19 PM, holger krekel hol...@merlinux.eu wrote: On Fri, Jun 21, 2013 at 01:05 +1000, Nick Coghlan wrote: On 21 Jun 2013 00:36, holger krekel hol...@merlinux.eu wrote: I still think the testing part of the interchange format between software publication and integration tools is underspecified. The dependencies alone will not be sufficient to allow the running of tests in many cases. Or am i missing something? If setup.py test works for a distribution today, it will still work under PEP 426. Anything more is deliberately deferred (as noted in the PEP). I am a little confused. Early in the PEP you say: The published metadata for distributions SHOULD allow integrators, with the aid of build and integration tools, to: ... * if supported by the distribution, run the distribution's automatic test suite on an installed instance of the distribution but a few sentences later: The current iteration of the metadata relies on the distutils commands system to support other necessary integration and deployment activities: ... python setup.py test: run the distribution's test suite on a built (but not yet installed) distribution And i am not sure what exactly PEP426 defines wrt to testing now. I saw the test_requires and test_may_requires fields and thought they were part of an idea already how to configure running of tests. But is there anything more in the current PEP426 that i missed? To help with my confusion, could you maybe describe how python setup.py test is to interoperate (if at all) with the new metadata format as is? thanks, holger python setup.py test does not change at all as a result of this work. The only interoperability that will happen is that an external test runner will better be able to ask an installer to install those dependencies before attempting to run the test, since installs-as-side-effects-of-setup.py are not so great. Later setup.py will be replaceable, and /that/ specification will define a standard entry point for testing. These metadata PEPs are mostly about the installer, database of installed files, and package indexes. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440)
On 20 June 2013 21:54, Daniel Holth dho...@gmail.com wrote: These metadata PEPs are mostly about the installer, database of installed files, and package indexes. In essence (and I freely admit I don't fully understand the details yet myself) I think the idea is that PEPs 426 and 440 define a set of data that can be published by packages. The *how* of publishing that data is still to be defined - the idea is that there will be the appropriate metadata JSON files available, but no tools currently produce that format. Equally, installers like pip will be able to read the metadata and ensure that dependencies are installed appropriately (for example) but again, that will be a future change to pip. For now, the PEPs define a standard for tools to *use*, but leave the creation of such tools (or modification of existing tools) as a later step. I don't think the PEPs say anything about a plan for implementing the spec (although I haven't read the PEPs in detail yet, so I may have missed something) but my assumption is that it'll go something like this: 1. Python setup.py sdist and python setup.py dist_info will be changed to generate pymeta.json files. But that will be for Python 3.4 only (there's a big problem if this doesn't make it into 3.4...). Unless there's a distutils backport or similar. 2. Pypi will be changed to publish metadata from pymeta.json, when it is present, which tools like pip can consume (see later). 3. Wheel will put pymeta.json into wheels, from the pymeta.json created by (1) and likely by converting legacy egg-info metadata as a fallback. Wheel will also write pymeta.json on install. 4. Pip will use PE 426 metadata from PyPI, write pymeta.json on install, and use installed pymeta.json (again probably falling back to legacy formats where pymeta.json isn't present). 5. Distlib will follow suit. The big gap here is older versions of Python. Maybe a setuptools plugin to generate the new metadata in sdists would be viable. Maybe package writers could provide pymeta.json by hand, although I seriously doubt many will bother. How close am I to reality here? Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440)
On Jun 20, 2013, at 5:40 PM, Paul Moore p.f.mo...@gmail.com wrote: On 20 June 2013 21:54, Daniel Holth dho...@gmail.com wrote: These metadata PEPs are mostly about the installer, database of installed files, and package indexes. In essence (and I freely admit I don't fully understand the details yet myself) I think the idea is that PEPs 426 and 440 define a set of data that can be published by packages. The *how* of publishing that data is still to be defined - the idea is that there will be the appropriate metadata JSON files available, but no tools currently produce that format. Equally, installers like pip will be able to read the metadata and ensure that dependencies are installed appropriately (for example) but again, that will be a future change to pip. For now, the PEPs define a standard for tools to *use*, but leave the creation of such tools (or modification of existing tools) as a later step. I don't think the PEPs say anything about a plan for implementing the spec (although I haven't read the PEPs in detail yet, so I may have missed something) but my assumption is that it'll go something like this: 1. Python setup.py sdist and python setup.py dist_info will be changed to generate pymeta.json files. But that will be for Python 3.4 only (there's a big problem if this doesn't make it into 3.4...). Unless there's a distutils backport or similar. sdist 2.0 is not part of the PEP afaik. There will be a separate PEP for that. Wheel's will be the only thing generating these files to begin with. 2. Pypi will be changed to publish metadata from pymeta.json, when it is present, which tools like pip can consume (see later). 3. Wheel will put pymeta.json into wheels, from the pymeta.json created by (1) and likely by converting legacy egg-info metadata as a fallback. Wheel will also write pymeta.json on install. 4. Pip will use PE 426 metadata from PyPI, write pymeta.json on install, and use installed pymeta.json (again probably falling back to legacy formats where pymeta.json isn't present). 5. Distlib will follow suit. The big gap here is older versions of Python. Maybe a setuptools plugin to generate the new metadata in sdists would be viable. Maybe package writers could provide pymeta.json by hand, although I seriously doubt many will bother. How close am I to reality here? Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig Basically the plan with PEP426 is to come up with the installer side of things. There's places that things like building (going from sdist to wheel) or development (going from checkout to sdist) can be incorporated. And some thoughts have been made for it but afaik the primary focus has been on making this possible with wheels while leaving us a path to doing the same with sdists. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440)
On 21 Jun 2013 06:19, holger krekel hol...@merlinux.eu wrote: On Fri, Jun 21, 2013 at 01:05 +1000, Nick Coghlan wrote: On 21 Jun 2013 00:36, holger krekel hol...@merlinux.eu wrote: I still think the testing part of the interchange format between software publication and integration tools is underspecified. The dependencies alone will not be sufficient to allow the running of tests in many cases. Or am i missing something? If setup.py test works for a distribution today, it will still work under PEP 426. Anything more is deliberately deferred (as noted in the PEP). I am a little confused. Early in the PEP you say: The published metadata for distributions SHOULD allow integrators, with the aid of build and integration tools, to: ... * if supported by the distribution, run the distribution's automatic test suite on an installed instance of the distribution It's just an editing error - this was explicitly rejected (and is listed as a rejected feature), I just missed one of the references that was in the previous draft. Cheers, Nick. but a few sentences later: The current iteration of the metadata relies on the distutils commands system to support other necessary integration and deployment activities: ... python setup.py test: run the distribution's test suite on a built (but not yet installed) distribution And i am not sure what exactly PEP426 defines wrt to testing now. I saw the test_requires and test_may_requires fields and thought they were part of an idea already how to configure running of tests. But is there anything more in the current PEP426 that i missed? To help with my confusion, could you maybe describe how python setup.py test is to interoperate (if at all) with the new metadata format as is? thanks, holger Cheers, Nick. best and thanks for your good work, holger On Thu, Jun 20, 2013 at 23:07 +1000, Nick Coghlan wrote: New versions of PEP 426 and PEP 440 are up on python.org: PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ PEP 440 (version spec): http://www.python.org/dev/peps/pep-0440/ (as before, not including them inline due to sheer length) The bulk of the changes are in this commit: http://hg.python.org/peps/rev/de305af601fa (there are a few minor tweaks in subsequent commits to the PEPs repo) There have been several significant changes to the main metadata spec in PEP 426: * Added a basic jsonschema based description of the format * The abbreviated metadata notion is gone, replaced by included documents. The metadata can now list the names of additional files included in the distinfo directory for the long description, the license and the change history. The contents are no longer embedded directly in the metadata (but will be extracted and made available by PyPI, with the markup format being determined from the file extension, just as it is by sites like GitHub) * The dependency system has been redesigned under the name Semantic dependencies (as they now allow a distribution to better say *why* a dependency is needed, rather than just saying I need this with almost no capacity to say why). There are now five kinds of dependencies (:meta:, :run:, :test:, :build: and :dev:) and they're integrated into the extras system so you can easily say you want to install some, none or all of them. * The * notation is added to extras to make it easier to say install all dependencies, along with the -extra_name notation to exclude the dependencies for specific extras * The - notation is added to extras to make it easier to install *just* a distribution's dependencies, without installing the distribution itself. * Install hooks are now a distinct concept from the still-hypothetical metabuild system, and place more constraints on their expected handling (installation tools are also explicitly permitted to refuse to run them, but doing so is required to fail noisily rather than silently appearing to succeed) The most significant change to PEP 440 is to the handling of pre-releases: whether or not pre-releases should be accepted by default is now determined solely by whether or not there is a stable release that *also* satisfies the version specifier. Reference a pre-release (or not) now has no effect on whether pre-releases are considered viable candidates. Pre-releases are now accepted if: * they're already installed * there's no other available option * the installation tool is told specifically to consider them Other less significant changes to PEP 426 include: * Longer introduction giving more context for the nature and purpose of the metadata * Separated various other things out into appendices * Various tweaks to definitions
Re: [Distutils] Updated drafts of metadata 2.0 (PEP 426 and PEP 440)
On 21 Jun 2013 07:40, Paul Moore p.f.mo...@gmail.com wrote: On 20 June 2013 21:54, Daniel Holth dho...@gmail.com wrote: These metadata PEPs are mostly about the installer, database of installed files, and package indexes. In essence (and I freely admit I don't fully understand the details yet myself) I think the idea is that PEPs 426 and 440 define a set of data that can be published by packages. The *how* of publishing that data is still to be defined - the idea is that there will be the appropriate metadata JSON files available, but no tools currently produce that format. Equally, installers like pip will be able to read the metadata and ensure that dependencies are installed appropriately (for example) but again, that will be a future change to pip. For now, the PEPs define a standard for tools to *use*, but leave the creation of such tools (or modification of existing tools) as a later step. I don't think the PEPs say anything about a plan for implementing the spec (although I haven't read the PEPs in detail yet, so I may have missed something) Right, the general implementation plan isn't part of the PEP, aside from the big note at the beginning about all the *other* PEPs that will be needed to finish bootstrapping this thing. but my assumption is that it'll go something like this: 1. Python setup.py sdist and python setup.py dist_info will be changed to generate pymeta.json files. But that will be for Python 3.4 only (there's a big problem if this doesn't make it into 3.4...). Unless there's a distutils backport or similar. The initial vehicles for migration will be Daniel's bdist_wheel command and upgrades to setuptools. Standard library support may not land for 3.4, but we will at least have the pip bootstrapping mechanism in place (which in turn will allow retrieval of setuptools). Until the updated packaging ecosystem stabilises, adopting the standard library's update rate would be a bad idea. Besides, a solution that doesn't work back as far Python 2.6 faces significant barriers to community adoption. 2. Pypi will be changed to publish metadata from pymeta.json, when it is present, which tools like pip can consume (see later). /me lets cat out of bag Donald's actually working on a second generation PyPI design that uses PEP 426 as its underlying data model for distributions. It will probably include the capability to derive 2.0 metadata from legacy metadata, so installers can benefit from the new mechanisms at install time, even for existing distributions. Many of the security constraints and backwards compatibility tweaks in the PEP actually come from that design work. With the spec now stabilised, we'll likely flip the public bit on that repo soon, and start working on some concrete plans to spin up a test instance alongside the existing PyPI service :) Along with Daniel ensuring that bdist_wheel can emit the format, Donald's work on the server side has been a key part of keeping the metadata spec grounded in reality. Once we flip the public bit on the repo, the next generation PyPI will serve alongside Daniel's existing wheel project as the reference implementations for the spec. 3. Wheel will put pymeta.json into wheels, from the pymeta.json created by (1) and likely by converting legacy egg-info metadata as a fallback. Wheel will also write pymeta.json on install. On install, it should be covered by the existing copy the dist-info contents from the wheel behaviour, but that's the general idea. sdist 2.0 will just deprecate PKG-INFO and replace it with a self identifying wheel-style dist-info subdirectory. 4. Pip will use PE 426 metadata from PyPI, write pymeta.json on install, and use installed pymeta.json (again probably falling back to legacy formats where pymeta.json isn't present). 5. Distlib will follow suit. Yup (although *definitely* falling back to legacy formats). The big gap here is older versions of Python. Maybe a setuptools plugin to generate the new metadata in sdists would be viable. Maybe package writers could provide pymeta.json by hand, although I seriously doubt many will bother. How close am I to reality here? Close, except that this is explicitly an ecosystem first plan, with the standard library playing catch up later. That's the big reason I sought (and received) agreement at this year's language summit for us to discuss and approve packaging related PEPs right here on distutils-sig rather than having to repost them to python-dev for ratification. The only must have on the list for 3.4 is the pip bootstrapping, and Richard and I hope to get that squared away at the PyCon AU sprints next month. Cheers, Nick. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
