Re: [Distutils] PEP449 - Removal of the PyPI Mirror Auto Discovery and Naming Scheme
On Aug 16, 2013, at 10:31 PM, Nick Coghlan wrote: > Heh, this got a "Usenet nod" from me. I wanted to mention this in a > talk tomorrow, so I posted the updated draft in the PEPs repo. > > Hopefully the February timeframe is generous enough for Christian to > either talk to the PSF about a cert for f.pypi.python.org, or else > migrate to a different domain name. I think the exact date is the main > open question now - I believe the PEP makes a solid case for why the > status quo can't continue indefinitely. > > Cheers, > Nick. I forgot that still needed updated, thanks! I know one person has said to me privately they plan on reviewing it still, but I haven't seen anything from Christian. Other then that I've not seen a lot of response so either people haven't noticed it or they were usenet nodding as well :D - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP449 - Removal of the PyPI Mirror Auto Discovery and Naming Scheme
Oh FWIW pip got a CVE assigned earlier today for --use-mirrors which is effectively an implementation of what this PEP is removing. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP449 - Removal of the PyPI Mirror Auto Discovery and Naming Scheme
Heh, this got a "Usenet nod" from me. I wanted to mention this in a talk tomorrow, so I posted the updated draft in the PEPs repo. Hopefully the February timeframe is generous enough for Christian to either talk to the PSF about a cert for f.pypi.python.org, or else migrate to a different domain name. I think the exact date is the main open question now - I believe the PEP makes a solid case for why the status quo can't continue indefinitely. Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Changing the "install hooks" mechanism for PEP 426
PJ Eby telecommunity.com> writes:
> I think you're thinking I'm describing a single level namespace; I'm
> referring to something like this:
>
> {group1: {anykey: export_or_mapping}}
I got that :-)
> "anykey" is not validated by the spec, only by registered validators
> for "group1". Of course it has to have some meaning that is
> interpretable by consumers of group1. The point is that the *spec*
> only defines the syntax of group names and export strings, and it's
> left to specific groups to define the syntax/semantics of the keys.
Ok, I get what you mean now. But an appropriate validator would be one
defined by the definer of of the group, right? That code may not be present
on the system. Any validators registered by third parties could disagree;
what then? It may be better to say that either a consumer ask for an entry
against a specific key which it understands, or iterate over all keys and
ignore those they don't recognise the meaning of.
> I gave one example already: i18n/l10n information that's about files
> contained in the distribution's data. It's quite possible to have
> distributions without any code, only data of that kind. A requirement
> to create code, just to specify the data seems rather pointless. In
> Nick's reply, he's listed other use cases.
Right, but your case here and the cases Nick mention belong, I think,
outside what I understand the scope of PEP 426 to be. That is to say,
metadata which describes elements of an installed distribution and
dependency information used when installing or uninstalling it. I don't see
any reason why we can't have an "extensions" subdict which is free-format in
the PEP 426 dict for holding other information not described further in the
spec, but that's just as a grab-bag for ancillary data.
I've probably been thinking at cross purposes when discussing the term
"extension". It's a somewhat overloaded term, what with C extensions and all :-)
> After this further discussion, I think that the use cases we're
> discussing really boil back down to exports vs. metadata extensions,
> and that maybe we should stick to them being separate.
I'm OK with that, but there is a lot of packaging metadata which properly
lives outside PEP 426, and which one wouldn't want to see ending up in the
metadata extensions just because they're there. For example, the information
conventionally passed in setuptools.setup(package_data=...), which is often
used to specify i18n/l10n data.
Regards,
Vinay Sajip
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Changing the "install hooks" mechanism for PEP 426
On Fri, Aug 16, 2013 at 8:04 AM, Nick Coghlan wrote: > Concrete extension use cases I have in mind that don't fit in the > exports/entry-point data model: > > - the mapping of prebuilt executable names to wheel contents > - platform specific external dependencies and other hints for conversion to > platform specific formats (e.g. Windows GUIDs) > - metadata for build purposes (e.g. the working directory to use when > building from a VCS checkout, so you can have multiple projects in the same > repo) > - project specific metadata (e.g. who actually did the release) > - security metadata (e.g. security contact address and email GPG > fingerprint) > > This is why extensions/exports were originally separate, and may still > remain that way. But exports should still be used for hooks defined by the spec; they are the Obvious Way to specify importable hooks, and *not* dogfooding there is still a bad idea. (To be clear, I was never exactly *enthused* about the idea of merging extensions and exports, just *unenthused* about the idea of the spec using extensions or custom metadata to do things that could be perfectly well expressed as exports from a reserved namespace.) I'm kind of toying with the idea that the core metadata itself could be carved into extension namespaces, have the core itself be just another extension, rather than nesting extensions and exports inside the core, so that the entire spec is just a relatively-flat collection of namespaces, in more human-digestible groups. There are some conceptual and practical advantages to that, at least in principle, but until I've played around with actually expressing some concepts from the PEP that way, I won't know whether it would actually pay off in practice. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Changing the "install hooks" mechanism for PEP 426
On Fri, Aug 16, 2013 at 6:21 AM, Vinay Sajip wrote:
> PJ Eby telecommunity.com> writes:
>
>> I guess I didn't explain it very well, because that's roughly what I
>> meant: a single namespace for all "extensions", structured as a
>> mapping from group names to submappings whose keys can be arbitrary
>> values, but whose values must then be either a string or a JSON
>> object, and if it's a string, then it should be an export specifier.
>
> Why should the keys be completely arbitrary?
By "arbitrary" I mean only that the PEP doesn't place syntactical
restrictions on them.
> I can't see a reason for this;
> the current constraint of the "prefixed name" seems sufficient. What would
> relaxing this constraint make possible which otherwise isn't?
I think you're thinking I'm describing a single level namespace; I'm
referring to something like this:
{group1: {anykey: export_or_mapping}}
"anykey" is not validated by the spec, only by registered validators
for "group1". Of course it has to have some meaning that is
interpretable by consumers of group1. The point is that the *spec*
only defines the syntax of group names and export strings, and it's
left to specific groups to define the syntax/semantics of the keys.
> On the values: an export specifier is just a more human-friendly version of
> a dict with module/content/extra keys. While of course the uses of
> importables in this area is well established, what specific use cases are we
> enabling by allowing arbitrary JSON? It certainly would clutter the metadata
> and render it less human-readable, and the only thing it provides is a dict
> which could be expressed in an importable form
I gave one example already: i18n/l10n information that's about files
contained in the distribution's data. It's quite possible to have
distributions without any code, only data of that kind. A requirement
to create code, just to specify the data seems rather pointless. In
Nick's reply, he's listed other use cases.
The main question is, should exports and extensions be treated
separately? Nick originally proposed merging the concepts and using
arbitrary JSON. My counterproposal was to say, let's distinguish
exports and extensions by restricting the spec to something which
spells out the distinction.
After this further discussion, I think that the use cases we're
discussing really boil back down to exports vs. metadata extensions,
and that maybe we should stick to them being separate.
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Changing the "install hooks" mechanism for PEP 426
Concrete extension use cases I have in mind that don't fit in the exports/entry-point data model: - the mapping of prebuilt executable names to wheel contents - platform specific external dependencies and other hints for conversion to platform specific formats (e.g. Windows GUIDs) - metadata for build purposes (e.g. the working directory to use when building from a VCS checkout, so you can have multiple projects in the same repo) - project specific metadata (e.g. who actually did the release) - security metadata (e.g. security contact address and email GPG fingerprint) This is why extensions/exports were originally separate, and may still remain that way. Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Changing the "install hooks" mechanism for PEP 426
PJ Eby telecommunity.com> writes: > I guess I didn't explain it very well, because that's roughly what I > meant: a single namespace for all "extensions", structured as a > mapping from group names to submappings whose keys can be arbitrary > values, but whose values must then be either a string or a JSON > object, and if it's a string, then it should be an export specifier. Why should the keys be completely arbitrary? I can't see a reason for this; the current constraint of the "prefixed name" seems sufficient. What would relaxing this constraint make possible which otherwise isn't? On the values: an export specifier is just a more human-friendly version of a dict with module/content/extra keys. While of course the uses of importables in this area is well established, what specific use cases are we enabling by allowing arbitrary JSON? It certainly would clutter the metadata and render it less human-readable, and the only thing it provides is a dict which could be expressed in an importable form "mypackage.mymodule:path.to.a.dict". Of course this incurs an import penalty for access to the data, but if the publisher is concerned about this, they can minimise that overhead by arranging their package hierarchy appropriately. What kinds of use cases would require that the data be fully expressed in the metadata to avoid import overhead? > To put it another way, I'm saying something slightly stronger than a > recommended convention: making it a requirement that strings at that > level be import specifiers, and only allowing mappings as an > alternative. I'd like to understand the use cases which allowing mappings here would facilitate, that need to avoid importing to access the mapping. > If you already know what keys go in an entry point group, there's a > good chance you're doing it wrong. Normally, the whole point of the > group is that the keys are defined by the publisher, not the consumer. > The normal pattern is that the consumer names the group (representing > a hook), and the publishers name the extensions (representing > implementations for the hook). But the general form of the keys needs to be agreed to some extent between the consumer and publisher. Otherwise, the consumer doesn't know how to interpret the values of those keys. Of course a consumer can get all of the key/value entries exported by a dist or all dists for a specific group, but then what do they do with it if they don't know what the individual entries mean? > which means I'd rather not see the PEP make up its own data structures > when they're not actually needed. +1 - identification of the needs should come before specific proposals to address them. > Don't get me wrong, I'm okay with allowing JSON structures for > extensions in place of export strings, but I don't think there's been > a single use case proposed as yet that actually *works better* as a > data structure. The forms are equivalent, modulo an import penalty which only occurs for actual use and not for just scanning to see what's available. > Way to do it is something like a setuptools entry point -- i.e. a > basic key-value pair in a consumer-defined namespace, mapping a > semantically-valued name to an importable object. > > And *most* use cases for extensions, that I'm aware of, fit that bill. > You have to be doing something pretty complex to need anything more > complicated, *and* there has to be a possibility that you're going to > avoid importing the related code or putting in on sys.path, or else > you don't actually *save* anything by putting it in the metadata. > IOW, if you're going to have to import it anyway, there is no point to > putting it in the metadata; you might as well import it. Agreed, so I would say that we need to identify these use cases before saying that arbitrary mappings should be allowed as values. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
