[Distutils] PEP449 - Removal of the PyPI Mirror Auto Discovery and Naming Scheme
Ok round two of this PEP. I've made a number of modifications to it: * Changed the title to be more relevant to the actual proposal. * Cleared up language to make it obvious that this was explicitly the naming and auto discovery being removed and not the mirrors in general. * Removed pointless distinctions between "Official" and "Unofficial" mirrors * Removed comments about staleness and points about the CDN which were not directly related to this PEP as the PEP doesn't attempt to solve issues with the mirroring protocol, just the discovery and naming scheme. * Incorporate feedback from the thread and simplify the proposal as well as extend the time frame. * Respond to some of the feedback from the thread that wasn't incorporated. I've explicitly CC'd Christian to this email so he can hopefully see it easier. Abstract This PEP provides a path to deprecate and ultimately remove the auto discovery of PyPI mirrors as well as the hard coded naming scheme which requires delegating a domain name under pypi.python.org to a third party. Rationale = The PyPI mirroring infrastructure (defined in `PEP381`_) provides a means to mirror the content of PyPI used by the automatic installers. It also provides a method for auto discovery of mirrors and a consistent naming scheme. There are a number of problems with the auto discovery protocol and the naming scheme: * They give control over a \*.python.org domain name to a third party, allowing that third party to set or read cookies on the pypi.python.org and python.org domain name. * The use of a sub domain of pypi.python.org means that the mirror operators will never be able to get a SSL certificate of their own, and giving them one for a python.org domain name is unlikely to happen. * The auto discovery uses an unauthenticated protocol (DNS). * The lack of a TLS certificate on these domains means that clients can not be sure that they have not been a victim of DNS poisoning or a MITM attack. * The auto discovery protocol was designed to enable a client to automatically select a mirror for use. This is no longer a requirement because the CDN that PyPI is now using a globally distributed network of servers which will automatically select one close to the client without any effort on the clients part. * The auto discovery protocol and use of the consistent naming scheme has only ever been implemented by one installer (pip), and its implementation, besides being insecure, has serious issues with performance and is slated for removal with it's next release (1.5). * While there are provisions in `PEP381`_ that would solve *some* of these issues for a dedicated client it would not solve the issues that affect a users browser. Additionally these provisions have not been implemented by any installer to date. Due to the number of issues, some of them very serious, and the CDN which provides most of the benefit of the auto discovery and consistent naming scheme this PEP proposes to first deprecate and then remove the [a..z].pypi.python.org names for mirrors and the last.pypi.python.org name for the auto discovery protocol. The ability to mirror and the method of mirror will not be affected and will continue to exist as written in `PEP381`_. Operators of existing mirrors are encouraged to acquire their own domains and certificates to use for their mirrors if they wish to continue hosting them. Plan for Deprecation & Removal == Immediately upon acceptance of this PEP documentation on PyPI will be updated to reflect the deprecated nature of the official public mirrors and will direct users to external resources like http://www.pypi-mirrors.org/ to discover unofficial public mirrors if they wish to use one. Mirror operators, if they wish to continue operating their mirror, should acquire a domain name to represent their mirror and, if they are able, a TLS certificate. Once they have acquired a domain they should redirect their assigned N.pypi.python.org domain name to their new domain. On Feb 15th, 2014 the DNS entries for [a..z].pypi.python.org and last.pypi.python.org will be removed. At any time prior to Feb 15th, 2014 a mirror operator may request that their domain name be reclaimed by PyPI and pointed back at the master. Why Feb 15th, 2014 -- The most critical decision of this PEP is the final cut off date. If the date is too soon then it needlessly punishes people by forcing them to drop everything to update their deployment scripts. If the date is too far away then the extended period of time does not help with the migration effort and merely puts off the migration until a later date. The date of Feb 15th, 2014 has been chosen because it is roughly 6 months from the date of the PEP. This should ensure a lengthy period of time to enable people to update their deployment procedures to point to the new domains names without merely padding the cut off date.
Re: [Distutils] Last PEP 426 update for a while
On 10 August 2013 13:12, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: > >> 6. Qualified names are currently restricted to Python 2 compatible >> identifiers. > > This constraint causes some problems in practice, some of which might be > more problematic than others: > > 1. Some existing distributions have group names which violate the constraint. >For example, nose uses group names such as "nose.plugins.0.10". Around a >hundred other distributions advertise plugins under group names like this. > > 2. Some distributions have package names which aren't actual packages but >which nevertheless contain source files. For example, Django lists >"django.utils.2to3_fixers" in its list of packages, though this is just >a directory containing fixers and not an actual package. This, of course, >also violates the constraint. > > It seems like it might be worth relaxing the restriction so that a qualified > name starts with a Python 2 compatible segment, but can have subsequent > components matching the [A-Za-z0-9]+ pattern. Done. Cheers, Nick. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Last PEP 426 update for a while
Nick Coghlan gmail.com> writes: > 6. Qualified names are currently restricted to Python 2 compatible > identifiers. This constraint causes some problems in practice, some of which might be more problematic than others: 1. Some existing distributions have group names which violate the constraint. For example, nose uses group names such as "nose.plugins.0.10". Around a hundred other distributions advertise plugins under group names like this. 2. Some distributions have package names which aren't actual packages but which nevertheless contain source files. For example, Django lists "django.utils.2to3_fixers" in its list of packages, though this is just a directory containing fixers and not an actual package. This, of course, also violates the constraint. It seems like it might be worth relaxing the restriction so that a qualified name starts with a Python 2 compatible segment, but can have subsequent components matching the [A-Za-z0-9]+ pattern. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Unexpected VersionConflict
On Fri, Aug 9, 2013 at 5:41 PM, Nick Coghlan wrote: > On 10 August 2013 04:06, PJ Eby wrote: >> Probably a better way would be to change the version resolution >> algorithm to be less "greedy", and simply rule out unacceptable >> versions as the process goes along, then picking the most recent >> versions left when everything necessary has been eliminated. >> (Ideally, such an algorithm would still track which distributions had >> the conflicting requirements, though.) > > The part I find most surprising is the fact that pkg_resources ignores > sys.path order entirely when choosing between multiple acceptable > versions. Technically, it doesn't ignore it: if a distribution is listed in sys.path, it takes precedence over any distribution listed later, or that has to be found *in* a directory on sys.path, and will in fact cause a VersionConflict if you ask for a version spec that it doesn't match. However, where the distributions aren't listed in sys.path, but merely *found in a directory on sys.path*, then sys.path has no bearing. It would make things a lot more complicated, and not just in an "implementation is hard to explain" kind of way. (In principle, you could write an Environment subclass that had a different precedence, but I'm not sure what benefit you would gain from the added complexity. The core version resolution algorithm wouldn't be affected, though, since it delegates the "find me something I haven't already got on sys.path" operation to an Environment instance's best_match() method.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Unexpected VersionConflict
That does indeed fix this problem, but requiring an egg writer to
interrogate all dependent packages (and their dependent packagesŠ) and
then hoist the dependencies up won't be robust if those dependent packages
change their requirements between the time the egg is written and the time
it's loaded.
It seems to me that if a requirement has no version specified, then it
shouldn't have a way to cause a VersionConflict. One possible way of
implementing this would be to have resolve() only check that a
distribution exists if no version is specified, do not update 'best'.
'to_activate' would need to be updated with 'generic' distributions only
if a requirement with a version specifier hadn't been seen.
On 8/8/13 7:44 PM, "PJ Eby" wrote:
>On Thu, Aug 8, 2013 at 3:19 PM, Townsend, Scott E. (GRC-RTM0)[Vantage
>Partners, LLC] wrote:
>> During easy_install of an egg where two versions of pyparsing were
>>available
>> (1.5.2 and 1.5.6), a VersionConflict was raised:
>>
>> pkg_resources.VersionConflict: (pyparsing 1.5.6
>> (/usr/lib/python2.7/dist-packages),
>>Requirement.parse('pyparsing==1.5.2'))
>>
>> This was unexpected since sys.path (via virtualenv) has version 1.5.2
>>before
>> 1.5.6. And the system gets 1.5.2 from 'import pyparsing', not 1.5.6.
>
>Have you tried declaring the 1.5.2 dependency from your main project?
>IIRC, that should make it take precedence over either of the indirect
>dependencies.
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
