I said that before we talked to a lawyer On Mar 10, 2013, at 3:54 PM, holger krekel <[email protected]> wrote:
> On Sun, Mar 10, 2013 at 14:29 -0400, Donald Stufft wrote: >> >> On Mar 10, 2013, at 2:18 PM, holger krekel <[email protected]> wrote: >> >>> On Sun, Mar 10, 2013 at 13:35 -0400, Donald Stufft wrote: >>>> On Mar 10, 2013, at 11:07 AM, holger krekel <[email protected]> wrote: >>>>> [...] >>>>> Transitioning to "pypi-cache" mode >>>>> ------------------------------------- >>>>> >>>>> When transitioning from the currently implicit "pypi-ext" mode to >>>>> "pypi-cache" for a given package, a package maintainer should >>>>> be able to retrieve/verify the historic release files which will >>>>> be cached from pypi.python.org. The UI should present this list >>>>> and have the maintainer accept it for completing the transition >>>>> to the "pypi-cache" mode. Upon future release registration actions, >>>>> pypi.python.org will perform crawling for the homepage/download sites >>>>> and cache release files *before* returning a success return code for >>>>> the release registration. >>>>> [...] >>>> >>>> Some concerns: >>>> >>>> 1. We cannot automatically switch people to pypi-cache. We _have_ to get >>>> explicit permission from them. >>> >>> Could you detail how you arrive at this conclusion? >>> (I've seen the claim before but not the underlying reasoning, maybe >>> i just missed it) >>> >>> There would be prior notifications to the package maintainers. If they >>> don't want to have their packages cached at pypi.python.org, they can set >>> the mode to "pypi-only" and leave manual instructions. I suspect there will >>> be very few people if anyone, objecting to pypi-cache mode. If that is >>> false we might need to prolong pypi-ext mode some more for them and >>> eventually switch them to pypi-only when we eventually decide to get >>> rid of external hosting. >> >> I asked VanL. His statement on re-hosting packages was: >> >> "We could do it if we had permission. The tricky part would be getting >> permission for already-existing packages." >> >> I'm pretty sure that emailing someone and assuming we have permission if >> they don't opt-out doesn't count as permission. > > Hum, i I saw Jesse Noller saying a few days ago "let them opt out". > But i guess VanL can trump that :) If that is true we could change the > notification to maintainers of B packages that hosting mode is going to > change to pypi-only, which would loose their release files unless they > opt-in to pypi-cache. As long as that is a no-brainer for them, we are > not asking for much and can count on most people's good will to not make > other people's installation life harder. > > Besides, admins could still set the "pypi-ext" mode if a maintainer can > explain why it's a problem for them to agree to "pypi-cache" or > "pypi-only". I'd really like to not have too many packages lingering > around in "pypi-ext" mode if it can be avoided. > >>> >>>> 2. The cache mechanism is going to be fragile, and in the long term leaves >>>> a window open for security issues. >>> >>> fragility: not sure it's too bad. Once the mode is activited release >>> registration ("submit" POST action on "/pypi" http endpoint) will only >>> succeed if according releases can be found through homepage/download. >>> Changing the mode to pypi-cache in the presence of historic release >>> files hosted elsewhere needs a good pypi.python.org UI interaction and >>> may take several tries if neccessary sites cannot be reached. Nevertheless, >>> this step is potentially fragile [X]. >> >> I see, so pypi-cache would only be triggered once during release creation. >> Cache makes it sound like we'd continuously monitor the given external urls >> instead of it actually being a pull based method of getting files. > > Right, we need to avoid cache invalidation problems by only allowing > updates at user-chosen point in times (there might also be an explicit > "update cache" button in case a maintainer pushes a egg/wheel later). > It's still technically a cache i think but the term "rehost" would > work as well i guess. > >> [...] >>> Back to pypi-cache: it is there to make it super-easy for package >>> maintainers. There are all kinds of release habits and scripts >>> pushing out things to google/bitbucket/github/other sites. With >>> "pypi-cache" they don't need to change any of that. They just need >>> to be fine with pypi.python.org pulling in the packages for caching. >> >> Yes I understand the goal here. The problem is that there's not really >> a good way to secure this without requiring changes to their workflow. >> At best they'll have to push information about every file so that PyPI >> is able to verify the files it is downloading, and if we are requiring >> them to push data about those files we might as well require them to >> push the files themselves. > > Is this about protection against package tampering? If so, I think a > proper solution involves maintainers signing their release files but > this is outside the intended scope of the PEP. > > Otherwise, the "re-hosting" process for pypi-cache mode is at least as > secure as currently where all hosts issuing pip/easy_install commands > visit external sites and can thus be MITM-attacked. For pypi-only > server packages it's safer because no crawling takes place. > > In any case, asking people to change their release process is not > a no-brainer. The PEP tries to avoid this source of friction. > That being said, i think we both agree to recommend maintainers to > (eventually) go for pypi-only and change their release processes > accordingly. This PEP is not the end of the story of evolving package > hosting and i'd like to be careful about asking maintainers to change > how they do things. > >> This also has the effect we can provide >> immediate feedback when files do not validate on PyPI. > > At release registration or switch-to-pypi-rehost time we could also do > package validation but i am inclined to see this as out of scope > for this PEP which tries to focus on the minimal steps to move > from pypi-ext to everything-hosted-through-pypi.python.org. > > cheers, > holger > >> >>> >>> We might think about phasing out pypi-cache after some larger time >>> frame so that we eventually only have pypi-only and things are eventually >>> simple and saner. >>> >>> best, >>> holger >>> >>> >>> >>>> These buttons would be one time and quit. Once your project has been >>>> switched to PyPI Only you cannot go back to Legacy mode. All new projects >>>> would be already switched to PyPI Only. After some amount of time switch >>>> all Projects to PyPI Only but _do not_ re-host their packages as we cannot >>>> legally do so without their permission. >>>> >>>> The above is simpler, still provides people an easy migration path, moves >>>> us to remove external hosting, and doesn't entangle us with legal issues. >>>> >>>> [1] There is still a small window here where someone could MITM PyPI >>>> fetching these files, however since it would be a one time and down deal >>>> this risk is minimal and is worth it to move to an pypi only solution. >>>> >>>> ----------------- >>>> Donald Stufft >>>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 >>>> DCFA >> >> >> ----------------- >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Catalog-SIG mailing list > [email protected] > http://mail.python.org/mailman/listinfo/catalog-sig _______________________________________________ Catalog-SIG mailing list [email protected] http://mail.python.org/mailman/listinfo/catalog-sig
