On 12 May 2014 16:57, Stefan Krah <[email protected]> wrote: > Thank you for your measured responses, and I agree with you that pip should > follow PEP 438. The main argument on python-dev was about *editorializing* > the contents of the PEP in both pip warning messages and posts to the mailing > lists (and by that I certainly do *not* mean your posts).
My feeling is that there was not as much deliberate editorializing going on as may have seemed the case at first. More that people interpreted the situation and the intent of the PEP differently - there is a *lot* of confusion and misunderstanding in this whole area. Many people, notably myself, have a fairly shallow understanding of security issues, and find the more security-oriented explanations pretty confusing (and, unfortunately, somewhat paranoid - I say unfortunately because it's all too easy to misunderstand or dismiss a key point simply because of a difference in mindset). Also, I'd have to say that packaging PEPs have traditionally been pretty divergent from reality, so people could be excused for thinking "it couldn't possibly mean X". I've definitely been guilty of arguing based on what I *think* is going on rather than checking the PEP and the code. Come to that, I honestly don't know if anyone has checked the pip implementation against the PEP (I believe they match, that's the best I can say). And from what I recall of the PEP discussion, I'm not sure it reflected the sort of issues being raised now, so I suspect more than one person either didn't get involved in the discussion, or didn't spot issues that affected them (from what I recall, the PEP discussion felt more like a technical PyPI internals issue than a significant user interface debate). The initial implementation in pip didn't help the situation at all, as it described the situation very badly (one example of this is the warning that you mention). We're improving that right now, but I doubt the next iteration will be perfect, just (hopefully) better. > The PEP appears to say: > > "Installers should provide a blanket option to allow installing any verifiable > external link." > > Perhaps something like --allow-verifiable-external would do? I would not be > unhappy if link-spidering were to be removed, I find it reasonable to provide > the explicit link. I believe that option has been there for a while as --allow-[all]-external. Again, naming and discoverability may be an issue, but the functionality is available. One issue *may* be that it's not clear to everyone what "verifiable" involves. In another thread I'm trying to clarify with MAL over his understanding of how the egenix packages are linked. There is a chain of links with hashes, but the PEP doesn't allow for a chain to be "verifiable", just a direct link to a downloadable file. Is that what you mean by link-spidering? If so, then as it stands link-spidering is allowed, but will never be verifiable. I could easily imagine some package maintainers feeling that characterising a 2-link chain as "not verifiable" is inaccurate and/or scaremongering. Suggestions for better terminology would be useful here. (Note that direct links vs link chains is an important distinction for pip, as it has performance implications as well as security ones - link spidering was a huge performance issue at one point, and a lot of the non-controversial benefits in PEP 438 are in terms of performance. It would be a shame to lose those.) > But don't take that too seriously, some important looking packages rely on > link spidering: > > https://pypi.python.org/pypi/bzr/2.6.0 > https://pypi.python.org/pypi/meliae/0.4.0.final.0 > https://pypi.python.org/pypi/CobraWinLDTP/4.0.0 > https://pypi.python.org/pypi/PloneIISApp/4.2 > https://pypi.python.org/pypi/which/1.1.0 > https://pypi.python.org/pypi/python-cjson-custom/1.0.3x7 > https://pypi.python.org/pypi/CAGE/1.1.4 > https://pypi.python.org/pypi/libavg/1.8.0 > https://pypi.python.org/pypi/BlogBackup/1.4 > https://pypi.python.org/pypi/Polygon3/3.0.6 > https://pypi.python.org/pypi/Pygame/1.8.0 > https://pypi.python.org/pypi/DOLFIN/1.3.0 > https://pypi.python.org/pypi/snippet/2.3.6.1 > https://pypi.python.org/pypi/savReaderWriter/ > https://pypi.python.org/pypi/PyCY/1.0.0 > https://pypi.python.org/pypi/WikklyText/1.6.0 > https://pypi.python.org/pypi/python-lzo/1.08 > https://pypi.python.org/pypi/blockcanvas/4.0.3 > https://pypi.python.org/pypi/boolopt/1.0 > https://pypi.python.org/pypi/egenix-mx-base/3.1.0 > https://pypi.python.org/pypi/egenix-mx-commercial/2.0.7 > https://pypi.python.org/pypi/python-musicbrainz2/0.7.2 > https://pypi.python.org/pypi/OpenAlea/0.9 Many of those I've never heard of, which shows how hard it is to spot "important" stuff. Right now, all of these will need an --allow-unverifiable flag - and nobody (yet...) has suggested that this rule is weakened. The only thing that will change the user experience is if the projects add direct links to PyPI (at which point they'll work with --allow-external/--allow-all-external). I won't say anything about how current proposals will change that, as frankly I've lost track of what is on the table right now. > Donald: >> Stefan was using it but has since removed it because he was upset >> over a *warning*. > > Not quite the sequence of events. -- I left the existing explicit link > for some time after the first posts to python-dev. Then serious security > issues were marginalized ("not a meaningful scenario"). I find this a > little surprising, since PEP 458 is precisely there to address them. > > The user base that cdecimal targets (banks, stock exchanges, scientists) > are able to verify checksums -- in fact in some places it might be a > firing offense not to do so. Personally, I don't recall ever seeing anything about a serious security issue. But on the one hand, I came in late to the discussion, and on the other, I'm not sure I'd have understood a superficial explanation anyway. Revisiting the details of that issue may be worthwhile - but maybe when some of the current heat has died down a little... > Mocking that procedure (as has happened in certain channels) is not very > productive. Agreed, tempers have been fairly short. It can be pretty exhausting trying to keep things calm and reasonable. People need to let off steam, and sometimes do so inappropriately. But hopefully we are managing to achieve something, if not without upsets. I've been around for some of the previous packaging debates - I don't know if anyone else involved has those battle scars - and I can say that the tone of this debate, for all its faults, has been far better. > Before the thread on python-dev switched to external vs. internal, my > major point was that pip users might get the impression that internal > packages are safe and external packages are unsafe. Yeah, "safe" and "unsafe" are not great terms. "Verifiable" and "unverifiable" are a little better, but still have similar connotations (especially to the naive user). The term "external" is neutral, but gives a sense of "we only support PyPI" that is not true. I have no idea what would be good terms, unfortunately, and I suspect that everyone would have differing ideas :-( Thanks for your reply, I hope the above helps explain some things. Paul. _______________________________________________ Distutils-SIG maillist - [email protected] https://mail.python.org/mailman/listinfo/distutils-sig
